From 3ba59f7362deae722b1ef3705768c62b909839ed Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 9 Oct 2025 17:48:28 +0800 Subject: [PATCH 001/891] chore(deps): Bump serde_with from 3.14.0 to 3.14.1 (#1727) Bumps [serde_with](https://github.com/jonasbb/serde_with) from 3.14.0 to 3.14.1.
Release notes

Sourced from serde_with's releases.

serde_with v3.14.1

Fixed

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=serde_with&package-manager=cargo&previous-version=3.14.0&new-version=3.14.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index aa30a87e89..596e943bc2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6587,9 +6587,9 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.14.0" +version = "3.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2c45cd61fefa9db6f254525d46e392b852e0e61d9a1fd36e5bd183450a556d5" +checksum = "c522100790450cf78eeac1507263d0a350d4d5b30df0c8e1fe051a10c22b376e" dependencies = [ "base64 0.22.1", "chrono", @@ -6607,11 +6607,11 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.14.0" +version = "3.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de90945e6565ce0d9a25098082ed4ee4002e047cb59892c318d66821e14bb30f" +checksum = "327ada00f7d64abaac1e55a6911e90cf665aa051b9a561c7006c157f4633135e" dependencies = [ - "darling 0.20.11", + "darling 0.21.3", "proc-macro2", "quote", "syn 2.0.106", From 42191e97c4b661b5098233bf6c4015a51a31a3b6 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Thu, 9 Oct 2025 03:35:34 -0700 Subject: [PATCH 002/891] refactor(writer): Refactor writers for the future partitioning writers (#1657) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #1650 ## What changes are included in this PR? Refactored the writer layers; from a bird’s-eye view, the structure now looks like this: ```mermaid flowchart TD subgraph PartitioningWriter PW[PartitioningWriter] subgraph DataFileWriter RW[DataFileWriter] subgraph RollingWriter DFW[RollingWriter] subgraph FileWriter FW[FileWriter] end DFW --> FW end RW --> DFW end PW --> RW end ``` ### Key Changes - Modified `RollingFileWriter` to handle location generator, file name generator, and partition keys directly - Simplified `ParquetWriterBuilder` interface to accept output files during build - Restructured `DataFileWriterBuilder` to use `RollingFileWriter` with partition keys - Updated DataFusion integration to work with the new writer architecture - NOTE: Technically DataFusion or any engine should use `TaskWriter` -> `PartitioningWriter` -> `RollingWriter` -> ..., but `TaskWriter` and `PartitioningWriter` are not included in this draft so far ## Are these changes tested? Not yet, but changing the existing tests accordingly should be enough --- crates/iceberg/src/spec/manifest/data_file.rs | 4 +- crates/iceberg/src/spec/partition.rs | 24 ++ .../writer/base_writer/data_file_writer.rs | 153 +++++++---- .../base_writer/equality_delete_writer.rs | 148 ++++++----- crates/iceberg/src/writer/file_writer/mod.rs | 4 +- .../src/writer/file_writer/parquet_writer.rs | 245 +++++++----------- .../src/writer/file_writer/rolling_writer.rs | 212 +++++++++++---- crates/iceberg/src/writer/mod.rs | 31 ++- .../shared_tests/append_data_file_test.rs | 7 +- .../append_partition_data_file_test.rs | 76 +++--- .../shared_tests/conflict_commit_test.rs | 7 +- .../tests/shared_tests/scan_all_type.rs | 7 +- .../datafusion/src/physical_plan/write.rs | 27 +- 13 files changed, 570 insertions(+), 375 deletions(-) diff --git a/crates/iceberg/src/spec/manifest/data_file.rs b/crates/iceberg/src/spec/manifest/data_file.rs index 931f9441e9..d7455b56fe 100644 --- a/crates/iceberg/src/spec/manifest/data_file.rs +++ b/crates/iceberg/src/spec/manifest/data_file.rs @@ -26,7 +26,7 @@ use serde_with::{DeserializeFromStr, SerializeDisplay}; use super::_serde::DataFileSerde; use super::{Datum, FormatVersion, Schema, data_file_schema_v1, data_file_schema_v2}; use crate::error::Result; -use crate::spec::{Struct, StructType}; +use crate::spec::{DEFAULT_PARTITION_SPEC_ID, Struct, StructType}; use crate::{Error, ErrorKind}; /// Data file carries data file path, partition tuple, metrics, … @@ -49,6 +49,7 @@ pub struct DataFile { /// /// Partition data tuple, schema based on the partition spec output using /// partition field ids for the struct field ids + #[builder(default = "Struct::empty()")] pub(crate) partition: Struct, /// field id: 103 /// @@ -156,6 +157,7 @@ pub struct DataFile { pub(crate) first_row_id: Option, /// This field is not included in spec. It is just store in memory representation used /// in process. + #[builder(default = "DEFAULT_PARTITION_SPEC_ID")] pub(crate) partition_spec_id: i32, /// field id: 143 /// diff --git a/crates/iceberg/src/spec/partition.rs b/crates/iceberg/src/spec/partition.rs index fff9b62d78..128db338af 100644 --- a/crates/iceberg/src/spec/partition.rs +++ b/crates/iceberg/src/spec/partition.rs @@ -194,6 +194,15 @@ impl PartitionKey { Self { spec, schema, data } } + /// Creates a new partition key from another partition key, with a new data field. + pub fn copy_with_data(&self, data: Struct) -> Self { + Self { + spec: self.spec.clone(), + schema: self.schema.clone(), + data, + } + } + /// Generates a partition path based on the partition values. pub fn to_path(&self) -> String { self.spec.partition_to_path(&self.data, self.schema.clone()) @@ -207,6 +216,21 @@ impl PartitionKey { Some(pk) => pk.spec.is_unpartitioned(), } } + + /// Returns the associated [`PartitionSpec`]. + pub fn spec(&self) -> &PartitionSpec { + &self.spec + } + + /// Returns the associated [`SchemaRef`]. + pub fn schema(&self) -> &SchemaRef { + &self.schema + } + + /// Returns the associated [`Struct`]. + pub fn data(&self) -> &Struct { + &self.data + } } /// Reference to [`UnboundPartitionSpec`]. diff --git a/crates/iceberg/src/writer/base_writer/data_file_writer.rs b/crates/iceberg/src/writer/base_writer/data_file_writer.rs index f215673df1..a950547d33 100644 --- a/crates/iceberg/src/writer/base_writer/data_file_writer.rs +++ b/crates/iceberg/src/writer/base_writer/data_file_writer.rs @@ -18,86 +18,126 @@ //! This module provide `DataFileWriter`. use arrow_array::RecordBatch; -use itertools::Itertools; -use crate::Result; -use crate::spec::{DataContentType, DataFile, Struct}; -use crate::writer::file_writer::{FileWriter, FileWriterBuilder}; +use crate::spec::{DataContentType, DataFile, PartitionKey}; +use crate::writer::file_writer::FileWriterBuilder; +use crate::writer::file_writer::location_generator::{FileNameGenerator, LocationGenerator}; +use crate::writer::file_writer::rolling_writer::{RollingFileWriter, RollingFileWriterBuilder}; use crate::writer::{CurrentFileStatus, IcebergWriter, IcebergWriterBuilder}; +use crate::{Error, ErrorKind, Result}; /// Builder for `DataFileWriter`. #[derive(Clone, Debug)] -pub struct DataFileWriterBuilder { - inner: B, - partition_value: Option, - partition_spec_id: i32, +pub struct DataFileWriterBuilder { + inner: RollingFileWriterBuilder, + partition_key: Option, } -impl DataFileWriterBuilder { - /// Create a new `DataFileWriterBuilder` using a `FileWriterBuilder`. - pub fn new(inner: B, partition_value: Option, partition_spec_id: i32) -> Self { +impl DataFileWriterBuilder +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + /// Create a new `DataFileWriterBuilder` using a `RollingFileWriterBuilder`. + pub fn new( + inner_builder: RollingFileWriterBuilder, + partition_key: Option, + ) -> Self { Self { - inner, - partition_value, - partition_spec_id, + inner: inner_builder, + partition_key, } } } #[async_trait::async_trait] -impl IcebergWriterBuilder for DataFileWriterBuilder { - type R = DataFileWriter; +impl IcebergWriterBuilder for DataFileWriterBuilder +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + type R = DataFileWriter; async fn build(self) -> Result { Ok(DataFileWriter { - inner_writer: Some(self.inner.clone().build().await?), - partition_value: self.partition_value.unwrap_or(Struct::empty()), - partition_spec_id: self.partition_spec_id, + inner: Some(self.inner.clone().build()), + partition_key: self.partition_key, }) } } /// A writer write data is within one spec/partition. #[derive(Debug)] -pub struct DataFileWriter { - inner_writer: Option, - partition_value: Struct, - partition_spec_id: i32, +pub struct DataFileWriter { + inner: Option>, + partition_key: Option, } #[async_trait::async_trait] -impl IcebergWriter for DataFileWriter { +impl IcebergWriter for DataFileWriter +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ async fn write(&mut self, batch: RecordBatch) -> Result<()> { - self.inner_writer.as_mut().unwrap().write(&batch).await + if let Some(writer) = self.inner.as_mut() { + writer.write(&self.partition_key, &batch).await + } else { + Err(Error::new( + ErrorKind::Unexpected, + "Writer is not initialized!", + )) + } } async fn close(&mut self) -> Result> { - let writer = self.inner_writer.take().unwrap(); - Ok(writer - .close() - .await? - .into_iter() - .map(|mut res| { - res.content(DataContentType::Data); - res.partition(self.partition_value.clone()); - res.partition_spec_id(self.partition_spec_id); - res.build().expect("Guaranteed to be valid") - }) - .collect_vec()) + if let Some(writer) = self.inner.take() { + writer + .close() + .await? + .into_iter() + .map(|mut res| { + res.content(DataContentType::Data); + if let Some(pk) = self.partition_key.as_ref() { + res.partition(pk.data().clone()); + res.partition_spec_id(pk.spec().spec_id()); + } + res.build().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to build data file: {}", e), + ) + }) + }) + .collect() + } else { + Err(Error::new( + ErrorKind::Unexpected, + "Data file writer has been closed.", + )) + } } } -impl CurrentFileStatus for DataFileWriter { +impl CurrentFileStatus for DataFileWriter +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ fn current_file_path(&self) -> String { - self.inner_writer.as_ref().unwrap().current_file_path() + self.inner.as_ref().unwrap().current_file_path() } fn current_row_num(&self) -> usize { - self.inner_writer.as_ref().unwrap().current_row_num() + self.inner.as_ref().unwrap().current_row_num() } fn current_written_size(&self) -> usize { - self.inner_writer.as_ref().unwrap().current_written_size() + self.inner.as_ref().unwrap().current_written_size() } } @@ -116,13 +156,15 @@ mod test { use crate::Result; use crate::io::FileIOBuilder; use crate::spec::{ - DataContentType, DataFileFormat, Literal, NestedField, PrimitiveType, Schema, Struct, Type, + DataContentType, DataFileFormat, Literal, NestedField, PartitionKey, PartitionSpec, + PrimitiveType, Schema, Struct, Type, }; use crate::writer::base_writer::data_file_writer::DataFileWriterBuilder; use crate::writer::file_writer::ParquetWriterBuilder; use crate::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; + use crate::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use crate::writer::{IcebergWriter, IcebergWriterBuilder, RecordBatch}; #[tokio::test] @@ -143,16 +185,16 @@ mod test { ]) .build()?; - let pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema), - None, + let pw = ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(schema)); + + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + pw, file_io.clone(), location_gen, file_name_gen, ); - let mut data_file_writer = DataFileWriterBuilder::new(pw, None, 0) + let mut data_file_writer = DataFileWriterBuilder::new(rolling_file_writer_builder, None) .build() .await .unwrap(); @@ -219,20 +261,27 @@ mod test { NestedField::required(6, "name", Type::Primitive(PrimitiveType::String)).into(), ]) .build()?; + let schema_ref = Arc::new(schema); let partition_value = Struct::from_iter([Some(Literal::int(1))]); + let partition_key = PartitionKey::new( + PartitionSpec::builder(schema_ref.clone()).build()?, + schema_ref.clone(), + partition_value.clone(), + ); - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema.clone()), - None, + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema_ref.clone()); + + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, file_io.clone(), location_gen, file_name_gen, ); let mut data_file_writer = - DataFileWriterBuilder::new(parquet_writer_builder, Some(partition_value.clone()), 0) + DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) .build() .await?; diff --git a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs index 765ff1cacd..6740ed435c 100644 --- a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs +++ b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs @@ -26,21 +26,35 @@ use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use crate::arrow::record_batch_projector::RecordBatchProjector; use crate::arrow::schema_to_arrow_schema; -use crate::spec::{DataFile, SchemaRef, Struct}; -use crate::writer::file_writer::{FileWriter, FileWriterBuilder}; +use crate::spec::{DataFile, PartitionKey, SchemaRef}; +use crate::writer::file_writer::FileWriterBuilder; +use crate::writer::file_writer::location_generator::{FileNameGenerator, LocationGenerator}; +use crate::writer::file_writer::rolling_writer::{RollingFileWriter, RollingFileWriterBuilder}; use crate::writer::{IcebergWriter, IcebergWriterBuilder}; use crate::{Error, ErrorKind, Result}; /// Builder for `EqualityDeleteWriter`. #[derive(Clone, Debug)] -pub struct EqualityDeleteFileWriterBuilder { - inner: B, +pub struct EqualityDeleteFileWriterBuilder< + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +> { + inner: RollingFileWriterBuilder, config: EqualityDeleteWriterConfig, } -impl EqualityDeleteFileWriterBuilder { - /// Create a new `EqualityDeleteFileWriterBuilder` using a `FileWriterBuilder`. - pub fn new(inner: B, config: EqualityDeleteWriterConfig) -> Self { +impl EqualityDeleteFileWriterBuilder +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + /// Create a new `EqualityDeleteFileWriterBuilder` using a `RollingFileWriterBuilder`. + pub fn new( + inner: RollingFileWriterBuilder, + config: EqualityDeleteWriterConfig, + ) -> Self { Self { inner, config } } } @@ -52,8 +66,7 @@ pub struct EqualityDeleteWriterConfig { equality_ids: Vec, // Projector used to project the data chunk into specific fields. projector: RecordBatchProjector, - partition_value: Struct, - partition_spec_id: i32, + partition_key: Option, } impl EqualityDeleteWriterConfig { @@ -61,8 +74,7 @@ impl EqualityDeleteWriterConfig { pub fn new( equality_ids: Vec, original_schema: SchemaRef, - partition_value: Option, - partition_spec_id: i32, + partition_key: Option, ) -> Result { let original_arrow_schema = Arc::new(schema_to_arrow_schema(&original_schema)?); let projector = RecordBatchProjector::new( @@ -98,8 +110,7 @@ impl EqualityDeleteWriterConfig { Ok(Self { equality_ids, projector, - partition_value: partition_value.unwrap_or(Struct::empty()), - partition_spec_id, + partition_key, }) } @@ -110,36 +121,48 @@ impl EqualityDeleteWriterConfig { } #[async_trait::async_trait] -impl IcebergWriterBuilder for EqualityDeleteFileWriterBuilder { - type R = EqualityDeleteFileWriter; +impl IcebergWriterBuilder for EqualityDeleteFileWriterBuilder +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + type R = EqualityDeleteFileWriter; async fn build(self) -> Result { Ok(EqualityDeleteFileWriter { - inner_writer: Some(self.inner.clone().build().await?), + inner: Some(self.inner.clone().build()), // todo revisit this, probably still need a builder for rolling writer projector: self.config.projector, equality_ids: self.config.equality_ids, - partition_value: self.config.partition_value, - partition_spec_id: self.config.partition_spec_id, + partition_key: self.config.partition_key, }) } } /// Writer used to write equality delete files. #[derive(Debug)] -pub struct EqualityDeleteFileWriter { - inner_writer: Option, +pub struct EqualityDeleteFileWriter< + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +> { + inner: Option>, projector: RecordBatchProjector, equality_ids: Vec, - partition_value: Struct, - partition_spec_id: i32, + partition_key: Option, } #[async_trait::async_trait] -impl IcebergWriter for EqualityDeleteFileWriter { +impl IcebergWriter for EqualityDeleteFileWriter +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ async fn write(&mut self, batch: RecordBatch) -> Result<()> { let batch = self.projector.project_batch(batch)?; - if let Some(writer) = self.inner_writer.as_mut() { - writer.write(&batch).await + if let Some(writer) = self.inner.as_mut() { + writer.write(&self.partition_key, &batch).await } else { Err(Error::new( ErrorKind::Unexpected, @@ -149,19 +172,26 @@ impl IcebergWriter for EqualityDeleteFileWriter { } async fn close(&mut self) -> Result> { - if let Some(writer) = self.inner_writer.take() { - Ok(writer + if let Some(writer) = self.inner.take() { + writer .close() .await? .into_iter() .map(|mut res| { res.content(crate::spec::DataContentType::EqualityDeletes); res.equality_ids(Some(self.equality_ids.iter().copied().collect_vec())); - res.partition(self.partition_value.clone()); - res.partition_spec_id(self.partition_spec_id); - res.build().expect("msg") + if let Some(pk) = self.partition_key.as_ref() { + res.partition(pk.data().clone()); + res.partition_spec_id(pk.spec().spec_id()); + } + res.build().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to build data file: {}", e), + ) + }) }) - .collect_vec()) + .collect() } else { Err(Error::new( ErrorKind::Unexpected, @@ -201,6 +231,7 @@ mod test { use crate::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; + use crate::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use crate::writer::{IcebergWriter, IcebergWriterBuilder}; async fn check_parquet_data_file_with_equality_delete_write( @@ -397,23 +428,24 @@ mod test { let equality_ids = vec![0_i32, 8]; let equality_config = - EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None, 0).unwrap(); + EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None).unwrap(); let delete_schema = arrow_schema_to_schema(equality_config.projected_arrow_schema_ref()).unwrap(); let projector = equality_config.projector.clone(); // prepare writer - let pb = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(delete_schema), - None, + let pb = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(delete_schema)); + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + pb, file_io.clone(), location_gen, file_name_gen, ); - let mut equality_delete_writer = EqualityDeleteFileWriterBuilder::new(pb, equality_config) - .build() - .await?; + let mut equality_delete_writer = + EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, equality_config) + .build() + .await?; // write equality_delete_writer.write(to_write.clone()).await?; @@ -499,19 +531,19 @@ mod test { .unwrap(), ); // Float and Double are not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![0], schema.clone(), None, 0).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![1], schema.clone(), None, 0).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![0], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![1], schema.clone(), None).is_err()); // Struct is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![3], schema.clone(), None, 0).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![3], schema.clone(), None).is_err()); // Nested field of struct is allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![4], schema.clone(), None, 0).is_ok()); + assert!(EqualityDeleteWriterConfig::new(vec![4], schema.clone(), None).is_ok()); // Nested field of map is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![7], schema.clone(), None, 0).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![8], schema.clone(), None, 0).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![9], schema.clone(), None, 0).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![7], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![8], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![9], schema.clone(), None).is_err()); // Nested field of list is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![10], schema.clone(), None, 0).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![11], schema.clone(), None, 0).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![10], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![11], schema.clone(), None).is_err()); Ok(()) } @@ -565,22 +597,22 @@ mod test { .unwrap(), ); let equality_ids = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]; - let config = - EqualityDeleteWriterConfig::new(equality_ids, schema.clone(), None, 0).unwrap(); + let config = EqualityDeleteWriterConfig::new(equality_ids, schema.clone(), None).unwrap(); let delete_arrow_schema = config.projected_arrow_schema_ref().clone(); let delete_schema = arrow_schema_to_schema(&delete_arrow_schema).unwrap(); - let pb = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(delete_schema), - None, + let pb = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(delete_schema)); + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + pb, file_io.clone(), location_gen, file_name_gen, ); - let mut equality_delete_writer = EqualityDeleteFileWriterBuilder::new(pb, config) - .build() - .await?; + let mut equality_delete_writer = + EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, config) + .build() + .await?; // prepare data let col0 = Arc::new(BooleanArray::from(vec![ @@ -763,7 +795,7 @@ mod test { let to_write = RecordBatch::try_new(arrow_schema.clone(), columns).unwrap(); let equality_ids = vec![0_i32, 2, 5]; let equality_config = - EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None, 0).unwrap(); + EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None).unwrap(); let projector = equality_config.projector.clone(); // check diff --git a/crates/iceberg/src/writer/file_writer/mod.rs b/crates/iceberg/src/writer/file_writer/mod.rs index 2a5a735534..2ed6414ce8 100644 --- a/crates/iceberg/src/writer/file_writer/mod.rs +++ b/crates/iceberg/src/writer/file_writer/mod.rs @@ -27,6 +27,8 @@ use crate::spec::DataFileBuilder; mod parquet_writer; pub use parquet_writer::{ParquetWriter, ParquetWriterBuilder}; +use crate::io::OutputFile; + pub mod location_generator; /// Module providing writers that can automatically roll over to new files based on size thresholds. pub mod rolling_writer; @@ -38,7 +40,7 @@ pub trait FileWriterBuilder: Send + Clone + 'static { /// The associated file writer type. type R: FileWriter; /// Build file writer. - fn build(self) -> impl Future> + Send; + fn build(self, output_file: OutputFile) -> impl Future> + Send; } /// File writer focus on writing record batch to different physical file format.(Such as parquet. orc) diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 0a8a095ea8..620f27df33 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -34,7 +34,6 @@ use parquet::format::FileMetaData; use parquet::thrift::{TCompactOutputProtocol, TSerializable}; use thrift::protocol::TOutputProtocol; -use super::location_generator::{FileNameGenerator, LocationGenerator}; use super::{FileWriter, FileWriterBuilder}; use crate::arrow::{ ArrowFileReader, DEFAULT_MAP_FIELD_NAME, FieldMatchMode, NanValueCountVisitor, @@ -43,8 +42,8 @@ use crate::arrow::{ use crate::io::{FileIO, FileWrite, OutputFile}; use crate::spec::{ DataContentType, DataFileBuilder, DataFileFormat, Datum, ListType, Literal, MapType, - NestedFieldRef, PartitionKey, PartitionSpec, PrimitiveType, Schema, SchemaRef, SchemaVisitor, - Struct, StructType, TableMetadata, Type, visit_schema, + NestedFieldRef, PartitionSpec, PrimitiveType, Schema, SchemaRef, SchemaVisitor, Struct, + StructType, TableMetadata, Type, visit_schema, }; use crate::transform::create_transform_function; use crate::writer::{CurrentFileStatus, DataFile}; @@ -52,78 +51,43 @@ use crate::{Error, ErrorKind, Result}; /// ParquetWriterBuilder is used to builder a [`ParquetWriter`] #[derive(Clone, Debug)] -pub struct ParquetWriterBuilder { +pub struct ParquetWriterBuilder { props: WriterProperties, schema: SchemaRef, - partition_key: Option, match_mode: FieldMatchMode, - - file_io: FileIO, - location_generator: T, - file_name_generator: F, } -impl ParquetWriterBuilder { +impl ParquetWriterBuilder { /// Create a new `ParquetWriterBuilder` /// To construct the write result, the schema should contain the `PARQUET_FIELD_ID_META_KEY` metadata for each field. - pub fn new( - props: WriterProperties, - schema: SchemaRef, - partition_key: Option, - file_io: FileIO, - location_generator: T, - file_name_generator: F, - ) -> Self { - Self::new_with_match_mode( - props, - schema, - partition_key, - FieldMatchMode::Id, - file_io, - location_generator, - file_name_generator, - ) + pub fn new(props: WriterProperties, schema: SchemaRef) -> Self { + Self::new_with_match_mode(props, schema, FieldMatchMode::Id) } /// Create a new `ParquetWriterBuilder` with custom match mode pub fn new_with_match_mode( props: WriterProperties, schema: SchemaRef, - partition_key: Option, match_mode: FieldMatchMode, - file_io: FileIO, - location_generator: T, - file_name_generator: F, ) -> Self { Self { props, schema, - partition_key, match_mode, - file_io, - location_generator, - file_name_generator, } } } -impl FileWriterBuilder for ParquetWriterBuilder { +impl FileWriterBuilder for ParquetWriterBuilder { type R = ParquetWriter; - async fn build(self) -> Result { - let out_file = self - .file_io - .new_output(self.location_generator.generate_location( - self.partition_key.as_ref(), - &self.file_name_generator.generate_file_name(), - ))?; - + async fn build(self, output_file: OutputFile) -> Result { Ok(ParquetWriter { schema: self.schema.clone(), inner_writer: None, writer_properties: self.props, current_row_num: 0, - out_file, + output_file, nan_value_count_visitor: NanValueCountVisitor::new_with_match_mode(self.match_mode), }) } @@ -250,7 +214,7 @@ impl SchemaVisitor for IndexByParquetPathName { /// `ParquetWriter`` is used to write arrow data into parquet file on storage. pub struct ParquetWriter { schema: SchemaRef, - out_file: OutputFile, + output_file: OutputFile, inner_writer: Option>>>, writer_properties: WriterProperties, current_row_num: usize, @@ -555,7 +519,7 @@ impl FileWriter for ParquetWriter { writer } else { let arrow_schema: ArrowSchemaRef = Arc::new(self.schema.as_ref().try_into()?); - let inner_writer = self.out_file.writer().await?; + let inner_writer = self.output_file.writer().await?; let async_writer = AsyncFileWriter::new(inner_writer); let writer = AsyncArrowWriter::try_new( async_writer, @@ -594,7 +558,7 @@ impl FileWriter for ParquetWriter { let written_size = writer.bytes_written(); if self.current_row_num == 0 { - self.out_file.delete().await.map_err(|err| { + self.output_file.delete().await.map_err(|err| { Error::new( ErrorKind::Unexpected, "Failed to delete empty parquet file.", @@ -616,7 +580,7 @@ impl FileWriter for ParquetWriter { self.schema, parquet_metadata, written_size, - self.out_file.location().to_string(), + self.output_file.location().to_string(), self.nan_value_count_visitor.nan_value_counts, )?]) } @@ -625,7 +589,7 @@ impl FileWriter for ParquetWriter { impl CurrentFileStatus for ParquetWriter { fn current_file_path(&self) -> String { - self.out_file.location().to_string() + self.output_file.location().to_string() } fn current_row_num(&self) -> usize { @@ -703,7 +667,7 @@ mod tests { use crate::io::FileIOBuilder; use crate::spec::{PrimitiveLiteral, Struct, *}; use crate::writer::file_writer::location_generator::{ - DefaultFileNameGenerator, DefaultLocationGenerator, + DefaultFileNameGenerator, DefaultLocationGenerator, FileNameGenerator, LocationGenerator, }; use crate::writer::tests::check_parquet_data_file; @@ -871,11 +835,13 @@ mod tests { // prepare data let schema = { - let fields = vec![ - arrow_schema::Field::new("col", arrow_schema::DataType::Int64, true).with_metadata( - HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "0".to_string())]), - ), - ]; + let fields = + vec![ + Field::new("col", DataType::Int64, true).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "0".to_string(), + )])), + ]; Arc::new(arrow_schema::Schema::new(fields)) }; let col = Arc::new(Int64Array::from_iter_values(0..1024)) as ArrayRef; @@ -883,18 +849,18 @@ mod tests { let to_write = RecordBatch::try_new(schema.clone(), vec![col]).unwrap(); let to_write_null = RecordBatch::try_new(schema.clone(), vec![null_col]).unwrap(); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; + // write data let mut pw = ParquetWriterBuilder::new( WriterProperties::builder() .set_max_row_group_size(128) .build(), Arc::new(to_write.schema().as_ref().try_into().unwrap()), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; pw.write(&to_write_null).await?; @@ -905,7 +871,7 @@ mod tests { .next() .unwrap() // Put dummy field for build successfully. - .content(crate::spec::DataContentType::Data) + .content(DataContentType::Data) .partition(Struct::empty()) .partition_spec_id(0) .build() @@ -1006,7 +972,7 @@ mod tests { None, )); let col5 = Arc::new({ - let mut map_array_builder = arrow_array::builder::MapBuilder::new( + let mut map_array_builder = MapBuilder::new( None, arrow_array::builder::StringBuilder::new(), arrow_array::builder::ListBuilder::new(arrow_array::builder::PrimitiveBuilder::< @@ -1083,18 +1049,15 @@ mod tests { col0, col1, col2, col3, col4, col5, ]) .unwrap(); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data - let mut pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema), - None, - file_io.clone(), - location_gen, - file_name_gen, - ) - .build() - .await?; + let mut pw = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(schema)) + .build(output_file) + .await?; pw.write(&to_write).await?; let res = pw.close().await?; assert_eq!(res.len(), 1); @@ -1161,7 +1124,7 @@ mod tests { async fn test_all_type_for_write() -> Result<()> { let temp_dir = TempDir::new().unwrap(); let file_io = FileIOBuilder::new_fs_io().build().unwrap(); - let loccation_gen = DefaultLocationGenerator::with_data_location( + let location_gen = DefaultLocationGenerator::with_data_location( temp_dir.path().to_str().unwrap().to_string(), ); let file_name_gen = @@ -1276,18 +1239,15 @@ mod tests { col14, col15, col16, ]) .unwrap(); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data - let mut pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema), - None, - file_io.clone(), - loccation_gen, - file_name_gen, - ) - .build() - .await?; + let mut pw = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(schema)) + .build(output_file) + .await?; pw.write(&to_write).await?; let res = pw.close().await?; assert_eq!(res.len(), 1); @@ -1405,7 +1365,7 @@ mod tests { async fn test_decimal_bound() -> Result<()> { let temp_dir = TempDir::new().unwrap(); let file_io = FileIOBuilder::new_fs_io().build().unwrap(); - let loccation_gen = DefaultLocationGenerator::with_data_location( + let location_gen = DefaultLocationGenerator::with_data_location( temp_dir.path().to_str().unwrap().to_string(), ); let file_name_gen = @@ -1429,16 +1389,12 @@ mod tests { .unwrap(), ); let arrow_schema: ArrowSchemaRef = Arc::new(schema_to_arrow_schema(&schema).unwrap()); - let mut pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - schema.clone(), - None, - file_io.clone(), - loccation_gen.clone(), - file_name_gen.clone(), - ) - .build() - .await?; + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; + let mut pw = ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()) + .build(output_file) + .await?; let col0 = Arc::new( Decimal128Array::from(vec![Some(22000000000), Some(11000000000)]) .with_data_type(DataType::Decimal128(28, 10)), @@ -1485,16 +1441,12 @@ mod tests { .unwrap(), ); let arrow_schema: ArrowSchemaRef = Arc::new(schema_to_arrow_schema(&schema).unwrap()); - let mut pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - schema.clone(), - None, - file_io.clone(), - loccation_gen.clone(), - file_name_gen.clone(), - ) - .build() - .await?; + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; + let mut pw = ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()) + .build(output_file) + .await?; let col0 = Arc::new( Decimal128Array::from(vec![Some(-22000000000), Some(-11000000000)]) .with_data_type(DataType::Decimal128(28, 10)), @@ -1544,16 +1496,12 @@ mod tests { .unwrap(), ); let arrow_schema: ArrowSchemaRef = Arc::new(schema_to_arrow_schema(&schema).unwrap()); - let mut pw = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - schema, - None, - file_io.clone(), - loccation_gen, - file_name_gen, - ) - .build() - .await?; + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; + let mut pw = ParquetWriterBuilder::new(WriterProperties::builder().build(), schema) + .build(output_file) + .await?; let col0 = Arc::new( Decimal128Array::from(vec![ Some(decimal_max.mantissa()), @@ -1676,37 +1624,31 @@ mod tests { }; let col = Arc::new(Int64Array::from_iter_values(0..1024)) as ArrayRef; let to_write = RecordBatch::try_new(schema.clone(), vec![col]).unwrap(); + let file_path = location_gen.generate_location(None, &file_name_gen.generate_file_name()); + let output_file = file_io.new_output(&file_path)?; let mut pw = ParquetWriterBuilder::new( WriterProperties::builder().build(), Arc::new(to_write.schema().as_ref().try_into().unwrap()), - None, - file_io.clone(), - location_gen.clone(), - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; - let file_path = pw.out_file.location().to_string(); pw.close().await.unwrap(); - assert!(file_io.exists(file_path).await.unwrap()); + assert!(file_io.exists(&file_path).await.unwrap()); // Test that file will not create if no data to write let file_name_gen = DefaultFileNameGenerator::new("test_empty".to_string(), None, DataFileFormat::Parquet); + let file_path = location_gen.generate_location(None, &file_name_gen.generate_file_name()); + let output_file = file_io.new_output(&file_path)?; let pw = ParquetWriterBuilder::new( WriterProperties::builder().build(), Arc::new(to_write.schema().as_ref().try_into().unwrap()), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; - let file_path = pw.out_file.location().to_string(); pw.close().await.unwrap(); - assert!(!file_io.exists(file_path).await.unwrap()); + assert!(!file_io.exists(&file_path).await.unwrap()); Ok(()) } @@ -1746,17 +1688,16 @@ mod tests { let to_write = RecordBatch::try_new(arrow_schema.clone(), vec![float_32_col, float_64_col]).unwrap(); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data let mut pw = ParquetWriterBuilder::new( WriterProperties::builder().build(), Arc::new(to_write.schema().as_ref().try_into().unwrap()), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; @@ -1887,17 +1828,16 @@ mod tests { struct_nested_float_field_col, ]) .unwrap(); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data let mut pw = ParquetWriterBuilder::new( WriterProperties::builder().build(), Arc::new(to_write.schema().as_ref().try_into().unwrap()), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; @@ -2047,6 +1987,9 @@ mod tests { // large_list_float_field_col, ]) .expect("Could not form record batch"); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data let mut pw = ParquetWriterBuilder::new( @@ -2058,12 +2001,8 @@ mod tests { .try_into() .expect("Could not convert iceberg schema"), ), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; @@ -2229,6 +2168,9 @@ mod tests { struct_list_float_field_col, ]) .expect("Could not form record batch"); + let output_file = file_io.new_output( + location_gen.generate_location(None, &file_name_gen.generate_file_name()), + )?; // write data let mut pw = ParquetWriterBuilder::new( @@ -2240,12 +2182,8 @@ mod tests { .try_into() .expect("Could not convert iceberg schema"), ), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await?; pw.write(&to_write).await?; @@ -2310,6 +2248,9 @@ mod tests { ); let file_name_gen = DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + let output_file = file_io + .new_output(location_gen.generate_location(None, &file_name_gen.generate_file_name())) + .unwrap(); // write data let pw = ParquetWriterBuilder::new( @@ -2325,12 +2266,8 @@ mod tests { .build() .expect("Failed to create schema"), ), - None, - file_io.clone(), - location_gen, - file_name_gen, ) - .build() + .build(output_file) .await .unwrap(); diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index 181205c371..0b9b105c5e 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -15,50 +15,104 @@ // specific language governing permissions and limitations // under the License. +use std::fmt::{Debug, Formatter}; + use arrow_array::RecordBatch; -use crate::spec::DataFileBuilder; +use crate::io::{FileIO, OutputFile}; +use crate::spec::{DataFileBuilder, PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, PartitionKey}; use crate::writer::CurrentFileStatus; +use crate::writer::file_writer::location_generator::{FileNameGenerator, LocationGenerator}; use crate::writer::file_writer::{FileWriter, FileWriterBuilder}; use crate::{Error, ErrorKind, Result}; -/// Builder for creating a `RollingFileWriter` that rolls over to a new file -/// when the data size exceeds a target threshold. -#[derive(Clone)] -pub struct RollingFileWriterBuilder { +/// Builder for [`RollingFileWriter`]. +#[derive(Clone, Debug)] +pub struct RollingFileWriterBuilder< + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +> { inner_builder: B, target_file_size: usize, + file_io: FileIO, + location_generator: L, + file_name_generator: F, } -impl RollingFileWriterBuilder { - /// Creates a new `RollingFileWriterBuilder` with the specified inner builder and target size. +impl RollingFileWriterBuilder +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + /// Creates a new `RollingFileWriterBuilder` with the specified target file size. /// - /// # Arguments + /// # Parameters /// /// * `inner_builder` - The builder for the underlying file writer - /// * `target_file_size` - The target size in bytes before rolling over to a new file + /// * `target_file_size` - The target file size in bytes that triggers rollover + /// * `file_io` - The file IO interface for creating output files + /// * `location_generator` - Generator for file locations + /// * `file_name_generator` - Generator for file names + /// + /// # Returns /// - /// NOTE: The `target_file_size` does not exactly reflect the final size on physical storage. - /// This is because the input size is based on the Arrow in-memory format and cannot precisely control rollover behavior. - /// The actual file size on disk is expected to be slightly larger than `target_file_size`. - pub fn new(inner_builder: B, target_file_size: usize) -> Self { + /// A new `RollingFileWriterBuilder` instance + pub fn new( + inner_builder: B, + target_file_size: usize, + file_io: FileIO, + location_generator: L, + file_name_generator: F, + ) -> Self { Self { inner_builder, target_file_size, + file_io, + location_generator, + file_name_generator, } } -} -impl FileWriterBuilder for RollingFileWriterBuilder { - type R = RollingFileWriter; + /// Creates a new `RollingFileWriterBuilder` with the default target file size. + /// + /// # Parameters + /// + /// * `inner_builder` - The builder for the underlying file writer + /// * `file_io` - The file IO interface for creating output files + /// * `location_generator` - Generator for file locations + /// * `file_name_generator` - Generator for file names + /// + /// # Returns + /// + /// A new `RollingFileWriterBuilder` instance with default target file size + pub fn new_with_default_file_size( + inner_builder: B, + file_io: FileIO, + location_generator: L, + file_name_generator: F, + ) -> Self { + Self { + inner_builder, + target_file_size: PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, + file_io, + location_generator, + file_name_generator, + } + } - async fn build(self) -> Result { - Ok(RollingFileWriter { + /// Build a new [`RollingFileWriter`]. + pub fn build(self) -> RollingFileWriter { + RollingFileWriter { inner: None, inner_builder: self.inner_builder, target_file_size: self.target_file_size, data_file_builders: vec![], - }) + file_io: self.file_io, + location_generator: self.location_generator, + file_name_generator: self.file_name_generator, + } } } @@ -68,14 +122,36 @@ impl FileWriterBuilder for RollingFileWriterBuilder { /// This writer wraps another file writer that tracks the amount of data written. /// When the data size exceeds the target size, it closes the current file and /// starts writing to a new one. -pub struct RollingFileWriter { +pub struct RollingFileWriter { inner: Option, inner_builder: B, target_file_size: usize, data_file_builders: Vec, + file_io: FileIO, + location_generator: L, + file_name_generator: F, +} + +impl Debug for RollingFileWriter +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + f.debug_struct("RollingFileWriter") + .field("target_file_size", &self.target_file_size) + .field("file_io", &self.file_io) + .finish() + } } -impl RollingFileWriter { +impl RollingFileWriter +where + B: FileWriterBuilder, + L: LocationGenerator, + F: FileNameGenerator, +{ /// Determines if the writer should roll over to a new file. /// /// # Returns @@ -84,13 +160,42 @@ impl RollingFileWriter { fn should_roll(&self) -> bool { self.current_written_size() > self.target_file_size } -} -impl FileWriter for RollingFileWriter { - async fn write(&mut self, input: &RecordBatch) -> Result<()> { + fn new_output_file(&self, partition_key: &Option) -> Result { + self.file_io + .new_output(self.location_generator.generate_location( + partition_key.as_ref(), + &self.file_name_generator.generate_file_name(), + )) + } + + /// Writes a record batch to the current file, rolling over to a new file if necessary. + /// + /// # Parameters + /// + /// * `partition_key` - Optional partition key for the data + /// * `input` - The record batch to write + /// + /// # Returns + /// + /// A `Result` indicating success or failure + /// + /// # Errors + /// + /// Returns an error if the writer is not initialized or if writing fails + pub async fn write( + &mut self, + partition_key: &Option, + input: &RecordBatch, + ) -> Result<()> { if self.inner.is_none() { // initialize inner writer - self.inner = Some(self.inner_builder.clone().build().await?); + self.inner = Some( + self.inner_builder + .clone() + .build(self.new_output_file(partition_key)?) + .await?, + ); } if self.should_roll() { @@ -99,7 +204,12 @@ impl FileWriter for RollingFileWriter { self.data_file_builders.extend(inner.close().await?); // start a new writer - self.inner = Some(self.inner_builder.clone().build().await?); + self.inner = Some( + self.inner_builder + .clone() + .build(self.new_output_file(partition_key)?) + .await?, + ); } } @@ -114,7 +224,13 @@ impl FileWriter for RollingFileWriter { } } - async fn close(mut self) -> Result> { + /// Closes the writer and returns all data file builders. + /// + /// # Returns + /// + /// A `Result` containing a vector of `DataFileBuilder` instances representing + /// all files that were written, including any that were created due to rollover + pub async fn close(mut self) -> Result> { // close the current writer and merge the output if let Some(current_writer) = self.inner { self.data_file_builders @@ -125,7 +241,9 @@ impl FileWriter for RollingFileWriter { } } -impl CurrentFileStatus for RollingFileWriter { +impl CurrentFileStatus + for RollingFileWriter +{ fn current_file_path(&self) -> String { self.inner.as_ref().unwrap().current_file_path() } @@ -199,22 +317,20 @@ mod tests { let schema = make_test_schema()?; // Create writer builders - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema), - None, - file_io.clone(), - location_gen, - file_name_gen, - ); + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(schema)); // Set a large target size so no rolling occurs - let rolling_writer_builder = RollingFileWriterBuilder::new( + let rolling_file_writer_builder = RollingFileWriterBuilder::new( parquet_writer_builder, - 1024 * 1024, // 1MB, large enough to not trigger rolling + 1024 * 1024, + file_io.clone(), + location_gen, + file_name_gen, ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None, 0); + let data_file_writer_builder = + DataFileWriterBuilder::new(rolling_file_writer_builder, None); // Create writer let mut writer = data_file_writer_builder.build().await?; @@ -260,19 +376,19 @@ mod tests { let schema = make_test_schema()?; // Create writer builders - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::builder().build(), - Arc::new(schema), - None, - file_io.clone(), + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), Arc::new(schema)); + + // Set a very small target size to trigger rolling + let rolling_writer_builder = RollingFileWriterBuilder::new( + parquet_writer_builder, + 1024, + file_io, location_gen, file_name_gen, ); - // Set a very small target size to trigger rolling - let rolling_writer_builder = RollingFileWriterBuilder::new(parquet_writer_builder, 1024); - - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None, 0); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None); // Create writer let mut writer = data_file_writer_builder.build().await?; diff --git a/crates/iceberg/src/writer/mod.rs b/crates/iceberg/src/writer/mod.rs index 8f17d50e27..d5a8a66861 100644 --- a/crates/iceberg/src/writer/mod.rs +++ b/crates/iceberg/src/writer/mod.rs @@ -60,6 +60,9 @@ //! async fn main() -> Result<()> { //! // Connect to a catalog. //! use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; +//! use iceberg::writer::file_writer::rolling_writer::{ +//! RollingFileWriter, RollingFileWriterBuilder, +//! }; //! let catalog = MemoryCatalogBuilder::default() //! .load( //! "memory", @@ -86,15 +89,21 @@ //! let parquet_writer_builder = ParquetWriterBuilder::new( //! WriterProperties::default(), //! table.metadata().current_schema().clone(), -//! None, +//! ); +//! +//! // Create a rolling file writer using parquet file writer builder. +//! let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( +//! parquet_writer_builder, //! table.file_io().clone(), //! location_generator.clone(), //! file_name_generator.clone(), //! ); +//! //! // Create a data file writer using parquet file writer builder. -//! let data_file_writer_builder = DataFileWriterBuilder::new(parquet_writer_builder, None, 0); +//! let data_file_writer_builder = +//! DataFileWriterBuilder::new(rolling_file_writer_builder, None); //! // Build the data file writer -//! let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); +//! let mut data_file_writer = data_file_writer_builder.build().await?; //! //! // Write the data using data_file_writer... //! @@ -174,6 +183,9 @@ //! // Connect to a catalog. //! use iceberg::memory::MEMORY_CATALOG_WAREHOUSE; //! use iceberg::spec::{Literal, PartitionKey, Struct}; +//! use iceberg::writer::file_writer::rolling_writer::{ +//! RollingFileWriter, RollingFileWriterBuilder, +//! }; //! //! let catalog = MemoryCatalogBuilder::default() //! .load( @@ -207,13 +219,20 @@ //! let parquet_writer_builder = ParquetWriterBuilder::new( //! WriterProperties::default(), //! table.metadata().current_schema().clone(), -//! Some(partition_key), +//! ); +//! +//! // Create a rolling file writer +//! let rolling_file_writer_builder = RollingFileWriterBuilder::new( +//! parquet_writer_builder, +//! 512 * 1024 * 1024, //! table.file_io().clone(), //! location_generator.clone(), //! file_name_generator.clone(), //! ); -//! // Create a data file writer builder using parquet file writer builder. -//! let data_file_writer_builder = DataFileWriterBuilder::new(parquet_writer_builder, None, 0); +//! +//! // Create a data file writer builder using rolling file writer. +//! let data_file_writer_builder = +//! DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)); //! // Create latency record writer using data file writer builder. //! let latency_record_builder = LatencyRecordWriterBuilder::new(data_file_writer_builder); //! // Build the final writer diff --git a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_data_file_test.rs index 9a3d62137e..f4cba959ee 100644 --- a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs +++ b/crates/integration_tests/tests/shared_tests/append_data_file_test.rs @@ -27,6 +27,7 @@ use iceberg::writer::file_writer::ParquetWriterBuilder; use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; +use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Catalog, CatalogBuilder, TableCreation}; use iceberg_catalog_rest::RestCatalogBuilder; @@ -74,12 +75,14 @@ async fn test_append_data_file() { let parquet_writer_builder = ParquetWriterBuilder::new( WriterProperties::default(), table.metadata().current_schema().clone(), - None, + ); + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, table.file_io().clone(), location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(parquet_writer_builder, None, 0); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); let col1 = StringArray::from(vec![Some("foo"), Some("bar"), None, Some("baz")]); let col2 = Int32Array::from(vec![Some(1), Some(2), Some(3), Some(4)]); diff --git a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs index c737357ee9..0da88f1a09 100644 --- a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs +++ b/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs @@ -31,6 +31,7 @@ use iceberg::writer::file_writer::ParquetWriterBuilder; use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; +use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Catalog, CatalogBuilder, TableCreation}; use iceberg_catalog_rest::RestCatalogBuilder; @@ -96,18 +97,18 @@ async fn test_append_partition_data_file() { let parquet_writer_builder = ParquetWriterBuilder::new( WriterProperties::default(), table.metadata().current_schema().clone(), - Some(partition_key), + ); + + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder.clone(), table.file_io().clone(), location_generator.clone(), file_name_generator.clone(), ); let mut data_file_writer_valid = DataFileWriterBuilder::new( - parquet_writer_builder.clone(), - Some(Struct::from_iter([Some(Literal::Primitive( - PrimitiveLiteral::Int(first_partition_id_value), - ))])), - 0, + rolling_file_writer_builder.clone(), + Some(partition_key.clone()), ) .build() .await @@ -148,44 +149,53 @@ async fn test_append_partition_data_file() { assert_eq!(batches.len(), 1); assert_eq!(batches[0], batch); + let partition_key = partition_key.copy_with_data(Struct::from_iter([Some( + Literal::Primitive(PrimitiveLiteral::Boolean(true)), + )])); test_schema_incompatible_partition_type( - parquet_writer_builder.clone(), + rolling_file_writer_builder.clone(), batch.clone(), + partition_key.clone(), table.clone(), &rest_catalog, ) .await; + let partition_key = partition_key.copy_with_data(Struct::from_iter([ + Some(Literal::Primitive(PrimitiveLiteral::Int( + first_partition_id_value, + ))), + Some(Literal::Primitive(PrimitiveLiteral::Int( + first_partition_id_value, + ))), + ])); test_schema_incompatible_partition_fields( - parquet_writer_builder, + rolling_file_writer_builder.clone(), batch, + partition_key, table, &rest_catalog, - first_partition_id_value, ) .await; } async fn test_schema_incompatible_partition_type( - parquet_writer_builder: ParquetWriterBuilder< + rolling_file_writer_builder: RollingFileWriterBuilder< + ParquetWriterBuilder, DefaultLocationGenerator, DefaultFileNameGenerator, >, batch: RecordBatch, + partition_key: PartitionKey, table: Table, catalog: &dyn Catalog, ) { // test writing different "type" of partition than mentioned in schema - let mut data_file_writer_invalid = DataFileWriterBuilder::new( - parquet_writer_builder.clone(), - Some(Struct::from_iter([Some(Literal::Primitive( - PrimitiveLiteral::Boolean(true), - ))])), - 0, - ) - .build() - .await - .unwrap(); + let mut data_file_writer_invalid = + DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) + .build() + .await + .unwrap(); data_file_writer_invalid.write(batch.clone()).await.unwrap(); let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); @@ -200,32 +210,22 @@ async fn test_schema_incompatible_partition_type( } async fn test_schema_incompatible_partition_fields( - parquet_writer_builder: ParquetWriterBuilder< + rolling_file_writer_builder: RollingFileWriterBuilder< + ParquetWriterBuilder, DefaultLocationGenerator, DefaultFileNameGenerator, >, batch: RecordBatch, + partition_key: PartitionKey, table: Table, catalog: &dyn Catalog, - first_partition_id_value: i32, ) { // test writing different number of partition fields than mentioned in schema - - let mut data_file_writer_invalid = DataFileWriterBuilder::new( - parquet_writer_builder, - Some(Struct::from_iter([ - Some(Literal::Primitive(PrimitiveLiteral::Int( - first_partition_id_value, - ))), - Some(Literal::Primitive(PrimitiveLiteral::Int( - first_partition_id_value, - ))), - ])), - 0, - ) - .build() - .await - .unwrap(); + let mut data_file_writer_invalid = + DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) + .build() + .await + .unwrap(); data_file_writer_invalid.write(batch.clone()).await.unwrap(); let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); diff --git a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs b/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs index 4b85612ff5..a248fa707a 100644 --- a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs +++ b/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs @@ -27,6 +27,7 @@ use iceberg::writer::file_writer::ParquetWriterBuilder; use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; +use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Catalog, CatalogBuilder, TableCreation}; use iceberg_catalog_rest::RestCatalogBuilder; @@ -73,12 +74,14 @@ async fn test_append_data_file_conflict() { let parquet_writer_builder = ParquetWriterBuilder::new( WriterProperties::default(), table.metadata().current_schema().clone(), - None, + ); + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, table.file_io().clone(), location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(parquet_writer_builder, None, 0); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); let col1 = StringArray::from(vec![Some("foo"), Some("bar"), None, Some("baz")]); let col2 = Int32Array::from(vec![Some(1), Some(2), Some(3), Some(4)]); diff --git a/crates/integration_tests/tests/shared_tests/scan_all_type.rs b/crates/integration_tests/tests/shared_tests/scan_all_type.rs index b8e7c5941e..1125de11a5 100644 --- a/crates/integration_tests/tests/shared_tests/scan_all_type.rs +++ b/crates/integration_tests/tests/shared_tests/scan_all_type.rs @@ -39,6 +39,7 @@ use iceberg::writer::file_writer::ParquetWriterBuilder; use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; +use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Catalog, CatalogBuilder, TableCreation}; use iceberg_catalog_rest::RestCatalogBuilder; @@ -155,12 +156,14 @@ async fn test_scan_all_type() { let parquet_writer_builder = ParquetWriterBuilder::new( WriterProperties::default(), table.metadata().current_schema().clone(), - None, + ); + let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, table.file_io().clone(), location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(parquet_writer_builder, None, 0); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); // Prepare data diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index 625405c95b..712da92b21 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -208,7 +208,6 @@ impl ExecutionPlan for IcebergWriteExec { )); } - let spec_id = self.table.metadata().default_partition_spec_id(); let partition_type = self.table.metadata().default_partition_type().clone(); let format_version = self.table.metadata().format_version(); @@ -235,13 +234,7 @@ impl ExecutionPlan for IcebergWriteExec { let parquet_file_writer_builder = ParquetWriterBuilder::new_with_match_mode( WriterProperties::default(), self.table.metadata().current_schema().clone(), - None, FieldMatchMode::Name, - self.table.file_io().clone(), - DefaultLocationGenerator::new(self.table.metadata().clone()) - .map_err(to_datafusion_error)?, - // todo filename prefix/suffix should be configurable - DefaultFileNameGenerator::new(Uuid::now_v7().to_string(), None, file_format), ); let target_file_size = match self .table @@ -261,10 +254,22 @@ impl ExecutionPlan for IcebergWriteExec { .map_err(to_datafusion_error)?, None => PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, }; - let rolling_writer_builder = - RollingFileWriterBuilder::new(parquet_file_writer_builder, target_file_size); - let data_file_writer_builder = - DataFileWriterBuilder::new(rolling_writer_builder, None, spec_id); + + let file_io = self.table.file_io().clone(); + let location_generator = DefaultLocationGenerator::new(self.table.metadata().clone()) + .map_err(to_datafusion_error)?; + // todo filename prefix/suffix should be configurable + let file_name_generator = + DefaultFileNameGenerator::new(Uuid::now_v7().to_string(), None, file_format); + let rolling_writer_builder = RollingFileWriterBuilder::new( + parquet_file_writer_builder, + target_file_size, + file_io, + location_generator, + file_name_generator, + ); + // todo specify partition key when partitioning writer is supported + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None); // Get input data let data = execute_input_stream( From 39812f8735ed05674883ac9412af5c5c9846f1fa Mon Sep 17 00:00:00 2001 From: Fokko Driesprong Date: Fri, 10 Oct 2025 09:33:32 +0200 Subject: [PATCH 003/891] Set lock on version of Pydantic (#1737) ## Which issue does this PR close? Got fixed upstream https://github.com/apache/iceberg-python/pull/2591 - Closes #1736 . ## What changes are included in this PR? ## Are these changes tested? --- bindings/python/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 36b5834228..41877e1952 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -49,7 +49,7 @@ include = [ ignore = ["F403", "F405"] [tool.hatch.envs.dev] -dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] +dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1", "pydantic<2.12.0"] [tool.hatch.envs.dev.scripts] build = "maturin build --out dist --sdist" From 441a9c4977e563202815393c4257c0e088b90d5d Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Fri, 10 Oct 2025 16:55:25 +0800 Subject: [PATCH 004/891] chore(deps): Bump crate-ci/typos from 1.36.3 to 1.37.2 (#1734) ## Which issue does this PR close? - Closes #1732. ## What changes are included in this PR? Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.36.3 to 1.37.2. ## Are these changes tested? Signed-off-by: Alan Tang Co-authored-by: Renjie Liu --- .github/workflows/ci_typos.yml | 2 +- crates/catalog/hms/src/utils.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 184b16ac29..488858a113 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v5 - name: Check typos - uses: crate-ci/typos@v1.36.3 + uses: crate-ci/typos@v1.37.2 diff --git a/crates/catalog/hms/src/utils.rs b/crates/catalog/hms/src/utils.rs index 1b65bf0ccb..32c28b82ee 100644 --- a/crates/catalog/hms/src/utils.rs +++ b/crates/catalog/hms/src/utils.rs @@ -37,7 +37,7 @@ const OWNER: &str = "owner"; const COMMENT: &str = "comment"; /// hive metatore `location` property const LOCATION: &str = "location"; -/// hive metatore `metadat_location` property +/// hive metatore `metadata_location` property const METADATA_LOCATION: &str = "metadata_location"; /// hive metatore `external` property const EXTERNAL: &str = "EXTERNAL"; From bdb8bf7c6a55d30ae474b12535b22024144dea56 Mon Sep 17 00:00:00 2001 From: Mrinal Paliwal Date: Sat, 11 Oct 2025 09:13:31 +0530 Subject: [PATCH 005/891] feat: support more partition transformations for PartitionSpec::partition_to_path (#1730) ## Which issue does this PR close? - Closes #1729 . ## What changes are included in this PR? - Support more transformation types in `Transform::to_human_string` - Change visibility of few methods to pub ## Are these changes tested? Updated existing unit test Co-authored-by: Renjie Liu --- crates/iceberg/src/spec/partition.rs | 20 +++++++++++++++++--- crates/iceberg/src/spec/transform.rs | 24 +++++++++++------------- crates/iceberg/src/spec/values.rs | 2 +- 3 files changed, 29 insertions(+), 17 deletions(-) diff --git a/crates/iceberg/src/spec/partition.rs b/crates/iceberg/src/spec/partition.rs index 128db338af..b594895ccd 100644 --- a/crates/iceberg/src/spec/partition.rs +++ b/crates/iceberg/src/spec/partition.rs @@ -155,7 +155,9 @@ impl PartitionSpec { true } - pub(crate) fn partition_to_path(&self, data: &Struct, schema: SchemaRef) -> String { + /// Returns partition path string containing partition type and partition + /// value as key-value pairs. + pub fn partition_to_path(&self, data: &Struct, schema: SchemaRef) -> String { let partition_type = self.partition_type(&schema).unwrap(); let field_types = partition_type.fields(); @@ -1813,6 +1815,9 @@ mod tests { .with_fields(vec![ NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "timestamp", Type::Primitive(PrimitiveType::Timestamp)) + .into(), + NestedField::required(4, "empty", Type::Primitive(PrimitiveType::String)).into(), ]) .build() .unwrap(); @@ -1822,14 +1827,23 @@ mod tests { .unwrap() .add_partition_field("name", "name", Transform::Identity) .unwrap() + .add_partition_field("timestamp", "ts_hour", Transform::Hour) + .unwrap() + .add_partition_field("empty", "empty_void", Transform::Void) + .unwrap() .build() .unwrap(); - let data = Struct::from_iter([Some(Literal::int(42)), Some(Literal::string("alice"))]); + let data = Struct::from_iter([ + Some(Literal::int(42)), + Some(Literal::string("alice")), + Some(Literal::int(1000)), + Some(Literal::string("empty")), + ]); assert_eq!( spec.partition_to_path(&data, schema.into()), - "id=42/name=alice" + "id=42/name=alice/ts_hour=1000/empty_void=null" ); } } diff --git a/crates/iceberg/src/spec/transform.rs b/crates/iceberg/src/spec/transform.rs index 8b6d2225a9..d69d15c288 100644 --- a/crates/iceberg/src/spec/transform.rs +++ b/crates/iceberg/src/spec/transform.rs @@ -137,19 +137,17 @@ pub enum Transform { impl Transform { /// Returns a human-readable String representation of a transformed value. pub fn to_human_string(&self, field_type: &Type, value: Option<&Literal>) -> String { - if let Some(value) = value { - if let Some(value) = value.as_primitive_literal() { - let field_type = field_type.as_primitive_type().unwrap(); - let datum = Datum::new(field_type.clone(), value); - match self { - Self::Identity => datum.to_human_string(), - Self::Void => "null".to_string(), - _ => { - todo!() - } - } - } else { - "null".to_string() + let Some(value) = value else { + return "null".to_string(); + }; + + if let Some(value) = value.as_primitive_literal() { + let field_type = field_type.as_primitive_type().unwrap(); + let datum = Datum::new(field_type.clone(), value); + + match self { + Self::Void => "null".to_string(), + _ => datum.to_human_string(), } } else { "null".to_string() diff --git a/crates/iceberg/src/spec/values.rs b/crates/iceberg/src/spec/values.rs index 7f214810e9..8d56604b1c 100644 --- a/crates/iceberg/src/spec/values.rs +++ b/crates/iceberg/src/spec/values.rs @@ -1271,7 +1271,7 @@ impl Datum { /// /// For string literals, this returns the raw string value without quotes. /// For all other literals, it falls back to [`to_string()`]. - pub(crate) fn to_human_string(&self) -> String { + pub fn to_human_string(&self) -> String { match self.literal() { PrimitiveLiteral::String(s) => s.to_string(), _ => self.to_string(), From deb9a9c752f16e1ed40dbf8a4f9b564e0462c8e0 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Sat, 11 Oct 2025 00:06:37 -0700 Subject: [PATCH 006/891] chore: Update website for 0.7.0 (#1738) ## Which issue does this PR close? - Closes #1631 ## What changes are included in this PR? - Update download links in `website` ## Are these changes tested? --- website/src/download.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/src/download.md b/website/src/download.md index fa29d98177..cc632c4c87 100644 --- a/website/src/download.md +++ b/website/src/download.md @@ -23,8 +23,8 @@ The official Apache Iceberg-Rust releases are provided as source artifacts. ## Releases -The latest source release is [0.6.0](https://www.apache.org/dyn/closer.lua/iceberg/apache-iceberg-rust-0.6.0/apache-iceberg-rust-0.6.0-src.tar.gz?action=download) ([asc](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.6.0/apache-iceberg-rust-0.6.0-src.tar.gz.asc), -[sha512](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.6.0/apache-iceberg-rust-0.6.0-src.tar.gz.sha512)). +The latest source release is [0.7.0](https://www.apache.org/dyn/closer.lua/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz) ([asc](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz.asc), +[sha512](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz.sha512)). For older releases, please check the [archive](https://archive.apache.org/dist/iceberg/). From 05d912235a6b6216d5aef02653f35fe380f635dd Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Sat, 11 Oct 2025 15:35:01 +0800 Subject: [PATCH 007/891] feat(sql-catalog): implement register table for sql catalog (#1724) ## Which issue does this PR close? - Closes #1518. ## What changes are included in this PR? - Implement `register_table` for sql catalog ## Are these changes tested? --------- Signed-off-by: Alan Tang --- crates/catalog/sql/src/catalog.rs | 88 ++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 6 deletions(-) diff --git a/crates/catalog/sql/src/catalog.rs b/crates/catalog/sql/src/catalog.rs index 35889d451a..c1040c65a0 100644 --- a/crates/catalog/sql/src/catalog.rs +++ b/crates/catalog/sql/src/catalog.rs @@ -765,13 +765,30 @@ impl Catalog for SqlCatalog { async fn register_table( &self, - _table_ident: &TableIdent, - _metadata_location: String, + table_ident: &TableIdent, + metadata_location: String, ) -> Result { - Err(Error::new( - ErrorKind::FeatureUnsupported, - "Registering a table is not supported yet", - )) + if self.table_exists(table_ident).await? { + return table_already_exists_err(table_ident); + } + + let metadata = TableMetadata::read_from(&self.fileio, &metadata_location).await?; + + let namespace = table_ident.namespace(); + let tbl_name = table_ident.name().to_string(); + + self.execute(&format!( + "INSERT INTO {CATALOG_TABLE_NAME} + ({CATALOG_FIELD_CATALOG_NAME}, {CATALOG_FIELD_TABLE_NAMESPACE}, {CATALOG_FIELD_TABLE_NAME}, {CATALOG_FIELD_METADATA_LOCATION_PROP}, {CATALOG_FIELD_RECORD_TYPE}) + VALUES (?, ?, ?, ?, ?) + "), vec![Some(&self.name), Some(&namespace.join(".")), Some(&tbl_name), Some(&metadata_location), Some(CATALOG_FIELD_TABLE_RECORD_TYPE)], None).await?; + + Ok(Table::builder() + .identifier(table_ident.clone()) + .metadata_location(metadata_location) + .metadata(metadata) + .file_io(self.fileio.clone()) + .build()?) } async fn update_table(&self, _commit: TableCommit) -> Result
{ @@ -1908,4 +1925,63 @@ mod tests { "Unexpected => No such table: TableIdent { namespace: NamespaceIdent([\"a\"]), name: \"tbl1\" }" ); } + + #[tokio::test] + async fn test_register_table_throws_error_if_table_with_same_name_already_exists() { + let warehouse_loc = temp_path(); + let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let namespace_ident = NamespaceIdent::new("a".into()); + create_namespace(&catalog, &namespace_ident).await; + let table_name = "tbl1"; + let table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); + create_table(&catalog, &table_ident).await; + + assert_eq!( + catalog + .register_table(&table_ident, warehouse_loc) + .await + .unwrap_err() + .to_string(), + format!("Unexpected => Table {:?} already exists.", &table_ident) + ); + } + + #[tokio::test] + async fn test_register_table() { + let warehouse_loc = temp_path(); + let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let namespace_ident = NamespaceIdent::new("a".into()); + create_namespace(&catalog, &namespace_ident).await; + + let table_name = "abc"; + let location = warehouse_loc.clone(); + let table_creation = TableCreation::builder() + .name(table_name.into()) + .location(location.clone()) + .schema(simple_table_schema()) + .build(); + + let table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); + let expected_table = catalog + .create_table(&namespace_ident, table_creation) + .await + .unwrap(); + + let metadata_location = expected_table + .metadata_location() + .expect("Expected metadata location to be set") + .to_string(); + + assert_table_eq(&expected_table, &table_ident, &simple_table_schema()); + + let _ = catalog.drop_table(&table_ident).await; + + let table = catalog + .register_table(&table_ident, metadata_location.clone()) + .await + .unwrap(); + + assert_eq!(table.identifier(), expected_table.identifier()); + assert_eq!(table.metadata_location(), Some(metadata_location.as_str())); + } } From 1de331593ea877e3a8b570e328d26f8362ad2035 Mon Sep 17 00:00:00 2001 From: Sergei Grebnov Date: Sat, 11 Oct 2025 08:44:54 +0000 Subject: [PATCH 008/891] fix: ensure CoalescePartitionsExec is enabled for IcebergCommitExec (#1723) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? PR fixes partial writes similar to reported [here](https://github.com/spiceai/spiceai/issues/7407). Despite the following code to enforce `CoalescePartitionsExec` (single input behavior) it can be removed by DataFusion optimizer. Unit test was added to demonstrate such behavior. https://github.com/apache/iceberg-rust/blob/dc349284a4204c1a56af47fb3177ace6f9e899a0/crates/integrations/datafusion/src/table/mod.rs#L196-L210 ```rust let write_plan = Arc::new(IcebergWriteExec::new( self.table.clone(), input, self.schema.clone(), )); // Merge the outputs of write_plan into one so we can commit all files together let coalesce_partitions = Arc::new(CoalescePartitionsExec::new(write_plan)); Ok(Arc::new(IcebergCommitExec::new( self.table.clone(), catalog, coalesce_partitions, self.schema.clone(), ))) ``` Example plan (observe no `CoalescePartitionsExec`) ```shell explain format tree insert into task_history_sink select * from runtime.task_history; +---------------+-------------------------------+ | plan_type | plan | +---------------+-------------------------------+ | physical_plan | ┌───────────────────────────┐ | | | │ IcebergCommitExec │ | | | │ -------------------- │ | | | │ IcebergCommitExec: table: │ | | | │ team_app.task_history │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ IcebergWriteExec │ | | | │ -------------------- │ | | | │ IcebergWriteExec: table: │ | | | │ team_app.task_history │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ ProjectionExec │ | | | │ -------------------- │ | | | │ captured_output: │ | | | │ captured_output │ | | | │ │ | | | │ end_time: │ | | | │ CAST(end_time AS Timestamp│ | | | │ (Microsecond, None)) │ | | | │ │ | | | │ error_message: │ | | | │ error_message │ | | | │ │ | | | │ execution_duration_ms: │ | | | │ execution_duration_ms │ | | | │ │ | | | │ input: input │ | | | │ │ | | | │ labels: │ | | | │ CAST(labels AS Map(Field {│ | | | │ name: "key_value", │ | | | │ data_type: Struct( │ | | | │ [Field { name: "key", │ | | | │ data_type: Utf8, │ | | | │ nullable: false, │ | | | │ dict_id: 0, │ | | | │ dict_is_ordered │ | | | │ : false, metadata: { │ | | | │ "PARQUET:field_id": │ | | | │ "12"} }, Field { name: │ | | | │ "value", data_type: │ | | | │ Utf8, nullable: true │ | | | │ ... │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ RepartitionExec │ | | | │ -------------------- │ | | | │ partition_count(in->out): │ | | | │ 1 -> 14 │ | | | │ │ | | | │ partitioning_scheme: │ | | | │ RoundRobinBatch(14) │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ BytesProcessedExec │ | | | │ -------------------- │ | | | │ BytesProcessedExec │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ SchemaCastScanExec │ | | | │ -------------------- │ | | | │ SchemaCastScanExec │ | | | └─────────────┬─────────────┘ | | | ┌─────────────┴─────────────┐ | | | │ DataSourceExec │ | | | │ -------------------- │ | | | │ bytes: 88176 │ | | | │ format: memory │ | | | │ rows: 6 │ | | | └───────────────────────────┘ | | | | +---------------+-------------------------------+ ``` ## What changes are included in this PR? PR adds `required_input_distribution` setting for `IcebergWriteExec` to ensure DataFusion coalesces input partitions automatically before commit. Similar to [DataFusion DataSinkExec](https://github.com/apache/datafusion/blob/a7b113c45509aae34595b6a62469b3173cac91bd/datafusion/datasource/src/sink.rs#L187) `test_datafusion_execution_partitioned_source` can be used to ovserve behavior before and after Before ```rust Physical plan: IcebergCommitExec: table=test_namespace.test_table_partitioning RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=3 IcebergWriteExec: table=test_namespace.test_table_partitioning DataSourceExec: partitions=3, partition_sizes=[1, 1, 1] ``` After ```rust IcebergCommitExec: table=test_namespace.test_table CoalescePartitionsExec IcebergWriteExec: table=test_namespace.test_table DataSourceExec: partitions=3, partition_sizes=[1, 1, 1] ``` ## Are these changes tested? Added `test_datafusion_execution_partitioned_source` unit test, tested manually --- .../datafusion/src/physical_plan/commit.rs | 111 +++++++++++++++++- .../datafusion/src/physical_plan/write.rs | 10 ++ 2 files changed, 120 insertions(+), 1 deletion(-) diff --git a/crates/integrations/datafusion/src/physical_plan/commit.rs b/crates/integrations/datafusion/src/physical_plan/commit.rs index 0ee0504faa..067049c122 100644 --- a/crates/integrations/datafusion/src/physical_plan/commit.rs +++ b/crates/integrations/datafusion/src/physical_plan/commit.rs @@ -136,6 +136,14 @@ impl ExecutionPlan for IcebergCommitExec { vec![&self.input] } + fn required_input_distribution(&self) -> Vec { + vec![datafusion::physical_plan::Distribution::SinglePartition; self.children().len()] + } + + fn benefits_from_input_partitioning(&self) -> Vec { + vec![false] + } + fn with_new_children( self: Arc, children: Vec>, @@ -262,14 +270,16 @@ mod tests { use std::fmt; use std::sync::Arc; - use datafusion::arrow::array::{ArrayRef, RecordBatch, StringArray}; + use datafusion::arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray, UInt64Array}; use datafusion::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; + use datafusion::datasource::MemTable; use datafusion::execution::context::TaskContext; use datafusion::physical_expr::{EquivalenceProperties, Partitioning}; use datafusion::physical_plan::common::collect; use datafusion::physical_plan::execution_plan::Boundedness; use datafusion::physical_plan::stream::RecordBatchStreamAdapter; use datafusion::physical_plan::{DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties}; + use datafusion::prelude::*; use futures::StreamExt; use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; use iceberg::spec::{ @@ -280,6 +290,7 @@ mod tests { use super::*; use crate::physical_plan::DATA_FILES_COL_NAME; + use crate::table::IcebergTableProvider; // A mock execution plan that returns record batches with serialized data files #[derive(Debug)] @@ -510,4 +521,102 @@ mod tests { Ok(()) } + + #[tokio::test] + async fn test_datafusion_execution_partitioned_source() -> Result<(), Box> + { + let catalog = Arc::new( + MemoryCatalogBuilder::default() + .load( + "memory", + HashMap::from([( + MEMORY_CATALOG_WAREHOUSE.to_string(), + "memory://root".to_string(), + )]), + ) + .await?, + ); + + let namespace = NamespaceIdent::new("test_namespace".to_string()); + catalog.create_namespace(&namespace, HashMap::new()).await?; + + let schema = Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build()?; + + let table_name = "test_table"; + let table_creation = TableCreation::builder() + .name(table_name.to_string()) + .schema(schema) + .location("memory://root/test_table".to_string()) + .properties(HashMap::new()) + .build(); + let _ = catalog.create_table(&namespace, table_creation).await?; + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let batches: Vec = (1..4) + .map(|idx| { + RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Int32Array::from(vec![idx])) as ArrayRef, + Arc::new(StringArray::from(vec![format!("Name{idx}")])) as ArrayRef, + ]) + }) + .collect::>()?; + + // Create DataFusion context with specific partition configuration + let mut config = SessionConfig::new(); + config = config.set_usize("datafusion.execution.target_partitions", 8); + let ctx = SessionContext::new_with_config(config); + + // Create multiple partitions - each batch becomes a separate partition + let partitions: Vec> = + batches.into_iter().map(|batch| vec![batch]).collect(); + let source_table = Arc::new(MemTable::try_new(Arc::clone(&arrow_schema), partitions)?); + ctx.register_table("source_table", source_table)?; + + let iceberg_table_provider = IcebergTableProvider::try_new( + catalog.clone(), + namespace.clone(), + table_name.to_string(), + ) + .await?; + ctx.register_table("iceberg_table", Arc::new(iceberg_table_provider))?; + + let insert_plan = ctx + .sql("INSERT INTO iceberg_table SELECT * FROM source_table") + .await?; + + let physical_plan = insert_plan.create_physical_plan().await?; + + let actual_plan = format!( + "{}", + datafusion::physical_plan::displayable(physical_plan.as_ref()).indent(false) + ); + + println!("Physical plan:\n{actual_plan}"); + + let expected_plan = "\ +IcebergCommitExec: table=test_namespace.test_table + CoalescePartitionsExec + IcebergWriteExec: table=test_namespace.test_table + DataSourceExec: partitions=3, partition_sizes=[1, 1, 1]"; + + assert_eq!( + actual_plan.trim(), + expected_plan.trim(), + "Physical plan does not match expected\n\nExpected:\n{}\n\nActual:\n{}", + expected_plan.trim(), + actual_plan.trim() + ); + + Ok(()) + } } diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index 712da92b21..759f1a8de2 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -144,6 +144,16 @@ impl ExecutionPlan for IcebergWriteExec { self } + /// Prevents the introduction of additional `RepartitionExec` and processing input in parallel. + fn benefits_from_input_partitioning(&self) -> Vec { + vec![false] + } + + fn maintains_input_order(&self) -> Vec { + // Maintains ordering in the sense that the written file will reflect the ordering of the input. + vec![true; self.children().len()] + } + fn properties(&self) -> &PlanProperties { &self.plan_properties } From 03a64055eedc535d158d8f824bcf201d165e031d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Oct 2025 15:33:48 +0800 Subject: [PATCH 009/891] chore(deps): Bump regex from 1.11.2 to 1.12.1 (#1741) Bumps [regex](https://github.com/rust-lang/regex) from 1.11.2 to 1.12.1.
Changelog

Sourced from regex's changelog.

1.12.1 (2025-10-10)

This release makes a bug fix in the new regex::Captures::get_match API introduced in 1.12.0. There was an oversight with the lifetime parameter for the Match returned. This is technically a breaking change, but given that it was caught almost immediately and I've yanked the 1.12.0 release, I think this is fine.

1.12.0 (2025-10-10)

This release contains a smattering of bug fixes, a fix for excessive memory consumption in some cases and a new regex::Captures::get_match API.

Improvements:

Bug fixes:

1.11.3 (2025-09-25)

This is a small patch release with an improvement in memory usage in some cases.

Improvements:

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=regex&package-manager=cargo&previous-version=1.11.2&new-version=1.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 596e943bc2..16ffef0654 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2838,7 +2838,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "778e2ac28f6c47af28e4907f13ffd1e1ddbd400980a9abd7c8df189bf578a5ad" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -4288,7 +4288,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07033963ba89ebaf1584d767badaa2e8fcec21aedea6b8c0346d487d49c28667" dependencies = [ "cfg-if", - "windows-targets 0.48.5", + "windows-targets 0.53.3", ] [[package]] @@ -5698,7 +5698,7 @@ dependencies = [ "once_cell", "socket2 0.6.0", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -5864,9 +5864,9 @@ dependencies = [ [[package]] name = "regex" -version = "1.11.2" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23d7fd106d8c02486a8d64e778353d1cffe08ce79ac2e82f540c86d0facf6912" +checksum = "4a52d8d02cacdb176ef4678de6c052efb4b3da14b78e4db683a4252762be5433" dependencies = [ "aho-corasick", "memchr", @@ -5876,9 +5876,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.10" +version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b9458fa0bfeeac22b5ca447c63aaf45f28439a709ccd244698632f9aa6394d6" +checksum = "722166aa0d7438abbaa4d5cc2c649dac844e8c56d82fb3d33e9c34b5cd268fc6" dependencies = [ "aho-corasick", "memchr", @@ -6186,7 +6186,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.9.4", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -7262,7 +7262,7 @@ dependencies = [ "getrandom 0.3.3", "once_cell", "rustix 1.0.8", - "windows-sys 0.59.0", + "windows-sys 0.61.0", ] [[package]] From b4a44a31e7fea50b443098cdb9c05c33288866da Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Oct 2025 15:34:18 +0800 Subject: [PATCH 010/891] chore(deps): Bump crate-ci/typos from 1.37.2 to 1.38.1 (#1740) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.37.2 to 1.38.1.
Release notes

Sourced from crate-ci/typos's releases.

v1.38.1

[1.38.1] - 2025-10-07

Fixes

  • Ignore common golang identifiers

v1.38.0

[1.38.0] - 2025-10-06

Features

  • Update type list

Fixes

  • Don't correct typ
  • Consistently error on unused config fields

v1.37.3

[1.37.3] - 2025-10-06

Fixes

  • Don't correct PN for bitbake file types
Changelog

Sourced from crate-ci/typos's changelog.

[1.38.1] - 2025-10-07

Fixes

  • Ignore common golang identifiers

[1.38.0] - 2025-10-06

Features

  • Update type list

Fixes

  • Don't correct typ
  • Consistently error on unused config fields

[1.37.3] - 2025-10-06

Fixes

  • Don't correct PN for bitbake file types
Commits
  • 80c8a49 chore: Release
  • c1008ce docs: Update changelog
  • 62a3b50 Merge pull request #1398 from ccoveille-forks/go-exclusions
  • e6bedbd fix(config): Add some Go exclusions
  • 90cacd6 docs(ref): Speak to glob ambiguity
  • b81b12e docs(ref): Clarify directories are not spell checked
  • eaf25df docs(ref): Speak to locale's behavior
  • a9735e2 docs(ref): Provide identifier/word config examples
  • 3c14191 docs(ref): Talk about include lists
  • d0f81dc docs(ref): Re-organize help more like cargo
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.37.2&new-version=1.38.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 488858a113..11564caa13 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v5 - name: Check typos - uses: crate-ci/typos@v1.37.2 + uses: crate-ci/typos@v1.38.1 From ec545f2a9f50dcbe694b2196e04b32de24f7a411 Mon Sep 17 00:00:00 2001 From: Sergei Grebnov Date: Tue, 14 Oct 2025 10:11:23 +0000 Subject: [PATCH 011/891] Improve `IcebergCommitExec` to correctly populate properties/schema (#1721) ## Which issue does this PR close? PR fixes schema mismatch errors (similar to the example shown below) when using `IcebergCommitExec` with DataFusion. This occurs when `IcebergCommitExec` is not the top-level plan but is instead wrapped as the input to another plan node, for example when added by a custom optimization rule (cache invalidation step for example). >An internal error occurred. Internal error: PhysicalOptimizer rule 'OutputRequirements' failed. Schema mismatch. Expected original schema: Schema { fields: [Field { name: "count", data_type: UInt64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, got new schema: Schema { fields: [Field { name: "r_regionkey", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, Field { name: "r_name", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, Field { name: "r_comment", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }], metadata: {} }. This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues ## What changes are included in this PR? PR updates `compute_properties` logic to use target (output) schema instead of input schema. Below is example DataFusion `DataSinkExec` implementation demonstrating that properties must be created based on target schema, not input. https://github.com/apache/datafusion/blob/4eacb6046773b759dae0b3d801fe8cb1c6b65c0f/datafusion/datasource/src/sink.rs#L101C1-L117C6 ```rust impl DataSinkExec { /// Create a plan to write to `sink` pub fn new( input: Arc, sink: Arc, sort_order: Option, ) -> Self { let count_schema = make_count_schema(); let cache = Self::create_schema(&input, count_schema); Self { input, sink, count_schema: make_count_schema(), sort_order, cache, } } .... fn properties(&self) -> &PlanProperties { &self.cache } ``` ## Are these changes tested? Tested manually, expanded existing test to verify output schema, tested as part of [Spice Iceberg write automated tests](https://github.com/spiceai/spiceai/blob/trunk/crates/runtime/tests/iceberg/write/mod.rs) --- .../integrations/datafusion/src/physical_plan/commit.rs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/crates/integrations/datafusion/src/physical_plan/commit.rs b/crates/integrations/datafusion/src/physical_plan/commit.rs index 067049c122..48373f11e2 100644 --- a/crates/integrations/datafusion/src/physical_plan/commit.rs +++ b/crates/integrations/datafusion/src/physical_plan/commit.rs @@ -57,14 +57,16 @@ impl IcebergCommitExec { input: Arc, schema: ArrowSchemaRef, ) -> Self { - let plan_properties = Self::compute_properties(schema.clone()); + let count_schema = Self::make_count_schema(); + + let plan_properties = Self::compute_properties(Arc::clone(&count_schema)); Self { table, catalog, input, schema, - count_schema: Self::make_count_schema(), + count_schema, plan_properties, } } @@ -469,6 +471,9 @@ mod tests { let commit_exec = IcebergCommitExec::new(table.clone(), catalog.clone(), input_exec, arrow_schema); + // Verify Execution Plan schema matches the count schema + assert_eq!(commit_exec.schema(), IcebergCommitExec::make_count_schema()); + // Execute the commit exec let task_ctx = Arc::new(TaskContext::default()); let stream = commit_exec.execute(0, task_ctx)?; From 36bd7eb36828be24acf08e97e2b2ff8e8e83d6f4 Mon Sep 17 00:00:00 2001 From: Kaushik Srinivasan Date: Tue, 14 Oct 2025 06:16:32 -0400 Subject: [PATCH 012/891] feat(spec): add `table_properties.rs` to spec (#1733) ## Which issue does this PR close? - Closes #1505 . ## What changes are included in this PR? - Adds `table_properties.rs` to hold and validate properties and set default values. Uses macros to simplify setting new properties. ## Are these changes tested? Yes --- Cargo.toml | 4 +- crates/iceberg/Cargo.toml | 2 +- crates/iceberg/src/spec/mod.rs | 2 + crates/iceberg/src/spec/table_metadata.rs | 85 ------ .../src/spec/table_metadata_builder.rs | 23 +- crates/iceberg/src/spec/table_properties.rs | 284 ++++++++++++++++++ crates/iceberg/src/transaction/mod.rs | 69 +---- crates/iceberg/src/transaction/snapshot.rs | 13 +- .../src/writer/file_writer/rolling_writer.rs | 4 +- .../datafusion/src/physical_plan/write.rs | 14 +- 10 files changed, 327 insertions(+), 173 deletions(-) create mode 100644 crates/iceberg/src/spec/table_properties.rs diff --git a/Cargo.toml b/Cargo.toml index 999b911753..46c99cc3d8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -76,10 +76,10 @@ futures = "0.3" hive_metastore = "0.2.0" http = "1.2" iceberg = { version = "0.7.0", path = "./crates/iceberg" } -iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } iceberg-catalog-glue = { version = "0.7.0", path = "./crates/catalog/glue" } -iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } iceberg-catalog-hms = { version = "0.7.0", path = "./crates/catalog/hms" } +iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } +iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } indicatif = "0.17" itertools = "0.13" diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index d592700b73..b831607aab 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -74,8 +74,8 @@ opendal = { workspace = true } ordered-float = { workspace = true } parquet = { workspace = true, features = ["async"] } rand = { workspace = true } -reqwest = { workspace = true } reqsign = { version = "0.16.3", optional = true, default-features = false } +reqwest = { workspace = true } roaring = { workspace = true } rust_decimal = { workspace = true } serde = { workspace = true } diff --git a/crates/iceberg/src/spec/mod.rs b/crates/iceberg/src/spec/mod.rs index 90dafcc110..44b35e5a6b 100644 --- a/crates/iceberg/src/spec/mod.rs +++ b/crates/iceberg/src/spec/mod.rs @@ -30,6 +30,7 @@ mod sort; mod statistic_file; mod table_metadata; mod table_metadata_builder; +mod table_properties; mod transform; mod values; mod view_metadata; @@ -48,6 +49,7 @@ pub use snapshot_summary::*; pub use sort::*; pub use statistic_file::*; pub use table_metadata::*; +pub use table_properties::*; pub use transform::*; pub use values::*; pub use view_metadata::*; diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index d7347d50ee..ca298f308d 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -46,91 +46,6 @@ pub(crate) static ONE_MINUTE_MS: i64 = 60_000; pub(crate) static EMPTY_SNAPSHOT_ID: i64 = -1; pub(crate) static INITIAL_SEQUENCE_NUMBER: i64 = 0; -/// Reserved table property for table format version. -/// -/// Iceberg will default a new table's format version to the latest stable and recommended -/// version. This reserved property keyword allows users to override the Iceberg format version of -/// the table metadata. -/// -/// If this table property exists when creating a table, the table will use the specified format -/// version. If a table updates this property, it will try to upgrade to the specified format -/// version. -pub const PROPERTY_FORMAT_VERSION: &str = "format-version"; -/// Reserved table property for table UUID. -pub const PROPERTY_UUID: &str = "uuid"; -/// Reserved table property for the total number of snapshots. -pub const PROPERTY_SNAPSHOT_COUNT: &str = "snapshot-count"; -/// Reserved table property for current snapshot summary. -pub const PROPERTY_CURRENT_SNAPSHOT_SUMMARY: &str = "current-snapshot-summary"; -/// Reserved table property for current snapshot id. -pub const PROPERTY_CURRENT_SNAPSHOT_ID: &str = "current-snapshot-id"; -/// Reserved table property for current snapshot timestamp. -pub const PROPERTY_CURRENT_SNAPSHOT_TIMESTAMP: &str = "current-snapshot-timestamp-ms"; -/// Reserved table property for the JSON representation of current schema. -pub const PROPERTY_CURRENT_SCHEMA: &str = "current-schema"; -/// Reserved table property for the JSON representation of current(default) partition spec. -pub const PROPERTY_DEFAULT_PARTITION_SPEC: &str = "default-partition-spec"; -/// Reserved table property for the JSON representation of current(default) sort order. -pub const PROPERTY_DEFAULT_SORT_ORDER: &str = "default-sort-order"; - -/// Property key for max number of previous versions to keep. -pub const PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX: &str = "write.metadata.previous-versions-max"; -/// Default value for max number of previous versions to keep. -pub const PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT: usize = 100; - -/// Property key for max number of partitions to keep summary stats for. -pub const PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT: &str = "write.summary.partition-limit"; -/// Default value for the max number of partitions to keep summary stats for. -pub const PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT: u64 = 0; - -/// Reserved Iceberg table properties list. -/// -/// Reserved table properties are only used to control behaviors when creating or updating a -/// table. The value of these properties are not persisted as a part of the table metadata. -pub const RESERVED_PROPERTIES: [&str; 9] = [ - PROPERTY_FORMAT_VERSION, - PROPERTY_UUID, - PROPERTY_SNAPSHOT_COUNT, - PROPERTY_CURRENT_SNAPSHOT_ID, - PROPERTY_CURRENT_SNAPSHOT_SUMMARY, - PROPERTY_CURRENT_SNAPSHOT_TIMESTAMP, - PROPERTY_CURRENT_SCHEMA, - PROPERTY_DEFAULT_PARTITION_SPEC, - PROPERTY_DEFAULT_SORT_ORDER, -]; - -/// Property key for number of commit retries. -pub const PROPERTY_COMMIT_NUM_RETRIES: &str = "commit.retry.num-retries"; -/// Default value for number of commit retries. -pub const PROPERTY_COMMIT_NUM_RETRIES_DEFAULT: usize = 4; - -/// Property key for minimum wait time (ms) between retries. -pub const PROPERTY_COMMIT_MIN_RETRY_WAIT_MS: &str = "commit.retry.min-wait-ms"; -/// Default value for minimum wait time (ms) between retries. -pub const PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT: u64 = 100; - -/// Property key for maximum wait time (ms) between retries. -pub const PROPERTY_COMMIT_MAX_RETRY_WAIT_MS: &str = "commit.retry.max-wait-ms"; -/// Default value for maximum wait time (ms) between retries. -pub const PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT: u64 = 60 * 1000; // 1 minute - -/// Property key for total maximum retry time (ms). -pub const PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS: &str = "commit.retry.total-timeout-ms"; -/// Default value for total maximum retry time (ms). -pub const PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT: u64 = 30 * 60 * 1000; // 30 minutes - -/// Default file format for data files -pub const PROPERTY_DEFAULT_FILE_FORMAT: &str = "write.format.default"; -/// Default file format for delete files -pub const PROPERTY_DELETE_DEFAULT_FILE_FORMAT: &str = "write.delete.format.default"; -/// Default value for data file format -pub const PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT: &str = "parquet"; - -/// Target file size for newly written files. -pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES: &str = "write.target-file-size-bytes"; -/// Default target file size -pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT: usize = 512 * 1024 * 1024; // 512 MB - /// Reference to [`TableMetadata`]. pub type TableMetadataRef = Arc; diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 068f2002f9..7881ebea40 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -22,11 +22,10 @@ use uuid::Uuid; use super::{ DEFAULT_PARTITION_SPEC_ID, DEFAULT_SCHEMA_ID, FormatVersion, MAIN_BRANCH, MetadataLog, - ONE_MINUTE_MS, PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX, - PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT, PartitionSpec, PartitionSpecBuilder, - PartitionStatisticsFile, RESERVED_PROPERTIES, Schema, SchemaRef, Snapshot, SnapshotLog, - SnapshotReference, SnapshotRetention, SortOrder, SortOrderRef, StatisticsFile, StructType, - TableMetadata, UNPARTITIONED_LAST_ASSIGNED_ID, UnboundPartitionSpec, + ONE_MINUTE_MS, PartitionSpec, PartitionSpecBuilder, PartitionStatisticsFile, Schema, SchemaRef, + Snapshot, SnapshotLog, SnapshotReference, SnapshotRetention, SortOrder, SortOrderRef, + StatisticsFile, StructType, TableMetadata, TableProperties, UNPARTITIONED_LAST_ASSIGNED_ID, + UnboundPartitionSpec, }; use crate::error::{Error, ErrorKind, Result}; use crate::{TableCreation, TableUpdate}; @@ -247,7 +246,7 @@ impl TableMetadataBuilder { // List of specified properties that are RESERVED and should not be persisted. let reserved_properties = properties .keys() - .filter(|key| RESERVED_PROPERTIES.contains(&key.as_str())) + .filter(|key| TableProperties::RESERVED_PROPERTIES.contains(&key.as_str())) .map(ToString::to_string) .collect::>(); @@ -285,7 +284,7 @@ impl TableMetadataBuilder { // disallow removal of reserved properties let reserved_properties = properties .iter() - .filter(|key| RESERVED_PROPERTIES.contains(&key.as_str())) + .filter(|key| TableProperties::RESERVED_PROPERTIES.contains(&key.as_str())) .map(ToString::to_string) .collect::>(); @@ -1061,9 +1060,9 @@ impl TableMetadataBuilder { let max_size = self .metadata .properties - .get(PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX) + .get(TableProperties::PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX) .and_then(|v| v.parse::().ok()) - .unwrap_or(PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT) + .unwrap_or(TableProperties::PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT) .max(1); if self.metadata.metadata_log.len() > max_size { @@ -1360,8 +1359,8 @@ mod tests { use crate::io::FileIOBuilder; use crate::spec::{ BlobMetadata, NestedField, NullOrder, Operation, PartitionSpec, PrimitiveType, Schema, - SnapshotRetention, SortDirection, SortField, StructType, Summary, Transform, Type, - UnboundPartitionField, + SnapshotRetention, SortDirection, SortField, StructType, Summary, TableProperties, + Transform, Type, UnboundPartitionField, }; use crate::table::Table; @@ -2299,7 +2298,7 @@ mod tests { let builder = builder_without_changes(FormatVersion::V2); let metadata = builder .set_properties(HashMap::from_iter(vec![( - PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX.to_string(), + TableProperties::PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX.to_string(), "2".to_string(), )])) .unwrap() diff --git a/crates/iceberg/src/spec/table_properties.rs b/crates/iceberg/src/spec/table_properties.rs new file mode 100644 index 0000000000..9aa789fed8 --- /dev/null +++ b/crates/iceberg/src/spec/table_properties.rs @@ -0,0 +1,284 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::collections::HashMap; + +// Helper function to parse a property from a HashMap +// If the property is not found, use the default value +fn parse_property( + properties: &HashMap, + key: &str, + default: T, +) -> Result +where + ::Err: std::fmt::Display, +{ + properties.get(key).map_or(Ok(default), |value| { + value + .parse::() + .map_err(|e| anyhow::anyhow!("Invalid value for {}: {}", key, e)) + }) +} + +/// TableProperties that contains the properties of a table. +#[derive(Debug)] +pub struct TableProperties { + /// The number of times to retry a commit. + pub commit_num_retries: usize, + /// The minimum wait time between retries. + pub commit_min_retry_wait_ms: u64, + /// The maximum wait time between retries. + pub commit_max_retry_wait_ms: u64, + /// The total timeout for commit retries. + pub commit_total_retry_timeout_ms: u64, + /// The default format for files. + pub write_format_default: String, + /// The target file size for files. + pub write_target_file_size_bytes: usize, +} + +impl TableProperties { + /// Reserved table property for table format version. + /// + /// Iceberg will default a new table's format version to the latest stable and recommended + /// version. This reserved property keyword allows users to override the Iceberg format version of + /// the table metadata. + /// + /// If this table property exists when creating a table, the table will use the specified format + /// version. If a table updates this property, it will try to upgrade to the specified format + /// version. + pub const PROPERTY_FORMAT_VERSION: &str = "format-version"; + /// Reserved table property for table UUID. + pub const PROPERTY_UUID: &str = "uuid"; + /// Reserved table property for the total number of snapshots. + pub const PROPERTY_SNAPSHOT_COUNT: &str = "snapshot-count"; + /// Reserved table property for current snapshot summary. + pub const PROPERTY_CURRENT_SNAPSHOT_SUMMARY: &str = "current-snapshot-summary"; + /// Reserved table property for current snapshot id. + pub const PROPERTY_CURRENT_SNAPSHOT_ID: &str = "current-snapshot-id"; + /// Reserved table property for current snapshot timestamp. + pub const PROPERTY_CURRENT_SNAPSHOT_TIMESTAMP: &str = "current-snapshot-timestamp-ms"; + /// Reserved table property for the JSON representation of current schema. + pub const PROPERTY_CURRENT_SCHEMA: &str = "current-schema"; + /// Reserved table property for the JSON representation of current(default) partition spec. + pub const PROPERTY_DEFAULT_PARTITION_SPEC: &str = "default-partition-spec"; + /// Reserved table property for the JSON representation of current(default) sort order. + pub const PROPERTY_DEFAULT_SORT_ORDER: &str = "default-sort-order"; + + /// Property key for max number of previous versions to keep. + pub const PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX: &str = + "write.metadata.previous-versions-max"; + /// Default value for max number of previous versions to keep. + pub const PROPERTY_METADATA_PREVIOUS_VERSIONS_MAX_DEFAULT: usize = 100; + + /// Property key for max number of partitions to keep summary stats for. + pub const PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT: &str = "write.summary.partition-limit"; + /// Default value for the max number of partitions to keep summary stats for. + pub const PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT: u64 = 0; + + /// Reserved Iceberg table properties list. + /// + /// Reserved table properties are only used to control behaviors when creating or updating a + /// table. The value of these properties are not persisted as a part of the table metadata. + pub const RESERVED_PROPERTIES: [&str; 9] = [ + Self::PROPERTY_FORMAT_VERSION, + Self::PROPERTY_UUID, + Self::PROPERTY_SNAPSHOT_COUNT, + Self::PROPERTY_CURRENT_SNAPSHOT_ID, + Self::PROPERTY_CURRENT_SNAPSHOT_SUMMARY, + Self::PROPERTY_CURRENT_SNAPSHOT_TIMESTAMP, + Self::PROPERTY_CURRENT_SCHEMA, + Self::PROPERTY_DEFAULT_PARTITION_SPEC, + Self::PROPERTY_DEFAULT_SORT_ORDER, + ]; + + /// Property key for number of commit retries. + pub const PROPERTY_COMMIT_NUM_RETRIES: &str = "commit.retry.num-retries"; + /// Default value for number of commit retries. + pub const PROPERTY_COMMIT_NUM_RETRIES_DEFAULT: usize = 4; + + /// Property key for minimum wait time (ms) between retries. + pub const PROPERTY_COMMIT_MIN_RETRY_WAIT_MS: &str = "commit.retry.min-wait-ms"; + /// Default value for minimum wait time (ms) between retries. + pub const PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT: u64 = 100; + + /// Property key for maximum wait time (ms) between retries. + pub const PROPERTY_COMMIT_MAX_RETRY_WAIT_MS: &str = "commit.retry.max-wait-ms"; + /// Default value for maximum wait time (ms) between retries. + pub const PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT: u64 = 60 * 1000; // 1 minute + + /// Property key for total maximum retry time (ms). + pub const PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS: &str = "commit.retry.total-timeout-ms"; + /// Default value for total maximum retry time (ms). + pub const PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT: u64 = 30 * 60 * 1000; // 30 minutes + + /// Default file format for data files + pub const PROPERTY_DEFAULT_FILE_FORMAT: &str = "write.format.default"; + /// Default file format for delete files + pub const PROPERTY_DELETE_DEFAULT_FILE_FORMAT: &str = "write.delete.format.default"; + /// Default value for data file format + pub const PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT: &str = "parquet"; + + /// Target file size for newly written files. + pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES: &str = "write.target-file-size-bytes"; + /// Default target file size + pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT: usize = 512 * 1024 * 1024; // 512 MB +} + +impl TryFrom<&HashMap> for TableProperties { + // parse by entry key or use default value + type Error = anyhow::Error; + + fn try_from(props: &HashMap) -> Result { + Ok(TableProperties { + commit_num_retries: parse_property( + props, + TableProperties::PROPERTY_COMMIT_NUM_RETRIES, + TableProperties::PROPERTY_COMMIT_NUM_RETRIES_DEFAULT, + )?, + commit_min_retry_wait_ms: parse_property( + props, + TableProperties::PROPERTY_COMMIT_MIN_RETRY_WAIT_MS, + TableProperties::PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT, + )?, + commit_max_retry_wait_ms: parse_property( + props, + TableProperties::PROPERTY_COMMIT_MAX_RETRY_WAIT_MS, + TableProperties::PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT, + )?, + commit_total_retry_timeout_ms: parse_property( + props, + TableProperties::PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS, + TableProperties::PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT, + )?, + write_format_default: parse_property( + props, + TableProperties::PROPERTY_DEFAULT_FILE_FORMAT, + TableProperties::PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT.to_string(), + )?, + write_target_file_size_bytes: parse_property( + props, + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES, + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, + )?, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_table_properties_default() { + let props = HashMap::new(); + let table_properties = TableProperties::try_from(&props).unwrap(); + assert_eq!( + table_properties.commit_num_retries, + TableProperties::PROPERTY_COMMIT_NUM_RETRIES_DEFAULT + ); + assert_eq!( + table_properties.commit_min_retry_wait_ms, + TableProperties::PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT + ); + assert_eq!( + table_properties.commit_max_retry_wait_ms, + TableProperties::PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT + ); + assert_eq!( + table_properties.write_format_default, + TableProperties::PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT.to_string() + ); + assert_eq!( + table_properties.write_target_file_size_bytes, + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT + ); + } + + #[test] + fn test_table_properties_valid() { + let props = HashMap::from([ + ( + TableProperties::PROPERTY_COMMIT_NUM_RETRIES.to_string(), + "10".to_string(), + ), + ( + TableProperties::PROPERTY_COMMIT_MAX_RETRY_WAIT_MS.to_string(), + "20".to_string(), + ), + ( + TableProperties::PROPERTY_DEFAULT_FILE_FORMAT.to_string(), + "avro".to_string(), + ), + ( + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES.to_string(), + "512".to_string(), + ), + ]); + let table_properties = TableProperties::try_from(&props).unwrap(); + assert_eq!(table_properties.commit_num_retries, 10); + assert_eq!(table_properties.commit_max_retry_wait_ms, 20); + assert_eq!(table_properties.write_format_default, "avro".to_string()); + assert_eq!(table_properties.write_target_file_size_bytes, 512); + } + + #[test] + fn test_table_properties_invalid() { + let invalid_retries = HashMap::from([( + TableProperties::PROPERTY_COMMIT_NUM_RETRIES.to_string(), + "abc".to_string(), + )]); + + let table_properties = TableProperties::try_from(&invalid_retries).unwrap_err(); + assert!( + table_properties.to_string().contains( + "Invalid value for commit.retry.num-retries: invalid digit found in string" + ) + ); + + let invalid_min_wait = HashMap::from([( + TableProperties::PROPERTY_COMMIT_MIN_RETRY_WAIT_MS.to_string(), + "abc".to_string(), + )]); + let table_properties = TableProperties::try_from(&invalid_min_wait).unwrap_err(); + assert!( + table_properties.to_string().contains( + "Invalid value for commit.retry.min-wait-ms: invalid digit found in string" + ) + ); + + let invalid_max_wait = HashMap::from([( + TableProperties::PROPERTY_COMMIT_MAX_RETRY_WAIT_MS.to_string(), + "abc".to_string(), + )]); + let table_properties = TableProperties::try_from(&invalid_max_wait).unwrap_err(); + assert!( + table_properties.to_string().contains( + "Invalid value for commit.retry.max-wait-ms: invalid digit found in string" + ) + ); + + let invalid_target_size = HashMap::from([( + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES.to_string(), + "abc".to_string(), + )]); + let table_properties = TableProperties::try_from(&invalid_target_size).unwrap_err(); + assert!(table_properties.to_string().contains( + "Invalid value for write.target-file-size-bytes: invalid digit found in string" + )); + } +} diff --git a/crates/iceberg/src/transaction/mod.rs b/crates/iceberg/src/transaction/mod.rs index 06549a95c5..26bd652226 100644 --- a/crates/iceberg/src/transaction/mod.rs +++ b/crates/iceberg/src/transaction/mod.rs @@ -52,8 +52,6 @@ /// that allows users to apply a transaction action to a `Transaction`. mod action; -use std::collections::HashMap; - pub use action::*; mod append; mod snapshot; @@ -69,12 +67,7 @@ use std::time::Duration; use backon::{BackoffBuilder, ExponentialBackoff, ExponentialBuilder, RetryableWithContext}; use crate::error::Result; -use crate::spec::{ - PROPERTY_COMMIT_MAX_RETRY_WAIT_MS, PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT, - PROPERTY_COMMIT_MIN_RETRY_WAIT_MS, PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT, - PROPERTY_COMMIT_NUM_RETRIES, PROPERTY_COMMIT_NUM_RETRIES_DEFAULT, - PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS, PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT, -}; +use crate::spec::TableProperties; use crate::table::Table; use crate::transaction::action::BoxedTransactionAction; use crate::transaction::append::FastAppendAction; @@ -170,7 +163,12 @@ impl Transaction { return Ok(self.table); } - let backoff = Self::build_backoff(self.table.metadata().properties())?; + let table_props = + TableProperties::try_from(self.table.metadata().properties()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Invalid table properties").with_source(e) + })?; + + let backoff = Self::build_backoff(table_props)?; let tx = self; (|mut tx: Transaction| async { @@ -185,53 +183,14 @@ impl Transaction { .1 } - fn build_backoff(props: &HashMap) -> Result { - let min_delay = match props.get(PROPERTY_COMMIT_MIN_RETRY_WAIT_MS) { - Some(value_str) => value_str.parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Invalid value for commit.retry.min-wait-ms", - ) - .with_source(e) - })?, - None => PROPERTY_COMMIT_MIN_RETRY_WAIT_MS_DEFAULT, - }; - let max_delay = match props.get(PROPERTY_COMMIT_MAX_RETRY_WAIT_MS) { - Some(value_str) => value_str.parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Invalid value for commit.retry.max-wait-ms", - ) - .with_source(e) - })?, - None => PROPERTY_COMMIT_MAX_RETRY_WAIT_MS_DEFAULT, - }; - let total_delay = match props.get(PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS) { - Some(value_str) => value_str.parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Invalid value for commit.retry.total-timeout-ms", - ) - .with_source(e) - })?, - None => PROPERTY_COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT, - }; - let max_times = match props.get(PROPERTY_COMMIT_NUM_RETRIES) { - Some(value_str) => value_str.parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Invalid value for commit.retry.num-retries", - ) - .with_source(e) - })?, - None => PROPERTY_COMMIT_NUM_RETRIES_DEFAULT, - }; - + fn build_backoff(props: TableProperties) -> Result { Ok(ExponentialBuilder::new() - .with_min_delay(Duration::from_millis(min_delay)) - .with_max_delay(Duration::from_millis(max_delay)) - .with_total_delay(Some(Duration::from_millis(total_delay))) - .with_max_times(max_times) + .with_min_delay(Duration::from_millis(props.commit_min_retry_wait_ms)) + .with_max_delay(Duration::from_millis(props.commit_max_retry_wait_ms)) + .with_total_delay(Some(Duration::from_millis( + props.commit_total_retry_timeout_ms, + ))) + .with_max_times(props.commit_num_retries) .with_factor(2.0) .build()) } diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index 48dc2b5b90..a03b8dc490 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -24,10 +24,9 @@ use uuid::Uuid; use crate::error::Result; use crate::spec::{ DataFile, DataFileFormat, FormatVersion, MAIN_BRANCH, ManifestContentType, ManifestEntry, - ManifestFile, ManifestListWriter, ManifestWriter, ManifestWriterBuilder, Operation, - PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT, PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT, - Snapshot, SnapshotReference, SnapshotRetention, SnapshotSummaryCollector, Struct, StructType, - Summary, update_snapshot_summaries, + ManifestFile, ManifestListWriter, ManifestWriter, ManifestWriterBuilder, Operation, Snapshot, + SnapshotReference, SnapshotRetention, SnapshotSummaryCollector, Struct, StructType, Summary, + TableProperties, update_snapshot_summaries, }; use crate::table::Table; use crate::transaction::ActionCommit; @@ -322,15 +321,15 @@ impl<'a> SnapshotProducer<'a> { let partition_summary_limit = if let Some(limit) = table_metadata .properties() - .get(PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT) + .get(TableProperties::PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT) { if let Ok(limit) = limit.parse::() { limit } else { - PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT + TableProperties::PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT } } else { - PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT + TableProperties::PROPERTY_WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT }; summary_collector.set_partition_summary_limit(partition_summary_limit); diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index 0b9b105c5e..0003617043 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -20,7 +20,7 @@ use std::fmt::{Debug, Formatter}; use arrow_array::RecordBatch; use crate::io::{FileIO, OutputFile}; -use crate::spec::{DataFileBuilder, PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, PartitionKey}; +use crate::spec::{DataFileBuilder, PartitionKey, TableProperties}; use crate::writer::CurrentFileStatus; use crate::writer::file_writer::location_generator::{FileNameGenerator, LocationGenerator}; use crate::writer::file_writer::{FileWriter, FileWriterBuilder}; @@ -95,7 +95,7 @@ where ) -> Self { Self { inner_builder, - target_file_size: PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, + target_file_size: TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, file_io, location_generator, file_name_generator, diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index 759f1a8de2..7bff239222 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -36,11 +36,7 @@ use datafusion::physical_plan::{ }; use futures::StreamExt; use iceberg::arrow::{FieldMatchMode, schema_to_arrow_schema}; -use iceberg::spec::{ - DataFileFormat, PROPERTY_DEFAULT_FILE_FORMAT, PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT, - PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES, PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, - serialize_data_file_to_json, -}; +use iceberg::spec::{DataFileFormat, TableProperties, serialize_data_file_to_json}; use iceberg::table::Table; use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; use iceberg::writer::file_writer::ParquetWriterBuilder; @@ -226,8 +222,8 @@ impl ExecutionPlan for IcebergWriteExec { self.table .metadata() .properties() - .get(PROPERTY_DEFAULT_FILE_FORMAT) - .unwrap_or(&PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT.to_string()), + .get(TableProperties::PROPERTY_DEFAULT_FILE_FORMAT) + .unwrap_or(&TableProperties::PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT.to_string()), ) .map_err(to_datafusion_error)?; if file_format != DataFileFormat::Parquet { @@ -250,7 +246,7 @@ impl ExecutionPlan for IcebergWriteExec { .table .metadata() .properties() - .get(PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES) + .get(TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES) { Some(value_str) => value_str .parse::() @@ -262,7 +258,7 @@ impl ExecutionPlan for IcebergWriteExec { .with_source(e) }) .map_err(to_datafusion_error)?, - None => PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, + None => TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, }; let file_io = self.table.file_io().clone(); From 32ecab0cf0a950be6b7edb35c38b493b946be5fe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Oct 2025 17:34:06 -0700 Subject: [PATCH 013/891] chore(deps): Bump actions/stale from 10.0.0 to 10.1.0 (#1726) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/stale](https://github.com/actions/stale) from 10.0.0 to 10.1.0.
Release notes

Sourced from actions/stale's releases.

v10.1.0

What's Changed

New Contributors

Full Changelog: https://github.com/actions/stale/compare/v10...v10.1.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/stale&package-manager=github_actions&previous-version=10.0.0&new-version=10.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/stale.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 4b79c33274..5e653cffe4 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -31,7 +31,7 @@ jobs: if: github.repository_owner == 'apache' runs-on: ubuntu-22.04 steps: - - uses: actions/stale@v10.0.0 + - uses: actions/stale@v10.1.0 with: stale-issue-label: 'stale' exempt-issue-labels: 'not-stale' From 471ef6b37f898ab4315efb5497714bb35c57ce11 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 14 Oct 2025 19:56:20 -0700 Subject: [PATCH 014/891] docs: remove -src suffix from artifact name (#1743) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Context: https://github.com/apache/iceberg-rust/issues/1631#issuecomment-3402485332 Aligns with other subproject's artifact naming convention ## Are these changes tested? --- scripts/release.sh | 2 +- website/src/release.md | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 1f66b35911..a790cdd8d1 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -47,7 +47,7 @@ python3 ./scripts/dependencies.py check # Generate and verify artifacts echo "> Start package" -git archive --format=tar.gz --output="dist/apache-iceberg-rust-$release_version-src.tar.gz" --prefix="apache-iceberg-rust-$release_version-src/" --add-file=Cargo.toml "$git_branch" +git archive --format=tar.gz --output="dist/apache-iceberg-rust-$release_version.tar.gz" --prefix="apache-iceberg-rust-$release_version/" --add-file=Cargo.toml "$git_branch" cd dist diff --git a/website/src/release.md b/website/src/release.md index 28b98f1700..78c0e57525 100644 --- a/website/src/release.md +++ b/website/src/release.md @@ -143,9 +143,9 @@ After GitHub Release has been created, we can start to create ASF Release. - This script will do the following things: - Create a new branch named by `release-${release_version}` from the tag - Generate the release candidate artifacts under `dist`, including: - - `apache-iceberg-rust-${release_version}-src.tar.gz` - - `apache-iceberg-rust-${release_version}-src.tar.gz.asc` - - `apache-iceberg-rust-${release_version}-src.tar.gz.sha512` + - `apache-iceberg-rust-${release_version}.tar.gz` + - `apache-iceberg-rust-${release_version}.tar.gz.asc` + - `apache-iceberg-rust-${release_version}.tar.gz.sha512` - Check the header of the source code. This step needs docker to run. This script will create a new release under `dist`. @@ -155,9 +155,9 @@ For example: ```shell > tree dist dist -├── apache-iceberg-rust-0.2.0-src.tar.gz -├── apache-iceberg-rust-0.2.0-src.tar.gz.asc -└── apache-iceberg-rust-0.2.0-src.tar.gz.sha512 +├── apache-iceberg-rust-0.2.0.tar.gz +├── apache-iceberg-rust-0.2.0.tar.gz.asc +└── apache-iceberg-rust-0.2.0.tar.gz.sha512 ``` ### Upload artifacts to the SVN dist repo @@ -325,7 +325,7 @@ After downloading them, here are the instructions on how to verify them. ```bash gpg --verify apache-iceberg-rust-*.tar.gz.asc ``` - Expects: `"apache-iceberg-rust-0.7.0-src.tar.gz: OK"` + Expects: `"apache-iceberg-rust-0.7.0.tar.gz: OK"` * Verify the checksums: ```bash From 273991eee390038325866edb3df87a16a00fc4a8 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Thu, 16 Oct 2025 06:01:32 -0400 Subject: [PATCH 015/891] feat(reader): Make ArrowReaderBuilder::new public (#1748) ## Which issue does this PR close? - Part of #1749. ## What changes are included in this PR? - Change `ArrowReaderBuilder::new` to be `pub` instead of `pub(crate)`. ## Are these changes tested? - No new tests for this. Currently being used in DataFusion Comet: https://github.com/apache/datafusion-comet/pull/2528 --- crates/iceberg/src/arrow/reader.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 05aa6a4c9f..564b281107 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -68,7 +68,7 @@ pub struct ArrowReaderBuilder { impl ArrowReaderBuilder { /// Create a new ArrowReaderBuilder - pub(crate) fn new(file_io: FileIO) -> Self { + pub fn new(file_io: FileIO) -> Self { let num_cpus = available_parallelism().get(); ArrowReaderBuilder { From 856597b60a846b9287788e35f5b4e113e71e2516 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Fri, 17 Oct 2025 03:02:33 -0700 Subject: [PATCH 016/891] feat(writer): Add clustered and fanout writer (#1735) ## Which issue does this PR close? - Closes #1572 #1573 ## What changes are included in this PR? New: - Added new `partitioning` module with `PartitioningWriter` trait - `ClusteredWriter`: Optimized for pre-sorted data, requires writing in partition order - `FanoutWriter`: Flexible writer that can handle data from any partition at any time Modification: - (BREAKING) Modified `DataFileWriterBuilder` to support dynamic partition assignment - Updated DataFusion integration to use the new writer API ## Are these changes tested? Added unit tests --- .../writer/base_writer/data_file_writer.rs | 26 +- .../base_writer/equality_delete_writer.rs | 42 +- .../src/writer/file_writer/rolling_writer.rs | 9 +- crates/iceberg/src/writer/mod.rs | 28 +- .../writer/partitioning/clustered_writer.rs | 520 ++++++++++++++++++ .../src/writer/partitioning/fanout_writer.rs | 385 +++++++++++++ crates/iceberg/src/writer/partitioning/mod.rs | 49 ++ .../shared_tests/append_data_file_test.rs | 4 +- .../append_partition_data_file_test.rs | 30 +- .../shared_tests/conflict_commit_test.rs | 4 +- .../tests/shared_tests/scan_all_type.rs | 4 +- .../datafusion/src/physical_plan/write.rs | 7 +- 12 files changed, 1024 insertions(+), 84 deletions(-) create mode 100644 crates/iceberg/src/writer/partitioning/clustered_writer.rs create mode 100644 crates/iceberg/src/writer/partitioning/fanout_writer.rs create mode 100644 crates/iceberg/src/writer/partitioning/mod.rs diff --git a/crates/iceberg/src/writer/base_writer/data_file_writer.rs b/crates/iceberg/src/writer/base_writer/data_file_writer.rs index a950547d33..b9de4efda7 100644 --- a/crates/iceberg/src/writer/base_writer/data_file_writer.rs +++ b/crates/iceberg/src/writer/base_writer/data_file_writer.rs @@ -30,7 +30,6 @@ use crate::{Error, ErrorKind, Result}; #[derive(Clone, Debug)] pub struct DataFileWriterBuilder { inner: RollingFileWriterBuilder, - partition_key: Option, } impl DataFileWriterBuilder @@ -40,14 +39,8 @@ where F: FileNameGenerator, { /// Create a new `DataFileWriterBuilder` using a `RollingFileWriterBuilder`. - pub fn new( - inner_builder: RollingFileWriterBuilder, - partition_key: Option, - ) -> Self { - Self { - inner: inner_builder, - partition_key, - } + pub fn new(inner: RollingFileWriterBuilder) -> Self { + Self { inner } } } @@ -60,10 +53,10 @@ where { type R = DataFileWriter; - async fn build(self) -> Result { + async fn build(self, partition_key: Option) -> Result { Ok(DataFileWriter { inner: Some(self.inner.clone().build()), - partition_key: self.partition_key, + partition_key, }) } } @@ -194,8 +187,8 @@ mod test { file_name_gen, ); - let mut data_file_writer = DataFileWriterBuilder::new(rolling_file_writer_builder, None) - .build() + let mut data_file_writer = DataFileWriterBuilder::new(rolling_file_writer_builder) + .build(None) .await .unwrap(); @@ -280,10 +273,9 @@ mod test { file_name_gen, ); - let mut data_file_writer = - DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) - .build() - .await?; + let mut data_file_writer = DataFileWriterBuilder::new(rolling_file_writer_builder) + .build(Some(partition_key)) + .await?; let arrow_schema = arrow_schema::Schema::new(vec![ Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( diff --git a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs index 6740ed435c..bd96c80bb3 100644 --- a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs +++ b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs @@ -66,16 +66,11 @@ pub struct EqualityDeleteWriterConfig { equality_ids: Vec, // Projector used to project the data chunk into specific fields. projector: RecordBatchProjector, - partition_key: Option, } impl EqualityDeleteWriterConfig { /// Create a new `DataFileWriterConfig` with equality ids. - pub fn new( - equality_ids: Vec, - original_schema: SchemaRef, - partition_key: Option, - ) -> Result { + pub fn new(equality_ids: Vec, original_schema: SchemaRef) -> Result { let original_arrow_schema = Arc::new(schema_to_arrow_schema(&original_schema)?); let projector = RecordBatchProjector::new( original_arrow_schema, @@ -110,7 +105,6 @@ impl EqualityDeleteWriterConfig { Ok(Self { equality_ids, projector, - partition_key, }) } @@ -129,12 +123,12 @@ where { type R = EqualityDeleteFileWriter; - async fn build(self) -> Result { + async fn build(self, partition_key: Option) -> Result { Ok(EqualityDeleteFileWriter { - inner: Some(self.inner.clone().build()), // todo revisit this, probably still need a builder for rolling writer + inner: Some(self.inner.clone().build()), projector: self.config.projector, equality_ids: self.config.equality_ids, - partition_key: self.config.partition_key, + partition_key, }) } } @@ -428,7 +422,7 @@ mod test { let equality_ids = vec![0_i32, 8]; let equality_config = - EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None).unwrap(); + EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema)).unwrap(); let delete_schema = arrow_schema_to_schema(equality_config.projected_arrow_schema_ref()).unwrap(); let projector = equality_config.projector.clone(); @@ -444,7 +438,7 @@ mod test { ); let mut equality_delete_writer = EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, equality_config) - .build() + .build(None) .await?; // write @@ -531,19 +525,19 @@ mod test { .unwrap(), ); // Float and Double are not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![0], schema.clone(), None).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![1], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![0], schema.clone()).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![1], schema.clone()).is_err()); // Struct is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![3], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![3], schema.clone()).is_err()); // Nested field of struct is allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![4], schema.clone(), None).is_ok()); + assert!(EqualityDeleteWriterConfig::new(vec![4], schema.clone()).is_ok()); // Nested field of map is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![7], schema.clone(), None).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![8], schema.clone(), None).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![9], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![7], schema.clone()).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![8], schema.clone()).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![9], schema.clone()).is_err()); // Nested field of list is not allowed to be used for equality delete - assert!(EqualityDeleteWriterConfig::new(vec![10], schema.clone(), None).is_err()); - assert!(EqualityDeleteWriterConfig::new(vec![11], schema.clone(), None).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![10], schema.clone()).is_err()); + assert!(EqualityDeleteWriterConfig::new(vec![11], schema.clone()).is_err()); Ok(()) } @@ -597,7 +591,7 @@ mod test { .unwrap(), ); let equality_ids = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]; - let config = EqualityDeleteWriterConfig::new(equality_ids, schema.clone(), None).unwrap(); + let config = EqualityDeleteWriterConfig::new(equality_ids, schema.clone()).unwrap(); let delete_arrow_schema = config.projected_arrow_schema_ref().clone(); let delete_schema = arrow_schema_to_schema(&delete_arrow_schema).unwrap(); @@ -611,7 +605,7 @@ mod test { ); let mut equality_delete_writer = EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, config) - .build() + .build(None) .await?; // prepare data @@ -795,7 +789,7 @@ mod test { let to_write = RecordBatch::try_new(arrow_schema.clone(), columns).unwrap(); let equality_ids = vec![0_i32, 2, 5]; let equality_config = - EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema), None).unwrap(); + EqualityDeleteWriterConfig::new(equality_ids, Arc::new(schema)).unwrap(); let projector = equality_config.projector.clone(); // check diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index 0003617043..b68010304c 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -329,11 +329,10 @@ mod tests { file_name_gen, ); - let data_file_writer_builder = - DataFileWriterBuilder::new(rolling_file_writer_builder, None); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); // Create writer - let mut writer = data_file_writer_builder.build().await?; + let mut writer = data_file_writer_builder.build(None).await?; // Create test data let arrow_schema = make_test_arrow_schema(); @@ -388,10 +387,10 @@ mod tests { file_name_gen, ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); // Create writer - let mut writer = data_file_writer_builder.build().await?; + let mut writer = data_file_writer_builder.build(None).await?; // Create test data let arrow_schema = make_test_arrow_schema(); diff --git a/crates/iceberg/src/writer/mod.rs b/crates/iceberg/src/writer/mod.rs index d5a8a66861..1da3fa6790 100644 --- a/crates/iceberg/src/writer/mod.rs +++ b/crates/iceberg/src/writer/mod.rs @@ -100,10 +100,9 @@ //! ); //! //! // Create a data file writer using parquet file writer builder. -//! let data_file_writer_builder = -//! DataFileWriterBuilder::new(rolling_file_writer_builder, None); +//! let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); //! // Build the data file writer -//! let mut data_file_writer = data_file_writer_builder.build().await?; +//! let mut data_file_writer = data_file_writer_builder.build(None).await?; //! //! // Write the data using data_file_writer... //! @@ -122,7 +121,7 @@ //! use arrow_array::RecordBatch; //! use iceberg::io::FileIOBuilder; //! use iceberg::memory::MemoryCatalogBuilder; -//! use iceberg::spec::DataFile; +//! use iceberg::spec::{DataFile, PartitionKey}; //! use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; //! use iceberg::writer::file_writer::ParquetWriterBuilder; //! use iceberg::writer::file_writer::location_generator::{ @@ -149,9 +148,9 @@ //! impl IcebergWriterBuilder for LatencyRecordWriterBuilder { //! type R = LatencyRecordWriter; //! -//! async fn build(self) -> Result { +//! async fn build(self, partition_key: Option) -> Result { //! Ok(LatencyRecordWriter { -//! inner_writer: self.inner_writer_builder.build().await?, +//! inner_writer: self.inner_writer_builder.build(partition_key).await?, //! }) //! } //! } @@ -231,12 +230,14 @@ //! ); //! //! // Create a data file writer builder using rolling file writer. -//! let data_file_writer_builder = -//! DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)); +//! let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); //! // Create latency record writer using data file writer builder. //! let latency_record_builder = LatencyRecordWriterBuilder::new(data_file_writer_builder); //! // Build the final writer -//! let mut latency_record_data_file_writer = latency_record_builder.build().await.unwrap(); +//! let mut latency_record_data_file_writer = latency_record_builder +//! .build(Some(partition_key)) +//! .await +//! .unwrap(); //! //! Ok(()) //! } @@ -244,11 +245,14 @@ pub mod base_writer; pub mod file_writer; +/// Provides partition-aware writers +/// TODO examples +pub mod partitioning; use arrow_array::RecordBatch; use crate::Result; -use crate::spec::DataFile; +use crate::spec::{DataFile, PartitionKey}; type DefaultInput = RecordBatch; type DefaultOutput = Vec; @@ -260,8 +264,8 @@ pub trait IcebergWriterBuilder: { /// The associated writer type. type R: IcebergWriter; - /// Build the iceberg writer. - async fn build(self) -> Result; + /// Build the iceberg writer with an optional partition key. + async fn build(self, partition_key: Option) -> Result; } /// The iceberg writer used to write data to iceberg table. diff --git a/crates/iceberg/src/writer/partitioning/clustered_writer.rs b/crates/iceberg/src/writer/partitioning/clustered_writer.rs new file mode 100644 index 0000000000..c141b212f5 --- /dev/null +++ b/crates/iceberg/src/writer/partitioning/clustered_writer.rs @@ -0,0 +1,520 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! This module provides the `ClusteredWriter` implementation. + +use std::collections::HashSet; +use std::marker::PhantomData; + +use async_trait::async_trait; + +use crate::spec::{PartitionKey, Struct}; +use crate::writer::partitioning::PartitioningWriter; +use crate::writer::{DefaultInput, DefaultOutput, IcebergWriter, IcebergWriterBuilder}; +use crate::{Error, ErrorKind, Result}; + +/// A writer that writes data to a single partition at a time. +/// +/// This writer expects input data to be sorted by partition key. It maintains only one +/// active writer at a time, making it memory efficient for sorted data. +/// +/// # Type Parameters +/// +/// * `B` - The inner writer builder type +/// * `I` - Input type (defaults to `RecordBatch`) +/// * `O` - Output collection type (defaults to `Vec`) +pub struct ClusteredWriter +where + B: IcebergWriterBuilder, + O: IntoIterator + FromIterator<::Item>, + ::Item: Clone, +{ + inner_builder: B, + current_writer: Option, + current_partition: Option, + closed_partitions: HashSet, + output: Vec<::Item>, + _phantom: PhantomData, +} + +impl ClusteredWriter +where + B: IcebergWriterBuilder, + I: Send + 'static, + O: IntoIterator + FromIterator<::Item>, + ::Item: Send + Clone, +{ + /// Create a new `ClusteredWriter`. + pub fn new(inner_builder: B) -> Self { + Self { + inner_builder, + current_writer: None, + current_partition: None, + closed_partitions: HashSet::new(), + output: Vec::new(), + _phantom: PhantomData, + } + } + + /// Closes the current writer if it exists, flushes the written data to output, and record closed partition. + async fn close_current_writer(&mut self) -> Result<()> { + if let Some(mut writer) = self.current_writer.take() { + self.output.extend(writer.close().await?); + + // Add the current partition to the set of closed partitions + if let Some(current_partition) = self.current_partition.take() { + self.closed_partitions.insert(current_partition); + } + } + + Ok(()) + } +} + +#[async_trait] +impl PartitioningWriter for ClusteredWriter +where + B: IcebergWriterBuilder, + I: Send + 'static, + O: IntoIterator + FromIterator<::Item> + Send + 'static, + ::Item: Send + Clone, +{ + async fn write(&mut self, partition_key: PartitionKey, input: I) -> Result<()> { + let partition_value = partition_key.data(); + + // Check if this partition has been closed already + if self.closed_partitions.contains(partition_value) { + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "The input is not sorted! Cannot write to partition that was previously closed: {:?}", + partition_key + ), + )); + } + + // Check if we need to switch to a new partition + let need_new_writer = match &self.current_partition { + Some(current) => current != partition_value, + None => true, + }; + + if need_new_writer { + self.close_current_writer().await?; + + // Create a new writer for the new partition + self.current_writer = Some( + self.inner_builder + .clone() + .build(Some(partition_key.clone())) + .await?, + ); + self.current_partition = Some(partition_value.clone()); + } + + // do write + self.current_writer + .as_mut() + .expect("Writer should be initialized") + .write(input) + .await + } + + async fn close(mut self) -> Result { + self.close_current_writer().await?; + + // Collect all output items into the output collection type + Ok(O::from_iter(self.output)) + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_schema::{DataType, Field, Schema}; + use parquet::arrow::PARQUET_FIELD_ID_META_KEY; + use parquet::file::properties::WriterProperties; + use tempfile::TempDir; + + use super::*; + use crate::io::FileIOBuilder; + use crate::spec::{DataFileFormat, NestedField, PrimitiveType, Type}; + use crate::writer::base_writer::data_file_writer::DataFileWriterBuilder; + use crate::writer::file_writer::ParquetWriterBuilder; + use crate::writer::file_writer::location_generator::{ + DefaultFileNameGenerator, DefaultLocationGenerator, + }; + use crate::writer::file_writer::rolling_writer::RollingFileWriterBuilder; + + #[tokio::test] + async fn test_clustered_writer_single_partition() -> Result<()> { + let temp_dir = TempDir::new()?; + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + + // Create schema with partition field + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + ); + + // Create partition spec and key + let partition_spec = crate::spec::PartitionSpec::builder(schema.clone()).build()?; + let partition_value = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("US"))]); + let partition_key = + crate::spec::PartitionKey::new(partition_spec, schema.clone(), partition_value.clone()); + + // Create writer builder + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + + // Create rolling file writer builder + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io.clone(), + location_gen, + file_name_gen, + ); + + // Create data file writer builder + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + // Create clustered writer + let mut writer = ClusteredWriter::new(data_file_writer_builder); + + // Create test data with proper field ID metadata + let arrow_schema = Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 1.to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 2.to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 3.to_string(), + )])), + ]); + + let batch1 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + let batch2 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + // Write data to the same partition (this should work) + writer.write(partition_key.clone(), batch1).await?; + writer.write(partition_key.clone(), batch2).await?; + + // Close writer and get data files + let data_files = writer.close().await?; + + // Verify at least one file was created + assert!( + !data_files.is_empty(), + "Expected at least one data file to be created" + ); + + // Verify that all data files have the correct partition value + for data_file in &data_files { + assert_eq!(data_file.partition, partition_value); + } + + Ok(()) + } + + #[tokio::test] + async fn test_clustered_writer_sorted_partitions() -> Result<()> { + let temp_dir = TempDir::new()?; + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + + // Create schema with partition field + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + ); + + // Create partition spec + let partition_spec = crate::spec::PartitionSpec::builder(schema.clone()).build()?; + + // Create partition keys for different regions (in sorted order) + let partition_value_asia = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("ASIA"))]); + let partition_key_asia = crate::spec::PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_asia.clone(), + ); + + let partition_value_eu = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("EU"))]); + let partition_key_eu = crate::spec::PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_eu.clone(), + ); + + let partition_value_us = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("US"))]); + let partition_key_us = crate::spec::PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_us.clone(), + ); + + // Create writer builder + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + + // Create rolling file writer builder + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io.clone(), + location_gen, + file_name_gen, + ); + + // Create data file writer builder + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + // Create clustered writer + let mut writer = ClusteredWriter::new(data_file_writer_builder); + + // Create test data with proper field ID metadata + let arrow_schema = Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 1.to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 2.to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 3.to_string(), + )])), + ]); + + // Create batches for different partitions (in sorted order) + let batch_asia = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + Arc::new(StringArray::from(vec!["ASIA", "ASIA"])), + ])?; + + let batch_eu = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["EU", "EU"])), + ])?; + + let batch_us = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![5, 6])), + Arc::new(StringArray::from(vec!["Eve", "Frank"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + // Write data in sorted partition order (this should work) + writer.write(partition_key_asia.clone(), batch_asia).await?; + writer.write(partition_key_eu.clone(), batch_eu).await?; + writer.write(partition_key_us.clone(), batch_us).await?; + + // Close writer and get data files + let data_files = writer.close().await?; + + // Verify files were created for all partitions + assert!( + data_files.len() >= 3, + "Expected at least 3 data files (one per partition), got {}", + data_files.len() + ); + + // Verify that we have files for each partition + let mut partitions_found = std::collections::HashSet::new(); + for data_file in &data_files { + partitions_found.insert(data_file.partition.clone()); + } + + assert!( + partitions_found.contains(&partition_value_asia), + "Missing ASIA partition" + ); + assert!( + partitions_found.contains(&partition_value_eu), + "Missing EU partition" + ); + assert!( + partitions_found.contains(&partition_value_us), + "Missing US partition" + ); + + Ok(()) + } + + #[tokio::test] + async fn test_clustered_writer_unsorted_partitions_error() -> Result<()> { + let temp_dir = TempDir::new()?; + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + + // Create schema with partition field + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + ); + + // Create partition spec + let partition_spec = crate::spec::PartitionSpec::builder(schema.clone()).build()?; + + // Create partition keys for different regions + let partition_value_us = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("US"))]); + let partition_key_us = crate::spec::PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_us.clone(), + ); + + let partition_value_eu = + crate::spec::Struct::from_iter([Some(crate::spec::Literal::string("EU"))]); + let partition_key_eu = crate::spec::PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_eu.clone(), + ); + + // Create writer builder + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + + // Create rolling file writer builder + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io.clone(), + location_gen, + file_name_gen, + ); + + // Create data file writer builder + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + // Create clustered writer + let mut writer = ClusteredWriter::new(data_file_writer_builder); + + // Create test data with proper field ID metadata + let arrow_schema = Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 1.to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 2.to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 3.to_string(), + )])), + ]); + + // Create batches for different partitions + let batch_us = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + let batch_eu = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["EU", "EU"])), + ])?; + + let batch_us2 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![5])), + Arc::new(StringArray::from(vec!["Eve"])), + Arc::new(StringArray::from(vec!["US"])), + ])?; + + // Write data to US partition first + writer.write(partition_key_us.clone(), batch_us).await?; + + // Write data to EU partition (this closes US partition) + writer.write(partition_key_eu.clone(), batch_eu).await?; + + // Try to write to US partition again - this should fail because data is not sorted + let result = writer.write(partition_key_us.clone(), batch_us2).await; + + assert!(result.is_err(), "Expected error when writing unsorted data"); + + let error = result.unwrap_err(); + assert!( + error.to_string().contains("The input is not sorted"), + "Expected 'input is not sorted' error, got: {}", + error + ); + + Ok(()) + } +} diff --git a/crates/iceberg/src/writer/partitioning/fanout_writer.rs b/crates/iceberg/src/writer/partitioning/fanout_writer.rs new file mode 100644 index 0000000000..796c1a4888 --- /dev/null +++ b/crates/iceberg/src/writer/partitioning/fanout_writer.rs @@ -0,0 +1,385 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! This module provides the `FanoutWriter` implementation. + +use std::collections::HashMap; +use std::marker::PhantomData; + +use async_trait::async_trait; + +use crate::spec::{PartitionKey, Struct}; +use crate::writer::partitioning::PartitioningWriter; +use crate::writer::{DefaultInput, DefaultOutput, IcebergWriter, IcebergWriterBuilder}; +use crate::{Error, ErrorKind, Result}; + +/// A writer that can write data to multiple partitions simultaneously. +/// +/// Unlike `ClusteredWriter` which expects sorted input and maintains only one active writer, +/// `FanoutWriter` can handle unsorted data by maintaining multiple active writers in a map. +/// This allows writing to any partition at any time, but uses more memory as all writers +/// remain active until the writer is closed. +/// +/// # Type Parameters +/// +/// * `B` - The inner writer builder type +/// * `I` - Input type (defaults to `RecordBatch`) +/// * `O` - Output collection type (defaults to `Vec`) +pub struct FanoutWriter +where + B: IcebergWriterBuilder, + O: IntoIterator + FromIterator<::Item>, + ::Item: Clone, +{ + inner_builder: B, + partition_writers: HashMap, + output: Vec<::Item>, + _phantom: PhantomData, +} + +impl FanoutWriter +where + B: IcebergWriterBuilder, + I: Send + 'static, + O: IntoIterator + FromIterator<::Item>, + ::Item: Send + Clone, +{ + /// Create a new `FanoutWriter`. + pub fn new(inner_builder: B) -> Self { + Self { + inner_builder, + partition_writers: HashMap::new(), + output: Vec::new(), + _phantom: PhantomData, + } + } + + /// Get or create a writer for the specified partition. + async fn get_or_create_writer(&mut self, partition_key: &PartitionKey) -> Result<&mut B::R> { + if !self.partition_writers.contains_key(partition_key.data()) { + let writer = self + .inner_builder + .clone() + .build(Some(partition_key.clone())) + .await?; + self.partition_writers + .insert(partition_key.data().clone(), writer); + } + + self.partition_writers + .get_mut(partition_key.data()) + .ok_or_else(|| { + Error::new( + ErrorKind::Unexpected, + "Failed to get partition writer after creation", + ) + }) + } +} + +#[async_trait] +impl PartitioningWriter for FanoutWriter +where + B: IcebergWriterBuilder, + I: Send + 'static, + O: IntoIterator + FromIterator<::Item> + Send + 'static, + ::Item: Send + Clone, +{ + async fn write(&mut self, partition_key: PartitionKey, input: I) -> Result<()> { + let writer = self.get_or_create_writer(&partition_key).await?; + writer.write(input).await + } + + async fn close(mut self) -> Result { + // Close all partition writers + for (_, mut writer) in self.partition_writers { + self.output.extend(writer.close().await?); + } + + // Collect all output items into the output collection type + Ok(O::from_iter(self.output)) + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_schema::{DataType, Field, Schema}; + use parquet::arrow::PARQUET_FIELD_ID_META_KEY; + use parquet::file::properties::WriterProperties; + use tempfile::TempDir; + + use super::*; + use crate::io::FileIOBuilder; + use crate::spec::{ + DataFileFormat, Literal, NestedField, PartitionKey, PartitionSpec, PrimitiveType, Struct, + Type, + }; + use crate::writer::base_writer::data_file_writer::DataFileWriterBuilder; + use crate::writer::file_writer::ParquetWriterBuilder; + use crate::writer::file_writer::location_generator::{ + DefaultFileNameGenerator, DefaultLocationGenerator, + }; + use crate::writer::file_writer::rolling_writer::RollingFileWriterBuilder; + + #[tokio::test] + async fn test_fanout_writer_single_partition() -> Result<()> { + let temp_dir = TempDir::new()?; + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + + // Create schema with partition field + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + ); + + // Create partition spec - using the same pattern as data_file_writer tests + let partition_spec = PartitionSpec::builder(schema.clone()).build()?; + let partition_value = Struct::from_iter([Some(Literal::string("US"))]); + let partition_key = + PartitionKey::new(partition_spec, schema.clone(), partition_value.clone()); + + // Create writer builder + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + + // Create rolling file writer builder + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io.clone(), + location_gen, + file_name_gen, + ); + + // Create data file writer builder + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + // Create fanout writer + let mut writer = FanoutWriter::new(data_file_writer_builder); + + // Create test data with proper field ID metadata + let arrow_schema = Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 1.to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 2.to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 3.to_string(), + )])), + ]); + + let batch1 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + let batch2 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + // Write data to the same partition + writer.write(partition_key.clone(), batch1).await?; + writer.write(partition_key.clone(), batch2).await?; + + // Close writer and get data files + let data_files = writer.close().await?; + + // Verify at least one file was created + assert!( + !data_files.is_empty(), + "Expected at least one data file to be created" + ); + + // Verify that all data files have the correct partition value + for data_file in &data_files { + assert_eq!(data_file.partition, partition_value); + } + + Ok(()) + } + + #[tokio::test] + async fn test_fanout_writer_multiple_partitions() -> Result<()> { + let temp_dir = TempDir::new()?; + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + + // Create schema with partition field + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + ); + + // Create partition spec + let partition_spec = PartitionSpec::builder(schema.clone()).build()?; + + // Create partition keys for different regions + let partition_value_us = Struct::from_iter([Some(Literal::string("US"))]); + let partition_key_us = PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_us.clone(), + ); + + let partition_value_eu = Struct::from_iter([Some(Literal::string("EU"))]); + let partition_key_eu = PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_eu.clone(), + ); + + let partition_value_asia = Struct::from_iter([Some(Literal::string("ASIA"))]); + let partition_key_asia = PartitionKey::new( + partition_spec.clone(), + schema.clone(), + partition_value_asia.clone(), + ); + + // Create writer builder + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + + // Create rolling file writer builder + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io.clone(), + location_gen, + file_name_gen, + ); + + // Create data file writer builder + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + // Create fanout writer + let mut writer = FanoutWriter::new(data_file_writer_builder); + + // Create test data with proper field ID metadata + let arrow_schema = Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 1.to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 2.to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + 3.to_string(), + )])), + ]); + + // Create batches for different partitions + let batch_us1 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + Arc::new(StringArray::from(vec!["US", "US"])), + ])?; + + let batch_eu1 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["EU", "EU"])), + ])?; + + let batch_us2 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![5])), + Arc::new(StringArray::from(vec!["Eve"])), + Arc::new(StringArray::from(vec!["US"])), + ])?; + + let batch_asia1 = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![ + Arc::new(Int32Array::from(vec![6, 7])), + Arc::new(StringArray::from(vec!["Frank", "Grace"])), + Arc::new(StringArray::from(vec!["ASIA", "ASIA"])), + ])?; + + // Write data in mixed partition order to demonstrate fanout capability + // This is the key difference from ClusteredWriter - we can write to any partition at any time + writer.write(partition_key_us.clone(), batch_us1).await?; + writer.write(partition_key_eu.clone(), batch_eu1).await?; + writer.write(partition_key_us.clone(), batch_us2).await?; // Back to US partition + writer + .write(partition_key_asia.clone(), batch_asia1) + .await?; + + // Close writer and get data files + let data_files = writer.close().await?; + + // Verify files were created for all partitions + assert!( + data_files.len() >= 3, + "Expected at least 3 data files (one per partition), got {}", + data_files.len() + ); + + // Verify that we have files for each partition + let mut partitions_found = std::collections::HashSet::new(); + for data_file in &data_files { + partitions_found.insert(data_file.partition.clone()); + } + + assert!( + partitions_found.contains(&partition_value_us), + "Missing US partition" + ); + assert!( + partitions_found.contains(&partition_value_eu), + "Missing EU partition" + ); + assert!( + partitions_found.contains(&partition_value_asia), + "Missing ASIA partition" + ); + + Ok(()) + } +} diff --git a/crates/iceberg/src/writer/partitioning/mod.rs b/crates/iceberg/src/writer/partitioning/mod.rs new file mode 100644 index 0000000000..36acb2a987 --- /dev/null +++ b/crates/iceberg/src/writer/partitioning/mod.rs @@ -0,0 +1,49 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +pub mod clustered_writer; +pub mod fanout_writer; + +use crate::Result; +use crate::spec::PartitionKey; +use crate::writer::{DefaultInput, DefaultOutput}; + +/// A writer that can write data to partitioned tables. +/// +/// This trait provides methods for writing data with partition keys and +/// closing the writer to retrieve the output. +#[async_trait::async_trait] +pub trait PartitioningWriter: Send + 'static { + /// Write data with a partition key. + /// + /// # Parameters + /// + /// * `partition_key` - Partition key to determine which partition to write to + /// * `input` - The input data to write + /// + /// # Returns + /// + /// `Ok(())` on success, or an error if the write operation fails. + async fn write(&mut self, partition_key: PartitionKey, input: I) -> Result<()>; + + /// Close the writer and return the output. + /// + /// # Returns + /// + /// The accumulated output from all write operations. + async fn close(self) -> Result; +} diff --git a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_data_file_test.rs index f4cba959ee..bedc975102 100644 --- a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs +++ b/crates/integration_tests/tests/shared_tests/append_data_file_test.rs @@ -82,8 +82,8 @@ async fn test_append_data_file() { location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); - let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); + let mut data_file_writer = data_file_writer_builder.build(None).await.unwrap(); let col1 = StringArray::from(vec![Some("foo"), Some("bar"), None, Some("baz")]); let col2 = Int32Array::from(vec![Some(1), Some(2), Some(3), Some(4)]); let col3 = BooleanArray::from(vec![Some(true), Some(false), None, Some(false)]); diff --git a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs index 0da88f1a09..a305ec0842 100644 --- a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs +++ b/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs @@ -106,13 +106,11 @@ async fn test_append_partition_data_file() { file_name_generator.clone(), ); - let mut data_file_writer_valid = DataFileWriterBuilder::new( - rolling_file_writer_builder.clone(), - Some(partition_key.clone()), - ) - .build() - .await - .unwrap(); + let mut data_file_writer_valid = + DataFileWriterBuilder::new(rolling_file_writer_builder.clone()) + .build(Some(partition_key.clone())) + .await + .unwrap(); let col1 = StringArray::from(vec![Some("foo1"), Some("foo2")]); let col2 = Int32Array::from(vec![ @@ -191,11 +189,10 @@ async fn test_schema_incompatible_partition_type( catalog: &dyn Catalog, ) { // test writing different "type" of partition than mentioned in schema - let mut data_file_writer_invalid = - DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) - .build() - .await - .unwrap(); + let mut data_file_writer_invalid = DataFileWriterBuilder::new(rolling_file_writer_builder) + .build(Some(partition_key)) + .await + .unwrap(); data_file_writer_invalid.write(batch.clone()).await.unwrap(); let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); @@ -221,11 +218,10 @@ async fn test_schema_incompatible_partition_fields( catalog: &dyn Catalog, ) { // test writing different number of partition fields than mentioned in schema - let mut data_file_writer_invalid = - DataFileWriterBuilder::new(rolling_file_writer_builder, Some(partition_key)) - .build() - .await - .unwrap(); + let mut data_file_writer_invalid = DataFileWriterBuilder::new(rolling_file_writer_builder) + .build(Some(partition_key)) + .await + .unwrap(); data_file_writer_invalid.write(batch.clone()).await.unwrap(); let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); diff --git a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs b/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs index a248fa707a..fc529cc3d2 100644 --- a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs +++ b/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs @@ -81,8 +81,8 @@ async fn test_append_data_file_conflict() { location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); - let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); + let mut data_file_writer = data_file_writer_builder.build(None).await.unwrap(); let col1 = StringArray::from(vec![Some("foo"), Some("bar"), None, Some("baz")]); let col2 = Int32Array::from(vec![Some(1), Some(2), Some(3), Some(4)]); let col3 = BooleanArray::from(vec![Some(true), Some(false), None, Some(false)]); diff --git a/crates/integration_tests/tests/shared_tests/scan_all_type.rs b/crates/integration_tests/tests/shared_tests/scan_all_type.rs index 1125de11a5..7a2907d4cb 100644 --- a/crates/integration_tests/tests/shared_tests/scan_all_type.rs +++ b/crates/integration_tests/tests/shared_tests/scan_all_type.rs @@ -163,8 +163,8 @@ async fn test_scan_all_type() { location_generator.clone(), file_name_generator.clone(), ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder, None); - let mut data_file_writer = data_file_writer_builder.build().await.unwrap(); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); + let mut data_file_writer = data_file_writer_builder.build(None).await.unwrap(); // Prepare data let col1 = Int32Array::from(vec![1, 2, 3, 4, 5]); diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index 7bff239222..dff40a3c0d 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -262,6 +262,7 @@ impl ExecutionPlan for IcebergWriteExec { }; let file_io = self.table.file_io().clone(); + // todo location_gen and file_name_gen should be configurable let location_generator = DefaultLocationGenerator::new(self.table.metadata().clone()) .map_err(to_datafusion_error)?; // todo filename prefix/suffix should be configurable @@ -274,8 +275,7 @@ impl ExecutionPlan for IcebergWriteExec { location_generator, file_name_generator, ); - // todo specify partition key when partitioning writer is supported - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder, None); + let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); // Get input data let data = execute_input_stream( @@ -291,7 +291,8 @@ impl ExecutionPlan for IcebergWriteExec { // Create write stream let stream = futures::stream::once(async move { let mut writer = data_file_writer_builder - .build() + // todo specify partition key when partitioning writer is supported + .build(None) .await .map_err(to_datafusion_error)?; let mut input_stream = data; From 826d98f94675c8dad0faae838a543cd3f00fd2be Mon Sep 17 00:00:00 2001 From: Yoshiki Kudo <335g@users.noreply.github.com> Date: Fri, 17 Oct 2025 19:04:10 +0900 Subject: [PATCH 017/891] feat(catalog): impl builder for SqlCatalog (#1666) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #1626 ## What changes are included in this PR? - Introduced a new `SqlCatalogBuilder` implementation for `iceberg::CatalogBuilder`. - By transitioning the builder implementation from `SqlCatalogConfig` to `SqlCatalogBuilder`, we've simplified the process for creating `SqlCatalog` objects. - Enabled generation of `Box` for the `SqlCatalog` via the `load` function (#1260) ## Are these changes tested? Yes (only tested whether it could be made) --------- Co-authored-by: Yoshiki Kudo --- Cargo.lock | 6 +- Cargo.toml | 2 + crates/catalog/loader/Cargo.toml | 5 + crates/catalog/loader/src/lib.rs | 34 +++ crates/catalog/sql/Cargo.toml | 3 +- crates/catalog/sql/src/catalog.rs | 374 ++++++++++++++++++++++++++++-- crates/catalog/sql/src/lib.rs | 37 +++ 7 files changed, 434 insertions(+), 27 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 16ffef0654..87b97b7c78 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3712,6 +3712,9 @@ dependencies = [ "iceberg-catalog-hms", "iceberg-catalog-rest", "iceberg-catalog-s3tables", + "iceberg-catalog-sql", + "sqlx", + "tempfile", "tokio", ] @@ -3757,13 +3760,12 @@ version = "0.7.0" dependencies = [ "async-trait", "iceberg", - "iceberg_test_utils", "itertools 0.13.0", "regex", "sqlx", + "strum 0.27.2", "tempfile", "tokio", - "typed-builder 0.20.1", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 46c99cc3d8..7735d702cf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -78,6 +78,7 @@ http = "1.2" iceberg = { version = "0.7.0", path = "./crates/iceberg" } iceberg-catalog-glue = { version = "0.7.0", path = "./crates/catalog/glue" } iceberg-catalog-hms = { version = "0.7.0", path = "./crates/catalog/hms" } +iceberg-catalog-sql = { version = "0.7.0", path = "./crates/catalog/sql" } iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } @@ -111,6 +112,7 @@ serde_repr = "0.1.16" serde_with = "3.4" smol = "2.0.2" sqllogictest = "0.28.3" +sqlx = { version = "0.8.1", default-features = false } stacker = "0.1.20" strum = "0.27.2" tempfile = "3.18" diff --git a/crates/catalog/loader/Cargo.toml b/crates/catalog/loader/Cargo.toml index 014f84a3a1..d4b925fb94 100644 --- a/crates/catalog/loader/Cargo.toml +++ b/crates/catalog/loader/Cargo.toml @@ -34,5 +34,10 @@ iceberg-catalog-rest = { workspace = true } iceberg-catalog-glue = { workspace = true } iceberg-catalog-s3tables = { workspace = true } iceberg-catalog-hms = { workspace = true } +iceberg-catalog-sql = { workspace = true } tokio = { workspace = true } async-trait = { workspace = true } + +[dev-dependencies] +sqlx = { workspace = true, features = ["runtime-tokio", "sqlite", "migrate"] } +tempfile = { workspace = true } diff --git a/crates/catalog/loader/src/lib.rs b/crates/catalog/loader/src/lib.rs index 9c18ab4e5f..e118ef86a9 100644 --- a/crates/catalog/loader/src/lib.rs +++ b/crates/catalog/loader/src/lib.rs @@ -24,6 +24,7 @@ use iceberg_catalog_glue::GlueCatalogBuilder; use iceberg_catalog_hms::HmsCatalogBuilder; use iceberg_catalog_rest::RestCatalogBuilder; use iceberg_catalog_s3tables::S3TablesCatalogBuilder; +use iceberg_catalog_sql::SqlCatalogBuilder; /// A CatalogBuilderFactory creating a new catalog builder. type CatalogBuilderFactory = fn() -> Box; @@ -34,6 +35,7 @@ static CATALOG_REGISTRY: &[(&str, CatalogBuilderFactory)] = &[ ("glue", || Box::new(GlueCatalogBuilder::default())), ("s3tables", || Box::new(S3TablesCatalogBuilder::default())), ("hms", || Box::new(HmsCatalogBuilder::default())), + ("sql", || Box::new(SqlCatalogBuilder::default())), ]; /// Return the list of supported catalog types. @@ -108,6 +110,9 @@ impl CatalogLoader<'_> { mod tests { use std::collections::HashMap; + use sqlx::migrate::MigrateDatabase; + use tempfile::TempDir; + use crate::{CatalogLoader, load}; #[tokio::test] @@ -220,6 +225,35 @@ mod tests { assert!(catalog.is_ok()); } + fn temp_path() -> String { + let temp_dir = TempDir::new().unwrap(); + temp_dir.path().to_str().unwrap().to_string() + } + + #[tokio::test] + async fn test_catalog_loader_pattern_sql_catalog() { + use iceberg_catalog_sql::{SQL_CATALOG_PROP_URI, SQL_CATALOG_PROP_WAREHOUSE}; + + let uri = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&uri).await.unwrap(); + + let catalog_loader = load("sql").unwrap(); + let catalog = catalog_loader + .load( + "sql".to_string(), + HashMap::from([ + (SQL_CATALOG_PROP_URI.to_string(), uri), + ( + SQL_CATALOG_PROP_WAREHOUSE.to_string(), + "s3://warehouse".to_string(), + ), + ]), + ) + .await; + + assert!(catalog.is_ok()); + } + #[tokio::test] async fn test_error_message_includes_supported_types() { let err = match load("does-not-exist") { diff --git a/crates/catalog/sql/Cargo.toml b/crates/catalog/sql/Cargo.toml index 33ca700bf7..262422c284 100644 --- a/crates/catalog/sql/Cargo.toml +++ b/crates/catalog/sql/Cargo.toml @@ -32,10 +32,9 @@ repository = { workspace = true } async-trait = { workspace = true } iceberg = { workspace = true } sqlx = { version = "0.8.1", features = ["any"], default-features = false } -typed-builder = { workspace = true } +strum = { workspace = true } [dev-dependencies] -iceberg_test_utils = { path = "../../test_utils", features = ["tests"] } itertools = { workspace = true } regex = "1.10.5" sqlx = { version = "0.8.1", features = [ diff --git a/crates/catalog/sql/src/catalog.rs b/crates/catalog/sql/src/catalog.rs index c1040c65a0..b6dc4f835b 100644 --- a/crates/catalog/sql/src/catalog.rs +++ b/crates/catalog/sql/src/catalog.rs @@ -16,6 +16,7 @@ // under the License. use std::collections::{HashMap, HashSet}; +use std::str::FromStr; use std::time::Duration; use async_trait::async_trait; @@ -23,17 +24,23 @@ use iceberg::io::FileIO; use iceberg::spec::{TableMetadata, TableMetadataBuilder}; use iceberg::table::Table; use iceberg::{ - Catalog, Error, ErrorKind, MetadataLocation, Namespace, NamespaceIdent, Result, TableCommit, - TableCreation, TableIdent, + Catalog, CatalogBuilder, Error, ErrorKind, MetadataLocation, Namespace, NamespaceIdent, Result, + TableCommit, TableCreation, TableIdent, }; use sqlx::any::{AnyPoolOptions, AnyQueryResult, AnyRow, install_default_drivers}; use sqlx::{Any, AnyPool, Row, Transaction}; -use typed_builder::TypedBuilder; use crate::error::{ from_sqlx_error, no_such_namespace_err, no_such_table_err, table_already_exists_err, }; +/// catalog URI +pub const SQL_CATALOG_PROP_URI: &str = "uri"; +/// catalog warehouse location +pub const SQL_CATALOG_PROP_WAREHOUSE: &str = "warehouse"; +/// catalog sql bind style +pub const SQL_CATALOG_PROP_BIND_STYLE: &str = "sql_bind_style"; + static CATALOG_TABLE_NAME: &str = "iceberg_tables"; static CATALOG_FIELD_CATALOG_NAME: &str = "catalog_name"; static CATALOG_FIELD_TABLE_NAME: &str = "table_name"; @@ -54,6 +61,126 @@ static MAX_CONNECTIONS: u32 = 10; // Default the SQL pool to 10 connections if n static IDLE_TIMEOUT: u64 = 10; // Default the maximum idle timeout per connection to 10s before it is closed static TEST_BEFORE_ACQUIRE: bool = true; // Default the health-check of each connection to enabled prior to returning +/// Builder for [`SqlCatalog`] +#[derive(Debug)] +pub struct SqlCatalogBuilder(SqlCatalogConfig); + +impl Default for SqlCatalogBuilder { + fn default() -> Self { + Self(SqlCatalogConfig { + uri: "".to_string(), + name: "".to_string(), + warehouse_location: "".to_string(), + sql_bind_style: SqlBindStyle::DollarNumeric, + props: HashMap::new(), + }) + } +} + +impl SqlCatalogBuilder { + /// Configure the database URI + /// + /// If `SQL_CATALOG_PROP_URI` has a value set in `props` during `SqlCatalogBuilder::load`, + /// that value takes precedence, and the value specified by this method will not be used. + pub fn uri(mut self, uri: impl Into) -> Self { + self.0.uri = uri.into(); + self + } + + /// Configure the warehouse location + /// + /// If `SQL_CATALOG_PROP_WAREHOUSE` has a value set in `props` during `SqlCatalogBuilder::load`, + /// that value takes precedence, and the value specified by this method will not be used. + pub fn warehouse_location(mut self, location: impl Into) -> Self { + self.0.warehouse_location = location.into(); + self + } + + /// Configure the bound SQL Statement + /// + /// If `SQL_CATALOG_PROP_BIND_STYLE` has a value set in `props` during `SqlCatalogBuilder::load`, + /// that value takes precedence, and the value specified by this method will not be used. + pub fn sql_bind_style(mut self, sql_bind_style: SqlBindStyle) -> Self { + self.0.sql_bind_style = sql_bind_style; + self + } + + /// Configure the any properties + /// + /// If the same key has values set in `props` during `SqlCatalogBuilder::load`, + /// those values will take precedence. + pub fn props(mut self, props: HashMap) -> Self { + for (k, v) in props { + self.0.props.insert(k, v); + } + self + } + + /// Set a new property on the property to be configured. + /// When multiple methods are executed with the same key, + /// the later-set value takes precedence. + /// + /// If the same key has values set in `props` during `SqlCatalogBuilder::load`, + /// those values will take precedence. + pub fn prop(mut self, key: impl Into, value: impl Into) -> Self { + self.0.props.insert(key.into(), value.into()); + self + } +} + +impl CatalogBuilder for SqlCatalogBuilder { + type C = SqlCatalog; + + fn load( + mut self, + name: impl Into, + props: HashMap, + ) -> impl Future> + Send { + let name = name.into(); + + for (k, v) in props { + self.0.props.insert(k, v); + } + + if let Some(uri) = self.0.props.remove(SQL_CATALOG_PROP_URI) { + self.0.uri = uri; + } + if let Some(warehouse_location) = self.0.props.remove(SQL_CATALOG_PROP_WAREHOUSE) { + self.0.warehouse_location = warehouse_location; + } + + let mut valid_sql_bind_style = true; + if let Some(sql_bind_style) = self.0.props.remove(SQL_CATALOG_PROP_BIND_STYLE) { + if let Ok(sql_bind_style) = SqlBindStyle::from_str(&sql_bind_style) { + self.0.sql_bind_style = sql_bind_style; + } else { + valid_sql_bind_style = false; + } + } + + async move { + if name.trim().is_empty() { + Err(Error::new( + ErrorKind::DataInvalid, + "Catalog name cannot be empty", + )) + } else if !valid_sql_bind_style { + Err(Error::new( + ErrorKind::DataInvalid, + format!( + "`{}` values are valid only if they're `{}` or `{}`", + SQL_CATALOG_PROP_BIND_STYLE, + SqlBindStyle::DollarNumeric, + SqlBindStyle::QMark + ), + )) + } else { + SqlCatalog::new(self.0).await + } + } + } +} + /// A struct representing the SQL catalog configuration. /// /// This struct contains various parameters that are used to configure a SQL catalog, @@ -62,14 +189,12 @@ static TEST_BEFORE_ACQUIRE: bool = true; // Default the health-check of each con /// The options available for this parameter include: /// - `SqlBindStyle::DollarNumeric`: Binds SQL statements using `$1`, `$2`, etc., as placeholders. This is for PostgreSQL databases. /// - `SqlBindStyle::QuestionMark`: Binds SQL statements using `?` as a placeholder. This is for MySQL and SQLite databases. -#[derive(Debug, TypedBuilder)] -pub struct SqlCatalogConfig { +#[derive(Debug)] +struct SqlCatalogConfig { uri: String, name: String, warehouse_location: String, - file_io: FileIO, sql_bind_style: SqlBindStyle, - #[builder(default)] props: HashMap, } @@ -83,7 +208,7 @@ pub struct SqlCatalog { sql_bind_style: SqlBindStyle, } -#[derive(Debug, PartialEq)] +#[derive(Debug, PartialEq, strum::EnumString, strum::Display)] /// Set the SQL parameter bind style to either $1..$N (Postgres style) or ? (SQLite/MySQL/MariaDB) pub enum SqlBindStyle { /// DollarNumeric uses parameters of the form `$1..$N``, which is the Postgres style @@ -94,7 +219,8 @@ pub enum SqlBindStyle { impl SqlCatalog { /// Create new sql catalog instance - pub async fn new(config: SqlCatalogConfig) -> Result { + async fn new(config: SqlCatalogConfig) -> Result { + let fileio = FileIO::from_path(&config.warehouse_location)?.build()?; install_default_drivers(); let max_connections: u32 = config .props @@ -150,7 +276,7 @@ impl SqlCatalog { name: config.name.to_owned(), connection: pool, warehouse_location: config.warehouse_location, - fileio: config.file_io, + fileio, sql_bind_style: config.sql_bind_style, }) } @@ -804,17 +930,19 @@ mod tests { use std::collections::{HashMap, HashSet}; use std::hash::Hash; - use iceberg::io::FileIOBuilder; use iceberg::spec::{NestedField, PartitionSpec, PrimitiveType, Schema, SortOrder, Type}; use iceberg::table::Table; - use iceberg::{Catalog, Namespace, NamespaceIdent, TableCreation, TableIdent}; + use iceberg::{Catalog, CatalogBuilder, Namespace, NamespaceIdent, TableCreation, TableIdent}; use itertools::Itertools; use regex::Regex; use sqlx::migrate::MigrateDatabase; use tempfile::TempDir; - use crate::catalog::NAMESPACE_LOCATION_PROPERTY_KEY; - use crate::{SqlBindStyle, SqlCatalog, SqlCatalogConfig}; + use crate::catalog::{ + NAMESPACE_LOCATION_PROPERTY_KEY, SQL_CATALOG_PROP_BIND_STYLE, SQL_CATALOG_PROP_URI, + SQL_CATALOG_PROP_WAREHOUSE, + }; + use crate::{SqlBindStyle, SqlCatalogBuilder}; const UUID_REGEX_STR: &str = "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"; @@ -835,15 +963,18 @@ mod tests { let sql_lite_uri = format!("sqlite:{}", temp_path()); sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); - let config = SqlCatalogConfig::builder() - .uri(sql_lite_uri.to_string()) - .name("iceberg".to_string()) - .warehouse_location(warehouse_location) - .file_io(FileIOBuilder::new_fs_io().build().unwrap()) - .sql_bind_style(SqlBindStyle::QMark) - .build(); - - SqlCatalog::new(config).await.unwrap() + let props = HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri.to_string()), + (SQL_CATALOG_PROP_WAREHOUSE.to_string(), warehouse_location), + ( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::DollarNumeric.to_string(), + ), + ]); + SqlCatalogBuilder::default() + .load("iceberg", props) + .await + .unwrap() } async fn create_namespace(catalog: &C, namespace_ident: &NamespaceIdent) { @@ -942,6 +1073,203 @@ mod tests { new_sql_catalog(warehouse_loc.clone()).await; } + #[tokio::test] + async fn test_builder_method() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + + let catalog = SqlCatalogBuilder::default() + .uri(sql_lite_uri.to_string()) + .warehouse_location(warehouse_location.clone()) + .sql_bind_style(SqlBindStyle::QMark) + .load("iceberg", HashMap::default()) + .await; + assert!(catalog.is_ok()); + + let catalog = catalog.unwrap(); + assert!(catalog.warehouse_location == warehouse_location); + assert!(catalog.sql_bind_style == SqlBindStyle::QMark); + } + + /// Overwriting an sqlite database with a non-existent path causes + /// catalog generation to fail + #[tokio::test] + async fn test_builder_props_non_existent_path_fails() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + let sql_lite_uri2 = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + + let catalog = SqlCatalogBuilder::default() + .uri(sql_lite_uri) + .warehouse_location(warehouse_location) + .load( + "iceberg", + HashMap::from_iter([(SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri2)]), + ) + .await; + assert!(catalog.is_err()); + } + + /// Even when an invalid URI is specified in a builder method, + /// it can be successfully overridden with a valid URI in props + /// for catalog generation to succeed. + #[tokio::test] + async fn test_builder_props_set_valid_uri() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + let sql_lite_uri2 = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + + let catalog = SqlCatalogBuilder::default() + .uri(sql_lite_uri2) + .warehouse_location(warehouse_location) + .load( + "iceberg", + HashMap::from_iter([(SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri.clone())]), + ) + .await; + assert!(catalog.is_ok()); + } + + /// values assigned via props take precedence + #[tokio::test] + async fn test_builder_props_take_precedence() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + let warehouse_location2 = temp_path(); + + let catalog = SqlCatalogBuilder::default() + .warehouse_location(warehouse_location2) + .sql_bind_style(SqlBindStyle::DollarNumeric) + .load( + "iceberg", + HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri), + ( + SQL_CATALOG_PROP_WAREHOUSE.to_string(), + warehouse_location.clone(), + ), + ( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::QMark.to_string(), + ), + ]), + ) + .await; + + assert!(catalog.is_ok()); + + let catalog = catalog.unwrap(); + assert!(catalog.warehouse_location == warehouse_location); + assert!(catalog.sql_bind_style == SqlBindStyle::QMark); + } + + /// values assigned via props take precedence + #[tokio::test] + async fn test_builder_props_take_precedence_props() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + let sql_lite_uri2 = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + let warehouse_location2 = temp_path(); + + let props = HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri.clone()), + ( + SQL_CATALOG_PROP_WAREHOUSE.to_string(), + warehouse_location.clone(), + ), + ( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::QMark.to_string(), + ), + ]); + let props2 = HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri2.clone()), + ( + SQL_CATALOG_PROP_WAREHOUSE.to_string(), + warehouse_location2.clone(), + ), + ( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::DollarNumeric.to_string(), + ), + ]); + + let catalog = SqlCatalogBuilder::default() + .props(props2) + .load("iceberg", props) + .await; + + assert!(catalog.is_ok()); + + let catalog = catalog.unwrap(); + assert!(catalog.warehouse_location == warehouse_location); + assert!(catalog.sql_bind_style == SqlBindStyle::QMark); + } + + /// values assigned via props take precedence + #[tokio::test] + async fn test_builder_props_take_precedence_prop() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + let sql_lite_uri2 = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + let warehouse_location2 = temp_path(); + + let props = HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri.clone()), + ( + SQL_CATALOG_PROP_WAREHOUSE.to_string(), + warehouse_location.clone(), + ), + ( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::QMark.to_string(), + ), + ]); + + let catalog = SqlCatalogBuilder::default() + .prop(SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri2) + .prop(SQL_CATALOG_PROP_WAREHOUSE.to_string(), warehouse_location2) + .prop( + SQL_CATALOG_PROP_BIND_STYLE.to_string(), + SqlBindStyle::DollarNumeric.to_string(), + ) + .load("iceberg", props) + .await; + + assert!(catalog.is_ok()); + + let catalog = catalog.unwrap(); + assert!(catalog.warehouse_location == warehouse_location); + assert!(catalog.sql_bind_style == SqlBindStyle::QMark); + } + + /// invalid value for `SqlBindStyle` causes catalog creation to fail + #[tokio::test] + async fn test_builder_props_invalid_bind_style_fails() { + let sql_lite_uri = format!("sqlite:{}", temp_path()); + sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); + let warehouse_location = temp_path(); + + let catalog = SqlCatalogBuilder::default() + .load( + "iceberg", + HashMap::from_iter([ + (SQL_CATALOG_PROP_URI.to_string(), sql_lite_uri), + (SQL_CATALOG_PROP_WAREHOUSE.to_string(), warehouse_location), + (SQL_CATALOG_PROP_BIND_STYLE.to_string(), "AAA".to_string()), + ]), + ) + .await; + + assert!(catalog.is_err()); + } + #[tokio::test] async fn test_list_namespaces_returns_empty_vector() { let warehouse_loc = temp_path(); diff --git a/crates/catalog/sql/src/lib.rs b/crates/catalog/sql/src/lib.rs index 6861dab3f8..b76006ed3b 100644 --- a/crates/catalog/sql/src/lib.rs +++ b/crates/catalog/sql/src/lib.rs @@ -16,6 +16,43 @@ // under the License. //! Iceberg sql catalog implementation. +//! +//! To build a sql catalog with configurations +//! # Example +//! +//! ```rust, no_run +//! use std::collections::HashMap; +//! +//! use iceberg::CatalogBuilder; +//! use iceberg_catalog_sql::{ +//! SQL_CATALOG_PROP_BIND_STYLE, SQL_CATALOG_PROP_URI, SQL_CATALOG_PROP_WAREHOUSE, +//! SqlBindStyle, SqlCatalogBuilder, +//! }; +//! +//! #[tokio::main] +//! async fn main() { +//! let catalog = SqlCatalogBuilder::default() +//! .load( +//! "sql", +//! HashMap::from_iter([ +//! ( +//! SQL_CATALOG_PROP_URI.to_string(), +//! "http://localhost:8181".to_string(), +//! ), +//! ( +//! SQL_CATALOG_PROP_WAREHOUSE.to_string(), +//! "s3://warehouse".to_string(), +//! ), +//! ( +//! SQL_CATALOG_PROP_BIND_STYLE.to_string(), +//! SqlBindStyle::QMark.to_string(), +//! ), +//! ]), +//! ) +//! .await +//! .unwrap(); +//! } +//! ``` #![deny(missing_docs)] From fa07ec6f26c785de332c75e4f19d87040d543dc3 Mon Sep 17 00:00:00 2001 From: Dylan Date: Fri, 17 Oct 2025 19:03:15 +0800 Subject: [PATCH 018/891] fix: fix read parquert file when schema change (#1750) --- crates/iceberg/src/arrow/reader.rs | 129 ++++++++++++++++++++++++----- 1 file changed, 106 insertions(+), 23 deletions(-) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 564b281107..ff4cff0a64 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -583,35 +583,25 @@ impl ArrowReader { true }); - if column_map.len() != leaf_field_ids.len() { - let missing_fields = leaf_field_ids - .iter() - .filter(|field_id| !column_map.contains_key(field_id)) - .collect::>(); - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Parquet schema {} and Iceberg schema {} do not match.", - iceberg_schema, iceberg_schema_of_task - ), - ) - .with_context("column_map", format! {"{:?}", column_map}) - .with_context("field_ids", format! {"{:?}", leaf_field_ids}) - .with_context("missing_fields", format! {"{:?}", missing_fields})); - } - + // Only project columns that exist in the Parquet file. + // Missing columns will be added by RecordBatchTransformer with default/NULL values. + // This supports schema evolution where new columns are added to the table schema + // but old Parquet files don't have them yet. let mut indices = vec![]; for field_id in leaf_field_ids { if let Some(col_idx) = column_map.get(&field_id) { indices.push(*col_idx); - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!("Field {} is not found in Parquet schema.", field_id), - )); } + // Skip fields that don't exist in the Parquet file - they will be added later + } + + if indices.is_empty() { + // If no columns from the projection exist in the file, project all columns + // This can happen if all requested columns are new and need to be added by the transformer + Ok(ProjectionMask::all()) + } else { + Ok(ProjectionMask::leaves(parquet_schema, indices)) } - Ok(ProjectionMask::leaves(parquet_schema, indices)) } } @@ -1958,4 +1948,97 @@ message schema { Arc::new(SchemaDescriptor::new(Arc::new(schema))) } + + /// Test schema evolution: reading old Parquet file (with only column 'a') + /// using a newer table schema (with columns 'a' and 'b'). + /// This tests that: + /// 1. get_arrow_projection_mask allows missing columns + /// 2. RecordBatchTransformer adds missing column 'b' with NULL values + #[tokio::test] + async fn test_schema_evolution_add_column() { + use arrow_array::{Array, Int32Array}; + + // New table schema: columns 'a' and 'b' (b was added later, file only has 'a') + let new_schema = Arc::new( + Schema::builder() + .with_schema_id(2) + .with_fields(vec![ + NestedField::required(1, "a", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "b", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + // Create Arrow schema for old Parquet file (only has column 'a') + let arrow_schema_old = Arc::new(ArrowSchema::new(vec![ + Field::new("a", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + ])); + + // Write old Parquet file with only column 'a' + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let data_a = Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef; + let to_write = RecordBatch::try_new(arrow_schema_old.clone(), vec![data_a]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + let file = File::create(format!("{}/old_file.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + // Read the old Parquet file using the NEW schema (with column 'b') + let reader = ArrowReaderBuilder::new(file_io).build(); + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/old_file.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: new_schema.clone(), + project_field_ids: vec![1, 2], // Request both columns 'a' and 'b' + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Verify we got the correct data + assert_eq!(result.len(), 1); + let batch = &result[0]; + + // Should have 2 columns now + assert_eq!(batch.num_columns(), 2); + assert_eq!(batch.num_rows(), 3); + + // Column 'a' should have the original data + let col_a = batch + .column(0) + .as_primitive::(); + assert_eq!(col_a.values(), &[1, 2, 3]); + + // Column 'b' should be all NULLs (it didn't exist in the old file) + let col_b = batch + .column(1) + .as_primitive::(); + assert_eq!(col_b.null_count(), 3); + assert!(col_b.is_null(0)); + assert!(col_b.is_null(1)); + assert!(col_b.is_null(2)); + } } From 790e0253a6d1ba83a54f51cf8a8ed62a5c2da7f5 Mon Sep 17 00:00:00 2001 From: Peter Nguyen Date: Fri, 17 Oct 2025 20:13:21 -0700 Subject: [PATCH 019/891] docs: Fix broken orbstack and podman links in CONTRIBUTING.md (#1757) ## Which issue does this PR close? - Closes #1756 ## What changes are included in this PR? Fix broken orbstack and podman links in CONTRIBUTING.md ## Are these changes tested? N/A --- CONTRIBUTING.md | 4 ++-- website/src/reference/orbstack.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e4b175b055..0d05c3ddac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -103,8 +103,8 @@ cargo 1.69.0 (6e9a83356 2023-04-12) Currently, iceberg-rust uses Docker to set up environment for integration tests. Native Docker has some limitations, please check (https://github.com/apache/iceberg-rust/pull/748). Please use Orbstack or Podman. -For MacOS users, you can install [OrbStack as a docker alternative](reference/orbstack.md). -For Podman users, refer to [Using Podman instead of Docker](reference/podman.md) +For MacOS users, you can install [OrbStack as a docker alternative](website/src/reference/orbstack.md). +For Podman users, refer to [Using Podman instead of Docker](website/src/reference/podman.md) ## Build diff --git a/website/src/reference/orbstack.md b/website/src/reference/orbstack.md index cb00849d6b..cbd6566359 100644 --- a/website/src/reference/orbstack.md +++ b/website/src/reference/orbstack.md @@ -25,7 +25,7 @@ 2. Migrate Docker data ```shell - orbstack migrate docker + orb migrate docker ``` 3. (Optional) Add registry mirrors @@ -36,4 +36,4 @@ { "registry-mirrors": [""] } - ``` \ No newline at end of file + ``` From 2861dabab8931ebb0e07bcf47c47990355b7ad55 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 19 Oct 2025 16:56:17 -0400 Subject: [PATCH 020/891] chore(deps): Bump tokio from 1.47.1 to 1.48.0 (#1763) Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.47.1 to 1.48.0.
Release notes

Sourced from tokio's releases.

Tokio v1.48.0

1.48.0 (October 14th, 2025)

The MSRV is increased to 1.71.

Added

  • fs: add File::max_buf_size (#7594)
  • io: export Chain of AsyncReadExt::chain (#7599)
  • net: add SocketAddr::as_abstract_name (#7491)
  • net: add TcpStream::quickack and TcpStream::set_quickack (#7490)
  • net: implement AsRef<Self> for TcpStream and UnixStream (#7573)
  • task: add LocalKey::try_get (#7666)
  • task: implement Ord for task::Id (#7530)

Changed

  • deps: bump windows-sys to version 0.61 (#7645)
  • fs: preserve max_buf_size when cloning a File (#7593)
  • macros: suppress clippy::unwrap_in_result in #[tokio::main] (#7651)
  • net: remove PollEvented noise from Debug formats (#7675)
  • process: upgrade Command::spawn_with to use FnOnce (#7511)
  • sync: remove inner mutex in SetOnce (#7554)
  • sync: use UnsafeCell::get_mut in Mutex::get_mut and RwLock::get_mut (#7569)
  • time: reduce the generated code size of Timeout<T>::poll (#7535)

Fixed

  • macros: fix hygiene issue in join! and try_join! (#7638)
  • net: fix copy/paste errors in udp peek methods (#7604)
  • process: fix error when runtime is shut down on nightly-2025-10-12 (#7672)
  • runtime: use release ordering in wake_by_ref() even if already woken (#7622)
  • sync: close the broadcast::Sender in broadcast::Sender::new() (#7629)
  • sync: fix implementation of unused RwLock::try_* methods (#7587)

Unstable

  • tokio: use cargo features instead of --cfg flags for taskdump and io_uring (#7655, #7621)
  • fs: support io_uring in fs::write (#7567)
  • fs: support io_uring with File::open() (#7617)
  • fs: support io_uring with OpenOptions (#7321)
  • macros: add local runtime flavor (#7375, #7597)

Documented

  • io: clarify the zero capacity case of AsyncRead::poll_read (#7580)
  • io: fix typos in the docs of AsyncFd readiness guards (#7583)
  • net: clarify socket gets closed on drop (#7526)
  • net: clarify the behavior of UCred::pid() on Cygwin (#7611)
  • net: clarify the supported platform of set_reuseport() and reuseport() (#7628)

... (truncated)

Commits
  • 556820f chore: prepare Tokio v1.48.0 (#7677)
  • fd1659a chore: prepare tokio-macros v2.6.0 (#7676)
  • 53e8aca ci: update nightly version to 2025-10-12 (#7670)
  • 9e5527d process: fix error when runtime is shut down on nightly-2025-10-12 (#7672)
  • 25a24de net: remove PollEvented noise from Debug formats (#7675)
  • c1fa25f task: clarify the behavior of several spawn_local methods (#7669)
  • e7e02fc fs: use FileOptions inside fs::File to support uring (#7617)
  • f7a7f62 ci: remove cargo-deny Unicode-DFS-2016 license exception config (#7619)
  • d1f1499 tokio: use cargo feature for taskdump support instead of cfg (#7655)
  • ad6f618 runtime: clarify the behavior of Handle::block_on (#7665)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tokio&package-manager=cargo&previous-version=1.47.1&new-version=1.48.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 83 ++++++++---------------------------------------------- 1 file changed, 12 insertions(+), 71 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 87b97b7c78..6a2a33238f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,15 +2,6 @@ # It is not intended for manual editing. version = 4 -[[package]] -name = "addr2line" -version = "0.24.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1" -dependencies = [ - "gimli", -] - [[package]] name = "adler2" version = "2.0.1" @@ -1060,21 +1051,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "backtrace" -version = "0.3.75" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6806a6321ec58106fea15becdad98371e28d92ccbc7c8f1b3b6dd724fe8f1002" -dependencies = [ - "addr2line", - "cfg-if", - "libc", - "miniz_oxide", - "object", - "rustc-demangle", - "windows-targets 0.52.6", -] - [[package]] name = "base64" version = "0.21.7" @@ -1566,7 +1542,7 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fde0e0ec90c9dfb3b4b1a0891a7dcd0e2bffde2f7efed5fe7c9bb00e5bfb915e" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.48.0", ] [[package]] @@ -2927,7 +2903,7 @@ checksum = "0ce92ff622d6dadf7349484f42c93271a0d49b7cc4d466a936405bacbe10aa78" dependencies = [ "cfg-if", "rustix 1.0.8", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -3186,12 +3162,6 @@ dependencies = [ "wasm-bindgen", ] -[[package]] -name = "gimli" -version = "0.31.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f" - [[package]] name = "glob" version = "0.3.3" @@ -3561,7 +3531,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.6.0", + "socket2 0.5.10", "tokio", "tower-service", "tracing", @@ -4049,17 +4019,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "io-uring" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "046fa2d4d00aea763528b4950358d0ead425372445dc8ff86312b3c69ff7727b" -dependencies = [ - "bitflags", - "cfg-if", - "libc", -] - [[package]] name = "ipnet" version = "2.11.0" @@ -4846,15 +4805,6 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" -[[package]] -name = "object" -version = "0.36.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87" -dependencies = [ - "memchr", -] - [[package]] name = "object_store" version = "0.12.3" @@ -5661,7 +5611,7 @@ dependencies = [ "quinn-udp", "rustc-hash 2.1.1", "rustls 0.23.31", - "socket2 0.6.0", + "socket2 0.5.10", "thiserror 2.0.16", "tokio", "tracing", @@ -5698,7 +5648,7 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.6.0", + "socket2 0.5.10", "tracing", "windows-sys 0.60.2", ] @@ -6135,12 +6085,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "rustc-demangle" -version = "0.1.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56f7d92ca342cea22a06f2121d944b4fd82af56988c270852495420f961d4ace" - [[package]] name = "rustc-hash" version = "1.1.0" @@ -6175,7 +6119,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.4.15", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -7093,7 +7037,7 @@ dependencies = [ "cfg-if", "libc", "psm", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -7432,29 +7376,26 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.47.1" +version = "1.48.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89e49afdadebb872d3145a5638b59eb0691ea23e46ca484037cfab3b76b95038" +checksum = "ff360e02eab121e0bc37a2d3b4d4dc622e6eda3a8e5253d5435ecf5bd4c68408" dependencies = [ - "backtrace", "bytes", - "io-uring", "libc", "mio", "parking_lot", "pin-project-lite", "signal-hook-registry", - "slab", "socket2 0.6.0", "tokio-macros", - "windows-sys 0.59.0", + "windows-sys 0.61.0", ] [[package]] name = "tokio-macros" -version = "2.5.0" +version = "2.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e06d43f1345a3bcd39f6a56dbb7dcab2ba47e68e8ac134855e7e2bdbaf8cab8" +checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", From 1244acd08ef56e5f15b0961c66a8492a6f6d83a2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 19 Oct 2025 16:57:05 -0400 Subject: [PATCH 021/891] chore(deps): Bump backon from 1.5.2 to 1.6.0 (#1762) Bumps [backon](https://github.com/Xuanwo/backon) from 1.5.2 to 1.6.0.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=backon&package-manager=cargo&previous-version=1.5.2&new-version=1.6.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 6a2a33238f..2adfb0a868 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1042,9 +1042,9 @@ dependencies = [ [[package]] name = "backon" -version = "1.5.2" +version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "592277618714fbcecda9a02ba7a8781f319d26532a88553bbacc77ba5d2b3a8d" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" dependencies = [ "fastrand", "gloo-timers", From d33f3bb77ede1bf481bf71d9ddb45cb4cdcbd858 Mon Sep 17 00:00:00 2001 From: Amogh Jahagirdar Date: Mon, 20 Oct 2025 10:48:51 -0600 Subject: [PATCH 022/891] fix: global eq delete matching should apply to only strictly older files, and fix partition scoped matching to consider spec id (#1758) ## Which issue does this PR close? - Closes #1759. ## What changes are included in this PR? This changes scan planning of equality deletes so that: 1. we only match against eq deletes which are strictly older than the data file. Right now it looks like we incorrectly over-apply based on the seq number for global equality deletes. 2. Partition scoped deletes (both equality and position) are compared correctly by also factoring in the spec ID. It's not quite enough to compare just off the tuple, we should also compare based off the spec ID as well. ## Are these changes tested? Added unit tests which are scoped to testing delete index matching logic. If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> --- crates/iceberg/src/delete_file_index.rs | 292 +++++++++++++++++++++++- 1 file changed, 283 insertions(+), 9 deletions(-) diff --git a/crates/iceberg/src/delete_file_index.rs b/crates/iceberg/src/delete_file_index.rs index d8f7a872e1..4f6fd28483 100644 --- a/crates/iceberg/src/delete_file_index.rs +++ b/crates/iceberg/src/delete_file_index.rs @@ -42,7 +42,7 @@ enum DeleteFileIndexState { #[derive(Debug)] struct PopulatedDeleteFileIndex { #[allow(dead_code)] - global_deletes: Vec>, + global_equality_deletes: Vec>, eq_deletes_by_partition: HashMap>>, pos_deletes_by_partition: HashMap>>, // TODO: do we need this? @@ -65,7 +65,8 @@ impl DeleteFileIndex { spawn({ let state = state.clone(); async move { - let delete_files = delete_file_stream.collect::>().await; + let delete_files: Vec = + delete_file_stream.collect::>().await; let populated_delete_file_index = PopulatedDeleteFileIndex::new(delete_files); @@ -114,7 +115,7 @@ impl PopulatedDeleteFileIndex { /// /// 1. The partition information is extracted from each delete file's manifest entry. /// 2. If the partition is empty and the delete file is not a positional delete, - /// it is added to the `global_deletes` vector + /// it is added to the `global_equality_deletes` vector /// 3. Otherwise, the delete file is added to one of two hash maps based on its content type. fn new(files: Vec) -> PopulatedDeleteFileIndex { let mut eq_deletes_by_partition: HashMap>> = @@ -122,7 +123,7 @@ impl PopulatedDeleteFileIndex { let mut pos_deletes_by_partition: HashMap>> = HashMap::default(); - let mut global_deletes: Vec> = vec![]; + let mut global_equality_deletes: Vec> = vec![]; files.into_iter().for_each(|ctx| { let arc_ctx = Arc::new(ctx); @@ -133,7 +134,7 @@ impl PopulatedDeleteFileIndex { if partition.fields().is_empty() { // TODO: confirm we're good to skip here if we encounter a pos del if arc_ctx.manifest_entry.content_type() != DataContentType::PositionDeletes { - global_deletes.push(arc_ctx); + global_equality_deletes.push(arc_ctx); return; } } @@ -153,7 +154,7 @@ impl PopulatedDeleteFileIndex { }); PopulatedDeleteFileIndex { - global_deletes, + global_equality_deletes, eq_deletes_by_partition, pos_deletes_by_partition, } @@ -167,12 +168,12 @@ impl PopulatedDeleteFileIndex { ) -> Vec { let mut results = vec![]; - self.global_deletes + self.global_equality_deletes .iter() - // filter that returns true if the provided delete file's sequence number is **greater than or equal to** `seq_num` + // filter that returns true if the provided delete file's sequence number is **greater than** `seq_num` .filter(|&delete| { seq_num - .map(|seq_num| delete.manifest_entry.sequence_number() >= Some(seq_num)) + .map(|seq_num| delete.manifest_entry.sequence_number() > Some(seq_num)) .unwrap_or_else(|| true) }) .for_each(|delete| results.push(delete.as_ref().into())); @@ -185,6 +186,7 @@ impl PopulatedDeleteFileIndex { seq_num .map(|seq_num| delete.manifest_entry.sequence_number() > Some(seq_num)) .unwrap_or_else(|| true) + && data_file.partition_spec_id == delete.partition_spec_id }) .for_each(|delete| results.push(delete.as_ref().into())); } @@ -201,6 +203,7 @@ impl PopulatedDeleteFileIndex { seq_num .map(|seq_num| delete.manifest_entry.sequence_number() >= Some(seq_num)) .unwrap_or_else(|| true) + && data_file.partition_spec_id == delete.partition_spec_id }) .for_each(|delete| results.push(delete.as_ref().into())); } @@ -208,3 +211,274 @@ impl PopulatedDeleteFileIndex { results } } + +#[cfg(test)] +mod tests { + use uuid::Uuid; + + use super::*; + use crate::spec::{ + DataContentType, DataFileBuilder, DataFileFormat, Literal, ManifestEntry, ManifestStatus, + Struct, + }; + + #[test] + fn test_delete_file_index_unpartitioned() { + let deletes: Vec = vec![ + build_added_manifest_entry(4, &build_unpartitioned_eq_delete()), + build_added_manifest_entry(6, &build_unpartitioned_eq_delete()), + build_added_manifest_entry(5, &build_unpartitioned_pos_delete()), + build_added_manifest_entry(6, &build_unpartitioned_pos_delete()), + ]; + + let delete_file_paths: Vec = deletes + .iter() + .map(|file| file.file_path().to_string()) + .collect(); + + let delete_contexts: Vec = deletes + .into_iter() + .map(|entry| DeleteFileContext { + manifest_entry: entry.into(), + partition_spec_id: 0, + }) + .collect(); + + let delete_file_index = PopulatedDeleteFileIndex::new(delete_contexts); + + let data_file = build_unpartitioned_data_file(); + + // All deletes apply to sequence 0 + let delete_files_to_apply_for_seq_0 = + delete_file_index.get_deletes_for_data_file(&data_file, Some(0)); + assert_eq!(delete_files_to_apply_for_seq_0.len(), 4); + + // All deletes apply to sequence 3 + let delete_files_to_apply_for_seq_3 = + delete_file_index.get_deletes_for_data_file(&data_file, Some(3)); + assert_eq!(delete_files_to_apply_for_seq_3.len(), 4); + + // Last 3 deletes apply to sequence 4 + let delete_files_to_apply_for_seq_4 = + delete_file_index.get_deletes_for_data_file(&data_file, Some(4)); + let actual_paths_to_apply_for_seq_4: Vec = delete_files_to_apply_for_seq_4 + .into_iter() + .map(|file| file.file_path) + .collect(); + + assert_eq!( + actual_paths_to_apply_for_seq_4, + delete_file_paths[delete_file_paths.len() - 3..] + ); + + // Last 3 deletes apply to sequence 5 + let delete_files_to_apply_for_seq_5 = + delete_file_index.get_deletes_for_data_file(&data_file, Some(5)); + let actual_paths_to_apply_for_seq_5: Vec = delete_files_to_apply_for_seq_5 + .into_iter() + .map(|file| file.file_path) + .collect(); + assert_eq!( + actual_paths_to_apply_for_seq_5, + delete_file_paths[delete_file_paths.len() - 3..] + ); + + // Only the last position delete applies to sequence 6 + let delete_files_to_apply_for_seq_6 = + delete_file_index.get_deletes_for_data_file(&data_file, Some(6)); + let actual_paths_to_apply_for_seq_6: Vec = delete_files_to_apply_for_seq_6 + .into_iter() + .map(|file| file.file_path) + .collect(); + assert_eq!( + actual_paths_to_apply_for_seq_6, + delete_file_paths[delete_file_paths.len() - 1..] + ); + + // The 2 global equality deletes should match against any partitioned file + let partitioned_file = + build_partitioned_data_file(&Struct::from_iter([Some(Literal::long(100))]), 1); + + let delete_files_to_apply_for_partitioned_file = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(0)); + let actual_paths_to_apply_for_partitioned_file: Vec = + delete_files_to_apply_for_partitioned_file + .into_iter() + .map(|file| file.file_path) + .collect(); + assert_eq!( + actual_paths_to_apply_for_partitioned_file, + delete_file_paths[..2] + ); + } + + #[test] + fn test_delete_file_index_partitioned() { + let partition_one = Struct::from_iter([Some(Literal::long(100))]); + let spec_id = 1; + let deletes: Vec = vec![ + build_added_manifest_entry(4, &build_partitioned_eq_delete(&partition_one, spec_id)), + build_added_manifest_entry(6, &build_partitioned_eq_delete(&partition_one, spec_id)), + build_added_manifest_entry(5, &build_partitioned_pos_delete(&partition_one, spec_id)), + build_added_manifest_entry(6, &build_partitioned_pos_delete(&partition_one, spec_id)), + ]; + + let delete_file_paths: Vec = deletes + .iter() + .map(|file| file.file_path().to_string()) + .collect(); + + let delete_contexts: Vec = deletes + .into_iter() + .map(|entry| DeleteFileContext { + manifest_entry: entry.into(), + partition_spec_id: spec_id, + }) + .collect(); + + let delete_file_index = PopulatedDeleteFileIndex::new(delete_contexts); + + let partitioned_file = + build_partitioned_data_file(&Struct::from_iter([Some(Literal::long(100))]), spec_id); + + // All deletes apply to sequence 0 + let delete_files_to_apply_for_seq_0 = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(0)); + assert_eq!(delete_files_to_apply_for_seq_0.len(), 4); + + // All deletes apply to sequence 3 + let delete_files_to_apply_for_seq_3 = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(3)); + assert_eq!(delete_files_to_apply_for_seq_3.len(), 4); + + // Last 3 deletes apply to sequence 4 + let delete_files_to_apply_for_seq_4 = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(4)); + let actual_paths_to_apply_for_seq_4: Vec = delete_files_to_apply_for_seq_4 + .into_iter() + .map(|file| file.file_path) + .collect(); + + assert_eq!( + actual_paths_to_apply_for_seq_4, + delete_file_paths[delete_file_paths.len() - 3..] + ); + + // Last 3 deletes apply to sequence 5 + let delete_files_to_apply_for_seq_5 = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(5)); + let actual_paths_to_apply_for_seq_5: Vec = delete_files_to_apply_for_seq_5 + .into_iter() + .map(|file| file.file_path) + .collect(); + assert_eq!( + actual_paths_to_apply_for_seq_5, + delete_file_paths[delete_file_paths.len() - 3..] + ); + + // Only the last position delete applies to sequence 6 + let delete_files_to_apply_for_seq_6 = + delete_file_index.get_deletes_for_data_file(&partitioned_file, Some(6)); + let actual_paths_to_apply_for_seq_6: Vec = delete_files_to_apply_for_seq_6 + .into_iter() + .map(|file| file.file_path) + .collect(); + assert_eq!( + actual_paths_to_apply_for_seq_6, + delete_file_paths[delete_file_paths.len() - 1..] + ); + + // Data file with different partition tuples does not match any delete files + let partitioned_second_file = + build_partitioned_data_file(&Struct::from_iter([Some(Literal::long(200))]), 1); + let delete_files_to_apply_for_different_partition = + delete_file_index.get_deletes_for_data_file(&partitioned_second_file, Some(0)); + let actual_paths_to_apply_for_different_partition: Vec = + delete_files_to_apply_for_different_partition + .into_iter() + .map(|file| file.file_path) + .collect(); + assert!(actual_paths_to_apply_for_different_partition.is_empty()); + + // Data file with same tuple but different spec ID does not match any delete files + let partitioned_different_spec = build_partitioned_data_file(&partition_one, 2); + let delete_files_to_apply_for_different_spec = + delete_file_index.get_deletes_for_data_file(&partitioned_different_spec, Some(0)); + let actual_paths_to_apply_for_different_spec: Vec = + delete_files_to_apply_for_different_spec + .into_iter() + .map(|file| file.file_path) + .collect(); + assert!(actual_paths_to_apply_for_different_spec.is_empty()); + } + + fn build_unpartitioned_eq_delete() -> DataFile { + build_partitioned_eq_delete(&Struct::empty(), 0) + } + + fn build_partitioned_eq_delete(partition: &Struct, spec_id: i32) -> DataFile { + DataFileBuilder::default() + .file_path(format!("{}_equality_delete.parquet", Uuid::new_v4())) + .file_format(DataFileFormat::Parquet) + .content(DataContentType::EqualityDeletes) + .equality_ids(Some(vec![1])) + .record_count(1) + .partition(partition.clone()) + .partition_spec_id(spec_id) + .file_size_in_bytes(100) + .build() + .unwrap() + } + + fn build_unpartitioned_pos_delete() -> DataFile { + build_partitioned_pos_delete(&Struct::empty(), 0) + } + + fn build_partitioned_pos_delete(partition: &Struct, spec_id: i32) -> DataFile { + DataFileBuilder::default() + .file_path(format!("{}-pos-delete.parquet", Uuid::new_v4())) + .file_format(DataFileFormat::Parquet) + .content(DataContentType::PositionDeletes) + .record_count(1) + .referenced_data_file(Some("/some-data-file.parquet".to_string())) + .partition(partition.clone()) + .partition_spec_id(spec_id) + .file_size_in_bytes(100) + .build() + .unwrap() + } + + fn build_unpartitioned_data_file() -> DataFile { + DataFileBuilder::default() + .file_path(format!("{}-data.parquet", Uuid::new_v4())) + .file_format(DataFileFormat::Parquet) + .content(DataContentType::Data) + .record_count(100) + .partition(Struct::empty()) + .partition_spec_id(0) + .file_size_in_bytes(100) + .build() + .unwrap() + } + + fn build_partitioned_data_file(partition_value: &Struct, spec_id: i32) -> DataFile { + DataFileBuilder::default() + .file_path(format!("{}-data.parquet", Uuid::new_v4())) + .file_format(DataFileFormat::Parquet) + .content(DataContentType::Data) + .record_count(100) + .partition(partition_value.clone()) + .partition_spec_id(spec_id) + .file_size_in_bytes(100) + .build() + .unwrap() + } + + fn build_added_manifest_entry(data_seq_number: i64, file: &DataFile) -> ManifestEntry { + ManifestEntry::builder() + .status(ManifestStatus::Added) + .sequence_number(data_seq_number) + .data_file(file.clone()) + .build() + } +} From 777d9dc3746759229a0c0b6a13c30c5414efe7e5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 Oct 2025 11:28:03 +0800 Subject: [PATCH 023/891] chore(deps): Bump apache/skywalking-eyes from 0.7.0 to 0.8.0 (#1760) Bumps [apache/skywalking-eyes](https://github.com/apache/skywalking-eyes) from 0.7.0 to 0.8.0.
Commits
  • 61275cc Add fsf-free and osi-approved options (#250)
  • d1a97df Update example workflow in the README for the Ruby scenario (#248)
  • 2ca7b4d fix: Gemfile / Gemspec parser to ignore commented dependencies (#249)
  • 454b8ea feat: add some compatibility licenses (#247)
  • 36f75de feat: Compatibility matrix: MIT & Ruby (#209)
  • 18cb362 GemfileLockResolver adjusted to exclude all specs when runtime dependencies a...
  • 5a19c65 Fixed a panic caused by nil dependency specs in Ruby Gemfile.lock resolver (#...
  • 0d9c4df fix: return error in license check, add MIT header (#206)
  • 3655e78 Ruby dependency license scanning support via Gemfile.lock. (#205)
  • 07a607f Bump golang.org/x/oauth2 from 0.5.0 to 0.27.0 (#204)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=apache/skywalking-eyes&package-manager=github_actions&previous-version=0.7.0&new-version=0.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aa3f54d317..3207ae87b1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -58,7 +58,7 @@ jobs: uses: ./.github/actions/setup-builder - name: Check License Header - uses: apache/skywalking-eyes/header@v0.7.0 + uses: apache/skywalking-eyes/header@v0.8.0 - name: Check toml format run: make check-toml From 4d20ff08d7c79abbe5bdc17d8e770f077b66dfbf Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 Oct 2025 12:58:50 +0800 Subject: [PATCH 024/891] chore(deps): Bump rust_decimal from 1.38.0 to 1.39.0 (#1761) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.38.0 to 1.39.0.
Release notes

Sourced from rust_decimal's releases.

1.39.0

What's Changed

New Contributors

Full Changelog: https://github.com/paupino/rust-decimal/compare/1.38.0...1.39.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rust_decimal&package-manager=cargo&previous-version=1.38.0&new-version=1.39.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 2adfb0a868..b048c3db56 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6070,9 +6070,9 @@ dependencies = [ [[package]] name = "rust_decimal" -version = "1.38.0" +version = "1.39.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8975fc98059f365204d635119cf9c5a60ae67b841ed49b5422a9a7e56cdfac0" +checksum = "35affe401787a9bd846712274d97654355d21b2a2c092a3139aabe31e9022282" dependencies = [ "arrayvec", "borsh", From b3b5afed126ee5962dc3a4ca23cd735694449df8 Mon Sep 17 00:00:00 2001 From: Florian Valeye Date: Tue, 21 Oct 2025 11:53:58 +0200 Subject: [PATCH 025/891] feat(datafusion): implement the project node to add the partition columns (#1602) ## Which issue does this PR close? - Closes #1542 ## What changes are included in this PR? Implement a physical execution plan node that projects Iceberg partition columns from source data, supporting nested fields and all Iceberg transforms. ## Are these changes tested? Yes, with unit tests --------- Signed-off-by: Florian Valeye --- crates/iceberg/src/arrow/mod.rs | 3 +- .../src/arrow/record_batch_projector.rs | 71 ++- crates/iceberg/src/transform/mod.rs | 4 +- .../datafusion/src/physical_plan/mod.rs | 2 + .../datafusion/src/physical_plan/project.rs | 488 ++++++++++++++++++ 5 files changed, 563 insertions(+), 5 deletions(-) create mode 100644 crates/integrations/datafusion/src/physical_plan/project.rs diff --git a/crates/iceberg/src/arrow/mod.rs b/crates/iceberg/src/arrow/mod.rs index d32cbeb8f9..28116a4b5e 100644 --- a/crates/iceberg/src/arrow/mod.rs +++ b/crates/iceberg/src/arrow/mod.rs @@ -28,7 +28,8 @@ pub mod delete_file_loader; pub(crate) mod delete_filter; mod reader; -pub(crate) mod record_batch_projector; +/// RecordBatch projection utilities +pub mod record_batch_projector; pub(crate) mod record_batch_transformer; mod value; diff --git a/crates/iceberg/src/arrow/record_batch_projector.rs b/crates/iceberg/src/arrow/record_batch_projector.rs index 7ca28c25c7..45de0212e8 100644 --- a/crates/iceberg/src/arrow/record_batch_projector.rs +++ b/crates/iceberg/src/arrow/record_batch_projector.rs @@ -20,13 +20,16 @@ use std::sync::Arc; use arrow_array::{ArrayRef, RecordBatch, StructArray, make_array}; use arrow_buffer::NullBuffer; use arrow_schema::{DataType, Field, FieldRef, Fields, Schema, SchemaRef}; +use parquet::arrow::PARQUET_FIELD_ID_META_KEY; +use crate::arrow::schema::schema_to_arrow_schema; use crate::error::Result; +use crate::spec::Schema as IcebergSchema; use crate::{Error, ErrorKind}; /// Help to project specific field from `RecordBatch`` according to the fields id. -#[derive(Clone, Debug)] -pub(crate) struct RecordBatchProjector { +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RecordBatchProjector { // A vector of vectors, where each inner vector represents the index path to access a specific field in a nested structure. // E.g. [[0], [1, 2]] means the first field is accessed directly from the first column, // while the second field is accessed from the second column and then from its third subcolumn (second column must be a struct column). @@ -77,6 +80,46 @@ impl RecordBatchProjector { }) } + /// Create RecordBatchProjector using Iceberg schema. + /// + /// This constructor converts the Iceberg schema to Arrow schema with field ID metadata, + /// then uses the standard field ID lookup for projection. + /// + /// # Arguments + /// * `iceberg_schema` - The Iceberg schema for field ID mapping + /// * `target_field_ids` - The field IDs to project + pub fn from_iceberg_schema( + iceberg_schema: Arc, + target_field_ids: &[i32], + ) -> Result { + let arrow_schema_with_ids = Arc::new(schema_to_arrow_schema(&iceberg_schema)?); + + let field_id_fetch_func = |field: &Field| -> Result> { + if let Some(value) = field.metadata().get(PARQUET_FIELD_ID_META_KEY) { + let field_id = value.parse::().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + "Failed to parse field id".to_string(), + ) + .with_context("value", value) + .with_source(e) + })?; + Ok(Some(field_id as i64)) + } else { + Ok(None) + } + }; + + let searchable_field_func = |_field: &Field| -> bool { true }; + + Self::new( + arrow_schema_with_ids, + target_field_ids, + field_id_fetch_func, + searchable_field_func, + ) + } + fn fetch_field_index( fields: &Fields, index_vec: &mut Vec, @@ -129,7 +172,7 @@ impl RecordBatchProjector { } /// Do projection with columns - pub(crate) fn project_column(&self, batch: &[ArrayRef]) -> Result> { + pub fn project_column(&self, batch: &[ArrayRef]) -> Result> { self.field_indices .iter() .map(|index_vec| Self::get_column_by_field_index(batch, index_vec)) @@ -166,6 +209,7 @@ mod test { use arrow_schema::{DataType, Field, Fields, Schema}; use crate::arrow::record_batch_projector::RecordBatchProjector; + use crate::spec::{NestedField, PrimitiveType, Schema as IcebergSchema, Type}; use crate::{Error, ErrorKind}; #[test] @@ -293,4 +337,25 @@ mod test { RecordBatchProjector::new(schema.clone(), &[3], field_id_fetch_func, |_| true); assert!(projector.is_ok()); } + + #[test] + fn test_from_iceberg_schema() { + let iceberg_schema = IcebergSchema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "age", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(); + + let projector = + RecordBatchProjector::from_iceberg_schema(Arc::new(iceberg_schema), &[1, 3]).unwrap(); + + assert_eq!(projector.field_indices.len(), 2); + assert_eq!(projector.projected_schema_ref().fields().len(), 2); + assert_eq!(projector.projected_schema_ref().field(0).name(), "id"); + assert_eq!(projector.projected_schema_ref().field(1).name(), "age"); + } } diff --git a/crates/iceberg/src/transform/mod.rs b/crates/iceberg/src/transform/mod.rs index 4cc0d1fe8f..809d2dafe0 100644 --- a/crates/iceberg/src/transform/mod.rs +++ b/crates/iceberg/src/transform/mod.rs @@ -17,6 +17,8 @@ //! Transform function used to compute partition values. +use std::fmt::Debug; + use arrow_array::ArrayRef; use crate::spec::{Datum, Transform}; @@ -29,7 +31,7 @@ mod truncate; mod void; /// TransformFunction is a trait that defines the interface for all transform functions. -pub trait TransformFunction: Send + Sync { +pub trait TransformFunction: Send + Sync + Debug { /// transform will take an input array and transform it into a new array. /// The implementation of this function will need to check and downcast the input to specific /// type. diff --git a/crates/integrations/datafusion/src/physical_plan/mod.rs b/crates/integrations/datafusion/src/physical_plan/mod.rs index fcfd11a453..ce923b8662 100644 --- a/crates/integrations/datafusion/src/physical_plan/mod.rs +++ b/crates/integrations/datafusion/src/physical_plan/mod.rs @@ -18,9 +18,11 @@ pub(crate) mod commit; pub(crate) mod expr_to_predicate; pub(crate) mod metadata_scan; +pub(crate) mod project; pub(crate) mod scan; pub(crate) mod write; pub(crate) const DATA_FILES_COL_NAME: &str = "data_files"; +pub use project::project_with_partition; pub use scan::IcebergTableScan; diff --git a/crates/integrations/datafusion/src/physical_plan/project.rs b/crates/integrations/datafusion/src/physical_plan/project.rs new file mode 100644 index 0000000000..4bfe8192b0 --- /dev/null +++ b/crates/integrations/datafusion/src/physical_plan/project.rs @@ -0,0 +1,488 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Partition value projection for Iceberg tables. + +use std::sync::Arc; + +use datafusion::arrow::array::{ArrayRef, RecordBatch, StructArray}; +use datafusion::arrow::datatypes::{DataType, Schema as ArrowSchema}; +use datafusion::common::Result as DFResult; +use datafusion::error::DataFusionError; +use datafusion::physical_expr::PhysicalExpr; +use datafusion::physical_expr::expressions::Column; +use datafusion::physical_plan::projection::ProjectionExec; +use datafusion::physical_plan::{ColumnarValue, ExecutionPlan}; +use iceberg::arrow::record_batch_projector::RecordBatchProjector; +use iceberg::spec::{PartitionSpec, Schema}; +use iceberg::table::Table; +use iceberg::transform::BoxedTransformFunction; + +use crate::to_datafusion_error; + +/// Column name for the combined partition values struct +const PARTITION_VALUES_COLUMN: &str = "_partition"; + +/// Extends an ExecutionPlan with partition value calculations for Iceberg tables. +/// +/// This function takes an input ExecutionPlan and extends it with an additional column +/// containing calculated partition values based on the table's partition specification. +/// For unpartitioned tables, returns the original plan unchanged. +/// +/// # Arguments +/// * `input` - The input ExecutionPlan to extend +/// * `table` - The Iceberg table with partition specification +/// +/// # Returns +/// * `Ok(Arc)` - Extended plan with partition values column +/// * `Err` - If partition spec is not found or transformation fails +pub fn project_with_partition( + input: Arc, + table: &Table, +) -> DFResult> { + let metadata = table.metadata(); + let partition_spec = metadata.default_partition_spec(); + let table_schema = metadata.current_schema(); + + if partition_spec.is_unpartitioned() { + return Ok(input); + } + + let input_schema = input.schema(); + // TODO: Validate that input_schema matches the Iceberg table schema. + // See: https://github.com/apache/iceberg-rust/issues/1752 + let partition_type = build_partition_type(partition_spec, table_schema.as_ref())?; + let calculator = PartitionValueCalculator::new( + partition_spec.as_ref().clone(), + table_schema.as_ref().clone(), + partition_type, + )?; + + let mut projection_exprs: Vec<(Arc, String)> = + Vec::with_capacity(input_schema.fields().len() + 1); + + for (index, field) in input_schema.fields().iter().enumerate() { + let column_expr = Arc::new(Column::new(field.name(), index)); + projection_exprs.push((column_expr, field.name().clone())); + } + + let partition_expr = Arc::new(PartitionExpr::new(calculator)); + projection_exprs.push((partition_expr, PARTITION_VALUES_COLUMN.to_string())); + + let projection = ProjectionExec::try_new(projection_exprs, input)?; + Ok(Arc::new(projection)) +} + +/// PhysicalExpr implementation for partition value calculation +#[derive(Debug, Clone)] +struct PartitionExpr { + calculator: Arc, +} + +impl PartitionExpr { + fn new(calculator: PartitionValueCalculator) -> Self { + Self { + calculator: Arc::new(calculator), + } + } +} + +// Manual PartialEq/Eq implementations for pointer-based equality +// (two PartitionExpr are equal if they share the same calculator instance) +impl PartialEq for PartitionExpr { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.calculator, &other.calculator) + } +} + +impl Eq for PartitionExpr {} + +impl PhysicalExpr for PartitionExpr { + fn as_any(&self) -> &dyn std::any::Any { + self + } + + fn data_type(&self, _input_schema: &ArrowSchema) -> DFResult { + Ok(self.calculator.partition_type.clone()) + } + + fn nullable(&self, _input_schema: &ArrowSchema) -> DFResult { + Ok(false) + } + + fn evaluate(&self, batch: &RecordBatch) -> DFResult { + let array = self.calculator.calculate(batch)?; + Ok(ColumnarValue::Array(array)) + } + + fn children(&self) -> Vec<&Arc> { + vec![] + } + + fn with_new_children( + self: Arc, + _children: Vec>, + ) -> DFResult> { + Ok(self) + } + + fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let field_names: Vec = self + .calculator + .partition_spec + .fields() + .iter() + .map(|pf| format!("{}({})", pf.transform, pf.name)) + .collect(); + write!(f, "iceberg_partition_values[{}]", field_names.join(", ")) + } +} + +impl std::fmt::Display for PartitionExpr { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let field_names: Vec<&str> = self + .calculator + .partition_spec + .fields() + .iter() + .map(|pf| pf.name.as_str()) + .collect(); + write!(f, "iceberg_partition_values({})", field_names.join(", ")) + } +} + +impl std::hash::Hash for PartitionExpr { + fn hash(&self, state: &mut H) { + // Two PartitionExpr are equal if they share the same calculator Arc + Arc::as_ptr(&self.calculator).hash(state); + } +} + +/// Calculator for partition values in Iceberg tables +#[derive(Debug)] +struct PartitionValueCalculator { + partition_spec: PartitionSpec, + partition_type: DataType, + projector: RecordBatchProjector, + transform_functions: Vec, +} + +impl PartitionValueCalculator { + fn new( + partition_spec: PartitionSpec, + table_schema: Schema, + partition_type: DataType, + ) -> DFResult { + if partition_spec.is_unpartitioned() { + return Err(DataFusionError::Internal( + "Cannot create partition calculator for unpartitioned table".to_string(), + )); + } + + let transform_functions: Result, _> = partition_spec + .fields() + .iter() + .map(|pf| iceberg::transform::create_transform_function(&pf.transform)) + .collect(); + + let transform_functions = transform_functions.map_err(to_datafusion_error)?; + + let source_field_ids: Vec = partition_spec + .fields() + .iter() + .map(|pf| pf.source_id) + .collect(); + + let projector = RecordBatchProjector::from_iceberg_schema( + Arc::new(table_schema.clone()), + &source_field_ids, + ) + .map_err(to_datafusion_error)?; + + Ok(Self { + partition_spec, + partition_type, + projector, + transform_functions, + }) + } + + fn calculate(&self, batch: &RecordBatch) -> DFResult { + let source_columns = self + .projector + .project_column(batch.columns()) + .map_err(to_datafusion_error)?; + + let expected_struct_fields = match &self.partition_type { + DataType::Struct(fields) => fields.clone(), + _ => { + return Err(DataFusionError::Internal( + "Expected partition type must be a struct".to_string(), + )); + } + }; + + let mut partition_values = Vec::with_capacity(self.partition_spec.fields().len()); + + for (source_column, transform_fn) in source_columns.iter().zip(&self.transform_functions) { + let partition_value = transform_fn + .transform(source_column.clone()) + .map_err(to_datafusion_error)?; + + partition_values.push(partition_value); + } + + let struct_array = StructArray::try_new(expected_struct_fields, partition_values, None) + .map_err(|e| DataFusionError::ArrowError(e, None))?; + + Ok(Arc::new(struct_array)) + } +} + +fn build_partition_type( + partition_spec: &PartitionSpec, + table_schema: &Schema, +) -> DFResult { + let partition_struct_type = partition_spec + .partition_type(table_schema) + .map_err(to_datafusion_error)?; + + iceberg::arrow::type_to_arrow_type(&iceberg::spec::Type::Struct(partition_struct_type)) + .map_err(to_datafusion_error) +} + +#[cfg(test)] +mod tests { + use datafusion::arrow::array::Int32Array; + use datafusion::arrow::datatypes::{Field, Fields}; + use datafusion::physical_plan::empty::EmptyExec; + use iceberg::spec::{NestedField, PrimitiveType, StructType, Transform, Type}; + + use super::*; + + #[test] + fn test_partition_calculator_basic() { + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let _arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); + let calculator = PartitionValueCalculator::new( + partition_spec.clone(), + table_schema, + partition_type.clone(), + ) + .unwrap(); + + assert_eq!(calculator.partition_type, partition_type); + } + + #[test] + fn test_partition_expr_with_projection() { + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let input = Arc::new(EmptyExec::new(arrow_schema.clone())); + + let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); + let calculator = + PartitionValueCalculator::new(partition_spec, table_schema, partition_type).unwrap(); + + let mut projection_exprs: Vec<(Arc, String)> = + Vec::with_capacity(arrow_schema.fields().len() + 1); + for (i, field) in arrow_schema.fields().iter().enumerate() { + let column_expr = Arc::new(Column::new(field.name(), i)); + projection_exprs.push((column_expr, field.name().clone())); + } + + let partition_expr = Arc::new(PartitionExpr::new(calculator)); + projection_exprs.push((partition_expr, PARTITION_VALUES_COLUMN.to_string())); + + let projection = ProjectionExec::try_new(projection_exprs, input).unwrap(); + let result = Arc::new(projection); + + assert_eq!(result.schema().fields().len(), 3); + assert_eq!(result.schema().field(0).name(), "id"); + assert_eq!(result.schema().field(1).name(), "name"); + assert_eq!(result.schema().field(2).name(), "_partition"); + } + + #[test] + fn test_partition_expr_evaluate() { + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "data", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("data", DataType::Utf8, false), + ])); + + let batch = RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Int32Array::from(vec![10, 20, 30])), + Arc::new(datafusion::arrow::array::StringArray::from(vec![ + "a", "b", "c", + ])), + ]) + .unwrap(); + + let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); + let calculator = + PartitionValueCalculator::new(partition_spec, table_schema, partition_type.clone()) + .unwrap(); + let expr = PartitionExpr::new(calculator); + + assert_eq!(expr.data_type(&arrow_schema).unwrap(), partition_type); + assert!(!expr.nullable(&arrow_schema).unwrap()); + + let result = expr.evaluate(&batch).unwrap(); + match result { + ColumnarValue::Array(array) => { + let struct_array = array.as_any().downcast_ref::().unwrap(); + let id_partition = struct_array + .column_by_name("id_partition") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_partition.value(0), 10); + assert_eq!(id_partition.value(1), 20); + assert_eq!(id_partition.value(2), 30); + } + _ => panic!("Expected array result"), + } + } + + #[test] + fn test_nested_partition() { + let address_struct = StructType::new(vec![ + NestedField::required(3, "street", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(4, "city", Type::Primitive(PrimitiveType::String)).into(), + ]); + + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "address", Type::Struct(address_struct)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) + .add_partition_field("address.city", "city_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let struct_fields = Fields::from(vec![ + Field::new("street", DataType::Utf8, false), + Field::new("city", DataType::Utf8, false), + ]); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("address", DataType::Struct(struct_fields), false), + ])); + + let street_array = Arc::new(datafusion::arrow::array::StringArray::from(vec![ + "123 Main St", + "456 Oak Ave", + ])); + let city_array = Arc::new(datafusion::arrow::array::StringArray::from(vec![ + "New York", + "Los Angeles", + ])); + + let struct_array = StructArray::from(vec![ + ( + Arc::new(Field::new("street", DataType::Utf8, false)), + street_array as ArrayRef, + ), + ( + Arc::new(Field::new("city", DataType::Utf8, false)), + city_array as ArrayRef, + ), + ]); + + let batch = RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(struct_array), + ]) + .unwrap(); + + let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); + let calculator = + PartitionValueCalculator::new(partition_spec, table_schema, partition_type).unwrap(); + let array = calculator.calculate(&batch).unwrap(); + + let struct_array = array.as_any().downcast_ref::().unwrap(); + let city_partition = struct_array + .column_by_name("city_partition") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(city_partition.value(0), "New York"); + assert_eq!(city_partition.value(1), "Los Angeles"); + } +} From 3aa7ade28e6e56304700c163e1cbfdd8b0069921 Mon Sep 17 00:00:00 2001 From: Raminder Singh Date: Wed, 22 Oct 2025 15:03:44 +0530 Subject: [PATCH 026/891] fix: snapshot was producing empty summary (#1767) ## Which issue does this PR close? - Closes #. There was no open issue for this. I noticed this while trying out iceberg-rust. ## What changes are included in this PR? The first line in the method `SnapshotProducer::write_added_manifest` (`let added_data_files = std::mem::take(&mut self.added_data_files);`) sets the `self.added_data_files` to an empty vec. `SnapshotProducer::write_added_manifest` is called via the call chain `SnapshotProducer::commit` -> `SnapshotProducer::manifest_file` -> `SnapshotProducer::write_added_manifest`). Hence, if `SnapshotProducer::summary` is called after calling `SnapshotProducer::manifest_file`, the summary produced was empty due to empty `self.added_data_files`. This PR rearranges the code in `SnapshotProducer::commit` to make sure the produced summary is not empty. ## Are these changes tested? No new tests have been added for this as there were no previous tests. --- crates/iceberg/src/transaction/snapshot.rs | 23 ++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index a03b8dc490..93dd819d5c 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -380,17 +380,8 @@ impl<'a> SnapshotProducer<'a> { snapshot_produce_operation: OP, process: MP, ) -> Result { - let new_manifests = self - .manifest_file(&snapshot_produce_operation, &process) - .await?; - let next_seq_num = self.table.metadata().next_sequence_number(); - - let summary = self.summary(&snapshot_produce_operation).map_err(|err| { - Error::new(ErrorKind::Unexpected, "Failed to create snapshot summary.").with_source(err) - })?; - let manifest_list_path = self.generate_manifest_list_file_path(0); - + let next_seq_num = self.table.metadata().next_sequence_number(); let mut manifest_list_writer = match self.table.metadata().format_version() { FormatVersion::V1 => ManifestListWriter::v1( self.table @@ -408,6 +399,18 @@ impl<'a> SnapshotProducer<'a> { next_seq_num, ), }; + + // Calling self.summary() before self.manifest_file() is important because self.added_data_files + // will be set to an empty vec after self.manifest_file() returns, resulting in an empty summary + // being generated. + let summary = self.summary(&snapshot_produce_operation).map_err(|err| { + Error::new(ErrorKind::Unexpected, "Failed to create snapshot summary.").with_source(err) + })?; + + let new_manifests = self + .manifest_file(&snapshot_produce_operation, &process) + .await?; + manifest_list_writer.add_manifests(new_manifests.into_iter())?; manifest_list_writer.close().await?; From 915778ebb3245f611ddb0b16076bc7c573a3ea56 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 22 Oct 2025 02:50:20 -0700 Subject: [PATCH 027/891] docs: Add examples for PartitioningWriter (#1754) ## Which issue does this PR close? - Closes #1745 ## What changes are included in this PR? - As title, added examples and documentation ## Are these changes tested? --- crates/iceberg/src/writer/mod.rs | 143 +++++++++++++++++- crates/iceberg/src/writer/partitioning/mod.rs | 6 + 2 files changed, 147 insertions(+), 2 deletions(-) diff --git a/crates/iceberg/src/writer/mod.rs b/crates/iceberg/src/writer/mod.rs index 1da3fa6790..a7892d49e1 100644 --- a/crates/iceberg/src/writer/mod.rs +++ b/crates/iceberg/src/writer/mod.rs @@ -242,11 +242,150 @@ //! Ok(()) //! } //! ``` +//! +//! # Adding Partitioning to Data File Writers +//! +//! You can wrap a `DataFileWriter` with partitioning writers to handle partitioned tables. +//! Iceberg provides two partitioning strategies: +//! +//! ## FanoutWriter - For Unsorted Data +//! +//! Wraps the data file writer to handle unsorted data by maintaining multiple active writers. +//! Use this when your data is not pre-sorted by partition key. Writes to different partitions +//! can happen in any order, even interleaved. +//! +//! ```rust, no_run +//! # // Same setup as the simple example above... +//! # use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; +//! # use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; +//! # use iceberg::{Catalog, CatalogBuilder, Result, TableIdent}; +//! # use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; +//! # use iceberg::writer::file_writer::ParquetWriterBuilder; +//! # use iceberg::writer::file_writer::location_generator::{ +//! # DefaultFileNameGenerator, DefaultLocationGenerator, +//! # }; +//! # use parquet::file::properties::WriterProperties; +//! # use std::collections::HashMap; +//! # #[tokio::main] +//! # async fn main() -> Result<()> { +//! # let catalog = MemoryCatalogBuilder::default() +//! # .load("memory", HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), "file:///path/to/warehouse".to_string())])) +//! # .await?; +//! # let table = catalog.load_table(&TableIdent::from_strs(["hello", "world"])?).await?; +//! # let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); +//! # let file_name_generator = DefaultFileNameGenerator::new("test".to_string(), None, iceberg::spec::DataFileFormat::Parquet); +//! # let parquet_writer_builder = ParquetWriterBuilder::new(WriterProperties::default(), table.metadata().current_schema().clone()); +//! # let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( +//! # parquet_writer_builder, table.file_io().clone(), location_generator, file_name_generator); +//! # let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); +//! +//! // Wrap the data file writer with FanoutWriter for partitioning +//! use iceberg::writer::partitioning::fanout_writer::FanoutWriter; +//! use iceberg::writer::partitioning::PartitioningWriter; +//! use iceberg::spec::{Literal, PartitionKey, Struct}; +//! +//! let mut fanout_writer = FanoutWriter::new(data_file_writer_builder); +//! +//! // Create partition keys for different regions +//! let schema = table.metadata().current_schema().clone(); +//! let partition_spec = table.metadata().default_partition_spec().as_ref().clone(); +//! +//! let partition_key_us = PartitionKey::new( +//! partition_spec.clone(), +//! schema.clone(), +//! Struct::from_iter([Some(Literal::string("US"))]), +//! ); +//! +//! let partition_key_eu = PartitionKey::new( +//! partition_spec.clone(), +//! schema.clone(), +//! Struct::from_iter([Some(Literal::string("EU"))]), +//! ); +//! +//! // Write to different partitions in any order - can interleave partition writes +//! // fanout_writer.write(partition_key_us.clone(), batch_us1).await?; +//! // fanout_writer.write(partition_key_eu.clone(), batch_eu1).await?; +//! // fanout_writer.write(partition_key_us.clone(), batch_us2).await?; // Back to US - OK! +//! // fanout_writer.write(partition_key_eu.clone(), batch_eu2).await?; // Back to EU - OK! +//! +//! let data_files = fanout_writer.close().await?; +//! # Ok(()) +//! # } +//! ``` +//! +//! ## ClusteredWriter - For Sorted Data +//! +//! Wraps the data file writer for pre-sorted data. More memory efficient as it maintains +//! only one active writer at a time, but requires input sorted by partition key. +//! +//! ```rust, no_run +//! # // Same setup as the simple example above... +//! # use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; +//! # use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; +//! # use iceberg::{Catalog, CatalogBuilder, Result, TableIdent}; +//! # use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; +//! # use iceberg::writer::file_writer::ParquetWriterBuilder; +//! # use iceberg::writer::file_writer::location_generator::{ +//! # DefaultFileNameGenerator, DefaultLocationGenerator, +//! # }; +//! # use parquet::file::properties::WriterProperties; +//! # use std::collections::HashMap; +//! # #[tokio::main] +//! # async fn main() -> Result<()> { +//! # let catalog = MemoryCatalogBuilder::default() +//! # .load("memory", HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), "file:///path/to/warehouse".to_string())])) +//! # .await?; +//! # let table = catalog.load_table(&TableIdent::from_strs(["hello", "world"])?).await?; +//! # let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); +//! # let file_name_generator = DefaultFileNameGenerator::new("test".to_string(), None, iceberg::spec::DataFileFormat::Parquet); +//! # let parquet_writer_builder = ParquetWriterBuilder::new(WriterProperties::default(), table.metadata().current_schema().clone()); +//! # let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( +//! # parquet_writer_builder, table.file_io().clone(), location_generator, file_name_generator); +//! # let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); +//! +//! // Wrap the data file writer with ClusteredWriter for sorted partitioning +//! use iceberg::writer::partitioning::clustered_writer::ClusteredWriter; +//! use iceberg::writer::partitioning::PartitioningWriter; +//! use iceberg::spec::{Literal, PartitionKey, Struct}; +//! +//! let mut clustered_writer = ClusteredWriter::new(data_file_writer_builder); +//! +//! // Create partition keys (must write in sorted order) +//! let schema = table.metadata().current_schema().clone(); +//! let partition_spec = table.metadata().default_partition_spec().as_ref().clone(); +//! +//! let partition_key_asia = PartitionKey::new( +//! partition_spec.clone(), +//! schema.clone(), +//! Struct::from_iter([Some(Literal::string("ASIA"))]), +//! ); +//! +//! let partition_key_eu = PartitionKey::new( +//! partition_spec.clone(), +//! schema.clone(), +//! Struct::from_iter([Some(Literal::string("EU"))]), +//! ); +//! +//! let partition_key_us = PartitionKey::new( +//! partition_spec.clone(), +//! schema.clone(), +//! Struct::from_iter([Some(Literal::string("US"))]), +//! ); +//! +//! // Write to partitions in sorted order (ASIA -> EU -> US) +//! // clustered_writer.write(partition_key_asia, batch_asia).await?; +//! // clustered_writer.write(partition_key_eu, batch_eu).await?; +//! // clustered_writer.write(partition_key_us, batch_us).await?; +//! // Writing back to ASIA would fail since data must be sorted! +//! +//! let data_files = clustered_writer.close().await?; +//! +//! Ok(()) +//! } +//! ``` pub mod base_writer; pub mod file_writer; -/// Provides partition-aware writers -/// TODO examples pub mod partitioning; use arrow_array::RecordBatch; diff --git a/crates/iceberg/src/writer/partitioning/mod.rs b/crates/iceberg/src/writer/partitioning/mod.rs index 36acb2a987..f63a9d0d26 100644 --- a/crates/iceberg/src/writer/partitioning/mod.rs +++ b/crates/iceberg/src/writer/partitioning/mod.rs @@ -15,6 +15,12 @@ // specific language governing permissions and limitations // under the License. +//! Partitioning writers for handling partitioned Iceberg tables. +//! +//! This module provides two strategies for writing to partitioned tables: +//! - [`FanoutWriter`](fanout_writer::FanoutWriter): Handles unsorted data by maintaining multiple active writers +//! - [`ClusteredWriter`](clustered_writer::ClusteredWriter): Optimized for pre-sorted data with single active writer + pub mod clustered_writer; pub mod fanout_writer; From 406d4168acac4f6164dde66ac579586f770b4298 Mon Sep 17 00:00:00 2001 From: Leon Lin Date: Thu, 23 Oct 2025 17:46:05 -0700 Subject: [PATCH 028/891] feat(sqllogictest): Add support for iceberg datafusion sqllogictest integration (#1764) ## Which issue does this PR close? - Relates to #1211 - Part of #1621 - Add support for iceberg datafusion integration with sqllogictests ## What changes are included in this PR? * Add sqllogictests entry point `sqllogictests.rs` * Update `Schedule` to track `EngineRunner` instead of `Engine` * Update datafusion engine to run slt file with sqllogictest ## Are these changes tested? * `cargo test --test sqllogictests` ``` Finished `test` profile [unoptimized + debuginfo] target(s) in 0.46s Running tests/sqllogictests.rs (target/debug/deps/sqllogictests-8ee6c1e0fdedebe4) running 1 test test schedule: df_test.toml ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` --------- Co-authored-by: Renjie Liu --- .github/workflows/ci.yml | 12 + Cargo.lock | 1286 ++++++++--------- Cargo.toml | 3 + crates/sqllogictest/Cargo.toml | 12 + crates/sqllogictest/src/engine/datafusion.rs | 56 +- crates/sqllogictest/src/engine/mod.rs | 73 +- crates/sqllogictest/src/error.rs | 6 + crates/sqllogictest/src/lib.rs | 7 +- crates/sqllogictest/src/schedule.rs | 24 +- .../testdata/schedules/df_test.toml | 23 + .../testdata/slts/df_test/show_tables.slt | 34 + crates/sqllogictest/tests/sqllogictests.rs | 87 ++ 12 files changed, 862 insertions(+), 761 deletions(-) create mode 100644 crates/sqllogictest/testdata/schedules/df_test.toml create mode 100644 crates/sqllogictest/testdata/slts/df_test/show_tables.slt create mode 100644 crates/sqllogictest/tests/sqllogictests.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3207ae87b1..d547397808 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -89,6 +89,18 @@ jobs: - macos-latest - windows-latest steps: + - name: Maximize build space (Ubuntu) + if: matrix.os == 'ubuntu-latest' + uses: easimon/maximize-build-space@master + with: + remove-dotnet: "true" + remove-android: "true" + remove-haskell: "true" + remove-codeql: "true" + remove-docker-images: "true" + root-reserve-mb: 10240 + temp-reserve-mb: 10240 + - uses: actions/checkout@v5 - name: Setup Rust toolchain diff --git a/Cargo.lock b/Cargo.lock index b048c3db56..e27e05129d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -44,7 +44,7 @@ checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" dependencies = [ "cfg-if", "const-random", - "getrandom 0.3.3", + "getrandom 0.3.4", "once_cell", "serde", "version_check", @@ -92,9 +92,9 @@ dependencies = [ [[package]] name = "anstream" -version = "0.6.20" +version = "0.6.21" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ae563653d1938f79b1ab1b5e668c87c76a9930414574a6583a7b7e11a8e6192" +checksum = "43d5b281e737544384e969a5ccad3f1cdd24b48086a0fc1b2a5262a26b8f4f4a" dependencies = [ "anstyle", "anstyle-parse", @@ -107,9 +107,9 @@ dependencies = [ [[package]] name = "anstyle" -version = "1.0.11" +version = "1.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "862ed96ca487e809f1c8e5a8447f6ee2cf102f846893800b20cebdf541fc6bbd" +checksum = "5192cca8006f1fd4f7237516f40fa183bb07f8fbdfedaa0036de5ea9b0b45e78" [[package]] name = "anstyle-parse" @@ -142,9 +142,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.99" +version = "1.0.100" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0674a1ddeecb70197781e945de4b3b8ffb61fa939a5597bcf48503737663100" +checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" [[package]] name = "apache-avro" @@ -195,7 +195,7 @@ dependencies = [ "serde_json", "strum 0.27.2", "strum_macros 0.27.2", - "thiserror 2.0.16", + "thiserror 2.0.17", "uuid", "zstd", ] @@ -356,7 +356,7 @@ dependencies = [ "arrow-schema", "chrono", "half", - "indexmap 2.11.0", + "indexmap 2.12.0", "lexical-core", "memchr", "num", @@ -505,9 +505,9 @@ dependencies = [ [[package]] name = "async-fs" -version = "2.1.3" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09f7e37c0ed80b2a977691c47dae8625cfb21e205827106c64f7c588766b2e50" +checksum = "8034a681df4aed8b8edbd7fbe472401ecf009251c8b40556b304567052e294c5" dependencies = [ "async-lock", "blocking", @@ -516,20 +516,20 @@ dependencies = [ [[package]] name = "async-io" -version = "2.5.0" +version = "2.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19634d6336019ef220f09fd31168ce5c184b295cbf80345437cc36094ef223ca" +checksum = "456b8a8feb6f42d237746d4b3e9a178494627745c3c56c6ea55d92ba50d026fc" dependencies = [ - "async-lock", + "autocfg", "cfg-if", "concurrent-queue", "futures-io", "futures-lite", "parking", "polling", - "rustix 1.0.8", + "rustix", "slab", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -556,9 +556,9 @@ dependencies = [ [[package]] name = "async-process" -version = "2.4.0" +version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65daa13722ad51e6ab1a1b9c01299142bc75135b337923cfa10e79bbbd669f00" +checksum = "fc50921ec0055cdd8a16de48773bfeec5c972598674347252c0399676be7da75" dependencies = [ "async-channel", "async-io", @@ -569,7 +569,7 @@ dependencies = [ "cfg-if", "event-listener", "futures-lite", - "rustix 1.0.8", + "rustix", ] [[package]] @@ -580,14 +580,14 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "async-signal" -version = "0.2.12" +version = "0.2.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f567af260ef69e1d52c2b560ce0ea230763e6fbb9214a85d768760a920e3e3c1" +checksum = "43c070bbf59cd3570b6b2dd54cd772527c7c3620fce8be898406dd3ed6adc64c" dependencies = [ "async-io", "async-lock", @@ -595,10 +595,10 @@ dependencies = [ "cfg-if", "futures-core", "futures-io", - "rustix 1.0.8", + "rustix", "signal-hook-registry", "slab", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -615,7 +615,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -641,9 +641,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-config" -version = "1.8.6" +version = "1.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bc1b40fb26027769f16960d2f4a6bc20c4bb755d403e552c8c1a73af433c246" +checksum = "37cf2b6af2a95a20e266782b4f76f1a5e12bf412a9db2de9c1e9123b9d8c0ad8" dependencies = [ "aws-credential-types", "aws-runtime", @@ -671,9 +671,9 @@ dependencies = [ [[package]] name = "aws-credential-types" -version = "1.2.6" +version = "1.2.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d025db5d9f52cbc413b167136afb3d8aeea708c0d8884783cf6253be5e22f6f2" +checksum = "faf26925f4a5b59eb76722b63c2892b1d70d06fa053c72e4a100ec308c1d47bc" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -683,9 +683,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.13.3" +version = "1.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c953fe1ba023e6b7730c0d4b031d06f267f23a46167dcbd40316644b10a17ba" +checksum = "879b6c89592deb404ba4dc0ae6b58ffd1795c78991cbb5b8bc441c48a070440d" dependencies = [ "aws-lc-sys", "zeroize", @@ -693,9 +693,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.30.0" +version = "0.32.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbfd150b5dbdb988bcc8fb1fe787eb6b7ee6180ca24da683b61ea5405f3d43ff" +checksum = "107a4e9d9cab9963e04e84bb8dee0e25f2a987f9a8bad5ed054abd439caa8f8c" dependencies = [ "bindgen", "cc", @@ -706,9 +706,9 @@ dependencies = [ [[package]] name = "aws-runtime" -version = "1.5.10" +version = "1.5.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c034a1bc1d70e16e7f4e4caf7e9f7693e4c9c24cd91cf17c2a0b21abaebc7c8b" +checksum = "bfa006bb32360ed90ac51203feafb9d02e3d21046e1fd3a450a404b90ea73e5d" dependencies = [ "aws-credential-types", "aws-sigv4", @@ -730,9 +730,9 @@ dependencies = [ [[package]] name = "aws-sdk-glue" -version = "1.119.0" +version = "1.125.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9170d379508a356718ca6c27ffa5c00b5077ecdf7ab5cb27a70834a58e00dd13" +checksum = "df436173c8acd8c207ecff1aedfd0992d44d0d6d1e84be73e30dab7746a20363" dependencies = [ "aws-credential-types", "aws-runtime", @@ -752,9 +752,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3tables" -version = "1.37.0" +version = "1.40.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba5b1975c398d8778a07ddd3de2bff50f03455907d4524c5e74eab05954ac95c" +checksum = "4f5b63d53d1536c2332c4ed11737c5de3eda1ef59c3565290b2f247d0d712a5e" dependencies = [ "aws-credential-types", "aws-runtime", @@ -774,9 +774,9 @@ dependencies = [ [[package]] name = "aws-sdk-sso" -version = "1.83.0" +version = "1.86.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "643cd43af212d2a1c4dedff6f044d7e1961e5d9e7cfe773d70f31d9842413886" +checksum = "4a0abbfab841446cce6e87af853a3ba2cc1bc9afcd3f3550dd556c43d434c86d" dependencies = [ "aws-credential-types", "aws-runtime", @@ -796,9 +796,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.84.0" +version = "1.88.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20ec4a95bd48e0db7a424356a161f8d87bd6a4f0af37204775f0da03d9e39fc3" +checksum = "9a68d675582afea0e94d38b6ca9c5aaae4ca14f1d36faa6edb19b42e687e70d7" dependencies = [ "aws-credential-types", "aws-runtime", @@ -818,9 +818,9 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.85.0" +version = "1.88.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "410309ad0df4606bc721aff0d89c3407682845453247213a0ccc5ff8801ee107" +checksum = "d30990923f4f675523c51eb1c0dec9b752fb267b36a61e83cbc219c9d86da715" dependencies = [ "aws-credential-types", "aws-runtime", @@ -841,9 +841,9 @@ dependencies = [ [[package]] name = "aws-sigv4" -version = "1.3.4" +version = "1.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "084c34162187d39e3740cb635acd73c4e3a551a36146ad6fe8883c929c9f876c" +checksum = "bffc03068fbb9c8dd5ce1c6fb240678a5cffb86fb2b7b1985c999c4b83c8df68" dependencies = [ "aws-credential-types", "aws-smithy-http", @@ -863,9 +863,9 @@ dependencies = [ [[package]] name = "aws-smithy-async" -version = "1.2.5" +version = "1.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e190749ea56f8c42bf15dd76c65e14f8f765233e6df9b0506d9d934ebef867c" +checksum = "127fcfad33b7dfc531141fda7e1c402ac65f88aca5511a4d31e2e3d2cd01ce9c" dependencies = [ "futures-util", "pin-project-lite", @@ -874,9 +874,9 @@ dependencies = [ [[package]] name = "aws-smithy-http" -version = "0.62.3" +version = "0.62.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c4dacf2d38996cf729f55e7a762b30918229917eca115de45dfa8dfb97796c9" +checksum = "3feafd437c763db26aa04e0cc7591185d0961e64c61885bece0fb9d50ceac671" dependencies = [ "aws-smithy-runtime-api", "aws-smithy-types", @@ -894,9 +894,9 @@ dependencies = [ [[package]] name = "aws-smithy-http-client" -version = "1.1.1" +version = "1.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147e8eea63a40315d704b97bf9bc9b8c1402ae94f89d5ad6f7550d963309da1b" +checksum = "1053b5e587e6fa40ce5a79ea27957b04ba660baa02b28b7436f64850152234f1" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -913,38 +913,38 @@ dependencies = [ "hyper-util", "pin-project-lite", "rustls 0.21.12", - "rustls 0.23.31", - "rustls-native-certs 0.8.1", + "rustls 0.23.33", + "rustls-native-certs 0.8.2", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.2", + "tokio-rustls 0.26.4", "tower", "tracing", ] [[package]] name = "aws-smithy-json" -version = "0.61.5" +version = "0.61.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eaa31b350998e703e9826b2104dd6f63be0508666e1aba88137af060e8944047" +checksum = "cff418fc8ec5cadf8173b10125f05c2e7e1d46771406187b2c878557d4503390" dependencies = [ "aws-smithy-types", ] [[package]] name = "aws-smithy-observability" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9364d5989ac4dd918e5cc4c4bdcc61c9be17dcd2586ea7f69e348fc7c6cab393" +checksum = "2d1881b1ea6d313f9890710d65c158bdab6fb08c91ea825f74c1c8c357baf4cc" dependencies = [ "aws-smithy-runtime-api", ] [[package]] name = "aws-smithy-query" -version = "0.60.7" +version = "0.60.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2fbd61ceb3fe8a1cb7352e42689cec5335833cd9f94103a61e98f9bb61c64bb" +checksum = "d28a63441360c477465f80c7abac3b9c4d075ca638f982e605b7dc2a2c7156c9" dependencies = [ "aws-smithy-types", "urlencoding", @@ -952,9 +952,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime" -version = "1.9.1" +version = "1.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3946acbe1ead1301ba6862e712c7903ca9bb230bdf1fbd1b5ac54158ef2ab1f" +checksum = "40ab99739082da5347660c556689256438defae3bcefd66c52b095905730e404" dependencies = [ "aws-smithy-async", "aws-smithy-http", @@ -976,9 +976,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime-api" -version = "1.9.0" +version = "1.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07f5e0fc8a6b3f2303f331b94504bbf754d85488f402d6f1dd7a6080f99afe56" +checksum = "3683c5b152d2ad753607179ed71988e8cfd52964443b4f74fd8e552d0bbfeb46" dependencies = [ "aws-smithy-async", "aws-smithy-types", @@ -993,9 +993,9 @@ dependencies = [ [[package]] name = "aws-smithy-types" -version = "1.3.2" +version = "1.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d498595448e43de7f4296b7b7a18a8a02c61ec9349128c80a368f7c3b4ab11a8" +checksum = "9f5b3a7486f6690ba25952cabf1e7d75e34d69eaff5081904a47bc79074d6457" dependencies = [ "base64-simd", "bytes", @@ -1019,18 +1019,18 @@ dependencies = [ [[package]] name = "aws-smithy-xml" -version = "0.60.10" +version = "0.60.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3db87b96cb1b16c024980f133968d52882ca0daaee3a086c6decc500f6c99728" +checksum = "e9c34127e8c624bc2999f3b657e749c1393bedc9cd97b92a804db8ced4d2e163" dependencies = [ "xmlparser", ] [[package]] name = "aws-types" -version = "1.3.8" +version = "1.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b069d19bf01e46298eaedd7c6f283fe565a59263e53eebec945f3e6398f42390" +checksum = "e2fd329bf0e901ff3f60425691410c69094dc2a1f34b331f37bfc4e9ac1565a1" dependencies = [ "aws-credential-types", "aws-smithy-async", @@ -1081,9 +1081,9 @@ checksum = "55248b47b0caf0546f7988906588779981c43bb1bc9d0c44087278f80cdb44ba" [[package]] name = "bigdecimal" -version = "0.4.8" +version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a22f228ab7a1b23027ccc6c350b72868017af7ea8356fbdf19f8d991c690013" +checksum = "560f42649de9fa436b73517378a147ec21f6c997a546581df4b4b31677828934" dependencies = [ "autocfg", "libm", @@ -1101,34 +1101,31 @@ checksum = "230c5f1ca6a325a32553f8640d31ac9b49f2411e901e427570154868b46da4f7" [[package]] name = "bindgen" -version = "0.69.5" +version = "0.72.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088" +checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" dependencies = [ "bitflags", "cexpr", "clang-sys", - "itertools 0.12.1", - "lazy_static", - "lazycell", + "itertools 0.13.0", "log", "prettyplease", "proc-macro2", "quote", "regex", - "rustc-hash 1.1.0", + "rustc-hash", "shlex", - "syn 2.0.106", - "which", + "syn 2.0.107", ] [[package]] name = "bitflags" -version = "2.9.4" +version = "2.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2261d10cca569e4643e526d8dc2e62e433cc8aba21ab764233731f8d369bf394" +checksum = "812e12b5285cc515a9c72a5c1d3b6d46a19dac5acfef5265968c166106e31dd3" dependencies = [ - "serde", + "serde_core", ] [[package]] @@ -1198,9 +1195,9 @@ dependencies = [ [[package]] name = "bon" -version = "3.7.2" +version = "3.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2529c31017402be841eb45892278a6c21a000c0a17643af326c73a73f83f0fb" +checksum = "ebeb9aaf9329dff6ceb65c689ca3db33dbf15f324909c60e4e5eef5701ce31b1" dependencies = [ "bon-macros", "rustversion", @@ -1208,9 +1205,9 @@ dependencies = [ [[package]] name = "bon-macros" -version = "3.7.2" +version = "3.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d82020dadcb845a345591863adb65d74fa8dc5c18a0b6d408470e13b7adc7005" +checksum = "77e9d642a7e3a318e37c2c9427b5a6a48aa1ad55dcd986f3034ab2239045a645" dependencies = [ "darling 0.21.3", "ident_case", @@ -1218,7 +1215,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1241,7 +1238,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1305,9 +1302,9 @@ dependencies = [ [[package]] name = "bytemuck" -version = "1.23.2" +version = "1.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3995eaeebcdf32f91f980d360f78732ddc061097ab4e39991ae7a6ace9194677" +checksum = "1fbdf580320f38b612e485521afda1ee26d10cc9884efaaa750d383e13e3c5f4" [[package]] name = "byteorder" @@ -1374,9 +1371,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.36" +version = "1.2.41" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5252b3d2648e5eedbc1a6f501e3c795e07025c1e93bbf8bbdd6eef7f447a6d54" +checksum = "ac9fe6cdbb24b6ade63616c0a0688e45bb56732262c158df3c0c4bea4ca47cb7" dependencies = [ "find-msvc-tools", "jobserver", @@ -1395,9 +1392,9 @@ dependencies = [ [[package]] name = "cfg-if" -version = "1.0.3" +version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2fd1289c04a9ea8cb22300a459a72a385d7c73d3259e2ed7dcb2af674838cfa9" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] name = "cfg_aliases" @@ -1416,7 +1413,7 @@ dependencies = [ "num-traits", "serde", "wasm-bindgen", - "windows-link 0.2.0", + "windows-link", ] [[package]] @@ -1474,9 +1471,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.47" +version = "4.5.49" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7eac00902d9d136acd712710d71823fb8ac8004ca445a89e73a41d45aa712931" +checksum = "f4512b90fa68d3a9932cea5184017c5d200f5921df706d45e853537dea51508f" dependencies = [ "clap_builder", "clap_derive", @@ -1484,9 +1481,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.47" +version = "4.5.49" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ad9bbf750e73b5884fb8a211a9424a1906c1e156724260fdae972f31d70e1d6" +checksum = "0025e98baa12e766c67ba13ff4695a887a1eba19569aad00a472546795bd6730" dependencies = [ "anstream", "anstyle", @@ -1496,21 +1493,21 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.5.47" +version = "4.5.49" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbfd7eae0b0f1a6e63d4b13c9c478de77c2eb546fba158ad50b4203dc24b9f9c" +checksum = "2a0b5487afeab2deb2ff4e03a807ad1a03ac532ff5a2cee5d86884440c7f7671" dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "clap_lex" -version = "0.7.5" +version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b94f61472cee1439c0b966b47e3aca9ae07e45d070759512cd390ea2bebc6675" +checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d" [[package]] name = "clipboard-win" @@ -1542,17 +1539,17 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fde0e0ec90c9dfb3b4b1a0891a7dcd0e2bffde2f7efed5fe7c9bb00e5bfb915e" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.59.0", ] [[package]] name = "comfy-table" -version = "7.2.0" +version = "7.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f8e18d0dca9578507f13f9803add0df13362b02c501c1c17734f0dbb52eaf0b" +checksum = "b03b7db8e0b4b2fdad6c551e634134e99ec000e5c8c3b6856c65e8bbaded7a3b" dependencies = [ "unicode-segmentation", - "unicode-width 0.2.1", + "unicode-width 0.2.2", ] [[package]] @@ -1573,7 +1570,7 @@ dependencies = [ "encode_unicode", "libc", "once_cell", - "unicode-width 0.2.1", + "unicode-width 0.2.2", "windows-sys 0.59.0", ] @@ -1747,21 +1744,21 @@ dependencies = [ [[package]] name = "csv" -version = "1.3.1" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "acdc4883a9c96732e4733212c01447ebd805833b7275a73ca3ee080fd77afdaf" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" dependencies = [ "csv-core", "itoa", "ryu", - "serde", + "serde_core", ] [[package]] name = "csv-core" -version = "0.1.12" +version = "0.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d02f3b0da4c6504f86e9cd789d8dbafab48c2321be74e9987593de5a894d93d" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" dependencies = [ "memchr", ] @@ -1773,7 +1770,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" dependencies = [ "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1807,7 +1804,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1821,7 +1818,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1832,7 +1829,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -1843,14 +1840,14 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "dary_heap" -version = "0.3.7" +version = "0.3.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04d2cd9c18b9f454ed67da600630b021a8a80bf33f8c95896ab33aaf1c26b728" +checksum = "06d2e3287df1c007e74221c49ca10a95d557349e54b3a75dc2fb14712c751f04" [[package]] name = "dashmap" @@ -2008,7 +2005,7 @@ dependencies = [ "base64 0.22.1", "half", "hashbrown 0.14.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "libc", "log", "object_store", @@ -2212,7 +2209,7 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", - "indexmap 2.11.0", + "indexmap 2.12.0", "paste", "recursive", "serde_json", @@ -2227,7 +2224,7 @@ checksum = "70fafb3a045ed6c49cfca0cd090f62cf871ca6326cc3355cb0aaf1260fa760b6" dependencies = [ "arrow", "datafusion-common", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "paste", ] @@ -2368,7 +2365,7 @@ checksum = "2821de7cb0362d12e75a5196b636a59ea3584ec1e1cc7dc6f5e34b9e8389d251" dependencies = [ "datafusion-expr", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -2382,7 +2379,7 @@ dependencies = [ "datafusion-common", "datafusion-expr", "datafusion-physical-expr", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", "recursive", @@ -2405,11 +2402,11 @@ dependencies = [ "datafusion-physical-expr-common", "half", "hashbrown 0.14.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", "paste", - "petgraph 0.8.2", + "petgraph 0.8.3", ] [[package]] @@ -2467,7 +2464,7 @@ dependencies = [ "futures", "half", "hashbrown 0.14.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", "parking_lot", @@ -2525,7 +2522,7 @@ dependencies = [ "bigdecimal", "datafusion-common", "datafusion-expr", - "indexmap 2.11.0", + "indexmap 2.12.0", "log", "recursive", "regex", @@ -2555,7 +2552,7 @@ dependencies = [ "sqllogictest", "sqlparser", "tempfile", - "thiserror 2.0.16", + "thiserror 2.0.17", "tokio", ] @@ -2591,12 +2588,12 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d630bccd429a5bb5a64b5e94f693bfc48c9f8566418fda4c494cc94f911f87cc" +checksum = "a41953f86f8a05768a6cda24def994fd2f424b04ec5c719cf89989779f199071" dependencies = [ "powerfmt", - "serde", + "serde_core", ] [[package]] @@ -2617,7 +2614,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -2627,7 +2624,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -2672,7 +2669,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.0", + "windows-sys 0.61.2", ] [[package]] @@ -2683,7 +2680,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -2734,7 +2731,7 @@ dependencies = [ "enum-ordinalize", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -2775,14 +2772,14 @@ checksum = "0d28318a75d4aead5c4db25382e8ef717932d0346600cacae6357eb5941bc5ff" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "env_filter" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "186e05a59d4c50738528153b83b0b0194d3a29507dfec16eccd4b342903397d0" +checksum = "1bf3c259d255ca70051b30e2e95b5446cdb8949ac4cd22c0d7fd634d89f568e2" dependencies = [ "log", "regex", @@ -2809,12 +2806,12 @@ checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" [[package]] name = "errno" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "778e2ac28f6c47af28e4907f13ffd1e1ddbd400980a9abd7c8df189bf578a5ad" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -2890,7 +2887,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baec6a0289d7f1fe5665586ef7340af82e3037207bef60f5785e57569776f0c8" dependencies = [ "bytes", - "rkyv 0.8.11", + "rkyv 0.8.12", "serde", "simdutf8", ] @@ -2902,15 +2899,15 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ce92ff622d6dadf7349484f42c93271a0d49b7cc4d466a936405bacbe10aa78" dependencies = [ "cfg-if", - "rustix 1.0.8", - "windows-sys 0.52.0", + "rustix", + "windows-sys 0.59.0", ] [[package]] name = "find-msvc-tools" -version = "0.1.1" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7fd99930f64d146689264c637b5af2f0233a933bef0d8570e2526bf9e083192d" +checksum = "52051878f80a721bb68ebfbc930e07b65ba72f2da88968ea5c06fd6ca3d3a127" [[package]] name = "fixedbitset" @@ -2920,9 +2917,9 @@ checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" [[package]] name = "flatbuffers" -version = "25.2.10" +version = "25.9.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1045398c1bfd89168b5fd3f1fc11f6e70b34f6f66300c87d44d3de849463abf1" +checksum = "09b6620799e7340ebd9968d2e0708eb82cf1971e9a16821e2091b6d6e475eed5" dependencies = [ "bitflags", "rustc_version", @@ -2930,9 +2927,9 @@ dependencies = [ [[package]] name = "flate2" -version = "1.1.2" +version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a3d7db9596fecd151c5f638c0ee5d5bd487b6e0ea232e5dc96d5250f6f94b1d" +checksum = "dc5a4e564e38c699f2880d3fda590bedc2e69f3f84cd48b457bd892ce61d0aa9" dependencies = [ "crc32fast", "libz-rs-sys", @@ -2979,9 +2976,9 @@ checksum = "28dd6caf6059519a65843af8fe2a3ae298b14b80179855aeb4adc2c1934ee619" [[package]] name = "fs-err" -version = "3.1.1" +version = "3.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88d7be93788013f265201256d58f04936a8079ad5dc898743aa20525f503b683" +checksum = "6ad492b2cf1d89d568a43508ab24f98501fe03f2f31c01e1d0fe7366a71745d2" dependencies = [ "autocfg", ] @@ -3078,7 +3075,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -3111,25 +3108,11 @@ dependencies = [ "slab", ] -[[package]] -name = "generator" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "605183a538e3e2a9c1038635cc5c2d194e2ee8fd0d1b66b8349fad7dbacce5a2" -dependencies = [ - "cc", - "cfg-if", - "libc", - "log", - "rustversion", - "windows", -] - [[package]] name = "generic-array" -version = "0.14.7" +version = "0.14.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2" dependencies = [ "typenum", "version_check", @@ -3144,21 +3127,21 @@ dependencies = [ "cfg-if", "js-sys", "libc", - "wasi 0.11.1+wasi-snapshot-preview1", + "wasi", "wasm-bindgen", ] [[package]] name = "getrandom" -version = "0.3.3" +version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26145e563e54f2cadc477553f1ec5ee650b00862f0a58bcd12cbdc5f0ea2d2f4" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", "js-sys", "libc", "r-efi", - "wasi 0.14.4+wasi-0.2.4", + "wasip2", "wasm-bindgen", ] @@ -3170,9 +3153,9 @@ checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" [[package]] name = "globset" -version = "0.4.16" +version = "0.4.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54a1028dfc5f5df5da8a56a73e6c153c9a9708ec57232470703592a3f18e49f5" +checksum = "eab69130804d941f8075cfd713bf8848a2c3b3f201a9457a11e6f87e1ab62305" dependencies = [ "aho-corasick", "bstr", @@ -3216,7 +3199,7 @@ dependencies = [ "futures-sink", "futures-util", "http 0.2.12", - "indexmap 2.11.0", + "indexmap 2.12.0", "slab", "tokio", "tokio-util", @@ -3235,7 +3218,7 @@ dependencies = [ "futures-core", "futures-sink", "http 1.3.1", - "indexmap 2.11.0", + "indexmap 2.12.0", "slab", "tokio", "tokio-util", @@ -3244,13 +3227,14 @@ dependencies = [ [[package]] name = "half" -version = "2.6.0" +version = "2.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "459196ed295495a68f7d7fe1d84f6c4b7ff0e21fe3017b2f283c6fac3ad803c9" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" dependencies = [ "cfg-if", "crunchy", "num-traits", + "zerocopy", ] [[package]] @@ -3283,6 +3267,12 @@ dependencies = [ "foldhash", ] +[[package]] +name = "hashbrown" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d" + [[package]] name = "hashlink" version = "0.10.0" @@ -3428,9 +3418,9 @@ dependencies = [ [[package]] name = "humantime" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b112acc8b3adf4b107a8ec20977da0273a8c386765a3ec0229bd500a1443f9f" +checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" [[package]] name = "hyper" @@ -3504,20 +3494,20 @@ dependencies = [ "http 1.3.1", "hyper 1.7.0", "hyper-util", - "rustls 0.23.31", - "rustls-native-certs 0.8.1", + "rustls 0.23.33", + "rustls-native-certs 0.8.2", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.2", + "tokio-rustls 0.26.4", "tower-service", - "webpki-roots 1.0.2", + "webpki-roots 1.0.3", ] [[package]] name = "hyper-util" -version = "0.1.16" +version = "0.1.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d9b05277c7e8da2c93a568989bb6207bef0112e8d17df7a6eda4a3cf143bc5e" +checksum = "3c6995591a8f1380fcb4ba966a252a4b29188d51d2b89e3a252f5305be65aea8" dependencies = [ "base64 0.22.1", "bytes", @@ -3531,7 +3521,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.1", "tokio", "tower-service", "tracing", @@ -3539,9 +3529,9 @@ dependencies = [ [[package]] name = "iana-time-zone" -version = "0.1.63" +version = "0.1.64" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0c919e5debc312ad217002b8048a17b7d83f80703865bbfcfebb0458b0b27d8" +checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb" dependencies = [ "android_system_properties", "core-foundation-sys", @@ -3813,7 +3803,12 @@ dependencies = [ "datafusion", "datafusion-sqllogictest", "enum-ordinalize", + "env_logger", + "iceberg", + "iceberg-datafusion", "indicatif", + "libtest-mimic", + "log", "serde", "sqllogictest", "tokio", @@ -3944,9 +3939,9 @@ dependencies = [ [[package]] name = "ignore" -version = "0.4.23" +version = "0.4.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d89fd380afde86567dfba715db065673989d6253f42b88179abd3eae47bda4b" +checksum = "81776e6f9464432afcc28d03e52eb101c93b6f0566f52aef2427663e700f0403" dependencies = [ "crossbeam-deque", "globset", @@ -3971,13 +3966,14 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.11.0" +version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2481980430f9f78649238835720ddccc57e52df14ffce1c6f37391d61b563e9" +checksum = "6717a8d2a5a929a1a2eb43a12812498ed141a0bcfb7e8f7844fbdbe4303bba9f" dependencies = [ "equivalent", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "serde", + "serde_core", ] [[package]] @@ -3989,7 +3985,7 @@ dependencies = [ "console", "number_prefix", "portable-atomic", - "unicode-width 0.2.1", + "unicode-width 0.2.2", "web-time", ] @@ -4041,15 +4037,6 @@ version = "1.70.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" -[[package]] -name = "itertools" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.13.0" @@ -4095,7 +4082,7 @@ checksum = "03343451ff899767262ec32146f6d559dd759fdadf42ff0e227c7c48f72594b4" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -4104,15 +4091,15 @@ version = "0.1.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", "libc", ] [[package]] name = "js-sys" -version = "0.3.78" +version = "0.3.81" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0b063578492ceec17683ef2f8c5e89121fbd0b172cbc280635ab7567db2738" +checksum = "ec48937a97411dcb524a265206ccd4c90bb711fca92b2792c407f268825b9305" dependencies = [ "once_cell", "wasm-bindgen", @@ -4142,17 +4129,11 @@ dependencies = [ "spin", ] -[[package]] -name = "lazycell" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55" - [[package]] name = "lexical-core" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b765c31809609075565a70b4b71402281283aeda7ecaf4818ac14a7b2ade8958" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" dependencies = [ "lexical-parse-float", "lexical-parse-integer", @@ -4163,60 +4144,53 @@ dependencies = [ [[package]] name = "lexical-parse-float" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de6f9cb01fb0b08060209a057c048fcbab8717b4c1ecd2eac66ebfe39a65b0f2" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" dependencies = [ "lexical-parse-integer", "lexical-util", - "static_assertions", ] [[package]] name = "lexical-parse-integer" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72207aae22fc0a121ba7b6d479e42cbfea549af1479c3f3a4f12c70dd66df12e" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" dependencies = [ "lexical-util", - "static_assertions", ] [[package]] name = "lexical-util" -version = "1.0.6" +version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a82e24bf537fd24c177ffbbdc6ebcc8d54732c35b50a3f28cc3f4e4c949a0b3" -dependencies = [ - "static_assertions", -] +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" [[package]] name = "lexical-write-float" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5afc668a27f460fb45a81a757b6bf2f43c2d7e30cb5a2dcd3abf294c78d62bd" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" dependencies = [ "lexical-util", "lexical-write-integer", - "static_assertions", ] [[package]] name = "lexical-write-integer" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "629ddff1a914a836fb245616a7888b62903aae58fa771e1d83943035efa0f978" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" dependencies = [ "lexical-util", - "static_assertions", ] [[package]] name = "libc" -version = "0.2.175" +version = "0.2.177" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a82ae493e598baaea5209805c49bbf2ea7de956d50d7da0da1164f9c6d28543" +checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976" [[package]] name = "libflate" @@ -4244,12 +4218,12 @@ dependencies = [ [[package]] name = "libloading" -version = "0.8.8" +version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07033963ba89ebaf1584d767badaa2e8fcec21aedea6b8c0346d487d49c28667" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" dependencies = [ "cfg-if", - "windows-targets 0.53.3", + "windows-link", ] [[package]] @@ -4270,9 +4244,9 @@ dependencies = [ [[package]] name = "libredox" -version = "0.1.9" +version = "0.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "391290121bad3d37fbddad76d8f5d1c1c314cfc646d143d7e07a3086ddff0ce3" +checksum = "416f7e718bdb06000964960ffa43b4335ad4012ae8b99060261aa4a8088d5ccb" dependencies = [ "bitflags", "libc", @@ -4330,15 +4304,9 @@ dependencies = [ [[package]] name = "linux-raw-sys" -version = "0.4.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d26c52dbd32dccf2d10cac7725f8eae5296885fb5703b261f7d0a0739ec807ab" - -[[package]] -name = "linux-raw-sys" -version = "0.9.4" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd945864f07fe9f5371a27ad7b52a172b4b499999f1d97574c9fa68373937e12" +checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039" [[package]] name = "litemap" @@ -4348,11 +4316,10 @@ checksum = "241eaef5fd12c88705a01fc1066c48c4b36e0dd4377dcdc7ec3942cea7a69956" [[package]] name = "lock_api" -version = "0.4.13" +version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96936507f153605bddfcda068dd804796c84324ed2510809e5b2a624c81da765" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" dependencies = [ - "autocfg", "scopeguard", ] @@ -4362,19 +4329,6 @@ version = "0.4.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432" -[[package]] -name = "loom" -version = "0.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca" -dependencies = [ - "cfg-if", - "generator", - "scoped-tls", - "tracing", - "tracing-subscriber", -] - [[package]] name = "lru-slab" version = "0.1.2" @@ -4401,15 +4355,6 @@ dependencies = [ "pkg-config", ] -[[package]] -name = "matchers" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" -dependencies = [ - "regex-automata", -] - [[package]] name = "md-5" version = "0.10.6" @@ -4422,9 +4367,9 @@ dependencies = [ [[package]] name = "memchr" -version = "2.7.5" +version = "2.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a282da65faaf38286cf3be983213fcf1d2e2a58700e808f83f4ea9a4804bc0" +checksum = "f52b00d39961fc5b2736ea853c9cc86238e165017a493d1d5c8eac6bdc4cc273" [[package]] name = "memoffset" @@ -4444,7 +4389,7 @@ dependencies = [ "ahash 0.8.12", "faststr", "paste", - "rustc-hash 2.1.1", + "rustc-hash", "tokio", ] @@ -4470,17 +4415,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" dependencies = [ "adler2", + "simd-adler32", ] [[package]] name = "mio" -version = "1.0.4" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78bed444cc8a2160f01cbcf811ef18cac863ad68ae8ca62092e8db51d51c761c" +checksum = "69d83b0086dc8ecf3ce9ae2874b2d1290252e2a30720bea58a5c6639b0092873" dependencies = [ "libc", - "wasi 0.11.1+wasi-snapshot-preview1", - "windows-sys 0.59.0", + "wasi", + "windows-sys 0.61.2", ] [[package]] @@ -4506,7 +4452,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -4535,23 +4481,22 @@ dependencies = [ [[package]] name = "moka" -version = "0.12.10" +version = "0.12.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9321642ca94a4282428e6ea4af8cc2ca4eac48ac7a6a4ea8f33f76d0ce70926" +checksum = "8261cd88c312e0004c1d51baad2980c66528dfdb2bee62003e643a4d8f86b077" dependencies = [ "async-lock", "crossbeam-channel", "crossbeam-epoch", "crossbeam-utils", + "equivalent", "event-listener", "futures-util", - "loom", "parking_lot", "portable-atomic", "rustc_version", "smallvec", "tagptr", - "thiserror 1.0.69", "uuid", ] @@ -4575,7 +4520,7 @@ checksum = "b40e46c845ac234bcba19db7ab252bc2778cbadd516a466d2f12b1580852d136" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -4586,22 +4531,22 @@ checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" [[package]] name = "munge" -version = "0.4.6" +version = "0.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7feb0b48aa0a25f9fe0899482c6e1379ee7a11b24a53073eacdecb9adb6dc60" +checksum = "5e17401f259eba956ca16491461b6e8f72913a0a114e39736ce404410f915a0c" dependencies = [ "munge_macro", ] [[package]] name = "munge_macro" -version = "0.4.6" +version = "0.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2e3795a5d2da581a8b252fec6022eee01aea10161a4d1bf237d4cbe47f7e988" +checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -4662,11 +4607,11 @@ dependencies = [ [[package]] name = "nu-ansi-term" -version = "0.50.1" +version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4a28e057d01f97e61255210fcff094d74ed0466038633e95017f5beb68e4399" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -4779,9 +4724,9 @@ dependencies = [ [[package]] name = "num_enum" -version = "0.7.4" +version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a973b4e44ce6cad84ce69d797acf9a044532e4184c4f267913d1b546a0727b7a" +checksum = "b1207a7e20ad57b847bbddc6776b968420d38292bbfe2089accff5e19e82454c" dependencies = [ "num_enum_derive", "rustversion", @@ -4789,14 +4734,14 @@ dependencies = [ [[package]] name = "num_enum_derive" -version = "0.7.4" +version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77e878c846a8abae00dd069496dbe8751b16ac1c3d6bd2a7283a938e8228f90d" +checksum = "ff32365de1b6743cb203b710788263c44a03de03802daf96092f2da4fe6ba4d7" dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -4807,9 +4752,9 @@ checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" [[package]] name = "object_store" -version = "0.12.3" +version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efc4f07659e11cd45a341cd24d71e683e3be65d9ff1f8150061678fe60437496" +checksum = "4c1be0c6c22ec0817cdc77d3842f721a17fd30ab6965001415b5402a74e6b740" dependencies = [ "async-trait", "base64 0.22.1", @@ -4833,7 +4778,7 @@ dependencies = [ "serde", "serde_json", "serde_urlencoded", - "thiserror 2.0.16", + "thiserror 2.0.17", "tokio", "tracing", "url", @@ -4856,9 +4801,9 @@ checksum = "a4895175b425cb1f87721b59f0f286c2092bd4af812243672510e1ac53e2e0ad" [[package]] name = "opendal" -version = "0.54.0" +version = "0.54.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffb9838d0575c6dbaf3fcec7255af8d5771996d4af900bbb6fa9a314dec00a1a" +checksum = "42afda58fa2cf50914402d132cc1caacff116a85d10c72ab2082bb7c50021754" dependencies = [ "anyhow", "backon", @@ -4873,7 +4818,7 @@ dependencies = [ "log", "md-5", "percent-encoding", - "quick-xml 0.37.5", + "quick-xml 0.38.3", "reqsign", "reqwest", "serde", @@ -4932,9 +4877,9 @@ checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" [[package]] name = "owo-colors" -version = "4.2.2" +version = "4.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48dd4f4a2c8405440fd0462561f0e5806bd0f77e86f51c761481bdd4018b545e" +checksum = "9c6901729fa79e91a0913333229e9ca5dc725089d1c363b2f4b4760709dc4a52" [[package]] name = "parking" @@ -4944,9 +4889,9 @@ checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" [[package]] name = "parking_lot" -version = "0.12.4" +version = "0.12.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70d58bf43669b5795d1576d0641cfb6fbb2057bf629506267a92807158584a13" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" dependencies = [ "lock_api", "parking_lot_core", @@ -4954,15 +4899,15 @@ dependencies = [ [[package]] name = "parking_lot_core" -version = "0.9.11" +version = "0.9.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc838d2a56b5b1a6c25f55575dfc605fabb63bb2365f6c2353ef9159aa69e4a5" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" dependencies = [ "cfg-if", "libc", "redox_syscall", "smallvec", - "windows-targets 0.52.6", + "windows-link", ] [[package]] @@ -5065,12 +5010,12 @@ dependencies = [ [[package]] name = "pem" -version = "3.0.5" +version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38af38e8470ac9dee3ce1bae1af9c1671fffc44ddfd8bd1d0a3445bf349a8ef3" +checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" dependencies = [ "base64 0.22.1", - "serde", + "serde_core", ] [[package]] @@ -5090,20 +5035,19 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "pest" -version = "2.8.1" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1db05f56d34358a8b1066f67cbb203ee3e7ed2ba674a6263a1d5ec6db2204323" +checksum = "989e7521a040efde50c3ab6bbadafbe15ab6dc042686926be59ac35d74607df4" dependencies = [ "memchr", - "thiserror 2.0.16", "ucd-trie", ] [[package]] name = "pest_derive" -version = "2.8.1" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb056d9e8ea77922845ec74a1c4e8fb17e7c218cc4fc11a15c5d25e189aa40bc" +checksum = "187da9a3030dbafabbbfb20cb323b976dc7b7ce91fcd84f2f74d6e31d378e2de" dependencies = [ "pest", "pest_generator", @@ -5111,22 +5055,22 @@ dependencies = [ [[package]] name = "pest_generator" -version = "2.8.1" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87e404e638f781eb3202dc82db6760c8ae8a1eeef7fb3fa8264b2ef280504966" +checksum = "49b401d98f5757ebe97a26085998d6c0eecec4995cad6ab7fc30ffdf4b052843" dependencies = [ "pest", "pest_meta", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "pest_meta" -version = "2.8.1" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edd1101f170f5903fde0914f899bb503d9ff5271d7ba76bbb70bea63690cc0d5" +checksum = "72f27a2cfee9f9039c4d86faa5af122a0ac3851441a34865b8a043b46be0065a" dependencies = [ "pest", "sha2", @@ -5139,18 +5083,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" dependencies = [ "fixedbitset", - "indexmap 2.11.0", + "indexmap 2.12.0", ] [[package]] name = "petgraph" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54acf3a685220b533e437e264e4d932cfbdc4cc7ec0cd232ed73c08d03b8a7ca" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset", "hashbrown 0.15.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "serde", ] @@ -5249,7 +5193,7 @@ checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -5321,16 +5265,16 @@ checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" [[package]] name = "polling" -version = "3.10.0" +version = "3.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5bd19146350fe804f7cb2669c851c03d69da628803dab0d98018142aaa5d829" +checksum = "5d0e4f59085d47d8241c88ead0f274e8a0cb551f3625263c05eb8dd897c34218" dependencies = [ "cfg-if", "concurrent-queue", "hermit-abi", "pin-project-lite", - "rustix 1.0.8", - "windows-sys 0.60.2", + "rustix", + "windows-sys 0.61.2", ] [[package]] @@ -5356,9 +5300,9 @@ dependencies = [ [[package]] name = "postgres-protocol" -version = "0.6.8" +version = "0.6.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76ff0abab4a9b844b93ef7b81f1efc0a366062aaef2cd702c76256b5dc075c54" +checksum = "fbef655056b916eb868048276cfd5d6a7dea4f81560dfd047f97c8c6fe3fcfd4" dependencies = [ "base64 0.22.1", "byteorder", @@ -5374,9 +5318,9 @@ dependencies = [ [[package]] name = "postgres-types" -version = "0.2.9" +version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613283563cd90e1dfc3518d548caee47e0e725455ed619881f5cf21f36de4b48" +checksum = "ef4605b7c057056dd35baeb6ac0c0338e4975b1f2bef0f65da953285eb007095" dependencies = [ "bytes", "fallible-iterator", @@ -5450,16 +5394,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "proc-macro-crate" -version = "3.3.0" +version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edce586971a4dfaa28950c6f18ed55e0406c1ab88bbce2c6f6293a7aaba73d35" +checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" dependencies = [ - "toml_edit", + "toml_edit 0.23.7", ] [[package]] @@ -5497,7 +5441,7 @@ dependencies = [ "prost", "prost-types", "regex", - "syn 2.0.106", + "syn 2.0.107", "tempfile", ] @@ -5511,7 +5455,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -5525,9 +5469,9 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.26" +version = "0.1.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e944464ec8536cd1beb0bbfd96987eb5e3b72f2ecdafdc5c769a37f1fa2ae1f" +checksum = "e66fcd288453b748497d8fb18bccc83a16b0518e3906d4b8df0a8d42d93dbb1c" dependencies = [ "cc", ] @@ -5543,11 +5487,11 @@ dependencies = [ [[package]] name = "ptr_meta" -version = "0.3.0" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fe9e76f66d3f9606f44e45598d155cb13ecf09f4a28199e48daf8c8fc937ea90" +checksum = "0b9a0cf95a1196af61d4f1cbdab967179516d9a4a4312af1f31948f8f6224a79" dependencies = [ - "ptr_meta_derive 0.3.0", + "ptr_meta_derive 0.3.1", ] [[package]] @@ -5563,13 +5507,13 @@ dependencies = [ [[package]] name = "ptr_meta_derive" -version = "0.3.0" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca414edb151b4c8d125c12566ab0d74dc9cdba36fb80eb7b848c15f495fd32d1" +checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -5609,10 +5553,10 @@ dependencies = [ "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash 2.1.1", - "rustls 0.23.31", - "socket2 0.5.10", - "thiserror 2.0.16", + "rustc-hash", + "rustls 0.23.33", + "socket2 0.6.1", + "thiserror 2.0.17", "tokio", "tracing", "web-time", @@ -5625,15 +5569,15 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1906b49b0c3bc04b5fe5d86a77925ae6524a19b816ae38ce1e426255f1d8a31" dependencies = [ "bytes", - "getrandom 0.3.3", + "getrandom 0.3.4", "lru-slab", "rand 0.9.2", "ring", - "rustc-hash 2.1.1", - "rustls 0.23.31", + "rustc-hash", + "rustls 0.23.33", "rustls-pki-types", "slab", - "thiserror 2.0.16", + "thiserror 2.0.17", "tinyvec", "tracing", "web-time", @@ -5648,16 +5592,16 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.1", "tracing", "windows-sys 0.60.2", ] [[package]] name = "quote" -version = "1.0.40" +version = "1.0.41" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1885c039570dc00dcb4ff087a89e185fd56bae234ddc7f056a945bf36467248d" +checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1" dependencies = [ "proc-macro2", ] @@ -5686,11 +5630,11 @@ dependencies = [ [[package]] name = "rancor" -version = "0.1.0" +version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "caf5f7161924b9d1cea0e4cabc97c372cea92b5f927fc13c6bca67157a0ad947" +checksum = "a063ea72381527c2a0561da9c80000ef822bdd7c3241b1cc1b12100e3df081ee" dependencies = [ - "ptr_meta 0.3.0", + "ptr_meta 0.3.1", ] [[package]] @@ -5751,7 +5695,7 @@ version = "0.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "99d9a13982dcf210057a8a78572b2217b667c3beacbf3a0d8b454f6f82837d38" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", ] [[package]] @@ -5771,14 +5715,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "redox_syscall" -version = "0.5.17" +version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5407465600fb0548f1442edf71dd20683c6ed326200ace4b1ef0763521bb3b77" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ "bitflags", ] @@ -5791,34 +5735,34 @@ checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac" dependencies = [ "getrandom 0.2.16", "libredox", - "thiserror 2.0.16", + "thiserror 2.0.17", ] [[package]] name = "ref-cast" -version = "1.0.24" +version = "1.0.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a0ae411dbe946a674d89546582cea4ba2bb8defac896622d6496f14c23ba5cf" +checksum = "f354300ae66f76f1c85c5f84693f0ce81d747e2c3f21a45fef496d89c960bf7d" dependencies = [ "ref-cast-impl", ] [[package]] name = "ref-cast-impl" -version = "1.0.24" +version = "1.0.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1165225c21bff1f3bbce98f5a1f889949bc902d3575308cc7b0de30b4f6d27c7" +checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "regex" -version = "1.12.1" +version = "1.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a52d8d02cacdb176ef4678de6c052efb4b3da14b78e4db683a4252762be5433" +checksum = "843bc0191f75f3e22651ae5f1e72939ab2f72a4bc30fa80a066bd66edefc24d4" dependencies = [ "aho-corasick", "memchr", @@ -5828,9 +5772,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.12" +version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "722166aa0d7438abbaa4d5cc2c649dac844e8c56d82fb3d33e9c34b5cd268fc6" +checksum = "5276caf25ac86c8d810222b3dbb938e512c55c6831a10f3e6ed1c93b84041f1c" dependencies = [ "aho-corasick", "memchr", @@ -5839,15 +5783,15 @@ dependencies = [ [[package]] name = "regex-lite" -version = "0.1.7" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "943f41321c63ef1c92fd763bfe054d2668f7f225a5c29f0105903dc2fc04ba30" +checksum = "8d942b98df5e658f56f20d592c7f868833fe38115e65c33003d8cd224b0155da" [[package]] name = "regex-syntax" -version = "0.8.6" +version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "caf4aa5b0f434c91fe5c7f1ecb6a5ece2130b02ad2a590589dda5146df959001" +checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58" [[package]] name = "regress" @@ -5870,9 +5814,9 @@ dependencies = [ [[package]] name = "rend" -version = "0.5.2" +version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a35e8a6bf28cd121053a66aa2e6a2e3eaffad4a60012179f0e864aa5ffeff215" +checksum = "cadadef317c2f20755a64d7fdc48f9e7178ee6b0e1f7fce33fa60f1d68a276e6" [[package]] name = "reqsign" @@ -5908,9 +5852,9 @@ dependencies = [ [[package]] name = "reqwest" -version = "0.12.23" +version = "0.12.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d429f34c8092b2d42c7c93cec323bb4adeb7c67698f70839adec842ec10c7ceb" +checksum = "9d0946410b9f7b082a427e4ef5c8ff541a88b357bc6c637c40db3a68ac70a36f" dependencies = [ "base64 0.22.1", "bytes", @@ -5928,15 +5872,15 @@ dependencies = [ "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.31", - "rustls-native-certs 0.8.1", + "rustls 0.23.33", + "rustls-native-certs 0.8.2", "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "sync_wrapper", "tokio", - "tokio-rustls 0.26.2", + "tokio-rustls 0.26.4", "tokio-util", "tower", "tower-http", @@ -5946,7 +5890,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams", "web-sys", - "webpki-roots 1.0.2", + "webpki-roots 1.0.3", ] [[package]] @@ -5983,18 +5927,18 @@ dependencies = [ [[package]] name = "rkyv" -version = "0.8.11" +version = "0.8.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19f5c3e5da784cd8c69d32cdc84673f3204536ca56e1fa01be31a74b92c932ac" +checksum = "35a640b26f007713818e9a9b65d34da1cf58538207b052916a83d80e43f3ffa4" dependencies = [ "bytes", "hashbrown 0.15.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "munge", - "ptr_meta 0.3.0", + "ptr_meta 0.3.1", "rancor", - "rend 0.5.2", - "rkyv_derive 0.8.11", + "rend 0.5.3", + "rkyv_derive 0.8.12", "tinyvec", "uuid", ] @@ -6012,13 +5956,13 @@ dependencies = [ [[package]] name = "rkyv_derive" -version = "0.8.11" +version = "0.8.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4270433626cffc9c4c1d3707dd681f2a2718d3d7b09ad754bec137acecda8d22" +checksum = "bd83f5f173ff41e00337d97f6572e416d022ef8a19f371817259ae960324c482" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6085,12 +6029,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "rustc-hash" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" - [[package]] name = "rustc-hash" version = "2.1.1" @@ -6111,28 +6049,15 @@ dependencies = [ [[package]] name = "rustix" -version = "0.38.44" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154" -dependencies = [ - "bitflags", - "errno", - "libc", - "linux-raw-sys 0.4.15", - "windows-sys 0.52.0", -] - -[[package]] -name = "rustix" -version = "1.0.8" +version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11181fbabf243db407ef8df94a6ce0b2f9a733bd8be4ad02b4eda9602296cac8" +checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e" dependencies = [ "bitflags", "errno", "libc", - "linux-raw-sys 0.9.4", - "windows-sys 0.60.2", + "linux-raw-sys", + "windows-sys 0.61.2", ] [[package]] @@ -6149,15 +6074,15 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.31" +version = "0.23.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0ebcbd2f03de0fc1122ad9bb24b127a5a6cd51d72604a3f3c50ac459762b6cc" +checksum = "751e04a496ca00bb97a5e043158d23d66b5aabf2e1d5aa2a0aaebb1aafe6f82c" dependencies = [ "aws-lc-rs", "once_cell", "ring", "rustls-pki-types", - "rustls-webpki 0.103.4", + "rustls-webpki 0.103.7", "subtle", "zeroize", ] @@ -6176,14 +6101,14 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7fcff2dd52b58a8d98a70243663a0d234c4e2b79235637849d15913394a247d3" +checksum = "9980d917ebb0c0536119ba501e90834767bffc3d60641457fd84a1f3fd337923" dependencies = [ "openssl-probe", "rustls-pki-types", "schannel", - "security-framework 3.4.0", + "security-framework 3.5.1", ] [[package]] @@ -6226,9 +6151,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.4" +version = "0.103.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a17884ae0c1b773f1ccd2bd4a8c72f16da897310a98b0e84bf349ad5ead92fc" +checksum = "e10b3f4191e8a80e6b43eebabfac91e5dcecebb27a71f04e820c47ec41d314bf" dependencies = [ "aws-lc-rs", "ring", @@ -6259,7 +6184,7 @@ dependencies = [ "nix 0.30.1", "radix_trie", "unicode-segmentation", - "unicode-width 0.2.1", + "unicode-width 0.2.2", "utf8parse", "windows-sys 0.59.0", ] @@ -6290,11 +6215,11 @@ dependencies = [ [[package]] name = "schannel" -version = "0.1.27" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f29ebaa345f945cec9fbbc532eb307f0fdad8161f281b6369539c8d84876b3d" +checksum = "891d81b926048e76efe18581bf793546b4c0eaf8448d72be8de2bbee5fd166e1" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6342,15 +6267,9 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.106", + "syn 2.0.107", ] -[[package]] -name = "scoped-tls" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" - [[package]] name = "scopeguard" version = "1.2.0" @@ -6399,9 +6318,9 @@ dependencies = [ [[package]] name = "security-framework" -version = "3.4.0" +version = "3.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60b369d18893388b345804dc0007963c99b7d665ae71d275812d828c6f089640" +checksum = "b3297343eaf830f66ede390ea39da1d462b6b0c1b000f420d0a83f898bbbe6ef" dependencies = [ "bitflags", "core-foundation 0.10.1", @@ -6422,11 +6341,12 @@ dependencies = [ [[package]] name = "semver" -version = "1.0.26" +version = "1.0.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56e6fa9c48d24d85fb3de5ad847117517440f6beceb7798af16b4a87d616b8d0" +checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" dependencies = [ "serde", + "serde_core", ] [[package]] @@ -6437,31 +6357,42 @@ checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" [[package]] name = "serde" -version = "1.0.219" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f0e2c6ed6606019b4e29e69dbaba95b11854410e5347d525002456dbbb786b6" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" dependencies = [ + "serde_core", "serde_derive", ] [[package]] name = "serde_bytes" -version = "0.11.17" +version = "0.11.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8437fd221bde2d4ca316d61b90e337e9e702b3820b87d63caa9ba6c02bd06d96" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" dependencies = [ "serde", + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.219" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b0276cf7f2c73365f7157c8123c21cd9a50fbbd844757af28ca1f5925fc2a00" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6472,19 +6403,20 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "serde_json" -version = "1.0.143" +version = "1.0.145" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d401abef1d108fbd9cbaebc3e46611f4b1021f714a0597a71f41ee463f5f4a5a" +checksum = "402a6f66d8c709116cf22f558eab210f5a50187f702eb4d7e5ef38d9a7f1c79c" dependencies = [ "itoa", "memchr", "ryu", "serde", + "serde_core", ] [[package]] @@ -6495,7 +6427,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6516,7 +6448,7 @@ dependencies = [ "proc-macro2", "quote", "serde", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6533,19 +6465,18 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.14.1" +version = "3.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c522100790450cf78eeac1507263d0a350d4d5b30df0c8e1fe051a10c22b376e" +checksum = "6093cd8c01b25262b84927e0f7151692158fab02d961e04c979d3903eba7ecc5" dependencies = [ "base64 0.22.1", "chrono", "hex", "indexmap 1.9.3", - "indexmap 2.11.0", + "indexmap 2.12.0", "schemars 0.9.0", "schemars 1.0.4", - "serde", - "serde_derive", + "serde_core", "serde_json", "serde_with_macros", "time", @@ -6553,14 +6484,14 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.14.1" +version = "3.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "327ada00f7d64abaac1e55a6911e90cf665aa051b9a561c7006c157f4633135e" +checksum = "a7e6c180db0816026a61afa1cff5344fb7ebded7e4d3062772179f2501481c27" dependencies = [ "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6569,7 +6500,7 @@ version = "0.9.34+deprecated" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" dependencies = [ - "indexmap 2.11.0", + "indexmap 2.12.0", "itoa", "ryu", "serde", @@ -6632,6 +6563,12 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd-adler32" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d66dc143e6b11c1eddc06d5c423cfc97062865baf299914ab64caa38182078fe" + [[package]] name = "simdutf8" version = "0.1.5" @@ -6652,7 +6589,7 @@ checksum = "297f631f50729c8c99b84667867963997ec0b50f32b2a7dbcab828ef0541e8bb" dependencies = [ "num-bigint", "num-traits", - "thiserror 2.0.16", + "thiserror 2.0.17", "time", ] @@ -6722,12 +6659,12 @@ dependencies = [ [[package]] name = "socket2" -version = "0.6.0" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "233504af464074f9d066d7b5416c5f9b894a5862a6506e306f7b816cdd6f1807" +checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -6756,7 +6693,7 @@ dependencies = [ "simdutf8", "sonic-number", "sonic-simd", - "thiserror 2.0.16", + "thiserror 2.0.17", ] [[package]] @@ -6808,7 +6745,7 @@ dependencies = [ "similar", "subst", "tempfile", - "thiserror 2.0.16", + "thiserror 2.0.17", "tracing", ] @@ -6831,7 +6768,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6865,17 +6802,17 @@ dependencies = [ "futures-util", "hashbrown 0.15.5", "hashlink", - "indexmap 2.11.0", + "indexmap 2.12.0", "log", "memchr", "once_cell", "percent-encoding", - "rustls 0.23.31", + "rustls 0.23.33", "serde", "serde_json", "sha2", "smallvec", - "thiserror 2.0.16", + "thiserror 2.0.17", "tokio", "tokio-stream", "tracing", @@ -6893,7 +6830,7 @@ dependencies = [ "quote", "sqlx-core", "sqlx-macros-core", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -6914,7 +6851,7 @@ dependencies = [ "sha2", "sqlx-core", "sqlx-sqlite", - "syn 2.0.106", + "syn 2.0.107", "tokio", "url", ] @@ -6955,7 +6892,7 @@ dependencies = [ "smallvec", "sqlx-core", "stringprep", - "thiserror 2.0.16", + "thiserror 2.0.17", "tracing", "whoami", ] @@ -6992,7 +6929,7 @@ dependencies = [ "smallvec", "sqlx-core", "stringprep", - "thiserror 2.0.16", + "thiserror 2.0.17", "tracing", "whoami", ] @@ -7016,36 +6953,30 @@ dependencies = [ "serde", "serde_urlencoded", "sqlx-core", - "thiserror 2.0.16", + "thiserror 2.0.17", "tracing", "url", ] [[package]] name = "stable_deref_trait" -version = "1.2.0" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" -version = "0.1.21" +version = "0.1.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cddb07e32ddb770749da91081d8d0ac3a16f1a569a18b20348cd371f5dead06b" +checksum = "e1f8b29fb42aafcea4edeeb6b2f2d7ecd0d969c48b4cf0d2e64aafc471dd6e59" dependencies = [ "cc", "cfg-if", "libc", "psm", - "windows-sys 0.52.0", + "windows-sys 0.59.0", ] -[[package]] -name = "static_assertions" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" - [[package]] name = "stringprep" version = "0.1.5" @@ -7088,7 +7019,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7100,7 +7031,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7133,7 +7064,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.106", + "syn 2.0.107", "typify", "walkdir", ] @@ -7157,9 +7088,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.106" +version = "2.0.107" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ede7c438028d4436d71104916910f5bb611972c5cfd7f89b8300a8186e6fada6" +checksum = "2a26dbd934e5451d21ef060c018dae56fc073894c5a7896f882928a76e6d081b" dependencies = [ "proc-macro2", "quote", @@ -7183,7 +7114,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7205,10 +7136,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16" dependencies = [ "fastrand", - "getrandom 0.3.3", + "getrandom 0.3.4", "once_cell", - "rustix 1.0.8", - "windows-sys 0.61.0", + "rustix", + "windows-sys 0.61.2", ] [[package]] @@ -7250,11 +7181,11 @@ dependencies = [ [[package]] name = "thiserror" -version = "2.0.16" +version = "2.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3467d614147380f2e4e374161426ff399c91084acd2363eaf549172b3d5e60c0" +checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8" dependencies = [ - "thiserror-impl 2.0.16", + "thiserror-impl 2.0.17", ] [[package]] @@ -7265,18 +7196,18 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "thiserror-impl" -version = "2.0.16" +version = "2.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c5e1be1c48b9172ee610da68fd9cd2770e7a4056cb3fc98710ee6906f0c7960" +checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7312,11 +7243,12 @@ dependencies = [ [[package]] name = "time" -version = "0.3.43" +version = "0.3.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83bde6f1ec10e72d583d91623c939f623002284ef622b87de38cfd546cbf2031" +checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" dependencies = [ "deranged", + "itoa", "num-conv", "powerfmt", "serde", @@ -7386,9 +7318,9 @@ dependencies = [ "parking_lot", "pin-project-lite", "signal-hook-registry", - "socket2 0.6.0", + "socket2 0.6.1", "tokio-macros", - "windows-sys 0.61.0", + "windows-sys 0.61.2", ] [[package]] @@ -7399,7 +7331,7 @@ checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7414,11 +7346,11 @@ dependencies = [ [[package]] name = "tokio-rustls" -version = "0.26.2" +version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e727b36a1a0e8b74c376ac2211e40c2c8af09fb4013c60d910495810f008e9b" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.31", + "rustls 0.23.33", "tokio", ] @@ -7454,8 +7386,8 @@ checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" dependencies = [ "serde", "serde_spanned", - "toml_datetime", - "toml_edit", + "toml_datetime 0.6.11", + "toml_edit 0.22.27", ] [[package]] @@ -7467,20 +7399,50 @@ dependencies = [ "serde", ] +[[package]] +name = "toml_datetime" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2cdb639ebbc97961c51720f858597f7f24c4fc295327923af55b74c3c724533" +dependencies = [ + "serde_core", +] + [[package]] name = "toml_edit" version = "0.22.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" dependencies = [ - "indexmap 2.11.0", + "indexmap 2.12.0", "serde", "serde_spanned", - "toml_datetime", + "toml_datetime 0.6.11", "toml_write", "winnow", ] +[[package]] +name = "toml_edit" +version = "0.23.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6485ef6d0d9b5d0ec17244ff7eb05310113c3f316f2d14200d4de56b3cb98f8d" +dependencies = [ + "indexmap 2.12.0", + "toml_datetime 0.7.3", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0cbe268d35bdb4bb5a56a2de88d0ad0eb70af5384a99d648cd4b3d04039800e" +dependencies = [ + "winnow", +] + [[package]] name = "toml_write" version = "0.1.2" @@ -7552,7 +7514,7 @@ checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7582,14 +7544,10 @@ version = "0.3.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2054a14f5307d601f88daf0553e1cbf472acc4f2c51afab632431cdcd72124d5" dependencies = [ - "matchers", "nu-ansi-term", - "once_cell", - "regex-automata", "sharded-slab", "smallvec", "thread_local", - "tracing", "tracing-core", "tracing-log", ] @@ -7632,7 +7590,7 @@ checksum = "f9534daa9fd3ed0bd911d462a37f172228077e7abf18c18a5f67199d959205f8" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -7643,14 +7601,14 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "typenum" -version = "1.18.0" +version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1dccffe3ce07af9386bfd29e80c0ab1a8205a2fc34e4bcd40364df902cfa8f3f" +checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" [[package]] name = "typify" @@ -7677,8 +7635,8 @@ dependencies = [ "semver", "serde", "serde_json", - "syn 2.0.106", - "thiserror 2.0.16", + "syn 2.0.107", + "thiserror 2.0.17", "unicode-ident", ] @@ -7695,7 +7653,7 @@ dependencies = [ "serde", "serde_json", "serde_tokenstream", - "syn 2.0.106", + "syn 2.0.107", "typify-impl", ] @@ -7763,9 +7721,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" -version = "1.0.18" +version = "1.0.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a5f39404a5da50712a4c1eecf25e90dd62b613502b7e925fd4e4d19b5c96512" +checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d" [[package]] name = "unicode-normalization" @@ -7796,9 +7754,9 @@ checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af" [[package]] name = "unicode-width" -version = "0.2.1" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1a07cc7db3810833284e8d372ccdc6da29741639ecc70c9ec107df0fa6154c" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" [[package]] name = "unsafe-libyaml" @@ -7848,7 +7806,7 @@ version = "1.18.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2f87b8aa10b915a06587d0dec516c282ff295b475d94abf425d62b57710070a2" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", "js-sys", "serde", "wasm-bindgen", @@ -7891,7 +7849,7 @@ dependencies = [ "pin-project", "rand 0.9.2", "socket2 0.5.10", - "thiserror 2.0.16", + "thiserror 2.0.17", "tokio", "tokio-stream", "tower", @@ -7919,10 +7877,10 @@ dependencies = [ "paste", "pilota", "pin-project", - "rustc-hash 2.1.1", + "rustc-hash", "scopeguard", "sonic-rs", - "thiserror 2.0.16", + "thiserror 2.0.17", "tokio", "tracing", "volo", @@ -7960,10 +7918,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] -name = "wasi" -version = "0.14.4+wasi-0.2.4" +name = "wasip2" +version = "1.0.1+wasi-0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88a5f4a424faf49c3c2c344f166f0662341d470ea185e939657aaff130f0ec4a" +checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" dependencies = [ "wit-bindgen", ] @@ -7976,9 +7934,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasm-bindgen" -version = "0.2.101" +version = "0.2.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e14915cadd45b529bb8d1f343c4ed0ac1de926144b746e2710f9cd05df6603b" +checksum = "c1da10c01ae9f1ae40cbfac0bac3b1e724b320abfcf52229f80b547c0d250e2d" dependencies = [ "cfg-if", "once_cell", @@ -7989,23 +7947,23 @@ dependencies = [ [[package]] name = "wasm-bindgen-backend" -version = "0.2.101" +version = "0.2.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e28d1ba982ca7923fd01448d5c30c6864d0a14109560296a162f80f305fb93bb" +checksum = "671c9a5a66f49d8a47345ab942e2cb93c7d1d0339065d4f8139c486121b43b19" dependencies = [ "bumpalo", "log", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-futures" -version = "0.4.51" +version = "0.4.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ca85039a9b469b38336411d6d6ced91f3fc87109a2a27b0c197663f5144dffe" +checksum = "7e038d41e478cc73bae0ff9b36c60cff1c98b8f38f8d7e8061e79ee63608ac5c" dependencies = [ "cfg-if", "js-sys", @@ -8016,9 +7974,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.101" +version = "0.2.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c3d463ae3eff775b0c45df9da45d68837702ac35af998361e2c84e7c5ec1b0d" +checksum = "7ca60477e4c59f5f2986c50191cd972e3a50d8a95603bc9434501cf156a9a119" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -8026,22 +7984,22 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.101" +version = "0.2.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7bb4ce89b08211f923caf51d527662b75bdc9c9c7aab40f86dcb9fb85ac552aa" +checksum = "9f07d2f20d4da7b26400c9f4a0511e6e0345b040694e8a75bd41d578fa4421d7" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", "wasm-bindgen-backend", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.101" +version = "0.2.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f143854a3b13752c6950862c906306adb27c7e839f7414cec8fea35beab624c1" +checksum = "bad67dc8b2a1a6e5448428adec4c3e84c43e561d8c9ee8a9e5aabeb193ec41d1" dependencies = [ "unicode-ident", ] @@ -8061,9 +8019,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.78" +version = "0.3.81" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77e4b637749ff0d92b8fad63aa1f7cff3cbe125fd49c175cd6345e7272638b12" +checksum = "9367c417a924a74cae129e6a2ae3b47fabb1f8995595ab474029da749a8be120" dependencies = [ "js-sys", "wasm-bindgen", @@ -8085,30 +8043,18 @@ version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.2", + "webpki-roots 1.0.3", ] [[package]] name = "webpki-roots" -version = "1.0.2" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e8983c3ab33d6fb807cfcdad2491c4ea8cbc8ed839181c7dfd9c67c83e261b2" +checksum = "32b130c0d2d49f8b6889abc456e795e82525204f27c42cf767cf0d7734e089b8" dependencies = [ "rustls-pki-types", ] -[[package]] -name = "which" -version = "4.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87ba24419a2078cd2b0f2ede2691b6c66d8e47836da3b6db8265ebad47afbfc7" -dependencies = [ - "either", - "home", - "once_cell", - "rustix 0.38.44", -] - [[package]] name = "whoami" version = "1.6.1" @@ -8125,115 +8071,66 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.0", -] - -[[package]] -name = "windows" -version = "0.61.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893" -dependencies = [ - "windows-collections", - "windows-core", - "windows-future", - "windows-link 0.1.3", - "windows-numerics", -] - -[[package]] -name = "windows-collections" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3beeceb5e5cfd9eb1d76b381630e82c4241ccd0d27f1a39ed41b2760b255c5e8" -dependencies = [ - "windows-core", + "windows-sys 0.61.2", ] [[package]] name = "windows-core" -version = "0.61.2" +version = "0.62.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0fdd3ddb90610c7638aa2b3a3ab2904fb9e5cdbecc643ddb3647212781c4ae3" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" dependencies = [ "windows-implement", "windows-interface", - "windows-link 0.1.3", + "windows-link", "windows-result", "windows-strings", ] -[[package]] -name = "windows-future" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e" -dependencies = [ - "windows-core", - "windows-link 0.1.3", - "windows-threading", -] - [[package]] name = "windows-implement" -version = "0.60.0" +version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a47fddd13af08290e67f4acabf4b459f647552718f683a7b415d290ac744a836" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "windows-interface" -version = "0.59.1" +version = "0.59.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bd9211b69f8dcdfa817bfd14bf1c97c9188afa36f4750130fcdf3f400eca9fa8" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] name = "windows-link" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e6ad25900d524eaabdbbb96d20b4311e1e7ae1699af4fb28c17ae66c80d798a" - -[[package]] -name = "windows-link" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "45e46c0661abb7180e7b9c281db115305d49ca1709ab8242adf09666d2173c65" - -[[package]] -name = "windows-numerics" -version = "0.2.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9150af68066c4c5c07ddc0ce30421554771e528bde427614c61038bc2c92c2b1" -dependencies = [ - "windows-core", - "windows-link 0.1.3", -] +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" [[package]] name = "windows-result" -version = "0.3.4" +version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56f42bd332cc6c8eac5af113fc0c1fd6a8fd2aa08a0119358686e5160d0586c6" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" dependencies = [ - "windows-link 0.1.3", + "windows-link", ] [[package]] name = "windows-strings" -version = "0.4.2" +version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56e6c93f3a0c3b36176cb1327a4958a0353d5d166c2a35cb268ace15e91d3b57" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" dependencies = [ - "windows-link 0.1.3", + "windows-link", ] [[package]] @@ -8269,16 +8166,16 @@ version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" dependencies = [ - "windows-targets 0.53.3", + "windows-targets 0.53.5", ] [[package]] name = "windows-sys" -version = "0.61.0" +version = "0.61.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e201184e40b2ede64bc2ea34968b28e33622acdbbf37104f0e4a33f7abe657aa" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" dependencies = [ - "windows-link 0.2.0", + "windows-link", ] [[package]] @@ -8314,28 +8211,19 @@ dependencies = [ [[package]] name = "windows-targets" -version = "0.53.3" +version = "0.53.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5fe6031c4041849d7c496a8ded650796e7b6ecc19df1a431c1a363342e5dc91" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" dependencies = [ - "windows-link 0.1.3", - "windows_aarch64_gnullvm 0.53.0", - "windows_aarch64_msvc 0.53.0", - "windows_i686_gnu 0.53.0", - "windows_i686_gnullvm 0.53.0", - "windows_i686_msvc 0.53.0", - "windows_x86_64_gnu 0.53.0", - "windows_x86_64_gnullvm 0.53.0", - "windows_x86_64_msvc 0.53.0", -] - -[[package]] -name = "windows-threading" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b66463ad2e0ea3bbf808b7f1d371311c80e115c0b71d60efc142cafbcfb057a6" -dependencies = [ - "windows-link 0.1.3", + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", ] [[package]] @@ -8352,9 +8240,9 @@ checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" [[package]] name = "windows_aarch64_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86b8d5f90ddd19cb4a147a5fa63ca848db3df085e25fee3cc10b39b6eebae764" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" [[package]] name = "windows_aarch64_msvc" @@ -8370,9 +8258,9 @@ checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" [[package]] name = "windows_aarch64_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7651a1f62a11b8cbd5e0d42526e55f2c99886c77e007179efff86c2b137e66c" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" [[package]] name = "windows_i686_gnu" @@ -8388,9 +8276,9 @@ checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" [[package]] name = "windows_i686_gnu" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1dc67659d35f387f5f6c479dc4e28f1d4bb90ddd1a5d3da2e5d97b42d6272c3" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" [[package]] name = "windows_i686_gnullvm" @@ -8400,9 +8288,9 @@ checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" [[package]] name = "windows_i686_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ce6ccbdedbf6d6354471319e781c0dfef054c81fbc7cf83f338a4296c0cae11" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" [[package]] name = "windows_i686_msvc" @@ -8418,9 +8306,9 @@ checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" [[package]] name = "windows_i686_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "581fee95406bb13382d2f65cd4a908ca7b1e4c2f1917f143ba16efe98a589b5d" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" [[package]] name = "windows_x86_64_gnu" @@ -8436,9 +8324,9 @@ checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" [[package]] name = "windows_x86_64_gnu" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e55b5ac9ea33f2fc1716d1742db15574fd6fc8dadc51caab1c16a3d3b4190ba" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" [[package]] name = "windows_x86_64_gnullvm" @@ -8454,9 +8342,9 @@ checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" [[package]] name = "windows_x86_64_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a6e035dd0599267ce1ee132e51c27dd29437f63325753051e71dd9e42406c57" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" [[package]] name = "windows_x86_64_msvc" @@ -8472,9 +8360,9 @@ checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" [[package]] name = "windows_x86_64_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "271414315aff87387382ec3d271b52d7ae78726f5d44ac98b4f4030c91880486" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" [[package]] name = "winnow" @@ -8487,9 +8375,9 @@ dependencies = [ [[package]] name = "wit-bindgen" -version = "0.45.1" +version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c573471f125075647d03df72e026074b7203790d41351cd6edc96f46bcccd36" +checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" [[package]] name = "writeable" @@ -8547,7 +8435,7 @@ checksum = "38da3c9736e16c5d3c8c597a9aaa5d1fa565d0532ae05e27c24aa62fb32c0ab6" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", "synstructure", ] @@ -8568,7 +8456,7 @@ checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] @@ -8588,15 +8476,15 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", "synstructure", ] [[package]] name = "zeroize" -version = "1.8.1" +version = "1.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ced3678a2879b30306d323f4542626697a464a97c0a07c9aebf7ebca65cd4dde" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" [[package]] name = "zerotrie" @@ -8628,7 +8516,7 @@ checksum = "5b96237efa0c878c64bd89c436f661be4e46b2f3eff1ebb976f7ef2321d2f58f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.107", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 7735d702cf..6167380594 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -68,6 +68,7 @@ datafusion-sqllogictest = "48.0.1" derive_builder = "0.20" dirs = "6" enum-ordinalize = "4.3.0" +env_logger = "0.11.8" expect-test = "1" faststr = "0.2.31" fnv = "1.0.7" @@ -84,7 +85,9 @@ iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3table iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } indicatif = "0.17" itertools = "0.13" +libtest-mimic = "0.8.1" linkedbytes = "0.1.8" +log = "0.4.28" metainfo = "0.7.14" mimalloc = "0.1.46" mockall = "0.13.1" diff --git a/crates/sqllogictest/Cargo.toml b/crates/sqllogictest/Cargo.toml index e335de433e..e826ad7ae0 100644 --- a/crates/sqllogictest/Cargo.toml +++ b/crates/sqllogictest/Cargo.toml @@ -30,13 +30,25 @@ async-trait = { workspace = true } datafusion = { workspace = true } datafusion-sqllogictest = { workspace = true } enum-ordinalize = { workspace = true } +env_logger = { workspace = true } +iceberg = { workspace = true } +iceberg-datafusion = { workspace = true } indicatif = { workspace = true } +log = { workspace = true } sqllogictest = { workspace = true } toml = { workspace = true } serde = { workspace = true } tracing = { workspace = true } tokio = { workspace = true } +[dev-dependencies] +libtest-mimic = { workspace = true } + +[[test]] +harness = false +name = "sqllogictests" +path = "tests/sqllogictests.rs" + [package.metadata.cargo-machete] # These dependencies are added to ensure minimal dependency version ignored = ["enum-ordinalize"] diff --git a/crates/sqllogictest/src/engine/datafusion.rs b/crates/sqllogictest/src/engine/datafusion.rs index 49d7273d60..b3e37d9206 100644 --- a/crates/sqllogictest/src/engine/datafusion.rs +++ b/crates/sqllogictest/src/engine/datafusion.rs @@ -15,53 +15,77 @@ // specific language governing permissions and limitations // under the License. +use std::collections::HashMap; use std::path::{Path, PathBuf}; use std::sync::Arc; -use anyhow::{Context, anyhow}; use datafusion::catalog::CatalogProvider; use datafusion::prelude::{SessionConfig, SessionContext}; use datafusion_sqllogictest::DataFusion; +use iceberg::CatalogBuilder; +use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; +use iceberg_datafusion::IcebergCatalogProvider; use indicatif::ProgressBar; -use sqllogictest::runner::AsyncDB; use toml::Table as TomlTable; -use crate::engine::EngineRunner; +use crate::engine::{EngineRunner, run_slt_with_runner}; use crate::error::Result; pub struct DataFusionEngine { - datafusion: DataFusion, + test_data_path: PathBuf, + session_context: SessionContext, } #[async_trait::async_trait] impl EngineRunner for DataFusionEngine { async fn run_slt_file(&mut self, path: &Path) -> Result<()> { - let content = std::fs::read_to_string(path) - .with_context(|| format!("Failed to read slt file {:?}", path)) - .map_err(|e| anyhow!(e))?; + let ctx = self.session_context.clone(); + let testdata = self.test_data_path.clone(); - self.datafusion - .run(content.as_str()) - .await - .with_context(|| format!("Failed to run slt file {:?}", path)) - .map_err(|e| anyhow!(e))?; + let runner = sqllogictest::Runner::new({ + move || { + let ctx = ctx.clone(); + let testdata = testdata.clone(); + async move { + // Everything here is owned; no `self` capture. + Ok(DataFusion::new(ctx, testdata, ProgressBar::new(100))) + } + } + }); - Ok(()) + run_slt_with_runner(runner, path).await } } impl DataFusionEngine { pub async fn new(config: TomlTable) -> Result { - let session_config = SessionConfig::new().with_target_partitions(4); + let session_config = SessionConfig::new() + .with_target_partitions(4) + .with_information_schema(true); let ctx = SessionContext::new_with_config(session_config); ctx.register_catalog("default", Self::create_catalog(&config).await?); Ok(Self { - datafusion: DataFusion::new(ctx, PathBuf::from("testdata"), ProgressBar::new(100)), + test_data_path: PathBuf::from("testdata"), + session_context: ctx, }) } async fn create_catalog(_: &TomlTable) -> anyhow::Result> { - todo!() + // TODO: support dynamic catalog configuration + // See: https://github.com/apache/iceberg-rust/issues/1780 + let catalog = MemoryCatalogBuilder::default() + .load( + "memory", + HashMap::from([( + MEMORY_CATALOG_WAREHOUSE.to_string(), + "memory://".to_string(), + )]), + ) + .await?; + + Ok(Arc::new( + IcebergCatalogProvider::try_new(Arc::new(catalog)).await?, + )) } } diff --git a/crates/sqllogictest/src/engine/mod.rs b/crates/sqllogictest/src/engine/mod.rs index a1d34dd9bc..f8a405a7cc 100644 --- a/crates/sqllogictest/src/engine/mod.rs +++ b/crates/sqllogictest/src/engine/mod.rs @@ -19,70 +19,75 @@ mod datafusion; use std::path::Path; +use anyhow::anyhow; +use sqllogictest::{AsyncDB, MakeConnection, Runner, parse_file}; use toml::Table as TomlTable; use crate::engine::datafusion::DataFusionEngine; -use crate::error::Result; +use crate::error::{Error, Result}; -const KEY_TYPE: &str = "type"; const TYPE_DATAFUSION: &str = "datafusion"; #[async_trait::async_trait] -pub trait EngineRunner: Sized { +pub trait EngineRunner: Send { async fn run_slt_file(&mut self, path: &Path) -> Result<()>; } -pub enum Engine { - DataFusion(DataFusionEngine), +pub async fn load_engine_runner( + engine_type: &str, + cfg: TomlTable, +) -> Result> { + match engine_type { + TYPE_DATAFUSION => Ok(Box::new(DataFusionEngine::new(cfg).await?)), + _ => Err(anyhow::anyhow!("Unsupported engine type: {}", engine_type).into()), + } } -impl Engine { - pub async fn new(config: TomlTable) -> Result { - let engine_type = config - .get(KEY_TYPE) - .ok_or_else(|| anyhow::anyhow!("Missing required key: {KEY_TYPE}"))? - .as_str() - .ok_or_else(|| anyhow::anyhow!("Config value for {KEY_TYPE} must be a string"))?; - - match engine_type { - TYPE_DATAFUSION => { - let engine = DataFusionEngine::new(config).await?; - Ok(Engine::DataFusion(engine)) - } - _ => Err(anyhow::anyhow!("Unsupported engine type: {engine_type}").into()), +pub async fn run_slt_with_runner( + mut runner: Runner, + step_slt_file: impl AsRef, +) -> Result<()> +where + D: AsyncDB + Send + 'static, + M: MakeConnection + Send + 'static, +{ + let path = step_slt_file.as_ref().canonicalize()?; + let records = parse_file(&path).map_err(|e| Error(anyhow!("parsing slt file failed: {e}")))?; + + for record in records { + if let Err(err) = runner.run_async(record).await { + return Err(Error(anyhow!("SLT record execution failed: {err}"))); } } - pub async fn run_slt_file(&mut self, path: &Path) -> Result<()> { - match self { - Engine::DataFusion(engine) => engine.run_slt_file(path).await, - } - } + Ok(()) } #[cfg(test)] mod tests { - use toml::Table as TomlTable; - - use crate::engine::Engine; + use crate::engine::{TYPE_DATAFUSION, load_engine_runner}; #[tokio::test] - async fn test_engine_new_missing_type_key() { - let config = TomlTable::new(); - let result = Engine::new(config).await; + async fn test_engine_invalid_type() { + let input = r#" + [engines] + random = { type = "random_engine", url = "http://localhost:8181" } + "#; + let tbl = toml::from_str(input).unwrap(); + let result = load_engine_runner("random_engine", tbl).await; assert!(result.is_err()); } #[tokio::test] - async fn test_engine_invalid_type() { + async fn test_load_datafusion() { let input = r#" [engines] - random = { type = "random_engine", url = "http://localhost:8181" } + df = { type = "datafusion" } "#; let tbl = toml::from_str(input).unwrap(); - let result = Engine::new(tbl).await; + let result = load_engine_runner(TYPE_DATAFUSION, tbl).await; - assert!(result.is_err()); + assert!(result.is_ok()); } } diff --git a/crates/sqllogictest/src/error.rs b/crates/sqllogictest/src/error.rs index 01eb364e2a..2bf5a09d6b 100644 --- a/crates/sqllogictest/src/error.rs +++ b/crates/sqllogictest/src/error.rs @@ -44,3 +44,9 @@ impl From for Error { Self(value) } } + +impl From for Error { + fn from(value: std::io::Error) -> Self { + Self(value.into()) + } +} diff --git a/crates/sqllogictest/src/lib.rs b/crates/sqllogictest/src/lib.rs index 7b17727d1a..34a3c654aa 100644 --- a/crates/sqllogictest/src/lib.rs +++ b/crates/sqllogictest/src/lib.rs @@ -18,9 +18,6 @@ // This lib contains codes copied from // [Apache Datafusion](https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest) -#[allow(dead_code)] mod engine; -#[allow(dead_code)] -mod error; -#[allow(dead_code)] -mod schedule; +pub mod error; +pub mod schedule; diff --git a/crates/sqllogictest/src/schedule.rs b/crates/sqllogictest/src/schedule.rs index d3284949bc..7c13ad4d12 100644 --- a/crates/sqllogictest/src/schedule.rs +++ b/crates/sqllogictest/src/schedule.rs @@ -24,11 +24,11 @@ use serde::{Deserialize, Serialize}; use toml::{Table as TomlTable, Value}; use tracing::info; -use crate::engine::Engine; +use crate::engine::{EngineRunner, load_engine_runner}; pub struct Schedule { /// Engine names to engine instances - engines: HashMap, + engines: HashMap>, /// List of test steps to run steps: Vec, /// Path of the schedule file @@ -44,7 +44,11 @@ pub struct Step { } impl Schedule { - pub fn new(engines: HashMap, steps: Vec, schedule_file: String) -> Self { + pub fn new( + engines: HashMap>, + steps: Vec, + schedule_file: String, + ) -> Self { Self { engines, steps, @@ -102,7 +106,9 @@ impl Schedule { Ok(()) } - async fn parse_engines(table: &TomlTable) -> anyhow::Result> { + async fn parse_engines( + table: &TomlTable, + ) -> anyhow::Result>> { let engines_tbl = table .get("engines") .with_context(|| "Schedule file must have an 'engines' table")? @@ -117,9 +123,13 @@ impl Schedule { .ok_or_else(|| anyhow!("Config of engine '{name}' is not a table"))? .clone(); - let engine = Engine::new(cfg_tbl) - .await - .with_context(|| format!("Failed to construct engine '{name}'"))?; + let engine_type = cfg_tbl + .get("type") + .ok_or_else(|| anyhow::anyhow!("Engine {name} doesn't have a 'type' field"))? + .as_str() + .ok_or_else(|| anyhow::anyhow!("Engine {name} type must be a string"))?; + + let engine = load_engine_runner(engine_type, cfg_tbl.clone()).await?; if engines.insert(name.clone(), engine).is_some() { return Err(anyhow!("Duplicate engine '{name}'")); diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml new file mode 100644 index 0000000000..0733744951 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/show_tables.slt" \ No newline at end of file diff --git a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt new file mode 100644 index 0000000000..34709d7359 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +query TTTT rowsort +SHOW TABLES +---- +datafusion information_schema columns VIEW +datafusion information_schema df_settings VIEW +datafusion information_schema parameters VIEW +datafusion information_schema routines VIEW +datafusion information_schema schemata VIEW +datafusion information_schema tables VIEW +datafusion information_schema views VIEW +default information_schema columns VIEW +default information_schema df_settings VIEW +default information_schema parameters VIEW +default information_schema routines VIEW +default information_schema schemata VIEW +default information_schema tables VIEW +default information_schema views VIEW \ No newline at end of file diff --git a/crates/sqllogictest/tests/sqllogictests.rs b/crates/sqllogictest/tests/sqllogictests.rs new file mode 100644 index 0000000000..434d9d78cc --- /dev/null +++ b/crates/sqllogictest/tests/sqllogictests.rs @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::fs; +use std::path::PathBuf; + +use iceberg_sqllogictest::schedule::Schedule; +use libtest_mimic::{Arguments, Trial}; +use tokio::runtime::Handle; + +pub fn main() { + env_logger::init(); + + let rt = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .unwrap(); + + // Parse command line arguments + let args = Arguments::from_args(); + + let tests = collect_trials(rt.handle().clone()).unwrap(); + let result = libtest_mimic::run(&args, tests); + + drop(rt); + + result.exit(); +} + +pub(crate) fn collect_trials(handle: Handle) -> anyhow::Result> { + let schedule_files = collect_schedule_files()?; + log::debug!( + "Found {} schedules files: {:?}", + schedule_files.len(), + &schedule_files + ); + let mut trials = Vec::with_capacity(schedule_files.len()); + for schedule_file in schedule_files { + let h = handle.clone(); + let trial_name = format!( + "schedule: {}", + schedule_file + .file_name() + .expect("Schedule file should have a name") + .to_string_lossy() + ); + let trial = Trial::test(trial_name, move || { + Ok(h.block_on(run_schedule(schedule_file.clone()))?) + }); + trials.push(trial); + } + Ok(trials) +} + +pub(crate) fn collect_schedule_files() -> anyhow::Result> { + let dir = PathBuf::from(format!("{}/testdata/schedules", env!("CARGO_MANIFEST_DIR"))); + let mut schedule_files = Vec::with_capacity(32); + for entry in fs::read_dir(&dir)? { + let entry = entry?; + let path = entry.path(); + if path.is_file() { + schedule_files.push(fs::canonicalize(dir.join(path))?); + } + } + Ok(schedule_files) +} + +pub(crate) async fn run_schedule(schedule_file: PathBuf) -> anyhow::Result<()> { + let schedules = Schedule::from_file(schedule_file).await?; + schedules.run().await?; + + Ok(()) +} From ddbcae46d4b6739b0d21fbf44de85b140b3d272a Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Fri, 24 Oct 2025 08:03:49 -0400 Subject: [PATCH 029/891] fix(build): Pin home version after merging #1764 (#1783) --- Cargo.toml | 1 + crates/integrations/playground/Cargo.toml | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/Cargo.toml b/Cargo.toml index 6167380594..1c007376ad 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -75,6 +75,7 @@ fnv = "1.0.7" fs-err = "3.1.0" futures = "0.3" hive_metastore = "0.2.0" +home = "=0.5.11" http = "1.2" iceberg = { version = "0.7.0", path = "./crates/iceberg" } iceberg-catalog-glue = { version = "0.7.0", path = "./crates/catalog/glue" } diff --git a/crates/integrations/playground/Cargo.toml b/crates/integrations/playground/Cargo.toml index 8ba983d058..3f6774be19 100644 --- a/crates/integrations/playground/Cargo.toml +++ b/crates/integrations/playground/Cargo.toml @@ -33,6 +33,7 @@ datafusion = { workspace = true } datafusion-cli = { workspace = true } dirs = { workspace = true } fs-err = { workspace = true } +home = { workspace = true } iceberg = { workspace = true } iceberg-catalog-rest = { workspace = true } iceberg-datafusion = { workspace = true } @@ -45,4 +46,4 @@ tracing-subscriber = { workspace = true } [package.metadata.cargo-machete] # These dependencies are added to ensure minimal dependency version -ignored = ["stacker", "mimalloc"] +ignored = ["stacker", "mimalloc", "home"] From 104a01679cecbe4f6c1fba4d8aaca5ce099a4693 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Sun, 26 Oct 2025 20:32:54 -0700 Subject: [PATCH 030/891] minor: Update Cargo.lock to add home (#1785) ## Which issue does this PR close? - A minor followup of #1783 ## What changes are included in this PR? - Update Cargo.lock to add home ## Are these changes tested? No --- Cargo.lock | 1 + 1 file changed, 1 insertion(+) diff --git a/Cargo.lock b/Cargo.lock index e27e05129d..1e520bd448 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3783,6 +3783,7 @@ dependencies = [ "datafusion-cli", "dirs", "fs-err", + "home", "iceberg", "iceberg-catalog-rest", "iceberg-datafusion", From 1691419c4ca42e7979373188ac239fd6569df885 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 27 Oct 2025 12:22:31 +0800 Subject: [PATCH 031/891] chore(deps): Bump aws-sdk-s3tables from 1.40.0 to 1.41.0 (#1790) --- Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1e520bd448..4895381610 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -752,9 +752,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3tables" -version = "1.40.0" +version = "1.41.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f5b63d53d1536c2332c4ed11737c5de3eda1ef59c3565290b2f247d0d712a5e" +checksum = "761f176da526badb4c3dbd67ee1da2faf3dc1e537ed229355f7590d80595ae35" dependencies = [ "aws-credential-types", "aws-runtime", @@ -2669,7 +2669,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2811,7 +2811,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -4612,7 +4612,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5595,7 +5595,7 @@ dependencies = [ "once_cell", "socket2 0.6.1", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.59.0", ] [[package]] @@ -6058,7 +6058,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -7140,7 +7140,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -8072,7 +8072,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] From 0141385c581cc1ae6dd47c7a4b293aa75deda8f0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 27 Oct 2025 13:09:34 +0800 Subject: [PATCH 032/891] chore(deps): Bump rand from 0.8.5 to 0.9.2 (#1789) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [rand](https://github.com/rust-random/rand) from 0.8.5 to 0.9.2.
Changelog

Sourced from rand's changelog.

[0.9.2 — 2025-07-20]

Deprecated

  • Deprecate rand::rngs::mock module and StepRng generator (#1634)

Additions

  • Enable WeightedIndex<usize> (de)serialization (#1646)

[0.9.1] - 2025-04-17

Security and unsafe

  • Revise "not a crypto library" policy again (#1565)
  • Remove zerocopy dependency from rand (#1579)

Fixes

  • Fix feature simd_support for recent nightly rust (#1586)

Changes

  • Allow fn rand::seq::index::sample_weighted and fn IndexedRandom::choose_multiple_weighted to return fewer than amount results (#1623), reverting an undocumented change (#1382) to the previous release.

Additions

  • Add rand::distr::Alphabetic distribution. (#1587)
  • Re-export rand_core (#1604)

[0.9.0] - 2025-01-27

Security and unsafe

  • Policy: "rand is not a crypto library" (#1514)
  • Remove fork-protection from ReseedingRng and ThreadRng. Instead, it is recommended to call ThreadRng::reseed on fork. (#1379)
  • Use zerocopy to replace some unsafe code (#1349, #1393, #1446, #1502)

Dependencies

  • Bump the MSRV to 1.63.0 (#1207, #1246, #1269, #1341, #1416, #1536); note that 1.60.0 may work for dependents when using --ignore-rust-version
  • Update to rand_core v0.9.0 (#1558)

Features

  • Support std feature without getrandom or rand_chacha (#1354)
  • Enable feature small_rng by default (#1455)
  • Remove implicit feature rand_chacha; use std_rng instead. (#1473)
  • Rename feature serde1 to serde (#1477)
  • Rename feature getrandom to os_rng (#1537)
  • Add feature thread_rng (#1547)

API changes: rand_core traits

  • Add fn RngCore::read_adapter implementing std::io::Read (#1267)
  • Add trait CryptoBlockRng: BlockRngCore; make trait CryptoRng: RngCore (#1273)
  • Add traits TryRngCore, TryCryptoRng (#1424, #1499)
  • Rename fn SeedableRng::from_rng -> try_from_rng and add infallible variant fn from_rng (#1424)
  • Rename fn SeedableRng::from_entropy -> from_os_rng and add fallible variant fn try_from_os_rng (#1424)
  • Add bounds Clone and AsRef to associated type SeedableRng::Seed (#1491)

API changes: Rng trait and top-level fns

  • Rename fn rand::thread_rng() to rand::rng() and remove from the prelude (#1506)

... (truncated)

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rand&package-manager=cargo&previous-version=0.8.5&new-version=0.9.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu From f1b1ddfcb3926052c32d20106aa4b4e31ff8d59f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 27 Oct 2025 13:51:34 +0800 Subject: [PATCH 033/891] chore(deps): Bump actions/download-artifact from 5 to 6 (#1788) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 5 to 6.
Release notes

Sourced from actions/download-artifact's releases.

v6.0.0

What's Changed

BREAKING CHANGE: this update supports Node v24.x. This is not a breaking change per-se but we're treating it as such.

New Contributors

Full Changelog: https://github.com/actions/download-artifact/compare/v5...v6.0.0

Commits
  • 018cc2c Merge pull request #438 from actions/danwkennedy/prepare-6.0.0
  • 815651c Revert "Remove github.dep.yml"
  • bb3a066 Remove github.dep.yml
  • fa1ce46 Prepare v6.0.0
  • 4a24838 Merge pull request #431 from danwkennedy/patch-1
  • 5e3251c Readme: spell out the first use of GHES
  • abefc31 Merge pull request #424 from actions/yacaovsnc/update_readme
  • ac43a60 Update README with artifact extraction details
  • de96f46 Merge pull request #417 from actions/yacaovsnc/update_readme
  • 7993cb4 Remove migration guide for artifact download changes
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/download-artifact&package-manager=github_actions&previous-version=5&new-version=6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/release_python.yml | 2 +- .github/workflows/release_python_nightly.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 828e4e16b3..3608614918 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -178,7 +178,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v5 + uses: actions/download-artifact@v6 with: pattern: wheels-* merge-multiple: true diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 5e6fc3ce38..6624925d58 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -120,7 +120,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v5 + uses: actions/download-artifact@v6 with: pattern: wheels-* merge-multiple: true From 044b45b9e7d2b1444c32463cc836c5a0e348fbb4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 27 Oct 2025 14:24:04 +0800 Subject: [PATCH 034/891] chore(deps): Bump actions/upload-artifact from 4 to 5 (#1787) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5.
Release notes

Sourced from actions/upload-artifact's releases.

v5.0.0

What's Changed

BREAKING CHANGE: this update supports Node v24.x. This is not a breaking change per-se but we're treating it as such.

New Contributors

Full Changelog: https://github.com/actions/upload-artifact/compare/v4...v5.0.0

v4.6.2

What's Changed

New Contributors

Full Changelog: https://github.com/actions/upload-artifact/compare/v4...v4.6.2

v4.6.1

What's Changed

Full Changelog: https://github.com/actions/upload-artifact/compare/v4...v4.6.1

v4.6.0

What's Changed

Full Changelog: https://github.com/actions/upload-artifact/compare/v4...v4.6.0

v4.5.0

What's Changed

New Contributors

... (truncated)

Commits
  • 330a01c Merge pull request #734 from actions/danwkennedy/prepare-5.0.0
  • 03f2824 Update github.dep.yml
  • 905a1ec Prepare v5.0.0
  • 2d9f9cd Merge pull request #725 from patrikpolyak/patch-1
  • 9687587 Merge branch 'main' into patch-1
  • 2848b2c Merge pull request #727 from danwkennedy/patch-1
  • 9b51177 Spell out the first use of GHES
  • cd231ca Update GHES guidance to include reference to Node 20 version
  • de65e23 Merge pull request #712 from actions/nebuk89-patch-1
  • 8747d8c Update README.md
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/upload-artifact&package-manager=github_actions&previous-version=4&new-version=5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/release_python.yml | 4 ++-- .github/workflows/release_python_nightly.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 3608614918..8208373cf0 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -107,7 +107,7 @@ jobs: command: sdist args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v5 with: name: wheels-sdist path: bindings/python/dist @@ -159,7 +159,7 @@ jobs: command: build args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v5 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 6624925d58..821f95ac4e 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -56,7 +56,7 @@ jobs: args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v5 with: name: wheels-sdist path: bindings/python/dist @@ -102,7 +102,7 @@ jobs: args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v5 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist From cd6005724ff511b8a2844cf78bd54c9bb5a5493c Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Mon, 27 Oct 2025 07:30:59 -0400 Subject: [PATCH 035/891] fix(reader): filter row groups when FileScanTask contains byte ranges (#1779) --- crates/iceberg/src/arrow/reader.rs | 254 ++++++++++++++++++++++++++++- 1 file changed, 248 insertions(+), 6 deletions(-) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index ff4cff0a64..720b9363b9 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -224,15 +224,16 @@ impl ArrowReader { } }; - // There are two possible sources both for potential lists of selected RowGroup indices, - // and for `RowSelection`s. - // Selected RowGroup index lists can come from two sources: + // There are three possible sources for potential lists of selected RowGroup indices, + // and two for `RowSelection`s. + // Selected RowGroup index lists can come from three sources: + // * When task.start and task.length specify a byte range (file splitting); // * When there are equality delete files that are applicable; // * When there is a scan predicate and row_group_filtering_enabled = true. // `RowSelection`s can be created in either or both of the following cases: // * When there are positional delete files that are applicable; // * When there is a scan predicate and row_selection_enabled = true - // Note that, in the former case we only perform row group filtering when + // Note that row group filtering from predicates only happens when // there is a scan predicate AND row_group_filtering_enabled = true, // but we perform row selection filtering if there are applicable // equality delete files OR (there is a scan predicate AND row_selection_enabled), @@ -241,6 +242,17 @@ impl ArrowReader { let mut selected_row_group_indices = None; let mut row_selection = None; + // Filter row groups based on byte range from task.start and task.length. + // If both start and length are 0, read the entire file (backwards compatibility). + if task.start != 0 || task.length != 0 { + let byte_range_filtered_row_groups = Self::filter_row_groups_by_byte_range( + record_batch_stream_builder.metadata(), + task.start, + task.length, + )?; + selected_row_group_indices = Some(byte_range_filtered_row_groups); + } + if let Some(predicate) = final_predicate { let (iceberg_field_ids, field_id_map) = Self::build_field_id_set_and_map( record_batch_stream_builder.parquet_schema(), @@ -256,14 +268,26 @@ impl ArrowReader { record_batch_stream_builder = record_batch_stream_builder.with_row_filter(row_filter); if row_group_filtering_enabled { - let result = Self::get_selected_row_group_indices( + let predicate_filtered_row_groups = Self::get_selected_row_group_indices( &predicate, record_batch_stream_builder.metadata(), &field_id_map, &task.schema, )?; - selected_row_group_indices = Some(result); + // Merge predicate-based filtering with byte range filtering (if present) + // by taking the intersection of both filters + selected_row_group_indices = match selected_row_group_indices { + Some(byte_range_filtered) => { + // Keep only row groups that are in both filters + let intersection: Vec = byte_range_filtered + .into_iter() + .filter(|idx| predicate_filtered_row_groups.contains(idx)) + .collect(); + Some(intersection) + } + None => Some(predicate_filtered_row_groups), + }; } if row_selection_enabled { @@ -717,6 +741,36 @@ impl ArrowReader { Ok(results.into_iter().flatten().collect::>().into()) } + + /// Filters row groups by byte range to support Iceberg's file splitting. + /// + /// Iceberg splits large files at row group boundaries, so we only read row groups + /// whose byte ranges overlap with [start, start+length). + fn filter_row_groups_by_byte_range( + parquet_metadata: &Arc, + start: u64, + length: u64, + ) -> Result> { + let row_groups = parquet_metadata.row_groups(); + let mut selected = Vec::new(); + let end = start + length; + + // Row groups are stored sequentially after the 4-byte magic header. + let mut current_byte_offset = 4u64; + + for (idx, row_group) in row_groups.iter().enumerate() { + let row_group_size = row_group.compressed_size() as u64; + let row_group_end = current_byte_offset + row_group_size; + + if current_byte_offset < end && start < row_group_end { + selected.push(idx); + } + + current_byte_offset = row_group_end; + } + + Ok(selected) + } } /// Build the map of parquet field id to Parquet column index in the schema. @@ -1949,6 +2003,194 @@ message schema { Arc::new(SchemaDescriptor::new(Arc::new(schema))) } + /// Verifies that file splits respect byte ranges and only read specific row groups. + #[tokio::test] + async fn test_file_splits_respect_byte_ranges() { + use arrow_array::Int32Array; + use parquet::file::reader::{FileReader, SerializedFileReader}; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_path = format!("{}/multi_row_group.parquet", &table_location); + + // Force each batch into its own row group for testing byte range filtering. + let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new(Int32Array::from( + (0..100).collect::>(), + ))]) + .unwrap(); + let batch2 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new(Int32Array::from( + (100..200).collect::>(), + ))]) + .unwrap(); + let batch3 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new(Int32Array::from( + (200..300).collect::>(), + ))]) + .unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .set_max_row_group_size(100) + .build(); + + let file = File::create(&file_path).unwrap(); + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + writer.write(&batch1).expect("Writing batch 1"); + writer.write(&batch2).expect("Writing batch 2"); + writer.write(&batch3).expect("Writing batch 3"); + writer.close().unwrap(); + + // Read the file metadata to get row group byte positions + let file = File::open(&file_path).unwrap(); + let reader = SerializedFileReader::new(file).unwrap(); + let metadata = reader.metadata(); + + println!("File has {} row groups", metadata.num_row_groups()); + assert_eq!(metadata.num_row_groups(), 3, "Expected 3 row groups"); + + // Get byte positions for each row group + let row_group_0 = metadata.row_group(0); + let row_group_1 = metadata.row_group(1); + let row_group_2 = metadata.row_group(2); + + let rg0_start = 4u64; // Parquet files start with 4-byte magic "PAR1" + let rg1_start = rg0_start + row_group_0.compressed_size() as u64; + let rg2_start = rg1_start + row_group_1.compressed_size() as u64; + let file_end = rg2_start + row_group_2.compressed_size() as u64; + + println!( + "Row group 0: {} rows, starts at byte {}, {} bytes compressed", + row_group_0.num_rows(), + rg0_start, + row_group_0.compressed_size() + ); + println!( + "Row group 1: {} rows, starts at byte {}, {} bytes compressed", + row_group_1.num_rows(), + rg1_start, + row_group_1.compressed_size() + ); + println!( + "Row group 2: {} rows, starts at byte {}, {} bytes compressed", + row_group_2.num_rows(), + rg2_start, + row_group_2.compressed_size() + ); + + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + let reader = ArrowReaderBuilder::new(file_io).build(); + + // Task 1: read only the first row group + let task1 = FileScanTask { + start: rg0_start, + length: row_group_0.compressed_size() as u64, + record_count: Some(100), + data_file_path: file_path.clone(), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1], + predicate: None, + deletes: vec![], + }; + + // Task 2: read the second and third row groups + let task2 = FileScanTask { + start: rg1_start, + length: file_end - rg1_start, + record_count: Some(200), + data_file_path: file_path.clone(), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1], + predicate: None, + deletes: vec![], + }; + + let tasks1 = Box::pin(futures::stream::iter(vec![Ok(task1)])) as FileScanTaskStream; + let result1 = reader + .clone() + .read(tasks1) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + let total_rows_task1: usize = result1.iter().map(|b| b.num_rows()).sum(); + println!( + "Task 1 (bytes {}-{}) returned {} rows", + rg0_start, + rg0_start + row_group_0.compressed_size() as u64, + total_rows_task1 + ); + + let tasks2 = Box::pin(futures::stream::iter(vec![Ok(task2)])) as FileScanTaskStream; + let result2 = reader + .read(tasks2) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + let total_rows_task2: usize = result2.iter().map(|b| b.num_rows()).sum(); + println!( + "Task 2 (bytes {}-{}) returned {} rows", + rg1_start, file_end, total_rows_task2 + ); + + assert_eq!( + total_rows_task1, 100, + "Task 1 should read only the first row group (100 rows), but got {} rows", + total_rows_task1 + ); + + assert_eq!( + total_rows_task2, 200, + "Task 2 should read only the second+third row groups (200 rows), but got {} rows", + total_rows_task2 + ); + + // Verify the actual data values are correct (not just the row count) + if total_rows_task1 > 0 { + let first_batch = &result1[0]; + let id_col = first_batch + .column(0) + .as_primitive::(); + let first_val = id_col.value(0); + let last_val = id_col.value(id_col.len() - 1); + println!("Task 1 data range: {} to {}", first_val, last_val); + + assert_eq!(first_val, 0, "Task 1 should start with id=0"); + assert_eq!(last_val, 99, "Task 1 should end with id=99"); + } + + if total_rows_task2 > 0 { + let first_batch = &result2[0]; + let id_col = first_batch + .column(0) + .as_primitive::(); + let first_val = id_col.value(0); + println!("Task 2 first value: {}", first_val); + + assert_eq!(first_val, 100, "Task 2 should start with id=100, not id=0"); + } + } + /// Test schema evolution: reading old Parquet file (with only column 'a') /// using a newer table schema (with columns 'a' and 'b'). /// This tests that: From 0a06c3e241a297998ebdcdf7c20f22d03c4e23f2 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 28 Oct 2025 02:28:38 -0700 Subject: [PATCH 036/891] refactor(arrow,datafusion): Reuse PartitionValueCalculator in RecordBatchPartitionSplitter (#1781) ## Which issue does this PR close? - Closes #1786 - Covered some of changes from the previous draft: #1769 ## What changes are included in this PR? - Move PartitionValueCalculator to core/arrow so it can be reused by RecordBatchPartitionSplitter - Allow skipping partition value calculation in partition splitter for projected batches - Return rather than pairs in RecordBatchPartitionSplitter::split ## Are these changes tested? Added uts --- crates/iceberg/src/arrow/mod.rs | 7 +- .../src/arrow/partition_value_calculator.rs | 254 ++++++++++++ .../arrow/record_batch_partition_splitter.rs | 377 +++++++++++++----- .../datafusion/src/physical_plan/project.rs | 190 ++------- 4 files changed, 580 insertions(+), 248 deletions(-) create mode 100644 crates/iceberg/src/arrow/partition_value_calculator.rs diff --git a/crates/iceberg/src/arrow/mod.rs b/crates/iceberg/src/arrow/mod.rs index 28116a4b5e..c091c45177 100644 --- a/crates/iceberg/src/arrow/mod.rs +++ b/crates/iceberg/src/arrow/mod.rs @@ -35,4 +35,9 @@ mod value; pub use reader::*; pub use value::*; -pub(crate) mod record_batch_partition_splitter; +/// Partition value calculator for computing partition values +pub mod partition_value_calculator; +pub use partition_value_calculator::*; +/// Record batch partition splitter for partitioned tables +pub mod record_batch_partition_splitter; +pub use record_batch_partition_splitter::*; diff --git a/crates/iceberg/src/arrow/partition_value_calculator.rs b/crates/iceberg/src/arrow/partition_value_calculator.rs new file mode 100644 index 0000000000..1409503451 --- /dev/null +++ b/crates/iceberg/src/arrow/partition_value_calculator.rs @@ -0,0 +1,254 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Partition value calculation for Iceberg tables. +//! +//! This module provides utilities for calculating partition values from record batches +//! based on a partition specification. + +use std::sync::Arc; + +use arrow_array::{ArrayRef, RecordBatch, StructArray}; +use arrow_schema::DataType; + +use super::record_batch_projector::RecordBatchProjector; +use super::type_to_arrow_type; +use crate::spec::{PartitionSpec, Schema, StructType, Type}; +use crate::transform::{BoxedTransformFunction, create_transform_function}; +use crate::{Error, ErrorKind, Result}; + +/// Calculator for partition values in Iceberg tables. +/// +/// This struct handles the projection of source columns and application of +/// partition transforms to compute partition values for a given record batch. +#[derive(Debug)] +pub struct PartitionValueCalculator { + projector: RecordBatchProjector, + transform_functions: Vec, + partition_type: StructType, + partition_arrow_type: DataType, +} + +impl PartitionValueCalculator { + /// Create a new PartitionValueCalculator. + /// + /// # Arguments + /// + /// * `partition_spec` - The partition specification + /// * `table_schema` - The Iceberg table schema + /// + /// # Returns + /// + /// Returns a new `PartitionValueCalculator` instance or an error if initialization fails. + /// + /// # Errors + /// + /// Returns an error if: + /// - The partition spec is unpartitioned + /// - Transform function creation fails + /// - Projector initialization fails + pub fn try_new(partition_spec: &PartitionSpec, table_schema: &Schema) -> Result { + if partition_spec.is_unpartitioned() { + return Err(Error::new( + ErrorKind::DataInvalid, + "Cannot create partition calculator for unpartitioned table", + )); + } + + // Create transform functions for each partition field + let transform_functions: Vec = partition_spec + .fields() + .iter() + .map(|pf| create_transform_function(&pf.transform)) + .collect::>>()?; + + // Extract source field IDs for projection + let source_field_ids: Vec = partition_spec + .fields() + .iter() + .map(|pf| pf.source_id) + .collect(); + + // Create projector for extracting source columns + let projector = RecordBatchProjector::from_iceberg_schema( + Arc::new(table_schema.clone()), + &source_field_ids, + )?; + + // Get partition type information + let partition_type = partition_spec.partition_type(table_schema)?; + let partition_arrow_type = type_to_arrow_type(&Type::Struct(partition_type.clone()))?; + + Ok(Self { + projector, + transform_functions, + partition_type, + partition_arrow_type, + }) + } + + /// Get the partition type as an Iceberg StructType. + pub fn partition_type(&self) -> &StructType { + &self.partition_type + } + + /// Get the partition type as an Arrow DataType. + pub fn partition_arrow_type(&self) -> &DataType { + &self.partition_arrow_type + } + + /// Calculate partition values for a record batch. + /// + /// This method: + /// 1. Projects the source columns from the batch + /// 2. Applies partition transforms to each source column + /// 3. Constructs a StructArray containing the partition values + /// + /// # Arguments + /// + /// * `batch` - The record batch to calculate partition values for + /// + /// # Returns + /// + /// Returns an ArrayRef containing a StructArray of partition values, or an error if calculation fails. + /// + /// # Errors + /// + /// Returns an error if: + /// - Column projection fails + /// - Transform application fails + /// - StructArray construction fails + pub fn calculate(&self, batch: &RecordBatch) -> Result { + // Project source columns from the batch + let source_columns = self.projector.project_column(batch.columns())?; + + // Get expected struct fields for the result + let expected_struct_fields = match &self.partition_arrow_type { + DataType::Struct(fields) => fields.clone(), + _ => { + return Err(Error::new( + ErrorKind::DataInvalid, + "Expected partition type must be a struct", + )); + } + }; + + // Apply transforms to each source column + let mut partition_values = Vec::with_capacity(self.transform_functions.len()); + for (source_column, transform_fn) in source_columns.iter().zip(&self.transform_functions) { + let partition_value = transform_fn.transform(source_column.clone())?; + partition_values.push(partition_value); + } + + // Construct the StructArray + let struct_array = StructArray::try_new(expected_struct_fields, partition_values, None) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to create partition struct array: {}", e), + ) + })?; + + Ok(Arc::new(struct_array)) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_schema::{Field, Schema as ArrowSchema}; + + use super::*; + use crate::spec::{NestedField, PartitionSpecBuilder, PrimitiveType, Transform}; + + #[test] + fn test_partition_calculator_identity_transform() { + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = PartitionSpecBuilder::new(Arc::new(table_schema.clone())) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let calculator = PartitionValueCalculator::try_new(&partition_spec, &table_schema).unwrap(); + + // Verify partition type + assert_eq!(calculator.partition_type().fields().len(), 1); + assert_eq!(calculator.partition_type().fields()[0].name, "id_partition"); + + // Create test batch + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let batch = RecordBatch::try_new(arrow_schema, vec![ + Arc::new(Int32Array::from(vec![10, 20, 30])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ]) + .unwrap(); + + // Calculate partition values + let result = calculator.calculate(&batch).unwrap(); + let struct_array = result.as_any().downcast_ref::().unwrap(); + + let id_partition = struct_array + .column_by_name("id_partition") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(id_partition.value(0), 10); + assert_eq!(id_partition.value(1), 20); + assert_eq!(id_partition.value(2), 30); + } + + #[test] + fn test_partition_calculator_unpartitioned_error() { + let table_schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = PartitionSpecBuilder::new(Arc::new(table_schema.clone())) + .build() + .unwrap(); + + let result = PartitionValueCalculator::try_new(&partition_spec, &table_schema); + assert!(result.is_err()); + assert!( + result + .unwrap_err() + .to_string() + .contains("unpartitioned table") + ); + } +} diff --git a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs index 704a4e9c15..66371fac16 100644 --- a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs +++ b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs @@ -19,137 +19,169 @@ use std::collections::HashMap; use std::sync::Arc; use arrow_array::{ArrayRef, BooleanArray, RecordBatch, StructArray}; -use arrow_schema::{DataType, SchemaRef as ArrowSchemaRef}; use arrow_select::filter::filter_record_batch; -use itertools::Itertools; -use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use super::arrow_struct_to_literal; -use super::record_batch_projector::RecordBatchProjector; -use crate::arrow::type_to_arrow_type; -use crate::spec::{Literal, PartitionSpecRef, SchemaRef, Struct, StructType, Type}; -use crate::transform::{BoxedTransformFunction, create_transform_function}; +use super::partition_value_calculator::PartitionValueCalculator; +use crate::spec::{Literal, PartitionKey, PartitionSpecRef, SchemaRef, StructType}; use crate::{Error, ErrorKind, Result}; +/// Column name for the projected partition values struct +pub const PROJECTED_PARTITION_VALUE_COLUMN: &str = "_partition"; + /// The splitter used to split the record batch into multiple record batches by the partition spec. /// 1. It will project and transform the input record batch based on the partition spec, get the partitioned record batch. /// 2. Split the input record batch into multiple record batches based on the partitioned record batch. +/// +/// # Partition Value Modes +/// +/// The splitter supports two modes for obtaining partition values: +/// - **Computed mode** (`calculator` is `Some`): Computes partition values from source columns using transforms +/// - **Pre-computed mode** (`calculator` is `None`): Expects a `_partition` column in the input batch // # TODO // Remove this after partition writer supported. #[allow(dead_code)] pub struct RecordBatchPartitionSplitter { schema: SchemaRef, partition_spec: PartitionSpecRef, - projector: RecordBatchProjector, - transform_functions: Vec, - + calculator: Option, partition_type: StructType, - partition_arrow_type: DataType, } // # TODO // Remove this after partition writer supported. #[allow(dead_code)] impl RecordBatchPartitionSplitter { + /// Create a new RecordBatchPartitionSplitter. + /// + /// # Arguments + /// + /// * `iceberg_schema` - The Iceberg schema reference + /// * `partition_spec` - The partition specification reference + /// * `calculator` - Optional calculator for computing partition values from source columns. + /// - `Some(calculator)`: Compute partition values from source columns using transforms + /// - `None`: Expect a pre-computed `_partition` column in the input batch + /// + /// # Returns + /// + /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. pub fn new( - input_schema: ArrowSchemaRef, iceberg_schema: SchemaRef, partition_spec: PartitionSpecRef, + calculator: Option, ) -> Result { - let projector = RecordBatchProjector::new( - input_schema, - &partition_spec - .fields() - .iter() - .map(|field| field.source_id) - .collect::>(), - // The source columns, selected by ids, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct. - // ref: https://iceberg.apache.org/spec/#partitioning - |field| { - if !field.data_type().is_primitive() { - return Ok(None); - } - field - .metadata() - .get(PARQUET_FIELD_ID_META_KEY) - .map(|s| { - s.parse::() - .map_err(|e| Error::new(ErrorKind::Unexpected, e.to_string())) - }) - .transpose() - }, - |_| true, - )?; - let transform_functions = partition_spec - .fields() - .iter() - .map(|field| create_transform_function(&field.transform)) - .collect::>>()?; - let partition_type = partition_spec.partition_type(&iceberg_schema)?; - let partition_arrow_type = type_to_arrow_type(&Type::Struct(partition_type.clone()))?; Ok(Self { schema: iceberg_schema, partition_spec, - projector, - transform_functions, + calculator, partition_type, - partition_arrow_type, }) } - fn partition_columns_to_struct(&self, partition_columns: Vec) -> Result> { - let arrow_struct_array = { - let partition_arrow_fields = { - let DataType::Struct(fields) = &self.partition_arrow_type else { - return Err(Error::new( - ErrorKind::DataInvalid, - "The partition arrow type is not a struct type", - )); - }; - fields.clone() - }; - Arc::new(StructArray::try_new( - partition_arrow_fields, - partition_columns, - None, - )?) as ArrayRef - }; - let struct_array = { - let struct_array = arrow_struct_to_literal(&arrow_struct_array, &self.partition_type)?; + /// Create a new RecordBatchPartitionSplitter with computed partition values. + /// + /// This is a convenience method that creates a calculator and initializes the splitter + /// to compute partition values from source columns. + /// + /// # Arguments + /// + /// * `iceberg_schema` - The Iceberg schema reference + /// * `partition_spec` - The partition specification reference + /// + /// # Returns + /// + /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. + pub fn new_with_computed_values( + iceberg_schema: SchemaRef, + partition_spec: PartitionSpecRef, + ) -> Result { + let calculator = PartitionValueCalculator::try_new(&partition_spec, &iceberg_schema)?; + Self::new(iceberg_schema, partition_spec, Some(calculator)) + } + + /// Create a new RecordBatchPartitionSplitter expecting pre-computed partition values. + /// + /// This is a convenience method that initializes the splitter to expect a `_partition` + /// column in the input batches. + /// + /// # Arguments + /// + /// * `iceberg_schema` - The Iceberg schema reference + /// * `partition_spec` - The partition specification reference + /// + /// # Returns + /// + /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. + pub fn new_with_precomputed_values( + iceberg_schema: SchemaRef, + partition_spec: PartitionSpecRef, + ) -> Result { + Self::new(iceberg_schema, partition_spec, None) + } + + /// Split the record batch into multiple record batches based on the partition spec. + pub fn split(&self, batch: &RecordBatch) -> Result> { + let partition_structs = if let Some(calculator) = &self.calculator { + // Compute partition values from source columns using calculator + let partition_array = calculator.calculate(batch)?; + let struct_array = arrow_struct_to_literal(&partition_array, &self.partition_type)?; + struct_array .into_iter() .map(|s| { - if let Some(s) = s { - if let Literal::Struct(s) = s { - Ok(s) - } else { - Err(Error::new( - ErrorKind::DataInvalid, - "The struct is not a struct literal", - )) - } + if let Some(Literal::Struct(s)) = s { + Ok(s) } else { - Err(Error::new(ErrorKind::DataInvalid, "The struct is null")) + Err(Error::new( + ErrorKind::DataInvalid, + "Partition value is not a struct literal or is null", + )) } }) .collect::>>()? - }; + } else { + // Extract partition values from pre-computed partition column + let partition_column = batch + .column_by_name(PROJECTED_PARTITION_VALUE_COLUMN) + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Partition column '{}' not found in batch", + PROJECTED_PARTITION_VALUE_COLUMN + ), + ) + })?; - Ok(struct_array) - } + let partition_struct_array = partition_column + .as_any() + .downcast_ref::() + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + "Partition column is not a StructArray", + ) + })?; - /// Split the record batch into multiple record batches based on the partition spec. - pub fn split(&self, batch: &RecordBatch) -> Result> { - let source_columns = self.projector.project_column(batch.columns())?; - let partition_columns = source_columns - .into_iter() - .zip_eq(self.transform_functions.iter()) - .map(|(source_column, transform_function)| transform_function.transform(source_column)) - .collect::>>()?; + let arrow_struct_array = Arc::new(partition_struct_array.clone()) as ArrayRef; + let struct_array = arrow_struct_to_literal(&arrow_struct_array, &self.partition_type)?; - let partition_structs = self.partition_columns_to_struct(partition_columns)?; + struct_array + .into_iter() + .map(|s| { + if let Some(Literal::Struct(s)) = s { + Ok(s) + } else { + Err(Error::new( + ErrorKind::DataInvalid, + "Partition value is not a struct literal or is null", + )) + } + }) + .collect::>>()? + }; // Group the batch by row value. let mut group_ids = HashMap::new(); @@ -172,8 +204,15 @@ impl RecordBatchPartitionSplitter { filter.into() }; + // Create PartitionKey from the partition struct + let partition_key = PartitionKey::new( + self.partition_spec.as_ref().clone(), + self.schema.clone(), + row, + ); + // filter the RecordBatch - partition_batches.push((row, filter_record_batch(batch, &filter_array)?)); + partition_batches.push((partition_key, filter_record_batch(batch, &filter_array)?)); } Ok(partition_batches) @@ -185,11 +224,13 @@ mod tests { use std::sync::Arc; use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_schema::DataType; + use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use super::*; use crate::arrow::schema_to_arrow_schema; use crate::spec::{ - NestedField, PartitionSpecBuilder, PrimitiveLiteral, Schema, Transform, + NestedField, PartitionSpecBuilder, PrimitiveLiteral, Schema, Struct, Transform, Type, UnboundPartitionField, }; @@ -227,14 +268,14 @@ mod tests { .build() .unwrap(), ); - let input_schema = Arc::new(schema_to_arrow_schema(&schema).unwrap()); let partition_splitter = - RecordBatchPartitionSplitter::new(input_schema.clone(), schema.clone(), partition_spec) + RecordBatchPartitionSplitter::new_with_computed_values(schema.clone(), partition_spec) .expect("Failed to create splitter"); + let arrow_schema = Arc::new(schema_to_arrow_schema(&schema).unwrap()); let id_array = Int32Array::from(vec![1, 2, 1, 3, 2, 3, 1]); let data_array = StringArray::from(vec!["a", "b", "c", "d", "e", "f", "g"]); - let batch = RecordBatch::try_new(input_schema.clone(), vec![ + let batch = RecordBatch::try_new(arrow_schema.clone(), vec![ Arc::new(id_array), Arc::new(data_array), ]) @@ -243,8 +284,8 @@ mod tests { let mut partitioned_batches = partition_splitter .split(&batch) .expect("Failed to split RecordBatch"); - partitioned_batches.sort_by_key(|(row, _)| { - if let PrimitiveLiteral::Int(i) = row.fields()[0] + partitioned_batches.sort_by_key(|(partition_key, _)| { + if let PrimitiveLiteral::Int(i) = partition_key.data().fields()[0] .as_ref() .unwrap() .as_primitive_literal() @@ -260,7 +301,7 @@ mod tests { // check the first partition let expected_id_array = Int32Array::from(vec![1, 1, 1]); let expected_data_array = StringArray::from(vec!["a", "c", "g"]); - let expected_batch = RecordBatch::try_new(input_schema.clone(), vec![ + let expected_batch = RecordBatch::try_new(arrow_schema.clone(), vec![ Arc::new(expected_id_array), Arc::new(expected_data_array), ]) @@ -271,7 +312,7 @@ mod tests { // check the second partition let expected_id_array = Int32Array::from(vec![2, 2]); let expected_data_array = StringArray::from(vec!["b", "e"]); - let expected_batch = RecordBatch::try_new(input_schema.clone(), vec![ + let expected_batch = RecordBatch::try_new(arrow_schema.clone(), vec![ Arc::new(expected_id_array), Arc::new(expected_data_array), ]) @@ -282,7 +323,7 @@ mod tests { // check the third partition let expected_id_array = Int32Array::from(vec![3, 3]); let expected_data_array = StringArray::from(vec!["d", "f"]); - let expected_batch = RecordBatch::try_new(input_schema.clone(), vec![ + let expected_batch = RecordBatch::try_new(arrow_schema.clone(), vec![ Arc::new(expected_id_array), Arc::new(expected_data_array), ]) @@ -292,7 +333,7 @@ mod tests { let partition_values = partitioned_batches .iter() - .map(|(row, _)| row.clone()) + .map(|(partition_key, _)| partition_key.data().clone()) .collect::>(); // check partition value is struct(1), struct(2), struct(3) assert_eq!(partition_values, vec![ @@ -301,4 +342,144 @@ mod tests { Struct::from_iter(vec![Some(Literal::int(3))]), ]); } + + #[test] + fn test_record_batch_partition_split_with_partition_column() { + use arrow_array::StructArray; + use arrow_schema::{Field, Schema as ArrowSchema}; + + let schema = Arc::new( + Schema::builder() + .with_fields(vec![ + NestedField::required( + 1, + "id", + Type::Primitive(crate::spec::PrimitiveType::Int), + ) + .into(), + NestedField::required( + 2, + "name", + Type::Primitive(crate::spec::PrimitiveType::String), + ) + .into(), + ]) + .build() + .unwrap(), + ); + let partition_spec = Arc::new( + PartitionSpecBuilder::new(schema.clone()) + .with_spec_id(1) + .add_unbound_field(UnboundPartitionField { + source_id: 1, + field_id: None, + name: "id_bucket".to_string(), + transform: Transform::Identity, + }) + .unwrap() + .build() + .unwrap(), + ); + + // Create input schema with _partition column + // Note: partition field IDs start from 1000 by default + let partition_field = Field::new("id_bucket", DataType::Int32, false).with_metadata( + HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "1000".to_string())]), + ); + let partition_struct_field = Field::new( + PROJECTED_PARTITION_VALUE_COLUMN, + DataType::Struct(vec![partition_field.clone()].into()), + false, + ); + + let input_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + partition_struct_field, + ])); + + // Create splitter expecting pre-computed partition column + let partition_splitter = RecordBatchPartitionSplitter::new_with_precomputed_values( + schema.clone(), + partition_spec, + ) + .expect("Failed to create splitter"); + + // Create test data with pre-computed partition column + let id_array = Int32Array::from(vec![1, 2, 1, 3, 2, 3, 1]); + let data_array = StringArray::from(vec!["a", "b", "c", "d", "e", "f", "g"]); + + // Create partition column (same values as id for Identity transform) + let partition_values = Int32Array::from(vec![1, 2, 1, 3, 2, 3, 1]); + let partition_struct = StructArray::from(vec![( + Arc::new(partition_field), + Arc::new(partition_values) as ArrayRef, + )]); + + let batch = RecordBatch::try_new(input_schema.clone(), vec![ + Arc::new(id_array), + Arc::new(data_array), + Arc::new(partition_struct), + ]) + .expect("Failed to create RecordBatch"); + + // Split using the pre-computed partition column + let mut partitioned_batches = partition_splitter + .split(&batch) + .expect("Failed to split RecordBatch"); + + partitioned_batches.sort_by_key(|(partition_key, _)| { + if let PrimitiveLiteral::Int(i) = partition_key.data().fields()[0] + .as_ref() + .unwrap() + .as_primitive_literal() + .unwrap() + { + i + } else { + panic!("The partition value is not a int"); + } + }); + + assert_eq!(partitioned_batches.len(), 3); + + // Helper to extract id and name values from a batch + let extract_values = |batch: &RecordBatch| -> (Vec, Vec) { + let id_col = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let name_col = batch + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + ( + id_col.values().to_vec(), + name_col.iter().map(|s| s.unwrap().to_string()).collect(), + ) + }; + + // Verify partition 1: id=1, names=["a", "c", "g"] + let (key, batch) = &partitioned_batches[0]; + assert_eq!(key.data(), &Struct::from_iter(vec![Some(Literal::int(1))])); + let (ids, names) = extract_values(batch); + assert_eq!(ids, vec![1, 1, 1]); + assert_eq!(names, vec!["a", "c", "g"]); + + // Verify partition 2: id=2, names=["b", "e"] + let (key, batch) = &partitioned_batches[1]; + assert_eq!(key.data(), &Struct::from_iter(vec![Some(Literal::int(2))])); + let (ids, names) = extract_values(batch); + assert_eq!(ids, vec![2, 2]); + assert_eq!(names, vec!["b", "e"]); + + // Verify partition 3: id=3, names=["d", "f"] + let (key, batch) = &partitioned_batches[2]; + assert_eq!(key.data(), &Struct::from_iter(vec![Some(Literal::int(3))])); + let (ids, names) = extract_values(batch); + assert_eq!(ids, vec![3, 3]); + assert_eq!(names, vec!["d", "f"]); + } } diff --git a/crates/integrations/datafusion/src/physical_plan/project.rs b/crates/integrations/datafusion/src/physical_plan/project.rs index 4bfe8192b0..17492176a4 100644 --- a/crates/integrations/datafusion/src/physical_plan/project.rs +++ b/crates/integrations/datafusion/src/physical_plan/project.rs @@ -19,24 +19,19 @@ use std::sync::Arc; -use datafusion::arrow::array::{ArrayRef, RecordBatch, StructArray}; +use datafusion::arrow::array::RecordBatch; use datafusion::arrow::datatypes::{DataType, Schema as ArrowSchema}; use datafusion::common::Result as DFResult; -use datafusion::error::DataFusionError; use datafusion::physical_expr::PhysicalExpr; use datafusion::physical_expr::expressions::Column; use datafusion::physical_plan::projection::ProjectionExec; use datafusion::physical_plan::{ColumnarValue, ExecutionPlan}; -use iceberg::arrow::record_batch_projector::RecordBatchProjector; -use iceberg::spec::{PartitionSpec, Schema}; +use iceberg::arrow::{PROJECTED_PARTITION_VALUE_COLUMN, PartitionValueCalculator}; +use iceberg::spec::PartitionSpec; use iceberg::table::Table; -use iceberg::transform::BoxedTransformFunction; use crate::to_datafusion_error; -/// Column name for the combined partition values struct -const PARTITION_VALUES_COLUMN: &str = "_partition"; - /// Extends an ExecutionPlan with partition value calculations for Iceberg tables. /// /// This function takes an input ExecutionPlan and extends it with an additional column @@ -65,12 +60,9 @@ pub fn project_with_partition( let input_schema = input.schema(); // TODO: Validate that input_schema matches the Iceberg table schema. // See: https://github.com/apache/iceberg-rust/issues/1752 - let partition_type = build_partition_type(partition_spec, table_schema.as_ref())?; - let calculator = PartitionValueCalculator::new( - partition_spec.as_ref().clone(), - table_schema.as_ref().clone(), - partition_type, - )?; + let calculator = + PartitionValueCalculator::try_new(partition_spec.as_ref(), table_schema.as_ref()) + .map_err(to_datafusion_error)?; let mut projection_exprs: Vec<(Arc, String)> = Vec::with_capacity(input_schema.fields().len() + 1); @@ -80,8 +72,8 @@ pub fn project_with_partition( projection_exprs.push((column_expr, field.name().clone())); } - let partition_expr = Arc::new(PartitionExpr::new(calculator)); - projection_exprs.push((partition_expr, PARTITION_VALUES_COLUMN.to_string())); + let partition_expr = Arc::new(PartitionExpr::new(calculator, partition_spec.clone())); + projection_exprs.push((partition_expr, PROJECTED_PARTITION_VALUE_COLUMN.to_string())); let projection = ProjectionExec::try_new(projection_exprs, input)?; Ok(Arc::new(projection)) @@ -91,21 +83,24 @@ pub fn project_with_partition( #[derive(Debug, Clone)] struct PartitionExpr { calculator: Arc, + partition_spec: Arc, } impl PartitionExpr { - fn new(calculator: PartitionValueCalculator) -> Self { + fn new(calculator: PartitionValueCalculator, partition_spec: Arc) -> Self { Self { calculator: Arc::new(calculator), + partition_spec, } } } // Manual PartialEq/Eq implementations for pointer-based equality -// (two PartitionExpr are equal if they share the same calculator instance) +// (two PartitionExpr are equal if they share the same calculator and partition_spec instances) impl PartialEq for PartitionExpr { fn eq(&self, other: &Self) -> bool { Arc::ptr_eq(&self.calculator, &other.calculator) + && Arc::ptr_eq(&self.partition_spec, &other.partition_spec) } } @@ -117,7 +112,7 @@ impl PhysicalExpr for PartitionExpr { } fn data_type(&self, _input_schema: &ArrowSchema) -> DFResult { - Ok(self.calculator.partition_type.clone()) + Ok(self.calculator.partition_arrow_type().clone()) } fn nullable(&self, _input_schema: &ArrowSchema) -> DFResult { @@ -125,7 +120,10 @@ impl PhysicalExpr for PartitionExpr { } fn evaluate(&self, batch: &RecordBatch) -> DFResult { - let array = self.calculator.calculate(batch)?; + let array = self + .calculator + .calculate(batch) + .map_err(to_datafusion_error)?; Ok(ColumnarValue::Array(array)) } @@ -142,7 +140,6 @@ impl PhysicalExpr for PartitionExpr { fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let field_names: Vec = self - .calculator .partition_spec .fields() .iter() @@ -155,7 +152,6 @@ impl PhysicalExpr for PartitionExpr { impl std::fmt::Display for PartitionExpr { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let field_names: Vec<&str> = self - .calculator .partition_spec .fields() .iter() @@ -167,110 +163,18 @@ impl std::fmt::Display for PartitionExpr { impl std::hash::Hash for PartitionExpr { fn hash(&self, state: &mut H) { - // Two PartitionExpr are equal if they share the same calculator Arc + // Two PartitionExpr are equal if they share the same calculator and partition_spec Arcs Arc::as_ptr(&self.calculator).hash(state); + Arc::as_ptr(&self.partition_spec).hash(state); } } -/// Calculator for partition values in Iceberg tables -#[derive(Debug)] -struct PartitionValueCalculator { - partition_spec: PartitionSpec, - partition_type: DataType, - projector: RecordBatchProjector, - transform_functions: Vec, -} - -impl PartitionValueCalculator { - fn new( - partition_spec: PartitionSpec, - table_schema: Schema, - partition_type: DataType, - ) -> DFResult { - if partition_spec.is_unpartitioned() { - return Err(DataFusionError::Internal( - "Cannot create partition calculator for unpartitioned table".to_string(), - )); - } - - let transform_functions: Result, _> = partition_spec - .fields() - .iter() - .map(|pf| iceberg::transform::create_transform_function(&pf.transform)) - .collect(); - - let transform_functions = transform_functions.map_err(to_datafusion_error)?; - - let source_field_ids: Vec = partition_spec - .fields() - .iter() - .map(|pf| pf.source_id) - .collect(); - - let projector = RecordBatchProjector::from_iceberg_schema( - Arc::new(table_schema.clone()), - &source_field_ids, - ) - .map_err(to_datafusion_error)?; - - Ok(Self { - partition_spec, - partition_type, - projector, - transform_functions, - }) - } - - fn calculate(&self, batch: &RecordBatch) -> DFResult { - let source_columns = self - .projector - .project_column(batch.columns()) - .map_err(to_datafusion_error)?; - - let expected_struct_fields = match &self.partition_type { - DataType::Struct(fields) => fields.clone(), - _ => { - return Err(DataFusionError::Internal( - "Expected partition type must be a struct".to_string(), - )); - } - }; - - let mut partition_values = Vec::with_capacity(self.partition_spec.fields().len()); - - for (source_column, transform_fn) in source_columns.iter().zip(&self.transform_functions) { - let partition_value = transform_fn - .transform(source_column.clone()) - .map_err(to_datafusion_error)?; - - partition_values.push(partition_value); - } - - let struct_array = StructArray::try_new(expected_struct_fields, partition_values, None) - .map_err(|e| DataFusionError::ArrowError(e, None))?; - - Ok(Arc::new(struct_array)) - } -} - -fn build_partition_type( - partition_spec: &PartitionSpec, - table_schema: &Schema, -) -> DFResult { - let partition_struct_type = partition_spec - .partition_type(table_schema) - .map_err(to_datafusion_error)?; - - iceberg::arrow::type_to_arrow_type(&iceberg::spec::Type::Struct(partition_struct_type)) - .map_err(to_datafusion_error) -} - #[cfg(test)] mod tests { - use datafusion::arrow::array::Int32Array; + use datafusion::arrow::array::{ArrayRef, Int32Array, StructArray}; use datafusion::arrow::datatypes::{Field, Fields}; use datafusion::physical_plan::empty::EmptyExec; - use iceberg::spec::{NestedField, PrimitiveType, StructType, Transform, Type}; + use iceberg::spec::{NestedField, PrimitiveType, Schema, StructType, Transform, Type}; use super::*; @@ -291,20 +195,11 @@ mod tests { .build() .unwrap(); - let _arrow_schema = Arc::new(ArrowSchema::new(vec![ - Field::new("id", DataType::Int32, false), - Field::new("name", DataType::Utf8, false), - ])); - - let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); - let calculator = PartitionValueCalculator::new( - partition_spec.clone(), - table_schema, - partition_type.clone(), - ) - .unwrap(); + let calculator = PartitionValueCalculator::try_new(&partition_spec, &table_schema).unwrap(); - assert_eq!(calculator.partition_type, partition_type); + // Verify partition type + assert_eq!(calculator.partition_type().fields().len(), 1); + assert_eq!(calculator.partition_type().fields()[0].name, "id_partition"); } #[test] @@ -318,11 +213,13 @@ mod tests { .build() .unwrap(); - let partition_spec = iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) - .add_partition_field("id", "id_partition", Transform::Identity) - .unwrap() - .build() - .unwrap(); + let partition_spec = Arc::new( + iceberg::spec::PartitionSpec::builder(Arc::new(table_schema.clone())) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(), + ); let arrow_schema = Arc::new(ArrowSchema::new(vec![ Field::new("id", DataType::Int32, false), @@ -331,9 +228,7 @@ mod tests { let input = Arc::new(EmptyExec::new(arrow_schema.clone())); - let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); - let calculator = - PartitionValueCalculator::new(partition_spec, table_schema, partition_type).unwrap(); + let calculator = PartitionValueCalculator::try_new(&partition_spec, &table_schema).unwrap(); let mut projection_exprs: Vec<(Arc, String)> = Vec::with_capacity(arrow_schema.fields().len() + 1); @@ -342,8 +237,8 @@ mod tests { projection_exprs.push((column_expr, field.name().clone())); } - let partition_expr = Arc::new(PartitionExpr::new(calculator)); - projection_exprs.push((partition_expr, PARTITION_VALUES_COLUMN.to_string())); + let partition_expr = Arc::new(PartitionExpr::new(calculator, partition_spec)); + projection_exprs.push((partition_expr, PROJECTED_PARTITION_VALUE_COLUMN.to_string())); let projection = ProjectionExec::try_new(projection_exprs, input).unwrap(); let result = Arc::new(projection); @@ -384,11 +279,10 @@ mod tests { ]) .unwrap(); - let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); - let calculator = - PartitionValueCalculator::new(partition_spec, table_schema, partition_type.clone()) - .unwrap(); - let expr = PartitionExpr::new(calculator); + let partition_spec = Arc::new(partition_spec); + let calculator = PartitionValueCalculator::try_new(&partition_spec, &table_schema).unwrap(); + let partition_type = calculator.partition_arrow_type().clone(); + let expr = PartitionExpr::new(calculator, partition_spec); assert_eq!(expr.data_type(&arrow_schema).unwrap(), partition_type); assert!(!expr.nullable(&arrow_schema).unwrap()); @@ -469,9 +363,7 @@ mod tests { ]) .unwrap(); - let partition_type = build_partition_type(&partition_spec, &table_schema).unwrap(); - let calculator = - PartitionValueCalculator::new(partition_spec, table_schema, partition_type).unwrap(); + let calculator = PartitionValueCalculator::try_new(&partition_spec, &table_schema).unwrap(); let array = calculator.calculate(&batch).unwrap(); let struct_array = array.as_any().downcast_ref::().unwrap(); From 127eeb8dec905dbbc48fb065d6c73a67b82cdb88 Mon Sep 17 00:00:00 2001 From: Jannik Steinmann Date: Tue, 28 Oct 2025 16:50:10 +0100 Subject: [PATCH 037/891] feat: Update Datafusion to v49 (#1704) ## Which issue does this PR close? Closes #1702 ## What changes are included in this PR? - a dependency upgrade from Datafusion 48 to 49 - a dependency specificity change from an exact pinned patch version to a broader major version (IMO this is better for broader compatibility) - fixes to breaking APIs ## Are these changes tested? --------- Signed-off-by: Xuanwo Co-authored-by: Xuanwo --- Cargo.lock | 185 +++++++++++------- Cargo.toml | 26 +-- bindings/python/Cargo.lock | 175 +++++++++++------ bindings/python/Cargo.toml | 4 +- bindings/python/pyproject.toml | 2 +- .../datafusion/src/physical_plan/commit.rs | 5 +- .../datafusion/src/physical_plan/write.rs | 5 +- .../src/table/table_provider_factory.rs | 2 +- 8 files changed, 250 insertions(+), 154 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 4895381610..f4085e59e3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -341,6 +341,7 @@ dependencies = [ "arrow-schema", "flatbuffers", "lz4_flex", + "zstd", ] [[package]] @@ -1350,6 +1351,15 @@ dependencies = [ "bzip2-sys", ] +[[package]] +name = "bzip2" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bea8dcd42434048e4f7a304411d9273a411f647446c1234a65ce0554923f4cff" +dependencies = [ + "libbz2-rs-sys", +] + [[package]] name = "bzip2-sys" version = "0.1.13+1.0.8" @@ -1865,16 +1875,16 @@ dependencies = [ [[package]] name = "datafusion" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a11e19a7ccc5bb979c95c1dceef663eab39c9061b3bbf8d1937faf0f03bf41f" +checksum = "69dfeda1633bf8ec75b068d9f6c27cdc392ffcf5ff83128d5dbab65b73c1fd02" dependencies = [ "arrow", "arrow-ipc", "arrow-schema", "async-trait", "bytes", - "bzip2 0.5.2", + "bzip2 0.6.0", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1902,6 +1912,7 @@ dependencies = [ "datafusion-sql", "flate2", "futures", + "hex", "itertools 0.14.0", "log", "object_store", @@ -1920,9 +1931,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94985e67cab97b1099db2a7af11f31a45008b282aba921c1e1d35327c212ec18" +checksum = "2848fd1e85e2953116dab9cc2eb109214b0888d7bbd2230e30c07f1794f642c0" dependencies = [ "arrow", "async-trait", @@ -1946,9 +1957,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e002df133bdb7b0b9b429d89a69aa77b35caeadee4498b2ce1c7c23a99516988" +checksum = "051a1634628c2d1296d4e326823e7536640d87a118966cdaff069b68821ad53b" dependencies = [ "arrow", "async-trait", @@ -1969,9 +1980,9 @@ dependencies = [ [[package]] name = "datafusion-cli" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85591b54815b0c9d3fbf3b0425b75e9fc49ef73e6886d6c19d622986e2551b53" +checksum = "5966ac4973bf66cf5c189046b1b29de90db9ba059ce7193137dd089e277495de" dependencies = [ "arrow", "async-trait", @@ -1982,6 +1993,7 @@ dependencies = [ "dirs", "env_logger", "futures", + "log", "mimalloc", "object_store", "parking_lot", @@ -1994,17 +2006,19 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13242fc58fd753787b0a538e5ae77d356cb9d0656fa85a591a33c5f106267f6" +checksum = "765e4ad4ef7a4500e389a3f1e738791b71ff4c29fd00912c2f541d62b25da096" dependencies = [ "ahash 0.8.12", "apache-avro 0.17.0", "arrow", "arrow-ipc", "base64 0.22.1", + "chrono", "half", "hashbrown 0.14.5", + "hex", "indexmap 2.12.0", "libc", "log", @@ -2019,9 +2033,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2239f964e95c3a5d6b4a8cde07e646de8995c1396a7fd62c6e784f5341db499" +checksum = "40a2ae8393051ce25d232a6065c4558ab5a535c9637d5373bacfd464ac88ea12" dependencies = [ "futures", "log", @@ -2030,15 +2044,15 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cf792579bc8bf07d1b2f68c2d5382f8a63679cce8fbebfd4ba95742b6e08864" +checksum = "90cd841a77f378bc1a5c4a1c37345e1885a9203b008203f9f4b3a769729bf330" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2 0.5.2", + "bzip2 0.6.0", "chrono", "datafusion-common", "datafusion-common-runtime", @@ -2066,9 +2080,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-avro" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4de733d231abb0fba663ff60fd37bf7171fa8b2e46e8a99e41362001821d116e" +checksum = "3cba1696aa919da9517d29164d45f5902d6cc281f718e8d3bfe98bd52cd1142c" dependencies = [ "apache-avro 0.17.0", "arrow", @@ -2091,9 +2105,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc114f9a1415174f3e8d2719c371fc72092ef2195a7955404cfe6b2ba29a706" +checksum = "77f4a2c64939c6f0dd15b246723a699fa30d59d0133eb36a86e8ff8c6e2a8dc6" dependencies = [ "arrow", "async-trait", @@ -2116,9 +2130,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d88dd5e215c420a52362b9988ecd4cefd71081b730663d4f7d886f706111fc75" +checksum = "11387aaf931b2993ad9273c63ddca33f05aef7d02df9b70fb757429b4b71cdae" dependencies = [ "arrow", "async-trait", @@ -2141,9 +2155,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33692acdd1fbe75280d14f4676fe43f39e9cb36296df56575aa2cac9a819e4cf" +checksum = "028f430c5185120bf806347848b8d8acd9823f4038875b3820eeefa35f2bb4a2" dependencies = [ "arrow", "async-trait", @@ -2159,8 +2173,10 @@ dependencies = [ "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", + "datafusion-pruning", "datafusion-session", "futures", + "hex", "itertools 0.14.0", "log", "object_store", @@ -2172,15 +2188,15 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0e7b648387b0c1937b83cb328533c06c923799e73a9e3750b762667f32662c0" +checksum = "8ff336d1d755399753a9e4fbab001180e346fc8bfa063a97f1214b82274c00f8" [[package]] name = "datafusion-execution" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9609d83d52ff8315283c6dad3b97566e877d8f366fab4c3297742f33dcd636c7" +checksum = "042ea192757d1b2d7dcf71643e7ff33f6542c7704f00228d8b85b40003fd8e0f" dependencies = [ "arrow", "dashmap", @@ -2197,11 +2213,12 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e75230cd67f650ef0399eb00f54d4a073698f2c0262948298e5299fc7324da63" +checksum = "025222545d6d7fab71e2ae2b356526a1df67a2872222cbae7535e557a42abd2e" dependencies = [ "arrow", + "async-trait", "chrono", "datafusion-common", "datafusion-doc", @@ -2218,9 +2235,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70fafb3a045ed6c49cfca0cd090f62cf871ca6326cc3355cb0aaf1260fa760b6" +checksum = "9d5c267104849d5fa6d81cf5ba88f35ecd58727729c5eb84066c25227b644ae2" dependencies = [ "arrow", "datafusion-common", @@ -2231,9 +2248,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdf9a9cf655265861a20453b1e58357147eab59bdc90ce7f2f68f1f35104d3bb" +checksum = "c620d105aa208fcee45c588765483314eb415f5571cfd6c1bae3a59c5b4d15bb" dependencies = [ "arrow", "arrow-buffer", @@ -2260,9 +2277,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f07e49733d847be0a05235e17b884d326a2fd402c97a89fe8bcf0bfba310005" +checksum = "35f61d5198a35ed368bf3aacac74f0d0fa33de7a7cb0c57e9f68ab1346d2f952" dependencies = [ "ahash 0.8.12", "arrow", @@ -2281,9 +2298,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4512607e10d72b0b0a1dc08f42cb5bd5284cb8348b7fea49dc83409493e32b1b" +checksum = "13efdb17362be39b5024f6da0d977ffe49c0212929ec36eec550e07e2bc7812f" dependencies = [ "ahash 0.8.12", "arrow", @@ -2294,9 +2311,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ab331806e34f5545e5f03396e4d5068077395b1665795d8f88c14ec4f1e0b7a" +checksum = "9187678af567d7c9e004b72a0b6dc5b0a00ebf4901cb3511ed2db4effe092e66" dependencies = [ "arrow", "arrow-ord", @@ -2306,6 +2323,7 @@ dependencies = [ "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", "datafusion-macros", "datafusion-physical-expr-common", "itertools 0.14.0", @@ -2315,9 +2333,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4ac2c0be983a06950ef077e34e0174aa0cb9e346f3aeae459823158037ade37" +checksum = "ecf156589cc21ef59fe39c7a9a841b4a97394549643bbfa88cc44e8588cf8fe5" dependencies = [ "arrow", "async-trait", @@ -2331,9 +2349,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36f3d92731de384c90906941d36dcadf6a86d4128409a9c5cd916662baed5f53" +checksum = "edcb25e3e369f1366ec9a261456e45b5aad6ea1c0c8b4ce546587207c501ed9e" dependencies = [ "arrow", "datafusion-common", @@ -2349,9 +2367,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c679f8bf0971704ec8fd4249fcbb2eb49d6a12cc3e7a840ac047b4928d3541b5" +checksum = "8996a8e11174d0bd7c62dc2f316485affc6ae5ffd5b8a68b508137ace2310294" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -2359,9 +2377,9 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2821de7cb0362d12e75a5196b636a59ea3584ec1e1cc7dc6f5e34b9e8389d251" +checksum = "95ee8d1be549eb7316f437035f2cec7ec42aba8374096d807c4de006a3b5d78a" dependencies = [ "datafusion-expr", "quote", @@ -2370,14 +2388,15 @@ dependencies = [ [[package]] name = "datafusion-optimizer" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1594c7a97219ede334f25347ad8d57056621e7f4f35a0693c8da876e10dd6a53" +checksum = "c9fa98671458254928af854e5f6c915e66b860a8bde505baea0ff2892deab74d" dependencies = [ "arrow", "chrono", "datafusion-common", "datafusion-expr", + "datafusion-expr-common", "datafusion-physical-expr", "indexmap 2.12.0", "itertools 0.14.0", @@ -2389,9 +2408,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc6da0f2412088d23f6b01929dedd687b5aee63b19b674eb73d00c3eb3c883b7" +checksum = "3515d51531cca5f7b5a6f3ea22742b71bb36fc378b465df124ff9a2fa349b002" dependencies = [ "ahash 0.8.12", "arrow", @@ -2411,9 +2430,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcb0dbd9213078a593c3fe28783beaa625a4e6c6a6c797856ee2ba234311fb96" +checksum = "24485475d9c618a1d33b2a3dad003d946dc7a7bbf0354d125301abc0a5a79e3e" dependencies = [ "ahash 0.8.12", "arrow", @@ -2425,9 +2444,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d140854b2db3ef8ac611caad12bfb2e1e1de827077429322a6188f18fc0026a" +checksum = "b9da411a0a64702f941a12af2b979434d14ec5d36c6f49296966b2c7639cbb3a" dependencies = [ "arrow", "datafusion-common", @@ -2437,6 +2456,7 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-pruning", "itertools 0.14.0", "log", "recursive", @@ -2444,9 +2464,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b46cbdf21a01206be76d467f325273b22c559c744a012ead5018dfe79597de08" +checksum = "a6d168282bb7b54880bb3159f89b51c047db4287f5014d60c3ef4c6e1468212b" dependencies = [ "ahash 0.8.12", "arrow", @@ -2472,11 +2492,29 @@ dependencies = [ "tokio", ] +[[package]] +name = "datafusion-pruning" +version = "49.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "391a457b9d23744c53eeb89edd1027424cba100581488d89800ed841182df905" +dependencies = [ + "arrow", + "arrow-schema", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "itertools 0.14.0", + "log", +] + [[package]] name = "datafusion-session" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a72733766ddb5b41534910926e8da5836622316f6283307fd9fb7e19811a59c" +checksum = "053201c2bb729c7938f85879034df2b5a52cfaba16f1b3b66ab8505c81b2aad3" dependencies = [ "arrow", "async-trait", @@ -2498,9 +2536,9 @@ dependencies = [ [[package]] name = "datafusion-spark" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f469b1301a5a2dec8b944eacb7b1194ed8c0f8203ce460b7451bb6fa2e89060" +checksum = "126ceb2436b498ddd66c0a9f58304bce6b903f622c681c0ca9e52cf229c55ea0" dependencies = [ "arrow", "datafusion-catalog", @@ -2514,9 +2552,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5162338cdec9cc7ea13a0e6015c361acad5ec1d88d83f7c86301f789473971f" +checksum = "9082779be8ce4882189b229c0cff4393bd0808282a7194130c9f32159f185e25" dependencies = [ "arrow", "bigdecimal", @@ -2531,9 +2569,9 @@ dependencies = [ [[package]] name = "datafusion-sqllogictest" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3a05ab096d3dd4c234f4ecf3c9c08b71396ac1810d4d57d3a13049bbf8a1b30" +checksum = "2c2358f74d54fb9895b3b962f14877e414327eaaad4a95154abacc3e6bdcd38b" dependencies = [ "arrow", "async-trait", @@ -2558,9 +2596,9 @@ dependencies = [ [[package]] name = "datafusion-substrait" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af6029c08002772fb2c38a191d21a8b03af78750f42d9b896222f6b0e16c84cf" +checksum = "4189e480ae42f06ade81a6d12853c5f3f84fbe237d73fb8795c712b6f1923afb" dependencies = [ "async-recursion", "async-trait", @@ -4187,6 +4225,12 @@ dependencies = [ "lexical-util", ] +[[package]] +name = "libbz2-rs-sys" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" + [[package]] name = "libc" version = "0.2.177" @@ -4938,6 +4982,7 @@ dependencies = [ "num-bigint", "object_store", "paste", + "ring", "seq-macro", "simdutf8", "snap", @@ -7047,9 +7092,9 @@ dependencies = [ [[package]] name = "substrait" -version = "0.56.0" +version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13de2e20128f2a018dab1cfa30be83ae069219a65968c6f89df66ad124de2397" +checksum = "de6d24c270c6c672a86c183c3a8439ba46c1936f93cf7296aa692de3b0ff0228" dependencies = [ "heck", "pbjson", diff --git a/Cargo.toml b/Cargo.toml index 1c007376ad..49eeff038a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -42,14 +42,14 @@ rust-version = "1.87" anyhow = "1.0.72" apache-avro = { version = "0.20", features = ["zstandard"] } array-init = "2" -arrow-arith = { version = "55.1" } -arrow-array = { version = "55.1" } -arrow-buffer = { version = "55.1" } -arrow-cast = { version = "55.1" } -arrow-ord = { version = "55.1" } -arrow-schema = { version = "55.1" } -arrow-select = { version = "55.1" } -arrow-string = { version = "55.1" } +arrow-arith = { version = "55.2" } +arrow-array = { version = "55.2" } +arrow-buffer = { version = "55.2" } +arrow-cast = { version = "55.2" } +arrow-ord = { version = "55.2" } +arrow-schema = { version = "55.2" } +arrow-select = { version = "55.2" } +arrow-string = { version = "55.2" } as-any = "0.3.2" async-trait = "0.1.88" aws-config = "1.8.1" @@ -62,9 +62,9 @@ bytes = "1.10" chrono = "0.4.41" clap = { version = "4.5.41", features = ["derive", "cargo"] } ctor = "0.2.8" -datafusion = "48.0.1" -datafusion-cli = "48.0.1" -datafusion-sqllogictest = "48.0.1" +datafusion = "49" +datafusion-cli = "49" +datafusion-sqllogictest = "49" derive_builder = "0.20" dirs = "6" enum-ordinalize = "4.3.0" @@ -99,7 +99,7 @@ num-bigint = "0.4.6" once_cell = "1.20" opendal = "0.54.0" ordered-float = "4" -parquet = "55.1" +parquet = "55.2" pilota = "0.11.10" port_scanner = "0.1.5" pretty_assertions = "1.4" @@ -107,7 +107,7 @@ rand = "0.8.5" regex = "1.10.5" reqwest = { version = "0.12.12", default-features = false, features = ["json"] } roaring = { version = "0.11" } -rust_decimal = "1.37.1" +rust_decimal = "1.37.2" serde = { version = "1.0.219", features = ["rc"] } serde_bytes = "0.11.17" serde_derive = "1.0.219" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 1a619dca0f..70e95a37aa 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -157,6 +157,7 @@ dependencies = [ "strum_macros", "thiserror 2.0.16", "uuid", + "zstd", ] [[package]] @@ -301,6 +302,7 @@ dependencies = [ "arrow-schema", "flatbuffers", "lz4_flex", + "zstd", ] [[package]] @@ -429,7 +431,7 @@ version = "0.4.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06575e6a9673580f52661c92107baabffbf41e2141373441cbcdc47cb733003c" dependencies = [ - "bzip2", + "bzip2 0.5.2", "flate2", "futures-core", "memchr", @@ -717,6 +719,15 @@ dependencies = [ "bzip2-sys", ] +[[package]] +name = "bzip2" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bea8dcd42434048e4f7a304411d9273a411f647446c1234a65ce0554923f4cff" +dependencies = [ + "libbz2-rs-sys", +] + [[package]] name = "bzip2-sys" version = "0.1.13+1.0.8" @@ -1030,16 +1041,16 @@ dependencies = [ [[package]] name = "datafusion" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a11e19a7ccc5bb979c95c1dceef663eab39c9061b3bbf8d1937faf0f03bf41f" +checksum = "69dfeda1633bf8ec75b068d9f6c27cdc392ffcf5ff83128d5dbab65b73c1fd02" dependencies = [ "arrow", "arrow-ipc", "arrow-schema", "async-trait", "bytes", - "bzip2", + "bzip2 0.6.0", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1066,6 +1077,7 @@ dependencies = [ "datafusion-sql", "flate2", "futures", + "hex", "itertools 0.14.0", "log", "object_store", @@ -1084,9 +1096,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94985e67cab97b1099db2a7af11f31a45008b282aba921c1e1d35327c212ec18" +checksum = "2848fd1e85e2953116dab9cc2eb109214b0888d7bbd2230e30c07f1794f642c0" dependencies = [ "arrow", "async-trait", @@ -1110,9 +1122,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e002df133bdb7b0b9b429d89a69aa77b35caeadee4498b2ce1c7c23a99516988" +checksum = "051a1634628c2d1296d4e326823e7536640d87a118966cdaff069b68821ad53b" dependencies = [ "arrow", "async-trait", @@ -1133,16 +1145,18 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13242fc58fd753787b0a538e5ae77d356cb9d0656fa85a591a33c5f106267f6" +checksum = "765e4ad4ef7a4500e389a3f1e738791b71ff4c29fd00912c2f541d62b25da096" dependencies = [ "ahash 0.8.12", "arrow", "arrow-ipc", "base64", + "chrono", "half", "hashbrown 0.14.5", + "hex", "indexmap 2.11.0", "libc", "log", @@ -1157,9 +1171,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2239f964e95c3a5d6b4a8cde07e646de8995c1396a7fd62c6e784f5341db499" +checksum = "40a2ae8393051ce25d232a6065c4558ab5a535c9637d5373bacfd464ac88ea12" dependencies = [ "futures", "log", @@ -1168,15 +1182,15 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cf792579bc8bf07d1b2f68c2d5382f8a63679cce8fbebfd4ba95742b6e08864" +checksum = "90cd841a77f378bc1a5c4a1c37345e1885a9203b008203f9f4b3a769729bf330" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2", + "bzip2 0.6.0", "chrono", "datafusion-common", "datafusion-common-runtime", @@ -1204,9 +1218,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc114f9a1415174f3e8d2719c371fc72092ef2195a7955404cfe6b2ba29a706" +checksum = "77f4a2c64939c6f0dd15b246723a699fa30d59d0133eb36a86e8ff8c6e2a8dc6" dependencies = [ "arrow", "async-trait", @@ -1229,9 +1243,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d88dd5e215c420a52362b9988ecd4cefd71081b730663d4f7d886f706111fc75" +checksum = "11387aaf931b2993ad9273c63ddca33f05aef7d02df9b70fb757429b4b71cdae" dependencies = [ "arrow", "async-trait", @@ -1254,9 +1268,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33692acdd1fbe75280d14f4676fe43f39e9cb36296df56575aa2cac9a819e4cf" +checksum = "028f430c5185120bf806347848b8d8acd9823f4038875b3820eeefa35f2bb4a2" dependencies = [ "arrow", "async-trait", @@ -1272,8 +1286,10 @@ dependencies = [ "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", + "datafusion-pruning", "datafusion-session", "futures", + "hex", "itertools 0.14.0", "log", "object_store", @@ -1285,15 +1301,15 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0e7b648387b0c1937b83cb328533c06c923799e73a9e3750b762667f32662c0" +checksum = "8ff336d1d755399753a9e4fbab001180e346fc8bfa063a97f1214b82274c00f8" [[package]] name = "datafusion-execution" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9609d83d52ff8315283c6dad3b97566e877d8f366fab4c3297742f33dcd636c7" +checksum = "042ea192757d1b2d7dcf71643e7ff33f6542c7704f00228d8b85b40003fd8e0f" dependencies = [ "arrow", "dashmap", @@ -1310,11 +1326,12 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e75230cd67f650ef0399eb00f54d4a073698f2c0262948298e5299fc7324da63" +checksum = "025222545d6d7fab71e2ae2b356526a1df67a2872222cbae7535e557a42abd2e" dependencies = [ "arrow", + "async-trait", "chrono", "datafusion-common", "datafusion-doc", @@ -1331,9 +1348,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70fafb3a045ed6c49cfca0cd090f62cf871ca6326cc3355cb0aaf1260fa760b6" +checksum = "9d5c267104849d5fa6d81cf5ba88f35ecd58727729c5eb84066c25227b644ae2" dependencies = [ "arrow", "datafusion-common", @@ -1344,9 +1361,9 @@ dependencies = [ [[package]] name = "datafusion-ffi" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "257de77f836e9bad3e0274d4ffc88555ebf559ae30d4e9c674b9809104c1cc3b" +checksum = "ec21805d9df2d834e4c6ddfbf8a1bed2bd460b89b01686fe0dcd1cee06d0b60f" dependencies = [ "abi_stable", "arrow", @@ -1366,9 +1383,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdf9a9cf655265861a20453b1e58357147eab59bdc90ce7f2f68f1f35104d3bb" +checksum = "c620d105aa208fcee45c588765483314eb415f5571cfd6c1bae3a59c5b4d15bb" dependencies = [ "arrow", "arrow-buffer", @@ -1395,9 +1412,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f07e49733d847be0a05235e17b884d326a2fd402c97a89fe8bcf0bfba310005" +checksum = "35f61d5198a35ed368bf3aacac74f0d0fa33de7a7cb0c57e9f68ab1346d2f952" dependencies = [ "ahash 0.8.12", "arrow", @@ -1416,9 +1433,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4512607e10d72b0b0a1dc08f42cb5bd5284cb8348b7fea49dc83409493e32b1b" +checksum = "13efdb17362be39b5024f6da0d977ffe49c0212929ec36eec550e07e2bc7812f" dependencies = [ "ahash 0.8.12", "arrow", @@ -1429,9 +1446,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ab331806e34f5545e5f03396e4d5068077395b1665795d8f88c14ec4f1e0b7a" +checksum = "9187678af567d7c9e004b72a0b6dc5b0a00ebf4901cb3511ed2db4effe092e66" dependencies = [ "arrow", "arrow-ord", @@ -1441,6 +1458,7 @@ dependencies = [ "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", "datafusion-macros", "datafusion-physical-expr-common", "itertools 0.14.0", @@ -1450,9 +1468,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4ac2c0be983a06950ef077e34e0174aa0cb9e346f3aeae459823158037ade37" +checksum = "ecf156589cc21ef59fe39c7a9a841b4a97394549643bbfa88cc44e8588cf8fe5" dependencies = [ "arrow", "async-trait", @@ -1466,9 +1484,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36f3d92731de384c90906941d36dcadf6a86d4128409a9c5cd916662baed5f53" +checksum = "edcb25e3e369f1366ec9a261456e45b5aad6ea1c0c8b4ce546587207c501ed9e" dependencies = [ "arrow", "datafusion-common", @@ -1484,9 +1502,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c679f8bf0971704ec8fd4249fcbb2eb49d6a12cc3e7a840ac047b4928d3541b5" +checksum = "8996a8e11174d0bd7c62dc2f316485affc6ae5ffd5b8a68b508137ace2310294" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1494,9 +1512,9 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2821de7cb0362d12e75a5196b636a59ea3584ec1e1cc7dc6f5e34b9e8389d251" +checksum = "95ee8d1be549eb7316f437035f2cec7ec42aba8374096d807c4de006a3b5d78a" dependencies = [ "datafusion-expr", "quote", @@ -1505,14 +1523,15 @@ dependencies = [ [[package]] name = "datafusion-optimizer" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1594c7a97219ede334f25347ad8d57056621e7f4f35a0693c8da876e10dd6a53" +checksum = "c9fa98671458254928af854e5f6c915e66b860a8bde505baea0ff2892deab74d" dependencies = [ "arrow", "chrono", "datafusion-common", "datafusion-expr", + "datafusion-expr-common", "datafusion-physical-expr", "indexmap 2.11.0", "itertools 0.14.0", @@ -1524,9 +1543,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc6da0f2412088d23f6b01929dedd687b5aee63b19b674eb73d00c3eb3c883b7" +checksum = "3515d51531cca5f7b5a6f3ea22742b71bb36fc378b465df124ff9a2fa349b002" dependencies = [ "ahash 0.8.12", "arrow", @@ -1546,9 +1565,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcb0dbd9213078a593c3fe28783beaa625a4e6c6a6c797856ee2ba234311fb96" +checksum = "24485475d9c618a1d33b2a3dad003d946dc7a7bbf0354d125301abc0a5a79e3e" dependencies = [ "ahash 0.8.12", "arrow", @@ -1560,9 +1579,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d140854b2db3ef8ac611caad12bfb2e1e1de827077429322a6188f18fc0026a" +checksum = "b9da411a0a64702f941a12af2b979434d14ec5d36c6f49296966b2c7639cbb3a" dependencies = [ "arrow", "datafusion-common", @@ -1572,6 +1591,7 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-pruning", "itertools 0.14.0", "log", "recursive", @@ -1579,9 +1599,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b46cbdf21a01206be76d467f325273b22c559c744a012ead5018dfe79597de08" +checksum = "a6d168282bb7b54880bb3159f89b51c047db4287f5014d60c3ef4c6e1468212b" dependencies = [ "ahash 0.8.12", "arrow", @@ -1609,9 +1629,9 @@ dependencies = [ [[package]] name = "datafusion-proto" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3fc7a2744332c2ef8804274c21f9fa664b4ca5889169250a6fd6b649ee5d16c" +checksum = "1b36a0c84f4500efd90487a004b533bd81de1f2bb3f143f71b7526f33b85d2e2" dependencies = [ "arrow", "chrono", @@ -1625,20 +1645,38 @@ dependencies = [ [[package]] name = "datafusion-proto-common" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "800add86852f12e3d249867425de2224c1e9fb7adc2930460548868781fbeded" +checksum = "2ec788be522806740ad6372c0a2f7e45fb37cb37f786d9b77933add49cdd058f" dependencies = [ "arrow", "datafusion-common", "prost", ] +[[package]] +name = "datafusion-pruning" +version = "49.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "391a457b9d23744c53eeb89edd1027424cba100581488d89800ed841182df905" +dependencies = [ + "arrow", + "arrow-schema", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "itertools 0.14.0", + "log", +] + [[package]] name = "datafusion-session" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a72733766ddb5b41534910926e8da5836622316f6283307fd9fb7e19811a59c" +checksum = "053201c2bb729c7938f85879034df2b5a52cfaba16f1b3b66ab8505c81b2aad3" dependencies = [ "arrow", "async-trait", @@ -1660,9 +1698,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "48.0.1" +version = "49.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5162338cdec9cc7ea13a0e6015c361acad5ec1d88d83f7c86301f789473971f" +checksum = "9082779be8ce4882189b229c0cff4393bd0808282a7194130c9f32159f185e25" dependencies = [ "arrow", "bigdecimal", @@ -2617,6 +2655,12 @@ dependencies = [ "static_assertions", ] +[[package]] +name = "libbz2-rs-sys" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" + [[package]] name = "libc" version = "0.2.175" @@ -3048,6 +3092,7 @@ dependencies = [ "num-bigint", "object_store", "paste", + "ring", "seq-macro", "simdutf8", "snap", diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index 879a1450b7..f7f33d0729 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -31,9 +31,9 @@ license = "Apache-2.0" crate-type = ["cdylib"] [dependencies] -arrow = { version = "55.1", features = ["pyarrow", "chrono-tz"] } +arrow = { version = "55.2", features = ["pyarrow", "chrono-tz"] } iceberg = { path = "../../crates/iceberg" } pyo3 = { version = "0.24.1", features = ["extension-module", "abi3-py39"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } -datafusion-ffi = { version = "48.0.1" } +datafusion-ffi = { version = "49" } tokio = { version = "1.46.1", default-features = false } diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 41877e1952..c659d9b1a9 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -49,7 +49,7 @@ include = [ ignore = ["F403", "F405"] [tool.hatch.envs.dev] -dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1", "pydantic<2.12.0"] +dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==49.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1", "pydantic<2.12.0"] [tool.hatch.envs.dev.scripts] build = "maturin build --out dist --sdist" diff --git a/crates/integrations/datafusion/src/physical_plan/commit.rs b/crates/integrations/datafusion/src/physical_plan/commit.rs index 48373f11e2..6bba89fce5 100644 --- a/crates/integrations/datafusion/src/physical_plan/commit.rs +++ b/crates/integrations/datafusion/src/physical_plan/commit.rs @@ -86,7 +86,10 @@ impl IcebergCommitExec { let count_array = Arc::new(UInt64Array::from(vec![count])) as ArrayRef; RecordBatch::try_from_iter_with_nullable(vec![("count", count_array, false)]).map_err(|e| { - DataFusionError::ArrowError(e, Some("Failed to make count batch!".to_string())) + DataFusionError::ArrowError( + Box::new(e), + Some("Failed to make count batch!".to_string()), + ) }) } diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index dff40a3c0d..e46019c541 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -96,7 +96,10 @@ impl IcebergWriteExec { let files_array = Arc::new(StringArray::from(data_files)) as ArrayRef; RecordBatch::try_new(Self::make_result_schema(), vec![files_array]).map_err(|e| { - DataFusionError::ArrowError(e, Some("Failed to make result batch".to_string())) + DataFusionError::ArrowError( + Box::new(e), + Some("Failed to make result batch".to_string()), + ) }) } diff --git a/crates/integrations/datafusion/src/table/table_provider_factory.rs b/crates/integrations/datafusion/src/table/table_provider_factory.rs index a6d3146e54..e8e87dd318 100644 --- a/crates/integrations/datafusion/src/table/table_provider_factory.rs +++ b/crates/integrations/datafusion/src/table/table_provider_factory.rs @@ -241,7 +241,7 @@ mod tests { options: Default::default(), table_partition_cols: Default::default(), order_exprs: Default::default(), - constraints: Constraints::empty(), + constraints: Constraints::default(), column_defaults: Default::default(), if_not_exists: Default::default(), temporary: false, From 16ddd0e242f34d806f39e2f91a08a6cb9592e88e Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 28 Oct 2025 09:12:19 -0700 Subject: [PATCH 038/891] deps: unpin pydantic (#1793) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Revert of #1737 pydantic release a fix in 2.12.3 which will be automatically used by resolver (see simliar in pyiceberg, https://github.com/apache/iceberg-python/pull/2635) ## Are these changes tested? --- bindings/python/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index c659d9b1a9..a9c3c290c7 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -49,7 +49,7 @@ include = [ ignore = ["F403", "F405"] [tool.hatch.envs.dev] -dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==49.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1", "pydantic<2.12.0"] +dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==49.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] [tool.hatch.envs.dev.scripts] build = "maturin build --out dist --sdist" From b20a2554a1a95e881e92ec4312fd4f75fad6da2f Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 28 Oct 2025 13:01:57 -0400 Subject: [PATCH 039/891] feat(reader): Add Date32 support to RecordBatchTransformer create_column (#1792) ## Which issue does this PR close? - Partially address #1749. Iceberg Java has a test that does a schema change requires the `RecordBatchTransformer` to add a Date32 column, which it currently does not support. ## What changes are included in this PR? Add match arms for `Date32` type in `create_column`. ## Are these changes tested? New test that mirrors Iceberg Java's `TestSelect.readAndWriteWithBranchAfterSchemaChange` --------- Co-authored-by: Xuanwo --- .../src/arrow/record_batch_transformer.rs | 82 ++++++++++++++++++- 1 file changed, 79 insertions(+), 3 deletions(-) diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index 779f1cc625..71fe59dea5 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -19,8 +19,8 @@ use std::collections::HashMap; use std::sync::Arc; use arrow_array::{ - Array as ArrowArray, ArrayRef, BinaryArray, BooleanArray, Float32Array, Float64Array, - Int32Array, Int64Array, NullArray, RecordBatch, RecordBatchOptions, StringArray, + Array as ArrowArray, ArrayRef, BinaryArray, BooleanArray, Date32Array, Float32Array, + Float64Array, Int32Array, Int64Array, NullArray, RecordBatch, RecordBatchOptions, StringArray, }; use arrow_cast::cast; use arrow_schema::{ @@ -401,6 +401,13 @@ impl RecordBatchTransformer { let vals: Vec> = vec![None; num_rows]; Arc::new(Int32Array::from(vals)) } + (DataType::Date32, Some(PrimitiveLiteral::Int(value))) => { + Arc::new(Date32Array::from(vec![*value; num_rows])) + } + (DataType::Date32, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Date32Array::from(vals)) + } (DataType::Int64, Some(PrimitiveLiteral::Long(value))) => { Arc::new(Int64Array::from(vec![*value; num_rows])) } @@ -453,7 +460,8 @@ mod test { use std::sync::Arc; use arrow_array::{ - Float32Array, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray, + Array, Date32Array, Float32Array, Float64Array, Int32Array, Int64Array, RecordBatch, + StringArray, }; use arrow_schema::{DataType, Field, Schema as ArrowSchema}; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; @@ -509,6 +517,74 @@ mod test { assert_eq!(result, expected); } + #[test] + fn schema_evolution_adds_date_column_with_nulls() { + // Reproduces TestSelect.readAndWriteWithBranchAfterSchemaChange from iceberg-spark. + // When reading old snapshots after adding a DATE column, the transformer must + // populate the new column with NULL values since old files lack this field. + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "date_col", Type::Primitive(PrimitiveType::Date)) + .into(), + ]) + .build() + .unwrap(), + ); + let projected_iceberg_field_ids = [1, 2, 3]; + + let mut transformer = + RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + + let file_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("name", DataType::Utf8, true, "2"), + ])); + + let file_batch = RecordBatch::try_new(file_schema, vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec![ + Some("Alice"), + Some("Bob"), + Some("Charlie"), + ])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(file_batch).unwrap(); + + assert_eq!(result.num_columns(), 3); + assert_eq!(result.num_rows(), 3); + + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_column.values(), &[1, 2, 3]); + + let name_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_column.value(0), "Alice"); + assert_eq!(name_column.value(1), "Bob"); + assert_eq!(name_column.value(2), "Charlie"); + + let date_column = result + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + assert!(date_column.is_null(0)); + assert!(date_column.is_null(1)); + assert!(date_column.is_null(2)); + } + pub fn source_record_batch() -> RecordBatch { RecordBatch::try_new( arrow_schema_promotion_addition_and_renaming_required(), From 87acf3cbb77ecd5c0ed3e1b2eb4e8327223bc706 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 28 Oct 2025 10:34:01 -0700 Subject: [PATCH 040/891] feat(catalog): Implement update_table for S3TablesCatalog (#1594) ## Which issue does this PR close? - Addresses the S3Table part of #1389 ## What changes are included in this PR? - Implemented update_table for S3TablesCatalog ## Are these changes tested? added a test --------- Signed-off-by: Xuanwo Co-authored-by: Xuanwo --- Cargo.lock | 1 + crates/catalog/s3tables/Cargo.toml | 1 + crates/catalog/s3tables/src/catalog.rs | 199 ++++++++++++++++++++----- 3 files changed, 167 insertions(+), 34 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index f4085e59e3..808c222dd3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3743,6 +3743,7 @@ dependencies = [ name = "iceberg-catalog-s3tables" version = "0.7.0" dependencies = [ + "anyhow", "async-trait", "aws-config", "aws-sdk-s3tables", diff --git a/crates/catalog/s3tables/Cargo.toml b/crates/catalog/s3tables/Cargo.toml index c5df489693..66fb70fefc 100644 --- a/crates/catalog/s3tables/Cargo.toml +++ b/crates/catalog/s3tables/Cargo.toml @@ -29,6 +29,7 @@ license = { workspace = true } repository = { workspace = true } [dependencies] +anyhow = { workspace = true } async-trait = { workspace = true } aws-config = { workspace = true } aws-sdk-s3tables = { workspace = true } diff --git a/crates/catalog/s3tables/src/catalog.rs b/crates/catalog/s3tables/src/catalog.rs index 858bb92896..daa6590551 100644 --- a/crates/catalog/s3tables/src/catalog.rs +++ b/crates/catalog/s3tables/src/catalog.rs @@ -23,6 +23,7 @@ use aws_sdk_s3tables::operation::create_table::CreateTableOutput; use aws_sdk_s3tables::operation::get_namespace::GetNamespaceOutput; use aws_sdk_s3tables::operation::get_table::GetTableOutput; use aws_sdk_s3tables::operation::list_tables::ListTablesOutput; +use aws_sdk_s3tables::operation::update_table_metadata_location::UpdateTableMetadataLocationError; use aws_sdk_s3tables::types::OpenTableFormat; use iceberg::io::{FileIO, FileIOBuilder}; use iceberg::spec::{TableMetadata, TableMetadataBuilder}; @@ -189,6 +190,39 @@ impl S3TablesCatalog { file_io, }) } + + async fn load_table_with_version_token( + &self, + table_ident: &TableIdent, + ) -> Result<(Table, String)> { + let req = self + .s3tables_client + .get_table() + .table_bucket_arn(self.config.table_bucket_arn.clone()) + .namespace(table_ident.namespace().to_url_string()) + .name(table_ident.name()); + let resp: GetTableOutput = req.send().await.map_err(from_aws_sdk_error)?; + + // when a table is created, it's possible that the metadata location is not set. + let metadata_location = resp.metadata_location().ok_or_else(|| { + Error::new( + ErrorKind::Unexpected, + format!( + "Table {} does not have metadata location", + table_ident.name() + ), + ) + })?; + let metadata = TableMetadata::read_from(&self.file_io, metadata_location).await?; + + let table = Table::builder() + .identifier(table_ident.clone()) + .metadata(metadata) + .metadata_location(metadata_location) + .file_io(self.file_io.clone()) + .build()?; + Ok((table, resp.version_token)) + } } #[async_trait] @@ -477,33 +511,7 @@ impl Catalog for S3TablesCatalog { /// - Errors from the underlying database query process, converted using /// `from_aws_sdk_error`. async fn load_table(&self, table_ident: &TableIdent) -> Result
{ - let req = self - .s3tables_client - .get_table() - .table_bucket_arn(self.config.table_bucket_arn.clone()) - .namespace(table_ident.namespace().to_url_string()) - .name(table_ident.name()); - let resp: GetTableOutput = req.send().await.map_err(from_aws_sdk_error)?; - - // when a table is created, it's possible that the metadata location is not set. - let metadata_location = resp.metadata_location().ok_or_else(|| { - Error::new( - ErrorKind::Unexpected, - format!( - "Table {} does not have metadata location", - table_ident.name() - ), - ) - })?; - let metadata = TableMetadata::read_from(&self.file_io, metadata_location).await?; - - let table = Table::builder() - .identifier(table_ident.clone()) - .metadata(metadata) - .metadata_location(metadata_location) - .file_io(self.file_io.clone()) - .build()?; - Ok(table) + Ok(self.load_table_with_version_token(table_ident).await?.0) } /// Drops an existing table from the s3tables catalog. @@ -589,13 +597,50 @@ impl Catalog for S3TablesCatalog { } /// Updates an existing table within the s3tables catalog. - /// - /// This function is still in development and will always return an error. - async fn update_table(&self, _commit: TableCommit) -> Result
{ - Err(Error::new( - ErrorKind::FeatureUnsupported, - "Updating a table is not supported yet", - )) + async fn update_table(&self, commit: TableCommit) -> Result
{ + let table_ident = commit.identifier().clone(); + let table_namespace = table_ident.namespace(); + let (current_table, version_token) = + self.load_table_with_version_token(&table_ident).await?; + + let staged_table = commit.apply(current_table)?; + let staged_metadata_location = staged_table.metadata_location_result()?; + + staged_table + .metadata() + .write_to(staged_table.file_io(), staged_metadata_location) + .await?; + + let builder = self + .s3tables_client + .update_table_metadata_location() + .table_bucket_arn(&self.config.table_bucket_arn) + .namespace(table_namespace.to_url_string()) + .name(table_ident.name()) + .version_token(version_token) + .metadata_location(staged_metadata_location); + + let _ = builder.send().await.map_err(|e| { + let error = e.into_service_error(); + match error { + UpdateTableMetadataLocationError::ConflictException(_) => Error::new( + ErrorKind::CatalogCommitConflicts, + format!("Commit conflicted for table: {table_ident}"), + ) + .with_retryable(true), + UpdateTableMetadataLocationError::NotFoundException(_) => Error::new( + ErrorKind::TableNotFound, + format!("Table {table_ident} is not found"), + ), + _ => Error::new( + ErrorKind::Unexpected, + "Operation failed for hitting aws sdk error", + ), + } + .with_source(anyhow::Error::msg(format!("aws sdk error: {:?}", error))) + })?; + + Ok(staged_table) } } @@ -611,6 +656,7 @@ where T: std::fmt::Debug { #[cfg(test)] mod tests { use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; + use iceberg::transaction::{ApplyTransactionAction, Transaction}; use super::*; @@ -737,6 +783,91 @@ mod tests { catalog.drop_namespace(&namespace).await.unwrap(); } + #[tokio::test] + async fn test_s3tables_update_table() { + let catalog = match load_s3tables_catalog_from_env().await { + Ok(Some(catalog)) => catalog, + Ok(None) => return, + Err(e) => panic!("Error loading catalog: {}", e), + }; + + // Create a test namespace and table + let namespace = NamespaceIdent::new("test_s3tables_update_table".to_string()); + let table_ident = + TableIdent::new(namespace.clone(), "test_s3tables_update_table".to_string()); + + // Clean up any existing resources from previous test runs + catalog.drop_table(&table_ident).await.ok(); + catalog.drop_namespace(&namespace).await.ok(); + + // Create namespace and table + catalog + .create_namespace(&namespace, HashMap::new()) + .await + .unwrap(); + + let creation = { + let schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "foo", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "bar", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + TableCreation::builder() + .name(table_ident.name().to_string()) + .properties(HashMap::new()) + .schema(schema) + .build() + }; + + let table = catalog.create_table(&namespace, creation).await.unwrap(); + + // Create a transaction to update the table + let tx = Transaction::new(&table); + + // Store the original metadata location for comparison + let original_metadata_location = table.metadata_location(); + + // Update table properties using the transaction + let tx = tx + .update_table_properties() + .set("test_property".to_string(), "test_value".to_string()) + .apply(tx) + .unwrap(); + + // Commit the transaction to the catalog + let updated_table = tx.commit(&catalog).await.unwrap(); + + // Verify the update was successful + assert_eq!( + updated_table.metadata().properties().get("test_property"), + Some(&"test_value".to_string()) + ); + + // Verify the metadata location has been updated + assert_ne!( + updated_table.metadata_location(), + original_metadata_location, + "Metadata location should be updated after commit" + ); + + // Load the table again from the catalog to verify changes were persisted + let reloaded_table = catalog.load_table(&table_ident).await.unwrap(); + + // Verify the reloaded table matches the updated table + assert_eq!( + reloaded_table.metadata().properties().get("test_property"), + Some(&"test_value".to_string()) + ); + assert_eq!( + reloaded_table.metadata_location(), + updated_table.metadata_location(), + "Reloaded table should have the same metadata location as the updated table" + ); + } + #[tokio::test] async fn test_builder_load_missing_bucket_arn() { let builder = S3TablesCatalogBuilder::default(); From 4a6ea15a7dd73e7c68b0a24441820b13bdd22a92 Mon Sep 17 00:00:00 2001 From: Jannik Steinmann Date: Tue, 28 Oct 2025 19:34:28 +0100 Subject: [PATCH 041/891] feat: Update Datafusion to v50 (#1728) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? --------- Signed-off-by: Xuanwo Co-authored-by: Xuanwo --- .github/workflows/bindings_python_ci.yml | 2 +- Cargo.lock | 763 ++++++------- Cargo.toml | 42 +- bindings/python/Cargo.lock | 1327 ++++++++++------------ bindings/python/Cargo.toml | 6 +- bindings/python/pyproject.toml | 2 +- crates/catalog/sql/Cargo.toml | 2 +- crates/iceberg/src/arrow/reader.rs | 11 +- 8 files changed, 956 insertions(+), 1199 deletions(-) diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index 5c904b319d..ae3a6d1a54 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -91,6 +91,6 @@ jobs: shell: bash run: | set -e - pip install hatch==1.12.0 + pip install hatch==1.14.2 hatch run dev:pip install dist/pyiceberg_core-*.whl --force-reinstall hatch run dev:test diff --git a/Cargo.lock b/Cargo.lock index 808c222dd3..cd18718290 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8,12 +8,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" -[[package]] -name = "adler32" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aae1277d39aeec15cb388266ecc24b11c80469deae6067e17a1a7aa9e5c1f234" - [[package]] name = "aes" version = "0.8.4" @@ -146,35 +140,6 @@ version = "1.0.100" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" -[[package]] -name = "apache-avro" -version = "0.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aef82843a0ec9f8b19567445ad2421ceeb1d711514384bdd3d49fe37102ee13" -dependencies = [ - "bigdecimal", - "bzip2 0.4.4", - "crc32fast", - "digest", - "libflate", - "log", - "num-bigint", - "quad-rand", - "rand 0.8.5", - "regex-lite", - "serde", - "serde_bytes", - "serde_json", - "snap", - "strum 0.26.3", - "strum_macros 0.26.4", - "thiserror 1.0.69", - "typed-builder 0.19.1", - "uuid", - "xz2", - "zstd", -] - [[package]] name = "apache-avro" version = "0.20.0" @@ -183,6 +148,8 @@ checksum = "3a033b4ced7c585199fb78ef50fca7fe2f444369ec48080c5fd072efa1a03cc7" dependencies = [ "bigdecimal", "bon", + "bzip2 0.6.1", + "crc32fast", "digest", "log", "miniz_oxide", @@ -193,13 +160,24 @@ dependencies = [ "serde", "serde_bytes", "serde_json", + "snap", "strum 0.27.2", "strum_macros 0.27.2", "thiserror 2.0.17", "uuid", + "xz2", "zstd", ] +[[package]] +name = "ar_archive_writer" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0c269894b6fe5e9d7ada0cf69b5bf847ff35bc25fc271f08e1d080fce80339a" +dependencies = [ + "object", +] + [[package]] name = "array-init" version = "2.1.0" @@ -220,9 +198,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3f15b4c6b148206ff3a2b35002e08929c2462467b62b9c02036d9c34f9ef994" +checksum = "6e833808ff2d94ed40d9379848a950d995043c7fb3e81a30b383f4c6033821cc" dependencies = [ "arrow-arith", "arrow-array", @@ -241,9 +219,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30feb679425110209ae35c3fbf82404a39a4c0436bb3ec36164d8bffed2a4ce4" +checksum = "ad08897b81588f60ba983e3ca39bda2b179bdd84dced378e7df81a5313802ef8" dependencies = [ "arrow-array", "arrow-buffer", @@ -255,9 +233,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70732f04d285d49054a48b72c54f791bb3424abae92d27aafdf776c98af161c8" +checksum = "8548ca7c070d8db9ce7aa43f37393e4bfcf3f2d3681df278490772fd1673d08d" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -266,15 +244,15 @@ dependencies = [ "chrono", "chrono-tz 0.10.4", "half", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "num", ] [[package]] name = "arrow-buffer" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "169b1d5d6cb390dd92ce582b06b23815c7953e9dfaaea75556e89d890d19993d" +checksum = "e003216336f70446457e280807a73899dd822feaf02087d31febca1363e2fccc" dependencies = [ "bytes", "half", @@ -283,9 +261,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4f12eccc3e1c05a766cafb31f6a60a46c2f8efec9b74c6e0648766d30686af8" +checksum = "919418a0681298d3a77d1a315f625916cb5678ad0d74b9c60108eb15fd083023" dependencies = [ "arrow-array", "arrow-buffer", @@ -304,9 +282,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "012c9fef3f4a11573b2c74aec53712ff9fdae4a95f4ce452d1bbf088ee00f06b" +checksum = "bfa9bf02705b5cf762b6f764c65f04ae9082c7cfc4e96e0c33548ee3f67012eb" dependencies = [ "arrow-array", "arrow-cast", @@ -319,9 +297,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de1ce212d803199684b658fc4ba55fb2d7e87b213de5af415308d2fee3619c2" +checksum = "a5c64fff1d142f833d78897a772f2e5b55b36cb3e6320376f0961ab0db7bd6d0" dependencies = [ "arrow-buffer", "arrow-schema", @@ -331,14 +309,15 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9ea5967e8b2af39aff5d9de2197df16e305f47f404781d3230b2dc672da5d92" +checksum = "1d3594dcddccc7f20fd069bc8e9828ce37220372680ff638c5e00dea427d88f5" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", + "arrow-select", "flatbuffers", "lz4_flex", "zstd", @@ -346,9 +325,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5709d974c4ea5be96d900c01576c7c0b99705f4a3eec343648cb1ca863988a9c" +checksum = "88cf36502b64a127dc659e3b305f1d993a544eab0d48cce704424e62074dc04b" dependencies = [ "arrow-array", "arrow-buffer", @@ -368,9 +347,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6506e3a059e3be23023f587f79c82ef0bcf6d293587e3272d20f2d30b969b5a7" +checksum = "3c8f82583eb4f8d84d4ee55fd1cb306720cddead7596edce95b50ee418edf66f" dependencies = [ "arrow-array", "arrow-buffer", @@ -381,9 +360,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52bf7393166beaf79b4bed9bfdf19e97472af32ce5b6b48169d321518a08cae2" +checksum = "9d07ba24522229d9085031df6b94605e0f4b26e099fb7cdeec37abd941a73753" dependencies = [ "arrow-array", "arrow-buffer", @@ -394,9 +373,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af7686986a3bf2254c9fb130c623cdcb2f8e1f15763e7c71c310f0834da3d292" +checksum = "b3aa9e59c611ebc291c28582077ef25c97f1975383f1479b12f3b9ffee2ffabe" dependencies = [ "serde", "serde_json", @@ -404,9 +383,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dd2b45757d6a2373faa3352d02ff5b54b098f5e21dccebc45a21806bc34501e5" +checksum = "8c41dbbd1e97bfcaee4fcb30e29105fb2c75e4d82ae4de70b792a5d3f66b2e7a" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -418,9 +397,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0377d532850babb4d927a06294314b316e23311503ed580ec6ce6a0158f49d40" +checksum = "53f5183c150fbc619eede22b861ea7c0eebed8eaac0333eaa7f6da5205fd504d" dependencies = [ "arrow-array", "arrow-buffer", @@ -581,7 +560,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -616,7 +595,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -797,9 +776,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.88.0" +version = "1.89.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a68d675582afea0e94d38b6ca9c5aaae4ca14f1d36faa6edb19b42e687e70d7" +checksum = "695dc67bb861ccb8426c9129b91c30e266a0e3d85650cafdf62fcca14c8fd338" dependencies = [ "aws-credential-types", "aws-runtime", @@ -914,7 +893,7 @@ dependencies = [ "hyper-util", "pin-project-lite", "rustls 0.21.12", - "rustls 0.23.33", + "rustls 0.23.34", "rustls-native-certs 0.8.2", "rustls-pki-types", "tokio", @@ -1117,7 +1096,7 @@ dependencies = [ "regex", "rustc-hash", "shlex", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1216,7 +1195,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1239,7 +1218,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1265,9 +1244,9 @@ dependencies = [ [[package]] name = "bstr" -version = "1.12.0" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "234113d19d0d7d613b40e86fb654acf958910802bcceab913a4f9e7cda03b1a4" +checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" dependencies = [ "memchr", "serde", @@ -1332,16 +1311,6 @@ dependencies = [ "either", ] -[[package]] -name = "bzip2" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bdb116a6ef3f6c3698828873ad02c3014b3c85cadb88496095628e3ef1e347f8" -dependencies = [ - "bzip2-sys", - "libc", -] - [[package]] name = "bzip2" version = "0.5.2" @@ -1353,9 +1322,9 @@ dependencies = [ [[package]] name = "bzip2" -version = "0.6.0" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bea8dcd42434048e4f7a304411d9273a411f647446c1234a65ce0554923f4cff" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" dependencies = [ "libbz2-rs-sys", ] @@ -1381,9 +1350,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.41" +version = "1.2.43" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac9fe6cdbb24b6ade63616c0a0688e45bb56732262c158df3c0c4bea4ca47cb7" +checksum = "739eb0f94557554b3ca9a86d2d37bebd49c5e6d0c1d2bda35ba5bdac830befc2" dependencies = [ "find-msvc-tools", "jobserver", @@ -1481,9 +1450,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.49" +version = "4.5.50" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4512b90fa68d3a9932cea5184017c5d200f5921df706d45e853537dea51508f" +checksum = "0c2cfd7bf8a6017ddaa4e32ffe7403d547790db06bd171c1c53926faab501623" dependencies = [ "clap_builder", "clap_derive", @@ -1491,9 +1460,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.49" +version = "4.5.50" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0025e98baa12e766c67ba13ff4695a887a1eba19569aad00a472546795bd6730" +checksum = "0a4c05b9e80c5ccd3a7ef080ad7b6ba7d6fc00a985b8b157197075677c82c7a0" dependencies = [ "anstream", "anstyle", @@ -1510,7 +1479,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1554,11 +1523,12 @@ dependencies = [ [[package]] name = "comfy-table" -version = "7.2.1" +version = "7.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b03b7db8e0b4b2fdad6c551e634134e99ec000e5c8c3b6856c65e8bbaded7a3b" +checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56" dependencies = [ - "unicode-segmentation", + "strum 0.26.3", + "strum_macros 0.26.4", "unicode-width 0.2.2", ] @@ -1573,15 +1543,15 @@ dependencies = [ [[package]] name = "console" -version = "0.15.11" +version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8" +checksum = "b430743a6eb14e9764d4260d4c0d8123087d504eeb9c48f2b2a5e810dd369df4" dependencies = [ "encode_unicode", "libc", "once_cell", "unicode-width 0.2.2", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -1642,15 +1612,6 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" -[[package]] -name = "core2" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b49ba7ef1ad6107f8824dbe97de947cbaac53c44e7f9756a1fba0d37c1eec505" -dependencies = [ - "memchr", -] - [[package]] name = "cpufeatures" version = "0.2.17" @@ -1780,7 +1741,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" dependencies = [ "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1814,7 +1775,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1828,7 +1789,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1839,7 +1800,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -1850,15 +1811,9 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.107", + "syn 2.0.108", ] -[[package]] -name = "dary_heap" -version = "0.3.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06d2e3287df1c007e74221c49ca10a95d557349e54b3a75dc2fb14712c751f04" - [[package]] name = "dashmap" version = "6.1.0" @@ -1875,16 +1830,16 @@ dependencies = [ [[package]] name = "datafusion" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69dfeda1633bf8ec75b068d9f6c27cdc392ffcf5ff83128d5dbab65b73c1fd02" +checksum = "2af15bb3c6ffa33011ef579f6b0bcbe7c26584688bd6c994f548e44df67f011a" dependencies = [ "arrow", "arrow-ipc", "arrow-schema", "async-trait", "bytes", - "bzip2 0.6.0", + "bzip2 0.6.1", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1905,6 +1860,7 @@ dependencies = [ "datafusion-functions-window", "datafusion-optimizer", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", @@ -1931,9 +1887,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2848fd1e85e2953116dab9cc2eb109214b0888d7bbd2230e30c07f1794f642c0" +checksum = "187622262ad8f7d16d3be9202b4c1e0116f1c9aa387e5074245538b755261621" dependencies = [ "arrow", "async-trait", @@ -1957,9 +1913,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "051a1634628c2d1296d4e326823e7536640d87a118966cdaff069b68821ad53b" +checksum = "9657314f0a32efd0382b9a46fdeb2d233273ece64baa68a7c45f5a192daf0f83" dependencies = [ "arrow", "async-trait", @@ -1980,9 +1936,9 @@ dependencies = [ [[package]] name = "datafusion-cli" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5966ac4973bf66cf5c189046b1b29de90db9ba059ce7193137dd089e277495de" +checksum = "6a0b9c821d14e79070f42ea3a6d6618ced04d94277f0a32301918d7a022c250f" dependencies = [ "arrow", "async-trait", @@ -2006,12 +1962,12 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "765e4ad4ef7a4500e389a3f1e738791b71ff4c29fd00912c2f541d62b25da096" +checksum = "5a83760d9a13122d025fbdb1d5d5aaf93dd9ada5e90ea229add92aa30898b2d1" dependencies = [ "ahash 0.8.12", - "apache-avro 0.17.0", + "apache-avro", "arrow", "arrow-ipc", "base64 0.22.1", @@ -2033,9 +1989,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40a2ae8393051ce25d232a6065c4558ab5a535c9637d5373bacfd464ac88ea12" +checksum = "5b6234a6c7173fe5db1c6c35c01a12b2aa0f803a3007feee53483218817f8b1e" dependencies = [ "futures", "log", @@ -2044,21 +2000,22 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90cd841a77f378bc1a5c4a1c37345e1885a9203b008203f9f4b3a769729bf330" +checksum = "7256c9cb27a78709dd42d0c80f0178494637209cac6e29d5c93edd09b6721b86" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2 0.6.0", + "bzip2 0.6.1", "chrono", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", @@ -2080,11 +2037,11 @@ dependencies = [ [[package]] name = "datafusion-datasource-avro" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3cba1696aa919da9517d29164d45f5902d6cc281f718e8d3bfe98bd52cd1142c" +checksum = "10d40b6953ebc9099b37adfd12fde97eb73ff0cee44355c6dea64b8a4537d561" dependencies = [ - "apache-avro 0.17.0", + "apache-avro", "arrow", "async-trait", "bytes", @@ -2105,9 +2062,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77f4a2c64939c6f0dd15b246723a699fa30d59d0133eb36a86e8ff8c6e2a8dc6" +checksum = "64533a90f78e1684bfb113d200b540f18f268134622d7c96bbebc91354d04825" dependencies = [ "arrow", "async-trait", @@ -2130,9 +2087,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11387aaf931b2993ad9273c63ddca33f05aef7d02df9b70fb757429b4b71cdae" +checksum = "8d7ebeb12c77df0aacad26f21b0d033aeede423a64b2b352f53048a75bf1d6e6" dependencies = [ "arrow", "async-trait", @@ -2155,9 +2112,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "028f430c5185120bf806347848b8d8acd9823f4038875b3820eeefa35f2bb4a2" +checksum = "09e783c4c7d7faa1199af2df4761c68530634521b176a8d1331ddbc5a5c75133" dependencies = [ "arrow", "async-trait", @@ -2170,6 +2127,7 @@ dependencies = [ "datafusion-expr", "datafusion-functions-aggregate", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", @@ -2188,17 +2146,18 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff336d1d755399753a9e4fbab001180e346fc8bfa063a97f1214b82274c00f8" +checksum = "99ee6b1d9a80d13f9deb2291f45c07044b8e62fb540dbde2453a18be17a36429" [[package]] name = "datafusion-execution" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "042ea192757d1b2d7dcf71643e7ff33f6542c7704f00228d8b85b40003fd8e0f" +checksum = "a4cec0a57653bec7b933fb248d3ffa3fa3ab3bd33bd140dc917f714ac036f531" dependencies = [ "arrow", + "async-trait", "dashmap", "datafusion-common", "datafusion-expr", @@ -2206,6 +2165,7 @@ dependencies = [ "log", "object_store", "parking_lot", + "parquet", "rand 0.9.2", "tempfile", "url", @@ -2213,9 +2173,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "025222545d6d7fab71e2ae2b356526a1df67a2872222cbae7535e557a42abd2e" +checksum = "ef76910bdca909722586389156d0aa4da4020e1631994d50fadd8ad4b1aa05fe" dependencies = [ "arrow", "async-trait", @@ -2235,9 +2195,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d5c267104849d5fa6d81cf5ba88f35ecd58727729c5eb84066c25227b644ae2" +checksum = "6d155ccbda29591ca71a1344dd6bed26c65a4438072b400df9db59447f590bb6" dependencies = [ "arrow", "datafusion-common", @@ -2248,9 +2208,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c620d105aa208fcee45c588765483314eb415f5571cfd6c1bae3a59c5b4d15bb" +checksum = "7de2782136bd6014670fd84fe3b0ca3b3e4106c96403c3ae05c0598577139977" dependencies = [ "arrow", "arrow-buffer", @@ -2277,9 +2237,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35f61d5198a35ed368bf3aacac74f0d0fa33de7a7cb0c57e9f68ab1346d2f952" +checksum = "07331fc13603a9da97b74fd8a273f4238222943dffdbbed1c4c6f862a30105bf" dependencies = [ "ahash 0.8.12", "arrow", @@ -2298,9 +2258,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13efdb17362be39b5024f6da0d977ffe49c0212929ec36eec550e07e2bc7812f" +checksum = "b5951e572a8610b89968a09b5420515a121fbc305c0258651f318dc07c97ab17" dependencies = [ "ahash 0.8.12", "arrow", @@ -2311,9 +2271,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9187678af567d7c9e004b72a0b6dc5b0a00ebf4901cb3511ed2db4effe092e66" +checksum = "fdacca9302c3d8fc03f3e94f338767e786a88a33f5ebad6ffc0e7b50364b9ea3" dependencies = [ "arrow", "arrow-ord", @@ -2333,9 +2293,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecf156589cc21ef59fe39c7a9a841b4a97394549643bbfa88cc44e8588cf8fe5" +checksum = "8c37ff8a99434fbbad604a7e0669717c58c7c4f14c472d45067c4b016621d981" dependencies = [ "arrow", "async-trait", @@ -2349,9 +2309,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edcb25e3e369f1366ec9a261456e45b5aad6ea1c0c8b4ce546587207c501ed9e" +checksum = "48e2aea7c79c926cffabb13dc27309d4eaeb130f4a21c8ba91cdd241c813652b" dependencies = [ "arrow", "datafusion-common", @@ -2367,9 +2327,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8996a8e11174d0bd7c62dc2f316485affc6ae5ffd5b8a68b508137ace2310294" +checksum = "0fead257ab5fd2ffc3b40fda64da307e20de0040fe43d49197241d9de82a487f" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -2377,20 +2337,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95ee8d1be549eb7316f437035f2cec7ec42aba8374096d807c4de006a3b5d78a" +checksum = "ec6f637bce95efac05cdfb9b6c19579ed4aa5f6b94d951cfa5bb054b7bb4f730" dependencies = [ "datafusion-expr", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] name = "datafusion-optimizer" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9fa98671458254928af854e5f6c915e66b860a8bde505baea0ff2892deab74d" +checksum = "c6583ef666ae000a613a837e69e456681a9faa96347bf3877661e9e89e141d8a" dependencies = [ "arrow", "chrono", @@ -2408,9 +2368,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3515d51531cca5f7b5a6f3ea22742b71bb36fc378b465df124ff9a2fa349b002" +checksum = "c8668103361a272cbbe3a61f72eca60c9b7c706e87cc3565bcf21e2b277b84f6" dependencies = [ "ahash 0.8.12", "arrow", @@ -2424,15 +2384,31 @@ dependencies = [ "indexmap 2.12.0", "itertools 0.14.0", "log", + "parking_lot", "paste", "petgraph 0.8.3", ] +[[package]] +name = "datafusion-physical-expr-adapter" +version = "50.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "815acced725d30601b397e39958e0e55630e0a10d66ef7769c14ae6597298bb0" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "itertools 0.14.0", +] + [[package]] name = "datafusion-physical-expr-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24485475d9c618a1d33b2a3dad003d946dc7a7bbf0354d125301abc0a5a79e3e" +checksum = "6652fe7b5bf87e85ed175f571745305565da2c0b599d98e697bcbedc7baa47c3" dependencies = [ "ahash 0.8.12", "arrow", @@ -2444,9 +2420,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9da411a0a64702f941a12af2b979434d14ec5d36c6f49296966b2c7639cbb3a" +checksum = "49b7d623eb6162a3332b564a0907ba00895c505d101b99af78345f1acf929b5c" dependencies = [ "arrow", "datafusion-common", @@ -2464,9 +2440,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6d168282bb7b54880bb3159f89b51c047db4287f5014d60c3ef4c6e1468212b" +checksum = "e2f7f778a1a838dec124efb96eae6144237d546945587557c9e6936b3414558c" dependencies = [ "ahash 0.8.12", "arrow", @@ -2478,6 +2454,7 @@ dependencies = [ "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", + "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", @@ -2494,9 +2471,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "49.0.0" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "391a457b9d23744c53eeb89edd1027424cba100581488d89800ed841182df905" +checksum = "cd1e59e2ca14fe3c30f141600b10ad8815e2856caa59ebbd0e3e07cd3d127a65" dependencies = [ "arrow", "arrow-schema", @@ -2512,9 +2489,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053201c2bb729c7938f85879034df2b5a52cfaba16f1b3b66ab8505c81b2aad3" +checksum = "21ef8e2745583619bd7a49474e8f45fbe98ebb31a133f27802217125a7b3d58d" dependencies = [ "arrow", "async-trait", @@ -2536,11 +2513,13 @@ dependencies = [ [[package]] name = "datafusion-spark" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "126ceb2436b498ddd66c0a9f58304bce6b903f622c681c0ca9e52cf229c55ea0" +checksum = "613efb6666a7d42fcb922b90cd0daa2b25ea486d141350e5d3e86e46df28309a" dependencies = [ "arrow", + "chrono", + "crc32fast", "datafusion-catalog", "datafusion-common", "datafusion-execution", @@ -2548,13 +2527,16 @@ dependencies = [ "datafusion-functions", "datafusion-macros", "log", + "sha1", + "url", + "xxhash-rust", ] [[package]] name = "datafusion-sql" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9082779be8ce4882189b229c0cff4393bd0808282a7194130c9f32159f185e25" +checksum = "89abd9868770386fede29e5a4b14f49c0bf48d652c3b9d7a8a0332329b87d50b" dependencies = [ "arrow", "bigdecimal", @@ -2569,9 +2551,9 @@ dependencies = [ [[package]] name = "datafusion-sqllogictest" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c2358f74d54fb9895b3b962f14877e414327eaaad4a95154abacc3e6bdcd38b" +checksum = "17598193dd875ca895400c51ccab1c30fceb1855220dc60aa415a4db7c95a2d7" dependencies = [ "arrow", "async-trait", @@ -2596,9 +2578,9 @@ dependencies = [ [[package]] name = "datafusion-substrait" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4189e480ae42f06ade81a6d12853c5f3f84fbe237d73fb8795c712b6f1923afb" +checksum = "eaa011a3814d91a03ab655ad41bbe5e57b203b2859281af8fe2c30aebbbcc5d9" dependencies = [ "async-recursion", "async-trait", @@ -2611,6 +2593,7 @@ dependencies = [ "substrait", "tokio", "url", + "uuid", ] [[package]] @@ -2626,9 +2609,9 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.4" +version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a41953f86f8a05768a6cda24def994fd2f424b04ec5c719cf89989779f199071" +checksum = "ececcb659e7ba858fb4f10388c250a7252eb0a27373f1a72b8748afdd248e587" dependencies = [ "powerfmt", "serde_core", @@ -2652,7 +2635,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -2662,7 +2645,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -2707,7 +2690,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2718,7 +2701,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -2769,7 +2752,7 @@ dependencies = [ "enum-ordinalize", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -2795,22 +2778,22 @@ checksum = "c34f04666d835ff5d62e058c3995147c06f42fe86ff053337632bca83e42702d" [[package]] name = "enum-ordinalize" -version = "4.3.0" +version = "4.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fea0dcfa4e54eeb516fe454635a95753ddd39acda650ce703031c6973e315dd5" +checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.3.1" +version = "4.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d28318a75d4aead5c4db25382e8ef717932d0346600cacae6357eb5941bc5ff" +checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -2849,7 +2832,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2965,9 +2948,9 @@ dependencies = [ [[package]] name = "flate2" -version = "1.1.4" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc5a4e564e38c699f2880d3fda590bedc2e69f3f84cd48b457bd892ce61d0aa9" +checksum = "bfe33edd8e85a12a67454e37f8c75e730830d83e313556ab9ebf9ee7fbeb3bfb" dependencies = [ "crc32fast", "libz-rs-sys", @@ -3113,7 +3096,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -3191,9 +3174,9 @@ checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" [[package]] name = "globset" -version = "0.4.17" +version = "0.4.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eab69130804d941f8075cfd713bf8848a2c3b3f201a9457a11e6f87e1ab62305" +checksum = "52dfc19153a48bde0cbd630453615c8151bce3a5adfac7a0aebfbf0a1e1f57e3" dependencies = [ "aho-corasick", "bstr", @@ -3532,7 +3515,7 @@ dependencies = [ "http 1.3.1", "hyper 1.7.0", "hyper-util", - "rustls 0.23.33", + "rustls 0.23.34", "rustls-native-certs 0.8.2", "rustls-pki-types", "tokio", @@ -3594,7 +3577,7 @@ name = "iceberg" version = "0.7.0" dependencies = [ "anyhow", - "apache-avro 0.20.0", + "apache-avro", "array-init", "arrow-arith", "arrow-array", @@ -3645,7 +3628,7 @@ dependencies = [ "tera", "thrift", "tokio", - "typed-builder 0.20.1", + "typed-builder", "url", "uuid", "zstd", @@ -3735,7 +3718,7 @@ dependencies = [ "serde_json", "tokio", "tracing", - "typed-builder 0.20.1", + "typed-builder", "uuid", ] @@ -3866,9 +3849,9 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "200072f5d0e3614556f94a9930d5dc3e0662a652823904c3a75dc3b0af7fee47" +checksum = "f578a71f2bfaf7ceb30b519a645ae48024b45f9eecbe060a31a004d7b4ba9462" dependencies = [ "displaydoc", "potential_utf", @@ -3879,9 +3862,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cde2700ccaed3872079a65fb1a78f6c0a36c91570f28755dda67bc8f7d9f00a" +checksum = "4c219b62bf5a06801012446193fdfcbd7970e876823aba4c62def2ce957dcb44" dependencies = [ "displaydoc", "litemap", @@ -3892,11 +3875,10 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "436880e8e18df4d7bbc06d58432329d6458cc84531f7ac5f024e93deadb37979" +checksum = "33747cecc725eebb47ac503fab725e395d50cb7889ae490a1359f130611d4cc5" dependencies = [ - "displaydoc", "icu_collections", "icu_normalizer_data", "icu_properties", @@ -3907,42 +3889,38 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00210d6893afc98edb752b664b8890f0ef174c8adbb8d0be9710fa66fbbf72d3" +checksum = "d6ce2d23e1b3c45624ba6a23e2c767e01c9680e0c0800b39c7abfff9565175d8" [[package]] name = "icu_properties" -version = "2.0.1" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "016c619c1eeb94efb86809b015c58f479963de65bdb6253345c1a1276f22e32b" +checksum = "6d70f9b6574c79f7a83ea5ce72cc88d271a3e77355c5f7748a107e751d8617fb" dependencies = [ - "displaydoc", "icu_collections", "icu_locale_core", "icu_properties_data", "icu_provider", - "potential_utf", "zerotrie", "zerovec", ] [[package]] name = "icu_properties_data" -version = "2.0.1" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "298459143998310acd25ffe6810ed544932242d3f07083eee1084d83a71bd632" +checksum = "17fa55bf868e28e638ed132bcee1e5c21ba2c1e52c15e7c78b781858e7b54342" [[package]] name = "icu_provider" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03c80da27b5f4187909049ee2d72f276f0d9f99a42c306bd0131ecfe04d8e5af" +checksum = "f64958e359123591ae1f17a27b5fc9ebdb50c98b04e0401146154de1d8fe3e44" dependencies = [ "displaydoc", "icu_locale_core", - "stable_deref_trait", - "tinystr", "writeable", "yoke", "zerofrom", @@ -4018,14 +3996,14 @@ dependencies = [ [[package]] name = "indicatif" -version = "0.17.11" +version = "0.18.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "183b3088984b400f4cfac3620d5e076c84da5364016b4f49473de574b2586235" +checksum = "ade6dfcba0dfb62ad59e59e7241ec8912af34fd29e0e743e3db992bd278e8b65" dependencies = [ "console", - "number_prefix", "portable-atomic", "unicode-width 0.2.2", + "unit-prefix", "web-time", ] @@ -4073,9 +4051,9 @@ dependencies = [ [[package]] name = "is_terminal_polyfill" -version = "1.70.1" +version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" [[package]] name = "itertools" @@ -4122,7 +4100,7 @@ checksum = "03343451ff899767262ec32146f6d559dd759fdadf42ff0e227c7c48f72594b4" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -4137,9 +4115,9 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.81" +version = "0.3.82" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec48937a97411dcb524a265206ccd4c90bb711fca92b2792c407f268825b9305" +checksum = "b011eec8cc36da2aab2d5cff675ec18454fad408585853910a202391cf9f8e65" dependencies = [ "once_cell", "wasm-bindgen", @@ -4238,30 +4216,6 @@ version = "0.2.177" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976" -[[package]] -name = "libflate" -version = "2.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "45d9dfdc14ea4ef0900c1cddbc8dcd553fbaacd8a4a282cf4018ae9dd04fb21e" -dependencies = [ - "adler32", - "core2", - "crc32fast", - "dary_heap", - "libflate_lz77", -] - -[[package]] -name = "libflate_lz77" -version = "2.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e0d73b369f386f1c44abd9c570d5318f55ccde816ff4b562fa452e5182863d" -dependencies = [ - "core2", - "hashbrown 0.14.5", - "rle-decode-fast", -] - [[package]] name = "libloading" version = "0.8.9" @@ -4356,9 +4310,9 @@ checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039" [[package]] name = "litemap" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "241eaef5fd12c88705a01fc1066c48c4b36e0dd4377dcdc7ec3942cea7a69956" +checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" [[package]] name = "lock_api" @@ -4498,7 +4452,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -4566,7 +4520,7 @@ checksum = "b40e46c845ac234bcba19db7ab252bc2778cbadd516a466d2f12b1580852d136" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -4592,7 +4546,7 @@ checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -4657,7 +4611,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4787,14 +4741,17 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] -name = "number_prefix" -version = "0.4.0" +name = "object" +version = "0.32.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" +checksum = "a6a622008b6e321afc04970976f62ee297fdbaa6f95318ca343e3eebb9648441" +dependencies = [ + "memchr", +] [[package]] name = "object_store" @@ -4841,9 +4798,9 @@ checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" [[package]] name = "once_cell_polyfill" -version = "1.70.1" +version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4895175b425cb1f87721b59f0f286c2092bd4af812243672510e1ac53e2e0ad" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "opendal" @@ -4958,9 +4915,9 @@ dependencies = [ [[package]] name = "parquet" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b17da4150748086bd43352bc77372efa9b6e3dbd06a04831d2a98c041c225cfa" +checksum = "f0dbd48ad52d7dccf8ea1b90a3ddbfaea4f69878dd7683e51c507d4bc52b5b27" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -4977,7 +4934,7 @@ dependencies = [ "flate2", "futures", "half", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "lz4_flex", "num", "num-bigint", @@ -5110,7 +5067,7 @@ dependencies = [ "pest_meta", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5240,7 +5197,7 @@ checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5376,9 +5333,9 @@ dependencies = [ [[package]] name = "potential_utf" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "84df19adbe5b5a0782edcab45899906947ab039ccf4573713735ee7de1e6b08a" +checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77" dependencies = [ "zerovec", ] @@ -5441,7 +5398,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5455,9 +5412,9 @@ dependencies = [ [[package]] name = "proc-macro2" -version = "1.0.101" +version = "1.0.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de" +checksum = "5ee95bc4ef87b8d5ba32e8b7714ccc834865276eab0aed5c9958d00ec45f49e8" dependencies = [ "unicode-ident", ] @@ -5488,7 +5445,7 @@ dependencies = [ "prost", "prost-types", "regex", - "syn 2.0.107", + "syn 2.0.108", "tempfile", ] @@ -5502,7 +5459,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5516,10 +5473,11 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.27" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e66fcd288453b748497d8fb18bccc83a16b0518e3906d4b8df0a8d42d93dbb1c" +checksum = "d11f2fedc3b7dafdc2851bc52f277377c5473d378859be234bc7ebb593144d01" dependencies = [ + "ar_archive_writer", "cc", ] @@ -5560,7 +5518,7 @@ checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5601,7 +5559,7 @@ dependencies = [ "quinn-proto", "quinn-udp", "rustc-hash", - "rustls 0.23.33", + "rustls 0.23.34", "socket2 0.6.1", "thiserror 2.0.17", "tokio", @@ -5621,7 +5579,7 @@ dependencies = [ "rand 0.9.2", "ring", "rustc-hash", - "rustls 0.23.33", + "rustls 0.23.34", "rustls-pki-types", "slab", "thiserror 2.0.17", @@ -5641,7 +5599,7 @@ dependencies = [ "once_cell", "socket2 0.6.1", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -5762,7 +5720,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5802,7 +5760,7 @@ checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -5919,7 +5877,7 @@ dependencies = [ "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.33", + "rustls 0.23.34", "rustls-native-certs 0.8.2", "rustls-pki-types", "serde", @@ -6009,15 +5967,9 @@ checksum = "bd83f5f173ff41e00337d97f6572e416d022ef8a19f371817259ae960324c482" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] -[[package]] -name = "rle-decode-fast" -version = "1.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3582f63211428f83597b51b2ddb88e2a91a9d52d12831f9d08f5e624e8977422" - [[package]] name = "roaring" version = "0.11.2" @@ -6104,7 +6056,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6121,9 +6073,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.33" +version = "0.23.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "751e04a496ca00bb97a5e043158d23d66b5aabf2e1d5aa2a0aaebb1aafe6f82c" +checksum = "6a9586e9ee2b4f8fab52a0048ca7334d7024eef48e2cb9407e3497bb7cab7fa7" dependencies = [ "aws-lc-rs", "once_cell", @@ -6178,9 +6130,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.12.0" +version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "229a4a4c221013e7e1f1a043678c5cc39fe5171437c88fb47151a21e6f5b5c79" +checksum = "94182ad936a0c91c324cd46c6511b9510ed16af436d7b5bab34beab0afd55f7a" dependencies = [ "web-time", "zeroize", @@ -6216,9 +6168,9 @@ checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" [[package]] name = "rustyline" -version = "16.0.0" +version = "17.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62fd9ca5ebc709e8535e8ef7c658eb51457987e48c98ead2be482172accc408d" +checksum = "e902948a25149d50edc1a8e0141aad50f54e22ba83ff988cf8f7c9ef07f50564" dependencies = [ "bitflags", "cfg-if", @@ -6233,7 +6185,7 @@ dependencies = [ "unicode-segmentation", "unicode-width 0.2.2", "utf8parse", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -6314,7 +6266,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6439,7 +6391,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6450,7 +6402,7 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6474,7 +6426,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6495,7 +6447,7 @@ dependencies = [ "proc-macro2", "quote", "serde", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6512,9 +6464,9 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.15.0" +version = "3.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6093cd8c01b25262b84927e0f7151692158fab02d961e04c979d3903eba7ecc5" +checksum = "aa66c845eee442168b2c8134fec70ac50dc20e760769c8ba0ad1319ca1959b04" dependencies = [ "base64 0.22.1", "chrono", @@ -6531,14 +6483,14 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.15.0" +version = "3.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7e6c180db0816026a61afa1cff5344fb7ebded7e4d3062772179f2501481c27" +checksum = "b91a903660542fced4e99881aa481bdbaec1634568ee02e0b8bd57c64cb38955" dependencies = [ "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6798,9 +6750,9 @@ dependencies = [ [[package]] name = "sqlparser" -version = "0.55.0" +version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4521174166bac1ff04fe16ef4524c70144cd29682a45978978ca3d7f4e0be11" +checksum = "ec4b661c54b1e4b603b37873a18c59920e4c51ea8ea2cf527d925424dbd4437c" dependencies = [ "log", "recursive", @@ -6815,7 +6767,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6854,7 +6806,7 @@ dependencies = [ "memchr", "once_cell", "percent-encoding", - "rustls 0.23.33", + "rustls 0.23.34", "serde", "serde_json", "sha2", @@ -6877,7 +6829,7 @@ dependencies = [ "quote", "sqlx-core", "sqlx-macros-core", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -6898,7 +6850,7 @@ dependencies = [ "sha2", "sqlx-core", "sqlx-sqlite", - "syn 2.0.107", + "syn 2.0.108", "tokio", "url", ] @@ -7066,7 +7018,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7078,7 +7030,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7111,7 +7063,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.107", + "syn 2.0.108", "typify", "walkdir", ] @@ -7135,9 +7087,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.107" +version = "2.0.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a26dbd934e5451d21ef060c018dae56fc073894c5a7896f882928a76e6d081b" +checksum = "da58917d35242480a05c2897064da0a80589a2a0476c9a3f2fdc83b53502e917" dependencies = [ "proc-macro2", "quote", @@ -7161,7 +7113,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7186,7 +7138,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -7243,7 +7195,7 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7254,7 +7206,7 @@ checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7330,9 +7282,9 @@ dependencies = [ [[package]] name = "tinystr" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d4f6d1145dcb577acf783d4e601bc1d76a13337bb54e6233add580b07344c8b" +checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869" dependencies = [ "displaydoc", "zerovec", @@ -7378,7 +7330,7 @@ checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7397,7 +7349,7 @@ version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.33", + "rustls 0.23.34", "tokio", ] @@ -7561,7 +7513,7 @@ checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7611,33 +7563,13 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" -[[package]] -name = "typed-builder" -version = "0.19.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a06fbd5b8de54c5f7c91f6fe4cebb949be2125d7758e630bb58b1d831dbce600" -dependencies = [ - "typed-builder-macro 0.19.1", -] - [[package]] name = "typed-builder" version = "0.20.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cd9d30e3a08026c78f246b173243cf07b3696d274debd26680773b6773c2afc7" dependencies = [ - "typed-builder-macro 0.20.1", -] - -[[package]] -name = "typed-builder-macro" -version = "0.19.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9534daa9fd3ed0bd911d462a37f172228077e7abf18c18a5f67199d959205f8" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.107", + "typed-builder-macro", ] [[package]] @@ -7648,7 +7580,7 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -7682,7 +7614,7 @@ dependencies = [ "semver", "serde", "serde_json", - "syn 2.0.107", + "syn 2.0.108", "thiserror 2.0.17", "unicode-ident", ] @@ -7700,7 +7632,7 @@ dependencies = [ "serde", "serde_json", "serde_tokenstream", - "syn 2.0.107", + "syn 2.0.108", "typify-impl", ] @@ -7768,9 +7700,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" -version = "1.0.19" +version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d" +checksum = "462eeb75aeb73aea900253ce739c8e18a67423fadf006037cd3ff27e82748a06" [[package]] name = "unicode-normalization" @@ -7805,6 +7737,12 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +[[package]] +name = "unit-prefix" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "323402cff2dd658f39ca17c789b502021b3f18707c91cdf22e3838e1b4023817" + [[package]] name = "unsafe-libyaml" version = "0.2.11" @@ -7981,9 +7919,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasm-bindgen" -version = "0.2.104" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1da10c01ae9f1ae40cbfac0bac3b1e724b320abfcf52229f80b547c0d250e2d" +checksum = "da95793dfc411fbbd93f5be7715b0578ec61fe87cb1a42b12eb625caa5c5ea60" dependencies = [ "cfg-if", "once_cell", @@ -7992,25 +7930,11 @@ dependencies = [ "wasm-bindgen-shared", ] -[[package]] -name = "wasm-bindgen-backend" -version = "0.2.104" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "671c9a5a66f49d8a47345ab942e2cb93c7d1d0339065d4f8139c486121b43b19" -dependencies = [ - "bumpalo", - "log", - "proc-macro2", - "quote", - "syn 2.0.107", - "wasm-bindgen-shared", -] - [[package]] name = "wasm-bindgen-futures" -version = "0.4.54" +version = "0.4.55" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e038d41e478cc73bae0ff9b36c60cff1c98b8f38f8d7e8061e79ee63608ac5c" +checksum = "551f88106c6d5e7ccc7cd9a16f312dd3b5d36ea8b4954304657d5dfba115d4a0" dependencies = [ "cfg-if", "js-sys", @@ -8021,9 +7945,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.104" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ca60477e4c59f5f2986c50191cd972e3a50d8a95603bc9434501cf156a9a119" +checksum = "04264334509e04a7bf8690f2384ef5265f05143a4bff3889ab7a3269adab59c2" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -8031,22 +7955,22 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.104" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f07d2f20d4da7b26400c9f4a0511e6e0345b040694e8a75bd41d578fa4421d7" +checksum = "420bc339d9f322e562942d52e115d57e950d12d88983a14c79b86859ee6c7ebc" dependencies = [ + "bumpalo", "proc-macro2", "quote", - "syn 2.0.107", - "wasm-bindgen-backend", + "syn 2.0.108", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.104" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad67dc8b2a1a6e5448428adec4c3e84c43e561d8c9ee8a9e5aabeb193ec41d1" +checksum = "76f218a38c84bcb33c25ec7059b07847d465ce0e0a76b995e134a45adcb6af76" dependencies = [ "unicode-ident", ] @@ -8066,9 +7990,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.81" +version = "0.3.82" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9367c417a924a74cae129e6a2ae3b47fabb1f8995595ab474029da749a8be120" +checksum = "3a1f95c0d03a47f4ae1f7a64643a6bb97465d9b740f0fa8f90ea33915c99a9a1" dependencies = [ "js-sys", "wasm-bindgen", @@ -8118,7 +8042,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -8142,7 +8066,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -8153,7 +8077,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -8428,9 +8352,9 @@ checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" [[package]] name = "writeable" -version = "0.6.1" +version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea2f10b9bb0928dfb1b42b65e1f9e36f7f54dbdf08457afefb38afcdec4fa2bb" +checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" [[package]] name = "wyz" @@ -8447,6 +8371,12 @@ version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" +[[package]] +name = "xxhash-rust" +version = "0.8.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3" + [[package]] name = "xz2" version = "0.1.7" @@ -8464,11 +8394,10 @@ checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" [[package]] name = "yoke" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f41bb01b8226ef4bfd589436a297c53d118f65921786300e427be8d487695cc" +checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954" dependencies = [ - "serde", "stable_deref_trait", "yoke-derive", "zerofrom", @@ -8476,13 +8405,13 @@ dependencies = [ [[package]] name = "yoke-derive" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38da3c9736e16c5d3c8c597a9aaa5d1fa565d0532ae05e27c24aa62fb32c0ab6" +checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", "synstructure", ] @@ -8503,7 +8432,7 @@ checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] @@ -8523,7 +8452,7 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", "synstructure", ] @@ -8535,9 +8464,9 @@ checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" [[package]] name = "zerotrie" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36f0bbd478583f79edad978b407914f61b2972f5af6fa089686016be8f9af595" +checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851" dependencies = [ "displaydoc", "yoke", @@ -8546,9 +8475,9 @@ dependencies = [ [[package]] name = "zerovec" -version = "0.11.4" +version = "0.11.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7aa2bd55086f1ab526693ecbe444205da57e25f4489879da80635a46d90e73b" +checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002" dependencies = [ "yoke", "zerofrom", @@ -8557,13 +8486,13 @@ dependencies = [ [[package]] name = "zerovec-derive" -version = "0.11.1" +version = "0.11.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b96237efa0c878c64bd89c436f661be4e46b2f3eff1ebb976f7ef2321d2f58f" +checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.107", + "syn 2.0.108", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 49eeff038a..6f4385cf1c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -42,17 +42,17 @@ rust-version = "1.87" anyhow = "1.0.72" apache-avro = { version = "0.20", features = ["zstandard"] } array-init = "2" -arrow-arith = { version = "55.2" } -arrow-array = { version = "55.2" } -arrow-buffer = { version = "55.2" } -arrow-cast = { version = "55.2" } -arrow-ord = { version = "55.2" } -arrow-schema = { version = "55.2" } -arrow-select = { version = "55.2" } -arrow-string = { version = "55.2" } +arrow-arith = "56.2" +arrow-array = "56.2" +arrow-buffer = "56.2" +arrow-cast = "56.2" +arrow-ord = "56.2" +arrow-schema = "56.2" +arrow-select = "56.2" +arrow-string = "56.2" as-any = "0.3.2" -async-trait = "0.1.88" -aws-config = "1.8.1" +async-trait = "0.1.89" +aws-config = "1.8.7" aws-sdk-glue = "1.39" aws-sdk-s3tables = "1.28.0" backon = "1.5.1" @@ -60,11 +60,11 @@ base64 = "0.22.1" bimap = "0.6" bytes = "1.10" chrono = "0.4.41" -clap = { version = "4.5.41", features = ["derive", "cargo"] } +clap = { version = "4.5.48", features = ["derive", "cargo"] } ctor = "0.2.8" -datafusion = "49" -datafusion-cli = "49" -datafusion-sqllogictest = "49" +datafusion = "50" +datafusion-cli = "50" +datafusion-sqllogictest = "50" derive_builder = "0.20" dirs = "6" enum-ordinalize = "4.3.0" @@ -84,7 +84,7 @@ iceberg-catalog-sql = { version = "0.7.0", path = "./crates/catalog/sql" } iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } -indicatif = "0.17" +indicatif = "0.18" itertools = "0.13" libtest-mimic = "0.8.1" linkedbytes = "0.1.8" @@ -99,12 +99,12 @@ num-bigint = "0.4.6" once_cell = "1.20" opendal = "0.54.0" ordered-float = "4" -parquet = "55.2" +parquet = "56.2" pilota = "0.11.10" port_scanner = "0.1.5" pretty_assertions = "1.4" rand = "0.8.5" -regex = "1.10.5" +regex = "1.11.3" reqwest = { version = "0.12.12", default-features = false, features = ["json"] } roaring = { version = "0.11" } rust_decimal = "1.37.2" @@ -122,12 +122,12 @@ strum = "0.27.2" tempfile = "3.18" tera = "1" thrift = "0.17.0" -tokio = { version = "1.46.1", default-features = false } +tokio = { version = "1.47", default-features = false } toml = "0.8" -tracing = "0.1.40" -tracing-subscriber = "0.3.8" +tracing = "0.1.41" +tracing-subscriber = "0.3.20" typed-builder = "0.20" -url = "2.5.4" +url = "2.5.7" uuid = { version = "1.18", features = ["v7"] } volo = "0.10.6" volo-thrift = "0.10.8" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 70e95a37aa..8249414b8d 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -50,15 +50,6 @@ dependencies = [ "core_extensions", ] -[[package]] -name = "addr2line" -version = "0.24.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1" -dependencies = [ - "gimli", -] - [[package]] name = "adler2" version = "2.0.1" @@ -84,7 +75,7 @@ checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" dependencies = [ "cfg-if", "const-random", - "getrandom 0.3.3", + "getrandom 0.3.4", "once_cell", "version_check", "zerocopy", @@ -131,9 +122,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.99" +version = "1.0.100" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0674a1ddeecb70197781e945de4b3b8ffb61fa939a5597bcf48503737663100" +checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" [[package]] name = "apache-avro" @@ -153,13 +144,22 @@ dependencies = [ "serde", "serde_bytes", "serde_json", - "strum", - "strum_macros", - "thiserror 2.0.16", + "strum 0.27.2", + "strum_macros 0.27.2", + "thiserror", "uuid", "zstd", ] +[[package]] +name = "ar_archive_writer" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0c269894b6fe5e9d7ada0cf69b5bf847ff35bc25fc271f08e1d080fce80339a" +dependencies = [ + "object", +] + [[package]] name = "array-init" version = "2.1.0" @@ -180,9 +180,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3f15b4c6b148206ff3a2b35002e08929c2462467b62b9c02036d9c34f9ef994" +checksum = "6e833808ff2d94ed40d9379848a950d995043c7fb3e81a30b383f4c6033821cc" dependencies = [ "arrow-arith", "arrow-array", @@ -202,9 +202,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30feb679425110209ae35c3fbf82404a39a4c0436bb3ec36164d8bffed2a4ce4" +checksum = "ad08897b81588f60ba983e3ca39bda2b179bdd84dced378e7df81a5313802ef8" dependencies = [ "arrow-array", "arrow-buffer", @@ -216,9 +216,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70732f04d285d49054a48b72c54f791bb3424abae92d27aafdf776c98af161c8" +checksum = "8548ca7c070d8db9ce7aa43f37393e4bfcf3f2d3681df278490772fd1673d08d" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -227,15 +227,15 @@ dependencies = [ "chrono", "chrono-tz", "half", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "num", ] [[package]] name = "arrow-buffer" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "169b1d5d6cb390dd92ce582b06b23815c7953e9dfaaea75556e89d890d19993d" +checksum = "e003216336f70446457e280807a73899dd822feaf02087d31febca1363e2fccc" dependencies = [ "bytes", "half", @@ -244,9 +244,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4f12eccc3e1c05a766cafb31f6a60a46c2f8efec9b74c6e0648766d30686af8" +checksum = "919418a0681298d3a77d1a315f625916cb5678ad0d74b9c60108eb15fd083023" dependencies = [ "arrow-array", "arrow-buffer", @@ -265,9 +265,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "012c9fef3f4a11573b2c74aec53712ff9fdae4a95f4ce452d1bbf088ee00f06b" +checksum = "bfa9bf02705b5cf762b6f764c65f04ae9082c7cfc4e96e0c33548ee3f67012eb" dependencies = [ "arrow-array", "arrow-cast", @@ -280,9 +280,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de1ce212d803199684b658fc4ba55fb2d7e87b213de5af415308d2fee3619c2" +checksum = "a5c64fff1d142f833d78897a772f2e5b55b36cb3e6320376f0961ab0db7bd6d0" dependencies = [ "arrow-buffer", "arrow-schema", @@ -292,14 +292,15 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9ea5967e8b2af39aff5d9de2197df16e305f47f404781d3230b2dc672da5d92" +checksum = "1d3594dcddccc7f20fd069bc8e9828ce37220372680ff638c5e00dea427d88f5" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", + "arrow-select", "flatbuffers", "lz4_flex", "zstd", @@ -307,9 +308,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5709d974c4ea5be96d900c01576c7c0b99705f4a3eec343648cb1ca863988a9c" +checksum = "88cf36502b64a127dc659e3b305f1d993a544eab0d48cce704424e62074dc04b" dependencies = [ "arrow-array", "arrow-buffer", @@ -318,7 +319,7 @@ dependencies = [ "arrow-schema", "chrono", "half", - "indexmap 2.11.0", + "indexmap 2.12.0", "lexical-core", "memchr", "num", @@ -329,9 +330,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6506e3a059e3be23023f587f79c82ef0bcf6d293587e3272d20f2d30b969b5a7" +checksum = "3c8f82583eb4f8d84d4ee55fd1cb306720cddead7596edce95b50ee418edf66f" dependencies = [ "arrow-array", "arrow-buffer", @@ -342,9 +343,9 @@ dependencies = [ [[package]] name = "arrow-pyarrow" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e55ecf16b9b61d433f6e63c72fc6afcf2597d7db96583de88ebb887d1822268" +checksum = "7d924b32e96f8bb74d94cd82bd97b313c432fcb0ea331689ef9e7c6b8be4b258" dependencies = [ "arrow-array", "arrow-data", @@ -354,9 +355,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52bf7393166beaf79b4bed9bfdf19e97472af32ce5b6b48169d321518a08cae2" +checksum = "9d07ba24522229d9085031df6b94605e0f4b26e099fb7cdeec37abd941a73753" dependencies = [ "arrow-array", "arrow-buffer", @@ -367,9 +368,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af7686986a3bf2254c9fb130c623cdcb2f8e1f15763e7c71c310f0834da3d292" +checksum = "b3aa9e59c611ebc291c28582077ef25c97f1975383f1479b12f3b9ffee2ffabe" dependencies = [ "bitflags", "serde", @@ -378,9 +379,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dd2b45757d6a2373faa3352d02ff5b54b098f5e21dccebc45a21806bc34501e5" +checksum = "8c41dbbd1e97bfcaee4fcb30e29105fb2c75e4d82ae4de70b792a5d3f66b2e7a" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -392,9 +393,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0377d532850babb4d927a06294314b316e23311503ed580ec6ce6a0158f49d40" +checksum = "53f5183c150fbc619eede22b861ea7c0eebed8eaac0333eaa7f6da5205fd504d" dependencies = [ "arrow-array", "arrow-buffer", @@ -470,7 +471,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -496,30 +497,15 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "backon" -version = "1.5.2" +version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "592277618714fbcecda9a02ba7a8781f319d26532a88553bbacc77ba5d2b3a8d" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" dependencies = [ "fastrand", "gloo-timers", "tokio", ] -[[package]] -name = "backtrace" -version = "0.3.75" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6806a6321ec58106fea15becdad98371e28d92ccbc7c8f1b3b6dd724fe8f1002" -dependencies = [ - "addr2line", - "cfg-if", - "libc", - "miniz_oxide", - "object", - "rustc-demangle", - "windows-targets 0.52.6", -] - [[package]] name = "base64" version = "0.22.1" @@ -528,9 +514,9 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bigdecimal" -version = "0.4.8" +version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a22f228ab7a1b23027ccc6c350b72868017af7ea8356fbdf19f8d991c690013" +checksum = "560f42649de9fa436b73517378a147ec21f6c997a546581df4b4b31677828934" dependencies = [ "autocfg", "libm", @@ -548,9 +534,9 @@ checksum = "230c5f1ca6a325a32553f8640d31ac9b49f2411e901e427570154868b46da4f7" [[package]] name = "bitflags" -version = "2.9.4" +version = "2.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2261d10cca569e4643e526d8dc2e62e433cc8aba21ab764233731f8d369bf394" +checksum = "812e12b5285cc515a9c72a5c1d3b6d46a19dac5acfef5265968c166106e31dd3" [[package]] name = "bitvec" @@ -597,9 +583,9 @@ dependencies = [ [[package]] name = "bon" -version = "3.7.2" +version = "3.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2529c31017402be841eb45892278a6c21a000c0a17643af326c73a73f83f0fb" +checksum = "ebeb9aaf9329dff6ceb65c689ca3db33dbf15f324909c60e4e5eef5701ce31b1" dependencies = [ "bon-macros", "rustversion", @@ -607,9 +593,9 @@ dependencies = [ [[package]] name = "bon-macros" -version = "3.7.2" +version = "3.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d82020dadcb845a345591863adb65d74fa8dc5c18a0b6d408470e13b7adc7005" +checksum = "77e9d642a7e3a318e37c2c9427b5a6a48aa1ad55dcd986f3034ab2239045a645" dependencies = [ "darling 0.21.3", "ident_case", @@ -617,7 +603,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -640,7 +626,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -694,9 +680,9 @@ dependencies = [ [[package]] name = "bytemuck" -version = "1.23.2" +version = "1.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3995eaeebcdf32f91f980d360f78732ddc061097ab4e39991ae7a6ace9194677" +checksum = "1fbdf580320f38b612e485521afda1ee26d10cc9884efaaa750d383e13e3c5f4" [[package]] name = "byteorder" @@ -721,9 +707,9 @@ dependencies = [ [[package]] name = "bzip2" -version = "0.6.0" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bea8dcd42434048e4f7a304411d9273a411f647446c1234a65ce0554923f4cff" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" dependencies = [ "libbz2-rs-sys", ] @@ -740,9 +726,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.36" +version = "1.2.43" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5252b3d2648e5eedbc1a6f501e3c795e07025c1e93bbf8bbdd6eef7f447a6d54" +checksum = "739eb0f94557554b3ca9a86d2d37bebd49c5e6d0c1d2bda35ba5bdac830befc2" dependencies = [ "find-msvc-tools", "jobserver", @@ -752,9 +738,9 @@ dependencies = [ [[package]] name = "cfg-if" -version = "1.0.3" +version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2fd1289c04a9ea8cb22300a459a72a385d7c73d3259e2ed7dcb2af674838cfa9" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] name = "cfg_aliases" @@ -773,7 +759,7 @@ dependencies = [ "num-traits", "serde", "wasm-bindgen", - "windows-link 0.2.0", + "windows-link", ] [[package]] @@ -788,11 +774,12 @@ dependencies = [ [[package]] name = "comfy-table" -version = "7.2.0" +version = "7.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f8e18d0dca9578507f13f9803add0df13362b02c501c1c17734f0dbb52eaf0b" +checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56" dependencies = [ - "unicode-segmentation", + "strum 0.26.3", + "strum_macros 0.26.4", "unicode-width", ] @@ -833,9 +820,9 @@ dependencies = [ [[package]] name = "const_panic" -version = "0.2.14" +version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb8a602185c3c95b52f86dc78e55a6df9a287a7a93ddbcf012509930880cf879" +checksum = "e262cdaac42494e3ae34c43969f9cdeb7da178bdb4b66fa6a1ea2edb4c8ae652" dependencies = [ "typewit", ] @@ -936,21 +923,21 @@ dependencies = [ [[package]] name = "csv" -version = "1.3.1" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "acdc4883a9c96732e4733212c01447ebd805833b7275a73ca3ee080fd77afdaf" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" dependencies = [ "csv-core", "itoa", "ryu", - "serde", + "serde_core", ] [[package]] name = "csv-core" -version = "0.1.12" +version = "0.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d02f3b0da4c6504f86e9cd789d8dbafab48c2321be74e9987593de5a894d93d" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" dependencies = [ "memchr", ] @@ -986,7 +973,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1000,7 +987,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1011,7 +998,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1022,7 +1009,7 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1041,16 +1028,16 @@ dependencies = [ [[package]] name = "datafusion" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69dfeda1633bf8ec75b068d9f6c27cdc392ffcf5ff83128d5dbab65b73c1fd02" +checksum = "2af15bb3c6ffa33011ef579f6b0bcbe7c26584688bd6c994f548e44df67f011a" dependencies = [ "arrow", "arrow-ipc", "arrow-schema", "async-trait", "bytes", - "bzip2 0.6.0", + "bzip2 0.6.1", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1070,6 +1057,7 @@ dependencies = [ "datafusion-functions-window", "datafusion-optimizer", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", @@ -1077,7 +1065,6 @@ dependencies = [ "datafusion-sql", "flate2", "futures", - "hex", "itertools 0.14.0", "log", "object_store", @@ -1096,9 +1083,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2848fd1e85e2953116dab9cc2eb109214b0888d7bbd2230e30c07f1794f642c0" +checksum = "187622262ad8f7d16d3be9202b4c1e0116f1c9aa387e5074245538b755261621" dependencies = [ "arrow", "async-trait", @@ -1122,9 +1109,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "051a1634628c2d1296d4e326823e7536640d87a118966cdaff069b68821ad53b" +checksum = "9657314f0a32efd0382b9a46fdeb2d233273ece64baa68a7c45f5a192daf0f83" dependencies = [ "arrow", "async-trait", @@ -1145,9 +1132,9 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "765e4ad4ef7a4500e389a3f1e738791b71ff4c29fd00912c2f541d62b25da096" +checksum = "5a83760d9a13122d025fbdb1d5d5aaf93dd9ada5e90ea229add92aa30898b2d1" dependencies = [ "ahash 0.8.12", "arrow", @@ -1156,8 +1143,7 @@ dependencies = [ "chrono", "half", "hashbrown 0.14.5", - "hex", - "indexmap 2.11.0", + "indexmap 2.12.0", "libc", "log", "object_store", @@ -1171,9 +1157,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40a2ae8393051ce25d232a6065c4558ab5a535c9637d5373bacfd464ac88ea12" +checksum = "5b6234a6c7173fe5db1c6c35c01a12b2aa0f803a3007feee53483218817f8b1e" dependencies = [ "futures", "log", @@ -1182,21 +1168,22 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90cd841a77f378bc1a5c4a1c37345e1885a9203b008203f9f4b3a769729bf330" +checksum = "7256c9cb27a78709dd42d0c80f0178494637209cac6e29d5c93edd09b6721b86" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2 0.6.0", + "bzip2 0.6.1", "chrono", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", @@ -1218,9 +1205,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77f4a2c64939c6f0dd15b246723a699fa30d59d0133eb36a86e8ff8c6e2a8dc6" +checksum = "64533a90f78e1684bfb113d200b540f18f268134622d7c96bbebc91354d04825" dependencies = [ "arrow", "async-trait", @@ -1243,9 +1230,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11387aaf931b2993ad9273c63ddca33f05aef7d02df9b70fb757429b4b71cdae" +checksum = "8d7ebeb12c77df0aacad26f21b0d033aeede423a64b2b352f53048a75bf1d6e6" dependencies = [ "arrow", "async-trait", @@ -1268,9 +1255,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "028f430c5185120bf806347848b8d8acd9823f4038875b3820eeefa35f2bb4a2" +checksum = "09e783c4c7d7faa1199af2df4761c68530634521b176a8d1331ddbc5a5c75133" dependencies = [ "arrow", "async-trait", @@ -1283,13 +1270,13 @@ dependencies = [ "datafusion-expr", "datafusion-functions-aggregate", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-pruning", "datafusion-session", "futures", - "hex", "itertools 0.14.0", "log", "object_store", @@ -1301,17 +1288,18 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff336d1d755399753a9e4fbab001180e346fc8bfa063a97f1214b82274c00f8" +checksum = "99ee6b1d9a80d13f9deb2291f45c07044b8e62fb540dbde2453a18be17a36429" [[package]] name = "datafusion-execution" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "042ea192757d1b2d7dcf71643e7ff33f6542c7704f00228d8b85b40003fd8e0f" +checksum = "a4cec0a57653bec7b933fb248d3ffa3fa3ab3bd33bd140dc917f714ac036f531" dependencies = [ "arrow", + "async-trait", "dashmap", "datafusion-common", "datafusion-expr", @@ -1326,9 +1314,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "025222545d6d7fab71e2ae2b356526a1df67a2872222cbae7535e557a42abd2e" +checksum = "ef76910bdca909722586389156d0aa4da4020e1631994d50fadd8ad4b1aa05fe" dependencies = [ "arrow", "async-trait", @@ -1339,7 +1327,7 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", - "indexmap 2.11.0", + "indexmap 2.12.0", "paste", "recursive", "serde_json", @@ -1348,22 +1336,22 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d5c267104849d5fa6d81cf5ba88f35ecd58727729c5eb84066c25227b644ae2" +checksum = "6d155ccbda29591ca71a1344dd6bed26c65a4438072b400df9db59447f590bb6" dependencies = [ "arrow", "datafusion-common", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "paste", ] [[package]] name = "datafusion-ffi" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec21805d9df2d834e4c6ddfbf8a1bed2bd460b89b01686fe0dcd1cee06d0b60f" +checksum = "25ddb7c4e645df080c27dad13a198d191da328dd1c98e198664a7a0f64b335cc" dependencies = [ "abi_stable", "arrow", @@ -1383,9 +1371,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c620d105aa208fcee45c588765483314eb415f5571cfd6c1bae3a59c5b4d15bb" +checksum = "7de2782136bd6014670fd84fe3b0ca3b3e4106c96403c3ae05c0598577139977" dependencies = [ "arrow", "arrow-buffer", @@ -1412,9 +1400,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35f61d5198a35ed368bf3aacac74f0d0fa33de7a7cb0c57e9f68ab1346d2f952" +checksum = "07331fc13603a9da97b74fd8a273f4238222943dffdbbed1c4c6f862a30105bf" dependencies = [ "ahash 0.8.12", "arrow", @@ -1433,9 +1421,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13efdb17362be39b5024f6da0d977ffe49c0212929ec36eec550e07e2bc7812f" +checksum = "b5951e572a8610b89968a09b5420515a121fbc305c0258651f318dc07c97ab17" dependencies = [ "ahash 0.8.12", "arrow", @@ -1446,9 +1434,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9187678af567d7c9e004b72a0b6dc5b0a00ebf4901cb3511ed2db4effe092e66" +checksum = "fdacca9302c3d8fc03f3e94f338767e786a88a33f5ebad6ffc0e7b50364b9ea3" dependencies = [ "arrow", "arrow-ord", @@ -1468,9 +1456,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecf156589cc21ef59fe39c7a9a841b4a97394549643bbfa88cc44e8588cf8fe5" +checksum = "8c37ff8a99434fbbad604a7e0669717c58c7c4f14c472d45067c4b016621d981" dependencies = [ "arrow", "async-trait", @@ -1484,9 +1472,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edcb25e3e369f1366ec9a261456e45b5aad6ea1c0c8b4ce546587207c501ed9e" +checksum = "48e2aea7c79c926cffabb13dc27309d4eaeb130f4a21c8ba91cdd241c813652b" dependencies = [ "arrow", "datafusion-common", @@ -1502,9 +1490,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8996a8e11174d0bd7c62dc2f316485affc6ae5ffd5b8a68b508137ace2310294" +checksum = "0fead257ab5fd2ffc3b40fda64da307e20de0040fe43d49197241d9de82a487f" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1512,20 +1500,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95ee8d1be549eb7316f437035f2cec7ec42aba8374096d807c4de006a3b5d78a" +checksum = "ec6f637bce95efac05cdfb9b6c19579ed4aa5f6b94d951cfa5bb054b7bb4f730" dependencies = [ "datafusion-expr", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "datafusion-optimizer" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9fa98671458254928af854e5f6c915e66b860a8bde505baea0ff2892deab74d" +checksum = "c6583ef666ae000a613a837e69e456681a9faa96347bf3877661e9e89e141d8a" dependencies = [ "arrow", "chrono", @@ -1533,7 +1521,7 @@ dependencies = [ "datafusion-expr", "datafusion-expr-common", "datafusion-physical-expr", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", "recursive", @@ -1543,9 +1531,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3515d51531cca5f7b5a6f3ea22742b71bb36fc378b465df124ff9a2fa349b002" +checksum = "c8668103361a272cbbe3a61f72eca60c9b7c706e87cc3565bcf21e2b277b84f6" dependencies = [ "ahash 0.8.12", "arrow", @@ -1556,18 +1544,34 @@ dependencies = [ "datafusion-physical-expr-common", "half", "hashbrown 0.14.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", + "parking_lot", "paste", "petgraph", ] +[[package]] +name = "datafusion-physical-expr-adapter" +version = "50.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "815acced725d30601b397e39958e0e55630e0a10d66ef7769c14ae6597298bb0" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "itertools 0.14.0", +] + [[package]] name = "datafusion-physical-expr-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24485475d9c618a1d33b2a3dad003d946dc7a7bbf0354d125301abc0a5a79e3e" +checksum = "6652fe7b5bf87e85ed175f571745305565da2c0b599d98e697bcbedc7baa47c3" dependencies = [ "ahash 0.8.12", "arrow", @@ -1579,9 +1583,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9da411a0a64702f941a12af2b979434d14ec5d36c6f49296966b2c7639cbb3a" +checksum = "49b7d623eb6162a3332b564a0907ba00895c505d101b99af78345f1acf929b5c" dependencies = [ "arrow", "datafusion-common", @@ -1599,9 +1603,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6d168282bb7b54880bb3159f89b51c047db4287f5014d60c3ef4c6e1468212b" +checksum = "e2f7f778a1a838dec124efb96eae6144237d546945587557c9e6936b3414558c" dependencies = [ "ahash 0.8.12", "arrow", @@ -1613,13 +1617,14 @@ dependencies = [ "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", + "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "futures", "half", "hashbrown 0.14.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "itertools 0.14.0", "log", "parking_lot", @@ -1629,9 +1634,9 @@ dependencies = [ [[package]] name = "datafusion-proto" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b36a0c84f4500efd90487a004b533bd81de1f2bb3f143f71b7526f33b85d2e2" +checksum = "a7df9f606892e6af45763d94d210634eec69b9bb6ced5353381682ff090028a3" dependencies = [ "arrow", "chrono", @@ -1645,9 +1650,9 @@ dependencies = [ [[package]] name = "datafusion-proto-common" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ec788be522806740ad6372c0a2f7e45fb37cb37f786d9b77933add49cdd058f" +checksum = "b4b14f288ca4ef77743d9672cafecf3adfffff0b9b04af9af79ecbeaaf736901" dependencies = [ "arrow", "datafusion-common", @@ -1656,9 +1661,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "49.0.0" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "391a457b9d23744c53eeb89edd1027424cba100581488d89800ed841182df905" +checksum = "cd1e59e2ca14fe3c30f141600b10ad8815e2856caa59ebbd0e3e07cd3d127a65" dependencies = [ "arrow", "arrow-schema", @@ -1674,9 +1679,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053201c2bb729c7938f85879034df2b5a52cfaba16f1b3b66ab8505c81b2aad3" +checksum = "21ef8e2745583619bd7a49474e8f45fbe98ebb31a133f27802217125a7b3d58d" dependencies = [ "arrow", "async-trait", @@ -1698,15 +1703,15 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "49.0.2" +version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9082779be8ce4882189b229c0cff4393bd0808282a7194130c9f32159f185e25" +checksum = "89abd9868770386fede29e5a4b14f49c0bf48d652c3b9d7a8a0332329b87d50b" dependencies = [ "arrow", "bigdecimal", "datafusion-common", "datafusion-expr", - "indexmap 2.11.0", + "indexmap 2.12.0", "log", "recursive", "regex", @@ -1715,12 +1720,12 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.3" +version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d630bccd429a5bb5a64b5e94f693bfc48c9f8566418fda4c494cc94f911f87cc" +checksum = "ececcb659e7ba858fb4f10388c250a7252eb0a27373f1a72b8748afdd248e587" dependencies = [ "powerfmt", - "serde", + "serde_core", ] [[package]] @@ -1741,7 +1746,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1751,7 +1756,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1774,7 +1779,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -1812,12 +1817,12 @@ checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" [[package]] name = "errno" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "778e2ac28f6c47af28e4907f13ffd1e1ddbd400980a9abd7c8df189bf578a5ad" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1859,9 +1864,9 @@ checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" [[package]] name = "find-msvc-tools" -version = "0.1.1" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7fd99930f64d146689264c637b5af2f0233a933bef0d8570e2526bf9e083192d" +checksum = "52051878f80a721bb68ebfbc930e07b65ba72f2da88968ea5c06fd6ca3d3a127" [[package]] name = "fixedbitset" @@ -1871,9 +1876,9 @@ checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" [[package]] name = "flatbuffers" -version = "25.2.10" +version = "25.9.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1045398c1bfd89168b5fd3f1fc11f6e70b34f6f66300c87d44d3de849463abf1" +checksum = "09b6620799e7340ebd9968d2e0708eb82cf1971e9a16821e2091b6d6e475eed5" dependencies = [ "bitflags", "rustc_version", @@ -1881,9 +1886,9 @@ dependencies = [ [[package]] name = "flate2" -version = "1.1.2" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a3d7db9596fecd151c5f638c0ee5d5bd487b6e0ea232e5dc96d5250f6f94b1d" +checksum = "bfe33edd8e85a12a67454e37f8c75e730830d83e313556ab9ebf9ee7fbeb3bfb" dependencies = [ "crc32fast", "libz-rs-sys", @@ -1973,7 +1978,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -2015,25 +2020,11 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "generator" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "605183a538e3e2a9c1038635cc5c2d194e2ee8fd0d1b66b8349fad7dbacce5a2" -dependencies = [ - "cc", - "cfg-if", - "libc", - "log", - "rustversion", - "windows", -] - [[package]] name = "generic-array" -version = "0.14.7" +version = "0.14.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2" dependencies = [ "typenum", "version_check", @@ -2048,30 +2039,24 @@ dependencies = [ "cfg-if", "js-sys", "libc", - "wasi 0.11.1+wasi-snapshot-preview1", + "wasi", "wasm-bindgen", ] [[package]] name = "getrandom" -version = "0.3.3" +version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26145e563e54f2cadc477553f1ec5ee650b00862f0a58bcd12cbdc5f0ea2d2f4" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", "js-sys", "libc", "r-efi", - "wasi 0.14.4+wasi-0.2.4", + "wasip2", "wasm-bindgen", ] -[[package]] -name = "gimli" -version = "0.31.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f" - [[package]] name = "glob" version = "0.3.3" @@ -2092,13 +2077,14 @@ dependencies = [ [[package]] name = "half" -version = "2.6.0" +version = "2.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "459196ed295495a68f7d7fe1d84f6c4b7ff0e21fe3017b2f283c6fac3ad803c9" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" dependencies = [ "cfg-if", "crunchy", "num-traits", + "zerocopy", ] [[package]] @@ -2126,11 +2112,15 @@ version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ - "allocator-api2", - "equivalent", "foldhash", ] +[[package]] +name = "hashbrown" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d" + [[package]] name = "heck" version = "0.5.0" @@ -2209,9 +2199,9 @@ checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" [[package]] name = "humantime" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b112acc8b3adf4b107a8ec20977da0273a8c386765a3ec0229bd500a1443f9f" +checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" [[package]] name = "hyper" @@ -2253,9 +2243,9 @@ dependencies = [ [[package]] name = "hyper-util" -version = "0.1.16" +version = "0.1.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d9b05277c7e8da2c93a568989bb6207bef0112e8d17df7a6eda4a3cf143bc5e" +checksum = "3c6995591a8f1380fcb4ba966a252a4b29188d51d2b89e3a252f5305be65aea8" dependencies = [ "base64", "bytes", @@ -2277,9 +2267,9 @@ dependencies = [ [[package]] name = "iana-time-zone" -version = "0.1.63" +version = "0.1.64" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0c919e5debc312ad217002b8048a17b7d83f80703865bbfcfebb0458b0b27d8" +checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb" dependencies = [ "android_system_properties", "core-foundation-sys", @@ -2344,7 +2334,7 @@ dependencies = [ "serde_json", "serde_repr", "serde_with", - "strum", + "strum 0.27.2", "thrift", "tokio", "typed-builder", @@ -2369,9 +2359,9 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "200072f5d0e3614556f94a9930d5dc3e0662a652823904c3a75dc3b0af7fee47" +checksum = "f578a71f2bfaf7ceb30b519a645ae48024b45f9eecbe060a31a004d7b4ba9462" dependencies = [ "displaydoc", "potential_utf", @@ -2382,9 +2372,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cde2700ccaed3872079a65fb1a78f6c0a36c91570f28755dda67bc8f7d9f00a" +checksum = "4c219b62bf5a06801012446193fdfcbd7970e876823aba4c62def2ce957dcb44" dependencies = [ "displaydoc", "litemap", @@ -2395,11 +2385,10 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "436880e8e18df4d7bbc06d58432329d6458cc84531f7ac5f024e93deadb37979" +checksum = "33747cecc725eebb47ac503fab725e395d50cb7889ae490a1359f130611d4cc5" dependencies = [ - "displaydoc", "icu_collections", "icu_normalizer_data", "icu_properties", @@ -2410,42 +2399,38 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00210d6893afc98edb752b664b8890f0ef174c8adbb8d0be9710fa66fbbf72d3" +checksum = "d6ce2d23e1b3c45624ba6a23e2c767e01c9680e0c0800b39c7abfff9565175d8" [[package]] name = "icu_properties" -version = "2.0.1" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "016c619c1eeb94efb86809b015c58f479963de65bdb6253345c1a1276f22e32b" +checksum = "6d70f9b6574c79f7a83ea5ce72cc88d271a3e77355c5f7748a107e751d8617fb" dependencies = [ - "displaydoc", "icu_collections", "icu_locale_core", "icu_properties_data", "icu_provider", - "potential_utf", "zerotrie", "zerovec", ] [[package]] name = "icu_properties_data" -version = "2.0.1" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "298459143998310acd25ffe6810ed544932242d3f07083eee1084d83a71bd632" +checksum = "17fa55bf868e28e638ed132bcee1e5c21ba2c1e52c15e7c78b781858e7b54342" [[package]] name = "icu_provider" -version = "2.0.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03c80da27b5f4187909049ee2d72f276f0d9f99a42c306bd0131ecfe04d8e5af" +checksum = "f64958e359123591ae1f17a27b5fc9ebdb50c98b04e0401146154de1d8fe3e44" dependencies = [ "displaydoc", "icu_locale_core", - "stable_deref_trait", - "tinystr", "writeable", "yoke", "zerofrom", @@ -2493,20 +2478,24 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.11.0" +version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2481980430f9f78649238835720ddccc57e52df14ffce1c6f37391d61b563e9" +checksum = "6717a8d2a5a929a1a2eb43a12812498ed141a0bcfb7e8f7844fbdbe4303bba9f" dependencies = [ "equivalent", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "serde", + "serde_core", ] [[package]] name = "indoc" -version = "2.0.6" +version = "2.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4c7245a08504955605670dbf141fceab975f15ca21570696aebe9d2e71576bd" +checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706" +dependencies = [ + "rustversion", +] [[package]] name = "integer-encoding" @@ -2514,17 +2503,6 @@ version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" -[[package]] -name = "io-uring" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "046fa2d4d00aea763528b4950358d0ead425372445dc8ff86312b3c69ff7727b" -dependencies = [ - "bitflags", - "cfg-if", - "libc", -] - [[package]] name = "ipnet" version = "2.11.0" @@ -2571,31 +2549,25 @@ version = "0.1.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", "libc", ] [[package]] name = "js-sys" -version = "0.3.78" +version = "0.3.82" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0b063578492ceec17683ef2f8c5e89121fbd0b172cbc280635ab7567db2738" +checksum = "b011eec8cc36da2aab2d5cff675ec18454fad408585853910a202391cf9f8e65" dependencies = [ "once_cell", "wasm-bindgen", ] -[[package]] -name = "lazy_static" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" - [[package]] name = "lexical-core" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b765c31809609075565a70b4b71402281283aeda7ecaf4818ac14a7b2ade8958" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" dependencies = [ "lexical-parse-float", "lexical-parse-integer", @@ -2606,53 +2578,46 @@ dependencies = [ [[package]] name = "lexical-parse-float" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de6f9cb01fb0b08060209a057c048fcbab8717b4c1ecd2eac66ebfe39a65b0f2" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" dependencies = [ "lexical-parse-integer", "lexical-util", - "static_assertions", ] [[package]] name = "lexical-parse-integer" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72207aae22fc0a121ba7b6d479e42cbfea549af1479c3f3a4f12c70dd66df12e" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" dependencies = [ "lexical-util", - "static_assertions", ] [[package]] name = "lexical-util" -version = "1.0.6" +version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a82e24bf537fd24c177ffbbdc6ebcc8d54732c35b50a3f28cc3f4e4c949a0b3" -dependencies = [ - "static_assertions", -] +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" [[package]] name = "lexical-write-float" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5afc668a27f460fb45a81a757b6bf2f43c2d7e30cb5a2dcd3abf294c78d62bd" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" dependencies = [ "lexical-util", "lexical-write-integer", - "static_assertions", ] [[package]] name = "lexical-write-integer" -version = "1.0.5" +version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "629ddff1a914a836fb245616a7888b62903aae58fa771e1d83943035efa0f978" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" dependencies = [ "lexical-util", - "static_assertions", ] [[package]] @@ -2663,9 +2628,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" [[package]] name = "libc" -version = "0.2.175" +version = "0.2.177" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a82ae493e598baaea5209805c49bbf2ea7de956d50d7da0da1164f9c6d28543" +checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976" [[package]] name = "libloading" @@ -2694,23 +2659,22 @@ dependencies = [ [[package]] name = "linux-raw-sys" -version = "0.9.4" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd945864f07fe9f5371a27ad7b52a172b4b499999f1d97574c9fa68373937e12" +checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039" [[package]] name = "litemap" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "241eaef5fd12c88705a01fc1066c48c4b36e0dd4377dcdc7ec3942cea7a69956" +checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" [[package]] name = "lock_api" -version = "0.4.13" +version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96936507f153605bddfcda068dd804796c84324ed2510809e5b2a624c81da765" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" dependencies = [ - "autocfg", "scopeguard", ] @@ -2720,19 +2684,6 @@ version = "0.4.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432" -[[package]] -name = "loom" -version = "0.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca" -dependencies = [ - "cfg-if", - "generator", - "scoped-tls", - "tracing", - "tracing-subscriber", -] - [[package]] name = "lru-slab" version = "0.1.2" @@ -2759,15 +2710,6 @@ dependencies = [ "pkg-config", ] -[[package]] -name = "matchers" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" -dependencies = [ - "regex-automata", -] - [[package]] name = "md-5" version = "0.10.6" @@ -2780,9 +2722,9 @@ dependencies = [ [[package]] name = "memchr" -version = "2.7.5" +version = "2.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a282da65faaf38286cf3be983213fcf1d2e2a58700e808f83f4ea9a4804bc0" +checksum = "f52b00d39961fc5b2736ea853c9cc86238e165017a493d1d5c8eac6bdc4cc273" [[package]] name = "memoffset" @@ -2800,38 +2742,38 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" dependencies = [ "adler2", + "simd-adler32", ] [[package]] name = "mio" -version = "1.0.4" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78bed444cc8a2160f01cbcf811ef18cac863ad68ae8ca62092e8db51d51c761c" +checksum = "69d83b0086dc8ecf3ce9ae2874b2d1290252e2a30720bea58a5c6639b0092873" dependencies = [ "libc", - "wasi 0.11.1+wasi-snapshot-preview1", - "windows-sys 0.59.0", + "wasi", + "windows-sys 0.61.2", ] [[package]] name = "moka" -version = "0.12.10" +version = "0.12.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9321642ca94a4282428e6ea4af8cc2ca4eac48ac7a6a4ea8f33f76d0ce70926" +checksum = "8261cd88c312e0004c1d51baad2980c66528dfdb2bee62003e643a4d8f86b077" dependencies = [ "async-lock", "crossbeam-channel", "crossbeam-epoch", "crossbeam-utils", + "equivalent", "event-listener", "futures-util", - "loom", "parking_lot", "portable-atomic", "rustc_version", "smallvec", "tagptr", - "thiserror 1.0.69", "uuid", ] @@ -2841,15 +2783,6 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9252111cf132ba0929b6f8e030cac2a24b507f3a4d6db6fb2896f27b354c714b" -[[package]] -name = "nu-ansi-term" -version = "0.50.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4a28e057d01f97e61255210fcff094d74ed0466038633e95017f5beb68e4399" -dependencies = [ - "windows-sys 0.52.0", -] - [[package]] name = "num" version = "0.4.3" @@ -2943,18 +2876,18 @@ dependencies = [ [[package]] name = "object" -version = "0.36.7" +version = "0.32.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87" +checksum = "a6a622008b6e321afc04970976f62ee297fdbaa6f95318ca343e3eebb9648441" dependencies = [ "memchr", ] [[package]] name = "object_store" -version = "0.12.3" +version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efc4f07659e11cd45a341cd24d71e683e3be65d9ff1f8150061678fe60437496" +checksum = "4c1be0c6c22ec0817cdc77d3842f721a17fd30ab6965001415b5402a74e6b740" dependencies = [ "async-trait", "bytes", @@ -2965,7 +2898,7 @@ dependencies = [ "itertools 0.14.0", "parking_lot", "percent-encoding", - "thiserror 2.0.16", + "thiserror", "tokio", "tracing", "url", @@ -2982,9 +2915,9 @@ checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" [[package]] name = "opendal" -version = "0.54.0" +version = "0.54.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffb9838d0575c6dbaf3fcec7255af8d5771996d4af900bbb6fa9a314dec00a1a" +checksum = "42afda58fa2cf50914402d132cc1caacff116a85d10c72ab2082bb7c50021754" dependencies = [ "anyhow", "backon", @@ -2999,7 +2932,7 @@ dependencies = [ "log", "md-5", "percent-encoding", - "quick-xml", + "quick-xml 0.38.3", "reqsign", "reqwest", "serde", @@ -3044,9 +2977,9 @@ checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" [[package]] name = "parking_lot" -version = "0.12.4" +version = "0.12.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70d58bf43669b5795d1576d0641cfb6fbb2057bf629506267a92807158584a13" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" dependencies = [ "lock_api", "parking_lot_core", @@ -3054,22 +2987,22 @@ dependencies = [ [[package]] name = "parking_lot_core" -version = "0.9.11" +version = "0.9.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc838d2a56b5b1a6c25f55575dfc605fabb63bb2365f6c2353ef9159aa69e4a5" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" dependencies = [ "cfg-if", "libc", "redox_syscall", "smallvec", - "windows-targets 0.52.6", + "windows-link", ] [[package]] name = "parquet" -version = "55.2.0" +version = "56.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b17da4150748086bd43352bc77372efa9b6e3dbd06a04831d2a98c041c225cfa" +checksum = "f0dbd48ad52d7dccf8ea1b90a3ddbfaea4f69878dd7683e51c507d4bc52b5b27" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -3086,7 +3019,7 @@ dependencies = [ "flate2", "futures", "half", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "lz4_flex", "num", "num-bigint", @@ -3116,13 +3049,13 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "petgraph" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54acf3a685220b533e437e264e4d932cfbdc4cc7ec0cd232ed73c08d03b8a7ca" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset", "hashbrown 0.15.5", - "indexmap 2.11.0", + "indexmap 2.12.0", "serde", ] @@ -3170,9 +3103,9 @@ checksum = "f84267b20a16ea918e43c6a88433c2d54fa145c92a811b5b047ccbe153674483" [[package]] name = "potential_utf" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "84df19adbe5b5a0782edcab45899906947ab039ccf4573713735ee7de1e6b08a" +checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77" dependencies = [ "zerovec", ] @@ -3199,23 +3132,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "proc-macro-crate" -version = "3.3.0" +version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edce586971a4dfaa28950c6f18ed55e0406c1ab88bbce2c6f6293a7aaba73d35" +checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" dependencies = [ "toml_edit", ] [[package]] name = "proc-macro2" -version = "1.0.101" +version = "1.0.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de" +checksum = "5ee95bc4ef87b8d5ba32e8b7714ccc834865276eab0aed5c9958d00ec45f49e8" dependencies = [ "unicode-ident", ] @@ -3240,15 +3173,16 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "psm" -version = "0.1.26" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e944464ec8536cd1beb0bbfd96987eb5e3b72f2ecdafdc5c769a37f1fa2ae1f" +checksum = "d11f2fedc3b7dafdc2851bc52f277377c5473d378859be234bc7ebb593144d01" dependencies = [ + "ar_archive_writer", "cc", ] @@ -3286,11 +3220,10 @@ dependencies = [ [[package]] name = "pyo3" -version = "0.24.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5203598f366b11a02b13aa20cab591229ff0a89fd121a308a5df751d5fc9219" +checksum = "8970a78afe0628a3e3430376fc5fd76b6b45c4d43360ffd6cdd40bdde72b682a" dependencies = [ - "cfg-if", "indoc", "libc", "memoffset", @@ -3304,9 +3237,9 @@ dependencies = [ [[package]] name = "pyo3-build-config" -version = "0.24.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99636d423fa2ca130fa5acde3059308006d46f98caac629418e53f7ebb1e9999" +checksum = "458eb0c55e7ece017adeba38f2248ff3ac615e53660d7c71a238d7d2a01c7598" dependencies = [ "once_cell", "target-lexicon", @@ -3314,9 +3247,9 @@ dependencies = [ [[package]] name = "pyo3-ffi" -version = "0.24.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78f9cf92ba9c409279bc3305b5409d90db2d2c22392d443a87df3a1adad59e33" +checksum = "7114fe5457c61b276ab77c5055f206295b812608083644a5c5b2640c3102565c" dependencies = [ "libc", "pyo3-build-config", @@ -3324,27 +3257,27 @@ dependencies = [ [[package]] name = "pyo3-macros" -version = "0.24.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b999cb1a6ce21f9a6b147dcf1be9ffedf02e0043aec74dc390f3007047cecd9" +checksum = "a8725c0a622b374d6cb051d11a0983786448f7785336139c3c94f5aa6bef7e50" dependencies = [ "proc-macro2", "pyo3-macros-backend", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "pyo3-macros-backend" -version = "0.24.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "822ece1c7e1012745607d5cf0bcb2874769f0f7cb34c4cde03b9358eb9ef911a" +checksum = "4109984c22491085343c05b0dbc54ddc405c3cf7b4374fc533f5c3313a572ccc" dependencies = [ "heck", "proc-macro2", "pyo3-build-config", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -3363,6 +3296,16 @@ dependencies = [ "serde", ] +[[package]] +name = "quick-xml" +version = "0.38.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42a232e7487fc2ef313d96dde7948e7a3c05101870d8985e4fd8d26aedd27b89" +dependencies = [ + "memchr", + "serde", +] + [[package]] name = "quinn" version = "0.11.9" @@ -3377,7 +3320,7 @@ dependencies = [ "rustc-hash", "rustls", "socket2", - "thiserror 2.0.16", + "thiserror", "tokio", "tracing", "web-time", @@ -3390,7 +3333,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1906b49b0c3bc04b5fe5d86a77925ae6524a19b816ae38ce1e426255f1d8a31" dependencies = [ "bytes", - "getrandom 0.3.3", + "getrandom 0.3.4", "lru-slab", "rand 0.9.2", "ring", @@ -3398,7 +3341,7 @@ dependencies = [ "rustls", "rustls-pki-types", "slab", - "thiserror 2.0.16", + "thiserror", "tinyvec", "tracing", "web-time", @@ -3420,9 +3363,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.40" +version = "1.0.41" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1885c039570dc00dcb4ff087a89e185fd56bae234ddc7f056a945bf36467248d" +checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1" dependencies = [ "proc-macro2", ] @@ -3495,7 +3438,7 @@ version = "0.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "99d9a13982dcf210057a8a78572b2217b667c3beacbf3a0d8b454f6f82837d38" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", ] [[package]] @@ -3515,43 +3458,43 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "redox_syscall" -version = "0.5.17" +version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5407465600fb0548f1442edf71dd20683c6ed326200ace4b1ef0763521bb3b77" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ "bitflags", ] [[package]] name = "ref-cast" -version = "1.0.24" +version = "1.0.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a0ae411dbe946a674d89546582cea4ba2bb8defac896622d6496f14c23ba5cf" +checksum = "f354300ae66f76f1c85c5f84693f0ce81d747e2c3f21a45fef496d89c960bf7d" dependencies = [ "ref-cast-impl", ] [[package]] name = "ref-cast-impl" -version = "1.0.24" +version = "1.0.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1165225c21bff1f3bbce98f5a1f889949bc902d3575308cc7b0de30b4f6d27c7" +checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "regex" -version = "1.11.2" +version = "1.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23d7fd106d8c02486a8d64e778353d1cffe08ce79ac2e82f540c86d0facf6912" +checksum = "843bc0191f75f3e22651ae5f1e72939ab2f72a4bc30fa80a066bd66edefc24d4" dependencies = [ "aho-corasick", "memchr", @@ -3561,9 +3504,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.10" +version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b9458fa0bfeeac22b5ca447c63aaf45f28439a709ccd244698632f9aa6394d6" +checksum = "5276caf25ac86c8d810222b3dbb938e512c55c6831a10f3e6ed1c93b84041f1c" dependencies = [ "aho-corasick", "memchr", @@ -3572,15 +3515,15 @@ dependencies = [ [[package]] name = "regex-lite" -version = "0.1.7" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "943f41321c63ef1c92fd763bfe054d2668f7f225a5c29f0105903dc2fc04ba30" +checksum = "8d942b98df5e658f56f20d592c7f868833fe38115e65c33003d8cd224b0155da" [[package]] name = "regex-syntax" -version = "0.8.6" +version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "caf4aa5b0f434c91fe5c7f1ecb6a5ece2130b02ad2a590589dda5146df959001" +checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58" [[package]] name = "rend" @@ -3618,7 +3561,7 @@ dependencies = [ "http", "log", "percent-encoding", - "quick-xml", + "quick-xml 0.37.5", "rand 0.8.5", "reqwest", "rust-ini", @@ -3631,9 +3574,9 @@ dependencies = [ [[package]] name = "reqwest" -version = "0.12.23" +version = "0.12.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d429f34c8092b2d42c7c93cec323bb4adeb7c67698f70839adec842ec10c7ceb" +checksum = "9d0946410b9f7b082a427e4ef5c8ff541a88b357bc6c637c40db3a68ac70a36f" dependencies = [ "base64", "bytes", @@ -3735,9 +3678,9 @@ dependencies = [ [[package]] name = "rust_decimal" -version = "1.37.2" +version = "1.39.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b203a6425500a03e0919c42d3c47caca51e79f1132046626d2c8871c5092035d" +checksum = "35affe401787a9bd846712274d97654355d21b2a2c092a3139aabe31e9022282" dependencies = [ "arrayvec", "borsh", @@ -3749,12 +3692,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "rustc-demangle" -version = "0.1.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56f7d92ca342cea22a06f2121d944b4fd82af56988c270852495420f961d4ace" - [[package]] name = "rustc-hash" version = "2.1.1" @@ -3772,22 +3709,22 @@ dependencies = [ [[package]] name = "rustix" -version = "1.0.8" +version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11181fbabf243db407ef8df94a6ce0b2f9a733bd8be4ad02b4eda9602296cac8" +checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e" dependencies = [ "bitflags", "errno", "libc", "linux-raw-sys", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] name = "rustls" -version = "0.23.31" +version = "0.23.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0ebcbd2f03de0fc1122ad9bb24b127a5a6cd51d72604a3f3c50ac459762b6cc" +checksum = "6a9586e9ee2b4f8fab52a0048ca7334d7024eef48e2cb9407e3497bb7cab7fa7" dependencies = [ "once_cell", "ring", @@ -3799,9 +3736,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.12.0" +version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "229a4a4c221013e7e1f1a043678c5cc39fe5171437c88fb47151a21e6f5b5c79" +checksum = "94182ad936a0c91c324cd46c6511b9510ed16af436d7b5bab34beab0afd55f7a" dependencies = [ "web-time", "zeroize", @@ -3809,9 +3746,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.4" +version = "0.103.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a17884ae0c1b773f1ccd2bd4a8c72f16da897310a98b0e84bf349ad5ead92fc" +checksum = "e10b3f4191e8a80e6b43eebabfac91e5dcecebb27a71f04e820c47ec41d314bf" dependencies = [ "ring", "rustls-pki-types", @@ -3863,12 +3800,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "scoped-tls" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" - [[package]] name = "scopeguard" version = "1.2.0" @@ -3883,9 +3814,9 @@ checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b" [[package]] name = "semver" -version = "1.0.26" +version = "1.0.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56e6fa9c48d24d85fb3de5ad847117517440f6beceb7798af16b4a87d616b8d0" +checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" [[package]] name = "seq-macro" @@ -3895,43 +3826,55 @@ checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" [[package]] name = "serde" -version = "1.0.219" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f0e2c6ed6606019b4e29e69dbaba95b11854410e5347d525002456dbbb786b6" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" dependencies = [ + "serde_core", "serde_derive", ] [[package]] name = "serde_bytes" -version = "0.11.17" +version = "0.11.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8437fd221bde2d4ca316d61b90e337e9e702b3820b87d63caa9ba6c02bd06d96" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" dependencies = [ "serde", + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.219" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b0276cf7f2c73365f7157c8123c21cd9a50fbbd844757af28ca1f5925fc2a00" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "serde_json" -version = "1.0.143" +version = "1.0.145" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d401abef1d108fbd9cbaebc3e46611f4b1021f714a0597a71f41ee463f5f4a5a" +checksum = "402a6f66d8c709116cf22f558eab210f5a50187f702eb4d7e5ef38d9a7f1c79c" dependencies = [ "itoa", "memchr", "ryu", "serde", + "serde_core", ] [[package]] @@ -3942,7 +3885,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -3959,19 +3902,18 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.14.0" +version = "3.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2c45cd61fefa9db6f254525d46e392b852e0e61d9a1fd36e5bd183450a556d5" +checksum = "aa66c845eee442168b2c8134fec70ac50dc20e760769c8ba0ad1319ca1959b04" dependencies = [ "base64", "chrono", "hex", "indexmap 1.9.3", - "indexmap 2.11.0", + "indexmap 2.12.0", "schemars 0.9.0", "schemars 1.0.4", - "serde", - "serde_derive", + "serde_core", "serde_json", "serde_with_macros", "time", @@ -3979,14 +3921,14 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.14.0" +version = "3.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de90945e6565ce0d9a25098082ed4ee4002e047cb59892c318d66821e14bb30f" +checksum = "b91a903660542fced4e99881aa481bdbaec1634568ee02e0b8bd57c64cb38955" dependencies = [ - "darling 0.20.11", + "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -4011,21 +3953,18 @@ dependencies = [ "digest", ] -[[package]] -name = "sharded-slab" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" -dependencies = [ - "lazy_static", -] - [[package]] name = "shlex" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" +[[package]] +name = "simd-adler32" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d66dc143e6b11c1eddc06d5c423cfc97062865baf299914ab64caa38182078fe" + [[package]] name = "simdutf8" version = "0.1.5" @@ -4058,19 +3997,19 @@ checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" [[package]] name = "socket2" -version = "0.6.0" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "233504af464074f9d066d7b5416c5f9b894a5862a6506e306f7b816cdd6f1807" +checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] name = "sqlparser" -version = "0.55.0" +version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4521174166bac1ff04fe16ef4524c70144cd29682a45978978ca3d7f4e0be11" +checksum = "ec4b661c54b1e4b603b37873a18c59920e4c51ea8ea2cf527d925424dbd4437c" dependencies = [ "log", "recursive", @@ -4085,20 +4024,20 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "stable_deref_trait" -version = "1.2.0" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" -version = "0.1.21" +version = "0.1.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cddb07e32ddb770749da91081d8d0ac3a16f1a569a18b20348cd371f5dead06b" +checksum = "e1f8b29fb42aafcea4edeeb6b2f2d7ecd0d969c48b4cf0d2e64aafc471dd6e59" dependencies = [ "cc", "cfg-if", @@ -4107,25 +4046,38 @@ dependencies = [ "windows-sys 0.59.0", ] -[[package]] -name = "static_assertions" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" - [[package]] name = "strsim" version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" + [[package]] name = "strum" version = "0.27.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" dependencies = [ - "strum_macros", + "strum_macros 0.27.2", +] + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.108", ] [[package]] @@ -4137,7 +4089,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -4159,9 +4111,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.106" +version = "2.0.108" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ede7c438028d4436d71104916910f5bb611972c5cfd7f89b8300a8186e6fada6" +checksum = "da58917d35242480a05c2897064da0a80589a2a0476c9a3f2fdc83b53502e917" dependencies = [ "proc-macro2", "quote", @@ -4185,7 +4137,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -4202,70 +4154,41 @@ checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" [[package]] name = "target-lexicon" -version = "0.13.2" +version = "0.13.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e502f78cdbb8ba4718f566c418c52bc729126ffd16baee5baa718cf25dd5a69a" +checksum = "df7f62577c25e07834649fc3b39fafdc597c0a3527dc1c60129201ccfcbaa50c" [[package]] name = "tempfile" -version = "3.21.0" +version = "3.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15b61f8f20e3a6f7e0649d825294eaf317edce30f82cf6026e7e4cb9222a7d1e" +checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16" dependencies = [ "fastrand", - "getrandom 0.3.3", + "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] name = "thiserror" -version = "1.0.69" +version = "2.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8" dependencies = [ - "thiserror-impl 1.0.69", -] - -[[package]] -name = "thiserror" -version = "2.0.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3467d614147380f2e4e374161426ff399c91084acd2363eaf549172b3d5e60c0" -dependencies = [ - "thiserror-impl 2.0.16", + "thiserror-impl", ] [[package]] name = "thiserror-impl" -version = "1.0.69" +version = "2.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c5e1be1c48b9172ee610da68fd9cd2770e7a4056cb3fc98710ee6906f0c7960" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.106", -] - -[[package]] -name = "thread_local" -version = "1.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" -dependencies = [ - "cfg-if", + "syn 2.0.108", ] [[package]] @@ -4292,11 +4215,12 @@ dependencies = [ [[package]] name = "time" -version = "0.3.43" +version = "0.3.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83bde6f1ec10e72d583d91623c939f623002284ef622b87de38cfd546cbf2031" +checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" dependencies = [ "deranged", + "itoa", "num-conv", "powerfmt", "serde", @@ -4331,9 +4255,9 @@ dependencies = [ [[package]] name = "tinystr" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d4f6d1145dcb577acf783d4e601bc1d76a13337bb54e6233add580b07344c8b" +checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869" dependencies = [ "displaydoc", "zerovec", @@ -4356,38 +4280,35 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.47.1" +version = "1.48.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89e49afdadebb872d3145a5638b59eb0691ea23e46ca484037cfab3b76b95038" +checksum = "ff360e02eab121e0bc37a2d3b4d4dc622e6eda3a8e5253d5435ecf5bd4c68408" dependencies = [ - "backtrace", "bytes", - "io-uring", "libc", "mio", "pin-project-lite", - "slab", "socket2", "tokio-macros", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] name = "tokio-macros" -version = "2.5.0" +version = "2.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e06d43f1345a3bcd39f6a56dbb7dcab2ba47e68e8ac134855e7e2bdbaf8cab8" +checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "tokio-rustls" -version = "0.26.2" +version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e727b36a1a0e8b74c376ac2211e40c2c8af09fb4013c60d910495810f008e9b" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ "rustls", "tokio", @@ -4408,18 +4329,31 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "0.6.11" +version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" +checksum = "f2cdb639ebbc97961c51720f858597f7f24c4fc295327923af55b74c3c724533" +dependencies = [ + "serde_core", +] [[package]] name = "toml_edit" -version = "0.22.27" +version = "0.23.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" +checksum = "6485ef6d0d9b5d0ec17244ff7eb05310113c3f316f2d14200d4de56b3cb98f8d" dependencies = [ - "indexmap 2.11.0", + "indexmap 2.12.0", "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0cbe268d35bdb4bb5a56a2de88d0ad0eb70af5384a99d648cd4b3d04039800e" +dependencies = [ "winnow", ] @@ -4487,7 +4421,7 @@ checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -4497,36 +4431,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9d12581f227e93f094d3af2ae690a574abb8a2b9b7a96e7cfe9647b2b617678" dependencies = [ "once_cell", - "valuable", -] - -[[package]] -name = "tracing-log" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" -dependencies = [ - "log", - "once_cell", - "tracing-core", -] - -[[package]] -name = "tracing-subscriber" -version = "0.3.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2054a14f5307d601f88daf0553e1cbf472acc4f2c51afab632431cdcd72124d5" -dependencies = [ - "matchers", - "nu-ansi-term", - "once_cell", - "regex-automata", - "sharded-slab", - "smallvec", - "thread_local", - "tracing", - "tracing-core", - "tracing-log", ] [[package]] @@ -4579,26 +4483,26 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "typenum" -version = "1.18.0" +version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1dccffe3ce07af9386bfd29e80c0ab1a8205a2fc34e4bcd40364df902cfa8f3f" +checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" [[package]] name = "typewit" -version = "1.14.1" +version = "1.14.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c98488b93df24b7c794d6a58c4198d7a2abde676324beaca84f7fb5b39c0811" +checksum = "f8c1ae7cc0fdb8b842d65d127cb981574b0d2b249b74d1c7a2986863dc134f71" [[package]] name = "unicode-ident" -version = "1.0.18" +version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a5f39404a5da50712a4c1eecf25e90dd62b613502b7e925fd4e4d19b5c96512" +checksum = "462eeb75aeb73aea900253ce739c8e18a67423fadf006037cd3ff27e82748a06" [[package]] name = "unicode-segmentation" @@ -4608,9 +4512,9 @@ checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" [[package]] name = "unicode-width" -version = "0.2.1" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1a07cc7db3810833284e8d372ccdc6da29741639ecc70c9ec107df0fa6154c" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" [[package]] name = "unindent" @@ -4648,18 +4552,12 @@ version = "1.18.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2f87b8aa10b915a06587d0dec516c282ff295b475d94abf425d62b57710070a2" dependencies = [ - "getrandom 0.3.3", + "getrandom 0.3.4", "js-sys", "serde", "wasm-bindgen", ] -[[package]] -name = "valuable" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" - [[package]] name = "version_check" version = "0.9.5" @@ -4692,19 +4590,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] -name = "wasi" -version = "0.14.4+wasi-0.2.4" +name = "wasip2" +version = "1.0.1+wasi-0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88a5f4a424faf49c3c2c344f166f0662341d470ea185e939657aaff130f0ec4a" +checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" dependencies = [ "wit-bindgen", ] [[package]] name = "wasm-bindgen" -version = "0.2.101" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e14915cadd45b529bb8d1f343c4ed0ac1de926144b746e2710f9cd05df6603b" +checksum = "da95793dfc411fbbd93f5be7715b0578ec61fe87cb1a42b12eb625caa5c5ea60" dependencies = [ "cfg-if", "once_cell", @@ -4713,25 +4611,11 @@ dependencies = [ "wasm-bindgen-shared", ] -[[package]] -name = "wasm-bindgen-backend" -version = "0.2.101" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e28d1ba982ca7923fd01448d5c30c6864d0a14109560296a162f80f305fb93bb" -dependencies = [ - "bumpalo", - "log", - "proc-macro2", - "quote", - "syn 2.0.106", - "wasm-bindgen-shared", -] - [[package]] name = "wasm-bindgen-futures" -version = "0.4.51" +version = "0.4.55" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ca85039a9b469b38336411d6d6ced91f3fc87109a2a27b0c197663f5144dffe" +checksum = "551f88106c6d5e7ccc7cd9a16f312dd3b5d36ea8b4954304657d5dfba115d4a0" dependencies = [ "cfg-if", "js-sys", @@ -4742,9 +4626,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.101" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c3d463ae3eff775b0c45df9da45d68837702ac35af998361e2c84e7c5ec1b0d" +checksum = "04264334509e04a7bf8690f2384ef5265f05143a4bff3889ab7a3269adab59c2" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -4752,22 +4636,22 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.101" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7bb4ce89b08211f923caf51d527662b75bdc9c9c7aab40f86dcb9fb85ac552aa" +checksum = "420bc339d9f322e562942d52e115d57e950d12d88983a14c79b86859ee6c7ebc" dependencies = [ + "bumpalo", "proc-macro2", "quote", - "syn 2.0.106", - "wasm-bindgen-backend", + "syn 2.0.108", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.101" +version = "0.2.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f143854a3b13752c6950862c906306adb27c7e839f7414cec8fea35beab624c1" +checksum = "76f218a38c84bcb33c25ec7059b07847d465ce0e0a76b995e134a45adcb6af76" dependencies = [ "unicode-ident", ] @@ -4787,9 +4671,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.78" +version = "0.3.82" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77e4b637749ff0d92b8fad63aa1f7cff3cbe125fd49c175cd6345e7272638b12" +checksum = "3a1f95c0d03a47f4ae1f7a64643a6bb97465d9b740f0fa8f90ea33915c99a9a1" dependencies = [ "js-sys", "wasm-bindgen", @@ -4807,9 +4691,9 @@ dependencies = [ [[package]] name = "webpki-roots" -version = "1.0.2" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e8983c3ab33d6fb807cfcdad2491c4ea8cbc8ed839181c7dfd9c67c83e261b2" +checksum = "32b130c0d2d49f8b6889abc456e795e82525204f27c42cf767cf0d7734e089b8" dependencies = [ "rustls-pki-types", ] @@ -4836,7 +4720,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.0", + "windows-sys 0.61.2", ] [[package]] @@ -4845,112 +4729,63 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" -[[package]] -name = "windows" -version = "0.61.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893" -dependencies = [ - "windows-collections", - "windows-core", - "windows-future", - "windows-link 0.1.3", - "windows-numerics", -] - -[[package]] -name = "windows-collections" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3beeceb5e5cfd9eb1d76b381630e82c4241ccd0d27f1a39ed41b2760b255c5e8" -dependencies = [ - "windows-core", -] - [[package]] name = "windows-core" -version = "0.61.2" +version = "0.62.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0fdd3ddb90610c7638aa2b3a3ab2904fb9e5cdbecc643ddb3647212781c4ae3" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" dependencies = [ "windows-implement", "windows-interface", - "windows-link 0.1.3", + "windows-link", "windows-result", "windows-strings", ] -[[package]] -name = "windows-future" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e" -dependencies = [ - "windows-core", - "windows-link 0.1.3", - "windows-threading", -] - [[package]] name = "windows-implement" -version = "0.60.0" +version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a47fddd13af08290e67f4acabf4b459f647552718f683a7b415d290ac744a836" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "windows-interface" -version = "0.59.1" +version = "0.59.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bd9211b69f8dcdfa817bfd14bf1c97c9188afa36f4750130fcdf3f400eca9fa8" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] name = "windows-link" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e6ad25900d524eaabdbbb96d20b4311e1e7ae1699af4fb28c17ae66c80d798a" - -[[package]] -name = "windows-link" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "45e46c0661abb7180e7b9c281db115305d49ca1709ab8242adf09666d2173c65" - -[[package]] -name = "windows-numerics" -version = "0.2.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9150af68066c4c5c07ddc0ce30421554771e528bde427614c61038bc2c92c2b1" -dependencies = [ - "windows-core", - "windows-link 0.1.3", -] +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" [[package]] name = "windows-result" -version = "0.3.4" +version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56f42bd332cc6c8eac5af113fc0c1fd6a8fd2aa08a0119358686e5160d0586c6" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" dependencies = [ - "windows-link 0.1.3", + "windows-link", ] [[package]] name = "windows-strings" -version = "0.4.2" +version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56e6c93f3a0c3b36176cb1327a4958a0353d5d166c2a35cb268ace15e91d3b57" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" dependencies = [ - "windows-link 0.1.3", + "windows-link", ] [[package]] @@ -4977,16 +4812,16 @@ version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" dependencies = [ - "windows-targets 0.53.3", + "windows-targets 0.53.5", ] [[package]] name = "windows-sys" -version = "0.61.0" +version = "0.61.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e201184e40b2ede64bc2ea34968b28e33622acdbbf37104f0e4a33f7abe657aa" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" dependencies = [ - "windows-link 0.2.0", + "windows-link", ] [[package]] @@ -5007,28 +4842,19 @@ dependencies = [ [[package]] name = "windows-targets" -version = "0.53.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5fe6031c4041849d7c496a8ded650796e7b6ecc19df1a431c1a363342e5dc91" -dependencies = [ - "windows-link 0.1.3", - "windows_aarch64_gnullvm 0.53.0", - "windows_aarch64_msvc 0.53.0", - "windows_i686_gnu 0.53.0", - "windows_i686_gnullvm 0.53.0", - "windows_i686_msvc 0.53.0", - "windows_x86_64_gnu 0.53.0", - "windows_x86_64_gnullvm 0.53.0", - "windows_x86_64_msvc 0.53.0", -] - -[[package]] -name = "windows-threading" -version = "0.1.0" +version = "0.53.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b66463ad2e0ea3bbf808b7f1d371311c80e115c0b71d60efc142cafbcfb057a6" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" dependencies = [ - "windows-link 0.1.3", + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", ] [[package]] @@ -5039,9 +4865,9 @@ checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" [[package]] name = "windows_aarch64_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86b8d5f90ddd19cb4a147a5fa63ca848db3df085e25fee3cc10b39b6eebae764" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" [[package]] name = "windows_aarch64_msvc" @@ -5051,9 +4877,9 @@ checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" [[package]] name = "windows_aarch64_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7651a1f62a11b8cbd5e0d42526e55f2c99886c77e007179efff86c2b137e66c" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" [[package]] name = "windows_i686_gnu" @@ -5063,9 +4889,9 @@ checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" [[package]] name = "windows_i686_gnu" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1dc67659d35f387f5f6c479dc4e28f1d4bb90ddd1a5d3da2e5d97b42d6272c3" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" [[package]] name = "windows_i686_gnullvm" @@ -5075,9 +4901,9 @@ checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" [[package]] name = "windows_i686_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ce6ccbdedbf6d6354471319e781c0dfef054c81fbc7cf83f338a4296c0cae11" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" [[package]] name = "windows_i686_msvc" @@ -5087,9 +4913,9 @@ checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" [[package]] name = "windows_i686_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "581fee95406bb13382d2f65cd4a908ca7b1e4c2f1917f143ba16efe98a589b5d" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" [[package]] name = "windows_x86_64_gnu" @@ -5099,9 +4925,9 @@ checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" [[package]] name = "windows_x86_64_gnu" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e55b5ac9ea33f2fc1716d1742db15574fd6fc8dadc51caab1c16a3d3b4190ba" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" [[package]] name = "windows_x86_64_gnullvm" @@ -5111,9 +4937,9 @@ checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" [[package]] name = "windows_x86_64_gnullvm" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a6e035dd0599267ce1ee132e51c27dd29437f63325753051e71dd9e42406c57" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" [[package]] name = "windows_x86_64_msvc" @@ -5123,9 +4949,9 @@ checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" [[package]] name = "windows_x86_64_msvc" -version = "0.53.0" +version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "271414315aff87387382ec3d271b52d7ae78726f5d44ac98b4f4030c91880486" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" [[package]] name = "winnow" @@ -5138,15 +4964,15 @@ dependencies = [ [[package]] name = "wit-bindgen" -version = "0.45.1" +version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c573471f125075647d03df72e026074b7203790d41351cd6edc96f46bcccd36" +checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" [[package]] name = "writeable" -version = "0.6.1" +version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea2f10b9bb0928dfb1b42b65e1f9e36f7f54dbdf08457afefb38afcdec4fa2bb" +checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" [[package]] name = "wyz" @@ -5168,11 +4994,10 @@ dependencies = [ [[package]] name = "yoke" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f41bb01b8226ef4bfd589436a297c53d118f65921786300e427be8d487695cc" +checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954" dependencies = [ - "serde", "stable_deref_trait", "yoke-derive", "zerofrom", @@ -5180,13 +5005,13 @@ dependencies = [ [[package]] name = "yoke-derive" -version = "0.8.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38da3c9736e16c5d3c8c597a9aaa5d1fa565d0532ae05e27c24aa62fb32c0ab6" +checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", "synstructure", ] @@ -5207,7 +5032,7 @@ checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] @@ -5227,21 +5052,21 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", "synstructure", ] [[package]] name = "zeroize" -version = "1.8.1" +version = "1.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ced3678a2879b30306d323f4542626697a464a97c0a07c9aebf7ebca65cd4dde" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" [[package]] name = "zerotrie" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36f0bbd478583f79edad978b407914f61b2972f5af6fa089686016be8f9af595" +checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851" dependencies = [ "displaydoc", "yoke", @@ -5250,9 +5075,9 @@ dependencies = [ [[package]] name = "zerovec" -version = "0.11.4" +version = "0.11.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7aa2bd55086f1ab526693ecbe444205da57e25f4489879da80635a46d90e73b" +checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002" dependencies = [ "yoke", "zerofrom", @@ -5261,13 +5086,13 @@ dependencies = [ [[package]] name = "zerovec-derive" -version = "0.11.1" +version = "0.11.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b96237efa0c878c64bd89c436f661be4e46b2f3eff1ebb976f7ef2321d2f58f" +checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.106", + "syn 2.0.108", ] [[package]] diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index f7f33d0729..b8c1efe694 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -31,9 +31,9 @@ license = "Apache-2.0" crate-type = ["cdylib"] [dependencies] -arrow = { version = "55.2", features = ["pyarrow", "chrono-tz"] } +arrow = { version = "56", features = ["pyarrow", "chrono-tz"] } iceberg = { path = "../../crates/iceberg" } -pyo3 = { version = "0.24.1", features = ["extension-module", "abi3-py39"] } +pyo3 = { version = "0.25", features = ["extension-module", "abi3-py39"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } -datafusion-ffi = { version = "49" } +datafusion-ffi = { version = "50" } tokio = { version = "1.46.1", default-features = false } diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index a9c3c290c7..516424e0b0 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -49,7 +49,7 @@ include = [ ignore = ["F403", "F405"] [tool.hatch.envs.dev] -dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==49.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] +dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==50.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] [tool.hatch.envs.dev.scripts] build = "maturin build --out dist --sdist" diff --git a/crates/catalog/sql/Cargo.toml b/crates/catalog/sql/Cargo.toml index 262422c284..eff605f466 100644 --- a/crates/catalog/sql/Cargo.toml +++ b/crates/catalog/sql/Cargo.toml @@ -36,7 +36,7 @@ strum = { workspace = true } [dev-dependencies] itertools = { workspace = true } -regex = "1.10.5" +regex = { workspace = true } sqlx = { version = "0.8.1", features = [ "tls-rustls", "runtime-tokio", diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 720b9363b9..c6f5af2f2a 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -39,7 +39,9 @@ use parquet::arrow::arrow_reader::{ }; use parquet::arrow::async_reader::AsyncFileReader; use parquet::arrow::{PARQUET_FIELD_ID_META_KEY, ParquetRecordBatchStreamBuilder, ProjectionMask}; -use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, RowGroupMetaData}; +use parquet::file::metadata::{ + PageIndexPolicy, ParquetMetaData, ParquetMetaDataReader, RowGroupMetaData, +}; use parquet::schema::types::{SchemaDescriptor, Type as ParquetType}; use crate::arrow::caching_delete_file_loader::CachingDeleteFileLoader; @@ -1434,9 +1436,10 @@ impl AsyncFileReader for ArrowFileReader { async move { let reader = ParquetMetaDataReader::new() .with_prefetch_hint(self.metadata_size_hint) - .with_column_indexes(self.preload_column_index) - .with_page_indexes(self.preload_page_index) - .with_offset_indexes(self.preload_offset_index); + // Set the page policy first because it updates both column and offset policies. + .with_page_index_policy(PageIndexPolicy::from(self.preload_page_index)) + .with_column_index_policy(PageIndexPolicy::from(self.preload_column_index)) + .with_offset_index_policy(PageIndexPolicy::from(self.preload_offset_index)); let size = self.meta.size; let meta = reader.load_and_finish(self, size).await?; From a76e73863b84845dca566ae3e73defa40592cb15 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 29 Oct 2025 09:25:48 +0800 Subject: [PATCH 042/891] ci: Migrate to uv for python (#1796) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1794 ## What changes are included in this PR? Migrate to uv ## Are these changes tested? --------- Signed-off-by: Xuanwo --- .github/workflows/bindings_python_ci.yml | 21 +- .licenserc.yaml | 19 +- bindings/python/README.md | 14 +- bindings/python/pyproject.toml | 18 +- bindings/python/uv.lock | 767 +++++++++++++++++++++++ 5 files changed, 811 insertions(+), 28 deletions(-) create mode 100644 bindings/python/uv.lock diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index ae3a6d1a54..9ea2ac0b90 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -80,17 +80,28 @@ jobs: - uses: actions/checkout@v5 - uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: 3.12 - uses: PyO3/maturin-action@v1 with: working-directory: "bindings/python" command: build args: --out dist --sdist + - name: Install uv + shell: bash + run: | + python -m pip install uv==0.9.3 + - name: Sync dependencies + working-directory: "bindings/python" + shell: bash + run: | + uv sync --group dev --no-install-project + - name: Install built wheel + working-directory: "bindings/python" + shell: bash + run: | + uv pip install --reinstall dist/pyiceberg_core-*.whl - name: Run tests working-directory: "bindings/python" shell: bash run: | - set -e - pip install hatch==1.14.2 - hatch run dev:pip install dist/pyiceberg_core-*.whl --force-reinstall - hatch run dev:test + uv run --group dev --no-sync pytest diff --git a/.licenserc.yaml b/.licenserc.yaml index 01625e32cb..da87374c3b 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -21,16 +21,17 @@ header: copyright-owner: Apache Software Foundation paths-ignore: - - 'LICENSE' - - 'NOTICE' - - '.gitattributes' - - '**/*.json' + - "LICENSE" + - "NOTICE" + - ".gitattributes" + - "**/*.json" # Generated content by mdbook - - 'website/book' + - "website/book" # Generated content by scripts - - '**/DEPENDENCIES.*.tsv' + - "**/DEPENDENCIES.*.tsv" # Release distributions - - 'dist/*' - - 'Cargo.lock' - - '.github/PULL_REQUEST_TEMPLATE.md' + - "dist/*" + - "Cargo.lock" + - "bindings/python/uv.lock" + - ".github/PULL_REQUEST_TEMPLATE.md" comment: on-failure diff --git a/bindings/python/README.md b/bindings/python/README.md index 2dddc71ac1..a91599d9a4 100644 --- a/bindings/python/README.md +++ b/bindings/python/README.md @@ -23,22 +23,26 @@ This project is used to build an Iceberg-rust powered core for [PyIceberg](https ## Setup -Install Hatch: +Install [uv](https://docs.astral.sh/uv/getting-started/installation/): ```shell -pip install hatch==1.12.0 +pip install uv==0.9.3 ``` -Hatch uses [uv](https://docs.astral.sh/uv/) as a backend by default, so [make sure that it is installed](https://docs.astral.sh/uv/getting-started/installation/) as well. +Set up the development environment: + +```shell +uv sync --group dev +``` ## Build ```shell -hatch run dev:develop +uv run --group dev maturin develop ``` ## Test ```shell -hatch run dev:test +uv run --group dev pytest ``` diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 516424e0b0..c79c9b520c 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -25,14 +25,13 @@ classifiers = [ "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", - "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ] name = "pyiceberg-core" readme = "project-description.md" -requires-python = "~=3.9" +requires-python = ">=3.10,<3.13" dynamic = ["version"] license = { file = "LICENSE" } @@ -48,10 +47,11 @@ include = [ [tool.ruff.lint] ignore = ["F403", "F405"] -[tool.hatch.envs.dev] -dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==50.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] - -[tool.hatch.envs.dev.scripts] -build = "maturin build --out dist --sdist" -develop = "maturin develop" -test = "pytest" +[dependency-groups] +dev = [ + "maturin>=1.0,<2.0", + "pytest>=8.3.2", + "datafusion==50.*", + "pyiceberg[sql-sqlite,pyarrow] @ git+https://github.com/apache/iceberg-python.git@d3eb149fe99b14e714a06e727b69cd617c6c052d", + "fastavro>=1.11.1", +] diff --git a/bindings/python/uv.lock b/bindings/python/uv.lock new file mode 100644 index 0000000000..614478d338 --- /dev/null +++ b/bindings/python/uv.lock @@ -0,0 +1,767 @@ +version = 1 +revision = 3 +requires-python = ">=3.10, <3.13" + +[[package]] +name = "annotated-types" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" }, +] + +[[package]] +name = "cachetools" +version = "6.2.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cc/7e/b975b5814bd36faf009faebe22c1072a1fa1168db34d285ef0ba071ad78c/cachetools-6.2.1.tar.gz", hash = "sha256:3f391e4bd8f8bf0931169baf7456cc822705f4e2a31f840d218f445b9a854201", size = 31325, upload-time = "2025-10-12T14:55:30.139Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/96/c5/1e741d26306c42e2bf6ab740b2202872727e0f606033c9dd713f8b93f5a8/cachetools-6.2.1-py3-none-any.whl", hash = "sha256:09868944b6dde876dfd44e1d47e18484541eaf12f26f29b7af91b26cc892d701", size = 11280, upload-time = "2025-10-12T14:55:28.382Z" }, +] + +[[package]] +name = "certifi" +version = "2025.10.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4c/5b/b6ce21586237c77ce67d01dc5507039d444b630dd76611bbca2d8e5dcd91/certifi-2025.10.5.tar.gz", hash = "sha256:47c09d31ccf2acf0be3f701ea53595ee7e0b8fa08801c6624be771df09ae7b43", size = 164519, upload-time = "2025-10-05T04:12:15.808Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e4/37/af0d2ef3967ac0d6113837b44a4f0bfe1328c2b9763bd5b1744520e5cfed/certifi-2025.10.5-py3-none-any.whl", hash = "sha256:0f212c2744a9bb6de0c56639a6f68afe01ecd92d91f14ae897c4fe7bbeeef0de", size = 163286, upload-time = "2025-10-05T04:12:14.03Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/13/69/33ddede1939fdd074bce5434295f38fae7136463422fe4fd3e0e89b98062/charset_normalizer-3.4.4.tar.gz", hash = "sha256:94537985111c35f28720e43603b8e7b43a6ecfb2ce1d3058bbe955b73404e21a", size = 129418, upload-time = "2025-10-14T04:42:32.879Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/b8/6d51fc1d52cbd52cd4ccedd5b5b2f0f6a11bbf6765c782298b0f3e808541/charset_normalizer-3.4.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e824f1492727fa856dd6eda4f7cee25f8518a12f3c4a56a74e8095695089cf6d", size = 209709, upload-time = "2025-10-14T04:40:11.385Z" }, + { url = "https://files.pythonhosted.org/packages/5c/af/1f9d7f7faafe2ddfb6f72a2e07a548a629c61ad510fe60f9630309908fef/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4bd5d4137d500351a30687c2d3971758aac9a19208fc110ccb9d7188fbe709e8", size = 148814, upload-time = "2025-10-14T04:40:13.135Z" }, + { url = "https://files.pythonhosted.org/packages/79/3d/f2e3ac2bbc056ca0c204298ea4e3d9db9b4afe437812638759db2c976b5f/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:027f6de494925c0ab2a55eab46ae5129951638a49a34d87f4c3eda90f696b4ad", size = 144467, upload-time = "2025-10-14T04:40:14.728Z" }, + { url = "https://files.pythonhosted.org/packages/ec/85/1bf997003815e60d57de7bd972c57dc6950446a3e4ccac43bc3070721856/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f820802628d2694cb7e56db99213f930856014862f3fd943d290ea8438d07ca8", size = 162280, upload-time = "2025-10-14T04:40:16.14Z" }, + { url = "https://files.pythonhosted.org/packages/3e/8e/6aa1952f56b192f54921c436b87f2aaf7c7a7c3d0d1a765547d64fd83c13/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:798d75d81754988d2565bff1b97ba5a44411867c0cf32b77a7e8f8d84796b10d", size = 159454, upload-time = "2025-10-14T04:40:17.567Z" }, + { url = "https://files.pythonhosted.org/packages/36/3b/60cbd1f8e93aa25d1c669c649b7a655b0b5fb4c571858910ea9332678558/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d1bb833febdff5c8927f922386db610b49db6e0d4f4ee29601d71e7c2694313", size = 153609, upload-time = "2025-10-14T04:40:19.08Z" }, + { url = "https://files.pythonhosted.org/packages/64/91/6a13396948b8fd3c4b4fd5bc74d045f5637d78c9675585e8e9fbe5636554/charset_normalizer-3.4.4-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cd98cdc06614a2f768d2b7286d66805f94c48cde050acdbbb7db2600ab3197e", size = 151849, upload-time = "2025-10-14T04:40:20.607Z" }, + { url = "https://files.pythonhosted.org/packages/b7/7a/59482e28b9981d105691e968c544cc0df3b7d6133152fb3dcdc8f135da7a/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:077fbb858e903c73f6c9db43374fd213b0b6a778106bc7032446a8e8b5b38b93", size = 151586, upload-time = "2025-10-14T04:40:21.719Z" }, + { url = "https://files.pythonhosted.org/packages/92/59/f64ef6a1c4bdd2baf892b04cd78792ed8684fbc48d4c2afe467d96b4df57/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:244bfb999c71b35de57821b8ea746b24e863398194a4014e4c76adc2bbdfeff0", size = 145290, upload-time = "2025-10-14T04:40:23.069Z" }, + { url = "https://files.pythonhosted.org/packages/6b/63/3bf9f279ddfa641ffa1962b0db6a57a9c294361cc2f5fcac997049a00e9c/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:64b55f9dce520635f018f907ff1b0df1fdc31f2795a922fb49dd14fbcdf48c84", size = 163663, upload-time = "2025-10-14T04:40:24.17Z" }, + { url = "https://files.pythonhosted.org/packages/ed/09/c9e38fc8fa9e0849b172b581fd9803bdf6e694041127933934184e19f8c3/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:faa3a41b2b66b6e50f84ae4a68c64fcd0c44355741c6374813a800cd6695db9e", size = 151964, upload-time = "2025-10-14T04:40:25.368Z" }, + { url = "https://files.pythonhosted.org/packages/d2/d1/d28b747e512d0da79d8b6a1ac18b7ab2ecfd81b2944c4c710e166d8dd09c/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:6515f3182dbe4ea06ced2d9e8666d97b46ef4c75e326b79bb624110f122551db", size = 161064, upload-time = "2025-10-14T04:40:26.806Z" }, + { url = "https://files.pythonhosted.org/packages/bb/9a/31d62b611d901c3b9e5500c36aab0ff5eb442043fb3a1c254200d3d397d9/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:cc00f04ed596e9dc0da42ed17ac5e596c6ccba999ba6bd92b0e0aef2f170f2d6", size = 155015, upload-time = "2025-10-14T04:40:28.284Z" }, + { url = "https://files.pythonhosted.org/packages/1f/f3/107e008fa2bff0c8b9319584174418e5e5285fef32f79d8ee6a430d0039c/charset_normalizer-3.4.4-cp310-cp310-win32.whl", hash = "sha256:f34be2938726fc13801220747472850852fe6b1ea75869a048d6f896838c896f", size = 99792, upload-time = "2025-10-14T04:40:29.613Z" }, + { url = "https://files.pythonhosted.org/packages/eb/66/e396e8a408843337d7315bab30dbf106c38966f1819f123257f5520f8a96/charset_normalizer-3.4.4-cp310-cp310-win_amd64.whl", hash = "sha256:a61900df84c667873b292c3de315a786dd8dac506704dea57bc957bd31e22c7d", size = 107198, upload-time = "2025-10-14T04:40:30.644Z" }, + { url = "https://files.pythonhosted.org/packages/b5/58/01b4f815bf0312704c267f2ccb6e5d42bcc7752340cd487bc9f8c3710597/charset_normalizer-3.4.4-cp310-cp310-win_arm64.whl", hash = "sha256:cead0978fc57397645f12578bfd2d5ea9138ea0fac82b2f63f7f7c6877986a69", size = 100262, upload-time = "2025-10-14T04:40:32.108Z" }, + { url = "https://files.pythonhosted.org/packages/ed/27/c6491ff4954e58a10f69ad90aca8a1b6fe9c5d3c6f380907af3c37435b59/charset_normalizer-3.4.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6e1fcf0720908f200cd21aa4e6750a48ff6ce4afe7ff5a79a90d5ed8a08296f8", size = 206988, upload-time = "2025-10-14T04:40:33.79Z" }, + { url = "https://files.pythonhosted.org/packages/94/59/2e87300fe67ab820b5428580a53cad894272dbb97f38a7a814a2a1ac1011/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f819d5fe9234f9f82d75bdfa9aef3a3d72c4d24a6e57aeaebba32a704553aa0", size = 147324, upload-time = "2025-10-14T04:40:34.961Z" }, + { url = "https://files.pythonhosted.org/packages/07/fb/0cf61dc84b2b088391830f6274cb57c82e4da8bbc2efeac8c025edb88772/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a59cb51917aa591b1c4e6a43c132f0cdc3c76dbad6155df4e28ee626cc77a0a3", size = 142742, upload-time = "2025-10-14T04:40:36.105Z" }, + { url = "https://files.pythonhosted.org/packages/62/8b/171935adf2312cd745d290ed93cf16cf0dfe320863ab7cbeeae1dcd6535f/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8ef3c867360f88ac904fd3f5e1f902f13307af9052646963ee08ff4f131adafc", size = 160863, upload-time = "2025-10-14T04:40:37.188Z" }, + { url = "https://files.pythonhosted.org/packages/09/73/ad875b192bda14f2173bfc1bc9a55e009808484a4b256748d931b6948442/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d9e45d7faa48ee908174d8fe84854479ef838fc6a705c9315372eacbc2f02897", size = 157837, upload-time = "2025-10-14T04:40:38.435Z" }, + { url = "https://files.pythonhosted.org/packages/6d/fc/de9cce525b2c5b94b47c70a4b4fb19f871b24995c728e957ee68ab1671ea/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:840c25fb618a231545cbab0564a799f101b63b9901f2569faecd6b222ac72381", size = 151550, upload-time = "2025-10-14T04:40:40.053Z" }, + { url = "https://files.pythonhosted.org/packages/55/c2/43edd615fdfba8c6f2dfbd459b25a6b3b551f24ea21981e23fb768503ce1/charset_normalizer-3.4.4-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ca5862d5b3928c4940729dacc329aa9102900382fea192fc5e52eb69d6093815", size = 149162, upload-time = "2025-10-14T04:40:41.163Z" }, + { url = "https://files.pythonhosted.org/packages/03/86/bde4ad8b4d0e9429a4e82c1e8f5c659993a9a863ad62c7df05cf7b678d75/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d9c7f57c3d666a53421049053eaacdd14bbd0a528e2186fcb2e672effd053bb0", size = 150019, upload-time = "2025-10-14T04:40:42.276Z" }, + { url = "https://files.pythonhosted.org/packages/1f/86/a151eb2af293a7e7bac3a739b81072585ce36ccfb4493039f49f1d3cae8c/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:277e970e750505ed74c832b4bf75dac7476262ee2a013f5574dd49075879e161", size = 143310, upload-time = "2025-10-14T04:40:43.439Z" }, + { url = "https://files.pythonhosted.org/packages/b5/fe/43dae6144a7e07b87478fdfc4dbe9efd5defb0e7ec29f5f58a55aeef7bf7/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:31fd66405eaf47bb62e8cd575dc621c56c668f27d46a61d975a249930dd5e2a4", size = 162022, upload-time = "2025-10-14T04:40:44.547Z" }, + { url = "https://files.pythonhosted.org/packages/80/e6/7aab83774f5d2bca81f42ac58d04caf44f0cc2b65fc6db2b3b2e8a05f3b3/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:0d3d8f15c07f86e9ff82319b3d9ef6f4bf907608f53fe9d92b28ea9ae3d1fd89", size = 149383, upload-time = "2025-10-14T04:40:46.018Z" }, + { url = "https://files.pythonhosted.org/packages/4f/e8/b289173b4edae05c0dde07f69f8db476a0b511eac556dfe0d6bda3c43384/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:9f7fcd74d410a36883701fafa2482a6af2ff5ba96b9a620e9e0721e28ead5569", size = 159098, upload-time = "2025-10-14T04:40:47.081Z" }, + { url = "https://files.pythonhosted.org/packages/d8/df/fe699727754cae3f8478493c7f45f777b17c3ef0600e28abfec8619eb49c/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ebf3e58c7ec8a8bed6d66a75d7fb37b55e5015b03ceae72a8e7c74495551e224", size = 152991, upload-time = "2025-10-14T04:40:48.246Z" }, + { url = "https://files.pythonhosted.org/packages/1a/86/584869fe4ddb6ffa3bd9f491b87a01568797fb9bd8933f557dba9771beaf/charset_normalizer-3.4.4-cp311-cp311-win32.whl", hash = "sha256:eecbc200c7fd5ddb9a7f16c7decb07b566c29fa2161a16cf67b8d068bd21690a", size = 99456, upload-time = "2025-10-14T04:40:49.376Z" }, + { url = "https://files.pythonhosted.org/packages/65/f6/62fdd5feb60530f50f7e38b4f6a1d5203f4d16ff4f9f0952962c044e919a/charset_normalizer-3.4.4-cp311-cp311-win_amd64.whl", hash = "sha256:5ae497466c7901d54b639cf42d5b8c1b6a4fead55215500d2f486d34db48d016", size = 106978, upload-time = "2025-10-14T04:40:50.844Z" }, + { url = "https://files.pythonhosted.org/packages/7a/9d/0710916e6c82948b3be62d9d398cb4fcf4e97b56d6a6aeccd66c4b2f2bd5/charset_normalizer-3.4.4-cp311-cp311-win_arm64.whl", hash = "sha256:65e2befcd84bc6f37095f5961e68a6f077bf44946771354a28ad434c2cce0ae1", size = 99969, upload-time = "2025-10-14T04:40:52.272Z" }, + { url = "https://files.pythonhosted.org/packages/f3/85/1637cd4af66fa687396e757dec650f28025f2a2f5a5531a3208dc0ec43f2/charset_normalizer-3.4.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:0a98e6759f854bd25a58a73fa88833fba3b7c491169f86ce1180c948ab3fd394", size = 208425, upload-time = "2025-10-14T04:40:53.353Z" }, + { url = "https://files.pythonhosted.org/packages/9d/6a/04130023fef2a0d9c62d0bae2649b69f7b7d8d24ea5536feef50551029df/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5b290ccc2a263e8d185130284f8501e3e36c5e02750fc6b6bdeb2e9e96f1e25", size = 148162, upload-time = "2025-10-14T04:40:54.558Z" }, + { url = "https://files.pythonhosted.org/packages/78/29/62328d79aa60da22c9e0b9a66539feae06ca0f5a4171ac4f7dc285b83688/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74bb723680f9f7a6234dcf67aea57e708ec1fbdf5699fb91dfd6f511b0a320ef", size = 144558, upload-time = "2025-10-14T04:40:55.677Z" }, + { url = "https://files.pythonhosted.org/packages/86/bb/b32194a4bf15b88403537c2e120b817c61cd4ecffa9b6876e941c3ee38fe/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f1e34719c6ed0b92f418c7c780480b26b5d9c50349e9a9af7d76bf757530350d", size = 161497, upload-time = "2025-10-14T04:40:57.217Z" }, + { url = "https://files.pythonhosted.org/packages/19/89/a54c82b253d5b9b111dc74aca196ba5ccfcca8242d0fb64146d4d3183ff1/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2437418e20515acec67d86e12bf70056a33abdacb5cb1655042f6538d6b085a8", size = 159240, upload-time = "2025-10-14T04:40:58.358Z" }, + { url = "https://files.pythonhosted.org/packages/c0/10/d20b513afe03acc89ec33948320a5544d31f21b05368436d580dec4e234d/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:11d694519d7f29d6cd09f6ac70028dba10f92f6cdd059096db198c283794ac86", size = 153471, upload-time = "2025-10-14T04:40:59.468Z" }, + { url = "https://files.pythonhosted.org/packages/61/fa/fbf177b55bdd727010f9c0a3c49eefa1d10f960e5f09d1d887bf93c2e698/charset_normalizer-3.4.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ac1c4a689edcc530fc9d9aa11f5774b9e2f33f9a0c6a57864e90908f5208d30a", size = 150864, upload-time = "2025-10-14T04:41:00.623Z" }, + { url = "https://files.pythonhosted.org/packages/05/12/9fbc6a4d39c0198adeebbde20b619790e9236557ca59fc40e0e3cebe6f40/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:21d142cc6c0ec30d2efee5068ca36c128a30b0f2c53c1c07bd78cb6bc1d3be5f", size = 150647, upload-time = "2025-10-14T04:41:01.754Z" }, + { url = "https://files.pythonhosted.org/packages/ad/1f/6a9a593d52e3e8c5d2b167daf8c6b968808efb57ef4c210acb907c365bc4/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:5dbe56a36425d26d6cfb40ce79c314a2e4dd6211d51d6d2191c00bed34f354cc", size = 145110, upload-time = "2025-10-14T04:41:03.231Z" }, + { url = "https://files.pythonhosted.org/packages/30/42/9a52c609e72471b0fc54386dc63c3781a387bb4fe61c20231a4ebcd58bdd/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:5bfbb1b9acf3334612667b61bd3002196fe2a1eb4dd74d247e0f2a4d50ec9bbf", size = 162839, upload-time = "2025-10-14T04:41:04.715Z" }, + { url = "https://files.pythonhosted.org/packages/c4/5b/c0682bbf9f11597073052628ddd38344a3d673fda35a36773f7d19344b23/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d055ec1e26e441f6187acf818b73564e6e6282709e9bcb5b63f5b23068356a15", size = 150667, upload-time = "2025-10-14T04:41:05.827Z" }, + { url = "https://files.pythonhosted.org/packages/e4/24/a41afeab6f990cf2daf6cb8c67419b63b48cf518e4f56022230840c9bfb2/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:af2d8c67d8e573d6de5bc30cdb27e9b95e49115cd9baad5ddbd1a6207aaa82a9", size = 160535, upload-time = "2025-10-14T04:41:06.938Z" }, + { url = "https://files.pythonhosted.org/packages/2a/e5/6a4ce77ed243c4a50a1fecca6aaaab419628c818a49434be428fe24c9957/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:780236ac706e66881f3b7f2f32dfe90507a09e67d1d454c762cf642e6e1586e0", size = 154816, upload-time = "2025-10-14T04:41:08.101Z" }, + { url = "https://files.pythonhosted.org/packages/a8/ef/89297262b8092b312d29cdb2517cb1237e51db8ecef2e9af5edbe7b683b1/charset_normalizer-3.4.4-cp312-cp312-win32.whl", hash = "sha256:5833d2c39d8896e4e19b689ffc198f08ea58116bee26dea51e362ecc7cd3ed26", size = 99694, upload-time = "2025-10-14T04:41:09.23Z" }, + { url = "https://files.pythonhosted.org/packages/3d/2d/1e5ed9dd3b3803994c155cd9aacb60c82c331bad84daf75bcb9c91b3295e/charset_normalizer-3.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:a79cfe37875f822425b89a82333404539ae63dbdddf97f84dcbc3d339aae9525", size = 107131, upload-time = "2025-10-14T04:41:10.467Z" }, + { url = "https://files.pythonhosted.org/packages/d0/d9/0ed4c7098a861482a7b6a95603edce4c0d9db2311af23da1fb2b75ec26fc/charset_normalizer-3.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:376bec83a63b8021bb5c8ea75e21c4ccb86e7e45ca4eb81146091b56599b80c3", size = 100390, upload-time = "2025-10-14T04:41:11.915Z" }, + { url = "https://files.pythonhosted.org/packages/0a/4c/925909008ed5a988ccbb72dcc897407e5d6d3bd72410d69e051fc0c14647/charset_normalizer-3.4.4-py3-none-any.whl", hash = "sha256:7a32c560861a02ff789ad905a2fe94e3f840803362c84fecf1851cb4cf3dc37f", size = 53402, upload-time = "2025-10-14T04:42:31.76Z" }, +] + +[[package]] +name = "click" +version = "8.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/46/61/de6cd827efad202d7057d93e0fed9294b96952e188f7384832791c7b2254/click-8.3.0.tar.gz", hash = "sha256:e7b8232224eba16f4ebe410c25ced9f7875cb5f3263ffc93cc3e8da705e229c4", size = 276943, upload-time = "2025-09-18T17:32:23.696Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/d3/9dcc0f5797f070ec8edf30fbadfb200e71d9db6b84d211e3b2085a7589a0/click-8.3.0-py3-none-any.whl", hash = "sha256:9b9f285302c6e3064f4330c05f05b81945b2a39544279343e6e7c5f27a9baddc", size = 107295, upload-time = "2025-09-18T17:32:22.42Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "datafusion" +version = "50.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyarrow" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fa/cc/e8e8f7c472e93e7a560203ac40ac319b926029007c0dad873dbba97f9f2d/datafusion-50.1.0.tar.gz", hash = "sha256:d8b8f027c7ce2498cda1589d3ce6d8720798963e031660fbe4d2e26e172442ec", size = 188103, upload-time = "2025-10-20T12:39:23.802Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/6e/f9e2d5d935024a79fd549b5ce1d05549d26a027aab800727d492ac036504/datafusion-50.1.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:aeaa3c7bcf630bbea962b8fe75d300d98eaf7e2a5edf98e6a0130a1bec3543ea", size = 29280689, upload-time = "2025-10-20T12:39:06.913Z" }, + { url = "https://files.pythonhosted.org/packages/db/58/2dc473240f552d3620186b527c04397f82b36f02243afaf49f0813c84a17/datafusion-50.1.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:85727df82c818103092c3ee18d198365833d3e44c2921d2b378d4d682798e511", size = 26140751, upload-time = "2025-10-20T12:39:09.95Z" }, + { url = "https://files.pythonhosted.org/packages/00/ba/8d8aa1df96e0666752e5c9d406d440495df2014d315b2a95bbef9856b23e/datafusion-50.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49f5bd0edb2bf2d00625beeb46a115e1421db2e1b14b535f7c17cc0927f36b8a", size = 32165290, upload-time = "2025-10-20T12:39:13.713Z" }, + { url = "https://files.pythonhosted.org/packages/11/9a/afce9586145b3ed153d75364b21102a6a95260940352e06b7c6709e9d2db/datafusion-50.1.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:5c9c2f70922ddedf54d8abd4ba9585a5026c3409438f5aafc1ad0428a67a4d1f", size = 29982398, upload-time = "2025-10-20T12:39:16.823Z" }, + { url = "https://files.pythonhosted.org/packages/51/a3/41ef1c565770ef0c4060ee3fd50367dd06816f70a5be1ef41fbd7c3975e8/datafusion-50.1.0-cp39-abi3-win_amd64.whl", hash = "sha256:145c8f2e969c9cc51dc6af8a185ec39739ebeb5d680f9fe0020e005564ed40a8", size = 31258359, upload-time = "2025-10-20T12:39:21.731Z" }, +] + +[[package]] +name = "exceptiongroup" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0b/9f/a65090624ecf468cdca03533906e7c69ed7588582240cfe7cc9e770b50eb/exceptiongroup-1.3.0.tar.gz", hash = "sha256:b241f5885f560bc56a59ee63ca4c6a8bfa46ae4ad651af316d4e81817bb9fd88", size = 29749, upload-time = "2025-05-10T17:42:51.123Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/36/f4/c6e662dade71f56cd2f3735141b265c3c79293c109549c1e6933b0651ffc/exceptiongroup-1.3.0-py3-none-any.whl", hash = "sha256:4d111e6e0c13d0644cad6ddaa7ed0261a0b36971f6d23e7ec9b4b9097da78a10", size = 16674, upload-time = "2025-05-10T17:42:49.33Z" }, +] + +[[package]] +name = "fastavro" +version = "1.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/65/8b/fa2d3287fd2267be6261d0177c6809a7fa12c5600ddb33490c8dc29e77b2/fastavro-1.12.1.tar.gz", hash = "sha256:2f285be49e45bc047ab2f6bed040bb349da85db3f3c87880e4b92595ea093b2b", size = 1025661, upload-time = "2025-10-10T15:40:55.41Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/40/a0/077fd7cbfc143152cb96780cb592ed6cb6696667d8bc1b977745eb2255a8/fastavro-1.12.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:00650ca533907361edda22e6ffe8cf87ab2091c5d8aee5c8000b0f2dcdda7ed3", size = 1000335, upload-time = "2025-10-10T15:40:59.834Z" }, + { url = "https://files.pythonhosted.org/packages/a0/ae/a115e027f3a75df237609701b03ecba0b7f0aa3d77fe0161df533fde1eb7/fastavro-1.12.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac76d6d95f909c72ee70d314b460b7e711d928845771531d823eb96a10952d26", size = 3221067, upload-time = "2025-10-10T15:41:04.399Z" }, + { url = "https://files.pythonhosted.org/packages/94/4e/c4991c3eec0175af9a8a0c161b88089cb7bf7fe353b3e3be1bc4cf9036b2/fastavro-1.12.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f55eef18c41d4476bd32a82ed5dd86aabc3f614e1b66bdb09ffa291612e1670", size = 3228979, upload-time = "2025-10-10T15:41:06.738Z" }, + { url = "https://files.pythonhosted.org/packages/21/0c/f2afb8eaea38799ccb1ed07d68bf2659f2e313f1902bbd36774cf6a1bef9/fastavro-1.12.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:81563e1f93570e6565487cdb01ba241a36a00e58cff9c5a0614af819d1155d8f", size = 3160740, upload-time = "2025-10-10T15:41:08.731Z" }, + { url = "https://files.pythonhosted.org/packages/0d/1a/f4d367924b40b86857862c1fa65f2afba94ddadf298b611e610a676a29e5/fastavro-1.12.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:bec207360f76f0b3de540758a297193c5390e8e081c43c3317f610b1414d8c8f", size = 3235787, upload-time = "2025-10-10T15:41:10.869Z" }, + { url = "https://files.pythonhosted.org/packages/90/ec/8db9331896e3dfe4f71b2b3c23f2e97fbbfd90129777467ca9f8bafccb74/fastavro-1.12.1-cp310-cp310-win_amd64.whl", hash = "sha256:c0390bfe4a9f8056a75ac6785fbbff8f5e317f5356481d2e29ec980877d2314b", size = 449350, upload-time = "2025-10-10T15:41:12.104Z" }, + { url = "https://files.pythonhosted.org/packages/a0/e9/31c64b47cefc0951099e7c0c8c8ea1c931edd1350f34d55c27cbfbb08df1/fastavro-1.12.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6b632b713bc5d03928a87d811fa4a11d5f25cd43e79c161e291c7d3f7aa740fd", size = 1016585, upload-time = "2025-10-10T15:41:13.717Z" }, + { url = "https://files.pythonhosted.org/packages/10/76/111560775b548f5d8d828c1b5285ff90e2d2745643fb80ecbf115344eea4/fastavro-1.12.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eaa7ab3769beadcebb60f0539054c7755f63bd9cf7666e2c15e615ab605f89a8", size = 3404629, upload-time = "2025-10-10T15:41:15.642Z" }, + { url = "https://files.pythonhosted.org/packages/b0/07/6bb93cb963932146c2b6c5c765903a0a547ad9f0f8b769a4a9aad8c06369/fastavro-1.12.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:123fb221df3164abd93f2d042c82f538a1d5a43ce41375f12c91ce1355a9141e", size = 3428594, upload-time = "2025-10-10T15:41:17.779Z" }, + { url = "https://files.pythonhosted.org/packages/d1/67/8115ec36b584197ea737ec79e3499e1f1b640b288d6c6ee295edd13b80f6/fastavro-1.12.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:632a4e3ff223f834ddb746baae0cc7cee1068eb12c32e4d982c2fee8a5b483d0", size = 3344145, upload-time = "2025-10-10T15:41:19.89Z" }, + { url = "https://files.pythonhosted.org/packages/9e/9e/a7cebb3af967e62539539897c10138fa0821668ec92525d1be88a9cd3ee6/fastavro-1.12.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:83e6caf4e7a8717d932a3b1ff31595ad169289bbe1128a216be070d3a8391671", size = 3431942, upload-time = "2025-10-10T15:41:22.076Z" }, + { url = "https://files.pythonhosted.org/packages/c0/d1/7774ddfb8781c5224294c01a593ebce2ad3289b948061c9701bd1903264d/fastavro-1.12.1-cp311-cp311-win_amd64.whl", hash = "sha256:b91a0fe5a173679a6c02d53ca22dcaad0a2c726b74507e0c1c2e71a7c3f79ef9", size = 450542, upload-time = "2025-10-10T15:41:23.333Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f0/10bd1a3d08667fa0739e2b451fe90e06df575ec8b8ba5d3135c70555c9bd/fastavro-1.12.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:509818cb24b98a804fc80be9c5fed90f660310ae3d59382fc811bfa187122167", size = 1009057, upload-time = "2025-10-10T15:41:24.556Z" }, + { url = "https://files.pythonhosted.org/packages/78/ad/0d985bc99e1fa9e74c636658000ba38a5cd7f5ab2708e9c62eaf736ecf1a/fastavro-1.12.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:089e155c0c76e0d418d7e79144ce000524dd345eab3bc1e9c5ae69d500f71b14", size = 3391866, upload-time = "2025-10-10T15:41:26.882Z" }, + { url = "https://files.pythonhosted.org/packages/0d/9e/b4951dc84ebc34aac69afcbfbb22ea4a91080422ec2bfd2c06076ff1d419/fastavro-1.12.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44cbff7518901c91a82aab476fcab13d102e4999499df219d481b9e15f61af34", size = 3458005, upload-time = "2025-10-10T15:41:29.017Z" }, + { url = "https://files.pythonhosted.org/packages/af/f8/5a8df450a9f55ca8441f22ea0351d8c77809fc121498b6970daaaf667a21/fastavro-1.12.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a275e48df0b1701bb764b18a8a21900b24cf882263cb03d35ecdba636bbc830b", size = 3295258, upload-time = "2025-10-10T15:41:31.564Z" }, + { url = "https://files.pythonhosted.org/packages/99/b2/40f25299111d737e58b85696e91138a66c25b7334f5357e7ac2b0e8966f8/fastavro-1.12.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2de72d786eb38be6b16d556b27232b1bf1b2797ea09599507938cdb7a9fe3e7c", size = 3430328, upload-time = "2025-10-10T15:41:33.689Z" }, + { url = "https://files.pythonhosted.org/packages/e0/07/85157a7c57c5f8b95507d7829b5946561e5ee656ff80e9dd9a757f53ddaf/fastavro-1.12.1-cp312-cp312-win_amd64.whl", hash = "sha256:9090f0dee63fe022ee9cc5147483366cc4171c821644c22da020d6b48f576b4f", size = 444140, upload-time = "2025-10-10T15:41:34.902Z" }, +] + +[[package]] +name = "fsspec" +version = "2025.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/de/e0/bab50af11c2d75c9c4a2a26a5254573c0bd97cea152254401510950486fa/fsspec-2025.9.0.tar.gz", hash = "sha256:19fd429483d25d28b65ec68f9f4adc16c17ea2c7c7bf54ec61360d478fb19c19", size = 304847, upload-time = "2025-09-02T19:10:49.215Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/47/71/70db47e4f6ce3e5c37a607355f80da8860a33226be640226ac52cb05ef2e/fsspec-2025.9.0-py3-none-any.whl", hash = "sha256:530dc2a2af60a414a832059574df4a6e10cce927f6f4a78209390fe38955cfb7", size = 199289, upload-time = "2025-09-02T19:10:47.708Z" }, +] + +[[package]] +name = "greenlet" +version = "3.2.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/03/b8/704d753a5a45507a7aab61f18db9509302ed3d0a27ac7e0359ec2905b1a6/greenlet-3.2.4.tar.gz", hash = "sha256:0dca0d95ff849f9a364385f36ab49f50065d76964944638be9691e1832e9f86d", size = 188260, upload-time = "2025-08-07T13:24:33.51Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7d/ed/6bfa4109fcb23a58819600392564fea69cdc6551ffd5e69ccf1d52a40cbc/greenlet-3.2.4-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:8c68325b0d0acf8d91dde4e6f930967dd52a5302cd4062932a6b2e7c2969f47c", size = 271061, upload-time = "2025-08-07T13:17:15.373Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fc/102ec1a2fc015b3a7652abab7acf3541d58c04d3d17a8d3d6a44adae1eb1/greenlet-3.2.4-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:94385f101946790ae13da500603491f04a76b6e4c059dab271b3ce2e283b2590", size = 629475, upload-time = "2025-08-07T13:42:54.009Z" }, + { url = "https://files.pythonhosted.org/packages/c5/26/80383131d55a4ac0fb08d71660fd77e7660b9db6bdb4e8884f46d9f2cc04/greenlet-3.2.4-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f10fd42b5ee276335863712fa3da6608e93f70629c631bf77145021600abc23c", size = 640802, upload-time = "2025-08-07T13:45:25.52Z" }, + { url = "https://files.pythonhosted.org/packages/9f/7c/e7833dbcd8f376f3326bd728c845d31dcde4c84268d3921afcae77d90d08/greenlet-3.2.4-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:c8c9e331e58180d0d83c5b7999255721b725913ff6bc6cf39fa2a45841a4fd4b", size = 636703, upload-time = "2025-08-07T13:53:12.622Z" }, + { url = "https://files.pythonhosted.org/packages/e9/49/547b93b7c0428ede7b3f309bc965986874759f7d89e4e04aeddbc9699acb/greenlet-3.2.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:58b97143c9cc7b86fc458f215bd0932f1757ce649e05b640fea2e79b54cedb31", size = 635417, upload-time = "2025-08-07T13:18:25.189Z" }, + { url = "https://files.pythonhosted.org/packages/7f/91/ae2eb6b7979e2f9b035a9f612cf70f1bf54aad4e1d125129bef1eae96f19/greenlet-3.2.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c2ca18a03a8cfb5b25bc1cbe20f3d9a4c80d8c3b13ba3df49ac3961af0b1018d", size = 584358, upload-time = "2025-08-07T13:18:23.708Z" }, + { url = "https://files.pythonhosted.org/packages/f7/85/433de0c9c0252b22b16d413c9407e6cb3b41df7389afc366ca204dbc1393/greenlet-3.2.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9fe0a28a7b952a21e2c062cd5756d34354117796c6d9215a87f55e38d15402c5", size = 1113550, upload-time = "2025-08-07T13:42:37.467Z" }, + { url = "https://files.pythonhosted.org/packages/a1/8d/88f3ebd2bc96bf7747093696f4335a0a8a4c5acfcf1b757717c0d2474ba3/greenlet-3.2.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8854167e06950ca75b898b104b63cc646573aa5fef1353d4508ecdd1ee76254f", size = 1137126, upload-time = "2025-08-07T13:18:20.239Z" }, + { url = "https://files.pythonhosted.org/packages/d6/6f/b60b0291d9623c496638c582297ead61f43c4b72eef5e9c926ef4565ec13/greenlet-3.2.4-cp310-cp310-win_amd64.whl", hash = "sha256:73f49b5368b5359d04e18d15828eecc1806033db5233397748f4ca813ff1056c", size = 298654, upload-time = "2025-08-07T13:50:00.469Z" }, + { url = "https://files.pythonhosted.org/packages/a4/de/f28ced0a67749cac23fecb02b694f6473f47686dff6afaa211d186e2ef9c/greenlet-3.2.4-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:96378df1de302bc38e99c3a9aa311967b7dc80ced1dcc6f171e99842987882a2", size = 272305, upload-time = "2025-08-07T13:15:41.288Z" }, + { url = "https://files.pythonhosted.org/packages/09/16/2c3792cba130000bf2a31c5272999113f4764fd9d874fb257ff588ac779a/greenlet-3.2.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1ee8fae0519a337f2329cb78bd7a8e128ec0f881073d43f023c7b8d4831d5246", size = 632472, upload-time = "2025-08-07T13:42:55.044Z" }, + { url = "https://files.pythonhosted.org/packages/ae/8f/95d48d7e3d433e6dae5b1682e4292242a53f22df82e6d3dda81b1701a960/greenlet-3.2.4-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:94abf90142c2a18151632371140b3dba4dee031633fe614cb592dbb6c9e17bc3", size = 644646, upload-time = "2025-08-07T13:45:26.523Z" }, + { url = "https://files.pythonhosted.org/packages/d5/5e/405965351aef8c76b8ef7ad370e5da58d57ef6068df197548b015464001a/greenlet-3.2.4-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:4d1378601b85e2e5171b99be8d2dc85f594c79967599328f95c1dc1a40f1c633", size = 640519, upload-time = "2025-08-07T13:53:13.928Z" }, + { url = "https://files.pythonhosted.org/packages/25/5d/382753b52006ce0218297ec1b628e048c4e64b155379331f25a7316eb749/greenlet-3.2.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0db5594dce18db94f7d1650d7489909b57afde4c580806b8d9203b6e79cdc079", size = 639707, upload-time = "2025-08-07T13:18:27.146Z" }, + { url = "https://files.pythonhosted.org/packages/1f/8e/abdd3f14d735b2929290a018ecf133c901be4874b858dd1c604b9319f064/greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2523e5246274f54fdadbce8494458a2ebdcdbc7b802318466ac5606d3cded1f8", size = 587684, upload-time = "2025-08-07T13:18:25.164Z" }, + { url = "https://files.pythonhosted.org/packages/5d/65/deb2a69c3e5996439b0176f6651e0052542bb6c8f8ec2e3fba97c9768805/greenlet-3.2.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:1987de92fec508535687fb807a5cea1560f6196285a4cde35c100b8cd632cc52", size = 1116647, upload-time = "2025-08-07T13:42:38.655Z" }, + { url = "https://files.pythonhosted.org/packages/3f/cc/b07000438a29ac5cfb2194bfc128151d52f333cee74dd7dfe3fb733fc16c/greenlet-3.2.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:55e9c5affaa6775e2c6b67659f3a71684de4c549b3dd9afca3bc773533d284fa", size = 1142073, upload-time = "2025-08-07T13:18:21.737Z" }, + { url = "https://files.pythonhosted.org/packages/d8/0f/30aef242fcab550b0b3520b8e3561156857c94288f0332a79928c31a52cf/greenlet-3.2.4-cp311-cp311-win_amd64.whl", hash = "sha256:9c40adce87eaa9ddb593ccb0fa6a07caf34015a29bf8d344811665b573138db9", size = 299100, upload-time = "2025-08-07T13:44:12.287Z" }, + { url = "https://files.pythonhosted.org/packages/44/69/9b804adb5fd0671f367781560eb5eb586c4d495277c93bde4307b9e28068/greenlet-3.2.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3b67ca49f54cede0186854a008109d6ee71f66bd57bb36abd6d0a0267b540cdd", size = 274079, upload-time = "2025-08-07T13:15:45.033Z" }, + { url = "https://files.pythonhosted.org/packages/46/e9/d2a80c99f19a153eff70bc451ab78615583b8dac0754cfb942223d2c1a0d/greenlet-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ddf9164e7a5b08e9d22511526865780a576f19ddd00d62f8a665949327fde8bb", size = 640997, upload-time = "2025-08-07T13:42:56.234Z" }, + { url = "https://files.pythonhosted.org/packages/3b/16/035dcfcc48715ccd345f3a93183267167cdd162ad123cd93067d86f27ce4/greenlet-3.2.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f28588772bb5fb869a8eb331374ec06f24a83a9c25bfa1f38b6993afe9c1e968", size = 655185, upload-time = "2025-08-07T13:45:27.624Z" }, + { url = "https://files.pythonhosted.org/packages/31/da/0386695eef69ffae1ad726881571dfe28b41970173947e7c558d9998de0f/greenlet-3.2.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:5c9320971821a7cb77cfab8d956fa8e39cd07ca44b6070db358ceb7f8797c8c9", size = 649926, upload-time = "2025-08-07T13:53:15.251Z" }, + { url = "https://files.pythonhosted.org/packages/68/88/69bf19fd4dc19981928ceacbc5fd4bb6bc2215d53199e367832e98d1d8fe/greenlet-3.2.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c60a6d84229b271d44b70fb6e5fa23781abb5d742af7b808ae3f6efd7c9c60f6", size = 651839, upload-time = "2025-08-07T13:18:30.281Z" }, + { url = "https://files.pythonhosted.org/packages/19/0d/6660d55f7373b2ff8152401a83e02084956da23ae58cddbfb0b330978fe9/greenlet-3.2.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b3812d8d0c9579967815af437d96623f45c0f2ae5f04e366de62a12d83a8fb0", size = 607586, upload-time = "2025-08-07T13:18:28.544Z" }, + { url = "https://files.pythonhosted.org/packages/8e/1a/c953fdedd22d81ee4629afbb38d2f9d71e37d23caace44775a3a969147d4/greenlet-3.2.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:abbf57b5a870d30c4675928c37278493044d7c14378350b3aa5d484fa65575f0", size = 1123281, upload-time = "2025-08-07T13:42:39.858Z" }, + { url = "https://files.pythonhosted.org/packages/3f/c7/12381b18e21aef2c6bd3a636da1088b888b97b7a0362fac2e4de92405f97/greenlet-3.2.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:20fb936b4652b6e307b8f347665e2c615540d4b42b3b4c8a321d8286da7e520f", size = 1151142, upload-time = "2025-08-07T13:18:22.981Z" }, + { url = "https://files.pythonhosted.org/packages/e9/08/b0814846b79399e585f974bbeebf5580fbe59e258ea7be64d9dfb253c84f/greenlet-3.2.4-cp312-cp312-win_amd64.whl", hash = "sha256:a7d4e128405eea3814a12cc2605e0e6aedb4035bf32697f72deca74de4105e02", size = 299899, upload-time = "2025-08-07T13:38:53.448Z" }, +] + +[[package]] +name = "idna" +version = "3.11" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5b/f5/4ec618ed16cc4f8fb3b701563655a69816155e79e24a17b651541804721d/markdown_it_py-4.0.0.tar.gz", hash = "sha256:cb0a2b4aa34f932c007117b194e945bd74e0ec24133ceb5bac59009cda1cb9f3", size = 73070, upload-time = "2025-08-11T12:57:52.854Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/54/e7d793b573f298e1c9013b8c4dade17d481164aa517d1d7148619c2cedbf/markdown_it_py-4.0.0-py3-none-any.whl", hash = "sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147", size = 87321, upload-time = "2025-08-11T12:57:51.923Z" }, +] + +[[package]] +name = "maturin" +version = "1.9.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tomli", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9a/35/c3370188492f4c139c7a318f438d01b8185c216303c49c4bc885c98b6afb/maturin-1.9.6.tar.gz", hash = "sha256:2c2ae37144811d365509889ed7220b0598487f1278c2441829c3abf56cc6324a", size = 214846, upload-time = "2025-10-07T12:45:08.408Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/55/5c/b435418ba4ba2647a1f7a95d53314991b1e556e656ae276dea993c3bce1d/maturin-1.9.6-py3-none-linux_armv6l.whl", hash = "sha256:26e3ab1a42a7145824210e9d763f6958f2c46afb1245ddd0bab7d78b1f59bb3f", size = 8134483, upload-time = "2025-10-07T12:44:44.274Z" }, + { url = "https://files.pythonhosted.org/packages/4d/1c/8e58eda6601f328b412cdeeaa88a9b6a10e591e2a73f313e8c0154d68385/maturin-1.9.6-py3-none-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:5263dda3f71feef2e4122baf5c4620e4b3710dbb7f2121f85a337182de214369", size = 15776470, upload-time = "2025-10-07T12:44:47.476Z" }, + { url = "https://files.pythonhosted.org/packages/6c/33/8c967cce6848cdd87a2e442c86120ac644b80c5ed4c32e3291bde6a17df8/maturin-1.9.6-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fe78262c2800c92f67d1ce3c0f6463f958a692cc67bfb572e5dbf5b4b696a8ba", size = 8226557, upload-time = "2025-10-07T12:44:49.844Z" }, + { url = "https://files.pythonhosted.org/packages/58/bd/3e2675cdc8b7270700ba30c663c852a35694441732a107ac30ebd6878bd8/maturin-1.9.6-py3-none-manylinux_2_12_i686.manylinux2010_i686.musllinux_1_1_i686.whl", hash = "sha256:7ab827c6e8c022eb2e1e7fb6deede54549c8460b20ccc2e9268cc6e8cde957a8", size = 8166544, upload-time = "2025-10-07T12:44:51.396Z" }, + { url = "https://files.pythonhosted.org/packages/58/1f/a2047ddf2230e700d5f8a13dd4b9af5ce806ad380c32e58105888205926e/maturin-1.9.6-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl", hash = "sha256:0246202377c49449315305209f45c8ecef6e2d6bd27a04b5b6f1ab3e4ea47238", size = 8641010, upload-time = "2025-10-07T12:44:53.658Z" }, + { url = "https://files.pythonhosted.org/packages/be/1f/265d63c7aa6faf363d4a3f23396f51bc6b4d5c7680a4190ae68dba25dea2/maturin-1.9.6-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:f5bac167700fbb6f8c8ed1a97b494522554b4432d7578e11403b894b6a91d99f", size = 7965945, upload-time = "2025-10-07T12:44:55.248Z" }, + { url = "https://files.pythonhosted.org/packages/4c/ca/a8e61979ccfe080948bcc1bddd79356157aee687134df7fb013050cec783/maturin-1.9.6-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.musllinux_1_1_armv7l.whl", hash = "sha256:7f53d3b1d8396d3fea3e1ee5fd37558bca5719090f3d194ba1c02b0b56327ae3", size = 7978820, upload-time = "2025-10-07T12:44:56.919Z" }, + { url = "https://files.pythonhosted.org/packages/bf/4a/81b412f8ad02a99801ef19ec059fba0822d1d28fb44cb6a92e722f05f278/maturin-1.9.6-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.musllinux_1_1_ppc64le.whl", hash = "sha256:7f506eb358386d94d6ec3208c003130cf4b69cab26034fc0cbbf8bf83afa4c2e", size = 10452064, upload-time = "2025-10-07T12:44:58.232Z" }, + { url = "https://files.pythonhosted.org/packages/5b/12/cc96c7a8cb51d8dcc9badd886c361caa1526fba7fa69d1e7892e613b71d4/maturin-1.9.6-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f2d6984ab690af509f525dbd2b130714207c06ebb14a5814edbe1e42b17ae0de", size = 8852401, upload-time = "2025-10-07T12:44:59.8Z" }, + { url = "https://files.pythonhosted.org/packages/51/8e/653ac3c9f2c25cdd81aefb0a2d17ff140ca5a14504f5e3c7f94dcfe4dbb7/maturin-1.9.6-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:5c2252b0956bb331460ac750c805ddf0d9b44442449fc1f16e3b66941689d0bc", size = 8425057, upload-time = "2025-10-07T12:45:01.711Z" }, + { url = "https://files.pythonhosted.org/packages/db/29/f13490328764ae9bfc1da55afc5b707cebe4fa75ad7a1573bfa82cfae0c6/maturin-1.9.6-py3-none-win32.whl", hash = "sha256:f2c58d29ebdd4346fd004e6be213d071fdd94a77a16aa91474a21a4f9dbf6309", size = 7165956, upload-time = "2025-10-07T12:45:03.766Z" }, + { url = "https://files.pythonhosted.org/packages/db/9f/dd51e5ac1fce47581b8efa03d77a03f928c0ef85b6e48a61dfa37b6b85a2/maturin-1.9.6-py3-none-win_amd64.whl", hash = "sha256:1b39a5d82572c240d20d9e8be024d722dfb311d330c5e28ddeb615211755941a", size = 8145722, upload-time = "2025-10-07T12:45:05.487Z" }, + { url = "https://files.pythonhosted.org/packages/65/f2/e97aaba6d0d78c5871771bf9dd71d4eb8dac15df9109cf452748d2207412/maturin-1.9.6-py3-none-win_arm64.whl", hash = "sha256:ac02a30083553d2a781c10cd6f5480119bf6692fd177e743267406cad2ad198c", size = 6857006, upload-time = "2025-10-07T12:45:06.813Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "mmh3" +version = "5.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a7/af/f28c2c2f51f31abb4725f9a64bc7863d5f491f6539bd26aee2a1d21a649e/mmh3-5.2.0.tar.gz", hash = "sha256:1efc8fec8478e9243a78bb993422cf79f8ff85cb4cf6b79647480a31e0d950a8", size = 33582, upload-time = "2025-07-29T07:43:48.49Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b9/2b/870f0ff5ecf312c58500f45950751f214b7068665e66e9bfd8bc2595587c/mmh3-5.2.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:81c504ad11c588c8629536b032940f2a359dda3b6cbfd4ad8f74cb24dcd1b0bc", size = 56119, upload-time = "2025-07-29T07:41:39.117Z" }, + { url = "https://files.pythonhosted.org/packages/3b/88/eb9a55b3f3cf43a74d6bfa8db0e2e209f966007777a1dc897c52c008314c/mmh3-5.2.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0b898cecff57442724a0f52bf42c2de42de63083a91008fb452887e372f9c328", size = 40634, upload-time = "2025-07-29T07:41:40.626Z" }, + { url = "https://files.pythonhosted.org/packages/d1/4c/8e4b3878bf8435c697d7ce99940a3784eb864521768069feaccaff884a17/mmh3-5.2.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:be1374df449465c9f2500e62eee73a39db62152a8bdfbe12ec5b5c1cd451344d", size = 40080, upload-time = "2025-07-29T07:41:41.791Z" }, + { url = "https://files.pythonhosted.org/packages/45/ac/0a254402c8c5ca424a0a9ebfe870f5665922f932830f0a11a517b6390a09/mmh3-5.2.0-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:b0d753ad566c721faa33db7e2e0eddd74b224cdd3eaf8481d76c926603c7a00e", size = 95321, upload-time = "2025-07-29T07:41:42.659Z" }, + { url = "https://files.pythonhosted.org/packages/39/8e/29306d5eca6dfda4b899d22c95b5420db4e0ffb7e0b6389b17379654ece5/mmh3-5.2.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:dfbead5575f6470c17e955b94f92d62a03dfc3d07f2e6f817d9b93dc211a1515", size = 101220, upload-time = "2025-07-29T07:41:43.572Z" }, + { url = "https://files.pythonhosted.org/packages/49/f7/0dd1368e531e52a17b5b8dd2f379cce813bff2d0978a7748a506f1231152/mmh3-5.2.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7434a27754049144539d2099a6d2da5d88b8bdeedf935180bf42ad59b3607aa3", size = 103991, upload-time = "2025-07-29T07:41:44.914Z" }, + { url = "https://files.pythonhosted.org/packages/35/06/abc7122c40f4abbfcef01d2dac6ec0b77ede9757e5be8b8a40a6265b1274/mmh3-5.2.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cadc16e8ea64b5d9a47363013e2bea469e121e6e7cb416a7593aeb24f2ad122e", size = 110894, upload-time = "2025-07-29T07:41:45.849Z" }, + { url = "https://files.pythonhosted.org/packages/f4/2f/837885759afa4baccb8e40456e1cf76a4f3eac835b878c727ae1286c5f82/mmh3-5.2.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d765058da196f68dc721116cab335e696e87e76720e6ef8ee5a24801af65e63d", size = 118327, upload-time = "2025-07-29T07:41:47.224Z" }, + { url = "https://files.pythonhosted.org/packages/40/cc/5683ba20a21bcfb3f1605b1c474f46d30354f728a7412201f59f453d405a/mmh3-5.2.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:8b0c53fe0994beade1ad7c0f13bd6fec980a0664bfbe5a6a7d64500b9ab76772", size = 101701, upload-time = "2025-07-29T07:41:48.259Z" }, + { url = "https://files.pythonhosted.org/packages/0e/24/99ab3fb940150aec8a26dbdfc39b200b5592f6aeb293ec268df93e054c30/mmh3-5.2.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:49037d417419863b222ae47ee562b2de9c3416add0a45c8d7f4e864be8dc4f89", size = 96712, upload-time = "2025-07-29T07:41:49.467Z" }, + { url = "https://files.pythonhosted.org/packages/61/04/d7c4cb18f1f001ede2e8aed0f9dbbfad03d161c9eea4fffb03f14f4523e5/mmh3-5.2.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:6ecb4e750d712abde046858ee6992b65c93f1f71b397fce7975c3860c07365d2", size = 110302, upload-time = "2025-07-29T07:41:50.387Z" }, + { url = "https://files.pythonhosted.org/packages/d8/bf/4dac37580cfda74425a4547500c36fa13ef581c8a756727c37af45e11e9a/mmh3-5.2.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:382a6bb3f8c6532ea084e7acc5be6ae0c6effa529240836d59352398f002e3fc", size = 111929, upload-time = "2025-07-29T07:41:51.348Z" }, + { url = "https://files.pythonhosted.org/packages/eb/b1/49f0a582c7a942fb71ddd1ec52b7d21d2544b37d2b2d994551346a15b4f6/mmh3-5.2.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:7733ec52296fc1ba22e9b90a245c821adbb943e98c91d8a330a2254612726106", size = 100111, upload-time = "2025-07-29T07:41:53.139Z" }, + { url = "https://files.pythonhosted.org/packages/dc/94/ccec09f438caeb2506f4c63bb3b99aa08a9e09880f8fc047295154756210/mmh3-5.2.0-cp310-cp310-win32.whl", hash = "sha256:127c95336f2a98c51e7682341ab7cb0be3adb9df0819ab8505a726ed1801876d", size = 40783, upload-time = "2025-07-29T07:41:54.463Z" }, + { url = "https://files.pythonhosted.org/packages/ea/f4/8d39a32c8203c1cdae88fdb04d1ea4aa178c20f159df97f4c5a2eaec702c/mmh3-5.2.0-cp310-cp310-win_amd64.whl", hash = "sha256:419005f84ba1cab47a77465a2a843562dadadd6671b8758bf179d82a15ca63eb", size = 41549, upload-time = "2025-07-29T07:41:55.295Z" }, + { url = "https://files.pythonhosted.org/packages/cc/a1/30efb1cd945e193f62574144dd92a0c9ee6463435e4e8ffce9b9e9f032f0/mmh3-5.2.0-cp310-cp310-win_arm64.whl", hash = "sha256:d22c9dcafed659fadc605538946c041722b6d1104fe619dbf5cc73b3c8a0ded8", size = 39335, upload-time = "2025-07-29T07:41:56.194Z" }, + { url = "https://files.pythonhosted.org/packages/f7/87/399567b3796e134352e11a8b973cd470c06b2ecfad5468fe580833be442b/mmh3-5.2.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:7901c893e704ee3c65f92d39b951f8f34ccf8e8566768c58103fb10e55afb8c1", size = 56107, upload-time = "2025-07-29T07:41:57.07Z" }, + { url = "https://files.pythonhosted.org/packages/c3/09/830af30adf8678955b247d97d3d9543dd2fd95684f3cd41c0cd9d291da9f/mmh3-5.2.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4a5f5536b1cbfa72318ab3bfc8a8188b949260baed186b75f0abc75b95d8c051", size = 40635, upload-time = "2025-07-29T07:41:57.903Z" }, + { url = "https://files.pythonhosted.org/packages/07/14/eaba79eef55b40d653321765ac5e8f6c9ac38780b8a7c2a2f8df8ee0fb72/mmh3-5.2.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:cedac4f4054b8f7859e5aed41aaa31ad03fce6851901a7fdc2af0275ac533c10", size = 40078, upload-time = "2025-07-29T07:41:58.772Z" }, + { url = "https://files.pythonhosted.org/packages/bb/26/83a0f852e763f81b2265d446b13ed6d49ee49e1fc0c47b9655977e6f3d81/mmh3-5.2.0-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:eb756caf8975882630ce4e9fbbeb9d3401242a72528230422c9ab3a0d278e60c", size = 97262, upload-time = "2025-07-29T07:41:59.678Z" }, + { url = "https://files.pythonhosted.org/packages/00/7d/b7133b10d12239aeaebf6878d7eaf0bf7d3738c44b4aba3c564588f6d802/mmh3-5.2.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:097e13c8b8a66c5753c6968b7640faefe85d8e38992703c1f666eda6ef4c3762", size = 103118, upload-time = "2025-07-29T07:42:01.197Z" }, + { url = "https://files.pythonhosted.org/packages/7b/3e/62f0b5dce2e22fd5b7d092aba285abd7959ea2b17148641e029f2eab1ffa/mmh3-5.2.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a7c0c7845566b9686480e6a7e9044db4afb60038d5fabd19227443f0104eeee4", size = 106072, upload-time = "2025-07-29T07:42:02.601Z" }, + { url = "https://files.pythonhosted.org/packages/66/84/ea88bb816edfe65052c757a1c3408d65c4201ddbd769d4a287b0f1a628b2/mmh3-5.2.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:61ac226af521a572700f863d6ecddc6ece97220ce7174e311948ff8c8919a363", size = 112925, upload-time = "2025-07-29T07:42:03.632Z" }, + { url = "https://files.pythonhosted.org/packages/2e/13/c9b1c022807db575fe4db806f442d5b5784547e2e82cff36133e58ea31c7/mmh3-5.2.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:582f9dbeefe15c32a5fa528b79b088b599a1dfe290a4436351c6090f90ddebb8", size = 120583, upload-time = "2025-07-29T07:42:04.991Z" }, + { url = "https://files.pythonhosted.org/packages/8a/5f/0e2dfe1a38f6a78788b7eb2b23432cee24623aeabbc907fed07fc17d6935/mmh3-5.2.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2ebfc46b39168ab1cd44670a32ea5489bcbc74a25795c61b6d888c5c2cf654ed", size = 99127, upload-time = "2025-07-29T07:42:05.929Z" }, + { url = "https://files.pythonhosted.org/packages/77/27/aefb7d663b67e6a0c4d61a513c83e39ba2237e8e4557fa7122a742a23de5/mmh3-5.2.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:1556e31e4bd0ac0c17eaf220be17a09c171d7396919c3794274cb3415a9d3646", size = 98544, upload-time = "2025-07-29T07:42:06.87Z" }, + { url = "https://files.pythonhosted.org/packages/ab/97/a21cc9b1a7c6e92205a1b5fa030cdf62277d177570c06a239eca7bd6dd32/mmh3-5.2.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:81df0dae22cd0da87f1c978602750f33d17fb3d21fb0f326c89dc89834fea79b", size = 106262, upload-time = "2025-07-29T07:42:07.804Z" }, + { url = "https://files.pythonhosted.org/packages/43/18/db19ae82ea63c8922a880e1498a75342311f8aa0c581c4dd07711473b5f7/mmh3-5.2.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:eba01ec3bd4a49b9ac5ca2bc6a73ff5f3af53374b8556fcc2966dd2af9eb7779", size = 109824, upload-time = "2025-07-29T07:42:08.735Z" }, + { url = "https://files.pythonhosted.org/packages/9f/f5/41dcf0d1969125fc6f61d8618b107c79130b5af50b18a4651210ea52ab40/mmh3-5.2.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:e9a011469b47b752e7d20de296bb34591cdfcbe76c99c2e863ceaa2aa61113d2", size = 97255, upload-time = "2025-07-29T07:42:09.706Z" }, + { url = "https://files.pythonhosted.org/packages/32/b3/cce9eaa0efac1f0e735bb178ef9d1d2887b4927fe0ec16609d5acd492dda/mmh3-5.2.0-cp311-cp311-win32.whl", hash = "sha256:bc44fc2b886243d7c0d8daeb37864e16f232e5b56aaec27cc781d848264cfd28", size = 40779, upload-time = "2025-07-29T07:42:10.546Z" }, + { url = "https://files.pythonhosted.org/packages/7c/e9/3fa0290122e6d5a7041b50ae500b8a9f4932478a51e48f209a3879fe0b9b/mmh3-5.2.0-cp311-cp311-win_amd64.whl", hash = "sha256:8ebf241072cf2777a492d0e09252f8cc2b3edd07dfdb9404b9757bffeb4f2cee", size = 41549, upload-time = "2025-07-29T07:42:11.399Z" }, + { url = "https://files.pythonhosted.org/packages/3a/54/c277475b4102588e6f06b2e9095ee758dfe31a149312cdbf62d39a9f5c30/mmh3-5.2.0-cp311-cp311-win_arm64.whl", hash = "sha256:b5f317a727bba0e633a12e71228bc6a4acb4f471a98b1c003163b917311ea9a9", size = 39336, upload-time = "2025-07-29T07:42:12.209Z" }, + { url = "https://files.pythonhosted.org/packages/bf/6a/d5aa7edb5c08e0bd24286c7d08341a0446f9a2fbbb97d96a8a6dd81935ee/mmh3-5.2.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:384eda9361a7bf83a85e09447e1feafe081034af9dd428893701b959230d84be", size = 56141, upload-time = "2025-07-29T07:42:13.456Z" }, + { url = "https://files.pythonhosted.org/packages/08/49/131d0fae6447bc4a7299ebdb1a6fb9d08c9f8dcf97d75ea93e8152ddf7ab/mmh3-5.2.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2c9da0d568569cc87315cb063486d761e38458b8ad513fedd3dc9263e1b81bcd", size = 40681, upload-time = "2025-07-29T07:42:14.306Z" }, + { url = "https://files.pythonhosted.org/packages/8f/6f/9221445a6bcc962b7f5ff3ba18ad55bba624bacdc7aa3fc0a518db7da8ec/mmh3-5.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:86d1be5d63232e6eb93c50881aea55ff06eb86d8e08f9b5417c8c9b10db9db96", size = 40062, upload-time = "2025-07-29T07:42:15.08Z" }, + { url = "https://files.pythonhosted.org/packages/1e/d4/6bb2d0fef81401e0bb4c297d1eb568b767de4ce6fc00890bc14d7b51ecc4/mmh3-5.2.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:bf7bee43e17e81671c447e9c83499f53d99bf440bc6d9dc26a841e21acfbe094", size = 97333, upload-time = "2025-07-29T07:42:16.436Z" }, + { url = "https://files.pythonhosted.org/packages/44/e0/ccf0daff8134efbb4fbc10a945ab53302e358c4b016ada9bf97a6bdd50c1/mmh3-5.2.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7aa18cdb58983ee660c9c400b46272e14fa253c675ed963d3812487f8ca42037", size = 103310, upload-time = "2025-07-29T07:42:17.796Z" }, + { url = "https://files.pythonhosted.org/packages/02/63/1965cb08a46533faca0e420e06aff8bbaf9690a6f0ac6ae6e5b2e4544687/mmh3-5.2.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ae9d032488fcec32d22be6542d1a836f00247f40f320844dbb361393b5b22773", size = 106178, upload-time = "2025-07-29T07:42:19.281Z" }, + { url = "https://files.pythonhosted.org/packages/c2/41/c883ad8e2c234013f27f92061200afc11554ea55edd1bcf5e1accd803a85/mmh3-5.2.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1861fb6b1d0453ed7293200139c0a9011eeb1376632e048e3766945b13313c5", size = 113035, upload-time = "2025-07-29T07:42:20.356Z" }, + { url = "https://files.pythonhosted.org/packages/df/b5/1ccade8b1fa625d634a18bab7bf08a87457e09d5ec8cf83ca07cbea9d400/mmh3-5.2.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:99bb6a4d809aa4e528ddfe2c85dd5239b78b9dd14be62cca0329db78505e7b50", size = 120784, upload-time = "2025-07-29T07:42:21.377Z" }, + { url = "https://files.pythonhosted.org/packages/77/1c/919d9171fcbdcdab242e06394464ccf546f7d0f3b31e0d1e3a630398782e/mmh3-5.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1f8d8b627799f4e2fcc7c034fed8f5f24dc7724ff52f69838a3d6d15f1ad4765", size = 99137, upload-time = "2025-07-29T07:42:22.344Z" }, + { url = "https://files.pythonhosted.org/packages/66/8a/1eebef5bd6633d36281d9fc83cf2e9ba1ba0e1a77dff92aacab83001cee4/mmh3-5.2.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:b5995088dd7023d2d9f310a0c67de5a2b2e06a570ecfd00f9ff4ab94a67cde43", size = 98664, upload-time = "2025-07-29T07:42:23.269Z" }, + { url = "https://files.pythonhosted.org/packages/13/41/a5d981563e2ee682b21fb65e29cc0f517a6734a02b581359edd67f9d0360/mmh3-5.2.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1a5f4d2e59d6bba8ef01b013c472741835ad961e7c28f50c82b27c57748744a4", size = 106459, upload-time = "2025-07-29T07:42:24.238Z" }, + { url = "https://files.pythonhosted.org/packages/24/31/342494cd6ab792d81e083680875a2c50fa0c5df475ebf0b67784f13e4647/mmh3-5.2.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fd6e6c3d90660d085f7e73710eab6f5545d4854b81b0135a3526e797009dbda3", size = 110038, upload-time = "2025-07-29T07:42:25.629Z" }, + { url = "https://files.pythonhosted.org/packages/28/44/efda282170a46bb4f19c3e2b90536513b1d821c414c28469a227ca5a1789/mmh3-5.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c4a2f3d83879e3de2eb8cbf562e71563a8ed15ee9b9c2e77ca5d9f73072ac15c", size = 97545, upload-time = "2025-07-29T07:42:27.04Z" }, + { url = "https://files.pythonhosted.org/packages/68/8f/534ae319c6e05d714f437e7206f78c17e66daca88164dff70286b0e8ea0c/mmh3-5.2.0-cp312-cp312-win32.whl", hash = "sha256:2421b9d665a0b1ad724ec7332fb5a98d075f50bc51a6ff854f3a1882bd650d49", size = 40805, upload-time = "2025-07-29T07:42:28.032Z" }, + { url = "https://files.pythonhosted.org/packages/b8/f6/f6abdcfefcedab3c964868048cfe472764ed358c2bf6819a70dd4ed4ed3a/mmh3-5.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:72d80005b7634a3a2220f81fbeb94775ebd12794623bb2e1451701ea732b4aa3", size = 41597, upload-time = "2025-07-29T07:42:28.894Z" }, + { url = "https://files.pythonhosted.org/packages/15/fd/f7420e8cbce45c259c770cac5718badf907b302d3a99ec587ba5ce030237/mmh3-5.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:3d6bfd9662a20c054bc216f861fa330c2dac7c81e7fb8307b5e32ab5b9b4d2e0", size = 39350, upload-time = "2025-07-29T07:42:29.794Z" }, +] + +[[package]] +name = "packaging" +version = "25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pyarrow" +version = "22.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/30/53/04a7fdc63e6056116c9ddc8b43bc28c12cdd181b85cbeadb79278475f3ae/pyarrow-22.0.0.tar.gz", hash = "sha256:3d600dc583260d845c7d8a6db540339dd883081925da2bd1c5cb808f720b3cd9", size = 1151151, upload-time = "2025-10-24T12:30:00.762Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/9b/cb3f7e0a345353def531ca879053e9ef6b9f38ed91aebcf68b09ba54dec0/pyarrow-22.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:77718810bd3066158db1e95a63c160ad7ce08c6b0710bc656055033e39cdad88", size = 34223968, upload-time = "2025-10-24T10:03:31.21Z" }, + { url = "https://files.pythonhosted.org/packages/6c/41/3184b8192a120306270c5307f105b70320fdaa592c99843c5ef78aaefdcf/pyarrow-22.0.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:44d2d26cda26d18f7af7db71453b7b783788322d756e81730acb98f24eb90ace", size = 35942085, upload-time = "2025-10-24T10:03:38.146Z" }, + { url = "https://files.pythonhosted.org/packages/d9/3d/a1eab2f6f08001f9fb714b8ed5cfb045e2fe3e3e3c0c221f2c9ed1e6d67d/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b9d71701ce97c95480fecb0039ec5bb889e75f110da72005743451339262f4ce", size = 44964613, upload-time = "2025-10-24T10:03:46.516Z" }, + { url = "https://files.pythonhosted.org/packages/46/46/a1d9c24baf21cfd9ce994ac820a24608decf2710521b29223d4334985127/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:710624ab925dc2b05a6229d47f6f0dac1c1155e6ed559be7109f684eba048a48", size = 47627059, upload-time = "2025-10-24T10:03:55.353Z" }, + { url = "https://files.pythonhosted.org/packages/3a/4c/f711acb13075c1391fd54bc17e078587672c575f8de2a6e62509af026dcf/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f963ba8c3b0199f9d6b794c90ec77545e05eadc83973897a4523c9e8d84e9340", size = 47947043, upload-time = "2025-10-24T10:04:05.408Z" }, + { url = "https://files.pythonhosted.org/packages/4e/70/1f3180dd7c2eab35c2aca2b29ace6c519f827dcd4cfeb8e0dca41612cf7a/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:bd0d42297ace400d8febe55f13fdf46e86754842b860c978dfec16f081e5c653", size = 50206505, upload-time = "2025-10-24T10:04:15.786Z" }, + { url = "https://files.pythonhosted.org/packages/80/07/fea6578112c8c60ffde55883a571e4c4c6bc7049f119d6b09333b5cc6f73/pyarrow-22.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:00626d9dc0f5ef3a75fe63fd68b9c7c8302d2b5bbc7f74ecaedba83447a24f84", size = 28101641, upload-time = "2025-10-24T10:04:22.57Z" }, + { url = "https://files.pythonhosted.org/packages/2e/b7/18f611a8cdc43417f9394a3ccd3eace2f32183c08b9eddc3d17681819f37/pyarrow-22.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:3e294c5eadfb93d78b0763e859a0c16d4051fc1c5231ae8956d61cb0b5666f5a", size = 34272022, upload-time = "2025-10-24T10:04:28.973Z" }, + { url = "https://files.pythonhosted.org/packages/26/5c/f259e2526c67eb4b9e511741b19870a02363a47a35edbebc55c3178db22d/pyarrow-22.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:69763ab2445f632d90b504a815a2a033f74332997052b721002298ed6de40f2e", size = 35995834, upload-time = "2025-10-24T10:04:35.467Z" }, + { url = "https://files.pythonhosted.org/packages/50/8d/281f0f9b9376d4b7f146913b26fac0aa2829cd1ee7e997f53a27411bbb92/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:b41f37cabfe2463232684de44bad753d6be08a7a072f6a83447eeaf0e4d2a215", size = 45030348, upload-time = "2025-10-24T10:04:43.366Z" }, + { url = "https://files.pythonhosted.org/packages/f5/e5/53c0a1c428f0976bf22f513d79c73000926cb00b9c138d8e02daf2102e18/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:35ad0f0378c9359b3f297299c3309778bb03b8612f987399a0333a560b43862d", size = 47699480, upload-time = "2025-10-24T10:04:51.486Z" }, + { url = "https://files.pythonhosted.org/packages/95/e1/9dbe4c465c3365959d183e6345d0a8d1dc5b02ca3f8db4760b3bc834cf25/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8382ad21458075c2e66a82a29d650f963ce51c7708c7c0ff313a8c206c4fd5e8", size = 48011148, upload-time = "2025-10-24T10:04:59.585Z" }, + { url = "https://files.pythonhosted.org/packages/c5/b4/7caf5d21930061444c3cf4fa7535c82faf5263e22ce43af7c2759ceb5b8b/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1a812a5b727bc09c3d7ea072c4eebf657c2f7066155506ba31ebf4792f88f016", size = 50276964, upload-time = "2025-10-24T10:05:08.175Z" }, + { url = "https://files.pythonhosted.org/packages/ae/f3/cec89bd99fa3abf826f14d4e53d3d11340ce6f6af4d14bdcd54cd83b6576/pyarrow-22.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:ec5d40dd494882704fb876c16fa7261a69791e784ae34e6b5992e977bd2e238c", size = 28106517, upload-time = "2025-10-24T10:05:14.314Z" }, + { url = "https://files.pythonhosted.org/packages/af/63/ba23862d69652f85b615ca14ad14f3bcfc5bf1b99ef3f0cd04ff93fdad5a/pyarrow-22.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:bea79263d55c24a32b0d79c00a1c58bb2ee5f0757ed95656b01c0fb310c5af3d", size = 34211578, upload-time = "2025-10-24T10:05:21.583Z" }, + { url = "https://files.pythonhosted.org/packages/b1/d0/f9ad86fe809efd2bcc8be32032fa72e8b0d112b01ae56a053006376c5930/pyarrow-22.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:12fe549c9b10ac98c91cf791d2945e878875d95508e1a5d14091a7aaa66d9cf8", size = 35989906, upload-time = "2025-10-24T10:05:29.485Z" }, + { url = "https://files.pythonhosted.org/packages/b4/a8/f910afcb14630e64d673f15904ec27dd31f1e009b77033c365c84e8c1e1d/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:334f900ff08ce0423407af97e6c26ad5d4e3b0763645559ece6fbf3747d6a8f5", size = 45021677, upload-time = "2025-10-24T10:05:38.274Z" }, + { url = "https://files.pythonhosted.org/packages/13/95/aec81f781c75cd10554dc17a25849c720d54feafb6f7847690478dcf5ef8/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:c6c791b09c57ed76a18b03f2631753a4960eefbbca80f846da8baefc6491fcfe", size = 47726315, upload-time = "2025-10-24T10:05:47.314Z" }, + { url = "https://files.pythonhosted.org/packages/bb/d4/74ac9f7a54cfde12ee42734ea25d5a3c9a45db78f9def949307a92720d37/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c3200cb41cdbc65156e5f8c908d739b0dfed57e890329413da2748d1a2cd1a4e", size = 47990906, upload-time = "2025-10-24T10:05:58.254Z" }, + { url = "https://files.pythonhosted.org/packages/2e/71/fedf2499bf7a95062eafc989ace56572f3343432570e1c54e6599d5b88da/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ac93252226cf288753d8b46280f4edf3433bf9508b6977f8dd8526b521a1bbb9", size = 50306783, upload-time = "2025-10-24T10:06:08.08Z" }, + { url = "https://files.pythonhosted.org/packages/68/ed/b202abd5a5b78f519722f3d29063dda03c114711093c1995a33b8e2e0f4b/pyarrow-22.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:44729980b6c50a5f2bfcc2668d36c569ce17f8b17bccaf470c4313dcbbf13c9d", size = 27972883, upload-time = "2025-10-24T10:06:14.204Z" }, +] + +[[package]] +name = "pydantic" +version = "2.11.10" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/54/ecab642b3bed45f7d5f59b38443dcb36ef50f85af192e6ece103dbfe9587/pydantic-2.11.10.tar.gz", hash = "sha256:dc280f0982fbda6c38fada4e476dc0a4f3aeaf9c6ad4c28df68a666ec3c61423", size = 788494, upload-time = "2025-10-04T10:40:41.338Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl", hash = "sha256:802a655709d49bd004c31e865ef37da30b540786a46bfce02333e0e24b5fe29a", size = 444823, upload-time = "2025-10-04T10:40:39.055Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.33.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ad/88/5f2260bdfae97aabf98f1778d43f69574390ad787afb646292a638c923d4/pydantic_core-2.33.2.tar.gz", hash = "sha256:7cb8bc3605c29176e1b105350d2e6474142d7c1bd1d9327c4a9bdb46bf827acc", size = 435195, upload-time = "2025-04-23T18:33:52.104Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/92/b31726561b5dae176c2d2c2dc43a9c5bfba5d32f96f8b4c0a600dd492447/pydantic_core-2.33.2-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:2b3d326aaef0c0399d9afffeb6367d5e26ddc24d351dbc9c636840ac355dc5d8", size = 2028817, upload-time = "2025-04-23T18:30:43.919Z" }, + { url = "https://files.pythonhosted.org/packages/a3/44/3f0b95fafdaca04a483c4e685fe437c6891001bf3ce8b2fded82b9ea3aa1/pydantic_core-2.33.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0e5b2671f05ba48b94cb90ce55d8bdcaaedb8ba00cc5359f6810fc918713983d", size = 1861357, upload-time = "2025-04-23T18:30:46.372Z" }, + { url = "https://files.pythonhosted.org/packages/30/97/e8f13b55766234caae05372826e8e4b3b96e7b248be3157f53237682e43c/pydantic_core-2.33.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0069c9acc3f3981b9ff4cdfaf088e98d83440a4c7ea1bc07460af3d4dc22e72d", size = 1898011, upload-time = "2025-04-23T18:30:47.591Z" }, + { url = "https://files.pythonhosted.org/packages/9b/a3/99c48cf7bafc991cc3ee66fd544c0aae8dc907b752f1dad2d79b1b5a471f/pydantic_core-2.33.2-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d53b22f2032c42eaaf025f7c40c2e3b94568ae077a606f006d206a463bc69572", size = 1982730, upload-time = "2025-04-23T18:30:49.328Z" }, + { url = "https://files.pythonhosted.org/packages/de/8e/a5b882ec4307010a840fb8b58bd9bf65d1840c92eae7534c7441709bf54b/pydantic_core-2.33.2-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0405262705a123b7ce9f0b92f123334d67b70fd1f20a9372b907ce1080c7ba02", size = 2136178, upload-time = "2025-04-23T18:30:50.907Z" }, + { url = "https://files.pythonhosted.org/packages/e4/bb/71e35fc3ed05af6834e890edb75968e2802fe98778971ab5cba20a162315/pydantic_core-2.33.2-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4b25d91e288e2c4e0662b8038a28c6a07eaac3e196cfc4ff69de4ea3db992a1b", size = 2736462, upload-time = "2025-04-23T18:30:52.083Z" }, + { url = "https://files.pythonhosted.org/packages/31/0d/c8f7593e6bc7066289bbc366f2235701dcbebcd1ff0ef8e64f6f239fb47d/pydantic_core-2.33.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6bdfe4b3789761f3bcb4b1ddf33355a71079858958e3a552f16d5af19768fef2", size = 2005652, upload-time = "2025-04-23T18:30:53.389Z" }, + { url = "https://files.pythonhosted.org/packages/d2/7a/996d8bd75f3eda405e3dd219ff5ff0a283cd8e34add39d8ef9157e722867/pydantic_core-2.33.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:efec8db3266b76ef9607c2c4c419bdb06bf335ae433b80816089ea7585816f6a", size = 2113306, upload-time = "2025-04-23T18:30:54.661Z" }, + { url = "https://files.pythonhosted.org/packages/ff/84/daf2a6fb2db40ffda6578a7e8c5a6e9c8affb251a05c233ae37098118788/pydantic_core-2.33.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:031c57d67ca86902726e0fae2214ce6770bbe2f710dc33063187a68744a5ecac", size = 2073720, upload-time = "2025-04-23T18:30:56.11Z" }, + { url = "https://files.pythonhosted.org/packages/77/fb/2258da019f4825128445ae79456a5499c032b55849dbd5bed78c95ccf163/pydantic_core-2.33.2-cp310-cp310-musllinux_1_1_armv7l.whl", hash = "sha256:f8de619080e944347f5f20de29a975c2d815d9ddd8be9b9b7268e2e3ef68605a", size = 2244915, upload-time = "2025-04-23T18:30:57.501Z" }, + { url = "https://files.pythonhosted.org/packages/d8/7a/925ff73756031289468326e355b6fa8316960d0d65f8b5d6b3a3e7866de7/pydantic_core-2.33.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:73662edf539e72a9440129f231ed3757faab89630d291b784ca99237fb94db2b", size = 2241884, upload-time = "2025-04-23T18:30:58.867Z" }, + { url = "https://files.pythonhosted.org/packages/0b/b0/249ee6d2646f1cdadcb813805fe76265745c4010cf20a8eba7b0e639d9b2/pydantic_core-2.33.2-cp310-cp310-win32.whl", hash = "sha256:0a39979dcbb70998b0e505fb1556a1d550a0781463ce84ebf915ba293ccb7e22", size = 1910496, upload-time = "2025-04-23T18:31:00.078Z" }, + { url = "https://files.pythonhosted.org/packages/66/ff/172ba8f12a42d4b552917aa65d1f2328990d3ccfc01d5b7c943ec084299f/pydantic_core-2.33.2-cp310-cp310-win_amd64.whl", hash = "sha256:b0379a2b24882fef529ec3b4987cb5d003b9cda32256024e6fe1586ac45fc640", size = 1955019, upload-time = "2025-04-23T18:31:01.335Z" }, + { url = "https://files.pythonhosted.org/packages/3f/8d/71db63483d518cbbf290261a1fc2839d17ff89fce7089e08cad07ccfce67/pydantic_core-2.33.2-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:4c5b0a576fb381edd6d27f0a85915c6daf2f8138dc5c267a57c08a62900758c7", size = 2028584, upload-time = "2025-04-23T18:31:03.106Z" }, + { url = "https://files.pythonhosted.org/packages/24/2f/3cfa7244ae292dd850989f328722d2aef313f74ffc471184dc509e1e4e5a/pydantic_core-2.33.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e799c050df38a639db758c617ec771fd8fb7a5f8eaaa4b27b101f266b216a246", size = 1855071, upload-time = "2025-04-23T18:31:04.621Z" }, + { url = "https://files.pythonhosted.org/packages/b3/d3/4ae42d33f5e3f50dd467761304be2fa0a9417fbf09735bc2cce003480f2a/pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dc46a01bf8d62f227d5ecee74178ffc448ff4e5197c756331f71efcc66dc980f", size = 1897823, upload-time = "2025-04-23T18:31:06.377Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f3/aa5976e8352b7695ff808599794b1fba2a9ae2ee954a3426855935799488/pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a144d4f717285c6d9234a66778059f33a89096dfb9b39117663fd8413d582dcc", size = 1983792, upload-time = "2025-04-23T18:31:07.93Z" }, + { url = "https://files.pythonhosted.org/packages/d5/7a/cda9b5a23c552037717f2b2a5257e9b2bfe45e687386df9591eff7b46d28/pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:73cf6373c21bc80b2e0dc88444f41ae60b2f070ed02095754eb5a01df12256de", size = 2136338, upload-time = "2025-04-23T18:31:09.283Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/b8f9ec8dd1417eb9da784e91e1667d58a2a4a7b7b34cf4af765ef663a7e5/pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3dc625f4aa79713512d1976fe9f0bc99f706a9dee21dfd1810b4bbbf228d0e8a", size = 2730998, upload-time = "2025-04-23T18:31:11.7Z" }, + { url = "https://files.pythonhosted.org/packages/47/bc/cd720e078576bdb8255d5032c5d63ee5c0bf4b7173dd955185a1d658c456/pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:881b21b5549499972441da4758d662aeea93f1923f953e9cbaff14b8b9565aef", size = 2003200, upload-time = "2025-04-23T18:31:13.536Z" }, + { url = "https://files.pythonhosted.org/packages/ca/22/3602b895ee2cd29d11a2b349372446ae9727c32e78a94b3d588a40fdf187/pydantic_core-2.33.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:bdc25f3681f7b78572699569514036afe3c243bc3059d3942624e936ec93450e", size = 2113890, upload-time = "2025-04-23T18:31:15.011Z" }, + { url = "https://files.pythonhosted.org/packages/ff/e6/e3c5908c03cf00d629eb38393a98fccc38ee0ce8ecce32f69fc7d7b558a7/pydantic_core-2.33.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:fe5b32187cbc0c862ee201ad66c30cf218e5ed468ec8dc1cf49dec66e160cc4d", size = 2073359, upload-time = "2025-04-23T18:31:16.393Z" }, + { url = "https://files.pythonhosted.org/packages/12/e7/6a36a07c59ebefc8777d1ffdaf5ae71b06b21952582e4b07eba88a421c79/pydantic_core-2.33.2-cp311-cp311-musllinux_1_1_armv7l.whl", hash = "sha256:bc7aee6f634a6f4a95676fcb5d6559a2c2a390330098dba5e5a5f28a2e4ada30", size = 2245883, upload-time = "2025-04-23T18:31:17.892Z" }, + { url = "https://files.pythonhosted.org/packages/16/3f/59b3187aaa6cc0c1e6616e8045b284de2b6a87b027cce2ffcea073adf1d2/pydantic_core-2.33.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:235f45e5dbcccf6bd99f9f472858849f73d11120d76ea8707115415f8e5ebebf", size = 2241074, upload-time = "2025-04-23T18:31:19.205Z" }, + { url = "https://files.pythonhosted.org/packages/e0/ed/55532bb88f674d5d8f67ab121a2a13c385df382de2a1677f30ad385f7438/pydantic_core-2.33.2-cp311-cp311-win32.whl", hash = "sha256:6368900c2d3ef09b69cb0b913f9f8263b03786e5b2a387706c5afb66800efd51", size = 1910538, upload-time = "2025-04-23T18:31:20.541Z" }, + { url = "https://files.pythonhosted.org/packages/fe/1b/25b7cccd4519c0b23c2dd636ad39d381abf113085ce4f7bec2b0dc755eb1/pydantic_core-2.33.2-cp311-cp311-win_amd64.whl", hash = "sha256:1e063337ef9e9820c77acc768546325ebe04ee38b08703244c1309cccc4f1bab", size = 1952909, upload-time = "2025-04-23T18:31:22.371Z" }, + { url = "https://files.pythonhosted.org/packages/49/a9/d809358e49126438055884c4366a1f6227f0f84f635a9014e2deb9b9de54/pydantic_core-2.33.2-cp311-cp311-win_arm64.whl", hash = "sha256:6b99022f1d19bc32a4c2a0d544fc9a76e3be90f0b3f4af413f87d38749300e65", size = 1897786, upload-time = "2025-04-23T18:31:24.161Z" }, + { url = "https://files.pythonhosted.org/packages/18/8a/2b41c97f554ec8c71f2a8a5f85cb56a8b0956addfe8b0efb5b3d77e8bdc3/pydantic_core-2.33.2-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a7ec89dc587667f22b6a0b6579c249fca9026ce7c333fc142ba42411fa243cdc", size = 2009000, upload-time = "2025-04-23T18:31:25.863Z" }, + { url = "https://files.pythonhosted.org/packages/a1/02/6224312aacb3c8ecbaa959897af57181fb6cf3a3d7917fd44d0f2917e6f2/pydantic_core-2.33.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3c6db6e52c6d70aa0d00d45cdb9b40f0433b96380071ea80b09277dba021ddf7", size = 1847996, upload-time = "2025-04-23T18:31:27.341Z" }, + { url = "https://files.pythonhosted.org/packages/d6/46/6dcdf084a523dbe0a0be59d054734b86a981726f221f4562aed313dbcb49/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e61206137cbc65e6d5256e1166f88331d3b6238e082d9f74613b9b765fb9025", size = 1880957, upload-time = "2025-04-23T18:31:28.956Z" }, + { url = "https://files.pythonhosted.org/packages/ec/6b/1ec2c03837ac00886ba8160ce041ce4e325b41d06a034adbef11339ae422/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:eb8c529b2819c37140eb51b914153063d27ed88e3bdc31b71198a198e921e011", size = 1964199, upload-time = "2025-04-23T18:31:31.025Z" }, + { url = "https://files.pythonhosted.org/packages/2d/1d/6bf34d6adb9debd9136bd197ca72642203ce9aaaa85cfcbfcf20f9696e83/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c52b02ad8b4e2cf14ca7b3d918f3eb0ee91e63b3167c32591e57c4317e134f8f", size = 2120296, upload-time = "2025-04-23T18:31:32.514Z" }, + { url = "https://files.pythonhosted.org/packages/e0/94/2bd0aaf5a591e974b32a9f7123f16637776c304471a0ab33cf263cf5591a/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:96081f1605125ba0855dfda83f6f3df5ec90c61195421ba72223de35ccfb2f88", size = 2676109, upload-time = "2025-04-23T18:31:33.958Z" }, + { url = "https://files.pythonhosted.org/packages/f9/41/4b043778cf9c4285d59742281a769eac371b9e47e35f98ad321349cc5d61/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f57a69461af2a5fa6e6bbd7a5f60d3b7e6cebb687f55106933188e79ad155c1", size = 2002028, upload-time = "2025-04-23T18:31:39.095Z" }, + { url = "https://files.pythonhosted.org/packages/cb/d5/7bb781bf2748ce3d03af04d5c969fa1308880e1dca35a9bd94e1a96a922e/pydantic_core-2.33.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:572c7e6c8bb4774d2ac88929e3d1f12bc45714ae5ee6d9a788a9fb35e60bb04b", size = 2100044, upload-time = "2025-04-23T18:31:41.034Z" }, + { url = "https://files.pythonhosted.org/packages/fe/36/def5e53e1eb0ad896785702a5bbfd25eed546cdcf4087ad285021a90ed53/pydantic_core-2.33.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:db4b41f9bd95fbe5acd76d89920336ba96f03e149097365afe1cb092fceb89a1", size = 2058881, upload-time = "2025-04-23T18:31:42.757Z" }, + { url = "https://files.pythonhosted.org/packages/01/6c/57f8d70b2ee57fc3dc8b9610315949837fa8c11d86927b9bb044f8705419/pydantic_core-2.33.2-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:fa854f5cf7e33842a892e5c73f45327760bc7bc516339fda888c75ae60edaeb6", size = 2227034, upload-time = "2025-04-23T18:31:44.304Z" }, + { url = "https://files.pythonhosted.org/packages/27/b9/9c17f0396a82b3d5cbea4c24d742083422639e7bb1d5bf600e12cb176a13/pydantic_core-2.33.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:5f483cfb75ff703095c59e365360cb73e00185e01aaea067cd19acffd2ab20ea", size = 2234187, upload-time = "2025-04-23T18:31:45.891Z" }, + { url = "https://files.pythonhosted.org/packages/b0/6a/adf5734ffd52bf86d865093ad70b2ce543415e0e356f6cacabbc0d9ad910/pydantic_core-2.33.2-cp312-cp312-win32.whl", hash = "sha256:9cb1da0f5a471435a7bc7e439b8a728e8b61e59784b2af70d7c169f8dd8ae290", size = 1892628, upload-time = "2025-04-23T18:31:47.819Z" }, + { url = "https://files.pythonhosted.org/packages/43/e4/5479fecb3606c1368d496a825d8411e126133c41224c1e7238be58b87d7e/pydantic_core-2.33.2-cp312-cp312-win_amd64.whl", hash = "sha256:f941635f2a3d96b2973e867144fde513665c87f13fe0e193c158ac51bfaaa7b2", size = 1955866, upload-time = "2025-04-23T18:31:49.635Z" }, + { url = "https://files.pythonhosted.org/packages/0d/24/8b11e8b3e2be9dd82df4b11408a67c61bb4dc4f8e11b5b0fc888b38118b5/pydantic_core-2.33.2-cp312-cp312-win_arm64.whl", hash = "sha256:cca3868ddfaccfbc4bfb1d608e2ccaaebe0ae628e1416aeb9c4d88c001bb45ab", size = 1888894, upload-time = "2025-04-23T18:31:51.609Z" }, + { url = "https://files.pythonhosted.org/packages/30/68/373d55e58b7e83ce371691f6eaa7175e3a24b956c44628eb25d7da007917/pydantic_core-2.33.2-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:5c4aa4e82353f65e548c476b37e64189783aa5384903bfea4f41580f255fddfa", size = 2023982, upload-time = "2025-04-23T18:32:53.14Z" }, + { url = "https://files.pythonhosted.org/packages/a4/16/145f54ac08c96a63d8ed6442f9dec17b2773d19920b627b18d4f10a061ea/pydantic_core-2.33.2-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:d946c8bf0d5c24bf4fe333af284c59a19358aa3ec18cb3dc4370080da1e8ad29", size = 1858412, upload-time = "2025-04-23T18:32:55.52Z" }, + { url = "https://files.pythonhosted.org/packages/41/b1/c6dc6c3e2de4516c0bb2c46f6a373b91b5660312342a0cf5826e38ad82fa/pydantic_core-2.33.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:87b31b6846e361ef83fedb187bb5b4372d0da3f7e28d85415efa92d6125d6e6d", size = 1892749, upload-time = "2025-04-23T18:32:57.546Z" }, + { url = "https://files.pythonhosted.org/packages/12/73/8cd57e20afba760b21b742106f9dbdfa6697f1570b189c7457a1af4cd8a0/pydantic_core-2.33.2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aa9d91b338f2df0508606f7009fde642391425189bba6d8c653afd80fd6bb64e", size = 2067527, upload-time = "2025-04-23T18:32:59.771Z" }, + { url = "https://files.pythonhosted.org/packages/e3/d5/0bb5d988cc019b3cba4a78f2d4b3854427fc47ee8ec8e9eaabf787da239c/pydantic_core-2.33.2-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2058a32994f1fde4ca0480ab9d1e75a0e8c87c22b53a3ae66554f9af78f2fe8c", size = 2108225, upload-time = "2025-04-23T18:33:04.51Z" }, + { url = "https://files.pythonhosted.org/packages/f1/c5/00c02d1571913d496aabf146106ad8239dc132485ee22efe08085084ff7c/pydantic_core-2.33.2-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:0e03262ab796d986f978f79c943fc5f620381be7287148b8010b4097f79a39ec", size = 2069490, upload-time = "2025-04-23T18:33:06.391Z" }, + { url = "https://files.pythonhosted.org/packages/22/a8/dccc38768274d3ed3a59b5d06f59ccb845778687652daa71df0cab4040d7/pydantic_core-2.33.2-pp310-pypy310_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:1a8695a8d00c73e50bff9dfda4d540b7dee29ff9b8053e38380426a85ef10052", size = 2237525, upload-time = "2025-04-23T18:33:08.44Z" }, + { url = "https://files.pythonhosted.org/packages/d4/e7/4f98c0b125dda7cf7ccd14ba936218397b44f50a56dd8c16a3091df116c3/pydantic_core-2.33.2-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:fa754d1850735a0b0e03bcffd9d4b4343eb417e47196e4485d9cca326073a42c", size = 2238446, upload-time = "2025-04-23T18:33:10.313Z" }, + { url = "https://files.pythonhosted.org/packages/ce/91/2ec36480fdb0b783cd9ef6795753c1dea13882f2e68e73bce76ae8c21e6a/pydantic_core-2.33.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:a11c8d26a50bfab49002947d3d237abe4d9e4b5bdc8846a63537b6488e197808", size = 2066678, upload-time = "2025-04-23T18:33:12.224Z" }, + { url = "https://files.pythonhosted.org/packages/7b/27/d4ae6487d73948d6f20dddcd94be4ea43e74349b56eba82e9bdee2d7494c/pydantic_core-2.33.2-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:dd14041875d09cc0f9308e37a6f8b65f5585cf2598a53aa0123df8b129d481f8", size = 2025200, upload-time = "2025-04-23T18:33:14.199Z" }, + { url = "https://files.pythonhosted.org/packages/f1/b8/b3cb95375f05d33801024079b9392a5ab45267a63400bf1866e7ce0f0de4/pydantic_core-2.33.2-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:d87c561733f66531dced0da6e864f44ebf89a8fba55f31407b00c2f7f9449593", size = 1859123, upload-time = "2025-04-23T18:33:16.555Z" }, + { url = "https://files.pythonhosted.org/packages/05/bc/0d0b5adeda59a261cd30a1235a445bf55c7e46ae44aea28f7bd6ed46e091/pydantic_core-2.33.2-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2f82865531efd18d6e07a04a17331af02cb7a651583c418df8266f17a63c6612", size = 1892852, upload-time = "2025-04-23T18:33:18.513Z" }, + { url = "https://files.pythonhosted.org/packages/3e/11/d37bdebbda2e449cb3f519f6ce950927b56d62f0b84fd9cb9e372a26a3d5/pydantic_core-2.33.2-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2bfb5112df54209d820d7bf9317c7a6c9025ea52e49f46b6a2060104bba37de7", size = 2067484, upload-time = "2025-04-23T18:33:20.475Z" }, + { url = "https://files.pythonhosted.org/packages/8c/55/1f95f0a05ce72ecb02a8a8a1c3be0579bbc29b1d5ab68f1378b7bebc5057/pydantic_core-2.33.2-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:64632ff9d614e5eecfb495796ad51b0ed98c453e447a76bcbeeb69615079fc7e", size = 2108896, upload-time = "2025-04-23T18:33:22.501Z" }, + { url = "https://files.pythonhosted.org/packages/53/89/2b2de6c81fa131f423246a9109d7b2a375e83968ad0800d6e57d0574629b/pydantic_core-2.33.2-pp311-pypy311_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:f889f7a40498cc077332c7ab6b4608d296d852182211787d4f3ee377aaae66e8", size = 2069475, upload-time = "2025-04-23T18:33:24.528Z" }, + { url = "https://files.pythonhosted.org/packages/b8/e9/1f7efbe20d0b2b10f6718944b5d8ece9152390904f29a78e68d4e7961159/pydantic_core-2.33.2-pp311-pypy311_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:de4b83bb311557e439b9e186f733f6c645b9417c84e2eb8203f3f820a4b988bf", size = 2239013, upload-time = "2025-04-23T18:33:26.621Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b2/5309c905a93811524a49b4e031e9851a6b00ff0fb668794472ea7746b448/pydantic_core-2.33.2-pp311-pypy311_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:82f68293f055f51b51ea42fafc74b6aad03e70e191799430b90c13d643059ebb", size = 2238715, upload-time = "2025-04-23T18:33:28.656Z" }, + { url = "https://files.pythonhosted.org/packages/32/56/8a7ca5d2cd2cda1d245d34b1c9a942920a718082ae8e54e5f3e5a58b7add/pydantic_core-2.33.2-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:329467cecfb529c925cf2bbd4d60d2c509bc2fb52a20c1045bf09bb70971a9c1", size = 2066757, upload-time = "2025-04-23T18:33:30.645Z" }, +] + +[[package]] +name = "pygments" +version = "2.19.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" }, +] + +[[package]] +name = "pyiceberg" +version = "0.10.0" +source = { git = "https://github.com/apache/iceberg-python.git?rev=d3eb149fe99b14e714a06e727b69cd617c6c052d#d3eb149fe99b14e714a06e727b69cd617c6c052d" } +dependencies = [ + { name = "cachetools" }, + { name = "click" }, + { name = "fsspec" }, + { name = "mmh3" }, + { name = "pydantic" }, + { name = "pyparsing" }, + { name = "pyroaring" }, + { name = "requests" }, + { name = "rich" }, + { name = "sortedcontainers" }, + { name = "strictyaml" }, + { name = "tenacity" }, +] + +[package.optional-dependencies] +pyarrow = [ + { name = "pyarrow" }, + { name = "pyiceberg-core" }, +] +sql-sqlite = [ + { name = "sqlalchemy" }, +] + +[[package]] +name = "pyiceberg-core" +source = { editable = "." } + +[package.dev-dependencies] +dev = [ + { name = "datafusion" }, + { name = "fastavro" }, + { name = "maturin" }, + { name = "pyiceberg", extra = ["pyarrow", "sql-sqlite"] }, + { name = "pytest" }, +] + +[package.metadata] + +[package.metadata.requires-dev] +dev = [ + { name = "datafusion", specifier = "==50.*" }, + { name = "fastavro", specifier = ">=1.11.1" }, + { name = "maturin", specifier = ">=1.0,<2.0" }, + { name = "pyiceberg", extras = ["sql-sqlite", "pyarrow"], git = "https://github.com/apache/iceberg-python.git?rev=d3eb149fe99b14e714a06e727b69cd617c6c052d" }, + { name = "pytest", specifier = ">=8.3.2" }, +] + +[[package]] +name = "pyparsing" +version = "3.2.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f2/a5/181488fc2b9d093e3972d2a472855aae8a03f000592dbfce716a512b3359/pyparsing-3.2.5.tar.gz", hash = "sha256:2df8d5b7b2802ef88e8d016a2eb9c7aeaa923529cd251ed0fe4608275d4105b6", size = 1099274, upload-time = "2025-09-21T04:11:06.277Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/5e/1aa9a93198c6b64513c9d7752de7422c06402de6600a8767da1524f9570b/pyparsing-3.2.5-py3-none-any.whl", hash = "sha256:e38a4f02064cf41fe6593d328d0512495ad1f3d8a91c4f73fc401b3079a59a5e", size = 113890, upload-time = "2025-09-21T04:11:04.117Z" }, +] + +[[package]] +name = "pyroaring" +version = "1.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0f/e4/975f0fa77fc3590820b4a3ac49704644b389795409bc12eb91729f845812/pyroaring-1.0.3.tar.gz", hash = "sha256:cd7392d1c010c9e41c11c62cd0610c8852e7e9698b1f7f6c2fcdefe50e7ef6da", size = 188688, upload-time = "2025-10-09T09:08:22.448Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/87/f19328d35f29b1d634f8f2127941927b37c6b68890912a4668e1a272d54d/pyroaring-1.0.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:c10e4cfbe203a578c78808406af491e3615d5e46cf69a7709050243346cd68bc", size = 670434, upload-time = "2025-10-09T09:06:34.89Z" }, + { url = "https://files.pythonhosted.org/packages/2f/fa/d933448844925728990e2256bbfa4f18cc3956d43d548270317128038015/pyroaring-1.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:cc329c62e504f2531c4008240f31736bcd2dee4339071f1eac0648068e6d17fa", size = 367332, upload-time = "2025-10-09T09:06:36.549Z" }, + { url = "https://files.pythonhosted.org/packages/39/e4/ab74296c3aac8f3ceb800354c2f811de2c4c518b3ca3a7a28e0599740cff/pyroaring-1.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8c7fb6ddf6ef31148f0939bc5c26b681d63df301ee1e372525012dd7bfe4a30a", size = 311443, upload-time = "2025-10-09T09:06:37.617Z" }, + { url = "https://files.pythonhosted.org/packages/6b/71/13ff623f3bba340ea7cc841883d7a0eaba1bec7e2d4e0d6759b89a4ce754/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cd18446832ea04a7d33bd6b78270b0be14eabcda5937af3428d6cb3d2bf98e54", size = 1853845, upload-time = "2025-10-09T09:06:38.706Z" }, + { url = "https://files.pythonhosted.org/packages/2b/05/40c0b37d78b16842c924b87b4fa491f4b20ed0e40c6255c872df45314247/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f0cbc766df2a24e28f23d69b66bbec64e691799219fd82c2f2236f03fc88e2e", size = 2045170, upload-time = "2025-10-09T09:06:40.229Z" }, + { url = "https://files.pythonhosted.org/packages/78/c6/6560c61d2f5c30fbb8f7b9a1c7d02068e78a7e8cd336eb294ec70896a80a/pyroaring-1.0.3-cp310-cp310-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:96a51e96f8f473381615f0f852f7238ad0a47f28e4a35e9f082468c5cfe4e9c3", size = 1791073, upload-time = "2025-10-09T09:06:42.092Z" }, + { url = "https://files.pythonhosted.org/packages/33/9e/0c91d4dbc4ec7bea9dcd3c203cfb8d96ed9df3c46981c0b22e9f17e98296/pyroaring-1.0.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:654af38b1f1c9bdc27b4f6d331fc5d91599df96e72a6df1886f4d95eea60ab29", size = 1789048, upload-time = "2025-10-09T09:06:43.392Z" }, + { url = "https://files.pythonhosted.org/packages/e9/48/51af418321cda2a1cfa64a48397ea0b73da74afe5c53a862525476d8a42c/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6721036afa31c07bdcbb4fcafa166660cf9c2eac695dcd495f8778549fa55899", size = 2838811, upload-time = "2025-10-09T09:06:44.588Z" }, + { url = "https://files.pythonhosted.org/packages/b2/69/7bc7070b35f72706e3870b5856e73656b9065bedae90268da5d77be00b15/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:0caa10f20329d09233fac6550b2adce4d9f173f748a9a9a5ea3b7033827dfe2d", size = 2640299, upload-time = "2025-10-09T09:06:46.159Z" }, + { url = "https://files.pythonhosted.org/packages/d0/87/5f2f590973d454e79ee8729aca888f9bb2d6018f7c9816bf66000cbc5e88/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:f109be8af937e85c52cb920d3fd120db52b172f59460852d2e3d2e3d13a4f52a", size = 2965427, upload-time = "2025-10-09T09:06:47.558Z" }, + { url = "https://files.pythonhosted.org/packages/e4/c8/1b425503141681db94941d9d3f41333bbd61975c3fc62d95122c372da85a/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:ddc80bfcd313c7c524a2742d263e73cae088b6a611b77dcc46fa90c306f6dace", size = 3084447, upload-time = "2025-10-09T09:06:48.727Z" }, + { url = "https://files.pythonhosted.org/packages/51/b9/2ac712ea90bd1e0d7e49e5e26c0c5aad1d77f21cf520a7edf46a477f5217/pyroaring-1.0.3-cp310-cp310-win32.whl", hash = "sha256:5a183f5ec069757fe5b60e37f7c6fa8a53178eacf0d76601b739e2890edee036", size = 204956, upload-time = "2025-10-09T09:06:49.801Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b9/1d4859c74d05f72b86dc0b308e6221e814a178459301cea9bcd084b4a92b/pyroaring-1.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:051bd9a66ce855a1143faa2b879ea6c6ca2905209e172ce9eedf79834897c730", size = 253778, upload-time = "2025-10-09T09:06:50.68Z" }, + { url = "https://files.pythonhosted.org/packages/64/20/b421100bd14b6a1074945af1418671630e1b8c4996ef000ac4e363785ead/pyroaring-1.0.3-cp310-cp310-win_arm64.whl", hash = "sha256:3043ff5c85375310ca3cd3e01944e03026e0ec07885e52dfabcfcd9dc303867f", size = 219330, upload-time = "2025-10-09T09:06:52.011Z" }, + { url = "https://files.pythonhosted.org/packages/39/ed/5e555dd99b12318ea1c7666b773fc4f097aeb609eeb1c1b3da519d445f71/pyroaring-1.0.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:755cdac1f9a1b7b5c621e570d4f6dbcf3b8e4a1e35a66f976104ecb35dce4ed2", size = 675916, upload-time = "2025-10-09T09:06:53.174Z" }, + { url = "https://files.pythonhosted.org/packages/da/06/dd8a9a87b90c4560f8384ab1dbafcd40c2a16f6777a07334a8e341bd7383/pyroaring-1.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ebab073db620f26f0ba11e13fa2f35e3b1298209fba47b6bc8cb6f0e2c9627f9", size = 369743, upload-time = "2025-10-09T09:06:54.421Z" }, + { url = "https://files.pythonhosted.org/packages/35/aa/da882011045ddacffe818a4fcbdd7e609a15f9c83d536222ec5b17af4aa9/pyroaring-1.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:684fb8dffe19bdb7f91897c65eac6eee23b1e46043c47eb24288f28a1170fe04", size = 313981, upload-time = "2025-10-09T09:06:55.514Z" }, + { url = "https://files.pythonhosted.org/packages/ed/3c/f6534844b02e2505ccdc9aae461c9838ab96f72b5688c045448761735512/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:678d31fc24e82945a1bfb14816c77823983382ffea76985d494782aa2f058427", size = 1923181, upload-time = "2025-10-09T09:06:56.897Z" }, + { url = "https://files.pythonhosted.org/packages/ea/82/9f1a85ba33e3d89b9cdb8183fb2fd2f25720d10742dd8827508ccccc13ae/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7d815f624e0285db3669f673d1725cb754b120ec70d0032d7c7166103a96c96d", size = 2113222, upload-time = "2025-10-09T09:06:58.388Z" }, + { url = "https://files.pythonhosted.org/packages/a7/f8/4d4340971cbc1379f987c847080bcb7f9765a57e122f392c3a3485c9587e/pyroaring-1.0.3-cp311-cp311-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:57fd5b80dacb8e888402b6b7508a734c6a527063e4e24e882ff2e0fd90721ada", size = 1837385, upload-time = "2025-10-09T09:06:59.449Z" }, + { url = "https://files.pythonhosted.org/packages/c6/58/d14cc561685e4c224af26b4fdb4f6c7e643294ac5a4b29f178b5cbb71af1/pyroaring-1.0.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ab26a7a45a0bb46c00394d1a60a9f2d57c220f84586e30d59b39784b0f94aee6", size = 1856170, upload-time = "2025-10-09T09:07:00.608Z" }, + { url = "https://files.pythonhosted.org/packages/d1/d2/d2d9790c373f6438d4d0958bc4c79f3dc77826d8553743ff3f64acdc9ab3/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9232f3f606315d59049c128154100fd05008d5c5c211e48b21848cd41ee64d26", size = 2909282, upload-time = "2025-10-09T09:07:02.124Z" }, + { url = "https://files.pythonhosted.org/packages/bc/28/4b2277982302b5b406998064ca1eaef1a79e4ea87185f511e33e7a7e3511/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f34b44b3ec3df97b978799f2901fefb2a48d367496fd1cde3cc5fe8b3bc13510", size = 2701034, upload-time = "2025-10-09T09:07:03.403Z" }, + { url = "https://files.pythonhosted.org/packages/d2/91/b2340193825fa2431cf735f0ecb23206fb31f386fecca38336935a294513/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:25a83ec6bac3106568bd3fdd316f0fee52aa0be8c72da565ad02b10ae7905924", size = 3028962, upload-time = "2025-10-09T09:07:05.558Z" }, + { url = "https://files.pythonhosted.org/packages/07/ea/ad79073cc5d8dcca35d1a955bb886d96905e9dacc58d1971fda012a5ad18/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c17d4ec53b5b6b333d9a9515051213a691293ada785dc8c025d3641482597ed3", size = 3152109, upload-time = "2025-10-09T09:07:06.887Z" }, + { url = "https://files.pythonhosted.org/packages/9a/de/f55a1093acb16d25ff9811546823e59078e4a3e56d2eb0ff5d10f696933d/pyroaring-1.0.3-cp311-cp311-win32.whl", hash = "sha256:d54024459ace600f1d1ffbc6dc3c60eb47cca3b678701f06148f59e10f6f8d7b", size = 204246, upload-time = "2025-10-09T09:07:08.036Z" }, + { url = "https://files.pythonhosted.org/packages/c6/e5/36bf3039733b8e00732892c9334b2f5309f38e72af0b3b40b8729b5857a3/pyroaring-1.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:c28750148ef579a7447a8cb60b39e5943e03f8c29bce8f2788728f6f23d1887a", size = 254637, upload-time = "2025-10-09T09:07:09.103Z" }, + { url = "https://files.pythonhosted.org/packages/d6/e8/e2b78e595b5a82a6014af327614756a55f17ec4120a2ab197f1762641316/pyroaring-1.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:535d8deccbd8db2c6bf38629243e9646756905574a742b2a72ff51d6461d616c", size = 219597, upload-time = "2025-10-09T09:07:10.38Z" }, + { url = "https://files.pythonhosted.org/packages/dd/09/a5376d55672e0535019ba1469888909d0046cea0cfb969a4aa1f99caaf22/pyroaring-1.0.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:add3e4c78eb590a76526ecce8d1566eecdd5822e351c36b3697997f4a80ed808", size = 681056, upload-time = "2025-10-09T09:07:11.497Z" }, + { url = "https://files.pythonhosted.org/packages/23/dd/78f59d361bd9ebf8de3660408b0c48664ade0a057ebcf4b207d99ac1a698/pyroaring-1.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ebaffe846cf4ba4f00ce6b8a9f39613f24e2d09447e77be4fa6e898bc36451b6", size = 375111, upload-time = "2025-10-09T09:07:12.597Z" }, + { url = "https://files.pythonhosted.org/packages/bf/03/10dc93f83a5453eb40a69c79106a8385b40aa12cf4531ca72bd9d7f45cb2/pyroaring-1.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a9459f27498f97d08031a34a5ead230b77eb0ab3cc3d85b7f54faa2fd548acd6", size = 314319, upload-time = "2025-10-09T09:07:13.579Z" }, + { url = "https://files.pythonhosted.org/packages/86/9e/b00c38a7e62a73e152055f593595c37152e61fc2896fd11538a7c71fbe4e/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2b2eb8bd1c35c772994889be9f7dda09477475d7aa1e2af9ab4ef18619326f6", size = 1869251, upload-time = "2025-10-09T09:07:14.584Z" }, + { url = "https://files.pythonhosted.org/packages/4f/33/f32d00ca105b66303deab43d027c3574c8ade8525dac0e5b50a9fb4d1b76/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d31f4c1c906f1af14ce61a3959d04a14a64c594f8a768399146a45bbd341f21f", size = 2071551, upload-time = "2025-10-09T09:07:15.713Z" }, + { url = "https://files.pythonhosted.org/packages/5d/89/e953cae181ba4c7523334855a1ca0ae8eeea3cee8d7cd39c56bd99709d3f/pyroaring-1.0.3-cp312-cp312-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53be988fc86698d56c11049bfe5113a2f6990adb1fa2782b29636509808b6aa7", size = 1781071, upload-time = "2025-10-09T09:07:17.19Z" }, + { url = "https://files.pythonhosted.org/packages/fa/db/65d4be532e68b62a84a9c89b24d0a1394f452f484fa29392142d9a3b9c48/pyroaring-1.0.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7df84d223424523b19a23781f4246cc247fd6d821e1bc0853c2f25669136f7d0", size = 1795670, upload-time = "2025-10-09T09:07:18.524Z" }, + { url = "https://files.pythonhosted.org/packages/f5/9e/684ea0568ce7d30fc4e01ad1c666e9ce1a5b1702fa630231f4f6bdb96539/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:34a781f1f9766897f63ef18be129827340ae37764015b83fdcff1efb9e29136d", size = 2849305, upload-time = "2025-10-09T09:07:20.388Z" }, + { url = "https://files.pythonhosted.org/packages/7c/fd/d7773a2adf91f45d8924197954c66b1694325afd2f27e02edaac07338402/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:1f414343b4ed0756734328cdf2a91022fc54503769e3f8d79bd0b672ea815a16", size = 2692843, upload-time = "2025-10-09T09:07:22.042Z" }, + { url = "https://files.pythonhosted.org/packages/13/72/b8a99ba138eebd8ff9bf8d15f3942e9e43e8e45723e2e6b7b09e542b7448/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:d16ae185c72dc64f76335dbe53e53a892e78115adc92194957d1b7ef74d230b9", size = 2983440, upload-time = "2025-10-09T09:07:23.419Z" }, + { url = "https://files.pythonhosted.org/packages/ca/94/e6ed1f682d850e039c71b2032bacdefc5082dc809796cf34b9e6f24c604d/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f888447bf22dde7759108bfe6dfbeb6bbb61b14948de9c4cb6843c4dd57e2215", size = 3117542, upload-time = "2025-10-09T09:07:25.104Z" }, + { url = "https://files.pythonhosted.org/packages/8f/89/d55b0ed3e098ef89c421b43b748afe3d90eb250cab50b9e53e3a3449ac58/pyroaring-1.0.3-cp312-cp312-win32.whl", hash = "sha256:fbbdc44c51a0a3efd7be3dbe04466278ce098fcd101aa1905849319042159770", size = 205118, upload-time = "2025-10-09T09:07:26.532Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e1/b71fef6a73efb50110d33d714235ff7059f4ebae98dc474b6549b322f48f/pyroaring-1.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:3b217c4b3ad953b4c759a0d2f9bd95316f0c345b9f7adb49e6ded7a1f5106bd4", size = 260629, upload-time = "2025-10-09T09:07:27.528Z" }, + { url = "https://files.pythonhosted.org/packages/57/33/66ee872079c9c47512d6e17d374bcad8d91350c24dc20fbe678c34b33745/pyroaring-1.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:e6bcf838564c21bab8fe6c2748b4990d4cd90612d8c470c04889def7bb5114ea", size = 219032, upload-time = "2025-10-09T09:07:28.754Z" }, +] + +[[package]] +name = "pytest" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "exceptiongroup", marker = "python_full_version < '3.11'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/5c/00a0e072241553e1a7496d638deababa67c5058571567b92a7eaa258397c/pytest-8.4.2.tar.gz", hash = "sha256:86c0d0b93306b961d58d62a4db4879f27fe25513d4b969df351abdddb3c30e01", size = 1519618, upload-time = "2025-09-04T14:34:22.711Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/a4/20da314d277121d6534b3a980b29035dcd51e6744bd79075a6ce8fa4eb8d/pytest-8.4.2-py3-none-any.whl", hash = "sha256:872f880de3fc3a5bdc88a11b39c9710c3497a547cfa9320bc3c5e62fbf272e79", size = 365750, upload-time = "2025-09-04T14:34:20.226Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "requests" +version = "2.32.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c9/74/b3ff8e6c8446842c3f5c837e9c3dfcfe2018ea6ecef224c710c85ef728f4/requests-2.32.5.tar.gz", hash = "sha256:dbba0bac56e100853db0ea71b82b4dfd5fe2bf6d3754a8893c3af500cec7d7cf", size = 134517, upload-time = "2025-08-18T20:46:02.573Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl", hash = "sha256:2462f94637a34fd532264295e186976db0f5d453d1cdd31473c85a6a161affb6", size = 64738, upload-time = "2025-08-18T20:46:00.542Z" }, +] + +[[package]] +name = "rich" +version = "14.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "sortedcontainers" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, +] + +[[package]] +name = "sqlalchemy" +version = "2.0.44" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f0/f2/840d7b9496825333f532d2e3976b8eadbf52034178aac53630d09fe6e1ef/sqlalchemy-2.0.44.tar.gz", hash = "sha256:0ae7454e1ab1d780aee69fd2aae7d6b8670a581d8847f2d1e0f7ddfbf47e5a22", size = 9819830, upload-time = "2025-10-10T14:39:12.935Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/a7/e9ccfa7eecaf34c6f57d8cb0bb7cbdeeff27017cc0f5d0ca90fdde7a7c0d/sqlalchemy-2.0.44-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:7c77f3080674fc529b1bd99489378c7f63fcb4ba7f8322b79732e0258f0ea3ce", size = 2137282, upload-time = "2025-10-10T15:36:10.965Z" }, + { url = "https://files.pythonhosted.org/packages/b1/e1/50bc121885bdf10833a4f65ecbe9fe229a3215f4d65a58da8a181734cae3/sqlalchemy-2.0.44-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4c26ef74ba842d61635b0152763d057c8d48215d5be9bb8b7604116a059e9985", size = 2127322, upload-time = "2025-10-10T15:36:12.428Z" }, + { url = "https://files.pythonhosted.org/packages/46/f2/a8573b7230a3ce5ee4b961a2d510d71b43872513647398e595b744344664/sqlalchemy-2.0.44-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f4a172b31785e2f00780eccab00bc240ccdbfdb8345f1e6063175b3ff12ad1b0", size = 3214772, upload-time = "2025-10-10T15:34:15.09Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d8/c63d8adb6a7edaf8dcb6f75a2b1e9f8577960a1e489606859c4d73e7d32b/sqlalchemy-2.0.44-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f9480c0740aabd8cb29c329b422fb65358049840b34aba0adf63162371d2a96e", size = 3214434, upload-time = "2025-10-10T15:47:00.473Z" }, + { url = "https://files.pythonhosted.org/packages/ee/a6/243d277a4b54fae74d4797957a7320a5c210c293487f931cbe036debb697/sqlalchemy-2.0.44-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:17835885016b9e4d0135720160db3095dc78c583e7b902b6be799fb21035e749", size = 3155365, upload-time = "2025-10-10T15:34:17.932Z" }, + { url = "https://files.pythonhosted.org/packages/5f/f8/6a39516ddd75429fd4ee5a0d72e4c80639fab329b2467c75f363c2ed9751/sqlalchemy-2.0.44-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:cbe4f85f50c656d753890f39468fcd8190c5f08282caf19219f684225bfd5fd2", size = 3178910, upload-time = "2025-10-10T15:47:02.346Z" }, + { url = "https://files.pythonhosted.org/packages/43/f0/118355d4ad3c39d9a2f5ee4c7304a9665b3571482777357fa9920cd7a6b4/sqlalchemy-2.0.44-cp310-cp310-win32.whl", hash = "sha256:2fcc4901a86ed81dc76703f3b93ff881e08761c63263c46991081fd7f034b165", size = 2105624, upload-time = "2025-10-10T15:38:15.552Z" }, + { url = "https://files.pythonhosted.org/packages/61/83/6ae5f9466f8aa5d0dcebfff8c9c33b98b27ce23292df3b990454b3d434fd/sqlalchemy-2.0.44-cp310-cp310-win_amd64.whl", hash = "sha256:9919e77403a483ab81e3423151e8ffc9dd992c20d2603bf17e4a8161111e55f5", size = 2129240, upload-time = "2025-10-10T15:38:17.175Z" }, + { url = "https://files.pythonhosted.org/packages/e3/81/15d7c161c9ddf0900b076b55345872ed04ff1ed6a0666e5e94ab44b0163c/sqlalchemy-2.0.44-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0fe3917059c7ab2ee3f35e77757062b1bea10a0b6ca633c58391e3f3c6c488dd", size = 2140517, upload-time = "2025-10-10T15:36:15.64Z" }, + { url = "https://files.pythonhosted.org/packages/d4/d5/4abd13b245c7d91bdf131d4916fd9e96a584dac74215f8b5bc945206a974/sqlalchemy-2.0.44-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:de4387a354ff230bc979b46b2207af841dc8bf29847b6c7dbe60af186d97aefa", size = 2130738, upload-time = "2025-10-10T15:36:16.91Z" }, + { url = "https://files.pythonhosted.org/packages/cb/3c/8418969879c26522019c1025171cefbb2a8586b6789ea13254ac602986c0/sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c3678a0fb72c8a6a29422b2732fe423db3ce119c34421b5f9955873eb9b62c1e", size = 3304145, upload-time = "2025-10-10T15:34:19.569Z" }, + { url = "https://files.pythonhosted.org/packages/94/2d/fdb9246d9d32518bda5d90f4b65030b9bf403a935cfe4c36a474846517cb/sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3cf6872a23601672d61a68f390e44703442639a12ee9dd5a88bbce52a695e46e", size = 3304511, upload-time = "2025-10-10T15:47:05.088Z" }, + { url = "https://files.pythonhosted.org/packages/7d/fb/40f2ad1da97d5c83f6c1269664678293d3fe28e90ad17a1093b735420549/sqlalchemy-2.0.44-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:329aa42d1be9929603f406186630135be1e7a42569540577ba2c69952b7cf399", size = 3235161, upload-time = "2025-10-10T15:34:21.193Z" }, + { url = "https://files.pythonhosted.org/packages/95/cb/7cf4078b46752dca917d18cf31910d4eff6076e5b513c2d66100c4293d83/sqlalchemy-2.0.44-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:70e03833faca7166e6a9927fbee7c27e6ecde436774cd0b24bbcc96353bce06b", size = 3261426, upload-time = "2025-10-10T15:47:07.196Z" }, + { url = "https://files.pythonhosted.org/packages/f8/3b/55c09b285cb2d55bdfa711e778bdffdd0dc3ffa052b0af41f1c5d6e582fa/sqlalchemy-2.0.44-cp311-cp311-win32.whl", hash = "sha256:253e2f29843fb303eca6b2fc645aca91fa7aa0aa70b38b6950da92d44ff267f3", size = 2105392, upload-time = "2025-10-10T15:38:20.051Z" }, + { url = "https://files.pythonhosted.org/packages/c7/23/907193c2f4d680aedbfbdf7bf24c13925e3c7c292e813326c1b84a0b878e/sqlalchemy-2.0.44-cp311-cp311-win_amd64.whl", hash = "sha256:7a8694107eb4308a13b425ca8c0e67112f8134c846b6e1f722698708741215d5", size = 2130293, upload-time = "2025-10-10T15:38:21.601Z" }, + { url = "https://files.pythonhosted.org/packages/62/c4/59c7c9b068e6813c898b771204aad36683c96318ed12d4233e1b18762164/sqlalchemy-2.0.44-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:72fea91746b5890f9e5e0997f16cbf3d53550580d76355ba2d998311b17b2250", size = 2139675, upload-time = "2025-10-10T16:03:31.064Z" }, + { url = "https://files.pythonhosted.org/packages/d6/ae/eeb0920537a6f9c5a3708e4a5fc55af25900216bdb4847ec29cfddf3bf3a/sqlalchemy-2.0.44-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:585c0c852a891450edbb1eaca8648408a3cc125f18cf433941fa6babcc359e29", size = 2127726, upload-time = "2025-10-10T16:03:35.934Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d5/2ebbabe0379418eda8041c06b0b551f213576bfe4c2f09d77c06c07c8cc5/sqlalchemy-2.0.44-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9b94843a102efa9ac68a7a30cd46df3ff1ed9c658100d30a725d10d9c60a2f44", size = 3327603, upload-time = "2025-10-10T15:35:28.322Z" }, + { url = "https://files.pythonhosted.org/packages/45/e5/5aa65852dadc24b7d8ae75b7efb8d19303ed6ac93482e60c44a585930ea5/sqlalchemy-2.0.44-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:119dc41e7a7defcefc57189cfa0e61b1bf9c228211aba432b53fb71ef367fda1", size = 3337842, upload-time = "2025-10-10T15:43:45.431Z" }, + { url = "https://files.pythonhosted.org/packages/41/92/648f1afd3f20b71e880ca797a960f638d39d243e233a7082c93093c22378/sqlalchemy-2.0.44-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0765e318ee9179b3718c4fd7ba35c434f4dd20332fbc6857a5e8df17719c24d7", size = 3264558, upload-time = "2025-10-10T15:35:29.93Z" }, + { url = "https://files.pythonhosted.org/packages/40/cf/e27d7ee61a10f74b17740918e23cbc5bc62011b48282170dc4c66da8ec0f/sqlalchemy-2.0.44-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2e7b5b079055e02d06a4308d0481658e4f06bc7ef211567edc8f7d5dce52018d", size = 3301570, upload-time = "2025-10-10T15:43:48.407Z" }, + { url = "https://files.pythonhosted.org/packages/3b/3d/3116a9a7b63e780fb402799b6da227435be878b6846b192f076d2f838654/sqlalchemy-2.0.44-cp312-cp312-win32.whl", hash = "sha256:846541e58b9a81cce7dee8329f352c318de25aa2f2bbe1e31587eb1f057448b4", size = 2103447, upload-time = "2025-10-10T15:03:21.678Z" }, + { url = "https://files.pythonhosted.org/packages/25/83/24690e9dfc241e6ab062df82cc0df7f4231c79ba98b273fa496fb3dd78ed/sqlalchemy-2.0.44-cp312-cp312-win_amd64.whl", hash = "sha256:7cbcb47fd66ab294703e1644f78971f6f2f1126424d2b300678f419aa73c7b6e", size = 2130912, upload-time = "2025-10-10T15:03:24.656Z" }, + { url = "https://files.pythonhosted.org/packages/9c/5e/6a29fa884d9fb7ddadf6b69490a9d45fded3b38541713010dad16b77d015/sqlalchemy-2.0.44-py3-none-any.whl", hash = "sha256:19de7ca1246fbef9f9d1bff8f1ab25641569df226364a0e40457dc5457c54b05", size = 1928718, upload-time = "2025-10-10T15:29:45.32Z" }, +] + +[[package]] +name = "strictyaml" +version = "1.7.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/08/efd28d49162ce89c2ad61a88bd80e11fb77bc9f6c145402589112d38f8af/strictyaml-1.7.3.tar.gz", hash = "sha256:22f854a5fcab42b5ddba8030a0e4be51ca89af0267961c8d6cfa86395586c407", size = 115206, upload-time = "2023-03-10T12:50:27.062Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/96/7c/a81ef5ef10978dd073a854e0fa93b5d8021d0594b639cc8f6453c3c78a1d/strictyaml-1.7.3-py3-none-any.whl", hash = "sha256:fb5c8a4edb43bebb765959e420f9b3978d7f1af88c80606c03fb420888f5d1c7", size = 123917, upload-time = "2023-03-10T12:50:17.242Z" }, +] + +[[package]] +name = "tenacity" +version = "9.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0a/d4/2b0cd0fe285e14b36db076e78c93766ff1d529d70408bd1d2a5a84f1d929/tenacity-9.1.2.tar.gz", hash = "sha256:1169d376c297e7de388d18b4481760d478b0e99a777cad3a9c86e556f4b697cb", size = 48036, upload-time = "2025-04-02T08:25:09.966Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/30/643397144bfbfec6f6ef821f36f33e57d35946c44a2352d3c9f0ae847619/tenacity-9.1.2-py3-none-any.whl", hash = "sha256:f77bf36710d8b73a50b2dd155c97b870017ad21afe6ab300326b0371b3b05138", size = 28248, upload-time = "2025-04-02T08:25:07.678Z" }, +] + +[[package]] +name = "tomli" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/52/ed/3f73f72945444548f33eba9a87fc7a6e969915e7b1acc8260b30e1f76a2f/tomli-2.3.0.tar.gz", hash = "sha256:64be704a875d2a59753d80ee8a533c3fe183e3f06807ff7dc2232938ccb01549", size = 17392, upload-time = "2025-10-08T22:01:47.119Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/2e/299f62b401438d5fe1624119c723f5d877acc86a4c2492da405626665f12/tomli-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:88bd15eb972f3664f5ed4b57c1634a97153b4bac4479dcb6a495f41921eb7f45", size = 153236, upload-time = "2025-10-08T22:01:00.137Z" }, + { url = "https://files.pythonhosted.org/packages/86/7f/d8fffe6a7aefdb61bced88fcb5e280cfd71e08939da5894161bd71bea022/tomli-2.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:883b1c0d6398a6a9d29b508c331fa56adbcdff647f6ace4dfca0f50e90dfd0ba", size = 148084, upload-time = "2025-10-08T22:01:01.63Z" }, + { url = "https://files.pythonhosted.org/packages/47/5c/24935fb6a2ee63e86d80e4d3b58b222dafaf438c416752c8b58537c8b89a/tomli-2.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d1381caf13ab9f300e30dd8feadb3de072aeb86f1d34a8569453ff32a7dea4bf", size = 234832, upload-time = "2025-10-08T22:01:02.543Z" }, + { url = "https://files.pythonhosted.org/packages/89/da/75dfd804fc11e6612846758a23f13271b76d577e299592b4371a4ca4cd09/tomli-2.3.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a0e285d2649b78c0d9027570d4da3425bdb49830a6156121360b3f8511ea3441", size = 242052, upload-time = "2025-10-08T22:01:03.836Z" }, + { url = "https://files.pythonhosted.org/packages/70/8c/f48ac899f7b3ca7eb13af73bacbc93aec37f9c954df3c08ad96991c8c373/tomli-2.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0a154a9ae14bfcf5d8917a59b51ffd5a3ac1fd149b71b47a3a104ca4edcfa845", size = 239555, upload-time = "2025-10-08T22:01:04.834Z" }, + { url = "https://files.pythonhosted.org/packages/ba/28/72f8afd73f1d0e7829bfc093f4cb98ce0a40ffc0cc997009ee1ed94ba705/tomli-2.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:74bf8464ff93e413514fefd2be591c3b0b23231a77f901db1eb30d6f712fc42c", size = 245128, upload-time = "2025-10-08T22:01:05.84Z" }, + { url = "https://files.pythonhosted.org/packages/b6/eb/a7679c8ac85208706d27436e8d421dfa39d4c914dcf5fa8083a9305f58d9/tomli-2.3.0-cp311-cp311-win32.whl", hash = "sha256:00b5f5d95bbfc7d12f91ad8c593a1659b6387b43f054104cda404be6bda62456", size = 96445, upload-time = "2025-10-08T22:01:06.896Z" }, + { url = "https://files.pythonhosted.org/packages/0a/fe/3d3420c4cb1ad9cb462fb52967080575f15898da97e21cb6f1361d505383/tomli-2.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:4dc4ce8483a5d429ab602f111a93a6ab1ed425eae3122032db7e9acf449451be", size = 107165, upload-time = "2025-10-08T22:01:08.107Z" }, + { url = "https://files.pythonhosted.org/packages/ff/b7/40f36368fcabc518bb11c8f06379a0fd631985046c038aca08c6d6a43c6e/tomli-2.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d7d86942e56ded512a594786a5ba0a5e521d02529b3826e7761a05138341a2ac", size = 154891, upload-time = "2025-10-08T22:01:09.082Z" }, + { url = "https://files.pythonhosted.org/packages/f9/3f/d9dd692199e3b3aab2e4e4dd948abd0f790d9ded8cd10cbaae276a898434/tomli-2.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:73ee0b47d4dad1c5e996e3cd33b8a76a50167ae5f96a2607cbe8cc773506ab22", size = 148796, upload-time = "2025-10-08T22:01:10.266Z" }, + { url = "https://files.pythonhosted.org/packages/60/83/59bff4996c2cf9f9387a0f5a3394629c7efa5ef16142076a23a90f1955fa/tomli-2.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:792262b94d5d0a466afb5bc63c7daa9d75520110971ee269152083270998316f", size = 242121, upload-time = "2025-10-08T22:01:11.332Z" }, + { url = "https://files.pythonhosted.org/packages/45/e5/7c5119ff39de8693d6baab6c0b6dcb556d192c165596e9fc231ea1052041/tomli-2.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4f195fe57ecceac95a66a75ac24d9d5fbc98ef0962e09b2eddec5d39375aae52", size = 250070, upload-time = "2025-10-08T22:01:12.498Z" }, + { url = "https://files.pythonhosted.org/packages/45/12/ad5126d3a278f27e6701abde51d342aa78d06e27ce2bb596a01f7709a5a2/tomli-2.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e31d432427dcbf4d86958c184b9bfd1e96b5b71f8eb17e6d02531f434fd335b8", size = 245859, upload-time = "2025-10-08T22:01:13.551Z" }, + { url = "https://files.pythonhosted.org/packages/fb/a1/4d6865da6a71c603cfe6ad0e6556c73c76548557a8d658f9e3b142df245f/tomli-2.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7b0882799624980785240ab732537fcfc372601015c00f7fc367c55308c186f6", size = 250296, upload-time = "2025-10-08T22:01:14.614Z" }, + { url = "https://files.pythonhosted.org/packages/a0/b7/a7a7042715d55c9ba6e8b196d65d2cb662578b4d8cd17d882d45322b0d78/tomli-2.3.0-cp312-cp312-win32.whl", hash = "sha256:ff72b71b5d10d22ecb084d345fc26f42b5143c5533db5e2eaba7d2d335358876", size = 97124, upload-time = "2025-10-08T22:01:15.629Z" }, + { url = "https://files.pythonhosted.org/packages/06/1e/f22f100db15a68b520664eb3328fb0ae4e90530887928558112c8d1f4515/tomli-2.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:1cb4ed918939151a03f33d4242ccd0aa5f11b3547d0cf30f7c74a408a5b99878", size = 107698, upload-time = "2025-10-08T22:01:16.51Z" }, + { url = "https://files.pythonhosted.org/packages/77/b8/0135fadc89e73be292b473cb820b4f5a08197779206b33191e801feeae40/tomli-2.3.0-py3-none-any.whl", hash = "sha256:e95b1af3c5b07d9e643909b5abbec77cd9f1217e6d0bca72b0234736b9fb1f1b", size = 14408, upload-time = "2025-10-08T22:01:46.04Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] + +[[package]] +name = "urllib3" +version = "2.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/15/22/9ee70a2574a4f4599c47dd506532914ce044817c7752a79b6a51286319bc/urllib3-2.5.0.tar.gz", hash = "sha256:3fc47733c7e419d4bc3f6b3dc2b4f890bb743906a30d56ba4a5bfa4bbff92760", size = 393185, upload-time = "2025-06-18T14:07:41.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/c2/fe1e52489ae3122415c51f387e221dd0773709bad6c6cdaa599e8a2c5185/urllib3-2.5.0-py3-none-any.whl", hash = "sha256:e6b01673c0fa6a13e374b50871808eb3bf7046c4b125b216f6bf1cc604cff0dc", size = 129795, upload-time = "2025-06-18T14:07:40.39Z" }, +] From 3cb18e789fb10b837b7449f6634f2b7e1c90bbb5 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 29 Oct 2025 10:58:02 +0800 Subject: [PATCH 043/891] ci: Relax msrv check thanks to rust 2024 (#1795) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This PR is going to relax the MSRV check thanks to rust 2024. Since starts from rust 2024, cargo has [msrv aware dep resolve](https://doc.rust-lang.org/edition-guide/rust-2024/cargo-resolver.html). This means that dep's upgrade no a problem now. ## Are these changes tested? Signed-off-by: Xuanwo Co-authored-by: Renjie Liu --- Makefile | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 1b98fe8fa1..ecdada3df5 100644 --- a/Makefile +++ b/Makefile @@ -45,10 +45,7 @@ NIGHTLY_VERSION := $(shell awk -F'"' '/^channel/ {print $$2}' rust-toolchain.tom MSRV_VERSION := $(shell awk -F'"' '/^rust-version/ {print $$2}' Cargo.toml) check-msrv: - @set -e; \ - trap 'git restore Cargo.lock' EXIT; \ - cargo +$(NIGHTLY_VERSION) generate-lockfile -Z direct-minimal-versions; \ - cargo +$(MSRV_VERSION) check --locked --workspace + cargo +$(MSRV_VERSION) check --workspace check: check-fmt check-clippy check-toml cargo-machete From f10c0ab0e4b30c44679614231e1f299a20205697 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 29 Oct 2025 11:05:05 +0800 Subject: [PATCH 044/891] ci: Don't dismiss stale review to make contribution easier (#1799) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? I made this change to avoid dismissing stale reviews to make contributions easier. iceberg is quite active, and we nearly have conflicts every day. It can be a pain for contributors to get approved, especially after resolving conflicts—they then have to wait for reviewers to give another approval. I think we can make it clear that trivial changes like resolving merge conflicts are allowed, but adding other code changes aren't. We can just add this to the docs so it’s easier for everyone. ## Are these changes tested? --- .asf.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.asf.yaml b/.asf.yaml index 1a766f5330..842ffc39a3 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -40,7 +40,7 @@ github: required_pull_request_reviews: required_approving_review_count: 1 - dismiss_stale_reviews: true + dismiss_stale_reviews: false required_linear_history: true del_branch_on_merge: true From d3d3127bf43bd6102acc13dbb064ae4633f8e923 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Wed, 29 Oct 2025 01:13:18 -0700 Subject: [PATCH 045/891] add Makefile to bindings/python (#1800) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Follow up to #1796 This PR adds ``` make install make build make test ``` to `bindings/python/` ## Are these changes tested? --- .github/workflows/bindings_python_ci.yml | 22 ++++++++++++-------- bindings/python/Makefile | 26 ++++++++++++++++++++++++ bindings/python/README.md | 6 +++--- 3 files changed, 42 insertions(+), 12 deletions(-) create mode 100644 bindings/python/Makefile diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index 9ea2ac0b90..55e7e5bbd0 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -56,17 +56,21 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 + - uses: astral-sh/setup-uv@v6 + with: + version: "0.9.3" + enable-cache: true - name: Install tools run: | - pip install ruff + uv tool install ruff - name: Check format working-directory: "bindings/python" run: | - ruff format . --diff + uvx ruff format . --diff - name: Check style working-directory: "bindings/python" run: | - ruff check . + uvx ruff check . test: runs-on: ${{ matrix.os }} @@ -86,15 +90,15 @@ jobs: working-directory: "bindings/python" command: build args: --out dist --sdist - - name: Install uv - shell: bash - run: | - python -m pip install uv==0.9.3 + - uses: astral-sh/setup-uv@v6 + with: + version: "0.9.3" + enable-cache: true - name: Sync dependencies working-directory: "bindings/python" shell: bash run: | - uv sync --group dev --no-install-project + make install - name: Install built wheel working-directory: "bindings/python" shell: bash @@ -104,4 +108,4 @@ jobs: working-directory: "bindings/python" shell: bash run: | - uv run --group dev --no-sync pytest + make test diff --git a/bindings/python/Makefile b/bindings/python/Makefile new file mode 100644 index 0000000000..c36a1091dd --- /dev/null +++ b/bindings/python/Makefile @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +install: + uv sync --group dev --no-install-project + +build: + uv run maturin develop + +test: + uv run --no-sync pytest diff --git a/bindings/python/README.md b/bindings/python/README.md index a91599d9a4..320a9a401b 100644 --- a/bindings/python/README.md +++ b/bindings/python/README.md @@ -32,17 +32,17 @@ pip install uv==0.9.3 Set up the development environment: ```shell -uv sync --group dev +make install ``` ## Build ```shell -uv run --group dev maturin develop +make build ``` ## Test ```shell -uv run --group dev pytest +make test ``` From 4eafd2ca22e032c3ee4d9e6abe6ff3eae93e0189 Mon Sep 17 00:00:00 2001 From: Colin Marc Date: Thu, 30 Oct 2025 15:29:00 +0100 Subject: [PATCH 046/891] chore: inline format args (#1805) The `uninlined_format_args` lint became default in a recent version of rust. I just ran `clippy --fix`. --- crates/catalog/glue/src/catalog.rs | 24 +++--- crates/catalog/glue/src/error.rs | 4 +- crates/catalog/glue/src/schema.rs | 6 +- crates/catalog/glue/src/utils.rs | 5 +- .../catalog/glue/tests/glue_catalog_test.rs | 4 +- crates/catalog/hms/src/catalog.rs | 4 +- crates/catalog/hms/src/error.rs | 4 +- crates/catalog/hms/src/schema.rs | 6 +- crates/catalog/hms/src/utils.rs | 12 ++- crates/catalog/hms/tests/hms_catalog_test.rs | 2 +- crates/catalog/rest/src/catalog.rs | 6 +- .../catalog/rest/tests/rest_catalog_test.rs | 2 +- crates/catalog/s3tables/src/catalog.rs | 18 ++--- crates/catalog/sql/src/catalog.rs | 49 ++++-------- crates/catalog/sql/src/error.rs | 6 +- crates/examples/src/rest_catalog_namespace.rs | 9 +-- crates/examples/src/rest_catalog_table.rs | 2 +- .../src/arrow/caching_delete_file_loader.rs | 2 +- .../iceberg/src/arrow/delete_file_loader.rs | 2 +- .../src/arrow/partition_value_calculator.rs | 2 +- crates/iceberg/src/arrow/reader.rs | 22 ++---- .../arrow/record_batch_partition_splitter.rs | 3 +- .../src/arrow/record_batch_transformer.rs | 6 +- crates/iceberg/src/arrow/schema.rs | 13 +--- crates/iceberg/src/arrow/value.rs | 6 +- crates/iceberg/src/catalog/memory/catalog.rs | 74 +++++-------------- .../src/catalog/memory/namespace_state.rs | 14 +--- .../iceberg/src/catalog/metadata_location.rs | 11 +-- crates/iceberg/src/catalog/mod.rs | 10 +-- crates/iceberg/src/expr/predicate.rs | 16 ++-- .../src/expr/visitors/page_index_evaluator.rs | 9 +-- .../expr/visitors/strict_metrics_evaluator.rs | 45 +++-------- crates/iceberg/src/io/object_cache.rs | 4 +- crates/iceberg/src/io/storage.rs | 2 +- crates/iceberg/src/io/storage_s3.rs | 5 +- crates/iceberg/src/puffin/metadata.rs | 13 +--- crates/iceberg/src/scan/cache.rs | 2 +- crates/iceberg/src/scan/mod.rs | 15 +--- crates/iceberg/src/spec/datatypes.rs | 10 +-- crates/iceberg/src/spec/manifest/data_file.rs | 4 +- crates/iceberg/src/spec/manifest/entry.rs | 2 +- crates/iceberg/src/spec/manifest_list.rs | 5 +- crates/iceberg/src/spec/partition.rs | 7 +- .../iceberg/src/spec/schema/id_reassigner.rs | 4 +- crates/iceberg/src/spec/schema/mod.rs | 11 +-- crates/iceberg/src/spec/schema/utils.rs | 5 +- crates/iceberg/src/spec/snapshot.rs | 2 +- crates/iceberg/src/spec/snapshot_summary.rs | 6 +- crates/iceberg/src/spec/table_metadata.rs | 18 ++--- .../src/spec/table_metadata_builder.rs | 13 +--- crates/iceberg/src/spec/table_properties.rs | 2 +- crates/iceberg/src/spec/transform.rs | 2 +- crates/iceberg/src/spec/values.rs | 50 +++++-------- crates/iceberg/src/spec/view_metadata.rs | 4 +- .../iceberg/src/spec/view_metadata_builder.rs | 8 +- .../src/transaction/update_properties.rs | 5 +- crates/iceberg/src/transform/mod.rs | 10 +-- crates/iceberg/src/transform/temporal.rs | 10 +-- crates/iceberg/src/transform/truncate.rs | 4 +- .../writer/base_writer/data_file_writer.rs | 2 +- .../base_writer/equality_delete_writer.rs | 2 +- .../writer/file_writer/location_generator.rs | 4 +- .../src/writer/file_writer/parquet_writer.rs | 11 +-- .../src/writer/file_writer/rolling_writer.rs | 3 +- .../writer/partitioning/clustered_writer.rs | 6 +- crates/iceberg/tests/file_io_s3_test.rs | 6 +- crates/integration_tests/src/lib.rs | 2 +- .../shared_tests/read_positional_deletes.rs | 4 +- crates/integrations/datafusion/src/error.rs | 2 +- .../datafusion/src/physical_plan/commit.rs | 3 +- .../datafusion/src/physical_plan/scan.rs | 2 +- .../datafusion/src/physical_plan/write.rs | 11 +-- crates/integrations/playground/src/main.rs | 2 +- crates/sqllogictest/src/engine/mod.rs | 2 +- 74 files changed, 241 insertions(+), 442 deletions(-) diff --git a/crates/catalog/glue/src/catalog.rs b/crates/catalog/glue/src/catalog.rs index 9b383df2b1..4514f2d7ab 100644 --- a/crates/catalog/glue/src/catalog.rs +++ b/crates/catalog/glue/src/catalog.rs @@ -293,7 +293,7 @@ impl Catalog for GlueCatalog { } None => Err(Error::new( ErrorKind::DataInvalid, - format!("Database with name: {} does not exist", db_name), + format!("Database with name: {db_name} does not exist"), )), } } @@ -531,8 +531,7 @@ impl Catalog for GlueCatalog { None => Err(Error::new( ErrorKind::TableNotFound, format!( - "Table object for database: {} and table: {} does not exist", - db_name, table_name + "Table object for database: {db_name} and table: {table_name} does not exist" ), )), Some(table) => { @@ -643,8 +642,7 @@ impl Catalog for GlueCatalog { None => Err(Error::new( ErrorKind::TableNotFound, format!( - "'Table' object for database: {} and table: {} does not exist", - src_db_name, src_table_name + "'Table' object for database: {src_db_name} and table: {src_table_name} does not exist" ), )), Some(table) => { @@ -672,10 +670,8 @@ impl Catalog for GlueCatalog { match drop_src_table_result { Ok(_) => Ok(()), Err(_) => { - let err_msg_src_table = format!( - "Failed to drop old table {}.{}.", - src_db_name, src_table_name - ); + let err_msg_src_table = + format!("Failed to drop old table {src_db_name}.{src_table_name}."); let drop_dest_table_result = self.drop_table(dest).await; @@ -683,15 +679,13 @@ impl Catalog for GlueCatalog { Ok(_) => Err(Error::new( ErrorKind::Unexpected, format!( - "{} Rolled back table creation for {}.{}.", - err_msg_src_table, dest_db_name, dest_table_name + "{err_msg_src_table} Rolled back table creation for {dest_db_name}.{dest_table_name}." ), )), Err(_) => Err(Error::new( ErrorKind::Unexpected, format!( - "{} Failed to roll back table creation for {}.{}. Please clean up manually.", - err_msg_src_table, dest_db_name, dest_table_name + "{err_msg_src_table} Failed to roll back table creation for {dest_db_name}.{dest_table_name}. Please clean up manually." ), )), } @@ -753,7 +747,7 @@ impl Catalog for GlueCatalog { format!("Failed to register table {table_ident} due to AWS SDK error"), ), } - .with_source(anyhow!("aws sdk error: {:?}", error)) + .with_source(anyhow!("aws sdk error: {error:?}")) })?; Ok(Table::builder() @@ -811,7 +805,7 @@ impl Catalog for GlueCatalog { format!("Operation failed for table: {table_ident} for hitting aws sdk error"), ), } - .with_source(anyhow!("aws sdk error: {:?}", error)) + .with_source(anyhow!("aws sdk error: {error:?}")) })?; Ok(staged_table) diff --git a/crates/catalog/glue/src/error.rs b/crates/catalog/glue/src/error.rs index 741d8c90c3..c936ff4edd 100644 --- a/crates/catalog/glue/src/error.rs +++ b/crates/catalog/glue/src/error.rs @@ -27,7 +27,7 @@ where T: Debug { ErrorKind::Unexpected, "Operation failed for hitting aws sdk error".to_string(), ) - .with_source(anyhow!("aws sdk error: {:?}", error)) + .with_source(anyhow!("aws sdk error: {error:?}")) } /// Format AWS Build error into iceberg error @@ -36,5 +36,5 @@ pub(crate) fn from_aws_build_error(error: aws_sdk_glue::error::BuildError) -> Er ErrorKind::Unexpected, "Operation failed for hitting aws build error".to_string(), ) - .with_source(anyhow!("aws build error: {:?}", error)) + .with_source(anyhow!("aws build error: {error:?}")) } diff --git a/crates/catalog/glue/src/schema.rs b/crates/catalog/glue/src/schema.rs index 43918484d5..cfd7487973 100644 --- a/crates/catalog/glue/src/schema.rs +++ b/crates/catalog/glue/src/schema.rs @@ -143,7 +143,7 @@ impl SchemaVisitor for GlueSchemaBuilder { } fn list(&mut self, _list: &iceberg::spec::ListType, value: String) -> iceberg::Result { - Ok(format!("array<{}>", value)) + Ok(format!("array<{value}>")) } fn map( @@ -152,7 +152,7 @@ impl SchemaVisitor for GlueSchemaBuilder { key_value: String, value: String, ) -> iceberg::Result { - Ok(format!("map<{},{}>", key_value, value)) + Ok(format!("map<{key_value},{value}>")) } fn primitive(&mut self, p: &iceberg::spec::PrimitiveType) -> iceberg::Result { @@ -171,7 +171,7 @@ impl SchemaVisitor for GlueSchemaBuilder { } PrimitiveType::Binary | PrimitiveType::Fixed(_) => "binary".to_string(), PrimitiveType::Decimal { precision, scale } => { - format!("decimal({},{})", precision, scale) + format!("decimal({precision},{scale})") } _ => { return Err(Error::new( diff --git a/crates/catalog/glue/src/utils.rs b/crates/catalog/glue/src/utils.rs index d2c21b4d74..457471b34a 100644 --- a/crates/catalog/glue/src/utils.rs +++ b/crates/catalog/glue/src/utils.rs @@ -185,8 +185,7 @@ pub(crate) fn validate_namespace(namespace: &NamespaceIdent) -> Result { return Err(Error::new( ErrorKind::DataInvalid, format!( - "Invalid database name: {:?}, hierarchical namespaces are not supported", - namespace + "Invalid database name: {namespace:?}, hierarchical namespaces are not supported" ), )); } @@ -236,7 +235,7 @@ pub(crate) fn get_metadata_location( Some(location) => Ok(location.to_string()), None => Err(Error::new( ErrorKind::DataInvalid, - format!("No '{}' set on table", METADATA_LOCATION), + format!("No '{METADATA_LOCATION}' set on table"), )), }, None => Err(Error::new( diff --git a/crates/catalog/glue/tests/glue_catalog_test.rs b/crates/catalog/glue/tests/glue_catalog_test.rs index 548fc4f0f3..4917032149 100644 --- a/crates/catalog/glue/tests/glue_catalog_test.rs +++ b/crates/catalog/glue/tests/glue_catalog_test.rs @@ -91,7 +91,7 @@ async fn get_catalog() -> GlueCatalog { (AWS_REGION_NAME.to_string(), "us-east-1".to_string()), ( S3_ENDPOINT.to_string(), - format!("http://{}", minio_socket_addr), + format!("http://{minio_socket_addr}"), ), (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), @@ -119,7 +119,7 @@ async fn get_catalog() -> GlueCatalog { let mut glue_props = HashMap::from([ ( GLUE_CATALOG_PROP_URI.to_string(), - format!("http://{}", glue_socket_addr), + format!("http://{glue_socket_addr}"), ), ( GLUE_CATALOG_PROP_WAREHOUSE.to_string(), diff --git a/crates/catalog/hms/src/catalog.rs b/crates/catalog/hms/src/catalog.rs index c8f046cb7e..b7d192210b 100644 --- a/crates/catalog/hms/src/catalog.rs +++ b/crates/catalog/hms/src/catalog.rs @@ -321,7 +321,7 @@ impl Catalog for HmsCatalog { ErrorKind::Unexpected, "Operation failed for hitting thrift error".to_string(), ) - .with_source(anyhow!("thrift error: {:?}", exception))), + .with_source(anyhow!("thrift error: {exception:?}"))), Err(err) => Err(from_thrift_error(err)), } } @@ -554,7 +554,7 @@ impl Catalog for HmsCatalog { ErrorKind::Unexpected, "Operation failed for hitting thrift error".to_string(), ) - .with_source(anyhow!("thrift error: {:?}", exception))), + .with_source(anyhow!("thrift error: {exception:?}"))), Err(err) => Err(from_thrift_error(err)), } } diff --git a/crates/catalog/hms/src/error.rs b/crates/catalog/hms/src/error.rs index 15da3eaf68..ad3cca070c 100644 --- a/crates/catalog/hms/src/error.rs +++ b/crates/catalog/hms/src/error.rs @@ -30,7 +30,7 @@ pub fn from_thrift_error(error: impl std::error::Error) -> Error { ErrorKind::Unexpected, "Operation failed for hitting thrift error".to_string(), ) - .with_source(anyhow!("thrift error: {:?}", error)) + .with_source(anyhow!("thrift error: {error:?}")) } /// Format a thrift exception into iceberg error. @@ -41,7 +41,7 @@ pub fn from_thrift_exception(value: MaybeException) -> Result ErrorKind::Unexpected, "Operation failed for hitting thrift error".to_string(), ) - .with_source(anyhow!("thrift error: {:?}", err))), + .with_source(anyhow!("thrift error: {err:?}"))), } } diff --git a/crates/catalog/hms/src/schema.rs b/crates/catalog/hms/src/schema.rs index 6b7f71c6a0..8893a80521 100644 --- a/crates/catalog/hms/src/schema.rs +++ b/crates/catalog/hms/src/schema.rs @@ -100,7 +100,7 @@ impl SchemaVisitor for HiveSchemaBuilder { } fn list(&mut self, _list: &iceberg::spec::ListType, value: String) -> iceberg::Result { - Ok(format!("array<{}>", value)) + Ok(format!("array<{value}>")) } fn map( @@ -109,7 +109,7 @@ impl SchemaVisitor for HiveSchemaBuilder { key_value: String, value: String, ) -> iceberg::Result { - Ok(format!("map<{},{}>", key_value, value)) + Ok(format!("map<{key_value},{value}>")) } fn primitive(&mut self, p: &iceberg::spec::PrimitiveType) -> iceberg::Result { @@ -128,7 +128,7 @@ impl SchemaVisitor for HiveSchemaBuilder { } PrimitiveType::Binary | PrimitiveType::Fixed(_) => "binary".to_string(), PrimitiveType::Decimal { precision, scale } => { - format!("decimal({},{})", precision, scale) + format!("decimal({precision},{scale})") } _ => { return Err(Error::new( diff --git a/crates/catalog/hms/src/utils.rs b/crates/catalog/hms/src/utils.rs index 32c28b82ee..096e792f61 100644 --- a/crates/catalog/hms/src/utils.rs +++ b/crates/catalog/hms/src/utils.rs @@ -128,7 +128,7 @@ pub(crate) fn convert_to_database( _ => { return Err(Error::new( ErrorKind::DataInvalid, - format!("Invalid value for setting 'owner_type': {}", v), + format!("Invalid value for setting 'owner_type': {v}"), )); } }; @@ -214,8 +214,7 @@ pub(crate) fn validate_namespace(namespace: &NamespaceIdent) -> Result { return Err(Error::new( ErrorKind::DataInvalid, format!( - "Invalid database name: {:?}, hierarchical namespaces are not supported", - namespace + "Invalid database name: {namespace:?}, hierarchical namespaces are not supported" ), )); } @@ -257,7 +256,7 @@ pub(crate) fn get_metadata_location( Some(location) => Ok(location.to_string()), None => Err(Error::new( ErrorKind::DataInvalid, - format!("No '{}' set on table", METADATA_LOCATION), + format!("No '{METADATA_LOCATION}' set on table"), )), }, None => Err(Error::new( @@ -272,7 +271,7 @@ fn format_location_uri(location: String) -> String { let mut location = location; if !location.starts_with('/') { - location = format!("/{}", location); + location = format!("/{location}"); } if location.ends_with('/') && location.len() > 1 { @@ -292,8 +291,7 @@ fn validate_owner_settings(properties: &HashMap) -> Result<()> { return Err(Error::new( ErrorKind::DataInvalid, format!( - "Setting '{}' without setting '{}' is not allowed", - HMS_DB_OWNER_TYPE, HMS_DB_OWNER + "Setting '{HMS_DB_OWNER_TYPE}' without setting '{HMS_DB_OWNER}' is not allowed" ), )); } diff --git a/crates/catalog/hms/tests/hms_catalog_test.rs b/crates/catalog/hms/tests/hms_catalog_test.rs index 2bf12779b1..9793b7f738 100644 --- a/crates/catalog/hms/tests/hms_catalog_test.rs +++ b/crates/catalog/hms/tests/hms_catalog_test.rs @@ -96,7 +96,7 @@ async fn get_catalog() -> HmsCatalog { ), ( S3_ENDPOINT.to_string(), - format!("http://{}", minio_socket_addr), + format!("http://{minio_socket_addr}"), ), (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), diff --git a/crates/catalog/rest/src/catalog.rs b/crates/catalog/rest/src/catalog.rs index 6cb4d93b99..39553f7554 100644 --- a/crates/catalog/rest/src/catalog.rs +++ b/crates/catalog/rest/src/catalog.rs @@ -243,7 +243,7 @@ impl RestCatalogConfig { ), ( header::USER_AGENT, - HeaderValue::from_str(&format!("iceberg-rs/{}", CARGO_PKG_VERSION)).unwrap(), + HeaderValue::from_str(&format!("iceberg-rs/{CARGO_PKG_VERSION}")).unwrap(), ), ]); @@ -1273,7 +1273,7 @@ mod tests { ), ( header::USER_AGENT, - HeaderValue::from_str(&format!("iceberg-rs/{}", CARGO_PKG_VERSION)).unwrap(), + HeaderValue::from_str(&format!("iceberg-rs/{CARGO_PKG_VERSION}")).unwrap(), ), ]); assert_eq!(headers, expected_headers); @@ -1310,7 +1310,7 @@ mod tests { ), ( header::USER_AGENT, - HeaderValue::from_str(&format!("iceberg-rs/{}", CARGO_PKG_VERSION)).unwrap(), + HeaderValue::from_str(&format!("iceberg-rs/{CARGO_PKG_VERSION}")).unwrap(), ), ( HeaderName::from_static("customized-header"), diff --git a/crates/catalog/rest/tests/rest_catalog_test.rs b/crates/catalog/rest/tests/rest_catalog_test.rs index 7ce76ec344..59fea0b51f 100644 --- a/crates/catalog/rest/tests/rest_catalog_test.rs +++ b/crates/catalog/rest/tests/rest_catalog_test.rs @@ -72,7 +72,7 @@ async fn get_catalog() -> RestCatalog { "rest", HashMap::from([( REST_CATALOG_PROP_URI.to_string(), - format!("http://{}", rest_socket_addr), + format!("http://{rest_socket_addr}"), )]), ) .await diff --git a/crates/catalog/s3tables/src/catalog.rs b/crates/catalog/s3tables/src/catalog.rs index daa6590551..3606fac99a 100644 --- a/crates/catalog/s3tables/src/catalog.rs +++ b/crates/catalog/s3tables/src/catalog.rs @@ -637,7 +637,7 @@ impl Catalog for S3TablesCatalog { "Operation failed for hitting aws sdk error", ), } - .with_source(anyhow::Error::msg(format!("aws sdk error: {:?}", error))) + .with_source(anyhow::Error::msg(format!("aws sdk error: {error:?}"))) })?; Ok(staged_table) @@ -649,7 +649,7 @@ pub(crate) fn from_aws_sdk_error(error: aws_sdk_s3tables::error::SdkError) where T: std::fmt::Debug { Error::new( ErrorKind::Unexpected, - format!("Operation failed for hitting aws sdk error: {:?}", error), + format!("Operation failed for hitting aws sdk error: {error:?}"), ) } @@ -682,7 +682,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; let namespaces = catalog.list_namespaces(None).await.unwrap(); @@ -694,7 +694,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; let tables = catalog @@ -709,7 +709,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; let table = catalog @@ -719,7 +719,7 @@ mod tests { )) .await .unwrap(); - println!("{:?}", table); + println!("{table:?}"); } #[tokio::test] @@ -727,7 +727,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; let namespace = NamespaceIdent::new("test_s3tables_create_delete_namespace".to_string()); @@ -745,7 +745,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; let creation = { @@ -788,7 +788,7 @@ mod tests { let catalog = match load_s3tables_catalog_from_env().await { Ok(Some(catalog)) => catalog, Ok(None) => return, - Err(e) => panic!("Error loading catalog: {}", e), + Err(e) => panic!("Error loading catalog: {e}"), }; // Create a test namespace and table diff --git a/crates/catalog/sql/src/catalog.rs b/crates/catalog/sql/src/catalog.rs index b6dc4f835b..77b35a228f 100644 --- a/crates/catalog/sql/src/catalog.rs +++ b/crates/catalog/sql/src/catalog.rs @@ -410,7 +410,7 @@ impl Catalog for SqlCatalog { if exists { return Err(Error::new( iceberg::ErrorKind::Unexpected, - format!("Namespace {:?} already exists", namespace), + format!("Namespace {namespace:?} already exists"), )); } @@ -1608,7 +1608,7 @@ mod tests { assert_eq!( err.message(), - format!("No such namespace: {:?}", namespace_ident) + format!("No such namespace: {namespace_ident:?}") ); } @@ -1630,7 +1630,7 @@ mod tests { assert_eq!( err.message(), - format!("No such namespace: {:?}", namespace_ident) + format!("No such namespace: {namespace_ident:?}") ); } @@ -1714,10 +1714,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "Unexpected => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("Unexpected => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -1735,10 +1732,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "Unexpected => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("Unexpected => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -1785,10 +1779,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "Unexpected => No such namespace: {:?}", - non_existent_namespace_ident - ), + format!("Unexpected => No such namespace: {non_existent_namespace_ident:?}"), ); } @@ -1850,10 +1841,8 @@ mod tests { let table_name = "tbl1"; let expected_table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); - let expected_table_metadata_location_regex = format!( - "^{}/tbl1/metadata/00000-{}.metadata.json$", - namespace_location, UUID_REGEX_STR, - ); + let expected_table_metadata_location_regex = + format!("^{namespace_location}/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$",); let table = catalog .create_table( @@ -1908,8 +1897,7 @@ mod tests { let expected_table_ident = TableIdent::new(nested_namespace_ident.clone(), table_name.into()); let expected_table_metadata_location_regex = format!( - "^{}/tbl1/metadata/00000-{}.metadata.json$", - nested_namespace_location, UUID_REGEX_STR, + "^{nested_namespace_location}/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$", ); let table = catalog @@ -1947,10 +1935,8 @@ mod tests { let table_name = "tbl1"; let expected_table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); - let expected_table_metadata_location_regex = format!( - "^{}/a/tbl1/metadata/00000-{}.metadata.json$", - warehouse_loc, UUID_REGEX_STR - ); + let expected_table_metadata_location_regex = + format!("^{warehouse_loc}/a/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$"); let table = catalog .create_table( @@ -1986,10 +1972,8 @@ mod tests { let table_name = "tbl1"; let expected_table_ident = TableIdent::new(nested_namespace_ident.clone(), table_name.into()); - let expected_table_metadata_location_regex = format!( - "^{}/a/b/tbl1/metadata/00000-{}.metadata.json$", - warehouse_loc, UUID_REGEX_STR - ); + let expected_table_metadata_location_regex = + format!("^{warehouse_loc}/a/b/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$"); let table = catalog .create_table( @@ -2152,10 +2136,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "Unexpected => No such namespace: {:?}", - non_existent_dst_namespace_ident - ), + format!("Unexpected => No such namespace: {non_existent_dst_namespace_ident:?}"), ); } @@ -2174,7 +2155,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!("Unexpected => No such table: {:?}", src_table_ident), + format!("Unexpected => No such table: {src_table_ident:?}"), ); } diff --git a/crates/catalog/sql/src/error.rs b/crates/catalog/sql/src/error.rs index 15b56e8e23..a08f755596 100644 --- a/crates/catalog/sql/src/error.rs +++ b/crates/catalog/sql/src/error.rs @@ -29,20 +29,20 @@ pub fn from_sqlx_error(error: sqlx::Error) -> Error { pub fn no_such_namespace_err(namespace: &NamespaceIdent) -> Result { Err(Error::new( ErrorKind::Unexpected, - format!("No such namespace: {:?}", namespace), + format!("No such namespace: {namespace:?}"), )) } pub fn no_such_table_err(table_ident: &TableIdent) -> Result { Err(Error::new( ErrorKind::Unexpected, - format!("No such table: {:?}", table_ident), + format!("No such table: {table_ident:?}"), )) } pub fn table_already_exists_err(table_ident: &TableIdent) -> Result { Err(Error::new( ErrorKind::Unexpected, - format!("Table {:?} already exists.", table_ident), + format!("Table {table_ident:?} already exists."), )) } diff --git a/crates/examples/src/rest_catalog_namespace.rs b/crates/examples/src/rest_catalog_namespace.rs index 27196a2696..d871de4387 100644 --- a/crates/examples/src/rest_catalog_namespace.rs +++ b/crates/examples/src/rest_catalog_namespace.rs @@ -45,10 +45,7 @@ async fn main() { // ANCHOR: list_all_namespace // List all namespaces already in the catalog. let existing_namespaces = catalog.list_namespaces(None).await.unwrap(); - println!( - "Namespaces alreading in the existing catalog: {:?}", - existing_namespaces - ); + println!("Namespaces alreading in the existing catalog: {existing_namespaces:?}"); // ANCHOR_END: list_all_namespace // ANCHOR: create_namespace @@ -70,9 +67,9 @@ async fn main() { ) .await .unwrap(); - println!("Namespace {:?} created!", namespace_ident); + println!("Namespace {namespace_ident:?} created!"); let loaded_namespace = catalog.get_namespace(&namespace_ident).await.unwrap(); - println!("Namespace loaded!\n\nNamespace: {:#?}", loaded_namespace,); + println!("Namespace loaded!\n\nNamespace: {loaded_namespace:#?}",); // ANCHOR_END: create_namespace } diff --git a/crates/examples/src/rest_catalog_table.rs b/crates/examples/src/rest_catalog_table.rs index 306d886ae6..64acb263f2 100644 --- a/crates/examples/src/rest_catalog_table.rs +++ b/crates/examples/src/rest_catalog_table.rs @@ -96,6 +96,6 @@ async fn main() { // Load the table back from the catalog. It should be identical to the created table. let loaded_table = catalog.load_table(&table_ident).await.unwrap(); - println!("Table {TABLE_NAME} loaded!\n\nTable: {:?}", loaded_table); + println!("Table {TABLE_NAME} loaded!\n\nTable: {loaded_table:?}"); // ANCHOR_END: load_table } diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 9cf605680e..77f29b7f1e 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -568,7 +568,7 @@ mod tests { ) .await .expect("error parsing batch stream"); - println!("{}", parsed_eq_delete); + println!("{parsed_eq_delete}"); let expected = "((((y != 1) OR (z != 100)) OR (a != \"HELP\")) OR (sa != 4)) AND ((((y != 2) OR (z IS NOT NULL)) OR (a IS NOT NULL)) OR (sa != 5))".to_string(); diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index 592ef2eb4a..54fc4a58a7 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -66,7 +66,7 @@ impl BasicDeleteFileLoader { ) .await? .build()? - .map_err(|e| Error::new(ErrorKind::Unexpected, format!("{}", e))); + .map_err(|e| Error::new(ErrorKind::Unexpected, format!("{e}"))); Ok(Box::pin(record_batch_stream) as ArrowRecordBatchStream) } diff --git a/crates/iceberg/src/arrow/partition_value_calculator.rs b/crates/iceberg/src/arrow/partition_value_calculator.rs index 1409503451..3520f75ac5 100644 --- a/crates/iceberg/src/arrow/partition_value_calculator.rs +++ b/crates/iceberg/src/arrow/partition_value_calculator.rs @@ -159,7 +159,7 @@ impl PartitionValueCalculator { .map_err(|e| { Error::new( ErrorKind::DataInvalid, - format!("Failed to create partition struct array: {}", e), + format!("Failed to create partition struct array: {e}"), ) })?; diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index c6f5af2f2a..3c18c0337b 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -799,8 +799,7 @@ fn build_field_id_map(parquet_schema: &SchemaDescriptor) -> Result(); let first_val = id_col.value(0); let last_val = id_col.value(id_col.len() - 1); - println!("Task 1 data range: {} to {}", first_val, last_val); + println!("Task 1 data range: {first_val} to {last_val}"); assert_eq!(first_val, 0, "Task 1 should start with id=0"); assert_eq!(last_val, 99, "Task 1 should end with id=99"); @@ -2188,7 +2182,7 @@ message schema { .column(0) .as_primitive::(); let first_val = id_col.value(0); - println!("Task 2 first value: {}", first_val); + println!("Task 2 first value: {first_val}"); assert_eq!(first_val, 100, "Task 2 should start with id=100, not id=0"); } @@ -2246,7 +2240,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/old_file.parquet", table_location), + data_file_path: format!("{table_location}/old_file.parquet"), data_file_format: DataFileFormat::Parquet, schema: new_schema.clone(), project_field_ids: vec![1, 2], // Request both columns 'a' and 'b' diff --git a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs index 66371fac16..2508a00320 100644 --- a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs +++ b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs @@ -149,8 +149,7 @@ impl RecordBatchPartitionSplitter { Error::new( ErrorKind::DataInvalid, format!( - "Partition column '{}' not found in batch", - PROJECTED_PARTITION_VALUE_COLUMN + "Partition column '{PROJECTED_PARTITION_VALUE_COLUMN}' not found in batch" ), ) })?; diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index 71fe59dea5..5fbbbb106a 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -307,7 +307,7 @@ impl RecordBatchTransformer { let Literal::Primitive(primitive_literal) = iceberg_default_value else { return Err(Error::new( ErrorKind::Unexpected, - format!("Default value for column must be primitive type, but encountered {:?}", iceberg_default_value) + format!("Default value for column must be primitive type, but encountered {iceberg_default_value:?}") )); }; Some(primitive_literal.clone()) @@ -341,7 +341,7 @@ impl RecordBatchTransformer { .map_err(|e| { Error::new( ErrorKind::DataInvalid, - format!("field id not parseable as an i32: {}", e), + format!("field id not parseable as an i32: {e}"), ) })?; @@ -447,7 +447,7 @@ impl RecordBatchTransformer { (dt, _) => { return Err(Error::new( ErrorKind::Unexpected, - format!("unexpected target column type {}", dt), + format!("unexpected target column type {dt}"), )); } }) diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index c214821dd0..e10db3a5aa 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -701,10 +701,7 @@ pub(crate) fn get_arrow_datum(datum: &Datum) -> Result Err(Error::new( ErrorKind::FeatureUnsupported, - format!( - "Converting datum from type {:?} to arrow not supported yet.", - primitive_type - ), + format!("Converting datum from type {primitive_type:?} to arrow not supported yet."), )), } } @@ -789,7 +786,7 @@ pub(crate) fn get_parquet_stat_min_as_datum( PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Can't convert bytes to i128: {:?}", bytes), + format!("Can't convert bytes to i128: {bytes:?}"), ) })?), )) @@ -936,7 +933,7 @@ pub(crate) fn get_parquet_stat_max_as_datum( PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Can't convert bytes to i128: {:?}", bytes), + format!("Can't convert bytes to i128: {bytes:?}"), ) })?), )) @@ -1745,9 +1742,7 @@ mod tests { assert!( matches!(iceberg_field.field_type.as_ref(), Type::Primitive(t) if *t == expected_iceberg_type), - "Expected {:?} to map to {:?}", - arrow_type, - expected_iceberg_type + "Expected {arrow_type:?} to map to {expected_iceberg_type:?}" ); } diff --git a/crates/iceberg/src/arrow/value.rs b/crates/iceberg/src/arrow/value.rs index cc3a561d57..f1cf225bb4 100644 --- a/crates/iceberg/src/arrow/value.rs +++ b/crates/iceberg/src/arrow/value.rs @@ -263,8 +263,7 @@ impl SchemaWithPartnerVisitor for ArrowArrayToIcebergStructConverter { return Err(Error::new( ErrorKind::DataInvalid, format!( - "The precision or scale ({},{}) of arrow decimal128 array is not compatible with iceberg decimal type ({},{})", - arrow_precision, arrow_scale, precision, scale + "The precision or scale ({arrow_precision},{arrow_scale}) of arrow decimal128 array is not compatible with iceberg decimal type ({precision},{scale})" ), )); } @@ -505,8 +504,7 @@ impl PartnerAccessor for ArrowArrayAccessor { Error::new( ErrorKind::DataInvalid, format!( - "The struct partner is not a struct array, partner: {:?}", - struct_partner + "The struct partner is not a struct array, partner: {struct_partner:?}" ), ) })?; diff --git a/crates/iceberg/src/catalog/memory/catalog.rs b/crates/iceberg/src/catalog/memory/catalog.rs index 6ce3111419..fdb495f6d3 100644 --- a/crates/iceberg/src/catalog/memory/catalog.rs +++ b/crates/iceberg/src/catalog/memory/catalog.rs @@ -506,9 +506,7 @@ mod tests { let regex = Regex::new(regex_str).unwrap(); assert!( regex.is_match(&actual), - "Expected metadata location to match regex, but got location: {} and regex: {}", - actual, - regex + "Expected metadata location to match regex, but got location: {actual} and regex: {regex}" ) } @@ -888,10 +886,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -975,10 +970,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -1058,10 +1050,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -1078,10 +1067,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ) + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}") ) } @@ -1168,10 +1154,8 @@ mod tests { let table_name = "tbl1"; let expected_table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); - let expected_table_metadata_location_regex = format!( - "^{}/tbl1/metadata/00000-{}.metadata.json$", - namespace_location, UUID_REGEX_STR, - ); + let expected_table_metadata_location_regex = + format!("^{namespace_location}/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$",); let table = catalog .create_table( @@ -1230,8 +1214,7 @@ mod tests { let expected_table_ident = TableIdent::new(nested_namespace_ident.clone(), table_name.into()); let expected_table_metadata_location_regex = format!( - "^{}/tbl1/metadata/00000-{}.metadata.json$", - nested_namespace_location, UUID_REGEX_STR, + "^{nested_namespace_location}/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$", ); let table = catalog @@ -1278,10 +1261,8 @@ mod tests { let table_name = "tbl1"; let expected_table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); - let expected_table_metadata_location_regex = format!( - "^{}/a/tbl1/metadata/00000-{}.metadata.json$", - warehouse_location, UUID_REGEX_STR - ); + let expected_table_metadata_location_regex = + format!("^{warehouse_location}/a/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$"); let table = catalog .create_table( @@ -1335,8 +1316,7 @@ mod tests { let expected_table_ident = TableIdent::new(nested_namespace_ident.clone(), table_name.into()); let expected_table_metadata_location_regex = format!( - "^{}/a/b/tbl1/metadata/00000-{}.metadata.json$", - warehouse_location, UUID_REGEX_STR + "^{warehouse_location}/a/b/tbl1/metadata/00000-{UUID_REGEX_STR}.metadata.json$" ); let table = catalog @@ -1499,10 +1479,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ), + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}"), ); } @@ -1549,10 +1526,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ), + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}"), ); } @@ -1570,10 +1544,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "TableNotFound => No such table: {:?}", - non_existent_table_ident - ), + format!("TableNotFound => No such table: {non_existent_table_ident:?}"), ); } @@ -1638,10 +1609,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_namespace_ident - ), + format!("NamespaceNotFound => No such namespace: {non_existent_namespace_ident:?}"), ); } @@ -1753,10 +1721,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_src_namespace_ident - ), + format!("NamespaceNotFound => No such namespace: {non_existent_src_namespace_ident:?}"), ); } @@ -1777,10 +1742,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!( - "NamespaceNotFound => No such namespace: {:?}", - non_existent_dst_namespace_ident - ), + format!("NamespaceNotFound => No such namespace: {non_existent_dst_namespace_ident:?}"), ); } @@ -1798,7 +1760,7 @@ mod tests { .await .unwrap_err() .to_string(), - format!("TableNotFound => No such table: {:?}", src_table_ident), + format!("TableNotFound => No such table: {src_table_ident:?}"), ); } diff --git a/crates/iceberg/src/catalog/memory/namespace_state.rs b/crates/iceberg/src/catalog/memory/namespace_state.rs index 2fc481b767..d7dd6c4b2e 100644 --- a/crates/iceberg/src/catalog/memory/namespace_state.rs +++ b/crates/iceberg/src/catalog/memory/namespace_state.rs @@ -36,34 +36,28 @@ pub(crate) struct NamespaceState { fn no_such_namespace_err(namespace_ident: &NamespaceIdent) -> Result { Err(Error::new( ErrorKind::NamespaceNotFound, - format!("No such namespace: {:?}", namespace_ident), + format!("No such namespace: {namespace_ident:?}"), )) } fn no_such_table_err(table_ident: &TableIdent) -> Result { Err(Error::new( ErrorKind::TableNotFound, - format!("No such table: {:?}", table_ident), + format!("No such table: {table_ident:?}"), )) } fn namespace_already_exists_err(namespace_ident: &NamespaceIdent) -> Result { Err(Error::new( ErrorKind::NamespaceAlreadyExists, - format!( - "Cannot create namespace {:?}. Namespace already exists.", - namespace_ident - ), + format!("Cannot create namespace {namespace_ident:?}. Namespace already exists."), )) } fn table_already_exists_err(table_ident: &TableIdent) -> Result { Err(Error::new( ErrorKind::TableAlreadyExists, - format!( - "Cannot create table {:?}. Table already exists.", - table_ident - ), + format!("Cannot create table {table_ident:?}. Table already exists."), )) } diff --git a/crates/iceberg/src/catalog/metadata_location.rs b/crates/iceberg/src/catalog/metadata_location.rs index 8cb5cb11d2..3705ee42dc 100644 --- a/crates/iceberg/src/catalog/metadata_location.rs +++ b/crates/iceberg/src/catalog/metadata_location.rs @@ -53,10 +53,7 @@ impl MetadataLocation { fn parse_metadata_path_prefix(path: &str) -> Result { let prefix = path.strip_suffix("/metadata").ok_or(Error::new( ErrorKind::Unexpected, - format!( - "Metadata location not under \"/metadata\" subdirectory: {}", - path - ), + format!("Metadata location not under \"/metadata\" subdirectory: {path}"), ))?; Ok(prefix.to_string()) @@ -68,12 +65,12 @@ impl MetadataLocation { .strip_suffix(".metadata.json") .ok_or(Error::new( ErrorKind::Unexpected, - format!("Invalid metadata file ending: {}", file_name), + format!("Invalid metadata file ending: {file_name}"), ))? .split_once('-') .ok_or(Error::new( ErrorKind::Unexpected, - format!("Invalid metadata file name format: {}", file_name), + format!("Invalid metadata file name format: {file_name}"), ))?; Ok((version.parse::()?, Uuid::parse_str(id)?)) @@ -96,7 +93,7 @@ impl FromStr for MetadataLocation { fn from_str(s: &str) -> Result { let (path, file_name) = s.rsplit_once('/').ok_or(Error::new( ErrorKind::Unexpected, - format!("Invalid metadata location: {}", s), + format!("Invalid metadata location: {s}"), ))?; let prefix = Self::parse_metadata_path_prefix(path)?; diff --git a/crates/iceberg/src/catalog/mod.rs b/crates/iceberg/src/catalog/mod.rs index 77b171ee30..ec4b77fe01 100644 --- a/crates/iceberg/src/catalog/mod.rs +++ b/crates/iceberg/src/catalog/mod.rs @@ -662,7 +662,7 @@ impl TableRequirement { let snapshot_ref = snapshot_ref.ok_or( Error::new( ErrorKind::CatalogCommitConflicts, - format!("Requirement failed: Branch or tag `{}` not found", r#ref), + format!("Requirement failed: Branch or tag `{ref}` not found"), ) .with_retryable(true), )?; @@ -670,8 +670,7 @@ impl TableRequirement { return Err(Error::new( ErrorKind::CatalogCommitConflicts, format!( - "Requirement failed: Branch or tag `{}`'s snapshot has changed", - r#ref + "Requirement failed: Branch or tag `{ref}`'s snapshot has changed" ), ) .with_context("expected", snapshot_id.to_string()) @@ -682,10 +681,7 @@ impl TableRequirement { // a null snapshot ID means the ref should not exist already return Err(Error::new( ErrorKind::CatalogCommitConflicts, - format!( - "Requirement failed: Branch or tag `{}` already exists", - r#ref - ), + format!("Requirement failed: Branch or tag `{ref}` already exists"), ) .with_retryable(true)); } diff --git a/crates/iceberg/src/expr/predicate.rs b/crates/iceberg/src/expr/predicate.rs index 123cfada5f..35e88ff3e7 100644 --- a/crates/iceberg/src/expr/predicate.rs +++ b/crates/iceberg/src/expr/predicate.rs @@ -54,7 +54,7 @@ impl<'de, T: Deserialize<'de>, const N: usize> Deserialize<'de> for LogicalExpre let inputs = Vec::>::deserialize(deserializer)?; Ok(LogicalExpression::new( array_init::from_iter(inputs.into_iter()).ok_or_else(|| { - serde::de::Error::custom(format!("Failed to deserialize LogicalExpression: the len of inputs is not match with the len of LogicalExpression {}",N)) + serde::de::Error::custom(format!("Failed to deserialize LogicalExpression: the len of inputs is not match with the len of LogicalExpression {N}")) })?, )) } @@ -310,7 +310,7 @@ impl Bind for SetExpression { impl Display for SetExpression { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - let mut literal_strs = self.literals.iter().map(|l| format!("{}", l)); + let mut literal_strs = self.literals.iter().map(|l| format!("{l}")); write!(f, "{} {} ({})", self.term, self.op, literal_strs.join(", ")) } @@ -529,13 +529,13 @@ impl Display for Predicate { write!(f, "NOT ({})", expr.inputs()[0]) } Predicate::Unary(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } Predicate::Binary(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } Predicate::Set(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } } } @@ -805,13 +805,13 @@ impl Display for BoundPredicate { write!(f, "NOT ({})", expr.inputs()[0]) } BoundPredicate::Unary(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } BoundPredicate::Binary(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } BoundPredicate::Set(expr) => { - write!(f, "{}", expr) + write!(f, "{expr}") } } } diff --git a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs index ea56c32c66..3745d94d18 100644 --- a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs @@ -150,17 +150,14 @@ impl<'a> PageIndexEvaluator<'a> { let Some(field) = self.snapshot_schema.field_by_id(field_id) else { return Err(Error::new( ErrorKind::Unexpected, - format!("Field with id {} missing from snapshot schema", field_id), + format!("Field with id {field_id} missing from snapshot schema"), )); }; let Some(field_type) = field.field_type.as_primitive_type() else { return Err(Error::new( ErrorKind::Unexpected, - format!( - "Field with id {} not convertible to primitive type", - field_id - ), + format!("Field with id {field_id} not convertible to primitive type"), )); }; @@ -180,7 +177,7 @@ impl<'a> PageIndexEvaluator<'a> { // if we have a column index, we should always have an offset index. return Err(Error::new( ErrorKind::Unexpected, - format!("Missing offset index for field id {}", field_id), + format!("Missing offset index for field id {field_id}"), )); }; diff --git a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs index e17c44c658..e9bed775ef 100644 --- a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs @@ -1045,8 +1045,7 @@ mod test { // For zero-record files, strict eval returns MUST_MATCH. assert!( result, - "Strict eval: Should read zero-record file for expression {:?}", - expr + "Strict eval: Should read zero-record file for expression {expr:?}" ); } } @@ -1142,11 +1141,7 @@ mod test { let result = StrictMetricsEvaluator::eval(&less_than_int("id", INT_MIN_VALUE), &file).unwrap(); - assert!( - !result, - "Strict eval: id < {} should be false", - INT_MIN_VALUE - ); + assert!(!result, "Strict eval: id < {INT_MIN_VALUE} should be false"); let result = StrictMetricsEvaluator::eval(&less_than_int("id", INT_MIN_VALUE + 1), &file).unwrap(); @@ -1158,11 +1153,7 @@ mod test { let result = StrictMetricsEvaluator::eval(&less_than_int("id", INT_MAX_VALUE), &file).unwrap(); - assert!( - !result, - "Strict eval: id < {} should be false", - INT_MAX_VALUE - ); + assert!(!result, "Strict eval: id < {INT_MAX_VALUE} should be false"); let result = StrictMetricsEvaluator::eval(&less_than_int("id", INT_MAX_VALUE + 1), &file).unwrap(); @@ -1200,18 +1191,13 @@ mod test { .unwrap(); assert!( !result, - "Strict eval: id <= {} should be false", - INT_MIN_VALUE + "Strict eval: id <= {INT_MIN_VALUE} should be false" ); let result = StrictMetricsEvaluator::eval(&less_than_or_equal_int("id", INT_MAX_VALUE), &file) .unwrap(); - assert!( - result, - "Strict eval: id <= {} should be true", - INT_MAX_VALUE - ); + assert!(result, "Strict eval: id <= {INT_MAX_VALUE} should be true"); let result = StrictMetricsEvaluator::eval(&less_than_or_equal_int("id", INT_MAX_VALUE + 1), &file) @@ -1238,19 +1224,11 @@ mod test { let result = StrictMetricsEvaluator::eval(&greater_than_int("id", INT_MAX_VALUE), &file).unwrap(); - assert!( - !result, - "Strict eval: id > {} should be false", - INT_MAX_VALUE - ); + assert!(!result, "Strict eval: id > {INT_MAX_VALUE} should be false"); let result = StrictMetricsEvaluator::eval(&greater_than_int("id", INT_MIN_VALUE), &file).unwrap(); - assert!( - !result, - "Strict eval: id > {} should be false", - INT_MIN_VALUE - ); + assert!(!result, "Strict eval: id > {INT_MIN_VALUE} should be false"); let result = StrictMetricsEvaluator::eval(&greater_than_int("id", INT_MIN_VALUE - 1), &file) @@ -1302,18 +1280,13 @@ mod test { .unwrap(); assert!( !result, - "Strict eval: id >= {} should be false", - INT_MAX_VALUE + "Strict eval: id >= {INT_MAX_VALUE} should be false" ); let result = StrictMetricsEvaluator::eval(&greater_than_or_equal_int("id", INT_MIN_VALUE), &file) .unwrap(); - assert!( - result, - "Strict eval: id >= {} should be true", - INT_MIN_VALUE - ); + assert!(result, "Strict eval: id >= {INT_MIN_VALUE} should be true"); let result = StrictMetricsEvaluator::eval( &greater_than_or_equal_int("id", INT_MIN_VALUE - 1), diff --git a/crates/iceberg/src/io/object_cache.rs b/crates/iceberg/src/io/object_cache.rs index a23ff36b32..cb9f03fa5a 100644 --- a/crates/iceberg/src/io/object_cache.rs +++ b/crates/iceberg/src/io/object_cache.rs @@ -113,7 +113,7 @@ impl ObjectCache { CachedItem::Manifest(arc_manifest) => Ok(arc_manifest), _ => Err(Error::new( ErrorKind::Unexpected, - format!("cached object for key '{:?}' is not a Manifest", key), + format!("cached object for key '{key:?}' is not a Manifest"), )), } } @@ -157,7 +157,7 @@ impl ObjectCache { CachedItem::ManifestList(arc_manifest_list) => Ok(arc_manifest_list), _ => Err(Error::new( ErrorKind::Unexpected, - format!("cached object for path '{:?}' is not a manifest list", key), + format!("cached object for path '{key:?}' is not a manifest list"), )), } } diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index 3de4f10dbf..d5f2ad8fab 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -162,7 +162,7 @@ impl Storage { } else { Err(Error::new( ErrorKind::DataInvalid, - format!("Invalid s3 url: {}, should start with {}", path, prefix), + format!("Invalid s3 url: {path}, should start with {prefix}"), )) } } diff --git a/crates/iceberg/src/io/storage_s3.rs b/crates/iceberg/src/io/storage_s3.rs index f2408331c5..fcf9afed1f 100644 --- a/crates/iceberg/src/io/storage_s3.rs +++ b/crates/iceberg/src/io/storage_s3.rs @@ -127,8 +127,7 @@ pub(crate) fn s3_config_parse(mut m: HashMap) -> Result Given string is not valid JSON, source: invalid value: integer `{}`, expected i32 at line 5 column 51", - out_of_i32_range_number + "DataInvalid => Given string is not valid JSON, source: invalid value: integer `{out_of_i32_range_number}`, expected i32 at line 5 column 51" ), ) } diff --git a/crates/iceberg/src/scan/cache.rs b/crates/iceberg/src/scan/cache.rs index fc7e559e61..66242e389b 100644 --- a/crates/iceberg/src/scan/cache.rs +++ b/crates/iceberg/src/scan/cache.rs @@ -66,7 +66,7 @@ impl PartitionFilterCache { .partition_spec_by_id(spec_id) .ok_or(Error::new( ErrorKind::Unexpected, - format!("Could not find partition spec for id {}", spec_id), + format!("Could not find partition spec for id {spec_id}"), ))?; let partition_type = partition_spec.partition_type(schema)?; diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 3d14b3cce4..92ee2018c0 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -193,7 +193,7 @@ impl<'a> TableScanBuilder<'a> { .ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Snapshot with id {} not found", snapshot_id), + format!("Snapshot with id {snapshot_id} not found"), ) })? .clone(), @@ -223,10 +223,7 @@ impl<'a> TableScanBuilder<'a> { if schema.field_by_name(column_name).is_none() { return Err(Error::new( ErrorKind::DataInvalid, - format!( - "Column {} not found in table. Schema: {}", - column_name, schema - ), + format!("Column {column_name} not found in table. Schema: {schema}"), )); } } @@ -246,10 +243,7 @@ impl<'a> TableScanBuilder<'a> { let field_id = schema.field_id_by_name(column_name).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "Column {} not found in table. Schema: {}", - column_name, schema - ), + format!("Column {column_name} not found in table. Schema: {schema}"), ) })?; @@ -260,8 +254,7 @@ impl<'a> TableScanBuilder<'a> { Error::new( ErrorKind::FeatureUnsupported, format!( - "Column {} is not a direct child of schema but a nested field, which is not supported now. Schema: {}", - column_name, schema + "Column {column_name} is not a direct child of schema but a nested field, which is not supported now. Schema: {schema}" ), ) })?; diff --git a/crates/iceberg/src/spec/datatypes.rs b/crates/iceberg/src/spec/datatypes.rs index 7c03001f88..19facfb017 100644 --- a/crates/iceberg/src/spec/datatypes.rs +++ b/crates/iceberg/src/spec/datatypes.rs @@ -95,8 +95,8 @@ pub enum Type { impl fmt::Display for Type { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self { - Type::Primitive(primitive) => write!(f, "{}", primitive), - Type::Struct(s) => write!(f, "{}", s), + Type::Primitive(primitive) => write!(f, "{primitive}"), + Type::Struct(s) => write!(f, "{s}"), Type::List(_) => write!(f, "list"), Type::Map(_) => write!(f, "map"), } @@ -370,7 +370,7 @@ impl fmt::Display for PrimitiveType { PrimitiveType::Float => write!(f, "float"), PrimitiveType::Double => write!(f, "double"), PrimitiveType::Decimal { precision, scale } => { - write!(f, "decimal({},{})", precision, scale) + write!(f, "decimal({precision},{scale})") } PrimitiveType::Date => write!(f, "date"), PrimitiveType::Time => write!(f, "time"), @@ -380,7 +380,7 @@ impl fmt::Display for PrimitiveType { PrimitiveType::TimestamptzNs => write!(f, "timestamptz_ns"), PrimitiveType::String => write!(f, "string"), PrimitiveType::Uuid => write!(f, "uuid"), - PrimitiveType::Fixed(size) => write!(f, "fixed({})", size), + PrimitiveType::Fixed(size) => write!(f, "fixed({size})"), PrimitiveType::Binary => write!(f, "binary"), } } @@ -673,7 +673,7 @@ impl fmt::Display for NestedField { } write!(f, "{} ", self.field_type)?; if let Some(doc) = &self.doc { - write!(f, "{}", doc)?; + write!(f, "{doc}")?; } Ok(()) } diff --git a/crates/iceberg/src/spec/manifest/data_file.rs b/crates/iceberg/src/spec/manifest/data_file.rs index d7455b56fe..6c63f62235 100644 --- a/crates/iceberg/src/spec/manifest/data_file.rs +++ b/crates/iceberg/src/spec/manifest/data_file.rs @@ -360,7 +360,7 @@ impl TryFrom for DataContentType { 2 => Ok(DataContentType::EqualityDeletes), _ => Err(Error::new( ErrorKind::DataInvalid, - format!("data content type {} is invalid", v), + format!("data content type {v} is invalid"), )), } } @@ -390,7 +390,7 @@ impl FromStr for DataFileFormat { "puffin" => Ok(Self::Puffin), _ => Err(Error::new( ErrorKind::DataInvalid, - format!("Unsupported data file format: {}", s), + format!("Unsupported data file format: {s}"), )), } } diff --git a/crates/iceberg/src/spec/manifest/entry.rs b/crates/iceberg/src/spec/manifest/entry.rs index 7ba9efb3b9..d11d8acfe8 100644 --- a/crates/iceberg/src/spec/manifest/entry.rs +++ b/crates/iceberg/src/spec/manifest/entry.rs @@ -172,7 +172,7 @@ impl TryFrom for ManifestStatus { 2 => Ok(ManifestStatus::Deleted), _ => Err(Error::new( ErrorKind::DataInvalid, - format!("manifest status {} is invalid", v), + format!("manifest status {v} is invalid"), )), } } diff --git a/crates/iceberg/src/spec/manifest_list.rs b/crates/iceberg/src/spec/manifest_list.rs index 43808bb249..16409ffe56 100644 --- a/crates/iceberg/src/spec/manifest_list.rs +++ b/crates/iceberg/src/spec/manifest_list.rs @@ -642,10 +642,7 @@ impl TryFrom for ManifestContentType { 1 => Ok(ManifestContentType::Deletes), _ => Err(Error::new( crate::ErrorKind::DataInvalid, - format!( - "Invalid manifest content type. Expected 0 or 1, got {}", - value - ), + format!("Invalid manifest content type. Expected 0 or 1, got {value}"), )), } } diff --git a/crates/iceberg/src/spec/partition.rs b/crates/iceberg/src/spec/partition.rs index b594895ccd..255aabd476 100644 --- a/crates/iceberg/src/spec/partition.rs +++ b/crates/iceberg/src/spec/partition.rs @@ -683,7 +683,7 @@ trait CorePartitionSpecValidator { if self.fields().iter().any(|f| f.name == name) { return Err(Error::new( ErrorKind::DataInvalid, - format!("Cannot use partition name more than once: {}", name), + format!("Cannot use partition name more than once: {name}"), )); } Ok(()) @@ -715,10 +715,7 @@ trait CorePartitionSpecValidator { if self.fields().iter().any(|f| f.field_id == Some(field_id)) { return Err(Error::new( ErrorKind::DataInvalid, - format!( - "Cannot use field id more than once in one PartitionSpec: {}", - field_id - ), + format!("Cannot use field id more than once in one PartitionSpec: {field_id}"), )); } diff --git a/crates/iceberg/src/spec/schema/id_reassigner.rs b/crates/iceberg/src/spec/schema/id_reassigner.rs index 0443831853..5dbb370001 100644 --- a/crates/iceberg/src/spec/schema/id_reassigner.rs +++ b/crates/iceberg/src/spec/schema/id_reassigner.rs @@ -122,7 +122,7 @@ impl ReassignFieldIds { self.old_to_new_id.get(&id).copied().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Identifier Field ID {} not found", id), + format!("Identifier Field ID {id} not found"), ) }) }) @@ -142,7 +142,7 @@ impl ReassignFieldIds { .ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Field with id {} for alias {} not found", id, name), + format!("Field with id {id} for alias {name} not found"), ) }) .map(|new_id| (name, new_id)) diff --git a/crates/iceberg/src/spec/schema/mod.rs b/crates/iceberg/src/spec/schema/mod.rs index 6554629987..7080b6e700 100644 --- a/crates/iceberg/src/spec/schema/mod.rs +++ b/crates/iceberg/src/spec/schema/mod.rs @@ -425,7 +425,7 @@ impl Display for Schema { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { writeln!(f, "table {{")?; for field in self.as_struct().fields() { - writeln!(f, " {}", field)?; + writeln!(f, " {field}")?; } writeln!(f, "}}") } @@ -731,8 +731,7 @@ table { assert_eq!( Some(name), schema.name_by_field_id(id), - "Column name for field id {} not match.", - id + "Column name for field id {id} not match." ); } } @@ -754,8 +753,7 @@ table { assert_eq!( Some(name), schema.name_by_field_id(id), - "Column name for field id {} not match.", - id + "Column name for field id {id} not match." ); } } @@ -956,8 +954,7 @@ table { assert_eq!( Some(&field), schema.field_by_id(id).map(|f| f.as_ref()), - "Field for {} not match.", - id + "Field for {id} not match." ); } } diff --git a/crates/iceberg/src/spec/schema/utils.rs b/crates/iceberg/src/spec/schema/utils.rs index 37e6833441..34ee39b099 100644 --- a/crates/iceberg/src/spec/schema/utils.rs +++ b/crates/iceberg/src/spec/schema/utils.rs @@ -25,10 +25,7 @@ pub fn try_insert_field(map: &mut HashMap, field_id: i32, value: V) - |_| { Err(Error::new( ErrorKind::DataInvalid, - format!( - "Found duplicate 'field.id' {}. Field ids must be unique.", - field_id - ), + format!("Found duplicate 'field.id' {field_id}. Field ids must be unique."), )) }, ) diff --git a/crates/iceberg/src/spec/snapshot.rs b/crates/iceberg/src/spec/snapshot.rs index 04a9e15b34..809bf0998a 100644 --- a/crates/iceberg/src/spec/snapshot.rs +++ b/crates/iceberg/src/spec/snapshot.rs @@ -166,7 +166,7 @@ impl Snapshot { .ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Schema with id {} not found", schema_id), + format!("Schema with id {schema_id} not found"), ) })? .clone(), diff --git a/crates/iceberg/src/spec/snapshot_summary.rs b/crates/iceberg/src/spec/snapshot_summary.rs index 1b07ce3fa0..a9dd569985 100644 --- a/crates/iceberg/src/spec/snapshot_summary.rs +++ b/crates/iceberg/src/spec/snapshot_summary.rs @@ -824,9 +824,9 @@ mod tests { assert!(props.contains_key(&partition_key)); let partition_summary = props.get(&partition_key).unwrap(); - assert!(partition_summary.contains(&format!("{}=200", ADDED_FILE_SIZE))); - assert!(partition_summary.contains(&format!("{}=1", ADDED_DATA_FILES))); - assert!(partition_summary.contains(&format!("{}=20", ADDED_RECORDS))); + assert!(partition_summary.contains(&format!("{ADDED_FILE_SIZE}=200"))); + assert!(partition_summary.contains(&format!("{ADDED_DATA_FILES}=1"))); + assert!(partition_summary.contains(&format!("{ADDED_RECORDS}=20"))); } #[test] diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index ca298f308d..a98dc4f437 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -315,7 +315,7 @@ impl TableMetadata { pub fn snapshot_for_ref(&self, ref_name: &str) -> Option<&SnapshotRef> { self.refs.get(ref_name).map(|r| { self.snapshot_by_id(r.snapshot_id) - .unwrap_or_else(|| panic!("Snapshot id of ref {} doesn't exist", ref_name)) + .unwrap_or_else(|| panic!("Snapshot id of ref {ref_name} doesn't exist")) }) } @@ -513,8 +513,7 @@ impl TableMetadata { return Err(Error::new( ErrorKind::DataInvalid, format!( - "Snapshot for current snapshot id {} does not exist in the existing snapshots list", - current_snapshot_id + "Snapshot for current snapshot id {current_snapshot_id} does not exist in the existing snapshots list" ), )); } @@ -956,10 +955,7 @@ pub(super) mod _serde { .ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "No schema exists with the current schema id {}.", - schema_id - ), + format!("No schema exists with the current schema id {schema_id}."), ) })? .clone(); @@ -1346,7 +1342,7 @@ mod tests { } fn get_test_table_metadata(file_name: &str) -> TableMetadata { - let path = format!("testdata/table_metadata/{}", file_name); + let path = format!("testdata/table_metadata/{file_name}"); let metadata: String = fs::read_to_string(path).unwrap(); serde_json::from_str(&metadata).unwrap() @@ -2118,7 +2114,6 @@ mod tests { "#; let err = serde_json::from_str::(data).unwrap_err(); - println!("{}", err); assert!(err.to_string().contains( "Invalid snapshot with id 3055729675574597004 and sequence number 4 greater than last sequence number 1" )); @@ -2805,8 +2800,7 @@ mod tests { let error_message = desered.unwrap_err().to_string(); assert!( error_message.contains("No valid schema configuration found"), - "Expected error about no valid schema configuration, got: {}", - error_message + "Expected error about no valid schema configuration, got: {error_message}" ); } @@ -3033,7 +3027,7 @@ mod tests { let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json"); // Define the metadata location - let metadata_location = format!("{}/metadata.json", temp_path); + let metadata_location = format!("{temp_path}/metadata.json"); // Write the metadata original_metadata diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 7881ebea40..25af8c30e5 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -653,10 +653,7 @@ impl TableMetadataBuilder { let _schema = self.metadata.schemas.get(&schema_id).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "Cannot set current schema to unknown schema with id: '{}'", - schema_id - ), + format!("Cannot set current schema to unknown schema with id: '{schema_id}'"), ) })?; @@ -706,9 +703,8 @@ impl TableMetadataBuilder { return Err(Error::new( ErrorKind::DataInvalid, format!( - "Cannot add schema field '{}' because it conflicts with existing partition field name. \ - Schema evolution cannot introduce field names that match existing partition field names.", - field_name + "Cannot add schema field '{field_name}' because it conflicts with existing partition field name. \ + Schema evolution cannot introduce field names that match existing partition field names." ), )); } @@ -1209,8 +1205,7 @@ impl TableMetadataBuilder { Error::new( ErrorKind::Unexpected, format!( - "Cannot find source column with name {} for sort column in re-assigned schema.", - source_field_name + "Cannot find source column with name {source_field_name} for sort column in re-assigned schema." ), ) })?.id; diff --git a/crates/iceberg/src/spec/table_properties.rs b/crates/iceberg/src/spec/table_properties.rs index 9aa789fed8..4975456010 100644 --- a/crates/iceberg/src/spec/table_properties.rs +++ b/crates/iceberg/src/spec/table_properties.rs @@ -30,7 +30,7 @@ where properties.get(key).map_or(Ok(default), |value| { value .parse::() - .map_err(|e| anyhow::anyhow!("Invalid value for {}: {}", key, e)) + .map_err(|e| anyhow::anyhow!("Invalid value for {key}: {e}")) }) } diff --git a/crates/iceberg/src/spec/transform.rs b/crates/iceberg/src/spec/transform.rs index d69d15c288..6068716eff 100644 --- a/crates/iceberg/src/spec/transform.rs +++ b/crates/iceberg/src/spec/transform.rs @@ -894,7 +894,7 @@ impl Transform { ) { return Err(Error::new( ErrorKind::DataInvalid, - format!("Expected a numeric literal, got: {:?}", boundary), + format!("Expected a numeric literal, got: {boundary:?}"), )); } diff --git a/crates/iceberg/src/spec/values.rs b/crates/iceberg/src/spec/values.rs index 8d56604b1c..d06e754dcc 100644 --- a/crates/iceberg/src/spec/values.rs +++ b/crates/iceberg/src/spec/values.rs @@ -351,11 +351,11 @@ impl PartialOrd for Datum { impl Display for Datum { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match (&self.r#type, &self.literal) { - (_, PrimitiveLiteral::Boolean(val)) => write!(f, "{}", val), - (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => write!(f, "{}", val), - (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => write!(f, "{}", val), - (_, PrimitiveLiteral::Float(val)) => write!(f, "{}", val), - (_, PrimitiveLiteral::Double(val)) => write!(f, "{}", val), + (_, PrimitiveLiteral::Boolean(val)) => write!(f, "{val}"), + (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => write!(f, "{val}"), + (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => write!(f, "{val}"), + (_, PrimitiveLiteral::Float(val)) => write!(f, "{val}"), + (_, PrimitiveLiteral::Double(val)) => write!(f, "{val}"), (PrimitiveType::Date, PrimitiveLiteral::Int(val)) => { write!(f, "{}", days_to_date(*val)) } @@ -374,7 +374,7 @@ impl Display for Datum { (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(val)) => { write!(f, "{}", nanoseconds_to_datetimetz(*val)) } - (_, PrimitiveLiteral::String(val)) => write!(f, r#""{}""#, val), + (_, PrimitiveLiteral::String(val)) => write!(f, r#""{val}""#), (PrimitiveType::Uuid, PrimitiveLiteral::UInt128(val)) => { write!(f, "{}", Uuid::from_u128(*val)) } @@ -398,7 +398,7 @@ impl Display for Datum { fn display_bytes(bytes: &[u8], f: &mut Formatter<'_>) -> std::fmt::Result { let mut s = String::with_capacity(bytes.len() * 2); for b in bytes { - s.push_str(&format!("{:02X}", b)); + s.push_str(&format!("{b:02X}")); } f.write_str(&s) } @@ -482,7 +482,7 @@ impl Datum { PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Can't convert bytes to i128: {:?}", bytes), + format!("Can't convert bytes to i128: {bytes:?}"), ) })?) } @@ -526,8 +526,7 @@ impl Datum { return Err(Error::new( ErrorKind::DataInvalid, format!( - "PrimitiveType Decimal must has valid precision but got {}", - precision + "PrimitiveType Decimal must has valid precision but got {precision}" ), )); }; @@ -1134,10 +1133,7 @@ impl Datum { if actual_bytes.len() > available_bytes { return Err(Error::new( ErrorKind::DataInvalid, - format!( - "Decimal value {} is too large for precision {}", - decimal, precision - ), + format!("Decimal value {decimal} is too large for precision {precision}"), )); } @@ -1933,10 +1929,7 @@ impl Literal { (_, JsonValue::Null) => Ok(None), (i, j) => Err(Error::new( crate::ErrorKind::DataInvalid, - format!( - "The json value {} doesn't fit to the iceberg type {}.", - j, i - ), + format!("The json value {j} doesn't fit to the iceberg type {i}."), )), }, Type::Struct(schema) => { @@ -2081,7 +2074,7 @@ impl Literal { (_, PrimitiveLiteral::Binary(val)) => Ok(JsonValue::String(val.iter().fold( String::new(), |mut acc, x| { - acc.push_str(&format!("{:x}", x)); + acc.push_str(&format!("{x:x}")); acc }, ))), @@ -2139,10 +2132,7 @@ impl Literal { } (value, r#type) => Err(Error::new( ErrorKind::DataInvalid, - format!( - "The iceberg value {:?} doesn't fit to the iceberg type {}.", - value, r#type - ), + format!("The iceberg value {value:?} doesn't fit to the iceberg type {type}."), )), } } @@ -2515,7 +2505,7 @@ mod _serde { } else { return Err(Error::new( ErrorKind::DataInvalid, - format!("Type {} should be a struct", ty), + format!("Type {ty} should be a struct"), )); } RawLiteralEnum::Record(Record { required, optional }) @@ -2538,7 +2528,7 @@ mod _serde { } else { return Err(Error::new( ErrorKind::DataInvalid, - format!("Type {} should be a list", ty), + format!("Type {ty} should be a list"), )); } } @@ -2606,7 +2596,7 @@ mod _serde { } else { return Err(Error::new( ErrorKind::DataInvalid, - format!("Type {} should be a map", ty), + format!("Type {ty} should be a map"), )); } } @@ -2619,8 +2609,7 @@ mod _serde { Error::new( ErrorKind::DataInvalid, format!( - "Unable to convert raw literal ({}) fail convert to type {} for: type mismatch", - v, ty + "Unable to convert raw literal ({v}) fail convert to type {ty} for: type mismatch" ), ) }; @@ -2628,8 +2617,7 @@ mod _serde { Error::new( ErrorKind::DataInvalid, format!( - "Unable to convert raw literal ({}) fail convert to type {} for: {}", - v, ty, reason + "Unable to convert raw literal ({v}) fail convert to type {ty} for: {reason}" ), ) }; @@ -3301,7 +3289,7 @@ mod tests { Decimal::new(decimal_num, expect_scale), expect_precision, ); - assert!(result.is_err(), "expect error but got {:?}", result); + assert!(result.is_err(), "expect error but got {result:?}"); assert_eq!( result.unwrap_err().kind(), ErrorKind::DataInvalid, diff --git a/crates/iceberg/src/spec/view_metadata.rs b/crates/iceberg/src/spec/view_metadata.rs index dafca4190e..3139c04081 100644 --- a/crates/iceberg/src/spec/view_metadata.rs +++ b/crates/iceberg/src/spec/view_metadata.rs @@ -186,7 +186,7 @@ impl ViewMetadata { if !self.schemas.contains_key(&schema_id) { return Err(Error::new( ErrorKind::DataInvalid, - format!("No schema exists with the schema id {}.", schema_id), + format!("No schema exists with the schema id {schema_id}."), )); } Ok(()) @@ -426,7 +426,7 @@ pub(crate) mod tests { } pub(crate) fn get_test_view_metadata(file_name: &str) -> ViewMetadata { - let path = format!("testdata/view_metadata/{}", file_name); + let path = format!("testdata/view_metadata/{file_name}"); let metadata: String = fs::read_to_string(path).unwrap(); serde_json::from_str(&metadata).unwrap() diff --git a/crates/iceberg/src/spec/view_metadata_builder.rs b/crates/iceberg/src/spec/view_metadata_builder.rs index dc5c104029..9f542a7c61 100644 --- a/crates/iceberg/src/spec/view_metadata_builder.rs +++ b/crates/iceberg/src/spec/view_metadata_builder.rs @@ -200,10 +200,7 @@ impl ViewMetadataBuilder { let version = self.metadata.versions.get(&version_id).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "Cannot set current version to unknown version with id: {}", - version_id - ), + format!("Cannot set current version to unknown version with id: {version_id}"), ) })?; @@ -433,8 +430,7 @@ impl ViewMetadataBuilder { return Err(Error::new( ErrorKind::DataInvalid, format!( - "{} must be positive but was {}", - VIEW_PROPERTY_VERSION_HISTORY_SIZE, num_versions_to_keep + "{VIEW_PROPERTY_VERSION_HISTORY_SIZE} must be positive but was {num_versions_to_keep}" ), )); } diff --git a/crates/iceberg/src/transaction/update_properties.rs b/crates/iceberg/src/transaction/update_properties.rs index 825d6bec8d..d85072555a 100644 --- a/crates/iceberg/src/transaction/update_properties.rs +++ b/crates/iceberg/src/transaction/update_properties.rs @@ -87,10 +87,7 @@ impl TransactionAction for UpdatePropertiesAction { { return Err(Error::new( ErrorKind::PreconditionFailed, - format!( - "Key {} is present in both removal set and update set", - overlapping_key - ), + format!("Key {overlapping_key} is present in both removal set and update set"), )); } diff --git a/crates/iceberg/src/transform/mod.rs b/crates/iceberg/src/transform/mod.rs index 809d2dafe0..f870a08729 100644 --- a/crates/iceberg/src/transform/mod.rs +++ b/crates/iceberg/src/transform/mod.rs @@ -44,7 +44,7 @@ pub trait TransformFunction: Send + Sync + Debug { self.transform_literal(input)?.ok_or_else(|| { Error::new( ErrorKind::Unexpected, - format!("Returns 'None' for literal {}", input), + format!("Returns 'None' for literal {input}"), ) }) } @@ -171,10 +171,7 @@ mod test { assert_eq!( satisfies_order_of, &trans.satisfies_order_of(other_trans), - "Failed to check satisfies order {}, {}, {}", - trans, - other_trans, - satisfies_order_of + "Failed to check satisfies order {trans}, {other_trans}, {satisfies_order_of}" ); } @@ -182,8 +179,7 @@ mod test { let actual = trans.result_type(input_type).ok(); assert_eq!( result_type, &actual, - "type mismatch at index {}, input: {}, expected: {:?}, actual: {:?}", - i, input_type, result_type, actual + "type mismatch at index {i}, input: {input_type}, expected: {result_type:?}, actual: {actual:?}" ); } } diff --git a/crates/iceberg/src/transform/temporal.rs b/crates/iceberg/src/transform/temporal.rs index 3b333d7bad..d0a0da249b 100644 --- a/crates/iceberg/src/transform/temporal.rs +++ b/crates/iceberg/src/transform/temporal.rs @@ -230,10 +230,7 @@ impl Day { let delta = Duration::new(secs, nanos).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "Failed to create 'TimeDelta' from seconds {} and nanos {}", - secs, nanos - ), + format!("Failed to create 'TimeDelta' from seconds {secs} and nanos {nanos}"), ) })?; @@ -259,10 +256,7 @@ impl Day { let delta = Duration::new(secs, nanos).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!( - "Failed to create 'TimeDelta' from seconds {} and nanos {}", - secs, nanos - ), + format!("Failed to create 'TimeDelta' from seconds {secs} and nanos {nanos}"), ) })?; diff --git a/crates/iceberg/src/transform/truncate.rs b/crates/iceberg/src/transform/truncate.rs index d712df518e..84ef7c0da3 100644 --- a/crates/iceberg/src/transform/truncate.rs +++ b/crates/iceberg/src/transform/truncate.rs @@ -322,7 +322,7 @@ mod test { fixture.assert_projection( &fixture.set_predicate(PredicateOperator::In, vec![ Datum::string(value), - Datum::string(format!("{}abc", value)), + Datum::string(format!("{value}abc")), ]), Some(r#"name IN ("abcde")"#), )?; @@ -330,7 +330,7 @@ mod test { fixture.assert_projection( &fixture.set_predicate(PredicateOperator::NotIn, vec![ Datum::string(value), - Datum::string(format!("{}abc", value)), + Datum::string(format!("{value}abc")), ]), None, )?; diff --git a/crates/iceberg/src/writer/base_writer/data_file_writer.rs b/crates/iceberg/src/writer/base_writer/data_file_writer.rs index b9de4efda7..dcaa56cc97 100644 --- a/crates/iceberg/src/writer/base_writer/data_file_writer.rs +++ b/crates/iceberg/src/writer/base_writer/data_file_writer.rs @@ -101,7 +101,7 @@ where res.build().map_err(|e| { Error::new( ErrorKind::DataInvalid, - format!("Failed to build data file: {}", e), + format!("Failed to build data file: {e}"), ) }) }) diff --git a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs index bd96c80bb3..664ea84334 100644 --- a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs +++ b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs @@ -181,7 +181,7 @@ where res.build().map_err(|e| { Error::new( ErrorKind::DataInvalid, - format!("Failed to build data file: {}", e), + format!("Failed to build data file: {e}"), ) }) }) diff --git a/crates/iceberg/src/writer/file_writer/location_generator.rs b/crates/iceberg/src/writer/file_writer/location_generator.rs index 4a73fc148d..4cfc27844d 100644 --- a/crates/iceberg/src/writer/file_writer/location_generator.rs +++ b/crates/iceberg/src/writer/file_writer/location_generator.rs @@ -63,7 +63,7 @@ impl DefaultLocationGenerator { let data_location = if let Some(data_location) = configured_data_location { data_location.clone() } else { - format!("{}{}", table_location, DEFAULT_DATA_DIR) + format!("{table_location}{DEFAULT_DATA_DIR}") }; Ok(Self { data_location }) } @@ -114,7 +114,7 @@ impl DefaultFileNameGenerator { /// Create a new `FileNameGenerator`. pub fn new(prefix: String, suffix: Option, format: DataFileFormat) -> Self { let suffix = if let Some(suffix) = suffix { - format!("-{}", suffix) + format!("-{suffix}") } else { "".to_string() }; diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 620f27df33..3e9d1715c9 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -274,10 +274,7 @@ impl MinMaxColAggregator { let Type::Primitive(ty) = ty.clone() else { return Err(Error::new( ErrorKind::Unexpected, - format!( - "Composed type {} is not supported for min max aggregation.", - ty - ), + format!("Composed type {ty} is not supported for min max aggregation."), )); }; @@ -285,7 +282,7 @@ impl MinMaxColAggregator { let Some(min_datum) = get_parquet_stat_min_as_datum(&ty, &value)? else { return Err(Error::new( ErrorKind::Unexpected, - format!("Statistics {} is not match with field type {}.", value, ty), + format!("Statistics {value} is not match with field type {ty}."), )); }; @@ -296,7 +293,7 @@ impl MinMaxColAggregator { let Some(max_datum) = get_parquet_stat_max_as_datum(&ty, &value)? else { return Err(Error::new( ErrorKind::Unexpected, - format!("Statistics {} is not match with field type {}.", value, ty), + format!("Statistics {value} is not match with field type {ty}."), )); }; @@ -333,7 +330,7 @@ impl ParquetWriter { let parquet_metadata = parquet_reader.get_metadata(None).await.map_err(|err| { Error::new( ErrorKind::DataInvalid, - format!("Error reading Parquet metadata: {}", err), + format!("Error reading Parquet metadata: {err}"), ) })?; let mut builder = ParquetWriter::parquet_to_data_file_builder( diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index b68010304c..8f03654786 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -436,8 +436,7 @@ mod tests { let total_records: u64 = data_files.iter().map(|file| file.record_count).sum(); assert_eq!( total_records, expected_rows as u64, - "Expected {} total records across all files", - expected_rows + "Expected {expected_rows} total records across all files" ); Ok(()) diff --git a/crates/iceberg/src/writer/partitioning/clustered_writer.rs b/crates/iceberg/src/writer/partitioning/clustered_writer.rs index c141b212f5..3587723965 100644 --- a/crates/iceberg/src/writer/partitioning/clustered_writer.rs +++ b/crates/iceberg/src/writer/partitioning/clustered_writer.rs @@ -101,8 +101,7 @@ where return Err(Error::new( ErrorKind::Unexpected, format!( - "The input is not sorted! Cannot write to partition that was previously closed: {:?}", - partition_key + "The input is not sorted! Cannot write to partition that was previously closed: {partition_key:?}" ), )); } @@ -511,8 +510,7 @@ mod tests { let error = result.unwrap_err(); assert!( error.to_string().contains("The input is not sorted"), - "Expected 'input is not sorted' error, got: {}", - error + "Expected 'input is not sorted' error, got: {error}" ); Ok(()) diff --git a/crates/iceberg/tests/file_io_s3_test.rs b/crates/iceberg/tests/file_io_s3_test.rs index b7c484de8f..b044128327 100644 --- a/crates/iceberg/tests/file_io_s3_test.rs +++ b/crates/iceberg/tests/file_io_s3_test.rs @@ -60,7 +60,7 @@ mod tests { FileIOBuilder::new("s3") .with_props(vec![ - (S3_ENDPOINT, format!("http://{}", minio_socket_addr)), + (S3_ENDPOINT, format!("http://{minio_socket_addr}")), (S3_ACCESS_KEY_ID, "admin".to_string()), (S3_SECRET_ACCESS_KEY, "password".to_string()), (S3_REGION, "us-east-1".to_string()), @@ -208,7 +208,7 @@ mod tests { let file_io_with_custom_creds = FileIOBuilder::new("s3") .with_extension(custom_loader) .with_props(vec![ - (S3_ENDPOINT, format!("http://{}", minio_socket_addr)), + (S3_ENDPOINT, format!("http://{minio_socket_addr}")), (S3_REGION, "us-east-1".to_string()), ]) .build() @@ -237,7 +237,7 @@ mod tests { let file_io_with_custom_creds = FileIOBuilder::new("s3") .with_extension(custom_loader) .with_props(vec![ - (S3_ENDPOINT, format!("http://{}", minio_socket_addr)), + (S3_ENDPOINT, format!("http://{minio_socket_addr}")), (S3_REGION, "us-east-1".to_string()), ]) .build() diff --git a/crates/integration_tests/src/lib.rs b/crates/integration_tests/src/lib.rs index 422cd55863..44f6c30240 100644 --- a/crates/integration_tests/src/lib.rs +++ b/crates/integration_tests/src/lib.rs @@ -46,7 +46,7 @@ pub fn set_test_fixture(func: &str) -> TestFixture { let catalog_config = HashMap::from([ ( REST_CATALOG_PROP_URI.to_string(), - format!("http://{}:{}", rest_catalog_ip, REST_CATALOG_PORT), + format!("http://{rest_catalog_ip}:{REST_CATALOG_PORT}"), ), ( S3_ENDPOINT.to_string(), diff --git a/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs b/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs index 7641810073..318b06bf0e 100644 --- a/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs +++ b/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs @@ -40,7 +40,7 @@ async fn test_read_table_with_positional_deletes() { .unwrap(); let scan = table.scan().build().unwrap(); - println!("{:?}", scan); + println!("{scan:?}"); let plan: Vec<_> = scan .plan_files() @@ -49,7 +49,7 @@ async fn test_read_table_with_positional_deletes() { .try_collect() .await .unwrap(); - println!("{:?}", plan); + println!("{plan:?}"); // Scan plan phase should include delete files in file plan // when with_delete_file_processing_enabled == true diff --git a/crates/integrations/datafusion/src/error.rs b/crates/integrations/datafusion/src/error.rs index 273d92fa62..5bc47013dd 100644 --- a/crates/integrations/datafusion/src/error.rs +++ b/crates/integrations/datafusion/src/error.rs @@ -24,7 +24,7 @@ pub fn from_datafusion_error(error: datafusion::error::DataFusionError) -> Error ErrorKind::Unexpected, "Operation failed for hitting datafusion error".to_string(), ) - .with_source(anyhow!("datafusion error: {:?}", error)) + .with_source(anyhow!("datafusion error: {error:?}")) } /// Converts an iceberg error into a datafusion error. pub fn to_datafusion_error(error: Error) -> datafusion::error::DataFusionError { diff --git a/crates/integrations/datafusion/src/physical_plan/commit.rs b/crates/integrations/datafusion/src/physical_plan/commit.rs index 6bba89fce5..f876908ae6 100644 --- a/crates/integrations/datafusion/src/physical_plan/commit.rs +++ b/crates/integrations/datafusion/src/physical_plan/commit.rs @@ -176,8 +176,7 @@ impl ExecutionPlan for IcebergCommitExec { // IcebergCommitExec only has one partition (partition 0) if partition != 0 { return Err(DataFusionError::Internal(format!( - "IcebergCommitExec only has one partition, but got partition {}", - partition + "IcebergCommitExec only has one partition, but got partition {partition}" ))); } diff --git a/crates/integrations/datafusion/src/physical_plan/scan.rs b/crates/integrations/datafusion/src/physical_plan/scan.rs index d4751a19c8..be92e93d25 100644 --- a/crates/integrations/datafusion/src/physical_plan/scan.rs +++ b/crates/integrations/datafusion/src/physical_plan/scan.rs @@ -167,7 +167,7 @@ impl DisplayAs for IcebergTableScan { .map_or(String::new(), |v| v.join(",")), self.predicates .clone() - .map_or(String::from(""), |p| format!("{}", p)) + .map_or(String::from(""), |p| format!("{p}")) ) } } diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index e46019c541..b9d1f02d1f 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -232,10 +232,7 @@ impl ExecutionPlan for IcebergWriteExec { if file_format != DataFileFormat::Parquet { return Err(to_datafusion_error(Error::new( ErrorKind::FeatureUnsupported, - format!( - "File format {} is not supported for insert_into yet!", - file_format - ), + format!("File format {file_format} is not supported for insert_into yet!"), ))); } @@ -515,7 +512,7 @@ mod tests { .map_err(|e| { Error::new( ErrorKind::Unexpected, - format!("Failed to create record batch: {}", e), + format!("Failed to create record batch: {e}"), ) })?; @@ -532,7 +529,7 @@ mod tests { let stream = write_exec.execute(0, task_ctx).map_err(|e| { Error::new( ErrorKind::Unexpected, - format!("Failed to execute plan: {}", e), + format!("Failed to execute plan: {e}"), ) })?; @@ -541,7 +538,7 @@ mod tests { let mut stream = stream; while let Some(batch) = stream.next().await { results.push(batch.map_err(|e| { - Error::new(ErrorKind::Unexpected, format!("Failed to get batch: {}", e)) + Error::new(ErrorKind::Unexpected, format!("Failed to get batch: {e}")) })?); } diff --git a/crates/integrations/playground/src/main.rs b/crates/integrations/playground/src/main.rs index 6f3de1bc17..c522209957 100644 --- a/crates/integrations/playground/src/main.rs +++ b/crates/integrations/playground/src/main.rs @@ -76,7 +76,7 @@ async fn main_inner() -> anyhow::Result<()> { let args = Args::parse(); if !args.quiet { - println!("ICEBERG PLAYGROUND v{}", ICEBERG_PLAYGROUND_VERSION); + println!("ICEBERG PLAYGROUND v{ICEBERG_PLAYGROUND_VERSION}"); } let session_config = SessionConfig::from_env()?.with_information_schema(true); diff --git a/crates/sqllogictest/src/engine/mod.rs b/crates/sqllogictest/src/engine/mod.rs index f8a405a7cc..724359fbe5 100644 --- a/crates/sqllogictest/src/engine/mod.rs +++ b/crates/sqllogictest/src/engine/mod.rs @@ -39,7 +39,7 @@ pub async fn load_engine_runner( ) -> Result> { match engine_type { TYPE_DATAFUSION => Ok(Box::new(DataFusionEngine::new(cfg).await?)), - _ => Err(anyhow::anyhow!("Unsupported engine type: {}", engine_type).into()), + _ => Err(anyhow::anyhow!("Unsupported engine type: {engine_type}").into()), } } From 833739eacf094fed8e33849ab68df0a0937e16da Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Thu, 30 Oct 2025 22:53:18 +0800 Subject: [PATCH 047/891] refactor: Migrate from tera to minijinja (#1798) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1772 ## What changes are included in this PR? Migrate from tera to minijinja for smaller deps and well-maintain crate. ## Are these changes tested? --- **This PR was primarily authored with Codex using GPT-5-Codex and then hand-reviewed by me. I AM responsible for every change made in this PR. I aimed to keep it aligned with our goals, though I may have missed minor issues. Please flag anything that feels off, I'll fix it quickly.** --------- Signed-off-by: Xuanwo Co-authored-by: Renjie Liu --- Cargo.lock | 336 +++----------------------- Cargo.toml | 2 +- crates/iceberg/Cargo.toml | 2 +- crates/iceberg/src/io/object_cache.rs | 22 +- crates/iceberg/src/scan/mod.rs | 44 ++-- 5 files changed, 75 insertions(+), 331 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index cd18718290..1cacc75fb9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -47,9 +47,9 @@ dependencies = [ [[package]] name = "aho-corasick" -version = "1.1.3" +version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" dependencies = [ "memchr", ] @@ -242,7 +242,7 @@ dependencies = [ "arrow-data", "arrow-schema", "chrono", - "chrono-tz 0.10.4", + "chrono-tz", "half", "hashbrown 0.16.0", "num", @@ -1242,16 +1242,6 @@ dependencies = [ "alloc-stdlib", ] -[[package]] -name = "bstr" -version = "1.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" -dependencies = [ - "memchr", - "serde", -] - [[package]] name = "bumpalo" version = "3.19.0" @@ -1395,17 +1385,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "chrono-tz" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93698b29de5e97ad0ae26447b344c482a7284c737d9ddc5f9e52b74a336671bb" -dependencies = [ - "chrono", - "chrono-tz-build", - "phf 0.11.3", -] - [[package]] name = "chrono-tz" version = "0.10.4" @@ -1413,18 +1392,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" dependencies = [ "chrono", - "phf 0.12.1", -] - -[[package]] -name = "chrono-tz-build" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c088aee841df9c3041febbb73934cfc39708749bf96dc827e3359cd39ef11b1" -dependencies = [ - "parse-zoneinfo", - "phf 0.11.3", - "phf_codegen", + "phf", ] [[package]] @@ -1663,16 +1631,6 @@ dependencies = [ "crossbeam-utils", ] -[[package]] -name = "crossbeam-deque" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" -dependencies = [ - "crossbeam-epoch", - "crossbeam-utils", -] - [[package]] name = "crossbeam-epoch" version = "0.9.18" @@ -2648,12 +2606,6 @@ dependencies = [ "syn 2.0.108", ] -[[package]] -name = "deunicode" -version = "1.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "abd57806937c9cc163efc8ea3910e00a62e2aeb0b8119f1793a978088f8f6b04" - [[package]] name = "diff" version = "0.1.13" @@ -3172,30 +3124,6 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" -[[package]] -name = "globset" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52dfc19153a48bde0cbd630453615c8151bce3a5adfac7a0aebfbf0a1e1f57e3" -dependencies = [ - "aho-corasick", - "bstr", - "log", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "globwalk" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0bf760ebf69878d9fd8f110c89703d90ce35095324d1f1edcb595c63945ee757" -dependencies = [ - "bitflags", - "ignore", - "walkdir", -] - [[package]] name = "gloo-timers" version = "0.3.0" @@ -3428,15 +3356,6 @@ version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" -[[package]] -name = "humansize" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cb51c9a029ddc91b07a787f1d86b53ccfa49b0e86688c946ebe8d3555685dd7" -dependencies = [ - "libm", -] - [[package]] name = "humantime" version = "2.3.0" @@ -3601,6 +3520,7 @@ dependencies = [ "futures", "iceberg_test_utils", "itertools 0.13.0", + "minijinja", "mockall", "moka", "murmur3", @@ -3625,7 +3545,6 @@ dependencies = [ "smol", "strum 0.27.2", "tempfile", - "tera", "thrift", "tokio", "typed-builder", @@ -3849,9 +3768,9 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f578a71f2bfaf7ceb30b519a645ae48024b45f9eecbe060a31a004d7b4ba9462" +checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43" dependencies = [ "displaydoc", "potential_utf", @@ -3862,9 +3781,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c219b62bf5a06801012446193fdfcbd7970e876823aba4c62def2ce957dcb44" +checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6" dependencies = [ "displaydoc", "litemap", @@ -3875,9 +3794,9 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33747cecc725eebb47ac503fab725e395d50cb7889ae490a1359f130611d4cc5" +checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599" dependencies = [ "icu_collections", "icu_normalizer_data", @@ -3889,15 +3808,15 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6ce2d23e1b3c45624ba6a23e2c767e01c9680e0c0800b39c7abfff9565175d8" +checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" [[package]] name = "icu_properties" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d70f9b6574c79f7a83ea5ce72cc88d271a3e77355c5f7748a107e751d8617fb" +checksum = "e93fcd3157766c0c8da2f8cff6ce651a31f0810eaa1c51ec363ef790bbb5fb99" dependencies = [ "icu_collections", "icu_locale_core", @@ -3909,15 +3828,15 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17fa55bf868e28e638ed132bcee1e5c21ba2c1e52c15e7c78b781858e7b54342" +checksum = "02845b3647bb045f1100ecd6480ff52f34c35f82d9880e029d329c21d1054899" [[package]] name = "icu_provider" -version = "2.1.0" +version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f64958e359123591ae1f17a27b5fc9ebdb50c98b04e0401146154de1d8fe3e44" +checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614" dependencies = [ "displaydoc", "icu_locale_core", @@ -3955,22 +3874,6 @@ dependencies = [ "icu_properties", ] -[[package]] -name = "ignore" -version = "0.4.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81776e6f9464432afcc28d03e52eb101c93b6f0566f52aef2427663e700f0403" -dependencies = [ - "crossbeam-deque", - "globset", - "log", - "memchr", - "regex-automata", - "same-file", - "walkdir", - "winapi-util", -] - [[package]] name = "indexmap" version = "1.9.3" @@ -4025,9 +3928,9 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" [[package]] name = "integer-encoding" -version = "4.0.2" +version = "4.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d762194228a2f1c11063e46e32e5acb96e66e906382b9eb5441f2e0504bbd5a" +checksum = "14c00403deb17c3221a1fe4fb571b9ed0370b3dcd116553c77fa294a3d918699" dependencies = [ "async-trait", "tokio", @@ -4402,6 +4305,15 @@ dependencies = [ "libmimalloc-sys", ] +[[package]] +name = "minijinja" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9f264d75233323f4b7d2f03aefe8a990690cdebfbfe26ea86bcbaec5e9ac990" +dependencies = [ + "serde", +] + [[package]] name = "minimal-lexical" version = "0.2.1" @@ -4950,15 +4862,6 @@ dependencies = [ "zstd", ] -[[package]] -name = "parse-zoneinfo" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f2a05b18d44e2957b88f96ba460715e295bc1d7510468a2f3d3b44535d26c24" -dependencies = [ - "regex", -] - [[package]] name = "paste" version = "1.0.15" @@ -5037,49 +4940,6 @@ version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" -[[package]] -name = "pest" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "989e7521a040efde50c3ab6bbadafbe15ab6dc042686926be59ac35d74607df4" -dependencies = [ - "memchr", - "ucd-trie", -] - -[[package]] -name = "pest_derive" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "187da9a3030dbafabbbfb20cb323b976dc7b7ce91fcd84f2f74d6e31d378e2de" -dependencies = [ - "pest", - "pest_generator", -] - -[[package]] -name = "pest_generator" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49b401d98f5757ebe97a26085998d6c0eecec4995cad6ab7fc30ffdf4b052843" -dependencies = [ - "pest", - "pest_meta", - "proc-macro2", - "quote", - "syn 2.0.108", -] - -[[package]] -name = "pest_meta" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f27a2cfee9f9039c4d86faa5af122a0ac3851441a34865b8a043b46be0065a" -dependencies = [ - "pest", - "sha2", -] - [[package]] name = "petgraph" version = "0.7.1" @@ -5102,51 +4962,13 @@ dependencies = [ "serde", ] -[[package]] -name = "phf" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078" -dependencies = [ - "phf_shared 0.11.3", -] - [[package]] name = "phf" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" dependencies = [ - "phf_shared 0.12.1", -] - -[[package]] -name = "phf_codegen" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aef8048c789fa5e851558d709946d6d79a8ff88c0440c587967f8e94bfb1216a" -dependencies = [ - "phf_generator", - "phf_shared 0.11.3", -] - -[[package]] -name = "phf_generator" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d" -dependencies = [ - "phf_shared 0.11.3", - "rand 0.8.5", -] - -[[package]] -name = "phf_shared" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5" -dependencies = [ - "siphasher", + "phf_shared", ] [[package]] @@ -5169,7 +4991,7 @@ dependencies = [ "async-recursion", "bytes", "faststr", - "integer-encoding 4.0.2", + "integer-encoding 4.1.0", "lazy_static", "linkedbytes", "ordered-float 4.6.0", @@ -6081,7 +5903,7 @@ dependencies = [ "once_cell", "ring", "rustls-pki-types", - "rustls-webpki 0.103.7", + "rustls-webpki 0.103.8", "subtle", "zeroize", ] @@ -6150,9 +5972,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.7" +version = "0.103.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e10b3f4191e8a80e6b43eebabfac91e5dcecebb27a71f04e820c47ec41d314bf" +checksum = "2ffdfa2f5286e2247234e03f680868ac2815974dc39e00ea15adc445d0aafe52" dependencies = [ "aws-lc-rs", "ring", @@ -6604,16 +6426,6 @@ version = "0.4.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a2ae44ef20feb57a68b23d846850f861394c2e02dc425a50098ae8c90267589" -[[package]] -name = "slug" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "882a80f72ee45de3cc9a5afeb2da0331d58df69e4e7d8eeb5d3c7784ae67e724" -dependencies = [ - "deunicode", - "wasm-bindgen", -] - [[package]] name = "smallvec" version = "1.15.1" @@ -7141,28 +6953,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "tera" -version = "1.20.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab9d851b45e865f178319da0abdbfe6acbc4328759ff18dafc3a41c16b4cd2ee" -dependencies = [ - "chrono", - "chrono-tz 0.9.0", - "globwalk", - "humansize", - "lazy_static", - "percent-encoding", - "pest", - "pest_derive", - "rand 0.8.5", - "regex", - "serde", - "serde_json", - "slug", - "unic-segment", -] - [[package]] name = "termtree" version = "0.5.1" @@ -7636,62 +7426,6 @@ dependencies = [ "typify-impl", ] -[[package]] -name = "ucd-trie" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" - -[[package]] -name = "unic-char-property" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8c57a407d9b6fa02b4795eb81c5b6652060a15a7903ea981f3d723e6c0be221" -dependencies = [ - "unic-char-range", -] - -[[package]] -name = "unic-char-range" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0398022d5f700414f6b899e10b8348231abf9173fa93144cbc1a43b9793c1fbc" - -[[package]] -name = "unic-common" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "80d7ff825a6a654ee85a63e80f92f054f904f21e7d12da4e22f9834a4aaa35bc" - -[[package]] -name = "unic-segment" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4ed5d26be57f84f176157270c112ef57b86debac9cd21daaabbe56db0f88f23" -dependencies = [ - "unic-ucd-segment", -] - -[[package]] -name = "unic-ucd-segment" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2079c122a62205b421f499da10f3ee0f7697f012f55b675e002483c73ea34700" -dependencies = [ - "unic-char-property", - "unic-char-range", - "unic-ucd-version", -] - -[[package]] -name = "unic-ucd-version" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96bd2f2237fe450fcd0a1d2f5f4e91711124f7857ba2e964247776ebeeb7b0c4" -dependencies = [ - "unic-common", -] - [[package]] name = "unicode-bidi" version = "0.3.18" diff --git a/Cargo.toml b/Cargo.toml index 6f4385cf1c..5161810be3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -91,6 +91,7 @@ linkedbytes = "0.1.8" log = "0.4.28" metainfo = "0.7.14" mimalloc = "0.1.46" +minijinja = "2.12.0" mockall = "0.13.1" mockito = "1" motore-macros = "0.4.3" @@ -120,7 +121,6 @@ sqlx = { version = "0.8.1", default-features = false } stacker = "0.1.20" strum = "0.27.2" tempfile = "3.18" -tera = "1" thrift = "0.17.0" tokio = { version = "1.47", default-features = false } toml = "0.8" diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index b831607aab..c9b1bb3173 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -102,7 +102,7 @@ pretty_assertions = { workspace = true } rand = { workspace = true } regex = { workspace = true } tempfile = { workspace = true } -tera = { workspace = true } +minijinja = { workspace = true } [package.metadata.cargo-machete] # These dependencies are added to ensure minimal dependency version diff --git a/crates/iceberg/src/io/object_cache.rs b/crates/iceberg/src/io/object_cache.rs index cb9f03fa5a..af297bebb5 100644 --- a/crates/iceberg/src/io/object_cache.rs +++ b/crates/iceberg/src/io/object_cache.rs @@ -185,8 +185,9 @@ impl ObjectCache { mod tests { use std::fs; + use minijinja::value::Value; + use minijinja::{AutoEscape, Environment, context}; use tempfile::TempDir; - use tera::{Context, Tera}; use uuid::Uuid; use super::*; @@ -198,6 +199,12 @@ mod tests { }; use crate::table::Table; + fn render_template(template: &str, ctx: Value) -> String { + let mut env = Environment::new(); + env.set_auto_escape_callback(|_| AutoEscape::None); + env.render_str(template, ctx).unwrap() + } + struct TableTestFixture { table_location: String, table: Table, @@ -222,13 +229,12 @@ mod tests { env!("CARGO_MANIFEST_DIR") )) .unwrap(); - let mut context = Context::new(); - context.insert("table_location", &table_location); - context.insert("manifest_list_1_location", &manifest_list1_location); - context.insert("manifest_list_2_location", &manifest_list2_location); - context.insert("table_metadata_1_location", &table_metadata1_location); - - let metadata_json = Tera::one_off(&template_json_str, &context, false).unwrap(); + let metadata_json = render_template(&template_json_str, context! { + table_location => &table_location, + manifest_list_1_location => &manifest_list1_location, + manifest_list_2_location => &manifest_list2_location, + table_metadata_1_location => &table_metadata1_location, + }); serde_json::from_str::(&metadata_json).unwrap() }; diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 92ee2018c0..6884e00b9b 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -563,11 +563,12 @@ pub mod tests { ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray, }; use futures::{TryStreamExt, stream}; + use minijinja::value::Value; + use minijinja::{AutoEscape, Environment, context}; use parquet::arrow::{ArrowWriter, PARQUET_FIELD_ID_META_KEY}; use parquet::basic::Compression; use parquet::file::properties::WriterProperties; use tempfile::TempDir; - use tera::{Context, Tera}; use uuid::Uuid; use crate::TableIdent; @@ -582,6 +583,12 @@ pub mod tests { }; use crate::table::Table; + fn render_template(template: &str, ctx: Value) -> String { + let mut env = Environment::new(); + env.set_auto_escape_callback(|_| AutoEscape::None); + env.render_str(template, ctx).unwrap() + } + pub struct TableTestFixture { pub table_location: String, pub table: Table, @@ -607,13 +614,12 @@ pub mod tests { env!("CARGO_MANIFEST_DIR") )) .unwrap(); - let mut context = Context::new(); - context.insert("table_location", &table_location); - context.insert("manifest_list_1_location", &manifest_list1_location); - context.insert("manifest_list_2_location", &manifest_list2_location); - context.insert("table_metadata_1_location", &table_metadata1_location); - - let metadata_json = Tera::one_off(&template_json_str, &context, false).unwrap(); + let metadata_json = render_template(&template_json_str, context! { + table_location => &table_location, + manifest_list_1_location => &manifest_list1_location, + manifest_list_2_location => &manifest_list2_location, + table_metadata_1_location => &table_metadata1_location, + }); serde_json::from_str::(&metadata_json).unwrap() }; @@ -648,11 +654,10 @@ pub mod tests { env!("CARGO_MANIFEST_DIR") )) .unwrap(); - let mut context = Context::new(); - context.insert("table_location", &table_location); - context.insert("table_metadata_1_location", &table_metadata1_location); - - let metadata_json = Tera::one_off(&template_json_str, &context, false).unwrap(); + let metadata_json = render_template(&template_json_str, context! { + table_location => &table_location, + table_metadata_1_location => &table_metadata1_location, + }); serde_json::from_str::(&metadata_json).unwrap() }; @@ -688,13 +693,12 @@ pub mod tests { env!("CARGO_MANIFEST_DIR") )) .unwrap(); - let mut context = Context::new(); - context.insert("table_location", &table_location); - context.insert("manifest_list_1_location", &manifest_list1_location); - context.insert("manifest_list_2_location", &manifest_list2_location); - context.insert("table_metadata_1_location", &table_metadata1_location); - - let metadata_json = Tera::one_off(&template_json_str, &context, false).unwrap(); + let metadata_json = render_template(&template_json_str, context! { + table_location => &table_location, + manifest_list_1_location => &manifest_list1_location, + manifest_list_2_location => &manifest_list2_location, + table_metadata_1_location => &table_metadata1_location, + }); serde_json::from_str::(&metadata_json).unwrap() }; From 45c82dfa9632228f183aab0d011955c6a356513c Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Fri, 31 Oct 2025 04:39:33 -0400 Subject: [PATCH 048/891] fix(reader): fix position delete bugs with row group skipping (#1806) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? Partially address #1749. ## What changes are included in this PR? This PR fixes two related correctness bugs in `ArrowReader::build_deletes_row_selection()` where position deletes targeting rows in skipped or skipped-to row groups were not being applied correctly. ### Background: How These Bugs Were Discovered While running Apache Spark + Apache Iceberg integration tests through DataFusion Comet, we discovered that the following tests were failing or hanging: - org.apache.iceberg.spark.extensions.TestMergeOnReadMerge - org.apache.iceberg.spark.extensions.TestMergeOnReadDelete - org.apache.iceberg.spark.extensions.TestMergeOnReadUpdate Investigation revealed that recent work to support Iceberg's file splitting feature (via `filter_row_groups_by_byte_range()`) exposed latent bugs in the position delete logic. While the byte range filtering code itself is correct, it exercises code paths that were previously untested, revealing these pre-existing issues. #### Bug 1: Missing base index increment when skipping row groups **The Issue:** When processing a Parquet file with multiple row groups, if a position delete targets a row in a later row group, the function would skip row groups without deletes but fail to increment `current_row_group_base_idx`. This caused the row index tracking to become desynchronized. **Example scenario:** - File with 2 row groups: rows 0-99 (group 0) and rows 100-199 (group 1) - Position delete targets row 199 (last row in group 1) - When processing group 0: delete (199) is beyond the group's range, so code hits `continue` at lines 469-471 - BUG: `current_row_group_base_idx` is NOT incremented, stays at 0 - When processing group 1: code thinks rows start at 0 instead of 100 - Delete at position 199 is never applied (thinks file only has rows 0-99) **The Fix:** Add `current_row_group_base_idx += row_group_num_rows` before the two `continue` statements at lines ~470 and ~481. This ensures row index tracking stays synchronized when skipping row groups. #### Bug 2: Stale cached delete index when skipping unselected row groups **The Issue:** When row group selection is active (e.g., via byte range filtering for file splits) and an unselected row group is skipped, the cached `next_deleted_row_idx_opt` variable can become stale, leading to either lost deletes or infinite loops depending on the scenario. The function maintains a cached value (`next_deleted_row_idx_opt`) containing the next delete to apply. When skipping unselected row groups, it calls `delete_vector_iter.advance_to(next_row_group_base_idx)` to position the iterator, but this doesn't automatically update the cached variable. **Two problematic scenarios:** 1. Stale cache causes infinite loop (the bug we hit): - File with 2 row groups: rows 0-99 (group 0) and rows 100-199 (group 1) - Position delete at row 0 (in group 0) - Row group selection: read ONLY group 1 - Initial state: `next_deleted_row_idx_opt = Some(0)` (cached) - Skip group 0: `advance_to(100)` positions iterator past delete at 0 - BUG: cached value still `Some(0)` - STALE! - Process group 1: loop condition `0 < 200` is `true`, but `current_idx (100) != next_deleted_row_idx (0)`, so neither branch executes could result in infinite loop 2. Unconditionally calling `next()` loses deletes: - File with 2 row groups: rows 0-99 (group 0) and rows 100-199 (group 1) - Position delete at row 199 (in group 1) - Row group selection: read ONLY group 1 - Initial state: `next_deleted_row_idx_opt = Some(199)` (cached, already correct!) - Skip group 0: `advance_to(100)` - iterator already positioned correctly - If we call `next()`: BUG - consumes delete at 199, advancing past it - Process group 1: iterator exhausted, delete is lost **The Fix:** - If `cached value < next_row_group_base_idx` (stale), update it, thus avoiding infinite loop - If `cached value >= next_row_group_base_idx` (still valid), keep it, thus preserving delete ## Are these changes tested? Yes. This PR adds two comprehensive unit tests in reader.rs: 1. `test_position_delete_across_multiple_row_groups` - Tests bug 1 (missing base index increment) 2. `test_position_delete_with_row_group_selection` - Tests bug 2 scenario where delete is in selected group 3. `test_position_delete_in_skipped_row_group` - Tests bug 2 scenario where delete is in skipped group (would hang without fix) Additionally, these fixes resolve failures in Iceberg Java's spark-extension tests when running with DataFusion Comet’s PR https://github.com/apache/datafusion-comet/pull/2528: - org.apache.iceberg.spark.extensions.TestMergeOnReadMerge - org.apache.iceberg.spark.extensions.TestMergeOnReadDelete - org.apache.iceberg.spark.extensions.TestMergeOnReadUpdate --- crates/iceberg/src/arrow/reader.rs | 617 ++++++++++++++++++++++++++++- 1 file changed, 612 insertions(+), 5 deletions(-) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 3c18c0337b..e0894ad6b4 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -406,10 +406,17 @@ impl ArrowReader { // selected row group selected_row_groups_idx += 1; } else { - // remove any positional deletes from the skipped page so that - // `positional.deletes.min()` can be used + // Advance iterator past all deletes in the skipped row group. + // advance_to() positions the iterator to the first delete >= next_row_group_base_idx. + // However, if our cached next_deleted_row_idx_opt is in the skipped range, + // we need to call next() to update the cache with the newly positioned value. delete_vector_iter.advance_to(next_row_group_base_idx); - next_deleted_row_idx_opt = delete_vector_iter.next(); + // Only update the cache if the cached value is stale (in the skipped range) + if let Some(cached_idx) = next_deleted_row_idx_opt { + if cached_idx < next_row_group_base_idx { + next_deleted_row_idx_opt = delete_vector_iter.next(); + } + } // still increment the current page base index but then skip to the next row group // in the file @@ -424,6 +431,7 @@ impl ArrowReader { // the remainder of this row group and skip to the next row group if next_deleted_row_idx >= next_row_group_base_idx { results.push(RowSelector::select(row_group_num_rows as usize)); + current_row_group_base_idx += row_group_num_rows; continue; } @@ -433,6 +441,7 @@ impl ArrowReader { // If there are no more pos deletes, add a selector for the entirety of this row group. _ => { results.push(RowSelector::select(row_group_num_rows as usize)); + current_row_group_base_idx += row_group_num_rows; continue; } }; @@ -1496,8 +1505,10 @@ mod tests { use crate::expr::visitors::bound_predicate_visitor::visit; use crate::expr::{Bind, Predicate, Reference}; use crate::io::FileIO; - use crate::scan::{FileScanTask, FileScanTaskStream}; - use crate::spec::{DataFileFormat, Datum, NestedField, PrimitiveType, Schema, SchemaRef, Type}; + use crate::scan::{FileScanTask, FileScanTaskDeleteFile, FileScanTaskStream}; + use crate::spec::{ + DataContentType, DataFileFormat, Datum, NestedField, PrimitiveType, Schema, SchemaRef, Type, + }; fn table_schema_simple() -> SchemaRef { Arc::new( @@ -2280,4 +2291,600 @@ message schema { assert!(col_b.is_null(1)); assert!(col_b.is_null(2)); } + + /// Test for bug where position deletes in later row groups are not applied correctly. + /// + /// When a file has multiple row groups and a position delete targets a row in a later + /// row group, the `build_deletes_row_selection` function had a bug where it would + /// fail to increment `current_row_group_base_idx` when skipping row groups. + /// + /// This test creates: + /// - A data file with 200 rows split into 2 row groups (0-99, 100-199) + /// - A position delete file that deletes row 199 (last row in second row group) + /// + /// Expected behavior: Should return 199 rows (with id=200 deleted) + /// Bug behavior: Returns 200 rows (delete is not applied) + /// + /// This bug was discovered while running Apache Spark + Apache Iceberg integration tests + /// through DataFusion Comet. The following Iceberg Java tests failed due to this bug: + /// - `org.apache.iceberg.spark.extensions.TestMergeOnReadDelete::testDeleteWithMultipleRowGroupsParquet` + /// - `org.apache.iceberg.spark.extensions.TestMergeOnReadUpdate::testUpdateWithMultipleRowGroupsParquet` + #[tokio::test] + async fn test_position_delete_across_multiple_row_groups() { + use arrow_array::{Int32Array, Int64Array}; + use parquet::file::reader::{FileReader, SerializedFileReader}; + + // Field IDs for positional delete schema + const FIELD_ID_POSITIONAL_DELETE_FILE_PATH: u64 = 2147483546; + const FIELD_ID_POSITIONAL_DELETE_POS: u64 = 2147483545; + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + + // Create table schema with a single 'id' column + let table_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + ])); + + // Step 1: Create data file with 200 rows in 2 row groups + // Row group 0: rows 0-99 (ids 1-100) + // Row group 1: rows 100-199 (ids 101-200) + let data_file_path = format!("{}/data.parquet", &table_location); + + let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(1..=100), + )]) + .unwrap(); + + let batch2 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(101..=200), + )]) + .unwrap(); + + // Force each batch into its own row group + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .set_max_row_group_size(100) + .build(); + + let file = File::create(&data_file_path).unwrap(); + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + writer.write(&batch1).expect("Writing batch 1"); + writer.write(&batch2).expect("Writing batch 2"); + writer.close().unwrap(); + + // Verify we created 2 row groups + let verify_file = File::open(&data_file_path).unwrap(); + let verify_reader = SerializedFileReader::new(verify_file).unwrap(); + assert_eq!( + verify_reader.metadata().num_row_groups(), + 2, + "Should have 2 row groups" + ); + + // Step 2: Create position delete file that deletes row 199 (id=200, last row in row group 1) + let delete_file_path = format!("{}/deletes.parquet", &table_location); + + let delete_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_FILE_PATH.to_string(), + )])), + Field::new("pos", DataType::Int64, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_POS.to_string(), + )])), + ])); + + // Delete row at position 199 (0-indexed, so it's the last row: id=200) + let delete_batch = RecordBatch::try_new(delete_schema.clone(), vec![ + Arc::new(StringArray::from_iter_values(vec![data_file_path.clone()])), + Arc::new(Int64Array::from_iter_values(vec![199i64])), + ]) + .unwrap(); + + let delete_props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let delete_file = File::create(&delete_file_path).unwrap(); + let mut delete_writer = + ArrowWriter::try_new(delete_file, delete_schema, Some(delete_props)).unwrap(); + delete_writer.write(&delete_batch).unwrap(); + delete_writer.close().unwrap(); + + // Step 3: Read the data file with the delete applied + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + let reader = ArrowReaderBuilder::new(file_io).build(); + + let task = FileScanTask { + start: 0, + length: 0, + record_count: Some(200), + data_file_path: data_file_path.clone(), + data_file_format: DataFileFormat::Parquet, + schema: table_schema.clone(), + project_field_ids: vec![1], + predicate: None, + deletes: vec![FileScanTaskDeleteFile { + file_path: delete_file_path, + file_type: DataContentType::PositionDeletes, + partition_spec_id: 0, + equality_ids: None, + }], + }; + + let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Step 4: Verify we got 199 rows (not 200) + let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); + + println!("Total rows read: {}", total_rows); + println!("Expected: 199 rows (deleted row 199 which had id=200)"); + + // This assertion will FAIL before the fix and PASS after the fix + assert_eq!( + total_rows, 199, + "Expected 199 rows after deleting row 199, but got {} rows. \ + The bug causes position deletes in later row groups to be ignored.", + total_rows + ); + + // Verify the deleted row (id=200) is not present + let all_ids: Vec = result + .iter() + .flat_map(|batch| { + batch + .column(0) + .as_primitive::() + .values() + .iter() + .copied() + }) + .collect(); + + assert!( + !all_ids.contains(&200), + "Row with id=200 should be deleted but was found in results" + ); + + // Verify we have all other ids (1-199) + let expected_ids: Vec = (1..=199).collect(); + assert_eq!( + all_ids, expected_ids, + "Should have ids 1-199 but got different values" + ); + } + + /// Test for bug where position deletes are lost when skipping unselected row groups. + /// + /// This is a variant of `test_position_delete_across_multiple_row_groups` that exercises + /// the row group selection code path (`selected_row_groups: Some([...])`). + /// + /// When a file has multiple row groups and only some are selected for reading, + /// the `build_deletes_row_selection` function must correctly skip over deletes in + /// unselected row groups WITHOUT consuming deletes that belong to selected row groups. + /// + /// This test creates: + /// - A data file with 200 rows split into 2 row groups (0-99, 100-199) + /// - A position delete file that deletes row 199 (last row in second row group) + /// - Row group selection that reads ONLY row group 1 (rows 100-199) + /// + /// Expected behavior: Should return 99 rows (with row 199 deleted) + /// Bug behavior: Returns 100 rows (delete is lost when skipping row group 0) + /// + /// The bug occurs when processing row group 0 (unselected): + /// ```rust + /// delete_vector_iter.advance_to(next_row_group_base_idx); // Position at first delete >= 100 + /// next_deleted_row_idx_opt = delete_vector_iter.next(); // BUG: Consumes delete at 199! + /// ``` + /// + /// The fix is to NOT call `next()` after `advance_to()` when skipping unselected row groups, + /// because `advance_to()` already positions the iterator correctly without consuming elements. + #[tokio::test] + async fn test_position_delete_with_row_group_selection() { + use arrow_array::{Int32Array, Int64Array}; + use parquet::file::reader::{FileReader, SerializedFileReader}; + + // Field IDs for positional delete schema + const FIELD_ID_POSITIONAL_DELETE_FILE_PATH: u64 = 2147483546; + const FIELD_ID_POSITIONAL_DELETE_POS: u64 = 2147483545; + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + + // Create table schema with a single 'id' column + let table_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + ])); + + // Step 1: Create data file with 200 rows in 2 row groups + // Row group 0: rows 0-99 (ids 1-100) + // Row group 1: rows 100-199 (ids 101-200) + let data_file_path = format!("{}/data.parquet", &table_location); + + let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(1..=100), + )]) + .unwrap(); + + let batch2 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(101..=200), + )]) + .unwrap(); + + // Force each batch into its own row group + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .set_max_row_group_size(100) + .build(); + + let file = File::create(&data_file_path).unwrap(); + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + writer.write(&batch1).expect("Writing batch 1"); + writer.write(&batch2).expect("Writing batch 2"); + writer.close().unwrap(); + + // Verify we created 2 row groups + let verify_file = File::open(&data_file_path).unwrap(); + let verify_reader = SerializedFileReader::new(verify_file).unwrap(); + assert_eq!( + verify_reader.metadata().num_row_groups(), + 2, + "Should have 2 row groups" + ); + + // Step 2: Create position delete file that deletes row 199 (id=200, last row in row group 1) + let delete_file_path = format!("{}/deletes.parquet", &table_location); + + let delete_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_FILE_PATH.to_string(), + )])), + Field::new("pos", DataType::Int64, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_POS.to_string(), + )])), + ])); + + // Delete row at position 199 (0-indexed, so it's the last row: id=200) + let delete_batch = RecordBatch::try_new(delete_schema.clone(), vec![ + Arc::new(StringArray::from_iter_values(vec![data_file_path.clone()])), + Arc::new(Int64Array::from_iter_values(vec![199i64])), + ]) + .unwrap(); + + let delete_props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let delete_file = File::create(&delete_file_path).unwrap(); + let mut delete_writer = + ArrowWriter::try_new(delete_file, delete_schema, Some(delete_props)).unwrap(); + delete_writer.write(&delete_batch).unwrap(); + delete_writer.close().unwrap(); + + // Step 3: Get byte ranges to read ONLY row group 1 (rows 100-199) + // This exercises the row group selection code path where row group 0 is skipped + let metadata_file = File::open(&data_file_path).unwrap(); + let metadata_reader = SerializedFileReader::new(metadata_file).unwrap(); + let metadata = metadata_reader.metadata(); + + let row_group_0 = metadata.row_group(0); + let row_group_1 = metadata.row_group(1); + + let rg0_start = 4u64; // Parquet files start with 4-byte magic "PAR1" + let rg1_start = rg0_start + row_group_0.compressed_size() as u64; + let rg1_length = row_group_1.compressed_size() as u64; + + println!( + "Row group 0: starts at byte {}, {} bytes compressed", + rg0_start, + row_group_0.compressed_size() + ); + println!( + "Row group 1: starts at byte {}, {} bytes compressed", + rg1_start, + row_group_1.compressed_size() + ); + + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + let reader = ArrowReaderBuilder::new(file_io).build(); + + // Create FileScanTask that reads ONLY row group 1 via byte range filtering + let task = FileScanTask { + start: rg1_start, + length: rg1_length, + record_count: Some(100), // Row group 1 has 100 rows + data_file_path: data_file_path.clone(), + data_file_format: DataFileFormat::Parquet, + schema: table_schema.clone(), + project_field_ids: vec![1], + predicate: None, + deletes: vec![FileScanTaskDeleteFile { + file_path: delete_file_path, + file_type: DataContentType::PositionDeletes, + partition_spec_id: 0, + equality_ids: None, + }], + }; + + let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Step 4: Verify we got 99 rows (not 100) + // Row group 1 has 100 rows (ids 101-200), minus 1 delete (id=200) = 99 rows + let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); + + println!("Total rows read from row group 1: {}", total_rows); + println!("Expected: 99 rows (row group 1 has 100 rows, 1 delete at position 199)"); + + // This assertion will FAIL before the fix and PASS after the fix + assert_eq!( + total_rows, 99, + "Expected 99 rows from row group 1 after deleting position 199, but got {} rows. \ + The bug causes position deletes to be lost when advance_to() is followed by next() \ + when skipping unselected row groups.", + total_rows + ); + + // Verify the deleted row (id=200) is not present + let all_ids: Vec = result + .iter() + .flat_map(|batch| { + batch + .column(0) + .as_primitive::() + .values() + .iter() + .copied() + }) + .collect(); + + assert!( + !all_ids.contains(&200), + "Row with id=200 should be deleted but was found in results" + ); + + // Verify we have ids 101-199 (not 101-200) + let expected_ids: Vec = (101..=199).collect(); + assert_eq!( + all_ids, expected_ids, + "Should have ids 101-199 but got different values" + ); + } + /// Test for bug where stale cached delete causes infinite loop when skipping row groups. + /// + /// This test exposes the inverse scenario of `test_position_delete_with_row_group_selection`: + /// - Position delete targets a row in the SKIPPED row group (not the selected one) + /// - After calling advance_to(), the cached delete index is stale + /// - Without updating the cache, the code enters an infinite loop + /// + /// This test creates: + /// - A data file with 200 rows split into 2 row groups (0-99, 100-199) + /// - A position delete file that deletes row 0 (first row in SKIPPED row group 0) + /// - Row group selection that reads ONLY row group 1 (rows 100-199) + /// + /// The bug occurs when skipping row group 0: + /// ```rust + /// let mut next_deleted_row_idx_opt = delete_vector_iter.next(); // Some(0) + /// // ... skip to row group 1 ... + /// delete_vector_iter.advance_to(100); // Iterator advances past delete at 0 + /// // BUG: next_deleted_row_idx_opt is still Some(0) - STALE! + /// // When processing row group 1: + /// // current_idx = 100, next_deleted_row_idx = 0, next_row_group_base_idx = 200 + /// // Loop condition: 0 < 200 (true) + /// // But: current_idx (100) > next_deleted_row_idx (0) + /// // And: current_idx (100) != next_deleted_row_idx (0) + /// // Neither branch executes -> INFINITE LOOP! + /// ``` + /// + /// Expected behavior: Should return 100 rows (delete at 0 doesn't affect row group 1) + /// Bug behavior: Infinite loop in build_deletes_row_selection + #[tokio::test] + async fn test_position_delete_in_skipped_row_group() { + use arrow_array::{Int32Array, Int64Array}; + use parquet::file::reader::{FileReader, SerializedFileReader}; + + // Field IDs for positional delete schema + const FIELD_ID_POSITIONAL_DELETE_FILE_PATH: u64 = 2147483546; + const FIELD_ID_POSITIONAL_DELETE_POS: u64 = 2147483545; + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + + // Create table schema with a single 'id' column + let table_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + ])); + + // Step 1: Create data file with 200 rows in 2 row groups + // Row group 0: rows 0-99 (ids 1-100) + // Row group 1: rows 100-199 (ids 101-200) + let data_file_path = format!("{}/data.parquet", &table_location); + + let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(1..=100), + )]) + .unwrap(); + + let batch2 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( + Int32Array::from_iter_values(101..=200), + )]) + .unwrap(); + + // Force each batch into its own row group + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .set_max_row_group_size(100) + .build(); + + let file = File::create(&data_file_path).unwrap(); + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + writer.write(&batch1).expect("Writing batch 1"); + writer.write(&batch2).expect("Writing batch 2"); + writer.close().unwrap(); + + // Verify we created 2 row groups + let verify_file = File::open(&data_file_path).unwrap(); + let verify_reader = SerializedFileReader::new(verify_file).unwrap(); + assert_eq!( + verify_reader.metadata().num_row_groups(), + 2, + "Should have 2 row groups" + ); + + // Step 2: Create position delete file that deletes row 0 (id=1, first row in row group 0) + let delete_file_path = format!("{}/deletes.parquet", &table_location); + + let delete_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_FILE_PATH.to_string(), + )])), + Field::new("pos", DataType::Int64, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + FIELD_ID_POSITIONAL_DELETE_POS.to_string(), + )])), + ])); + + // Delete row at position 0 (0-indexed, so it's the first row: id=1) + let delete_batch = RecordBatch::try_new(delete_schema.clone(), vec![ + Arc::new(StringArray::from_iter_values(vec![data_file_path.clone()])), + Arc::new(Int64Array::from_iter_values(vec![0i64])), + ]) + .unwrap(); + + let delete_props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let delete_file = File::create(&delete_file_path).unwrap(); + let mut delete_writer = + ArrowWriter::try_new(delete_file, delete_schema, Some(delete_props)).unwrap(); + delete_writer.write(&delete_batch).unwrap(); + delete_writer.close().unwrap(); + + // Step 3: Get byte ranges to read ONLY row group 1 (rows 100-199) + // This exercises the row group selection code path where row group 0 is skipped + let metadata_file = File::open(&data_file_path).unwrap(); + let metadata_reader = SerializedFileReader::new(metadata_file).unwrap(); + let metadata = metadata_reader.metadata(); + + let row_group_0 = metadata.row_group(0); + let row_group_1 = metadata.row_group(1); + + let rg0_start = 4u64; // Parquet files start with 4-byte magic "PAR1" + let rg1_start = rg0_start + row_group_0.compressed_size() as u64; + let rg1_length = row_group_1.compressed_size() as u64; + + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + let reader = ArrowReaderBuilder::new(file_io).build(); + + // Create FileScanTask that reads ONLY row group 1 via byte range filtering + let task = FileScanTask { + start: rg1_start, + length: rg1_length, + record_count: Some(100), // Row group 1 has 100 rows + data_file_path: data_file_path.clone(), + data_file_format: DataFileFormat::Parquet, + schema: table_schema.clone(), + project_field_ids: vec![1], + predicate: None, + deletes: vec![FileScanTaskDeleteFile { + file_path: delete_file_path, + file_type: DataContentType::PositionDeletes, + partition_spec_id: 0, + equality_ids: None, + }], + }; + + let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Step 4: Verify we got 100 rows (all of row group 1) + // The delete at position 0 is in row group 0, which is skipped, so it doesn't affect us + let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); + + assert_eq!( + total_rows, 100, + "Expected 100 rows from row group 1 (delete at position 0 is in skipped row group 0). \ + If this hangs or fails, it indicates the cached delete index was not updated after advance_to()." + ); + + // Verify we have all ids from row group 1 (101-200) + let all_ids: Vec = result + .iter() + .flat_map(|batch| { + batch + .column(0) + .as_primitive::() + .values() + .iter() + .copied() + }) + .collect(); + + let expected_ids: Vec = (101..=200).collect(); + assert_eq!( + all_ids, expected_ids, + "Should have ids 101-200 (all of row group 1)" + ); + } } From 7ea6713c738db2b9738778a6bed92b97d3a55257 Mon Sep 17 00:00:00 2001 From: Florian Valeye Date: Fri, 31 Oct 2025 10:33:17 +0100 Subject: [PATCH 049/891] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning (#1620) ## Which issue does this PR close? - Closes #1543 ## What changes are included in this PR? Implement a physical execution repartition node that determines the relevant DataFusion partitioning strategy based on the Iceberg table schema and metadata. 1. Unpartitioned tables: Uses round-robin partitioning 2. Partitioned tables: It depends on the transform type: - Identity or Bucket transforms: Uses hash partitioning on the _partition column - Temporal transforms (Year, Month, Day, Hour): Uses round-robin partitioning _Minor change: I created a new `schema_ref()` helper method._ ## Are these changes tested? Yes, with unit tests --------- Signed-off-by: Florian Valeye --- crates/iceberg/src/table.rs | 7 +- .../datafusion/src/physical_plan/mod.rs | 1 + .../src/physical_plan/repartition.rs | 886 ++++++++++++++++++ 3 files changed, 893 insertions(+), 1 deletion(-) create mode 100644 crates/integrations/datafusion/src/physical_plan/repartition.rs diff --git a/crates/iceberg/src/table.rs b/crates/iceberg/src/table.rs index d4e696ce84..80e10b2fe2 100644 --- a/crates/iceberg/src/table.rs +++ b/crates/iceberg/src/table.rs @@ -24,7 +24,7 @@ use crate::inspect::MetadataTable; use crate::io::FileIO; use crate::io::object_cache::ObjectCache; use crate::scan::TableScanBuilder; -use crate::spec::{TableMetadata, TableMetadataRef}; +use crate::spec::{SchemaRef, TableMetadata, TableMetadataRef}; use crate::{Error, ErrorKind, Result, TableIdent}; /// Builder to create table scan. @@ -235,6 +235,11 @@ impl Table { self.readonly } + /// Returns the current schema as a shared reference. + pub fn current_schema_ref(&self) -> SchemaRef { + self.metadata.current_schema().clone() + } + /// Create a reader for the table. pub fn reader_builder(&self) -> ArrowReaderBuilder { ArrowReaderBuilder::new(self.file_io.clone()) diff --git a/crates/integrations/datafusion/src/physical_plan/mod.rs b/crates/integrations/datafusion/src/physical_plan/mod.rs index ce923b8662..eb58082fe5 100644 --- a/crates/integrations/datafusion/src/physical_plan/mod.rs +++ b/crates/integrations/datafusion/src/physical_plan/mod.rs @@ -19,6 +19,7 @@ pub(crate) mod commit; pub(crate) mod expr_to_predicate; pub(crate) mod metadata_scan; pub(crate) mod project; +pub(crate) mod repartition; pub(crate) mod scan; pub(crate) mod write; diff --git a/crates/integrations/datafusion/src/physical_plan/repartition.rs b/crates/integrations/datafusion/src/physical_plan/repartition.rs new file mode 100644 index 0000000000..95cdc8472e --- /dev/null +++ b/crates/integrations/datafusion/src/physical_plan/repartition.rs @@ -0,0 +1,886 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::num::NonZeroUsize; +use std::sync::Arc; + +use datafusion::error::{DataFusionError, Result as DFResult}; +use datafusion::physical_expr::PhysicalExpr; +use datafusion::physical_plan::expressions::Column; +use datafusion::physical_plan::repartition::RepartitionExec; +use datafusion::physical_plan::{ExecutionPlan, Partitioning}; +use iceberg::arrow::PROJECTED_PARTITION_VALUE_COLUMN; +use iceberg::spec::{TableMetadata, TableMetadataRef, Transform}; +/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution +/// for parallel processing while respecting Iceberg table partitioning semantics. +/// +/// Automatically determines the optimal partitioning strategy based on the table's +/// partition specification. +/// +/// ## Partitioning Strategies +/// +/// - **Partitioned tables with Identity/Bucket transforms** – Uses hash partitioning on the +/// `_partition` column for optimal data distribution and file clustering. Ensures that rows +/// with the same partition values are co-located in the same task. +/// +/// - **Partitioned tables with temporal transforms** – Uses round-robin partitioning for +/// temporal transforms (Year, Month, Day, Hour) that don't provide uniform hash distribution. +/// +/// - **Unpartitioned tables** – Uses round-robin distribution to balance load evenly across workers. +/// +/// ## Requirements +/// +/// - **For partitioned tables**: The input MUST include the `_partition` column. +/// Add it by calling [`project_with_partition`](crate::physical_plan::project_with_partition) before [`repartition`]. +/// - **For unpartitioned tables**: No special preparation needed. +/// - Returns an error if a partitioned table is missing the `_partition` column. +/// +/// ## Performance Notes +/// +/// - Only adds repartitioning when the input partitioning differs from the target. +/// - Requires an explicit target partition count for deterministic behavior. +/// +/// # Arguments +/// +/// * `input` - The input [`ExecutionPlan`]. For partitioned tables, must include the `_partition` +/// column (added via [`project_with_partition`](crate::physical_plan::project_with_partition)). +/// * `table_metadata` - Iceberg table metadata containing partition spec. +/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0). +/// +/// # Returns +/// +/// An [`ExecutionPlan`] that applies the optimal partitioning strategy, or the original input plan +/// if repartitioning is not needed. +/// +/// # Errors +/// +/// Returns [`DataFusionError::Plan`] if a partitioned table input is missing the `_partition` column. +/// +/// # Examples +/// +/// For partitioned tables, first add the `_partition` column: +/// +/// ```ignore +/// use std::num::NonZeroUsize; +/// use iceberg_datafusion::physical_plan::project_with_partition; +/// +/// let plan_with_partition = project_with_partition(input_plan, &table)?; +/// +/// let repartitioned_plan = repartition( +/// plan_with_partition, +/// table.metadata_ref(), +/// NonZeroUsize::new(4).unwrap(), +/// )?; +/// ``` +#[allow(dead_code)] +pub(crate) fn repartition( + input: Arc, + table_metadata: TableMetadataRef, + target_partitions: NonZeroUsize, +) -> DFResult> { + let partitioning_strategy = + determine_partitioning_strategy(&input, &table_metadata, target_partitions)?; + + Ok(Arc::new(RepartitionExec::try_new( + input, + partitioning_strategy, + )?)) +} + +/// Determine the optimal partitioning strategy based on table metadata. +/// +/// Analyzes the table's partition specification to select the most appropriate +/// DataFusion partitioning strategy for insert operations. +/// +/// ## Partitioning Strategy +/// +/// - **Partitioned tables**: Must have the `_partition` column in the input schema (added via +/// `project_with_partition`). Uses hash partitioning if the partition spec contains Identity +/// or Bucket transforms for good data distribution. Falls back to round-robin for temporal +/// transforms (Year, Month, Day, Hour) that don't provide uniform hash distribution. +/// +/// - **Unpartitioned tables**: Always uses round-robin batch partitioning to ensure even load +/// distribution across workers. +/// +/// ## Requirements +/// +/// - **For partitioned tables**: The input MUST include the `_partition` column +/// (added via `project_with_partition()`). +/// - **For unpartitioned tables**: No special preparation needed. +/// - Returns an error if a partitioned table is missing the `_partition` column. +fn determine_partitioning_strategy( + input: &Arc, + table_metadata: &TableMetadata, + target_partitions: NonZeroUsize, +) -> DFResult { + let partition_spec = table_metadata.default_partition_spec(); + let input_schema = input.schema(); + let target_partition_count = target_partitions.get(); + + // Check if partition spec has transforms suitable for hash partitioning + let has_hash_friendly_transforms = partition_spec + .fields() + .iter() + .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_))); + + let partition_col_result = input_schema.index_of(PROJECTED_PARTITION_VALUE_COLUMN); + let is_partitioned_table = !partition_spec.is_unpartitioned(); + + match (is_partitioned_table, partition_col_result) { + // Case 1: Partitioned table with _partition column present + (true, Ok(partition_col_idx)) => { + let partition_expr = Arc::new(Column::new( + PROJECTED_PARTITION_VALUE_COLUMN, + partition_col_idx, + )) as Arc; + + if has_hash_friendly_transforms { + Ok(Partitioning::Hash( + vec![partition_expr], + target_partition_count, + )) + } else { + Ok(Partitioning::RoundRobinBatch(target_partition_count)) + } + } + + // Case 2: Partitioned table missing _partition column (normally this should not happen) + (true, Err(_)) => Err(DataFusionError::Plan(format!( + "Partitioned table input missing {} column. \ + Ensure projection happens before repartitioning.", + PROJECTED_PARTITION_VALUE_COLUMN + ))), + + // Case 3: Unpartitioned table, always use RoundRobinBatch + (false, _) => Ok(Partitioning::RoundRobinBatch(target_partition_count)), + } +} + +#[cfg(test)] +mod tests { + use datafusion::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Fields, Schema as ArrowSchema, TimeUnit, + }; + use datafusion::execution::TaskContext; + use datafusion::physical_plan::empty::EmptyExec; + use iceberg::TableIdent; + use iceberg::io::FileIO; + use iceberg::spec::{ + NestedField, NullOrder, PrimitiveType, Schema, SortDirection, SortField, SortOrder, + Transform, Type, + }; + use iceberg::table::Table; + + use super::*; + + fn create_test_table() -> Table { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::required( + 2, + "data", + Type::Primitive(PrimitiveType::String), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .build() + .unwrap(); + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/table".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + + Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/metadata.json".to_string()) + .build() + .unwrap() + } + + fn create_test_arrow_schema() -> Arc { + Arc::new(ArrowSchema::new(vec![ + ArrowField::new("id", ArrowDataType::Int64, false), + ArrowField::new("data", ArrowDataType::Utf8, false), + ])) + } + + #[tokio::test] + async fn test_repartition_unpartitioned_table() { + let table = create_test_table(); + let input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + + let repartitioned_plan = repartition( + input.clone(), + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + assert_ne!(input.name(), repartitioned_plan.name()); + assert_eq!(repartitioned_plan.name(), "RepartitionExec"); + } + + #[tokio::test] + async fn test_repartition_explicit_partitions() { + let table = create_test_table(); + let input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(8).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + match partitioning { + Partitioning::RoundRobinBatch(n) => { + assert_eq!(*n, 8); + } + _ => panic!("Expected RoundRobinBatch partitioning"), + } + } + + #[tokio::test] + async fn test_repartition_zero_partitions_fails() { + let _table = create_test_table(); + let _input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + + let result = std::num::NonZeroUsize::new(0); + assert!(result.is_none(), "NonZeroUsize::new(0) should return None"); + + // Test that we can't call repartition with 0 partitions + // This is prevented at compile time by NonZeroUsize + let _ = result; // This would be None, so we can't call repartition + } + + #[tokio::test] + async fn test_partition_count_validation() { + let table = create_test_table(); + let input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + + let target_partitions = 16; + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(target_partitions).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + match partitioning { + Partitioning::RoundRobinBatch(n) => { + assert_eq!(*n, target_partitions); + } + _ => panic!("Expected RoundRobinBatch partitioning"), + } + } + + #[tokio::test] + async fn test_datafusion_repartitioning_integration() { + let table = create_test_table(); + let input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(3).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + match partitioning { + Partitioning::RoundRobinBatch(n) => { + assert_eq!(*n, 3, "Should use round-robin for unpartitioned table"); + } + _ => panic!("Expected RoundRobinBatch partitioning for unpartitioned table"), + } + + let task_ctx = Arc::new(TaskContext::default()); + let stream = repartitioned_plan.execute(0, task_ctx.clone()).unwrap(); + + assert!(!stream.schema().fields().is_empty()); + } + + #[tokio::test] + async fn test_bucket_aware_partitioning() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::required( + 2, + "category", + Type::Primitive(PrimitiveType::String), + )), + ]) + .build() + .unwrap(); + + let sort_order = SortOrder::builder() + .with_order_id(1) + .with_sort_field(SortField { + source_id: 2, + transform: Transform::Bucket(4), + direction: SortDirection::Ascending, + null_order: NullOrder::First, + }) + .build(&schema) + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .build() + .unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/bucketed_table".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "bucketed_table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/bucketed_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("id", ArrowDataType::Int64, false), + ArrowField::new("category", ArrowDataType::Utf8, false), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + // For bucketed tables without _partition column, should use round-robin + // since the new logic prioritizes _partition column when available + match partitioning { + Partitioning::Hash(_, _) => { + // This would happen if _partition column is present + } + Partitioning::RoundRobinBatch(_) => { + // This happens when _partition column is not present + } + _ => panic!("Unexpected partitioning strategy"), + } + } + + #[tokio::test] + async fn test_combined_partition_and_bucket_strategy() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "date", + Type::Primitive(PrimitiveType::Date), + )), + Arc::new(NestedField::required( + 2, + "user_id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::required( + 3, + "amount", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .add_partition_field("date", "date", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = SortOrder::builder() + .with_order_id(1) + .with_sort_field(SortField { + source_id: 2, + transform: Transform::Bucket(8), + direction: SortDirection::Ascending, + null_order: NullOrder::First, + }) + .build(&schema) + .unwrap(); + + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/partitioned_bucketed_table".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "partitioned_bucketed_table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/partitioned_bucketed_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("date", ArrowDataType::Date32, false), + ArrowField::new("user_id", ArrowDataType::Int64, false), + ArrowField::new("amount", ArrowDataType::Int64, false), + ArrowField::new( + PROJECTED_PARTITION_VALUE_COLUMN, + ArrowDataType::Struct(Fields::empty()), + false, + ), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + match partitioning { + Partitioning::Hash(exprs, _) => { + // Should use _partition column for hash partitioning + assert_eq!( + exprs.len(), + 1, + "Should have exactly one hash column (_partition)" + ); + + let column_names: Vec = exprs + .iter() + .filter_map(|expr| { + expr.as_any() + .downcast_ref::() + .map(|col| col.name().to_string()) + }) + .collect(); + + assert!( + column_names.contains(&PROJECTED_PARTITION_VALUE_COLUMN.to_string()), + "Should use _partition column, got: {:?}", + column_names + ); + } + _ => panic!("Expected Hash partitioning with Identity transform"), + } + } + + #[tokio::test] + async fn test_none_distribution_mode_fallback() { + let schema = Schema::builder() + .with_fields(vec![Arc::new(NestedField::required( + 1, + "id", + Type::Primitive(PrimitiveType::Long), + ))]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .build() + .unwrap(); + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + + let mut properties = std::collections::HashMap::new(); + properties.insert("write.distribution-mode".to_string(), "none".to_string()); + + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/none_table".to_string(), + iceberg::spec::FormatVersion::V2, + properties, + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "none_table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/none_metadata.json".to_string()) + .build() + .unwrap(); + + let input = Arc::new(EmptyExec::new(create_test_arrow_schema())); + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + assert!( + matches!(partitioning, Partitioning::RoundRobinBatch(_)), + "Should use round-robin for 'none' distribution mode" + ); + } + + #[tokio::test] + async fn test_schema_ref_convenience_method() { + let table = create_test_table(); + + let schema_ref_1 = table.current_schema_ref(); + let schema_ref_2 = Arc::clone(table.metadata().current_schema()); + + assert!( + Arc::ptr_eq(&schema_ref_1, &schema_ref_2), + "current_schema_ref() should return the same Arc as manual approach" + ); + } + + #[tokio::test] + async fn test_range_only_partitions_use_round_robin() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "date", + Type::Primitive(PrimitiveType::Date), + )), + Arc::new(NestedField::required( + 2, + "amount", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .add_partition_field("date", "date_day", Transform::Day) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/range_only_table".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "range_only_table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/range_only_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("date", ArrowDataType::Date32, false), + ArrowField::new("amount", ArrowDataType::Int64, false), + ArrowField::new( + PROJECTED_PARTITION_VALUE_COLUMN, + ArrowDataType::Struct(Fields::empty()), + false, + ), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + assert!( + matches!(partitioning, Partitioning::RoundRobinBatch(_)), + "Should use round-robin for temporal transforms (Day) that don't provide good hash distribution" + ); + } + + #[tokio::test] + async fn test_mixed_transforms_use_hash_partitioning() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "date", + Type::Primitive(PrimitiveType::Date), + )), + Arc::new(NestedField::required( + 2, + "user_id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::required( + 3, + "amount", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .add_partition_field("date", "date_day", Transform::Day) + .unwrap() + .add_partition_field("user_id", "user_id", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/mixed_transforms_table".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "mixed_transforms_table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/mixed_transforms_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("date", ArrowDataType::Date32, false), + ArrowField::new("user_id", ArrowDataType::Int64, false), + ArrowField::new("amount", ArrowDataType::Int64, false), + ArrowField::new( + PROJECTED_PARTITION_VALUE_COLUMN, + ArrowDataType::Struct(Fields::empty()), + false, + ), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + match partitioning { + Partitioning::Hash(exprs, _) => { + assert_eq!(exprs.len(), 1, "Should have one hash column (_partition)"); + let column_names: Vec = exprs + .iter() + .filter_map(|expr| { + expr.as_any() + .downcast_ref::() + .map(|col| col.name().to_string()) + }) + .collect(); + assert!( + column_names.contains(&PROJECTED_PARTITION_VALUE_COLUMN.to_string()), + "Should use _partition column for mixed transforms with Identity, got: {:?}", + column_names + ); + } + _ => panic!("Expected Hash partitioning for table with identity transforms"), + } + } + + #[tokio::test] + async fn test_partition_column_with_temporal_transforms_uses_round_robin() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "event_time", + Type::Primitive(PrimitiveType::Timestamp), + )), + Arc::new(NestedField::required( + 2, + "amount", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .add_partition_field("event_time", "event_month", Transform::Month) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/temporal_partition".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "temporal_partition"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/temporal_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new( + "event_time", + ArrowDataType::Timestamp(TimeUnit::Microsecond, None), + false, + ), + ArrowField::new("amount", ArrowDataType::Int64, false), + ArrowField::new( + PROJECTED_PARTITION_VALUE_COLUMN, + ArrowDataType::Struct(Fields::empty()), + false, + ), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + assert!( + matches!(partitioning, Partitioning::RoundRobinBatch(_)), + "Should use round-robin for _partition column with temporal transforms, not Hash" + ); + } + + #[tokio::test] + async fn test_partition_column_with_identity_transforms_uses_hash() { + let schema = Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::required( + 1, + "user_id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::required( + 2, + "amount", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = iceberg::spec::PartitionSpec::builder(schema.clone()) + .add_partition_field("user_id", "user_id", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder().build(&schema).unwrap(); + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + schema, + partition_spec, + sort_order, + "/test/identity_partition".to_string(), + iceberg::spec::FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + let table = Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "identity_partition"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/identity_metadata.json".to_string()) + .build() + .unwrap(); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("user_id", ArrowDataType::Int64, false), + ArrowField::new("amount", ArrowDataType::Int64, false), + ArrowField::new( + PROJECTED_PARTITION_VALUE_COLUMN, + ArrowDataType::Struct(Fields::empty()), + false, + ), + ])); + let input = Arc::new(EmptyExec::new(arrow_schema)); + + let repartitioned_plan = repartition( + input, + table.metadata_ref(), + std::num::NonZeroUsize::new(4).unwrap(), + ) + .unwrap(); + + let partitioning = repartitioned_plan.properties().output_partitioning(); + assert!( + matches!(partitioning, Partitioning::Hash(_, _)), + "Should use Hash partitioning for _partition column with Identity transforms" + ); + } +} From 043ca21cd425096a4b25b66090887d03699df696 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Fri, 31 Oct 2025 06:58:02 -0400 Subject: [PATCH 050/891] feat(reader): Date32 from days since epoch for Literal:try_from_json (#1803) --- crates/iceberg/src/spec/values.rs | 51 +++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/crates/iceberg/src/spec/values.rs b/crates/iceberg/src/spec/values.rs index d06e754dcc..4c763a3941 100644 --- a/crates/iceberg/src/spec/values.rs +++ b/crates/iceberg/src/spec/values.rs @@ -1888,6 +1888,17 @@ impl Literal { date::date_to_days(&NaiveDate::parse_from_str(&s, "%Y-%m-%d")?), )))) } + (PrimitiveType::Date, JsonValue::Number(number)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( + number + .as_i64() + .ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to date (days since epoch)", + ))? + .try_into()?, + )))) + } (PrimitiveType::Time, JsonValue::String(s)) => { Ok(Some(Literal::Primitive(PrimitiveLiteral::Long( time::time_to_microseconds(&NaiveTime::parse_from_str(&s, "%H:%M:%S%.f")?), @@ -3942,4 +3953,44 @@ mod tests { assert_eq!(double_sorted, double_expected); } + + /// Test Date deserialization from JSON as number (days since epoch). + /// + /// This reproduces the scenario from Iceberg Java's TestAddFilesProcedure where: + /// - Date partition columns have initial_default values in manifests + /// - These values are serialized as days since epoch (e.g., 18628 for 2021-01-01) + /// - The JSON schema includes: {"type":"date","initial-default":18628} + /// + /// Prior to this fix, Date values in JSON were only parsed from String format ("2021-01-01"), + /// causing initial_default values to be lost during schema deserialization. + /// + /// This test ensures both formats are supported: + /// - String format: "2021-01-01" (used in table metadata) + /// - Number format: 18628 (used in initial-default values from add_files) + /// + /// See: Iceberg Java TestAddFilesProcedure.addDataPartitionedByDateToPartitioned() + #[test] + fn test_date_from_json_as_number() { + use serde_json::json; + + // Test Date as number (days since epoch) - used in initial-default from add_files + let date_number = json!(18628); // 2021-01-01 is 18628 days since 1970-01-01 + let result = + Literal::try_from_json(date_number, &Type::Primitive(PrimitiveType::Date)).unwrap(); + assert_eq!( + result, + Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) + ); + + // Test Date as string - traditional format + let date_string = json!("2021-01-01"); + let result = + Literal::try_from_json(date_string, &Type::Primitive(PrimitiveType::Date)).unwrap(); + assert_eq!( + result, + Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) + ); + + // Both formats should produce the same Literal value + } } From e8f7a20082cc9b07b600bc82b6c1eeeb0a8f015b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Nov 2025 09:27:59 +0800 Subject: [PATCH 051/891] chore(deps): Bump aws-sdk-glue from 1.125.0 to 1.126.0 (#1812) Bumps [aws-sdk-glue](https://github.com/awslabs/aws-sdk-rust) from 1.125.0 to 1.126.0.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aws-sdk-glue&package-manager=cargo&previous-version=1.125.0&new-version=1.126.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1cacc75fb9..38d284e52d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -710,9 +710,9 @@ dependencies = [ [[package]] name = "aws-sdk-glue" -version = "1.125.0" +version = "1.126.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df436173c8acd8c207ecff1aedfd0992d44d0d6d1e84be73e30dab7746a20363" +checksum = "fd9c10a11584c0b619c9e478143072c4028c39017f98534e206156a7e94188be" dependencies = [ "aws-credential-types", "aws-runtime", @@ -2642,7 +2642,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2784,7 +2784,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -4523,7 +4523,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5421,7 +5421,7 @@ dependencies = [ "once_cell", "socket2 0.6.1", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.59.0", ] [[package]] @@ -5878,7 +5878,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -6950,7 +6950,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -7776,7 +7776,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] From b46ecd23f1e33f3b0dcf2a50c6a24a3dbab7ffc2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Nov 2025 11:21:05 +0800 Subject: [PATCH 052/891] chore(deps): Bump astral-sh/setup-uv from 6 to 7 (#1811) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7.
Release notes

Sourced from astral-sh/setup-uv's releases.

v7.0.0 🌈 node24 and a lot of bugfixes

Changes

This release comes with a load of bug fixes and a speed up. Because of switching from node20 to node24 it is also a breaking change. If you are running on GitHub hosted runners this will just work, if you are using self-hosted runners make sure, that your runners are up to date. If you followed the normal installation instructions your self-hosted runner will keep itself updated.

This release also removes the deprecated input server-url which was used to download uv releases from a different server. The manifest-file input supersedes that functionality by adding a flexible way to define available versions and where they should be downloaded from.

Fixes

  • The action now respects when the environment variable UV_CACHE_DIR is already set and does not overwrite it. It now also finds cache-dir settings in config files if you set them.
  • Some users encountered problems that cache pruning took forever because they had some uv processes running in the background. Starting with uv version 0.8.24 this action uses uv cache prune --ci --force to ignore the running processes
  • If you just want to install uv but not have it available in path, this action now respects UV_NO_MODIFY_PATH
  • Some other actions also set the env var UV_CACHE_DIR. This action can now deal with that but as this could lead to unwanted behavior in some edgecases a warning is now displayed.

Improvements

If you are using minimum version specifiers for the version of uv to install for example

[tool.uv]
required-version = ">=0.8.17"

This action now detects that and directly uses the latest version. Previously it would download all available releases from the uv repo to determine the highest matching candidate for the version specifier, which took much more time.

If you are using other specifiers like 0.8.x this action still needs to download all available releases because the specifier defines an upper bound (not 0.9.0 or later) and "latest" would possibly not satisfy that.

🚨 Breaking changes

🐛 Bug fixes

🚀 Enhancements

🧰 Maintenance

... (truncated)

Commits
  • 8585678 Bump dependencies (#664)
  • 22d500a Bump github/codeql-action from 4.30.8 to 4.30.9 (#652)
  • 14d5571 chore: update known checksums for 0.9.5 (#663)
  • 29cd235 Use tar for extracting the uv zip file on Windows too (#660)
  • 2ddd2b9 chore: update known checksums for 0.9.4 (#651)
  • b7bf789 Fix "lowest" resolution strategy with lower-bound only (#649)
  • cb6c0a5 Change version in docs to v7 (#647)
  • dffc629 Use working-directory to detect empty workdir (#645)
  • 6e346e1 chore: update known checksums for 0.9.3 (#644)
  • 3ccd0fd Bump github/codeql-action from 4.30.7 to 4.30.8 (#639)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=astral-sh/setup-uv&package-manager=github_actions&previous-version=6&new-version=7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/bindings_python_ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index 55e7e5bbd0..e9eabda2cb 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -56,7 +56,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@v7 with: version: "0.9.3" enable-cache: true @@ -90,7 +90,7 @@ jobs: working-directory: "bindings/python" command: build args: --out dist --sdist - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@v7 with: version: "0.9.3" enable-cache: true From ea51429da9c3d74a8e89112e30d9dd80ef19330a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Nov 2025 12:11:25 +0800 Subject: [PATCH 053/891] chore(deps): Bump crate-ci/typos from 1.38.1 to 1.39.0 (#1810) --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 11564caa13..693199eda6 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v5 - name: Check typos - uses: crate-ci/typos@v1.38.1 + uses: crate-ci/typos@v1.39.0 From 05ba2d38ccd93b7536addbc97fda89dcaccd49f8 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Mon, 3 Nov 2025 05:09:35 -0500 Subject: [PATCH 054/891] feat(reader): position-based column projection for Parquet files without field IDs (migrated tables) (#1777) ## What issue does this PR close? Partially address #1749. ## Rationale for this change **Background**: This issue was discovered when running Iceberg Java's test suite against our [experimental DataFusion Comet branch that uses iceberg-rust](https://github.com/apache/datafusion-comet/pull/2528). Many failures occurred in `TestMigrateTableAction.java`, which tests reading Parquet files from migrated tables (_e.g.,_ from Hive or Spark) that lack embedded field ID metadata. **Problem**: The Rust ArrowReader was unable to read these files, while Iceberg Java handles them using a position-based fallback where top-level field ID N maps to top-level Parquet column position N-1, and entire columns (including nested content) are projected. ## What changes are included in this PR? This PR implements position-based column projection for Parquet files without field IDs, enabling iceberg-rust to read migrated tables. **Solution**: Implemented fallback projection in `ArrowReader::get_arrow_projection_mask_fallback()` that matches Java's `ParquetSchemaUtil.pruneColumnsFallback()` behavior: - Detects Parquet files without field IDs by checking Arrow schema metadata - Maps top-level field IDs to top-level column positions (field IDs are 1-indexed, positions are 0-indexed) - Uses `ProjectionMask::roots()` to project entire columns including nested content (structs, lists, maps) - Adds field ID metadata to the projected schema for `RecordBatchTransformer` - Supports schema evolution by allowing missing columns (filled with default values by `RecordBatchTransformer`) This implementation now matches Iceberg Java's behavior for reading migrated tables, enabling interoperability with Java-based tooling and workflows. ## Are these changes tested? Yes, comprehensive unit tests were added to verify the fallback path works correctly: - `test_read_parquet_file_without_field_ids` - Basic projection with primitive columns using position-based mapping - `test_read_parquet_without_field_ids_partial_projection` - Project subset of columns - `test_read_parquet_without_field_ids_schema_evolution` - Handle missing columns with NULL values - `test_read_parquet_without_field_ids_multiple_row_groups` - Verify behavior across row group boundaries - `test_read_parquet_without_field_ids_with_struct` - Project structs with nested fields (entire top-level column) - `test_read_parquet_without_field_ids_filter_eliminates_all_rows` - Comet saw a panic when all row groups were filtered out, this reproduces that scenario - `test_read_parquet_without_field_ids_schema_evolution_add_column_in_middle` - Schema evolution with a column in the middle caused a panic at one point All tests verify that behavior matches Iceberg Java's `pruneColumnsFallback()` implementation in `/parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java`. --------- Co-authored-by: Renjie Liu --- .../iceberg/src/arrow/delete_file_loader.rs | 1 + crates/iceberg/src/arrow/reader.rs | 1007 +++++++++++++++-- 2 files changed, 915 insertions(+), 93 deletions(-) diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index 54fc4a58a7..f2c63cc59c 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -63,6 +63,7 @@ impl BasicDeleteFileLoader { data_file_path, self.file_io.clone(), false, + None, ) .await? .build()? diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index e0894ad6b4..fed8f19c05 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -183,22 +183,54 @@ impl ArrowReader { let delete_filter_rx = delete_file_loader.load_deletes(&task.deletes, task.schema.clone()); - let mut record_batch_stream_builder = Self::create_parquet_record_batch_stream_builder( + // Migrated tables lack field IDs, requiring us to inspect the schema to choose + // between field-ID-based or position-based projection + let initial_stream_builder = Self::create_parquet_record_batch_stream_builder( &task.data_file_path, file_io.clone(), should_load_page_index, + None, ) .await?; - // Create a projection mask for the batch stream to select which columns in the - // Parquet file that we want in the response + // Parquet files from Hive/Spark migrations lack field IDs in their metadata + let missing_field_ids = initial_stream_builder + .schema() + .fields() + .iter() + .next() + .is_some_and(|f| f.metadata().get(PARQUET_FIELD_ID_META_KEY).is_none()); + + // Adding position-based fallback IDs at schema level (not per-batch) enables projection + // on files that lack embedded field IDs. We recreate the builder to apply the modified schema. + let mut record_batch_stream_builder = if missing_field_ids { + let arrow_schema = + add_fallback_field_ids_to_arrow_schema(initial_stream_builder.schema()); + let options = ArrowReaderOptions::new().with_schema(arrow_schema); + + Self::create_parquet_record_batch_stream_builder( + &task.data_file_path, + file_io.clone(), + should_load_page_index, + Some(options), + ) + .await? + } else { + initial_stream_builder + }; + + // Fallback IDs don't match Parquet's embedded field IDs (since they don't exist), + // so we must use position-based projection instead of field-ID matching let projection_mask = Self::get_arrow_projection_mask( &task.project_field_ids, &task.schema, record_batch_stream_builder.parquet_schema(), record_batch_stream_builder.schema(), + missing_field_ids, // Whether to use position-based (true) or field-ID-based (false) projection )?; - record_batch_stream_builder = record_batch_stream_builder.with_projection(projection_mask); + + record_batch_stream_builder = + record_batch_stream_builder.with_projection(projection_mask.clone()); // RecordBatchTransformer performs any transformations required on the RecordBatches // that come back from the file, such as type promotion, default column insertion @@ -353,6 +385,7 @@ impl ArrowReader { data_file_path: &str, file_io: FileIO, should_load_page_index: bool, + arrow_reader_options: Option, ) -> Result>> { // Get the metadata for the Parquet file we need to read and build // a reader for the data within @@ -365,11 +398,9 @@ impl ArrowReader { .with_preload_page_index(should_load_page_index); // Create the record batch stream builder, which wraps the parquet file reader - let record_batch_stream_builder = ParquetRecordBatchStreamBuilder::new_with_options( - parquet_file_reader, - ArrowReaderOptions::new(), - ) - .await?; + let options = arrow_reader_options.unwrap_or_default(); + let record_batch_stream_builder = + ParquetRecordBatchStreamBuilder::new_with_options(parquet_file_reader, options).await?; Ok(record_batch_stream_builder) } @@ -503,13 +534,18 @@ impl ArrowReader { visit(&mut collector, predicate)?; let iceberg_field_ids = collector.field_ids(); - let field_id_map = build_field_id_map(parquet_schema)?; + + // Without embedded field IDs, we fall back to position-based mapping for compatibility + let field_id_map = match build_field_id_map(parquet_schema)? { + Some(map) => map, + None => build_fallback_field_id_map(parquet_schema), + }; Ok((iceberg_field_ids, field_id_map)) } - /// Insert the leaf field id into the field_ids using for projection. - /// For nested type, it will recursively insert the leaf field id. + /// Recursively extract leaf field IDs because Parquet projection works at the leaf column level. + /// Nested types (struct/list/map) are flattened in Parquet's columnar format. fn include_leaf_field_id(field: &NestedField, field_ids: &mut Vec) { match field.field_type.as_ref() { Type::Primitive(_) => { @@ -535,6 +571,7 @@ impl ArrowReader { iceberg_schema_of_task: &Schema, parquet_schema: &SchemaDescriptor, arrow_schema: &ArrowSchemaRef, + use_fallback: bool, // Whether file lacks embedded field IDs (e.g., migrated from Hive/Spark) ) -> Result { fn type_promotion_is_valid( file_type: Option<&PrimitiveType>, @@ -560,83 +597,132 @@ impl ArrowReader { } } - let mut leaf_field_ids = vec![]; - for field_id in field_ids { - let field = iceberg_schema_of_task.field_by_id(*field_id); - if let Some(field) = field { - Self::include_leaf_field_id(field, &mut leaf_field_ids); - } + if field_ids.is_empty() { + return Ok(ProjectionMask::all()); } - if leaf_field_ids.is_empty() { - Ok(ProjectionMask::all()) + if use_fallback { + // Position-based projection necessary because file lacks embedded field IDs + Self::get_arrow_projection_mask_fallback(field_ids, parquet_schema) } else { - // Build the map between field id and column index in Parquet schema. - let mut column_map = HashMap::new(); - - let fields = arrow_schema.fields(); - // Pre-project only the fields that have been selected, possibly avoiding converting - // some Arrow types that are not yet supported. - let mut projected_fields: HashMap = HashMap::new(); - let projected_arrow_schema = ArrowSchema::new_with_metadata( - fields.filter_leaves(|_, f| { - f.metadata() - .get(PARQUET_FIELD_ID_META_KEY) - .and_then(|field_id| i32::from_str(field_id).ok()) - .is_some_and(|field_id| { - projected_fields.insert((*f).clone(), field_id); - leaf_field_ids.contains(&field_id) - }) - }), - arrow_schema.metadata().clone(), - ); - let iceberg_schema = arrow_schema_to_schema(&projected_arrow_schema)?; - - fields.filter_leaves(|idx, field| { - let Some(field_id) = projected_fields.get(field).cloned() else { - return false; - }; + // Field-ID-based projection using embedded field IDs from Parquet metadata + + // Parquet's columnar format requires leaf-level (not top-level struct/list/map) projection + let mut leaf_field_ids = vec![]; + for field_id in field_ids { + let field = iceberg_schema_of_task.field_by_id(*field_id); + if let Some(field) = field { + Self::include_leaf_field_id(field, &mut leaf_field_ids); + } + } + + Self::get_arrow_projection_mask_with_field_ids( + &leaf_field_ids, + iceberg_schema_of_task, + parquet_schema, + arrow_schema, + type_promotion_is_valid, + ) + } + } + + /// Standard projection using embedded field IDs from Parquet metadata. + /// For iceberg-java compatibility with ParquetSchemaUtil.pruneColumns(). + fn get_arrow_projection_mask_with_field_ids( + leaf_field_ids: &[i32], + iceberg_schema_of_task: &Schema, + parquet_schema: &SchemaDescriptor, + arrow_schema: &ArrowSchemaRef, + type_promotion_is_valid: fn(Option<&PrimitiveType>, Option<&PrimitiveType>) -> bool, + ) -> Result { + let mut column_map = HashMap::new(); + let fields = arrow_schema.fields(); + + // Pre-project only the fields that have been selected, possibly avoiding converting + // some Arrow types that are not yet supported. + let mut projected_fields: HashMap = HashMap::new(); + let projected_arrow_schema = ArrowSchema::new_with_metadata( + fields.filter_leaves(|_, f| { + f.metadata() + .get(PARQUET_FIELD_ID_META_KEY) + .and_then(|field_id| i32::from_str(field_id).ok()) + .is_some_and(|field_id| { + projected_fields.insert((*f).clone(), field_id); + leaf_field_ids.contains(&field_id) + }) + }), + arrow_schema.metadata().clone(), + ); + let iceberg_schema = arrow_schema_to_schema(&projected_arrow_schema)?; - let iceberg_field = iceberg_schema_of_task.field_by_id(field_id); - let parquet_iceberg_field = iceberg_schema.field_by_id(field_id); + fields.filter_leaves(|idx, field| { + let Some(field_id) = projected_fields.get(field).cloned() else { + return false; + }; - if iceberg_field.is_none() || parquet_iceberg_field.is_none() { - return false; - } + let iceberg_field = iceberg_schema_of_task.field_by_id(field_id); + let parquet_iceberg_field = iceberg_schema.field_by_id(field_id); - if !type_promotion_is_valid( - parquet_iceberg_field - .unwrap() - .field_type - .as_primitive_type(), - iceberg_field.unwrap().field_type.as_primitive_type(), - ) { - return false; - } + if iceberg_field.is_none() || parquet_iceberg_field.is_none() { + return false; + } - column_map.insert(field_id, idx); - true - }); - - // Only project columns that exist in the Parquet file. - // Missing columns will be added by RecordBatchTransformer with default/NULL values. - // This supports schema evolution where new columns are added to the table schema - // but old Parquet files don't have them yet. - let mut indices = vec![]; - for field_id in leaf_field_ids { - if let Some(col_idx) = column_map.get(&field_id) { - indices.push(*col_idx); - } - // Skip fields that don't exist in the Parquet file - they will be added later + if !type_promotion_is_valid( + parquet_iceberg_field + .unwrap() + .field_type + .as_primitive_type(), + iceberg_field.unwrap().field_type.as_primitive_type(), + ) { + return false; + } + + column_map.insert(field_id, idx); + true + }); + + // Schema evolution: New columns may not exist in old Parquet files. + // We only project existing columns; RecordBatchTransformer adds default/NULL values. + let mut indices = vec![]; + for field_id in leaf_field_ids { + if let Some(col_idx) = column_map.get(field_id) { + indices.push(*col_idx); } + } + + if indices.is_empty() { + // Edge case: All requested columns are new (don't exist in file). + // Project all columns so RecordBatchTransformer has a batch to transform. + Ok(ProjectionMask::all()) + } else { + Ok(ProjectionMask::leaves(parquet_schema, indices)) + } + } + + /// Fallback projection for Parquet files without field IDs. + /// Uses position-based matching: field ID N → column position N-1. + /// Projects entire top-level columns (including nested content) for iceberg-java compatibility. + fn get_arrow_projection_mask_fallback( + field_ids: &[i32], + parquet_schema: &SchemaDescriptor, + ) -> Result { + // Position-based: field_id N → column N-1 (field IDs are 1-indexed) + let parquet_root_fields = parquet_schema.root_schema().get_fields(); + let mut root_indices = vec![]; + + for field_id in field_ids.iter() { + let parquet_pos = (*field_id - 1) as usize; - if indices.is_empty() { - // If no columns from the projection exist in the file, project all columns - // This can happen if all requested columns are new and need to be added by the transformer - Ok(ProjectionMask::all()) - } else { - Ok(ProjectionMask::leaves(parquet_schema, indices)) + if parquet_pos < parquet_root_fields.len() { + root_indices.push(parquet_pos); } + // RecordBatchTransformer adds missing columns with NULL values + } + + if root_indices.is_empty() { + Ok(ProjectionMask::all()) + } else { + Ok(ProjectionMask::roots(parquet_schema, root_indices)) } } @@ -713,6 +799,13 @@ impl ArrowReader { )); }; + // If all row groups were filtered out, return an empty RowSelection (select no rows) + if let Some(selected_row_groups) = selected_row_groups { + if selected_row_groups.is_empty() { + return Ok(RowSelection::from(Vec::new())); + } + } + let mut selected_row_groups_idx = 0; let page_index = column_index @@ -785,22 +878,16 @@ impl ArrowReader { } /// Build the map of parquet field id to Parquet column index in the schema. -fn build_field_id_map(parquet_schema: &SchemaDescriptor) -> Result> { +/// Returns None if the Parquet file doesn't have field IDs embedded (e.g., migrated tables). +fn build_field_id_map(parquet_schema: &SchemaDescriptor) -> Result>> { let mut column_map = HashMap::new(); + for (idx, field) in parquet_schema.columns().iter().enumerate() { let field_type = field.self_type(); match field_type { ParquetType::PrimitiveType { basic_info, .. } => { if !basic_info.has_id() { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Leave column idx: {}, name: {}, type {:?} in schema doesn't have field id", - idx, - basic_info.name(), - field_type - ), - )); + return Ok(None); } column_map.insert(basic_info.id(), idx); } @@ -815,7 +902,59 @@ fn build_field_id_map(parquet_schema: &SchemaDescriptor) -> Result HashMap { + let mut column_map = HashMap::new(); + + // 1-indexed to match iceberg-java's convention + for (idx, _field) in parquet_schema.columns().iter().enumerate() { + let field_id = (idx + 1) as i32; + column_map.insert(field_id, idx); + } + + column_map +} + +/// Add position-based fallback field IDs to Arrow schema for Parquet files lacking them. +/// Enables projection on migrated files (e.g., from Hive/Spark). +/// +/// Why at schema level (not per-batch): Efficiency - avoids repeated schema modification. +/// Why only top-level: Nested projection uses leaf column indices, not parent struct IDs. +/// Why 1-indexed: Compatibility with iceberg-java's ParquetSchemaUtil.addFallbackIds(). +fn add_fallback_field_ids_to_arrow_schema(arrow_schema: &ArrowSchemaRef) -> Arc { + debug_assert!( + arrow_schema + .fields() + .iter() + .next() + .is_none_or(|f| f.metadata().get(PARQUET_FIELD_ID_META_KEY).is_none()), + "Schema already has field IDs" + ); + + use arrow_schema::Field; + + let fields_with_fallback_ids: Vec<_> = arrow_schema + .fields() + .iter() + .enumerate() + .map(|(pos, field)| { + let mut metadata = field.metadata().clone(); + let field_id = (pos + 1) as i32; // 1-indexed for Java compatibility + metadata.insert(PARQUET_FIELD_ID_META_KEY.to_string(), field_id.to_string()); + + Field::new(field.name(), field.data_type().clone(), field.is_nullable()) + .with_metadata(metadata) + }) + .collect(); + + Arc::new(ArrowSchema::new_with_metadata( + fields_with_fallback_ids, + arrow_schema.metadata().clone(), + )) } /// A visitor to collect field ids from bound predicates. @@ -1637,6 +1776,7 @@ message schema { &schema, &parquet_schema, &arrow_schema, + false, ) .unwrap_err(); @@ -1652,6 +1792,7 @@ message schema { &schema, &parquet_schema, &arrow_schema, + false, ) .unwrap_err(); @@ -1662,9 +1803,14 @@ message schema { ); // Finally avoid selecting fields with unsupported data types - let mask = - ArrowReader::get_arrow_projection_mask(&[1], &schema, &parquet_schema, &arrow_schema) - .expect("Some ProjectionMask"); + let mask = ArrowReader::get_arrow_projection_mask( + &[1], + &schema, + &parquet_schema, + &arrow_schema, + false, + ) + .expect("Some ProjectionMask"); assert_eq!(mask, ProjectionMask::leaves(&parquet_schema, vec![0])); } @@ -2887,4 +3033,679 @@ message schema { "Should have ids 101-200 (all of row group 1)" ); } + + /// Test reading Parquet files without field ID metadata (e.g., migrated tables). + /// This exercises the position-based fallback path. + /// + /// Corresponds to Java's ParquetSchemaUtil.addFallbackIds() + pruneColumnsFallback() + /// in /parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java + #[tokio::test] + async fn test_read_parquet_file_without_field_ids() { + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(2, "age", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + // Parquet file from a migrated table - no field ID metadata + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("name", DataType::Utf8, false), + Field::new("age", DataType::Int32, false), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let name_data = vec!["Alice", "Bob", "Charlie"]; + let age_data = vec![30, 25, 35]; + + use arrow_array::Int32Array; + let name_col = Arc::new(StringArray::from(name_data.clone())) as ArrayRef; + let age_col = Arc::new(Int32Array::from(age_data.clone())) as ArrayRef; + + let to_write = RecordBatch::try_new(arrow_schema.clone(), vec![name_col, age_col]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let batch = &result[0]; + assert_eq!(batch.num_rows(), 3); + assert_eq!(batch.num_columns(), 2); + + // Verify position-based mapping: field_id 1 → position 0, field_id 2 → position 1 + let name_array = batch.column(0).as_string::(); + assert_eq!(name_array.value(0), "Alice"); + assert_eq!(name_array.value(1), "Bob"); + assert_eq!(name_array.value(2), "Charlie"); + + let age_array = batch + .column(1) + .as_primitive::(); + assert_eq!(age_array.value(0), 30); + assert_eq!(age_array.value(1), 25); + assert_eq!(age_array.value(2), 35); + } + + /// Test reading Parquet files without field IDs with partial projection. + /// Only a subset of columns are requested, verifying position-based fallback + /// handles column selection correctly. + #[tokio::test] + async fn test_read_parquet_without_field_ids_partial_projection() { + use arrow_array::Int32Array; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "col1", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(2, "col2", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(3, "col3", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(4, "col4", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("col1", DataType::Utf8, false), + Field::new("col2", DataType::Int32, false), + Field::new("col3", DataType::Utf8, false), + Field::new("col4", DataType::Int32, false), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let col1_data = Arc::new(StringArray::from(vec!["a", "b"])) as ArrayRef; + let col2_data = Arc::new(Int32Array::from(vec![10, 20])) as ArrayRef; + let col3_data = Arc::new(StringArray::from(vec!["c", "d"])) as ArrayRef; + let col4_data = Arc::new(Int32Array::from(vec![30, 40])) as ArrayRef; + + let to_write = RecordBatch::try_new(arrow_schema.clone(), vec![ + col1_data, col2_data, col3_data, col4_data, + ]) + .unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 3], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let batch = &result[0]; + assert_eq!(batch.num_rows(), 2); + assert_eq!(batch.num_columns(), 2); + + let col1_array = batch.column(0).as_string::(); + assert_eq!(col1_array.value(0), "a"); + assert_eq!(col1_array.value(1), "b"); + + let col3_array = batch.column(1).as_string::(); + assert_eq!(col3_array.value(0), "c"); + assert_eq!(col3_array.value(1), "d"); + } + + /// Test reading Parquet files without field IDs with schema evolution. + /// The Iceberg schema has more fields than the Parquet file, testing that + /// missing columns are filled with NULLs. + #[tokio::test] + async fn test_read_parquet_without_field_ids_schema_evolution() { + use arrow_array::{Array, Int32Array}; + + // Schema with field 3 added after the file was written + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(2, "age", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(3, "city", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("name", DataType::Utf8, false), + Field::new("age", DataType::Int32, false), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let name_data = Arc::new(StringArray::from(vec!["Alice", "Bob"])) as ArrayRef; + let age_data = Arc::new(Int32Array::from(vec![30, 25])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema.clone(), vec![name_data, age_data]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2, 3], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let batch = &result[0]; + assert_eq!(batch.num_rows(), 2); + assert_eq!(batch.num_columns(), 3); + + let name_array = batch.column(0).as_string::(); + assert_eq!(name_array.value(0), "Alice"); + assert_eq!(name_array.value(1), "Bob"); + + let age_array = batch + .column(1) + .as_primitive::(); + assert_eq!(age_array.value(0), 30); + assert_eq!(age_array.value(1), 25); + + // Verify missing column filled with NULLs + let city_array = batch.column(2).as_string::(); + assert_eq!(city_array.null_count(), 2); + assert!(city_array.is_null(0)); + assert!(city_array.is_null(1)); + } + + /// Test reading Parquet files without field IDs that have multiple row groups. + /// This ensures the position-based fallback works correctly across row group boundaries. + #[tokio::test] + async fn test_read_parquet_without_field_ids_multiple_row_groups() { + use arrow_array::Int32Array; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(2, "value", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("name", DataType::Utf8, false), + Field::new("value", DataType::Int32, false), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + // Small row group size to create multiple row groups + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .set_write_batch_size(2) + .set_max_row_group_size(2) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + + // Write 6 rows in 3 batches (will create 3 row groups) + for batch_num in 0..3 { + let name_data = Arc::new(StringArray::from(vec![ + format!("name_{}", batch_num * 2), + format!("name_{}", batch_num * 2 + 1), + ])) as ArrayRef; + let value_data = + Arc::new(Int32Array::from(vec![batch_num * 2, batch_num * 2 + 1])) as ArrayRef; + + let batch = + RecordBatch::try_new(arrow_schema.clone(), vec![name_data, value_data]).unwrap(); + writer.write(&batch).expect("Writing batch"); + } + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert!(!result.is_empty()); + + let mut all_names = Vec::new(); + let mut all_values = Vec::new(); + + for batch in &result { + let name_array = batch.column(0).as_string::(); + let value_array = batch + .column(1) + .as_primitive::(); + + for i in 0..batch.num_rows() { + all_names.push(name_array.value(i).to_string()); + all_values.push(value_array.value(i)); + } + } + + assert_eq!(all_names.len(), 6); + assert_eq!(all_values.len(), 6); + + for i in 0..6 { + assert_eq!(all_names[i], format!("name_{}", i)); + assert_eq!(all_values[i], i as i32); + } + } + + /// Test reading Parquet files without field IDs with nested types (struct). + /// Java's pruneColumnsFallback() projects entire top-level columns including nested content. + /// This test verifies that a top-level struct field is projected correctly with all its nested fields. + #[tokio::test] + async fn test_read_parquet_without_field_ids_with_struct() { + use arrow_array::{Int32Array, StructArray}; + use arrow_schema::Fields; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required( + 2, + "person", + Type::Struct(crate::spec::StructType::new(vec![ + NestedField::required( + 3, + "name", + Type::Primitive(PrimitiveType::String), + ) + .into(), + NestedField::required(4, "age", Type::Primitive(PrimitiveType::Int)) + .into(), + ])), + ) + .into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new( + "person", + DataType::Struct(Fields::from(vec![ + Field::new("name", DataType::Utf8, false), + Field::new("age", DataType::Int32, false), + ])), + false, + ), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let id_data = Arc::new(Int32Array::from(vec![1, 2])) as ArrayRef; + let name_data = Arc::new(StringArray::from(vec!["Alice", "Bob"])) as ArrayRef; + let age_data = Arc::new(Int32Array::from(vec![30, 25])) as ArrayRef; + let person_data = Arc::new(StructArray::from(vec![ + ( + Arc::new(Field::new("name", DataType::Utf8, false)), + name_data, + ), + ( + Arc::new(Field::new("age", DataType::Int32, false)), + age_data, + ), + ])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema.clone(), vec![id_data, person_data]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let batch = &result[0]; + assert_eq!(batch.num_rows(), 2); + assert_eq!(batch.num_columns(), 2); + + let id_array = batch + .column(0) + .as_primitive::(); + assert_eq!(id_array.value(0), 1); + assert_eq!(id_array.value(1), 2); + + let person_array = batch.column(1).as_struct(); + assert_eq!(person_array.num_columns(), 2); + + let name_array = person_array.column(0).as_string::(); + assert_eq!(name_array.value(0), "Alice"); + assert_eq!(name_array.value(1), "Bob"); + + let age_array = person_array + .column(1) + .as_primitive::(); + assert_eq!(age_array.value(0), 30); + assert_eq!(age_array.value(1), 25); + } + + /// Test reading Parquet files without field IDs with schema evolution - column added in the middle. + /// When a new column is inserted between existing columns in the schema order, + /// the fallback projection must correctly map field IDs to output positions. + #[tokio::test] + async fn test_read_parquet_without_field_ids_schema_evolution_add_column_in_middle() { + use arrow_array::{Array, Int32Array}; + + let arrow_schema_old = Arc::new(ArrowSchema::new(vec![ + Field::new("col0", DataType::Int32, true), + Field::new("col1", DataType::Int32, true), + ])); + + // New column added between existing columns: col0 (id=1), newCol (id=5), col1 (id=2) + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::optional(1, "col0", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(5, "newCol", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "col1", Type::Primitive(PrimitiveType::Int)).into(), + ]) + .build() + .unwrap(), + ); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let col0_data = Arc::new(Int32Array::from(vec![1, 2])) as ArrayRef; + let col1_data = Arc::new(Int32Array::from(vec![10, 20])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema_old.clone(), vec![col0_data, col1_data]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + let reader = ArrowReaderBuilder::new(file_io).build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 5, 2], + predicate: None, + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let batch = &result[0]; + assert_eq!(batch.num_rows(), 2); + assert_eq!(batch.num_columns(), 3); + + let result_col0 = batch + .column(0) + .as_primitive::(); + assert_eq!(result_col0.value(0), 1); + assert_eq!(result_col0.value(1), 2); + + // New column should be NULL (doesn't exist in old file) + let result_newcol = batch + .column(1) + .as_primitive::(); + assert_eq!(result_newcol.null_count(), 2); + assert!(result_newcol.is_null(0)); + assert!(result_newcol.is_null(1)); + + let result_col1 = batch + .column(2) + .as_primitive::(); + assert_eq!(result_col1.value(0), 10); + assert_eq!(result_col1.value(1), 20); + } + + /// Test reading Parquet files without field IDs with a filter that eliminates all row groups. + /// During development of field ID mapping, we saw a panic when row_selection_enabled=true and + /// all row groups are filtered out. + #[tokio::test] + async fn test_read_parquet_without_field_ids_filter_eliminates_all_rows() { + use arrow_array::{Float64Array, Int32Array}; + + // Schema with fields that will use fallback IDs 1, 2, 3 + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "value", Type::Primitive(PrimitiveType::Double)) + .into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + Field::new("value", DataType::Float64, false), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + // Write data where all ids are >= 10 + let id_data = Arc::new(Int32Array::from(vec![10, 11, 12])) as ArrayRef; + let name_data = Arc::new(StringArray::from(vec!["a", "b", "c"])) as ArrayRef; + let value_data = Arc::new(Float64Array::from(vec![100.0, 200.0, 300.0])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema.clone(), vec![id_data, name_data, value_data]) + .unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + // Filter that eliminates all row groups: id < 5 + let predicate = Reference::new("id").less_than(Datum::int(5)); + + // Enable both row_group_filtering and row_selection - triggered the panic + let reader = ArrowReaderBuilder::new(file_io) + .with_row_group_filtering_enabled(true) + .with_row_selection_enabled(true) + .build(); + + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/1.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2, 3], + predicate: Some(predicate.bind(schema, true).unwrap()), + deletes: vec![], + })] + .into_iter(), + )) as FileScanTaskStream; + + // Should no longer panic + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Should return empty results + assert!(result.is_empty() || result.iter().all(|batch| batch.num_rows() == 0)); + } } From 835f528f4dd1d700158ce5339e14bc1da04626d1 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 4 Nov 2025 04:06:05 -0500 Subject: [PATCH 055/891] fix(reader): Equality delete files with partial schemas (containing only equality columns) (#1782) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What issue does this PR close? Partially address #1749. ## Rationale for this change Equality delete files with partial schemas (containing only equality columns) were hitting Arrow validation errors: "Column 'id' is declared as non-nullable but contains null values” in Iceberg Java’s TestSparkReaderDeletes suite. The bug occurs because `evolve_schema()` adds missing columns with NULL values, which fails when those columns are declared REQUIRED in the table schema. ## What changes are included in this PR? Change the `evolve_schema()` call to take `equality_ids` because per the Iceberg spec, evolve schema for equality deletes but only for the `equality_ids` columns, not all table columns. ## Are these changes tested? `test_partial_schema_equality_deletes_evolve_succeeds` --- .../src/arrow/caching_delete_file_loader.rs | 118 ++++++++++++++++-- .../iceberg/src/arrow/delete_file_loader.rs | 24 ++-- 2 files changed, 124 insertions(+), 18 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 77f29b7f1e..078635c9be 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -224,16 +224,22 @@ impl CachingDeleteFileLoader { let (sender, receiver) = channel(); del_filter.insert_equality_delete(&task.file_path, receiver); + // Per the Iceberg spec, evolve schema for equality deletes but only for the + // equality_ids columns, not all table columns. + let equality_ids_vec = task.equality_ids.clone().unwrap(); + let evolved_stream = BasicDeleteFileLoader::evolve_schema( + basic_delete_file_loader + .parquet_to_batch_stream(&task.file_path) + .await?, + schema, + &equality_ids_vec, + ) + .await?; + Ok(DeleteFileContext::FreshEqDel { - batch_stream: BasicDeleteFileLoader::evolve_schema( - basic_delete_file_loader - .parquet_to_batch_stream(&task.file_path) - .await?, - schema, - ) - .await?, + batch_stream: evolved_stream, sender, - equality_ids: HashSet::from_iter(task.equality_ids.clone().unwrap()), + equality_ids: HashSet::from_iter(equality_ids_vec), }) } @@ -536,6 +542,7 @@ mod tests { use std::fs::File; use std::sync::Arc; + use arrow_array::cast::AsArray; use arrow_array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringArray, StructArray}; use arrow_schema::{DataType, Field, Fields}; use parquet::arrow::{ArrowWriter, PARQUET_FIELD_ID_META_KEY}; @@ -686,4 +693,99 @@ mod tests { let result = delete_filter.get_delete_vector(&file_scan_tasks[1]); assert!(result.is_none()); // no pos dels for file 3 } + + /// Verifies that evolve_schema on partial-schema equality deletes works correctly + /// when only equality_ids columns are evolved, not all table columns. + /// + /// Per the [Iceberg spec](https://iceberg.apache.org/spec/#equality-delete-files), + /// equality delete files can contain only a subset of columns. + #[tokio::test] + async fn test_partial_schema_equality_deletes_evolve_succeeds() { + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().as_os_str().to_str().unwrap(); + + // Create table schema with REQUIRED fields + let table_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + crate::spec::NestedField::required( + 1, + "id", + crate::spec::Type::Primitive(crate::spec::PrimitiveType::Int), + ) + .into(), + crate::spec::NestedField::required( + 2, + "data", + crate::spec::Type::Primitive(crate::spec::PrimitiveType::String), + ) + .into(), + ]) + .build() + .unwrap(), + ); + + // Write equality delete file with PARTIAL schema (only 'data' column) + let delete_file_path = { + let data_vals = vec!["a", "d", "g"]; + let data_col = Arc::new(StringArray::from(data_vals)) as ArrayRef; + + let delete_schema = Arc::new(arrow_schema::Schema::new(vec![simple_field( + "data", + DataType::Utf8, + false, + "2", // field ID + )])); + + let delete_batch = RecordBatch::try_new(delete_schema.clone(), vec![data_col]).unwrap(); + + let path = format!("{}/partial-eq-deletes.parquet", &table_location); + let file = File::create(&path).unwrap(); + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + let mut writer = + ArrowWriter::try_new(file, delete_batch.schema(), Some(props)).unwrap(); + writer.write(&delete_batch).expect("Writing batch"); + writer.close().unwrap(); + path + }; + + let file_io = FileIO::from_path(table_location).unwrap().build().unwrap(); + let basic_delete_file_loader = BasicDeleteFileLoader::new(file_io.clone()); + + let batch_stream = basic_delete_file_loader + .parquet_to_batch_stream(&delete_file_path) + .await + .unwrap(); + + // Only evolve the equality_ids columns (field 2), not all table columns + let equality_ids = vec![2]; + let evolved_stream = + BasicDeleteFileLoader::evolve_schema(batch_stream, table_schema, &equality_ids) + .await + .unwrap(); + + let result = evolved_stream.try_collect::>().await; + + assert!( + result.is_ok(), + "Expected success when evolving only equality_ids columns, got error: {:?}", + result.err() + ); + + let batches = result.unwrap(); + assert_eq!(batches.len(), 1); + + let batch = &batches[0]; + assert_eq!(batch.num_rows(), 3); + assert_eq!(batch.num_columns(), 1); // Only 'data' column + + // Verify the actual values are preserved after schema evolution + let data_col = batch.column(0).as_string::(); + assert_eq!(data_col.value(0), "a"); + assert_eq!(data_col.value(1), "d"); + assert_eq!(data_col.value(2), "g"); + } } diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index f2c63cc59c..c0b1392dc6 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -72,20 +72,17 @@ impl BasicDeleteFileLoader { Ok(Box::pin(record_batch_stream) as ArrowRecordBatchStream) } - /// Evolves the schema of the RecordBatches from an equality delete file + /// Evolves the schema of the RecordBatches from an equality delete file. + /// + /// Per the [Iceberg spec](https://iceberg.apache.org/spec/#equality-delete-files), + /// only evolves the specified `equality_ids` columns, not all table columns. pub(crate) async fn evolve_schema( record_batch_stream: ArrowRecordBatchStream, target_schema: Arc, + equality_ids: &[i32], ) -> Result { - let eq_ids = target_schema - .as_ref() - .field_id_to_name_map() - .keys() - .cloned() - .collect::>(); - let mut record_batch_transformer = - RecordBatchTransformer::build(target_schema.clone(), &eq_ids); + RecordBatchTransformer::build(target_schema.clone(), equality_ids); let record_batch_stream = record_batch_stream.map(move |record_batch| { record_batch.and_then(|record_batch| { @@ -106,7 +103,14 @@ impl DeleteFileLoader for BasicDeleteFileLoader { ) -> Result { let raw_batch_stream = self.parquet_to_batch_stream(&task.file_path).await?; - Self::evolve_schema(raw_batch_stream, schema).await + // For equality deletes, only evolve the equality_ids columns. + // For positional deletes (equality_ids is None), use all field IDs. + let field_ids = match &task.equality_ids { + Some(ids) => ids.clone(), + None => schema.field_id_to_name_map().keys().cloned().collect(), + }; + + Self::evolve_schema(raw_batch_stream, schema, &field_ids).await } } From e08bc61985a0a458ce088ec9d0b2429cd4fb2a20 Mon Sep 17 00:00:00 2001 From: Drew Gallardo Date: Tue, 4 Nov 2025 01:18:21 -0800 Subject: [PATCH 056/891] infra: use apache/hive:4.0.0 as hive Dockerfile base image (#1823) ## What changes are included in this PR? We had some failures in the Pyiceberg repo with the hive docker file here: https://github.com/apache/iceberg-python/pull/2697, so I'm porting this over here. The HMS test Dockerfile was using a deprecated `openjdk:8-jre-slim` base image that has very **recently** been removed from Docker Hub, causing build failures: ``` #7 ERROR: docker.io/library/openjdk:8-jre-slim: not found ``` Simplified the Dockerfile to use apache/hive:3.1.3 as the base image directly, also eliminating the multi-stage build pattern. Which removes the dependency on a deprecated OpenJDK image and will use what's included in hive, and maintains the same functionality for HMS integration tests. Inspired by [](https://github.com/trinodb/docker-images/blob/master/testing/hive4.0-hive/Dockerfile) --- .../hms/testdata/hms_catalog/Dockerfile | 22 +++++++------------ 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/crates/catalog/hms/testdata/hms_catalog/Dockerfile b/crates/catalog/hms/testdata/hms_catalog/Dockerfile index 8392e174a8..65ebc3d6d5 100644 --- a/crates/catalog/hms/testdata/hms_catalog/Dockerfile +++ b/crates/catalog/hms/testdata/hms_catalog/Dockerfile @@ -13,24 +13,18 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM --platform=$BUILDPLATFORM openjdk:8-jre-slim AS build - -ARG BUILDPLATFORM - -RUN apt-get update -qq && apt-get -qq -y install curl +FROM apache/hive:3.1.3 ENV AWSSDK_VERSION=2.20.18 ENV HADOOP_VERSION=3.1.0 -RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.271/aws-java-sdk-bundle-1.11.271.jar -Lo /tmp/aws-java-sdk-bundle-1.11.271.jar -RUN curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar -Lo /tmp/hadoop-aws-${HADOOP_VERSION}.jar +USER root +RUN apt-get update -qq && apt-get -qq -y install curl && \ + curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar -Lo /opt/hive/lib/hadoop-aws-${HADOOP_VERSION}.jar && \ + curl https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.271/aws-java-sdk-bundle-1.11.271.jar -Lo /opt/hive/lib/aws-java-sdk-bundle-1.11.271.jar && \ + apt-get clean && rm -rf /var/lib/apt/lists/* -FROM apache/hive:3.1.3 - -ENV AWSSDK_VERSION=2.20.18 -ENV HADOOP_VERSION=3.1.0 +COPY core-site.xml /opt/hadoop/etc/hadoop/core-site.xml -COPY --from=build /tmp/hadoop-aws-${HADOOP_VERSION}.jar /opt/hive/lib/hadoop-aws-${HADOOP_VERSION}.jar -COPY --from=build /tmp/aws-java-sdk-bundle-1.11.271.jar /opt/hive/lib/aws-java-sdk-bundle-1.11.271.jar -COPY core-site.xml /opt/hadoop/etc/hadoop/core-site.xml \ No newline at end of file +USER hive From b3519d5456e5dbba36b1b48f522175baf9718a47 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 4 Nov 2025 04:57:58 -0500 Subject: [PATCH 057/891] fix: StructType fails to deserialize JSON with type field (#1822) This fix enables proper interoperability with Iceberg implementations that include the type discriminator field in their JSON serialization (such as Apache Iceberg Java). ## Which issue does this PR close? Partially address #1749. ## What changes are included in this PR? Fixes a bug in the `StructType` deserializer that causes JSON deserialization to fail with the error `expected ',' or '}' at line 1 column 8`. **Root Cause:** The `StructType::Deserialize` implementation was not properly consuming the `"type"` field value when deserializing JSON like: ```json {"type":"struct","fields":[{"id":1,"name":"foo","required":true,"type":"string"}]} ``` In serde's visitor pattern, when you call `map.next_key()`, you must call `map.next_value()` to consume the corresponding value, even if you're discarding it. The deserializer was calling `next_key()` for the "type" field but not consuming its value, causing the parser to remain stuck at the value position and fail when encountering the next field. Changes: - Modified `StructTypeVisitor::visit_map` to properly consume the type field value - Added validation that the type field equals "struct" ## Are these changes tested? Yes, two new unit tests have been added to `crates/iceberg/src/spec/datatypes.rs`: 1. **`struct_type_with_type_field`** - Verifies that `StructType` successfully deserializes from JSON containing the `"type":"struct"` field. This test would fail before the fix with the error `expected ',' or '}' at line 1 column 8` 2. **`struct_type_rejects_wrong_type`** - Validates that the deserializer properly rejects JSON with incorrect type values (e.g., `"type":"list"`) --------- Co-authored-by: Renjie Liu --- crates/iceberg/src/spec/datatypes.rs | 55 +++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/spec/datatypes.rs b/crates/iceberg/src/spec/datatypes.rs index 19facfb017..456b754408 100644 --- a/crates/iceberg/src/spec/datatypes.rs +++ b/crates/iceberg/src/spec/datatypes.rs @@ -423,7 +423,15 @@ impl<'de> Deserialize<'de> for StructType { let mut fields = None; while let Some(key) = map.next_key()? { match key { - Field::Type => (), + Field::Type => { + let type_val: String = map.next_value()?; + if type_val != "struct" { + return Err(serde::de::Error::custom(format!( + "expected type 'struct', got '{}'", + type_val + ))); + } + } Field::Fields => { if fields.is_some() { return Err(serde::de::Error::duplicate_field("fields")); @@ -1208,4 +1216,49 @@ mod tests { assert!(ty.compatible(&literal)); } } + + #[test] + fn struct_type_with_type_field() { + // Test that StructType properly deserializes JSON with "type":"struct" field + // This was previously broken because the deserializer wasn't consuming the type field value + let json = r#" + { + "type": "struct", + "fields": [ + {"id": 1, "name": "field1", "required": true, "type": "string"} + ] + } + "#; + + let struct_type: StructType = serde_json::from_str(json) + .expect("Should successfully deserialize StructType with type field"); + + assert_eq!(struct_type.fields().len(), 1); + assert_eq!(struct_type.fields()[0].name, "field1"); + } + + #[test] + fn struct_type_rejects_wrong_type() { + // Test that StructType validation rejects incorrect type field values + let json = r#" + { + "type": "list", + "fields": [ + {"id": 1, "name": "field1", "required": true, "type": "string"} + ] + } + "#; + + let result = serde_json::from_str::(json); + assert!( + result.is_err(), + "Should reject StructType with wrong type field" + ); + assert!( + result + .unwrap_err() + .to_string() + .contains("expected type 'struct'") + ); + } } From b6dc4cfb8bd68c777a14e73afcbd4cf8a1a093e1 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 4 Nov 2025 17:35:46 +0100 Subject: [PATCH 058/891] feat: Support for V3 Metadata (#1682) ## Which issue does this PR close? Towards V3 Support! ## What changes are included in this PR? Introduce V3 FormatVersion accross Iceberg Metadata. ## Are these changes tested? Yes. Java has [a few more tests](https://github.com/apache/iceberg/blob/ee90c10e39cec0ccceb9425e03a3e0b5690daf3b/core/src/test/java/org/apache/iceberg/TestRowLineageMetadata.java#L36) regarding other table operations (different deletes, remove file), but we don't have those yet in our Transaction interface. --------- Signed-off-by: Xuanwo Co-authored-by: Fokko Driesprong Co-authored-by: Xuanwo --- crates/iceberg/src/catalog/memory/catalog.rs | 4 +- crates/iceberg/src/catalog/mod.rs | 219 +++++- .../src/expr/visitors/manifest_evaluator.rs | 1 + crates/iceberg/src/spec/encrypted_key.rs | 16 +- crates/iceberg/src/spec/manifest/data_file.rs | 6 +- crates/iceberg/src/spec/manifest/entry.rs | 38 + crates/iceberg/src/spec/manifest/mod.rs | 3 +- crates/iceberg/src/spec/manifest/writer.rs | 76 +- crates/iceberg/src/spec/manifest_list.rs | 671 +++++++++++++++++- crates/iceberg/src/spec/snapshot.rs | 138 +++- crates/iceberg/src/spec/snapshot_summary.rs | 2 + crates/iceberg/src/spec/table_metadata.rs | 541 +++++++++++++- .../src/spec/table_metadata_builder.rs | 453 +++++++++++- crates/iceberg/src/transaction/mod.rs | 122 +++- crates/iceberg/src/transaction/snapshot.rs | 36 +- .../writer/file_writer/location_generator.rs | 2 + .../TableMetadataV3ValidMinimal.json | 74 ++ 17 files changed, 2276 insertions(+), 126 deletions(-) create mode 100644 crates/iceberg/testdata/table_metadata/TableMetadataV3ValidMinimal.json diff --git a/crates/iceberg/src/catalog/memory/catalog.rs b/crates/iceberg/src/catalog/memory/catalog.rs index fdb495f6d3..cfa3dc6b52 100644 --- a/crates/iceberg/src/catalog/memory/catalog.rs +++ b/crates/iceberg/src/catalog/memory/catalog.rs @@ -377,7 +377,7 @@ impl Catalog for MemoryCatalog { } #[cfg(test)] -mod tests { +pub(crate) mod tests { use std::collections::HashSet; use std::hash::Hash; use std::iter::FromIterator; @@ -396,7 +396,7 @@ mod tests { temp_dir.path().to_str().unwrap().to_string() } - async fn new_memory_catalog() -> impl Catalog { + pub(crate) async fn new_memory_catalog() -> impl Catalog { let warehouse_location = temp_path(); MemoryCatalogBuilder::default() .load( diff --git a/crates/iceberg/src/catalog/mod.rs b/crates/iceberg/src/catalog/mod.rs index ec4b77fe01..27d5edaedb 100644 --- a/crates/iceberg/src/catalog/mod.rs +++ b/crates/iceberg/src/catalog/mod.rs @@ -28,7 +28,7 @@ use std::ops::Deref; use std::str::FromStr; use std::sync::Arc; -use _serde::deserialize_snapshot; +use _serde::{deserialize_snapshot, serialize_snapshot}; use async_trait::async_trait; pub use memory::MemoryCatalog; pub use metadata_location::*; @@ -39,9 +39,9 @@ use typed_builder::TypedBuilder; use uuid::Uuid; use crate::spec::{ - FormatVersion, PartitionStatisticsFile, Schema, SchemaId, Snapshot, SnapshotReference, - SortOrder, StatisticsFile, TableMetadata, TableMetadataBuilder, UnboundPartitionSpec, - ViewFormatVersion, ViewRepresentations, ViewVersion, + EncryptedKey, FormatVersion, PartitionStatisticsFile, Schema, SchemaId, Snapshot, + SnapshotReference, SortOrder, StatisticsFile, TableMetadata, TableMetadataBuilder, + UnboundPartitionSpec, ViewFormatVersion, ViewRepresentations, ViewVersion, }; use crate::table::Table; use crate::{Error, ErrorKind, Result}; @@ -291,6 +291,9 @@ pub struct TableCreation { props.into_iter().collect() }))] pub properties: HashMap, + /// Format version of the table. Defaults to V2. + #[builder(default = FormatVersion::V2)] + pub format_version: FormatVersion, } /// TableCommit represents the commit of a table in the catalog. @@ -479,7 +482,10 @@ pub enum TableUpdate { #[serde(rename_all = "kebab-case")] AddSnapshot { /// Snapshot to add. - #[serde(deserialize_with = "deserialize_snapshot")] + #[serde( + deserialize_with = "deserialize_snapshot", + serialize_with = "serialize_snapshot" + )] snapshot: Snapshot, }, /// Set table's snapshot ref. @@ -554,6 +560,18 @@ pub enum TableUpdate { /// Schema IDs to remove. schema_ids: Vec, }, + /// Add an encryption key + #[serde(rename_all = "kebab-case")] + AddEncryptionKey { + /// The encryption key to add. + encryption_key: EncryptedKey, + }, + /// Remove an encryption key + #[serde(rename_all = "kebab-case")] + RemoveEncryptionKey { + /// The id of the encryption key to remove. + key_id: String, + }, } impl TableUpdate { @@ -598,6 +616,12 @@ impl TableUpdate { Ok(builder.remove_partition_statistics(snapshot_id)) } TableUpdate::RemoveSchemas { schema_ids } => builder.remove_schemas(&schema_ids), + TableUpdate::AddEncryptionKey { encryption_key } => { + Ok(builder.add_encryption_key(encryption_key)) + } + TableUpdate::RemoveEncryptionKey { key_id } => { + Ok(builder.remove_encryption_key(&key_id)) + } } } } @@ -742,7 +766,7 @@ impl TableRequirement { } pub(super) mod _serde { - use serde::{Deserialize as _, Deserializer}; + use serde::{Deserialize as _, Deserializer, Serialize as _}; use super::*; use crate::spec::{SchemaId, Summary}; @@ -755,7 +779,18 @@ pub(super) mod _serde { Ok(buf.into()) } - #[derive(Debug, Deserialize, PartialEq, Eq)] + pub(super) fn serialize_snapshot( + snapshot: &Snapshot, + serializer: S, + ) -> std::result::Result + where + S: serde::Serializer, + { + let buf: CatalogSnapshot = snapshot.clone().into(); + buf.serialize(serializer) + } + + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] /// Defines the structure of a v2 snapshot for the catalog. /// Main difference to SnapshotV2 is that sequence-number is optional @@ -771,6 +806,12 @@ pub(super) mod _serde { summary: Summary, #[serde(skip_serializing_if = "Option::is_none")] schema_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + first_row_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + added_rows: Option, + #[serde(skip_serializing_if = "Option::is_none")] + key_id: Option, } impl From for Snapshot { @@ -783,6 +824,9 @@ pub(super) mod _serde { manifest_list, schema_id, summary, + first_row_id, + added_rows, + key_id, } = snapshot; let builder = Snapshot::builder() .with_snapshot_id(snapshot_id) @@ -790,11 +834,49 @@ pub(super) mod _serde { .with_sequence_number(sequence_number) .with_timestamp_ms(timestamp_ms) .with_manifest_list(manifest_list) - .with_summary(summary); - if let Some(schema_id) = schema_id { - builder.with_schema_id(schema_id).build() - } else { - builder.build() + .with_summary(summary) + .with_encryption_key_id(key_id); + let row_range = first_row_id.zip(added_rows); + match (schema_id, row_range) { + (None, None) => builder.build(), + (Some(schema_id), None) => builder.with_schema_id(schema_id).build(), + (None, Some((first_row_id, last_row_id))) => { + builder.with_row_range(first_row_id, last_row_id).build() + } + (Some(schema_id), Some((first_row_id, last_row_id))) => builder + .with_schema_id(schema_id) + .with_row_range(first_row_id, last_row_id) + .build(), + } + } + } + + impl From for CatalogSnapshot { + fn from(snapshot: Snapshot) -> Self { + let first_row_id = snapshot.first_row_id(); + let added_rows = snapshot.added_rows_count(); + let Snapshot { + snapshot_id, + parent_snapshot_id, + sequence_number, + timestamp_ms, + manifest_list, + summary, + schema_id, + row_range: _, + encryption_key_id: key_id, + } = snapshot; + CatalogSnapshot { + snapshot_id, + parent_snapshot_id, + sequence_number, + timestamp_ms, + manifest_list, + summary, + schema_id, + first_row_id, + added_rows, + key_id, } } } @@ -938,6 +1020,7 @@ mod tests { use std::fs::File; use std::io::BufReader; + use base64::Engine as _; use serde::Serialize; use serde::de::DeserializeOwned; use uuid::uuid; @@ -945,7 +1028,7 @@ mod tests { use super::ViewUpdate; use crate::io::FileIOBuilder; use crate::spec::{ - BlobMetadata, FormatVersion, MAIN_BRANCH, NestedField, NullOrder, Operation, + BlobMetadata, EncryptedKey, FormatVersion, MAIN_BRANCH, NestedField, NullOrder, Operation, PartitionStatisticsFile, PrimitiveType, Schema, Snapshot, SnapshotReference, SnapshotRetention, SortDirection, SortField, SortOrder, SqlViewRepresentation, StatisticsFile, Summary, TableMetadata, TableMetadataBuilder, Transform, Type, @@ -1075,20 +1158,18 @@ mod tests { assert!(requirement.check(Some(&metadata)).is_ok()); // Add snapshot - let record = r#" - { - "snapshot-id": 3051729675574597004, - "sequence-number": 10, - "timestamp-ms": 9992191116217, - "summary": { - "operation": "append" - }, - "manifest-list": "s3://b/wh/.../s1.avro", - "schema-id": 0 - } - "#; + let snapshot = Snapshot::builder() + .with_snapshot_id(3051729675574597004) + .with_sequence_number(10) + .with_timestamp_ms(9992191116217) + .with_manifest_list("s3://b/wh/.../s1.avro".to_string()) + .with_schema_id(0) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .build(); - let snapshot = serde_json::from_str::(record).unwrap(); let builder = metadata.into_builder(None); let builder = TableUpdate::AddSnapshot { snapshot: snapshot.clone(), @@ -1666,6 +1747,50 @@ mod tests { assert_eq!(actual, update, "Parsed value is not equal to expected"); } + #[test] + fn test_add_snapshot_v3() { + let json = serde_json::json!( + { + "action": "add-snapshot", + "snapshot": { + "snapshot-id": 3055729675574597000i64, + "parent-snapshot-id": 3051729675574597000i64, + "timestamp-ms": 1555100955770i64, + "first-row-id":0, + "added-rows":2, + "key-id":"key123", + "summary": { + "operation": "append" + }, + "manifest-list": "s3://a/b/2.avro" + } + }); + + let update = TableUpdate::AddSnapshot { + snapshot: Snapshot::builder() + .with_snapshot_id(3055729675574597000) + .with_parent_snapshot_id(Some(3051729675574597000)) + .with_timestamp_ms(1555100955770) + .with_sequence_number(0) + .with_manifest_list("s3://a/b/2.avro") + .with_row_range(0, 2) + .with_encryption_key_id(Some("key123".to_string())) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::default(), + }) + .build(), + }; + + let actual: TableUpdate = serde_json::from_value(json).expect("Failed to parse from json"); + assert_eq!(actual, update, "Parsed value is not equal to expected"); + let restored: TableUpdate = serde_json::from_str( + &serde_json::to_string(&actual).expect("Failed to serialize to json"), + ) + .expect("Failed to parse from serialized json"); + assert_eq!(restored, update); + } + #[test] fn test_remove_snapshots() { let json = r#" @@ -2169,6 +2294,48 @@ mod tests { ); } + #[test] + fn test_add_encryption_key() { + let key_bytes = "key".as_bytes(); + let encoded_key = base64::engine::general_purpose::STANDARD.encode(key_bytes); + test_serde_json( + format!( + r#" + {{ + "action": "add-encryption-key", + "encryption-key": {{ + "key-id": "a", + "encrypted-key-metadata": "{encoded_key}", + "encrypted-by-id": "b" + }} + }} + "# + ), + TableUpdate::AddEncryptionKey { + encryption_key: EncryptedKey::builder() + .key_id("a") + .encrypted_key_metadata(key_bytes.to_vec()) + .encrypted_by_id("b") + .build(), + }, + ); + } + + #[test] + fn test_remove_encryption_key() { + test_serde_json( + r#" + { + "action": "remove-encryption-key", + "key-id": "a" + } + "#, + TableUpdate::RemoveEncryptionKey { + key_id: "a".to_string(), + }, + ); + } + #[test] fn test_table_commit() { let table = { diff --git a/crates/iceberg/src/expr/visitors/manifest_evaluator.rs b/crates/iceberg/src/expr/visitors/manifest_evaluator.rs index 903c9ea4a7..abbd136cb1 100644 --- a/crates/iceberg/src/expr/visitors/manifest_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/manifest_evaluator.rs @@ -703,6 +703,7 @@ mod test { deleted_rows_count: None, partitions: Some(partitions), key_metadata: None, + first_row_id: None, } } diff --git a/crates/iceberg/src/spec/encrypted_key.rs b/crates/iceberg/src/spec/encrypted_key.rs index db19a023b2..6908ade1d5 100644 --- a/crates/iceberg/src/spec/encrypted_key.rs +++ b/crates/iceberg/src/spec/encrypted_key.rs @@ -26,16 +26,16 @@ use serde::{Deserialize, Serialize}; pub struct EncryptedKey { /// Unique identifier for the key #[builder(setter(into))] - key_id: String, + pub(crate) key_id: String, /// Encrypted key metadata as binary data #[builder(setter(into))] - encrypted_key_metadata: Vec, + pub(crate) encrypted_key_metadata: Vec, /// Identifier of the entity that encrypted this key - #[builder(setter(into))] - encrypted_by_id: String, + #[builder(default, setter(into, strip_option))] + pub(crate) encrypted_by_id: Option, /// Additional properties associated with the key #[builder(default)] - properties: HashMap, + pub(crate) properties: HashMap, } impl EncryptedKey { @@ -50,8 +50,8 @@ impl EncryptedKey { } /// Returns the ID of the entity that encrypted this key - pub fn encrypted_by_id(&self) -> &str { - &self.encrypted_by_id + pub fn encrypted_by_id(&self) -> Option<&str> { + self.encrypted_by_id.as_deref() } /// Returns the properties map @@ -72,7 +72,7 @@ pub(super) mod _serde { pub(super) struct EncryptedKeySerde { pub key_id: String, pub encrypted_key_metadata: String, // Base64 encoded - pub encrypted_by_id: String, + pub encrypted_by_id: Option, #[serde(default, skip_serializing_if = "HashMap::is_empty")] pub properties: HashMap, } diff --git a/crates/iceberg/src/spec/manifest/data_file.rs b/crates/iceberg/src/spec/manifest/data_file.rs index 6c63f62235..a9c041f540 100644 --- a/crates/iceberg/src/spec/manifest/data_file.rs +++ b/crates/iceberg/src/spec/manifest/data_file.rs @@ -24,7 +24,9 @@ use serde_derive::{Deserialize, Serialize}; use serde_with::{DeserializeFromStr, SerializeDisplay}; use super::_serde::DataFileSerde; -use super::{Datum, FormatVersion, Schema, data_file_schema_v1, data_file_schema_v2}; +use super::{ + Datum, FormatVersion, Schema, data_file_schema_v1, data_file_schema_v2, data_file_schema_v3, +}; use crate::error::Result; use crate::spec::{DEFAULT_PARTITION_SPEC_ID, Struct, StructType}; use crate::{Error, ErrorKind}; @@ -295,6 +297,7 @@ pub fn write_data_files_to_avro( let avro_schema = match version { FormatVersion::V1 => data_file_schema_v1(partition_type).unwrap(), FormatVersion::V2 => data_file_schema_v2(partition_type).unwrap(), + FormatVersion::V3 => data_file_schema_v3(partition_type).unwrap(), }; let mut writer = AvroWriter::new(&avro_schema, writer); @@ -322,6 +325,7 @@ pub fn read_data_files_from_avro( let avro_schema = match version { FormatVersion::V1 => data_file_schema_v1(partition_type).unwrap(), FormatVersion::V2 => data_file_schema_v2(partition_type).unwrap(), + FormatVersion::V3 => data_file_schema_v3(partition_type).unwrap(), }; let reader = AvroReader::with_schema(&avro_schema, reader)?; diff --git a/crates/iceberg/src/spec/manifest/entry.rs b/crates/iceberg/src/spec/manifest/entry.rs index d11d8acfe8..e8fe0f223a 100644 --- a/crates/iceberg/src/spec/manifest/entry.rs +++ b/crates/iceberg/src/spec/manifest/entry.rs @@ -509,6 +509,42 @@ static CONTENT_SIZE_IN_BYTES: Lazy = { }) }; +fn data_file_fields_v3(partition_type: &StructType) -> Vec { + vec![ + CONTENT.clone(), + FILE_PATH.clone(), + FILE_FORMAT.clone(), + Arc::new(NestedField::required( + 102, + "partition", + Type::Struct(partition_type.clone()), + )), + RECORD_COUNT.clone(), + FILE_SIZE_IN_BYTES.clone(), + COLUMN_SIZES.clone(), + VALUE_COUNTS.clone(), + NULL_VALUE_COUNTS.clone(), + NAN_VALUE_COUNTS.clone(), + LOWER_BOUNDS.clone(), + UPPER_BOUNDS.clone(), + KEY_METADATA.clone(), + SPLIT_OFFSETS.clone(), + EQUALITY_IDS.clone(), + SORT_ORDER_ID.clone(), + FIRST_ROW_ID.clone(), + REFERENCE_DATA_FILE.clone(), + CONTENT_OFFSET.clone(), + CONTENT_SIZE_IN_BYTES.clone(), + ] +} + +pub(super) fn data_file_schema_v3(partition_type: &StructType) -> Result { + let schema = Schema::builder() + .with_fields(data_file_fields_v3(partition_type)) + .build()?; + schema_to_avro_schema("data_file", &schema) +} + fn data_file_fields_v2(partition_type: &StructType) -> Vec { vec![ CONTENT.clone(), @@ -533,6 +569,8 @@ fn data_file_fields_v2(partition_type: &StructType) -> Vec { SORT_ORDER_ID.clone(), FIRST_ROW_ID.clone(), REFERENCE_DATA_FILE.clone(), + // Why are the following two fields here in the existing v2 schema? + // In the spec, they are not even listed as optional for v2. CONTENT_OFFSET.clone(), CONTENT_SIZE_IN_BYTES.clone(), ] diff --git a/crates/iceberg/src/spec/manifest/mod.rs b/crates/iceberg/src/spec/manifest/mod.rs index a1a5612c36..51219bfdb7 100644 --- a/crates/iceberg/src/spec/manifest/mod.rs +++ b/crates/iceberg/src/spec/manifest/mod.rs @@ -70,7 +70,8 @@ impl Manifest { }) .collect::>>()? } - FormatVersion::V2 => { + // Manifest Schema & Manifest Entry did not change between V2 and V3 + FormatVersion::V2 | FormatVersion::V3 => { let schema = manifest_schema_v2(&partition_type)?; let reader = AvroReader::with_schema(&schema, bs)?; reader diff --git a/crates/iceberg/src/spec/manifest/writer.rs b/crates/iceberg/src/spec/manifest/writer.rs index 673f8b5d2a..ebb0590bcf 100644 --- a/crates/iceberg/src/spec/manifest/writer.rs +++ b/crates/iceberg/src/spec/manifest/writer.rs @@ -72,7 +72,13 @@ impl ManifestWriterBuilder { .format_version(FormatVersion::V1) .content(ManifestContentType::Data) .build(); - ManifestWriter::new(self.output, self.snapshot_id, self.key_metadata, metadata) + ManifestWriter::new( + self.output, + self.snapshot_id, + self.key_metadata, + metadata, + None, + ) } /// Build a [`ManifestWriter`] for format version 2, data content. @@ -84,7 +90,13 @@ impl ManifestWriterBuilder { .format_version(FormatVersion::V2) .content(ManifestContentType::Data) .build(); - ManifestWriter::new(self.output, self.snapshot_id, self.key_metadata, metadata) + ManifestWriter::new( + self.output, + self.snapshot_id, + self.key_metadata, + metadata, + None, + ) } /// Build a [`ManifestWriter`] for format version 2, deletes content. @@ -96,7 +108,51 @@ impl ManifestWriterBuilder { .format_version(FormatVersion::V2) .content(ManifestContentType::Deletes) .build(); - ManifestWriter::new(self.output, self.snapshot_id, self.key_metadata, metadata) + ManifestWriter::new( + self.output, + self.snapshot_id, + self.key_metadata, + metadata, + None, + ) + } + + /// Build a [`ManifestWriter`] for format version 2, data content. + pub fn build_v3_data(self) -> ManifestWriter { + let metadata = ManifestMetadata::builder() + .schema_id(self.schema.schema_id()) + .schema(self.schema) + .partition_spec(self.partition_spec) + .format_version(FormatVersion::V3) + .content(ManifestContentType::Data) + .build(); + ManifestWriter::new( + self.output, + self.snapshot_id, + self.key_metadata, + metadata, + // First row id is assigned by the [`ManifestListWriter`] when the manifest + // is added to the list. + None, + ) + } + + /// Build a [`ManifestWriter`] for format version 3, deletes content. + pub fn build_v3_deletes(self) -> ManifestWriter { + let metadata = ManifestMetadata::builder() + .schema_id(self.schema.schema_id()) + .schema(self.schema) + .partition_spec(self.partition_spec) + .format_version(FormatVersion::V3) + .content(ManifestContentType::Deletes) + .build(); + ManifestWriter::new( + self.output, + self.snapshot_id, + self.key_metadata, + metadata, + None, + ) } } @@ -112,6 +168,7 @@ pub struct ManifestWriter { existing_rows: u64, deleted_files: u32, deleted_rows: u64, + first_row_id: Option, min_seq_num: Option, @@ -129,6 +186,7 @@ impl ManifestWriter { snapshot_id: Option, key_metadata: Option>, metadata: ManifestMetadata, + first_row_id: Option, ) -> Self { Self { output, @@ -139,6 +197,7 @@ impl ManifestWriter { existing_rows: 0, deleted_files: 0, deleted_rows: 0, + first_row_id, min_seq_num: None, key_metadata, manifest_entries: Vec::new(), @@ -348,7 +407,8 @@ impl ManifestWriter { let table_schema = &self.metadata.schema; let avro_schema = match self.metadata.format_version { FormatVersion::V1 => manifest_schema_v1(&partition_type)?, - FormatVersion::V2 => manifest_schema_v2(&partition_type)?, + // Manifest schema did not change between V2 and V3 + FormatVersion::V2 | FormatVersion::V3 => manifest_schema_v2(&partition_type)?, }; let mut avro_writer = AvroWriter::new(&avro_schema, Vec::new()); avro_writer.add_user_metadata( @@ -388,8 +448,11 @@ impl ManifestWriter { let value = match self.metadata.format_version { FormatVersion::V1 => to_value(ManifestEntryV1::try_from(entry, &partition_type)?)? .resolve(&avro_schema)?, - FormatVersion::V2 => to_value(ManifestEntryV2::try_from(entry, &partition_type)?)? - .resolve(&avro_schema)?, + // Manifest entry format did not change between V2 and V3 + FormatVersion::V2 | FormatVersion::V3 => { + to_value(ManifestEntryV2::try_from(entry, &partition_type)?)? + .resolve(&avro_schema)? + } }; avro_writer.append(value)?; @@ -417,6 +480,7 @@ impl ManifestWriter { deleted_rows_count: Some(self.deleted_rows), partitions: Some(partition_summary), key_metadata: self.key_metadata, + first_row_id: self.first_row_id, }) } } diff --git a/crates/iceberg/src/spec/manifest_list.rs b/crates/iceberg/src/spec/manifest_list.rs index 16409ffe56..5e97e5466e 100644 --- a/crates/iceberg/src/spec/manifest_list.rs +++ b/crates/iceberg/src/spec/manifest_list.rs @@ -31,6 +31,8 @@ use self::_serde::{ManifestFileV1, ManifestFileV2}; use super::{FormatVersion, Manifest}; use crate::error::Result; use crate::io::{FileIO, OutputFile}; +use crate::spec::manifest_list::_const_schema::MANIFEST_LIST_AVRO_SCHEMA_V3; +use crate::spec::manifest_list::_serde::ManifestFileV3; use crate::{Error, ErrorKind}; /// Placeholder for sequence number. The field with this value must be replaced with the actual sequence number before it write. @@ -69,6 +71,11 @@ impl ManifestList { let values = Value::Array(reader.collect::, _>>()?); from_value::<_serde::ManifestListV2>(&values)?.try_into() } + FormatVersion::V3 => { + let reader = Reader::new(bs)?; + let values = Value::Array(reader.collect::, _>>()?); + from_value::<_serde::ManifestListV3>(&values)?.try_into() + } } } @@ -90,6 +97,7 @@ pub struct ManifestListWriter { avro_writer: Writer<'static, Vec>, sequence_number: i64, snapshot_id: i64, + next_row_id: Option, } impl std::fmt::Debug for ManifestListWriter { @@ -103,6 +111,11 @@ impl std::fmt::Debug for ManifestListWriter { } impl ManifestListWriter { + /// Get the next row ID that will be assigned to the next data manifest added. + pub fn next_row_id(&self) -> Option { + self.next_row_id + } + /// Construct a v1 [`ManifestListWriter`] that writes to a provided [`OutputFile`]. pub fn v1(output_file: OutputFile, snapshot_id: i64, parent_snapshot_id: Option) -> Self { let mut metadata = HashMap::from_iter([ @@ -115,7 +128,14 @@ impl ManifestListWriter { parent_snapshot_id.to_string(), ); } - Self::new(FormatVersion::V1, output_file, metadata, 0, snapshot_id) + Self::new( + FormatVersion::V1, + output_file, + metadata, + 0, + snapshot_id, + None, + ) } /// Construct a v2 [`ManifestListWriter`] that writes to a provided [`OutputFile`]. @@ -142,6 +162,42 @@ impl ManifestListWriter { metadata, sequence_number, snapshot_id, + None, + ) + } + + /// Construct a v3 [`ManifestListWriter`] that writes to a provided [`OutputFile`]. + pub fn v3( + output_file: OutputFile, + snapshot_id: i64, + parent_snapshot_id: Option, + sequence_number: i64, + first_row_id: Option, // Always None for delete manifests + ) -> Self { + let mut metadata = HashMap::from_iter([ + ("snapshot-id".to_string(), snapshot_id.to_string()), + ("sequence-number".to_string(), sequence_number.to_string()), + ("format-version".to_string(), "3".to_string()), + ]); + metadata.insert( + "parent-snapshot-id".to_string(), + parent_snapshot_id + .map(|v| v.to_string()) + .unwrap_or("null".to_string()), + ); + metadata.insert( + "first-row-id".to_string(), + first_row_id + .map(|v| v.to_string()) + .unwrap_or("null".to_string()), + ); + Self::new( + FormatVersion::V3, + output_file, + metadata, + sequence_number, + snapshot_id, + first_row_id, ) } @@ -151,10 +207,12 @@ impl ManifestListWriter { metadata: HashMap, sequence_number: i64, snapshot_id: i64, + first_row_id: Option, ) -> Self { let avro_schema = match format_version { FormatVersion::V1 => &MANIFEST_LIST_AVRO_SCHEMA_V1, FormatVersion::V2 => &MANIFEST_LIST_AVRO_SCHEMA_V2, + FormatVersion::V3 => &MANIFEST_LIST_AVRO_SCHEMA_V3, }; let mut avro_writer = Writer::new(avro_schema, Vec::new()); for (key, value) in metadata { @@ -168,46 +226,35 @@ impl ManifestListWriter { avro_writer, sequence_number, snapshot_id, + next_row_id: first_row_id, } } /// Append manifests to be written. + /// + /// If V3 Manifests are added and the `first_row_id` of any data manifest is unassigned, + /// it will be assigned based on the `next_row_id` of the writer, and the `next_row_id` of the writer will be updated accordingly. + /// If `first_row_id` is already assigned, it will be validated against the `next_row_id` of the writer. pub fn add_manifests(&mut self, manifests: impl Iterator) -> Result<()> { match self.format_version { FormatVersion::V1 => { for manifest in manifests { - let manifes: ManifestFileV1 = manifest.try_into()?; - self.avro_writer.append_ser(manifes)?; + let manifests: ManifestFileV1 = manifest.try_into()?; + self.avro_writer.append_ser(manifests)?; } } - FormatVersion::V2 => { + FormatVersion::V2 | FormatVersion::V3 => { for mut manifest in manifests { - if manifest.sequence_number == UNASSIGNED_SEQUENCE_NUMBER { - if manifest.added_snapshot_id != self.snapshot_id { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Found unassigned sequence number for a manifest from snapshot {}.", - manifest.added_snapshot_id - ), - )); - } - manifest.sequence_number = self.sequence_number; + self.assign_sequence_numbers(&mut manifest)?; + + if self.format_version == FormatVersion::V2 { + let manifest_entry: ManifestFileV2 = manifest.try_into()?; + self.avro_writer.append_ser(manifest_entry)?; + } else if self.format_version == FormatVersion::V3 { + self.assign_first_row_id(&mut manifest)?; + let manifest_entry: ManifestFileV3 = manifest.try_into()?; + self.avro_writer.append_ser(manifest_entry)?; } - if manifest.min_sequence_number == UNASSIGNED_SEQUENCE_NUMBER { - if manifest.added_snapshot_id != self.snapshot_id { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Found unassigned sequence number for a manifest from snapshot {}.", - manifest.added_snapshot_id - ), - )); - } - manifest.min_sequence_number = self.sequence_number; - } - let manifest_entry: ManifestFileV2 = manifest.try_into()?; - self.avro_writer.append_ser(manifest_entry)?; } } } @@ -222,6 +269,112 @@ impl ManifestListWriter { writer.close().await?; Ok(()) } + + /// Assign sequence numbers to manifest if they are unassigned + fn assign_sequence_numbers(&self, manifest: &mut ManifestFile) -> Result<()> { + if manifest.sequence_number == UNASSIGNED_SEQUENCE_NUMBER { + if manifest.added_snapshot_id != self.snapshot_id { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Found unassigned sequence number for a manifest from snapshot {}.", + manifest.added_snapshot_id + ), + )); + } + manifest.sequence_number = self.sequence_number; + } + + if manifest.min_sequence_number == UNASSIGNED_SEQUENCE_NUMBER { + if manifest.added_snapshot_id != self.snapshot_id { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Found unassigned sequence number for a manifest from snapshot {}.", + manifest.added_snapshot_id + ), + )); + } + manifest.min_sequence_number = self.sequence_number; + } + + Ok(()) + } + + /// Returns number of newly assigned first-row-ids, if any. + fn assign_first_row_id(&mut self, manifest: &mut ManifestFile) -> Result<()> { + match manifest.content { + ManifestContentType::Data => { + match (self.next_row_id, manifest.first_row_id) { + (Some(_), Some(_)) => { + // Case: Manifest with already assigned first row ID. + // No need to increase next_row_id, as this manifest is already assigned. + } + (None, Some(manifest_first_row_id)) => { + // Case: Assigned first row ID for data manifest, but the writer does not have a next-row-id assigned. + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "Found invalid first-row-id assignment for Manifest {}. Writer does not have a next-row-id assigned, but the manifest has first-row-id assigned to {}.", + manifest.manifest_path, manifest_first_row_id, + ), + )); + } + (Some(writer_next_row_id), None) => { + // Case: Unassigned first row ID for data manifest. This is either a new + // manifest, or a manifest from a pre-v3 snapshot. We need to assign one. + let (existing_rows_count, added_rows_count) = + require_row_counts_in_manifest(manifest)?; + manifest.first_row_id = Some(writer_next_row_id); + + self.next_row_id = writer_next_row_id + .checked_add(existing_rows_count) + .and_then(|sum| sum.checked_add(added_rows_count)) + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Row ID overflow when computing next row ID for Manifest {}. Next Row ID: {writer_next_row_id}, Existing Rows Count: {existing_rows_count}, Added Rows Count: {added_rows_count}", + manifest.manifest_path + ), + ) + }).map(Some)?; + } + (None, None) => { + // Case: Table without row lineage. No action needed. + } + } + } + ManifestContentType::Deletes => { + // Deletes never have a first-row-id assigned. + manifest.first_row_id = None; + } + }; + + Ok(()) + } +} + +fn require_row_counts_in_manifest(manifest: &ManifestFile) -> Result<(u64, u64)> { + let existing_rows_count = manifest.existing_rows_count.ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Cannot include a Manifest without existing-rows-count to a table with row lineage enabled. Manifest path: {}", + manifest.manifest_path, + ), + ) + })?; + let added_rows_count = manifest.added_rows_count.ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Cannot include a Manifest without added-rows-count to a table with row lineage enabled. Manifest path: {}", + manifest.manifest_path, + ), + ) + })?; + Ok((existing_rows_count, added_rows_count)) } /// This is a helper module that defines the schema field of the manifest list entry. @@ -453,6 +606,15 @@ mod _const_schema { )) }) }; + static FIRST_ROW_ID: Lazy = { + Lazy::new(|| { + Arc::new(NestedField::optional( + 520, + "first_row_id", + Type::Primitive(PrimitiveType::Long), + )) + }) + }; static V1_SCHEMA: Lazy = { Lazy::new(|| { @@ -497,11 +659,38 @@ mod _const_schema { }) }; + static V3_SCHEMA: Lazy = { + Lazy::new(|| { + let fields = vec![ + MANIFEST_PATH.clone(), + MANIFEST_LENGTH.clone(), + PARTITION_SPEC_ID.clone(), + CONTENT.clone(), + SEQUENCE_NUMBER.clone(), + MIN_SEQUENCE_NUMBER.clone(), + ADDED_SNAPSHOT_ID.clone(), + ADDED_FILES_COUNT_V2.clone(), + EXISTING_FILES_COUNT_V2.clone(), + DELETED_FILES_COUNT_V2.clone(), + ADDED_ROWS_COUNT_V2.clone(), + EXISTING_ROWS_COUNT_V2.clone(), + DELETED_ROWS_COUNT_V2.clone(), + PARTITIONS.clone(), + KEY_METADATA.clone(), + FIRST_ROW_ID.clone(), + ]; + Schema::builder().with_fields(fields).build().unwrap() + }) + }; + pub(super) static MANIFEST_LIST_AVRO_SCHEMA_V1: Lazy = Lazy::new(|| schema_to_avro_schema("manifest_file", &V1_SCHEMA).unwrap()); pub(super) static MANIFEST_LIST_AVRO_SCHEMA_V2: Lazy = Lazy::new(|| schema_to_avro_schema("manifest_file", &V2_SCHEMA).unwrap()); + + pub(super) static MANIFEST_LIST_AVRO_SCHEMA_V3: Lazy = + Lazy::new(|| schema_to_avro_schema("manifest_file", &V3_SCHEMA).unwrap()); } /// Entry in a manifest list. @@ -580,6 +769,10 @@ pub struct ManifestFile { /// /// Implementation-specific key metadata for encryption pub key_metadata: Option>, + /// field 520 + /// + /// The starting _row_id to assign to rows added by ADDED data files + pub first_row_id: Option, } impl ManifestFile { @@ -703,6 +896,12 @@ pub(super) mod _serde { use crate::error::Result; use crate::spec::FieldSummary; + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] + #[serde(transparent)] + pub(crate) struct ManifestListV3 { + entries: Vec, + } + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(transparent)] pub(crate) struct ManifestListV2 { @@ -715,6 +914,33 @@ pub(super) mod _serde { entries: Vec, } + impl ManifestListV3 { + /// Converts the [ManifestListV3] into a [ManifestList]. + pub fn try_into(self) -> Result { + Ok(super::ManifestList { + entries: self + .entries + .into_iter() + .map(|v| v.try_into()) + .collect::>>()?, + }) + } + } + + impl TryFrom for ManifestListV3 { + type Error = Error; + + fn try_from(value: super::ManifestList) -> std::result::Result { + Ok(Self { + entries: value + .entries + .into_iter() + .map(|v| v.try_into()) + .collect::, _>>()?, + }) + } + } + impl ManifestListV2 { /// Converts the [ManifestListV2] into a [ManifestList]. pub fn try_into(self) -> Result { @@ -813,6 +1039,58 @@ pub(super) mod _serde { pub key_metadata: Option, } + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] + pub(super) struct ManifestFileV3 { + pub manifest_path: String, + pub manifest_length: i64, + pub partition_spec_id: i32, + #[serde(default = "v2_default_content_for_v1")] + pub content: i32, + #[serde(default = "v2_default_sequence_number_for_v1")] + pub sequence_number: i64, + #[serde(default = "v2_default_min_sequence_number_for_v1")] + pub min_sequence_number: i64, + pub added_snapshot_id: i64, + #[serde(alias = "added_data_files_count", alias = "added_files_count")] + pub added_files_count: i32, + #[serde(alias = "existing_data_files_count", alias = "existing_files_count")] + pub existing_files_count: i32, + #[serde(alias = "deleted_data_files_count", alias = "deleted_files_count")] + pub deleted_files_count: i32, + pub added_rows_count: i64, + pub existing_rows_count: i64, + pub deleted_rows_count: i64, + pub partitions: Option>, + pub key_metadata: Option, + pub first_row_id: Option, + } + + impl ManifestFileV3 { + /// Converts the [ManifestFileV3] into a [ManifestFile]. + pub fn try_into(self) -> Result { + let manifest_file = ManifestFile { + manifest_path: self.manifest_path, + manifest_length: self.manifest_length, + partition_spec_id: self.partition_spec_id, + content: self.content.try_into()?, + sequence_number: self.sequence_number, + min_sequence_number: self.min_sequence_number, + added_snapshot_id: self.added_snapshot_id, + added_files_count: Some(self.added_files_count.try_into()?), + existing_files_count: Some(self.existing_files_count.try_into()?), + deleted_files_count: Some(self.deleted_files_count.try_into()?), + added_rows_count: Some(self.added_rows_count.try_into()?), + existing_rows_count: Some(self.existing_rows_count.try_into()?), + deleted_rows_count: Some(self.deleted_rows_count.try_into()?), + partitions: self.partitions, + key_metadata: self.key_metadata.map(|b| b.into_vec()), + first_row_id: self.first_row_id, + }; + + Ok(manifest_file) + } + } + impl ManifestFileV2 { /// Converts the [ManifestFileV2] into a [ManifestFile]. pub fn try_into(self) -> Result { @@ -832,6 +1110,7 @@ pub(super) mod _serde { deleted_rows_count: Some(self.deleted_rows_count.try_into()?), partitions: self.partitions, key_metadata: self.key_metadata.map(|b| b.into_vec()), + first_row_id: None, }) } } @@ -881,6 +1160,7 @@ pub(super) mod _serde { content: super::ManifestContentType::Data, sequence_number: 0, min_sequence_number: 0, + first_row_id: None, }) } } @@ -892,6 +1172,80 @@ pub(super) mod _serde { } } + impl TryFrom for ManifestFileV3 { + type Error = Error; + + fn try_from(value: ManifestFile) -> std::result::Result { + let key_metadata = convert_to_serde_key_metadata(value.key_metadata); + Ok(Self { + manifest_path: value.manifest_path, + manifest_length: value.manifest_length, + partition_spec_id: value.partition_spec_id, + content: value.content as i32, + sequence_number: value.sequence_number, + min_sequence_number: value.min_sequence_number, + added_snapshot_id: value.added_snapshot_id, + added_files_count: value + .added_files_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "added_data_files_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + existing_files_count: value + .existing_files_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "existing_data_files_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + deleted_files_count: value + .deleted_files_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "deleted_data_files_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + added_rows_count: value + .added_rows_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "added_rows_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + existing_rows_count: value + .existing_rows_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "existing_rows_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + deleted_rows_count: value + .deleted_rows_count + .ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "deleted_rows_count in ManifestFileV3 is required", + ) + })? + .try_into()?, + partitions: value.partitions, + key_metadata, + first_row_id: value.first_row_id, + }) + } + } + impl TryFrom for ManifestFileV2 { type Error = Error; @@ -1012,7 +1366,7 @@ mod test { use super::_serde::ManifestListV2; use crate::io::FileIOBuilder; - use crate::spec::manifest_list::_serde::ManifestListV1; + use crate::spec::manifest_list::_serde::{ManifestListV1, ManifestListV3}; use crate::spec::{ Datum, FieldSummary, ManifestContentType, ManifestFile, ManifestList, ManifestListWriter, UNASSIGNED_SEQUENCE_NUMBER, @@ -1038,6 +1392,7 @@ mod test { deleted_rows_count: Some(0), partitions: Some(vec![]), key_metadata: None, + first_row_id: None, } ] }; @@ -1089,6 +1444,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] ), key_metadata: None, + first_row_id: None, }, ManifestFile { manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m1.avro".to_string(), @@ -1108,6 +1464,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::float(1.1).to_bytes().unwrap()), upper_bound: Some(Datum::float(2.1).to_bytes().unwrap())}] ), key_metadata: None, + first_row_id: None, } ] }; @@ -1138,6 +1495,80 @@ mod test { assert_eq!(manifest_list, parsed_manifest_list); } + #[tokio::test] + async fn test_parse_manifest_list_v3() { + let manifest_list = ManifestList { + entries: vec![ + ManifestFile { + manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m0.avro".to_string(), + manifest_length: 6926, + partition_spec_id: 1, + content: ManifestContentType::Data, + sequence_number: 1, + min_sequence_number: 1, + added_snapshot_id: 377075049360453639, + added_files_count: Some(1), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: Some(10), + }, + ManifestFile { + manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m1.avro".to_string(), + manifest_length: 6926, + partition_spec_id: 2, + content: ManifestContentType::Data, + sequence_number: 1, + min_sequence_number: 1, + added_snapshot_id: 377075049360453639, + added_files_count: Some(1), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::float(1.1).to_bytes().unwrap()), upper_bound: Some(Datum::float(2.1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: Some(13), + } + ] + }; + + let file_io = FileIOBuilder::new_fs_io().build().unwrap(); + + let tmp_dir = TempDir::new().unwrap(); + let file_name = "simple_manifest_list_v3.avro"; + let full_path = format!("{}/{}", tmp_dir.path().to_str().unwrap(), file_name); + + let mut writer = ManifestListWriter::v3( + file_io.new_output(full_path.clone()).unwrap(), + 377075049360453639, + Some(377075049360453639), + 1, + Some(10), + ); + + writer + .add_manifests(manifest_list.entries.clone().into_iter()) + .unwrap(); + writer.close().await.unwrap(); + + let bs = fs::read(full_path).expect("read_file must succeed"); + + let parsed_manifest_list = + ManifestList::parse_with_version(&bs, crate::spec::FormatVersion::V3).unwrap(); + + assert_eq!(manifest_list, parsed_manifest_list); + } + #[test] fn test_serialize_manifest_list_v1() { let manifest_list:ManifestListV1 = ManifestList { @@ -1157,6 +1588,7 @@ mod test { deleted_rows_count: Some(0), partitions: None, key_metadata: None, + first_row_id: None, }] }.try_into().unwrap(); let result = serde_json::to_string(&manifest_list).unwrap(); @@ -1187,6 +1619,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] ), key_metadata: None, + first_row_id: None, }] }.try_into().unwrap(); let result = serde_json::to_string(&manifest_list).unwrap(); @@ -1196,6 +1629,37 @@ mod test { ); } + #[test] + fn test_serialize_manifest_list_v3() { + let manifest_list: ManifestListV3 = ManifestList { + entries: vec![ManifestFile { + manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m0.avro".to_string(), + manifest_length: 6926, + partition_spec_id: 1, + content: ManifestContentType::Data, + sequence_number: 1, + min_sequence_number: 1, + added_snapshot_id: 377075049360453639, + added_files_count: Some(1), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: Some(10), + }] + }.try_into().unwrap(); + let result = serde_json::to_string(&manifest_list).unwrap(); + assert_eq!( + result, + r#"[{"manifest_path":"s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m0.avro","manifest_length":6926,"partition_spec_id":1,"content":0,"sequence_number":1,"min_sequence_number":1,"added_snapshot_id":377075049360453639,"added_files_count":1,"existing_files_count":0,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":0,"deleted_rows_count":0,"partitions":[{"contains_null":false,"contains_nan":false,"lower_bound":[1,0,0,0,0,0,0,0],"upper_bound":[1,0,0,0,0,0,0,0]}],"key_metadata":null,"first_row_id":10}]"# + ); + } + #[tokio::test] async fn test_manifest_list_writer_v1() { let expected_manifest_list = ManifestList { @@ -1217,6 +1681,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}], ), key_metadata: None, + first_row_id: None, }] }; @@ -1263,6 +1728,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] ), key_metadata: None, + first_row_id: None, }] }; @@ -1287,6 +1753,56 @@ mod test { temp_dir.close().unwrap(); } + #[tokio::test] + async fn test_manifest_list_writer_v3() { + let snapshot_id = 377075049360453639; + let seq_num = 1; + let mut expected_manifest_list = ManifestList { + entries: vec![ManifestFile { + manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m0.avro".to_string(), + manifest_length: 6926, + partition_spec_id: 1, + content: ManifestContentType::Data, + sequence_number: UNASSIGNED_SEQUENCE_NUMBER, + min_sequence_number: UNASSIGNED_SEQUENCE_NUMBER, + added_snapshot_id: snapshot_id, + added_files_count: Some(1), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: Some(10), + }] + }; + + let temp_dir = TempDir::new().unwrap(); + let path = temp_dir.path().join("manifest_list_v2.avro"); + let io = FileIOBuilder::new_fs_io().build().unwrap(); + let output_file = io.new_output(path.to_str().unwrap()).unwrap(); + + let mut writer = + ManifestListWriter::v3(output_file, snapshot_id, Some(0), seq_num, Some(10)); + writer + .add_manifests(expected_manifest_list.entries.clone().into_iter()) + .unwrap(); + writer.close().await.unwrap(); + + let bs = fs::read(path).unwrap(); + let manifest_list = + ManifestList::parse_with_version(&bs, crate::spec::FormatVersion::V3).unwrap(); + expected_manifest_list.entries[0].sequence_number = seq_num; + expected_manifest_list.entries[0].min_sequence_number = seq_num; + expected_manifest_list.entries[0].first_row_id = Some(10); + assert_eq!(manifest_list, expected_manifest_list); + + temp_dir.close().unwrap(); + } + #[tokio::test] async fn test_manifest_list_writer_v1_as_v2() { let expected_manifest_list = ManifestList { @@ -1308,6 +1824,7 @@ mod test { vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] ), key_metadata: None, + first_row_id: None, }] }; @@ -1331,6 +1848,100 @@ mod test { temp_dir.close().unwrap(); } + #[tokio::test] + async fn test_manifest_list_writer_v1_as_v3() { + let expected_manifest_list = ManifestList { + entries: vec![ManifestFile { + manifest_path: "/opt/bitnami/spark/warehouse/db/table/metadata/10d28031-9739-484c-92db-cdf2975cead4-m0.avro".to_string(), + manifest_length: 5806, + partition_spec_id: 1, + content: ManifestContentType::Data, + sequence_number: 0, + min_sequence_number: 0, + added_snapshot_id: 1646658105718557341, + added_files_count: Some(3), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: None, + }] + }; + + let temp_dir = TempDir::new().unwrap(); + let path = temp_dir.path().join("manifest_list_v1.avro"); + let io = FileIOBuilder::new_fs_io().build().unwrap(); + let output_file = io.new_output(path.to_str().unwrap()).unwrap(); + + let mut writer = ManifestListWriter::v1(output_file, 1646658105718557341, Some(0)); + writer + .add_manifests(expected_manifest_list.entries.clone().into_iter()) + .unwrap(); + writer.close().await.unwrap(); + + let bs = fs::read(path).unwrap(); + + let manifest_list = + ManifestList::parse_with_version(&bs, crate::spec::FormatVersion::V3).unwrap(); + assert_eq!(manifest_list, expected_manifest_list); + + temp_dir.close().unwrap(); + } + + #[tokio::test] + async fn test_manifest_list_writer_v2_as_v3() { + let snapshot_id = 377075049360453639; + let seq_num = 1; + let mut expected_manifest_list = ManifestList { + entries: vec![ManifestFile { + manifest_path: "s3a://icebergdata/demo/s1/t1/metadata/05ffe08b-810f-49b3-a8f4-e88fc99b254a-m0.avro".to_string(), + manifest_length: 6926, + partition_spec_id: 1, + content: ManifestContentType::Data, + sequence_number: UNASSIGNED_SEQUENCE_NUMBER, + min_sequence_number: UNASSIGNED_SEQUENCE_NUMBER, + added_snapshot_id: snapshot_id, + added_files_count: Some(1), + existing_files_count: Some(0), + deleted_files_count: Some(0), + added_rows_count: Some(3), + existing_rows_count: Some(0), + deleted_rows_count: Some(0), + partitions: Some( + vec![FieldSummary { contains_null: false, contains_nan: Some(false), lower_bound: Some(Datum::long(1).to_bytes().unwrap()), upper_bound: Some(Datum::long(1).to_bytes().unwrap())}] + ), + key_metadata: None, + first_row_id: None, + }] + }; + + let temp_dir = TempDir::new().unwrap(); + let path = temp_dir.path().join("manifest_list_v2.avro"); + let io = FileIOBuilder::new_fs_io().build().unwrap(); + let output_file = io.new_output(path.to_str().unwrap()).unwrap(); + + let mut writer = ManifestListWriter::v2(output_file, snapshot_id, Some(0), seq_num); + writer + .add_manifests(expected_manifest_list.entries.clone().into_iter()) + .unwrap(); + writer.close().await.unwrap(); + + let bs = fs::read(path).unwrap(); + + let manifest_list = + ManifestList::parse_with_version(&bs, crate::spec::FormatVersion::V3).unwrap(); + expected_manifest_list.entries[0].sequence_number = seq_num; + expected_manifest_list.entries[0].min_sequence_number = seq_num; + assert_eq!(manifest_list, expected_manifest_list); + + temp_dir.close().unwrap(); + } + #[tokio::test] async fn test_manifest_list_v2_deserializer_aliases() { // reading avro manifest file generated by iceberg 1.4.0 diff --git a/crates/iceberg/src/spec/snapshot.rs b/crates/iceberg/src/spec/snapshot.rs index 809bf0998a..5371cf68f2 100644 --- a/crates/iceberg/src/spec/snapshot.rs +++ b/crates/iceberg/src/spec/snapshot.rs @@ -21,7 +21,6 @@ use std::collections::HashMap; use std::sync::Arc; -use _serde::SnapshotV2; use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; use typed_builder::TypedBuilder; @@ -82,33 +81,52 @@ impl Default for Operation { } } -#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize, TypedBuilder)] -#[serde(from = "SnapshotV2", into = "SnapshotV2")] +#[derive(Debug, PartialEq, Eq, Clone)] +/// Row range of a snapshot, contains first_row_id and added_rows_count. +pub struct SnapshotRowRange { + /// The first _row_id assigned to the first row in the first data file in the first manifest. + pub first_row_id: u64, + /// The upper bound of the number of rows with assigned row IDs + pub added_rows: u64, +} + +#[derive(Debug, PartialEq, Eq, Clone, TypedBuilder)] #[builder(field_defaults(setter(prefix = "with_")))] /// A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table. pub struct Snapshot { /// A unique long ID - snapshot_id: i64, + pub(crate) snapshot_id: i64, /// The snapshot ID of the snapshot’s parent. /// Omitted for any snapshot with no parent #[builder(default = None)] - parent_snapshot_id: Option, + pub(crate) parent_snapshot_id: Option, /// A monotonically increasing long that tracks the order of /// changes to a table. - sequence_number: i64, + pub(crate) sequence_number: i64, /// A timestamp when the snapshot was created, used for garbage /// collection and table inspection - timestamp_ms: i64, + pub(crate) timestamp_ms: i64, /// The location of a manifest list for this snapshot that /// tracks manifest files with additional metadata. /// Currently we only support manifest list file, and manifest files are not supported. #[builder(setter(into))] - manifest_list: String, + pub(crate) manifest_list: String, /// A string map that summarizes the snapshot changes, including operation. - summary: Summary, + pub(crate) summary: Summary, /// ID of the table’s current schema when the snapshot was created. #[builder(setter(strip_option(fallback = schema_id_opt)), default = None)] - schema_id: Option, + pub(crate) schema_id: Option, + /// Encryption Key ID + #[builder(default)] + pub(crate) encryption_key_id: Option, + /// Row range of this snapshot, required when the table version supports row lineage. + /// Specify as a tuple of (first_row_id, added_rows_count) + #[builder(default, setter(!strip_option, transform = |first_row_id: u64, added_rows: u64| Some(SnapshotRowRange { first_row_id, added_rows })))] + // This is specified as a struct instead of two separate fields to ensure that both fields are either set or not set. + // The java implementations uses two separate fields, then sets `added_row_counts` to Null if `first_row_id` is set to Null. + // It throws an error if `added_row_counts` is set but `first_row_id` is not set, or if either of the two is negative. + // We handle all cases infallible using the rust type system. + pub(crate) row_range: Option, } impl Snapshot { @@ -205,6 +223,37 @@ impl Snapshot { snapshot_id: self.snapshot_id, } } + + /// The row-id of the first newly added row in this snapshot. All rows added in this snapshot will + /// have a row-id assigned to them greater than this value. All rows with a row-id less than this + /// value were created in a snapshot that was added to the table (but not necessarily committed to + /// this branch) in the past. + /// + /// This field is optional but is required when the table version supports row lineage. + pub fn first_row_id(&self) -> Option { + self.row_range.as_ref().map(|r| r.first_row_id) + } + + /// The total number of newly added rows in this snapshot. It should be the summation of {@link + /// ManifestFile#ADDED_ROWS_COUNT} for every manifest added in this snapshot. + /// + /// This field is optional but is required when the table version supports row lineage. + pub fn added_rows_count(&self) -> Option { + self.row_range.as_ref().map(|r| r.added_rows) + } + + /// Returns the row range of this snapshot, if available. + /// This is a tuple containing (first_row_id, added_rows_count). + pub fn row_range(&self) -> Option<(u64, u64)> { + self.row_range + .as_ref() + .map(|r| (r.first_row_id, r.added_rows)) + } + + /// Get encryption key id, if available. + pub fn encryption_key_id(&self) -> Option<&str> { + self.encryption_key_id.as_deref() + } } pub(super) mod _serde { @@ -219,6 +268,26 @@ pub(super) mod _serde { use super::{Operation, Snapshot, Summary}; use crate::Error; use crate::spec::SchemaId; + use crate::spec::snapshot::SnapshotRowRange; + + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "kebab-case")] + /// Defines the structure of a v2 snapshot for serialization/deserialization + pub(crate) struct SnapshotV3 { + pub snapshot_id: i64, + #[serde(skip_serializing_if = "Option::is_none")] + pub parent_snapshot_id: Option, + pub sequence_number: i64, + pub timestamp_ms: i64, + pub manifest_list: String, + pub summary: Summary, + #[serde(skip_serializing_if = "Option::is_none")] + pub schema_id: Option, + pub first_row_id: u64, + pub added_rows: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub key_id: Option, + } #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] @@ -253,6 +322,51 @@ pub(super) mod _serde { pub schema_id: Option, } + impl From for Snapshot { + fn from(s: SnapshotV3) -> Self { + Snapshot { + snapshot_id: s.snapshot_id, + parent_snapshot_id: s.parent_snapshot_id, + sequence_number: s.sequence_number, + timestamp_ms: s.timestamp_ms, + manifest_list: s.manifest_list, + summary: s.summary, + schema_id: s.schema_id, + encryption_key_id: s.key_id, + row_range: Some(SnapshotRowRange { + first_row_id: s.first_row_id, + added_rows: s.added_rows, + }), + } + } + } + + impl TryFrom for SnapshotV3 { + type Error = Error; + + fn try_from(s: Snapshot) -> Result { + let row_range = s.row_range.ok_or_else(|| { + Error::new( + crate::ErrorKind::DataInvalid, + "v3 Snapshots must have first-row-id and rows-added fields set.".to_string(), + ) + })?; + + Ok(SnapshotV3 { + snapshot_id: s.snapshot_id, + parent_snapshot_id: s.parent_snapshot_id, + sequence_number: s.sequence_number, + timestamp_ms: s.timestamp_ms, + manifest_list: s.manifest_list, + summary: s.summary, + schema_id: s.schema_id, + first_row_id: row_range.first_row_id, + added_rows: row_range.added_rows, + key_id: s.encryption_key_id, + }) + } + } + impl From for Snapshot { fn from(v2: SnapshotV2) -> Self { Snapshot { @@ -263,6 +377,8 @@ pub(super) mod _serde { manifest_list: v2.manifest_list, summary: v2.summary, schema_id: v2.schema_id, + encryption_key_id: None, + row_range: None, } } } @@ -300,6 +416,8 @@ pub(super) mod _serde { additional_properties: HashMap::new(), }), schema_id: v1.schema_id, + encryption_key_id: None, + row_range: None, }) } } diff --git a/crates/iceberg/src/spec/snapshot_summary.rs b/crates/iceberg/src/spec/snapshot_summary.rs index a9dd569985..4cd3715e06 100644 --- a/crates/iceberg/src/spec/snapshot_summary.rs +++ b/crates/iceberg/src/spec/snapshot_summary.rs @@ -850,6 +850,7 @@ mod tests { deleted_rows_count: Some(50), partitions: Some(Vec::new()), key_metadata: None, + first_row_id: None, }; collector @@ -974,6 +975,7 @@ mod tests { deleted_rows_count: Some(0), partitions: Some(Vec::new()), key_metadata: None, + first_row_id: None, }); summary_four.add_file( diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index a98dc4f437..437b0df542 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -38,6 +38,7 @@ use super::{ }; use crate::error::{Result, timestamp_ms_to_utc}; use crate::io::FileIO; +use crate::spec::EncryptedKey; use crate::{Error, ErrorKind}; static MAIN_BRANCH: &str = "main"; @@ -46,6 +47,10 @@ pub(crate) static ONE_MINUTE_MS: i64 = 60_000; pub(crate) static EMPTY_SNAPSHOT_ID: i64 = -1; pub(crate) static INITIAL_SEQUENCE_NUMBER: i64 = 0; +/// Initial row id for row lineage for new v3 tables and older tables upgrading to v3. +pub const INITIAL_ROW_ID: u64 = 0; +/// Minimum format version that supports row lineage (v3). +pub const MIN_FORMAT_VERSION_ROW_LINEAGE: FormatVersion = FormatVersion::V3; /// Reference to [`TableMetadata`]. pub type TableMetadataRef = Arc; @@ -123,8 +128,10 @@ pub struct TableMetadata { pub(crate) statistics: HashMap, /// Mapping of snapshot ids to partition statistics files. pub(crate) partition_statistics: HashMap, - /// Encryption Keys - pub(crate) encryption_keys: HashMap, + /// Encryption Keys - map of key id to the actual key + pub(crate) encryption_keys: HashMap, + /// Next row id to be assigned for Row Lineage (v3) + pub(crate) next_row_id: u64, } impl TableMetadata { @@ -398,16 +405,22 @@ impl TableMetadata { /// Iterate over all encryption keys #[inline] - pub fn encryption_keys_iter(&self) -> impl ExactSizeIterator { - self.encryption_keys.iter() + pub fn encryption_keys_iter(&self) -> impl ExactSizeIterator { + self.encryption_keys.values() } /// Get the encryption key for a given key id #[inline] - pub fn encryption_key(&self, key_id: &str) -> Option<&String> { + pub fn encryption_key(&self, key_id: &str) -> Option<&EncryptedKey> { self.encryption_keys.get(key_id) } + /// Get the next row id to be assigned + #[inline] + pub fn next_row_id(&self) -> u64 { + self.next_row_id + } + /// Read table metadata from the given location. pub async fn read_from( file_io: &FileIO, @@ -673,16 +686,18 @@ pub(super) mod _serde { TableMetadata, }; use crate::spec::schema::_serde::{SchemaV1, SchemaV2}; - use crate::spec::snapshot::_serde::{SnapshotV1, SnapshotV2}; + use crate::spec::snapshot::_serde::{SnapshotV1, SnapshotV2, SnapshotV3}; use crate::spec::{ - PartitionField, PartitionSpec, PartitionSpecRef, PartitionStatisticsFile, Schema, - SchemaRef, Snapshot, SnapshotReference, SnapshotRetention, SortOrder, StatisticsFile, + EncryptedKey, INITIAL_ROW_ID, PartitionField, PartitionSpec, PartitionSpecRef, + PartitionStatisticsFile, Schema, SchemaRef, Snapshot, SnapshotReference, SnapshotRetention, + SortOrder, StatisticsFile, }; use crate::{Error, ErrorKind}; #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(untagged)] pub(super) enum TableMetadataEnum { + V3(TableMetadataV3), V2(TableMetadataV2), V1(TableMetadataV1), } @@ -690,8 +705,21 @@ pub(super) mod _serde { #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] /// Defines the structure of a v2 table metadata for serialization/deserialization - pub(super) struct TableMetadataV2 { - pub format_version: VersionNumber<2>, + pub(super) struct TableMetadataV3 { + pub format_version: VersionNumber<3>, + #[serde(flatten)] + pub shared: TableMetadataV2V3Shared, + pub next_row_id: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub encryption_keys: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + pub snapshots: Option>, + } + + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "kebab-case")] + /// Defines the structure of a v2 table metadata for serialization/deserialization + pub(super) struct TableMetadataV2V3Shared { pub table_uuid: Uuid, pub location: String, pub last_sequence_number: i64, @@ -707,8 +735,6 @@ pub(super) mod _serde { #[serde(skip_serializing_if = "Option::is_none")] pub current_snapshot_id: Option, #[serde(skip_serializing_if = "Option::is_none")] - pub snapshots: Option>, - #[serde(skip_serializing_if = "Option::is_none")] pub snapshot_log: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub metadata_log: Option>, @@ -722,6 +748,17 @@ pub(super) mod _serde { pub partition_statistics: Vec, } + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "kebab-case")] + /// Defines the structure of a v2 table metadata for serialization/deserialization + pub(super) struct TableMetadataV2 { + pub format_version: VersionNumber<2>, + #[serde(flatten)] + pub shared: TableMetadataV2V3Shared, + #[serde(skip_serializing_if = "Option::is_none")] + pub snapshots: Option>, + } + #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] /// Defines the structure of a v1 table metadata for serialization/deserialization @@ -802,6 +839,7 @@ pub(super) mod _serde { type Error = Error; fn try_from(value: TableMetadataEnum) -> Result { match value { + TableMetadataEnum::V3(value) => value.try_into(), TableMetadataEnum::V2(value) => value.try_into(), TableMetadataEnum::V1(value) => value.try_into(), } @@ -812,15 +850,136 @@ pub(super) mod _serde { type Error = Error; fn try_from(value: TableMetadata) -> Result { Ok(match value.format_version { + FormatVersion::V3 => TableMetadataEnum::V3(value.try_into()?), FormatVersion::V2 => TableMetadataEnum::V2(value.into()), FormatVersion::V1 => TableMetadataEnum::V1(value.try_into()?), }) } } + impl TryFrom for TableMetadata { + type Error = Error; + fn try_from(value: TableMetadataV3) -> Result { + let TableMetadataV3 { + format_version: _, + shared: value, + next_row_id, + encryption_keys, + snapshots, + } = value; + let current_snapshot_id = if let &Some(-1) = &value.current_snapshot_id { + None + } else { + value.current_snapshot_id + }; + let schemas = HashMap::from_iter( + value + .schemas + .into_iter() + .map(|schema| Ok((schema.schema_id, Arc::new(schema.try_into()?)))) + .collect::, Error>>()?, + ); + + let current_schema: &SchemaRef = + schemas.get(&value.current_schema_id).ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!( + "No schema exists with the current schema id {}.", + value.current_schema_id + ), + ) + })?; + let partition_specs = HashMap::from_iter( + value + .partition_specs + .into_iter() + .map(|x| (x.spec_id(), Arc::new(x))), + ); + let default_spec_id = value.default_spec_id; + let default_spec: PartitionSpecRef = partition_specs + .get(&value.default_spec_id) + .map(|spec| (**spec).clone()) + .or_else(|| { + (DEFAULT_PARTITION_SPEC_ID == default_spec_id) + .then(PartitionSpec::unpartition_spec) + }) + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!("Default partition spec {default_spec_id} not found"), + ) + })? + .into(); + let default_partition_type = default_spec.partition_type(current_schema)?; + + let mut metadata = TableMetadata { + format_version: FormatVersion::V3, + table_uuid: value.table_uuid, + location: value.location, + last_sequence_number: value.last_sequence_number, + last_updated_ms: value.last_updated_ms, + last_column_id: value.last_column_id, + current_schema_id: value.current_schema_id, + schemas, + partition_specs, + default_partition_type, + default_spec, + last_partition_id: value.last_partition_id, + properties: value.properties.unwrap_or_default(), + current_snapshot_id, + snapshots: snapshots + .map(|snapshots| { + HashMap::from_iter( + snapshots + .into_iter() + .map(|x| (x.snapshot_id, Arc::new(x.into()))), + ) + }) + .unwrap_or_default(), + snapshot_log: value.snapshot_log.unwrap_or_default(), + metadata_log: value.metadata_log.unwrap_or_default(), + sort_orders: HashMap::from_iter( + value + .sort_orders + .into_iter() + .map(|x| (x.order_id, Arc::new(x))), + ), + default_sort_order_id: value.default_sort_order_id, + refs: value.refs.unwrap_or_else(|| { + if let Some(snapshot_id) = current_snapshot_id { + HashMap::from_iter(vec![(MAIN_BRANCH.to_string(), SnapshotReference { + snapshot_id, + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: None, + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + })]) + } else { + HashMap::new() + } + }), + statistics: index_statistics(value.statistics), + partition_statistics: index_partition_statistics(value.partition_statistics), + encryption_keys: encryption_keys + .map(|keys| { + HashMap::from_iter(keys.into_iter().map(|key| (key.key_id.clone(), key))) + }) + .unwrap_or_default(), + next_row_id, + }; + + metadata.borrow_mut().try_normalize()?; + Ok(metadata) + } + } + impl TryFrom for TableMetadata { type Error = Error; fn try_from(value: TableMetadataV2) -> Result { + let snapshots = value.snapshots; + let value = value.shared; let current_snapshot_id = if let &Some(-1) = &value.current_snapshot_id { None } else { @@ -882,8 +1041,7 @@ pub(super) mod _serde { last_partition_id: value.last_partition_id, properties: value.properties.unwrap_or_default(), current_snapshot_id, - snapshots: value - .snapshots + snapshots: snapshots .map(|snapshots| { HashMap::from_iter( snapshots @@ -918,6 +1076,7 @@ pub(super) mod _serde { statistics: index_statistics(value.statistics), partition_statistics: index_partition_statistics(value.partition_statistics), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; metadata.borrow_mut().try_normalize()?; @@ -1072,6 +1231,7 @@ pub(super) mod _serde { statistics: index_statistics(value.statistics), partition_statistics: index_partition_statistics(value.partition_statistics), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, // v1 has no row lineage }; metadata.borrow_mut().try_normalize()?; @@ -1079,10 +1239,63 @@ pub(super) mod _serde { } } + impl TryFrom for TableMetadataV3 { + type Error = Error; + + fn try_from(mut v: TableMetadata) -> Result { + let next_row_id = v.next_row_id; + let encryption_keys = std::mem::take(&mut v.encryption_keys); + let snapshots = std::mem::take(&mut v.snapshots); + let shared = v.into(); + + Ok(TableMetadataV3 { + format_version: VersionNumber::<3>, + shared, + next_row_id, + encryption_keys: if encryption_keys.is_empty() { + None + } else { + Some(encryption_keys.into_values().collect()) + }, + snapshots: if snapshots.is_empty() { + None + } else { + Some( + snapshots + .into_values() + .map(|s| SnapshotV3::try_from(Arc::unwrap_or_clone(s))) + .collect::>()?, + ) + }, + }) + } + } + impl From for TableMetadataV2 { - fn from(v: TableMetadata) -> Self { + fn from(mut v: TableMetadata) -> Self { + let snapshots = std::mem::take(&mut v.snapshots); + let shared = v.into(); + TableMetadataV2 { format_version: VersionNumber::<2>, + shared, + snapshots: if snapshots.is_empty() { + None + } else { + Some( + snapshots + .into_values() + .map(|s| SnapshotV2::from(Arc::unwrap_or_clone(s))) + .collect(), + ) + }, + } + } + } + + impl From for TableMetadataV2V3Shared { + fn from(v: TableMetadata) -> Self { + TableMetadataV2V3Shared { table_uuid: v.table_uuid, location: v.location, last_sequence_number: v.last_sequence_number, @@ -1111,20 +1324,6 @@ pub(super) mod _serde { Some(v.properties) }, current_snapshot_id: v.current_snapshot_id, - snapshots: if v.snapshots.is_empty() { - None - } else { - Some( - v.snapshots - .into_values() - .map(|x| { - Arc::try_unwrap(x) - .unwrap_or_else(|snapshot| snapshot.as_ref().clone()) - .into() - }) - .collect(), - ) - }, snapshot_log: if v.snapshot_log.is_empty() { None } else { @@ -1254,6 +1453,8 @@ pub enum FormatVersion { V1 = 1u8, /// Iceberg spec version 2 V2 = 2u8, + /// Iceberg spec version 3 + V3 = 3u8, } impl PartialOrd for FormatVersion { @@ -1273,6 +1474,7 @@ impl Display for FormatVersion { match self { FormatVersion::V1 => write!(f, "v1"), FormatVersion::V2 => write!(f, "v2"), + FormatVersion::V3 => write!(f, "v3"), } } } @@ -1317,6 +1519,7 @@ mod tests { use std::sync::Arc; use anyhow::Result; + use base64::Engine as _; use pretty_assertions::assert_eq; use tempfile::TempDir; use uuid::Uuid; @@ -1326,9 +1529,10 @@ mod tests { use crate::io::FileIOBuilder; use crate::spec::table_metadata::TableMetadata; use crate::spec::{ - BlobMetadata, NestedField, NullOrder, Operation, PartitionSpec, PartitionStatisticsFile, - PrimitiveType, Schema, Snapshot, SnapshotReference, SnapshotRetention, SortDirection, - SortField, SortOrder, StatisticsFile, Summary, Transform, Type, UnboundPartitionField, + BlobMetadata, EncryptedKey, INITIAL_ROW_ID, Literal, NestedField, NullOrder, Operation, + PartitionSpec, PartitionStatisticsFile, PrimitiveLiteral, PrimitiveType, Schema, Snapshot, + SnapshotReference, SnapshotRetention, SortDirection, SortField, SortOrder, StatisticsFile, + Summary, Transform, Type, UnboundPartitionField, }; fn check_table_metadata_serde(json: &str, expected_type: TableMetadata) { @@ -1474,6 +1678,183 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, + }; + + let expected_json_value = serde_json::to_value(&expected).unwrap(); + check_table_metadata_serde(data, expected); + + let json_value = serde_json::from_str::(data).unwrap(); + assert_eq!(json_value, expected_json_value); + } + + #[test] + fn test_table_data_v3() { + let data = r#" + { + "format-version" : 3, + "table-uuid": "fb072c92-a02b-11e9-ae9c-1bb7bc9eca94", + "location": "s3://b/wh/data.db/table", + "last-sequence-number" : 1, + "last-updated-ms": 1515100955770, + "last-column-id": 1, + "next-row-id": 5, + "schemas": [ + { + "schema-id" : 1, + "type" : "struct", + "fields" :[ + { + "id": 4, + "name": "ts", + "required": true, + "type": "timestamp" + } + ] + } + ], + "current-schema-id" : 1, + "partition-specs": [ + { + "spec-id": 0, + "fields": [ + { + "source-id": 4, + "field-id": 1000, + "name": "ts_day", + "transform": "day" + } + ] + } + ], + "default-spec-id": 0, + "last-partition-id": 1000, + "properties": { + "commit.retry.num-retries": "1" + }, + "metadata-log": [ + { + "metadata-file": "s3://bucket/.../v1.json", + "timestamp-ms": 1515100 + } + ], + "refs": {}, + "snapshots" : [ { + "snapshot-id" : 1, + "timestamp-ms" : 1662532818843, + "sequence-number" : 0, + "first-row-id" : 0, + "added-rows" : 4, + "key-id" : "key1", + "summary" : { + "operation" : "append" + }, + "manifest-list" : "/home/iceberg/warehouse/nyc/taxis/metadata/snap-638933773299822130-1-7e6760f0-4f6c-4b23-b907-0a5a174e3863.avro", + "schema-id" : 0 + } + ], + "encryption-keys": [ + { + "key-id": "key1", + "encrypted-by-id": "KMS", + "encrypted-key-metadata": "c29tZS1lbmNyeXB0aW9uLWtleQ==", + "properties": { + "p1": "v1" + } + } + ], + "sort-orders": [ + { + "order-id": 0, + "fields": [] + } + ], + "default-sort-order-id": 0 + } + "#; + + let schema = Schema::builder() + .with_schema_id(1) + .with_fields(vec![Arc::new(NestedField::required( + 4, + "ts", + Type::Primitive(PrimitiveType::Timestamp), + ))]) + .build() + .unwrap(); + + let partition_spec = PartitionSpec::builder(schema.clone()) + .with_spec_id(0) + .add_unbound_field(UnboundPartitionField { + name: "ts_day".to_string(), + transform: Transform::Day, + source_id: 4, + field_id: Some(1000), + }) + .unwrap() + .build() + .unwrap(); + + let snapshot = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(1662532818843) + .with_sequence_number(0) + .with_row_range(0, 4) + .with_encryption_key_id(Some("key1".to_string())) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_manifest_list("/home/iceberg/warehouse/nyc/taxis/metadata/snap-638933773299822130-1-7e6760f0-4f6c-4b23-b907-0a5a174e3863.avro".to_string()) + .with_schema_id(0) + .build(); + + let encryption_key = EncryptedKey::builder() + .key_id("key1".to_string()) + .encrypted_by_id("KMS".to_string()) + .encrypted_key_metadata( + base64::prelude::BASE64_STANDARD + .decode("c29tZS1lbmNyeXB0aW9uLWtleQ==") + .unwrap(), + ) + .properties(HashMap::from_iter(vec![( + "p1".to_string(), + "v1".to_string(), + )])) + .build(); + + let default_partition_type = partition_spec.partition_type(&schema).unwrap(); + let expected = TableMetadata { + format_version: FormatVersion::V3, + table_uuid: Uuid::parse_str("fb072c92-a02b-11e9-ae9c-1bb7bc9eca94").unwrap(), + location: "s3://b/wh/data.db/table".to_string(), + last_updated_ms: 1515100955770, + last_column_id: 1, + schemas: HashMap::from_iter(vec![(1, Arc::new(schema))]), + current_schema_id: 1, + partition_specs: HashMap::from_iter(vec![(0, partition_spec.clone().into())]), + default_partition_type, + default_spec: partition_spec.into(), + last_partition_id: 1000, + default_sort_order_id: 0, + sort_orders: HashMap::from_iter(vec![(0, SortOrder::unsorted_order().into())]), + snapshots: HashMap::from_iter(vec![(1, snapshot.into())]), + current_snapshot_id: None, + last_sequence_number: 1, + properties: HashMap::from_iter(vec![( + "commit.retry.num-retries".to_string(), + "1".to_string(), + )]), + snapshot_log: Vec::new(), + metadata_log: vec![MetadataLog { + metadata_file: "s3://bucket/.../v1.json".to_string(), + timestamp_ms: 1515100, + }], + refs: HashMap::new(), + statistics: HashMap::new(), + partition_statistics: HashMap::new(), + encryption_keys: HashMap::from_iter(vec![("key1".to_string(), encryption_key)]), + next_row_id: 5, }; let expected_json_value = serde_json::to_value(&expected).unwrap(); @@ -1650,6 +2031,7 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(data, expected); @@ -1748,6 +2130,7 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; let expected_json_value = serde_json::to_value(&expected).unwrap(); @@ -2282,6 +2665,7 @@ mod tests { }, })]), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(data, expected); @@ -2417,6 +2801,7 @@ mod tests { }, })]), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(data, expected); @@ -2445,6 +2830,95 @@ mod tests { Ok(()) } + #[test] + fn test_table_metadata_v3_valid_minimal() { + let metadata_str = + fs::read_to_string("testdata/table_metadata/TableMetadataV3ValidMinimal.json").unwrap(); + + let table_metadata = serde_json::from_str::(&metadata_str).unwrap(); + assert_eq!(table_metadata.format_version, FormatVersion::V3); + + let schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + Arc::new( + NestedField::required(1, "x", Type::Primitive(PrimitiveType::Long)) + .with_initial_default(Literal::Primitive(PrimitiveLiteral::Long(1))) + .with_write_default(Literal::Primitive(PrimitiveLiteral::Long(1))), + ), + Arc::new( + NestedField::required(2, "y", Type::Primitive(PrimitiveType::Long)) + .with_doc("comment"), + ), + Arc::new(NestedField::required( + 3, + "z", + Type::Primitive(PrimitiveType::Long), + )), + ]) + .build() + .unwrap(); + + let partition_spec = PartitionSpec::builder(schema.clone()) + .with_spec_id(0) + .add_unbound_field(UnboundPartitionField { + name: "x".to_string(), + transform: Transform::Identity, + source_id: 1, + field_id: Some(1000), + }) + .unwrap() + .build() + .unwrap(); + + let sort_order = SortOrder::builder() + .with_order_id(3) + .with_sort_field(SortField { + source_id: 2, + transform: Transform::Identity, + direction: SortDirection::Ascending, + null_order: NullOrder::First, + }) + .with_sort_field(SortField { + source_id: 3, + transform: Transform::Bucket(4), + direction: SortDirection::Descending, + null_order: NullOrder::Last, + }) + .build_unbound() + .unwrap(); + + let default_partition_type = partition_spec.partition_type(&schema).unwrap(); + let expected = TableMetadata { + format_version: FormatVersion::V3, + table_uuid: Uuid::parse_str("9c12d441-03fe-4693-9a96-a0705ddf69c1").unwrap(), + location: "s3://bucket/test/location".to_string(), + last_updated_ms: 1602638573590, + last_column_id: 3, + schemas: HashMap::from_iter(vec![(0, Arc::new(schema))]), + current_schema_id: 0, + partition_specs: HashMap::from_iter(vec![(0, partition_spec.clone().into())]), + default_spec: Arc::new(partition_spec), + default_partition_type, + last_partition_id: 1000, + default_sort_order_id: 3, + sort_orders: HashMap::from_iter(vec![(3, sort_order.into())]), + snapshots: HashMap::default(), + current_snapshot_id: None, + last_sequence_number: 34, + properties: HashMap::new(), + snapshot_log: Vec::new(), + metadata_log: Vec::new(), + refs: HashMap::new(), + statistics: HashMap::new(), + partition_statistics: HashMap::new(), + encryption_keys: HashMap::new(), + next_row_id: 0, // V3 specific field from the JSON + }; + + check_table_metadata_serde(&metadata_str, expected); + } + #[test] fn test_table_metadata_v2_file_valid() { let metadata = @@ -2579,6 +3053,7 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(&metadata, expected); @@ -2664,6 +3139,7 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(&metadata, expected); @@ -2733,6 +3209,7 @@ mod tests { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }; check_table_metadata_serde(&metadata, expected); diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 25af8c30e5..6b8ce1e6a5 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -28,6 +28,7 @@ use super::{ UnboundPartitionSpec, }; use crate::error::{Error, ErrorKind, Result}; +use crate::spec::{EncryptedKey, INITIAL_ROW_ID, MIN_FORMAT_VERSION_ROW_LINEAGE}; use crate::{TableCreation, TableUpdate}; const FIRST_FIELD_ID: u32 = 1; @@ -120,6 +121,7 @@ impl TableMetadataBuilder { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: INITIAL_ROW_ID, }, last_updated_ms: None, changes: vec![], @@ -170,6 +172,7 @@ impl TableMetadataBuilder { partition_spec, sort_order, properties, + format_version, } = table_creation; let location = location.ok_or_else(|| { @@ -188,7 +191,7 @@ impl TableMetadataBuilder { partition_spec, sort_order.unwrap_or(SortOrder::unsorted_order()), location, - FormatVersion::V2, + format_version, properties, ) } @@ -228,6 +231,11 @@ impl TableMetadataBuilder { self.changes .push(TableUpdate::UpgradeFormatVersion { format_version }); } + FormatVersion::V3 => { + self.metadata.format_version = format_version; + self.changes + .push(TableUpdate::UpgradeFormatVersion { format_version }); + } } } @@ -329,6 +337,9 @@ impl TableMetadataBuilder { /// # Errors /// - Snapshot id already exists. /// - For format version > 1: the sequence number of the snapshot is lower than the highest sequence number specified so far. + /// - For format version >= 3: the first-row-id of the snapshot is lower than the next-row-id of the table. + /// - For format version >= 3: added-rows is null or first-row-id is null. + /// - For format version >= 3: next-row-id would overflow when adding added-rows. pub fn add_snapshot(mut self, snapshot: Snapshot) -> Result { if self .metadata @@ -385,6 +396,43 @@ impl TableMetadataBuilder { )); } + let mut added_rows = None; + if self.metadata.format_version >= MIN_FORMAT_VERSION_ROW_LINEAGE { + if let Some((first_row_id, added_rows_count)) = snapshot.row_range() { + if first_row_id < self.metadata.next_row_id { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Cannot add a snapshot, first-row-id is behind table next-row-id: {first_row_id} < {}", + self.metadata.next_row_id + ), + )); + } + + added_rows = Some(added_rows_count); + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Cannot add a snapshot: first-row-id is null. first-row-id must be set for format version >= {MIN_FORMAT_VERSION_ROW_LINEAGE}", + ), + )); + } + } + + if let Some(added_rows) = added_rows { + self.metadata.next_row_id = self + .metadata + .next_row_id + .checked_add(added_rows) + .ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + "Cannot add snapshot: next-row-id overflowed when adding added-rows", + ) + })?; + } + // Mutation happens in next line - must be infallible from here self.changes.push(TableUpdate::AddSnapshot { snapshot: snapshot.clone(), @@ -1014,6 +1062,31 @@ impl TableMetadataBuilder { .set_default_sort_order(Self::LAST_ADDED as i64) } + /// Add an encryption key to the table metadata. + pub fn add_encryption_key(mut self, key: EncryptedKey) -> Self { + let key_id = key.key_id().to_string(); + if self.metadata.encryption_keys.contains_key(&key_id) { + // already exists + return self; + } + + self.metadata.encryption_keys.insert(key_id, key.clone()); + self.changes.push(TableUpdate::AddEncryptionKey { + encryption_key: key, + }); + self + } + + /// Remove an encryption key from the table metadata. + pub fn remove_encryption_key(mut self, key_id: &str) -> Self { + if self.metadata.encryption_keys.remove(key_id).is_some() { + self.changes.push(TableUpdate::RemoveEncryptionKey { + key_id: key_id.to_string(), + }); + } + self + } + /// Build the table metadata. pub fn build(mut self) -> Result { self.metadata.last_updated_ms = self @@ -2988,4 +3061,382 @@ mod tests { assert!(result.is_ok()); } + + #[test] + fn test_row_lineage_addition() { + let new_rows = 30; + let base = builder_without_changes(FormatVersion::V3) + .build() + .unwrap() + .metadata; + let add_rows = Snapshot::builder() + .with_snapshot_id(0) + .with_timestamp_ms(base.last_updated_ms + 1) + .with_sequence_number(0) + .with_schema_id(0) + .with_manifest_list("foo") + .with_parent_snapshot_id(None) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(base.next_row_id(), new_rows) + .build(); + + let first_addition = base + .into_builder(None) + .add_snapshot(add_rows.clone()) + .unwrap() + .build() + .unwrap() + .metadata; + + assert_eq!(first_addition.next_row_id(), new_rows); + + let add_more_rows = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(first_addition.last_updated_ms + 1) + .with_sequence_number(1) + .with_schema_id(0) + .with_manifest_list("foo") + .with_parent_snapshot_id(Some(0)) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(first_addition.next_row_id(), new_rows) + .build(); + + let second_addition = first_addition + .into_builder(None) + .add_snapshot(add_more_rows) + .unwrap() + .build() + .unwrap() + .metadata; + assert_eq!(second_addition.next_row_id(), new_rows * 2); + } + + #[test] + fn test_row_lineage_invalid_snapshot() { + let new_rows = 30; + let base = builder_without_changes(FormatVersion::V3) + .build() + .unwrap() + .metadata; + + // add rows to check TableMetadata validation; Snapshot rejects negative next-row-id + let add_rows = Snapshot::builder() + .with_snapshot_id(0) + .with_timestamp_ms(base.last_updated_ms + 1) + .with_sequence_number(0) + .with_schema_id(0) + .with_manifest_list("foo") + .with_parent_snapshot_id(None) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(base.next_row_id(), new_rows) + .build(); + + let added = base + .into_builder(None) + .add_snapshot(add_rows) + .unwrap() + .build() + .unwrap() + .metadata; + + let invalid_new_rows = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(added.last_updated_ms + 1) + .with_sequence_number(1) + .with_schema_id(0) + .with_manifest_list("foo") + .with_parent_snapshot_id(Some(0)) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + // first_row_id is behind table next_row_id + .with_row_range(added.next_row_id() - 1, 10) + .build(); + + let err = added + .into_builder(None) + .add_snapshot(invalid_new_rows) + .unwrap_err(); + assert!( + err.to_string().contains( + "Cannot add a snapshot, first-row-id is behind table next-row-id: 29 < 30" + ) + ); + } + + #[test] + fn test_row_lineage_append_branch() { + // Appends to a branch should still change last-row-id even if not on main, these changes + // should also affect commits to main + + let branch = "some_branch"; + + // Start with V3 metadata to support row lineage + let base = builder_without_changes(FormatVersion::V3) + .build() + .unwrap() + .metadata; + + // Initial next_row_id should be 0 + assert_eq!(base.next_row_id(), 0); + + // Write to Branch - append 30 rows + let branch_snapshot_1 = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(base.last_updated_ms + 1) + .with_sequence_number(0) + .with_schema_id(0) + .with_manifest_list("foo") + .with_parent_snapshot_id(None) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(base.next_row_id(), 30) + .build(); + + let table_after_branch_1 = base + .into_builder(None) + .set_branch_snapshot(branch_snapshot_1.clone(), branch) + .unwrap() + .build() + .unwrap() + .metadata; + + // Current snapshot should be null (no main branch snapshot yet) + assert!(table_after_branch_1.current_snapshot().is_none()); + + // Branch snapshot should have first_row_id = 0 + let branch_ref = table_after_branch_1.refs.get(branch).unwrap(); + let branch_snap_1 = table_after_branch_1 + .snapshots + .get(&branch_ref.snapshot_id) + .unwrap(); + assert_eq!(branch_snap_1.first_row_id(), Some(0)); + + // Next row id should be 30 + assert_eq!(table_after_branch_1.next_row_id(), 30); + + // Write to Main - append 28 rows + let main_snapshot = Snapshot::builder() + .with_snapshot_id(2) + .with_timestamp_ms(table_after_branch_1.last_updated_ms + 1) + .with_sequence_number(1) + .with_schema_id(0) + .with_manifest_list("bar") + .with_parent_snapshot_id(None) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(table_after_branch_1.next_row_id(), 28) + .build(); + + let table_after_main = table_after_branch_1 + .into_builder(None) + .add_snapshot(main_snapshot.clone()) + .unwrap() + .set_ref(MAIN_BRANCH, SnapshotReference { + snapshot_id: main_snapshot.snapshot_id(), + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: None, + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + }) + .unwrap() + .build() + .unwrap() + .metadata; + + // Main snapshot should have first_row_id = 30 + let current_snapshot = table_after_main.current_snapshot().unwrap(); + assert_eq!(current_snapshot.first_row_id(), Some(30)); + + // Next row id should be 58 (30 + 28) + assert_eq!(table_after_main.next_row_id(), 58); + + // Write again to branch - append 21 rows + let branch_snapshot_2 = Snapshot::builder() + .with_snapshot_id(3) + .with_timestamp_ms(table_after_main.last_updated_ms + 1) + .with_sequence_number(2) + .with_schema_id(0) + .with_manifest_list("baz") + .with_parent_snapshot_id(Some(branch_snapshot_1.snapshot_id())) + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::new(), + }) + .with_row_range(table_after_main.next_row_id(), 21) + .build(); + + let table_after_branch_2 = table_after_main + .into_builder(None) + .set_branch_snapshot(branch_snapshot_2.clone(), branch) + .unwrap() + .build() + .unwrap() + .metadata; + + // Branch snapshot should have first_row_id = 58 (30 + 28) + let branch_ref_2 = table_after_branch_2.refs.get(branch).unwrap(); + let branch_snap_2 = table_after_branch_2 + .snapshots + .get(&branch_ref_2.snapshot_id) + .unwrap(); + assert_eq!(branch_snap_2.first_row_id(), Some(58)); + + // Next row id should be 79 (30 + 28 + 21) + assert_eq!(table_after_branch_2.next_row_id(), 79); + } + + #[test] + fn test_encryption_keys() { + let builder = builder_without_changes(FormatVersion::V2); + + // Create test encryption keys + let encryption_key_1 = EncryptedKey::builder() + .key_id("key-1") + .encrypted_key_metadata(vec![1, 2, 3, 4]) + .encrypted_by_id("encryption-service-1") + .properties(HashMap::from_iter(vec![( + "algorithm".to_string(), + "AES-256".to_string(), + )])) + .build(); + + let encryption_key_2 = EncryptedKey::builder() + .key_id("key-2") + .encrypted_key_metadata(vec![5, 6, 7, 8]) + .encrypted_by_id("encryption-service-2") + .properties(HashMap::new()) + .build(); + + // Add first encryption key + let build_result = builder + .add_encryption_key(encryption_key_1.clone()) + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 1); + assert_eq!(build_result.metadata.encryption_keys.len(), 1); + assert_eq!( + build_result.metadata.encryption_key("key-1"), + Some(&encryption_key_1) + ); + assert_eq!(build_result.changes[0], TableUpdate::AddEncryptionKey { + encryption_key: encryption_key_1.clone() + }); + + // Add second encryption key + let build_result = build_result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata1.json".to_string(), + )) + .add_encryption_key(encryption_key_2.clone()) + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 1); + assert_eq!(build_result.metadata.encryption_keys.len(), 2); + assert_eq!( + build_result.metadata.encryption_key("key-1"), + Some(&encryption_key_1) + ); + assert_eq!( + build_result.metadata.encryption_key("key-2"), + Some(&encryption_key_2) + ); + assert_eq!(build_result.changes[0], TableUpdate::AddEncryptionKey { + encryption_key: encryption_key_2.clone() + }); + + // Try to add duplicate key - should not create a change + let build_result = build_result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata2.json".to_string(), + )) + .add_encryption_key(encryption_key_1.clone()) + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 0); + assert_eq!(build_result.metadata.encryption_keys.len(), 2); + + // Remove first encryption key + let build_result = build_result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata3.json".to_string(), + )) + .remove_encryption_key("key-1") + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 1); + assert_eq!(build_result.metadata.encryption_keys.len(), 1); + assert_eq!(build_result.metadata.encryption_key("key-1"), None); + assert_eq!( + build_result.metadata.encryption_key("key-2"), + Some(&encryption_key_2) + ); + assert_eq!(build_result.changes[0], TableUpdate::RemoveEncryptionKey { + key_id: "key-1".to_string() + }); + + // Try to remove non-existent key - should not create a change + let build_result = build_result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata4.json".to_string(), + )) + .remove_encryption_key("non-existent-key") + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 0); + assert_eq!(build_result.metadata.encryption_keys.len(), 1); + + // Test encryption_keys_iter() + let keys = build_result + .metadata + .encryption_keys_iter() + .collect::>(); + assert_eq!(keys.len(), 1); + assert_eq!(keys[0], &encryption_key_2); + + // Remove last encryption key + let build_result = build_result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata5.json".to_string(), + )) + .remove_encryption_key("key-2") + .build() + .unwrap(); + + assert_eq!(build_result.changes.len(), 1); + assert_eq!(build_result.metadata.encryption_keys.len(), 0); + assert_eq!(build_result.metadata.encryption_key("key-2"), None); + assert_eq!(build_result.changes[0], TableUpdate::RemoveEncryptionKey { + key_id: "key-2".to_string() + }); + + // Verify empty encryption_keys_iter() + let keys = build_result.metadata.encryption_keys_iter(); + assert_eq!(keys.len(), 0); + } } diff --git a/crates/iceberg/src/transaction/mod.rs b/crates/iceberg/src/transaction/mod.rs index 26bd652226..4116264a14 100644 --- a/crates/iceberg/src/transaction/mod.rs +++ b/crates/iceberg/src/transaction/mod.rs @@ -243,7 +243,7 @@ mod tests { use crate::spec::TableMetadata; use crate::table::Table; use crate::transaction::{ApplyTransactionAction, Transaction}; - use crate::{Error, ErrorKind, TableIdent}; + use crate::{Catalog, Error, ErrorKind, TableCreation, TableIdent}; pub fn make_v1_table() -> Table { let file = File::open(format!( @@ -302,8 +302,41 @@ mod tests { .unwrap() } + pub(crate) async fn make_v3_minimal_table_in_catalog(catalog: &impl Catalog) -> Table { + let table_ident = + TableIdent::from_strs([format!("ns1-{}", uuid::Uuid::new_v4()), "test1".to_string()]) + .unwrap(); + + catalog + .create_namespace(table_ident.namespace(), HashMap::new()) + .await + .unwrap(); + + let file = File::open(format!( + "{}/testdata/table_metadata/{}", + env!("CARGO_MANIFEST_DIR"), + "TableMetadataV3ValidMinimal.json" + )) + .unwrap(); + let reader = BufReader::new(file); + let base_metadata = serde_json::from_reader::<_, TableMetadata>(reader).unwrap(); + + let table_creation = TableCreation::builder() + .schema((**base_metadata.current_schema()).clone()) + .partition_spec((**base_metadata.default_partition_spec()).clone()) + .sort_order((**base_metadata.default_sort_order()).clone()) + .name(table_ident.name().to_string()) + .format_version(crate::spec::FormatVersion::V3) + .build(); + + catalog + .create_table(table_ident.namespace(), table_creation) + .await + .unwrap() + } + /// Helper function to create a test table with retry properties - fn setup_test_table(num_retries: &str) -> Table { + pub(super) fn setup_test_table(num_retries: &str) -> Table { let table = make_v2_table(); // Set retry properties @@ -469,3 +502,88 @@ mod tests { } } } + +#[cfg(test)] +mod test_row_lineage { + use crate::memory::tests::new_memory_catalog; + use crate::spec::{ + DataContentType, DataFile, DataFileBuilder, DataFileFormat, Literal, Struct, + }; + use crate::transaction::tests::make_v3_minimal_table_in_catalog; + use crate::transaction::{ApplyTransactionAction, Transaction}; + + #[tokio::test] + async fn test_fast_append_with_row_lineage() { + // Helper function to create a data file with specified number of rows + fn file_with_rows(record_count: u64) -> DataFile { + DataFileBuilder::default() + .content(DataContentType::Data) + .file_path(format!("test/{}.parquet", record_count)) + .file_format(DataFileFormat::Parquet) + .file_size_in_bytes(100) + .record_count(record_count) + .partition(Struct::from_iter([Some(Literal::long(0))])) + .partition_spec_id(0) + .build() + .unwrap() + } + let catalog = new_memory_catalog().await; + + let table = make_v3_minimal_table_in_catalog(&catalog).await; + + // Check initial state - next_row_id should be 0 + assert_eq!(table.metadata().next_row_id(), 0); + + // First fast append with 30 rows + let tx = Transaction::new(&table); + let data_file_30 = file_with_rows(30); + let action = tx.fast_append().add_data_files(vec![data_file_30]); + let tx = action.apply(tx).unwrap(); + let table = tx.commit(&catalog).await.unwrap(); + + // Check snapshot and table state after first append + let snapshot = table.metadata().current_snapshot().unwrap(); + assert_eq!(snapshot.first_row_id(), Some(0)); + assert_eq!(table.metadata().next_row_id(), 30); + + // Check written manifest for first_row_id + let manifest_list = table + .metadata() + .current_snapshot() + .unwrap() + .load_manifest_list(table.file_io(), table.metadata()) + .await + .unwrap(); + + assert_eq!(manifest_list.entries().len(), 1); + let manifest_file = &manifest_list.entries()[0]; + assert_eq!(manifest_file.first_row_id, Some(0)); + + // Second fast append with 17 and 11 rows + let tx = Transaction::new(&table); + let data_file_17 = file_with_rows(17); + let data_file_11 = file_with_rows(11); + let action = tx + .fast_append() + .add_data_files(vec![data_file_17, data_file_11]); + let tx = action.apply(tx).unwrap(); + let table = tx.commit(&catalog).await.unwrap(); + + // Check snapshot and table state after second append + let snapshot = table.metadata().current_snapshot().unwrap(); + assert_eq!(snapshot.first_row_id(), Some(30)); + assert_eq!(table.metadata().next_row_id(), 30 + 17 + 11); + + // Check written manifest for first_row_id + let manifest_list = table + .metadata() + .current_snapshot() + .unwrap() + .load_manifest_list(table.file_io(), table.metadata()) + .await + .unwrap(); + assert_eq!(manifest_list.entries().len(), 2); + let manifest_file = &manifest_list.entries()[1]; + assert_eq!(manifest_file.first_row_id, Some(30)); + } +} diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index 93dd819d5c..4f85962ff1 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -206,13 +206,16 @@ impl<'a> SnapshotProducer<'a> { .as_ref() .clone(), ); - if self.table.metadata().format_version() == FormatVersion::V1 { - Ok(builder.build_v1()) - } else { - match content { + match self.table.metadata().format_version() { + FormatVersion::V1 => Ok(builder.build_v1()), + FormatVersion::V2 => match content { ManifestContentType::Data => Ok(builder.build_v2_data()), ManifestContentType::Deletes => Ok(builder.build_v2_deletes()), - } + }, + FormatVersion::V3 => match content { + ManifestContentType::Data => Ok(builder.build_v3_data()), + ManifestContentType::Deletes => Ok(builder.build_v3_deletes()), + }, } } @@ -382,6 +385,7 @@ impl<'a> SnapshotProducer<'a> { ) -> Result { let manifest_list_path = self.generate_manifest_list_file_path(0); let next_seq_num = self.table.metadata().next_sequence_number(); + let first_row_id = self.table.metadata().next_row_id(); let mut manifest_list_writer = match self.table.metadata().format_version() { FormatVersion::V1 => ManifestListWriter::v1( self.table @@ -398,6 +402,15 @@ impl<'a> SnapshotProducer<'a> { self.table.metadata().current_snapshot_id(), next_seq_num, ), + FormatVersion::V3 => ManifestListWriter::v3( + self.table + .file_io() + .new_output(manifest_list_path.clone())?, + self.snapshot_id, + self.table.metadata().current_snapshot_id(), + next_seq_num, + Some(first_row_id), + ), }; // Calling self.summary() before self.manifest_file() is important because self.added_data_files @@ -412,6 +425,7 @@ impl<'a> SnapshotProducer<'a> { .await?; manifest_list_writer.add_manifests(new_manifests.into_iter())?; + let writer_next_row_id = manifest_list_writer.next_row_id(); manifest_list_writer.close().await?; let commit_ts = chrono::Utc::now().timestamp_millis(); @@ -422,8 +436,16 @@ impl<'a> SnapshotProducer<'a> { .with_sequence_number(next_seq_num) .with_summary(summary) .with_schema_id(self.table.metadata().current_schema_id()) - .with_timestamp_ms(commit_ts) - .build(); + .with_timestamp_ms(commit_ts); + + let new_snapshot = if let Some(writer_next_row_id) = writer_next_row_id { + let assigned_rows = writer_next_row_id - self.table.metadata().next_row_id(); + new_snapshot + .with_row_range(first_row_id, assigned_rows) + .build() + } else { + new_snapshot.build() + }; let updates = vec![ TableUpdate::AddSnapshot { diff --git a/crates/iceberg/src/writer/file_writer/location_generator.rs b/crates/iceberg/src/writer/file_writer/location_generator.rs index 4cfc27844d..a5cfc28292 100644 --- a/crates/iceberg/src/writer/file_writer/location_generator.rs +++ b/crates/iceberg/src/writer/file_writer/location_generator.rs @@ -183,6 +183,7 @@ pub(crate) mod test { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: 0, }; let file_name_generator = super::DefaultFileNameGenerator::new( @@ -297,6 +298,7 @@ pub(crate) mod test { statistics: HashMap::new(), partition_statistics: HashMap::new(), encryption_keys: HashMap::new(), + next_row_id: 0, }; // Test with DefaultLocationGenerator diff --git a/crates/iceberg/testdata/table_metadata/TableMetadataV3ValidMinimal.json b/crates/iceberg/testdata/table_metadata/TableMetadataV3ValidMinimal.json new file mode 100644 index 0000000000..bf85114c02 --- /dev/null +++ b/crates/iceberg/testdata/table_metadata/TableMetadataV3ValidMinimal.json @@ -0,0 +1,74 @@ +{ + "format-version": 3, + "table-uuid": "9c12d441-03fe-4693-9a96-a0705ddf69c1", + "location": "s3://bucket/test/location", + "last-sequence-number": 34, + "last-updated-ms": 1602638573590, + "last-column-id": 3, + "current-schema-id": 0, + "next-row-id": 0, + "schemas": [ + { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "x", + "required": true, + "type": "long", + "initial-default": 1, + "write-default": 1 + }, + { + "id": 2, + "name": "y", + "required": true, + "type": "long", + "doc": "comment" + }, + { + "id": 3, + "name": "z", + "required": true, + "type": "long" + } + ] + } + ], + "default-spec-id": 0, + "partition-specs": [ + { + "spec-id": 0, + "fields": [ + { + "name": "x", + "transform": "identity", + "source-id": 1, + "field-id": 1000 + } + ] + } + ], + "last-partition-id": 1000, + "default-sort-order-id": 3, + "sort-orders": [ + { + "order-id": 3, + "fields": [ + { + "transform": "identity", + "source-id": 2, + "direction": "asc", + "null-order": "nulls-first" + }, + { + "transform": "bucket[4]", + "source-id": 3, + "direction": "desc", + "null-order": "nulls-last" + } + ] + } + ] +} \ No newline at end of file From fd089164bff19fdd4b0d297363fc7808b6f6ea18 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 4 Nov 2025 20:41:29 -0500 Subject: [PATCH 059/891] fix(reader): Support both position and equality delete files on the same FileScanTask (#1778) ## What issue does this PR close? Partially address #1749. ## Rationale for this change This PR fixes a bug in delete file loading when a `FileScanTask` contains both positional and equality delete files. We hit this when running Iceberg Java test suite via Comet in https://github.com/apache/datafusion-comet/pull/2528. The tests that failed were ``` TestSparkExecutorCache > testMergeOnReadUpdate() TestSparkExecutorCache > testMergeOnReadMerge() TestSparkExecutorCache > testMergeOnReadDelete() ``` **The Bug:** The condition in `try_start_eq_del_load` (delete_filter.rs:71-73) was inverted. It returned `None` when the equality delete file was not in the cache, causing the loader to skip loading it. When `build_equality_delete_predicate` was later called, it would fail with "Missing predicate for equality delete file". ## What changes are included in this PR? **The Fix:** - Inverted the condition so it returns `None` when the file is already in the cache (being loaded or loaded), preventing duplicate work across concurrent tasks - When the file is not in the cache, mark it as Loading and proceed with loading **Additional Changes:** - Added test case `test_load_deletes_with_mixed_types` that reproduces the bug scenario ## Are these changes tested? Yes, this PR includes a new unit test `test_load_deletes_with_mixed_types` that: - Creates a `FileScanTask` with both a positional delete file and an equality delete file - Verifies that `load_deletes` successfully processes both types - Verifies that `build_equality_delete_predicate` succeeds without the "Missing predicate" error - We hit this when running Iceberg Java test suite via Comet in https://github.com/apache/datafusion-comet/pull/2528. I also confirmed that it fixes the tests in Iceberg Java's suite. The test would fail before this fix and passes after. --- .../src/arrow/caching_delete_file_loader.rs | 113 ++++++++++++++++++ crates/iceberg/src/arrow/delete_filter.rs | 4 +- 2 files changed, 116 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 078635c9be..8a3ab3a955 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -788,4 +788,117 @@ mod tests { assert_eq!(data_col.value(1), "d"); assert_eq!(data_col.value(2), "g"); } + + /// Test loading a FileScanTask with BOTH positional and equality deletes. + /// Verifies the fix for the inverted condition that caused "Missing predicate for equality delete file" errors. + #[tokio::test] + async fn test_load_deletes_with_mixed_types() { + use crate::scan::FileScanTask; + use crate::spec::{DataFileFormat, Schema}; + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path(); + let file_io = FileIO::from_path(table_location.as_os_str().to_str().unwrap()) + .unwrap() + .build() + .unwrap(); + + // Create the data file schema + let data_file_schema = Arc::new( + Schema::builder() + .with_fields(vec![ + crate::spec::NestedField::optional( + 2, + "y", + crate::spec::Type::Primitive(crate::spec::PrimitiveType::Long), + ) + .into(), + crate::spec::NestedField::optional( + 3, + "z", + crate::spec::Type::Primitive(crate::spec::PrimitiveType::Long), + ) + .into(), + ]) + .build() + .unwrap(), + ); + + // Write positional delete file + let positional_delete_schema = crate::arrow::delete_filter::tests::create_pos_del_schema(); + let file_path_values = + vec![format!("{}/data-1.parquet", table_location.to_str().unwrap()); 4]; + let file_path_col = Arc::new(StringArray::from_iter_values(&file_path_values)); + let pos_col = Arc::new(Int64Array::from_iter_values(vec![0i64, 1, 2, 3])); + + let positional_deletes_to_write = + RecordBatch::try_new(positional_delete_schema.clone(), vec![ + file_path_col, + pos_col, + ]) + .unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + let pos_del_path = format!("{}/pos-del-mixed.parquet", table_location.to_str().unwrap()); + let file = File::create(&pos_del_path).unwrap(); + let mut writer = ArrowWriter::try_new( + file, + positional_deletes_to_write.schema(), + Some(props.clone()), + ) + .unwrap(); + writer.write(&positional_deletes_to_write).unwrap(); + writer.close().unwrap(); + + // Write equality delete file + let eq_delete_path = setup_write_equality_delete_file_1(table_location.to_str().unwrap()); + + // Create FileScanTask with BOTH positional and equality deletes + let pos_del = FileScanTaskDeleteFile { + file_path: pos_del_path, + file_type: DataContentType::PositionDeletes, + partition_spec_id: 0, + equality_ids: None, + }; + + let eq_del = FileScanTaskDeleteFile { + file_path: eq_delete_path.clone(), + file_type: DataContentType::EqualityDeletes, + partition_spec_id: 0, + equality_ids: Some(vec![2, 3]), // Only use field IDs that exist in both schemas + }; + + let file_scan_task = FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/data-1.parquet", table_location.to_str().unwrap()), + data_file_format: DataFileFormat::Parquet, + schema: data_file_schema.clone(), + project_field_ids: vec![2, 3], + predicate: None, + deletes: vec![pos_del, eq_del], + }; + + // Load the deletes - should handle both types without error + let delete_file_loader = CachingDeleteFileLoader::new(file_io.clone(), 10); + let delete_filter = delete_file_loader + .load_deletes(&file_scan_task.deletes, file_scan_task.schema_ref()) + .await + .unwrap() + .unwrap(); + + // Verify both delete types can be processed together + let result = delete_filter + .build_equality_delete_predicate(&file_scan_task) + .await; + assert!( + result.is_ok(), + "Failed to build equality delete predicate: {:?}", + result.err() + ); + } } diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index b853baa993..4250974bcd 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -68,10 +68,12 @@ impl DeleteFilter { pub(crate) fn try_start_eq_del_load(&self, file_path: &str) -> Option> { let mut state = self.state.write().unwrap(); - if !state.equality_deletes.contains_key(file_path) { + // Skip if already loaded/loading - another task owns it + if state.equality_deletes.contains_key(file_path) { return None; } + // Mark as loading to prevent duplicate work let notifier = Arc::new(Notify::new()); state .equality_deletes From a970a0c02f7d50e88648dd01eb50fec55373cf65 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 5 Nov 2025 01:11:48 -0800 Subject: [PATCH 060/891] feat(datafusion): Add TaskWriter for DataFusion (#1769) ## Which issue does this PR close? - Closes #1770 ## What changes are included in this PR? - Added TaskWriter to leverage `RecordBatchPartitionSplitter` and projected partition values - Add UnpartitionedWriter to help write unpartitioned data ## Are these changes tested? Added unit tests --- .../arrow/record_batch_partition_splitter.rs | 6 - crates/iceberg/src/writer/partitioning/mod.rs | 1 + .../partitioning/unpartitioned_writer.rs | 198 +++++++ crates/integrations/datafusion/src/lib.rs | 2 + .../datafusion/src/task_writer.rs | 534 ++++++++++++++++++ 5 files changed, 735 insertions(+), 6 deletions(-) create mode 100644 crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs create mode 100644 crates/integrations/datafusion/src/task_writer.rs diff --git a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs index 2508a00320..dcdd9c68ee 100644 --- a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs +++ b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs @@ -38,9 +38,6 @@ pub const PROJECTED_PARTITION_VALUE_COLUMN: &str = "_partition"; /// The splitter supports two modes for obtaining partition values: /// - **Computed mode** (`calculator` is `Some`): Computes partition values from source columns using transforms /// - **Pre-computed mode** (`calculator` is `None`): Expects a `_partition` column in the input batch -// # TODO -// Remove this after partition writer supported. -#[allow(dead_code)] pub struct RecordBatchPartitionSplitter { schema: SchemaRef, partition_spec: PartitionSpecRef, @@ -48,9 +45,6 @@ pub struct RecordBatchPartitionSplitter { partition_type: StructType, } -// # TODO -// Remove this after partition writer supported. -#[allow(dead_code)] impl RecordBatchPartitionSplitter { /// Create a new RecordBatchPartitionSplitter. /// diff --git a/crates/iceberg/src/writer/partitioning/mod.rs b/crates/iceberg/src/writer/partitioning/mod.rs index f63a9d0d26..c8106041ac 100644 --- a/crates/iceberg/src/writer/partitioning/mod.rs +++ b/crates/iceberg/src/writer/partitioning/mod.rs @@ -23,6 +23,7 @@ pub mod clustered_writer; pub mod fanout_writer; +pub mod unpartitioned_writer; use crate::Result; use crate::spec::PartitionKey; diff --git a/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs b/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs new file mode 100644 index 0000000000..0fb9cba3f1 --- /dev/null +++ b/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs @@ -0,0 +1,198 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! This module provides the `UnpartitionedWriter` implementation. + +use std::marker::PhantomData; + +use crate::Result; +use crate::writer::{DefaultInput, DefaultOutput, IcebergWriter, IcebergWriterBuilder}; + +/// A simple wrapper around `IcebergWriterBuilder` for unpartitioned tables. +/// +/// This writer lazily creates the underlying writer on the first write operation +/// and writes all data to a single file (or set of files if rolling). +/// +/// # Type Parameters +/// +/// * `B` - The inner writer builder type +/// * `I` - Input type (defaults to `RecordBatch`) +/// * `O` - Output collection type (defaults to `Vec`) +pub struct UnpartitionedWriter +where + B: IcebergWriterBuilder, + O: IntoIterator + FromIterator<::Item>, + ::Item: Clone, +{ + inner_builder: B, + writer: Option, + output: Vec<::Item>, + _phantom: PhantomData, +} + +impl UnpartitionedWriter +where + B: IcebergWriterBuilder, + I: Send + 'static, + O: IntoIterator + FromIterator<::Item>, + ::Item: Send + Clone, +{ + /// Create a new `UnpartitionedWriter`. + pub fn new(inner_builder: B) -> Self { + Self { + inner_builder, + writer: None, + output: Vec::new(), + _phantom: PhantomData, + } + } + + /// Write data to the writer. + /// + /// The underlying writer is lazily created on the first write operation. + /// + /// # Parameters + /// + /// * `input` - The input data to write + /// + /// # Returns + /// + /// `Ok(())` on success, or an error if the write operation fails. + pub async fn write(&mut self, input: I) -> Result<()> { + // Lazily create writer on first write + if self.writer.is_none() { + self.writer = Some(self.inner_builder.clone().build(None).await?); + } + + // Write directly to inner writer + self.writer + .as_mut() + .expect("Writer should be initialized") + .write(input) + .await + } + + /// Close the writer and return all written data files. + /// + /// This method consumes the writer to prevent further use. + /// + /// # Returns + /// + /// The accumulated output from all write operations, or an empty collection + /// if no data was written. + pub async fn close(mut self) -> Result { + if let Some(mut writer) = self.writer.take() { + self.output.extend(writer.close().await?); + } + Ok(O::from_iter(self.output)) + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_schema::{DataType, Field, Schema}; + use parquet::arrow::PARQUET_FIELD_ID_META_KEY; + use parquet::file::properties::WriterProperties; + use tempfile::TempDir; + + use super::*; + use crate::Result; + use crate::io::FileIOBuilder; + use crate::spec::{DataFileFormat, NestedField, PrimitiveType, Struct, Type}; + use crate::writer::base_writer::data_file_writer::DataFileWriterBuilder; + use crate::writer::file_writer::ParquetWriterBuilder; + use crate::writer::file_writer::location_generator::{ + DefaultFileNameGenerator, DefaultLocationGenerator, + }; + use crate::writer::file_writer::rolling_writer::RollingFileWriterBuilder; + + #[tokio::test] + async fn test_unpartitioned_writer() -> Result<()> { + let temp_dir = TempDir::new()?; + + // Build Iceberg schema + let schema = Arc::new( + crate::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build()?, + ); + + // Build Arrow schema + let arrow_schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + ])); + + // Build writer + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema.clone()); + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io, + location_gen, + file_name_gen, + ); + let writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + + let mut writer = UnpartitionedWriter::new(writer_builder); + + // Write two batches + let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + ])?; + let batch2 = RecordBatch::try_new(arrow_schema, vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["Charlie", "Dave"])), + ])?; + + writer.write(batch1).await?; + writer.write(batch2).await?; + + let data_files = writer.close().await?; + + // Verify files have empty partition and correct format + assert!(!data_files.is_empty()); + for file in &data_files { + assert_eq!(file.partition, Struct::empty()); + assert_eq!(file.file_format, DataFileFormat::Parquet); + assert_eq!(file.record_count, 4); + } + + Ok(()) + } +} diff --git a/crates/integrations/datafusion/src/lib.rs b/crates/integrations/datafusion/src/lib.rs index 09d1cac4ce..4b0ea8606d 100644 --- a/crates/integrations/datafusion/src/lib.rs +++ b/crates/integrations/datafusion/src/lib.rs @@ -26,3 +26,5 @@ mod schema; pub mod table; pub use table::table_provider_factory::IcebergTableProviderFactory; pub use table::*; + +pub(crate) mod task_writer; diff --git a/crates/integrations/datafusion/src/task_writer.rs b/crates/integrations/datafusion/src/task_writer.rs new file mode 100644 index 0000000000..d27b2e6fbf --- /dev/null +++ b/crates/integrations/datafusion/src/task_writer.rs @@ -0,0 +1,534 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! TaskWriter for DataFusion integration. +//! +//! This module provides a high-level writer that handles partitioning and routing +//! of RecordBatch data to Iceberg tables. + +use datafusion::arrow::array::RecordBatch; +use iceberg::Result; +use iceberg::arrow::RecordBatchPartitionSplitter; +use iceberg::spec::{DataFile, PartitionSpecRef, SchemaRef}; +use iceberg::writer::IcebergWriterBuilder; +use iceberg::writer::partitioning::PartitioningWriter; +use iceberg::writer::partitioning::clustered_writer::ClusteredWriter; +use iceberg::writer::partitioning::fanout_writer::FanoutWriter; +use iceberg::writer::partitioning::unpartitioned_writer::UnpartitionedWriter; + +/// High-level writer for DataFusion that handles partitioning and routing of RecordBatch data. +/// +/// TaskWriter coordinates writing data to Iceberg tables by: +/// - Selecting the appropriate partitioning strategy (unpartitioned, fanout, or clustered) +/// - Lazily initializing the partition splitter on first write +/// - Routing data to the underlying writer +/// - Collecting all written data files +/// +/// # Type Parameters +/// +/// * `B` - The IcebergWriterBuilder type used to create underlying writers +/// +/// # Example +/// +/// ```rust,ignore +/// use iceberg::spec::{PartitionSpec, Schema}; +/// use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; +/// use iceberg_datafusion::writer::task_writer::TaskWriter; +/// +/// // Create a TaskWriter for an unpartitioned table +/// let task_writer = TaskWriter::new( +/// data_file_writer_builder, +/// false, // fanout_enabled +/// schema, +/// partition_spec, +/// ); +/// +/// // Write data +/// task_writer.write(record_batch).await?; +/// +/// // Close and get data files +/// let data_files = task_writer.close().await?; +/// ``` +#[allow(dead_code)] +pub(crate) struct TaskWriter { + /// The underlying writer (UnpartitionedWriter, FanoutWriter, or ClusteredWriter) + writer: SupportedWriter, + /// Lazily initialized partition splitter for partitioned tables + partition_splitter: Option, + /// Iceberg schema reference + schema: SchemaRef, + /// Partition specification reference + partition_spec: PartitionSpecRef, +} + +/// Internal enum to hold the different writer types. +/// +/// This enum allows TaskWriter to work with different partitioning strategies +/// while maintaining a unified interface. +#[allow(dead_code)] +enum SupportedWriter { + /// Writer for unpartitioned tables + Unpartitioned(UnpartitionedWriter), + /// Writer for partitioned tables with unsorted data (maintains multiple active writers) + Fanout(FanoutWriter), + /// Writer for partitioned tables with sorted data (maintains single active writer) + Clustered(ClusteredWriter), +} + +#[allow(dead_code)] +impl TaskWriter { + /// Create a new TaskWriter. + /// + /// # Parameters + /// + /// * `writer_builder` - The IcebergWriterBuilder to use for creating underlying writers + /// * `fanout_enabled` - If true, use FanoutWriter for partitioned tables; otherwise use ClusteredWriter + /// * `schema` - The Iceberg schema reference + /// * `partition_spec` - The partition specification reference + /// + /// # Returns + /// + /// Returns a new TaskWriter instance. + /// + /// # Writer Selection Logic + /// + /// - If partition_spec is unpartitioned: creates UnpartitionedWriter + /// - If partition_spec is partitioned AND fanout_enabled is true: creates FanoutWriter + /// - If partition_spec is partitioned AND fanout_enabled is false: creates ClusteredWriter + /// + /// # Example + /// + /// ```rust,ignore + /// use iceberg::spec::{PartitionSpec, Schema}; + /// use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; + /// use iceberg_datafusion::writer::task_writer::TaskWriter; + /// + /// // Create a TaskWriter for an unpartitioned table + /// let task_writer = TaskWriter::new( + /// data_file_writer_builder, + /// false, // fanout_enabled + /// schema, + /// partition_spec, + /// ); + /// ``` + pub fn new( + writer_builder: B, + fanout_enabled: bool, + schema: SchemaRef, + partition_spec: PartitionSpecRef, + ) -> Self { + let writer = if partition_spec.is_unpartitioned() { + SupportedWriter::Unpartitioned(UnpartitionedWriter::new(writer_builder)) + } else if fanout_enabled { + SupportedWriter::Fanout(FanoutWriter::new(writer_builder)) + } else { + SupportedWriter::Clustered(ClusteredWriter::new(writer_builder)) + }; + + Self { + writer, + partition_splitter: None, + schema, + partition_spec, + } + } + + /// Write a RecordBatch to the TaskWriter. + /// + /// For the first write to a partitioned table, this method initializes the partition splitter. + /// For unpartitioned tables, data is written directly without splitting. + /// + /// # Parameters + /// + /// * `batch` - The RecordBatch to write + /// + /// # Returns + /// + /// Returns `Ok(())` on success, or an error if the write fails. + /// + /// # Errors + /// + /// This method will return an error if: + /// - Partition splitter initialization fails + /// - Splitting the batch by partition fails + /// - Writing to the underlying writer fails + /// + /// # Example + /// + /// ```rust,ignore + /// use arrow_array::RecordBatch; + /// use iceberg_datafusion::writer::task_writer::TaskWriter; + /// + /// // Write a RecordBatch + /// task_writer.write(record_batch).await?; + /// ``` + pub async fn write(&mut self, batch: RecordBatch) -> Result<()> { + match &mut self.writer { + SupportedWriter::Unpartitioned(writer) => { + // Unpartitioned: write directly without splitting + writer.write(batch).await + } + SupportedWriter::Fanout(writer) => { + // Initialize splitter on first write if needed + if self.partition_splitter.is_none() { + self.partition_splitter = + Some(RecordBatchPartitionSplitter::new_with_precomputed_values( + self.schema.clone(), + self.partition_spec.clone(), + )?); + } + + // Split and write partitioned data + Self::write_partitioned_batches(writer, &self.partition_splitter, &batch).await + } + SupportedWriter::Clustered(writer) => { + // Initialize splitter on first write if needed + if self.partition_splitter.is_none() { + self.partition_splitter = + Some(RecordBatchPartitionSplitter::new_with_precomputed_values( + self.schema.clone(), + self.partition_spec.clone(), + )?); + } + + // Split and write partitioned data + Self::write_partitioned_batches(writer, &self.partition_splitter, &batch).await + } + } + } + + /// Helper method to split and write partitioned data. + /// + /// This method handles the common logic for both FanoutWriter and ClusteredWriter: + /// - Splits the batch by partition key using the provided splitter + /// - Writes each partition to the underlying writer + /// + /// # Parameters + /// + /// * `writer` - The underlying PartitioningWriter (FanoutWriter or ClusteredWriter) + /// * `partition_splitter` - The partition splitter (must be initialized) + /// * `batch` - The RecordBatch to write + /// + /// # Returns + /// + /// Returns `Ok(())` on success, or an error if the operation fails. + async fn write_partitioned_batches( + writer: &mut W, + partition_splitter: &Option, + batch: &RecordBatch, + ) -> Result<()> { + // Split batch by partition + let splitter = partition_splitter + .as_ref() + .expect("Partition splitter should be initialized"); + let partitioned_batches = splitter.split(batch)?; + + // Write each partition + for (partition_key, partition_batch) in partitioned_batches { + writer.write(partition_key, partition_batch).await?; + } + + Ok(()) + } + + /// Close the TaskWriter and return all written data files. + /// + /// This method consumes the TaskWriter to prevent further use. + /// + /// # Returns + /// + /// Returns a `Vec` containing all written files, or an error if closing fails. + /// + /// # Errors + /// + /// This method will return an error if: + /// - Closing the underlying writer fails + /// - Any I/O operation fails during the close process + /// + /// # Example + /// + /// ```rust,ignore + /// use iceberg_datafusion::writer::task_writer::TaskWriter; + /// + /// // Close the writer and get all data files + /// let data_files = task_writer.close().await?; + /// ``` + pub async fn close(self) -> Result> { + match self.writer { + SupportedWriter::Unpartitioned(writer) => writer.close().await, + SupportedWriter::Fanout(writer) => writer.close().await, + SupportedWriter::Clustered(writer) => writer.close().await, + } + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use datafusion::arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray, StructArray}; + use datafusion::arrow::datatypes::{DataType, Field, Schema}; + use iceberg::arrow::PROJECTED_PARTITION_VALUE_COLUMN; + use iceberg::io::FileIOBuilder; + use iceberg::spec::{DataFileFormat, NestedField, PartitionSpec, PrimitiveType, Type}; + use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; + use iceberg::writer::file_writer::ParquetWriterBuilder; + use iceberg::writer::file_writer::location_generator::{ + DefaultFileNameGenerator, DefaultLocationGenerator, + }; + use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; + use parquet::arrow::PARQUET_FIELD_ID_META_KEY; + use parquet::file::properties::WriterProperties; + use tempfile::TempDir; + + use super::*; + + fn create_test_schema() -> Result> { + Ok(Arc::new( + iceberg::spec::Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "region", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build()?, + )) + } + + fn create_arrow_schema() -> Arc { + Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "3".to_string(), + )])), + ])) + } + + fn create_arrow_schema_with_partition() -> Arc { + let partition_field = Field::new("region", DataType::Utf8, false).with_metadata( + HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "1000".to_string())]), + ); + let partition_struct_field = Field::new( + PROJECTED_PARTITION_VALUE_COLUMN, + DataType::Struct(vec![partition_field].into()), + false, + ); + + Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + Field::new("name", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + Field::new("region", DataType::Utf8, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "3".to_string(), + )])), + partition_struct_field, + ])) + } + + fn create_writer_builder( + temp_dir: &TempDir, + schema: Arc, + ) -> Result< + DataFileWriterBuilder< + ParquetWriterBuilder, + DefaultLocationGenerator, + DefaultFileNameGenerator, + >, + > { + let file_io = FileIOBuilder::new_fs_io().build()?; + let location_gen = DefaultLocationGenerator::with_data_location( + temp_dir.path().to_str().unwrap().to_string(), + ); + let file_name_gen = + DefaultFileNameGenerator::new("test".to_string(), None, DataFileFormat::Parquet); + let parquet_writer_builder = + ParquetWriterBuilder::new(WriterProperties::builder().build(), schema); + let rolling_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( + parquet_writer_builder, + file_io, + location_gen, + file_name_gen, + ); + Ok(DataFileWriterBuilder::new(rolling_writer_builder)) + } + + #[tokio::test] + async fn test_task_writer_unpartitioned() -> Result<()> { + let temp_dir = TempDir::new()?; + let schema = create_test_schema()?; + let arrow_schema = create_arrow_schema(); + + // Create unpartitioned spec + let partition_spec = Arc::new(PartitionSpec::builder(schema.clone()).build()?); + + let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; + let mut task_writer = TaskWriter::new(writer_builder, false, schema, partition_spec); + + // Write data + let batch = RecordBatch::try_new(arrow_schema, vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + Arc::new(StringArray::from(vec!["US", "EU", "US"])), + ])?; + + task_writer.write(batch).await?; + let data_files = task_writer.close().await?; + + // Verify results + assert!(!data_files.is_empty()); + assert_eq!(data_files[0].record_count(), 3); + + Ok(()) + } + + /// Helper to verify partition data files + fn verify_partition_files( + data_files: &[iceberg::spec::DataFile], + expected_total: u64, + ) -> HashMap { + let total_records: u64 = data_files.iter().map(|f| f.record_count()).sum(); + assert_eq!(total_records, expected_total, "Total record count mismatch"); + + let mut partition_counts = HashMap::new(); + for data_file in data_files { + let partition_value = data_file.partition(); + let region_literal = partition_value.fields()[0] + .as_ref() + .expect("Partition value should not be null"); + let region = match region_literal + .as_primitive_literal() + .expect("Should be primitive literal") + { + iceberg::spec::PrimitiveLiteral::String(s) => s.clone(), + _ => panic!("Expected string partition value"), + }; + + *partition_counts.entry(region.clone()).or_insert(0) += data_file.record_count(); + + // Verify file path contains partition information + assert!( + data_file.file_path().contains("region="), + "File path should contain partition info" + ); + } + partition_counts + } + + #[tokio::test] + async fn test_task_writer_partitioned_fanout() -> Result<()> { + let temp_dir = TempDir::new()?; + let schema = create_test_schema()?; + let arrow_schema = create_arrow_schema_with_partition(); + + let partition_spec = Arc::new( + PartitionSpec::builder(schema.clone()) + .with_spec_id(1) + .add_partition_field("region", "region", iceberg::spec::Transform::Identity)? + .build()?, + ); + + let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; + let mut task_writer = TaskWriter::new(writer_builder, true, schema, partition_spec); + + // Create partition column + let partition_field = Field::new("region", DataType::Utf8, false).with_metadata( + HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "1000".to_string())]), + ); + let partition_values = StringArray::from(vec!["US", "EU", "US", "EU"]); + let partition_struct = StructArray::from(vec![( + Arc::new(partition_field), + Arc::new(partition_values) as ArrayRef, + )]); + + let batch = RecordBatch::try_new(arrow_schema, vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["US", "EU", "US", "EU"])), + Arc::new(partition_struct), + ])?; + + task_writer.write(batch).await?; + let data_files = task_writer.close().await?; + + let partition_counts = verify_partition_files(&data_files, 4); + assert_eq!(partition_counts.get("US"), Some(&2)); + assert_eq!(partition_counts.get("EU"), Some(&2)); + + Ok(()) + } + + #[tokio::test] + async fn test_task_writer_partitioned_clustered() -> Result<()> { + let temp_dir = TempDir::new()?; + let schema = create_test_schema()?; + let arrow_schema = create_arrow_schema_with_partition(); + + let partition_spec = Arc::new( + PartitionSpec::builder(schema.clone()) + .with_spec_id(1) + .add_partition_field("region", "region", iceberg::spec::Transform::Identity)? + .build()?, + ); + + let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; + let mut task_writer = TaskWriter::new(writer_builder, false, schema, partition_spec); + + // Create partition column + let partition_field = Field::new("region", DataType::Utf8, false).with_metadata( + HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "1000".to_string())]), + ); + let partition_values = StringArray::from(vec!["ASIA", "ASIA", "EU", "EU"]); + let partition_struct = StructArray::from(vec![( + Arc::new(partition_field), + Arc::new(partition_values) as ArrayRef, + )]); + + // ClusteredWriter expects data to be pre-sorted by partition + let batch = RecordBatch::try_new(arrow_schema, vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie", "Dave"])), + Arc::new(StringArray::from(vec!["ASIA", "ASIA", "EU", "EU"])), + Arc::new(partition_struct), + ])?; + + task_writer.write(batch).await?; + let data_files = task_writer.close().await?; + + let partition_counts = verify_partition_files(&data_files, 4); + assert_eq!(partition_counts.get("ASIA"), Some(&2)); + assert_eq!(partition_counts.get("EU"), Some(&2)); + + Ok(()) + } +} From 76d8e2d31e36d4c5013f24f8bac51eaab1d9c1db Mon Sep 17 00:00:00 2001 From: Colin Marc Date: Wed, 5 Nov 2025 11:27:04 +0100 Subject: [PATCH 061/891] fix: support reading compressed metadata (#1802) The spec mentions this naming convention here: https://iceberg.apache.org/spec/#naming-for-gzip-compressed-metadata-json-files ## Which issue does this PR close? - Closes #1801 ## What changes are included in this PR? Support for reading compressed metadata. ## Are these changes tested? Yes. Co-authored-by: Renjie Liu --- Cargo.lock | 1 + Cargo.toml | 1 + crates/iceberg/Cargo.toml | 1 + crates/iceberg/src/spec/table_metadata.rs | 50 ++++++++++++++++++++++- crates/iceberg/src/table.rs | 4 +- 5 files changed, 53 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 38d284e52d..2edc51a4ea 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3516,6 +3516,7 @@ dependencies = [ "ctor", "derive_builder", "expect-test", + "flate2", "fnv", "futures", "iceberg_test_utils", diff --git a/Cargo.toml b/Cargo.toml index 5161810be3..c10c01d94a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -71,6 +71,7 @@ enum-ordinalize = "4.3.0" env_logger = "0.11.8" expect-test = "1" faststr = "0.2.31" +flate2 = "1.1.5" fnv = "1.0.7" fs-err = "3.1.0" futures = "0.3" diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index c9b1bb3173..895a5cf5e4 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -63,6 +63,7 @@ bytes = { workspace = true } chrono = { workspace = true } derive_builder = { workspace = true } expect-test = { workspace = true } +flate2 = { workspace = true } fnv = { workspace = true } futures = { workspace = true } itertools = { workspace = true } diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index 437b0df542..06b32cc847 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -22,10 +22,12 @@ use std::cmp::Ordering; use std::collections::HashMap; use std::fmt::{Display, Formatter}; use std::hash::Hash; +use std::io::Read as _; use std::sync::Arc; use _serde::TableMetadataEnum; use chrono::{DateTime, Utc}; +use flate2::read::GzDecoder; use serde::{Deserialize, Serialize}; use serde_repr::{Deserialize_repr, Serialize_repr}; use uuid::Uuid; @@ -426,9 +428,30 @@ impl TableMetadata { file_io: &FileIO, metadata_location: impl AsRef, ) -> Result { + let metadata_location = metadata_location.as_ref(); let input_file = file_io.new_input(metadata_location)?; let metadata_content = input_file.read().await?; - let metadata = serde_json::from_slice::(&metadata_content)?; + + // Check if the file is compressed by looking for the gzip "magic number". + let metadata = if metadata_content.len() > 2 + && metadata_content[0] == 0x1F + && metadata_content[1] == 0x8B + { + let mut decoder = GzDecoder::new(metadata_content.as_ref()); + let mut decompressed_data = Vec::new(); + decoder.read_to_end(&mut decompressed_data).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + "Trying to read compressed metadata file", + ) + .with_context("file_path", metadata_location) + .with_source(e) + })?; + serde_json::from_slice(&decompressed_data)? + } else { + serde_json::from_slice(&metadata_content)? + }; + Ok(metadata) } @@ -1516,6 +1539,7 @@ impl SnapshotLog { mod tests { use std::collections::HashMap; use std::fs; + use std::io::Write as _; use std::sync::Arc; use anyhow::Result; @@ -3524,6 +3548,30 @@ mod tests { assert_eq!(read_metadata, original_metadata); } + #[tokio::test] + async fn test_table_metadata_read_compressed() { + let temp_dir = TempDir::new().unwrap(); + let metadata_location = temp_dir.path().join("v1.gz.metadata.json"); + + let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json"); + let json = serde_json::to_string(&original_metadata).unwrap(); + + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(json.as_bytes()).unwrap(); + std::fs::write(&metadata_location, encoder.finish().unwrap()) + .expect("failed to write metadata"); + + // Read the metadata back + let file_io = FileIOBuilder::new_fs_io().build().unwrap(); + let metadata_location = metadata_location.to_str().unwrap(); + let read_metadata = TableMetadata::read_from(&file_io, metadata_location) + .await + .unwrap(); + + // Verify the metadata matches + assert_eq!(read_metadata, original_metadata); + } + #[tokio::test] async fn test_table_metadata_read_nonexistent_file() { // Create a FileIO instance diff --git a/crates/iceberg/src/table.rs b/crates/iceberg/src/table.rs index 80e10b2fe2..9c789e2186 100644 --- a/crates/iceberg/src/table.rs +++ b/crates/iceberg/src/table.rs @@ -297,9 +297,7 @@ impl StaticTable { table_ident: TableIdent, file_io: FileIO, ) -> Result { - let metadata_file = file_io.new_input(metadata_location)?; - let metadata_file_content = metadata_file.read().await?; - let metadata = serde_json::from_slice::(&metadata_file_content)?; + let metadata = TableMetadata::read_from(&file_io, metadata_location).await?; let table = Table::builder() .metadata(metadata) From bb607dc5f550b694dc1741b1d95294500b5959c6 Mon Sep 17 00:00:00 2001 From: Fokko Driesprong Date: Mon, 10 Nov 2025 01:38:14 +0000 Subject: [PATCH 062/891] Support deserializing bytes (#1820) ## Which issue does this PR close? - Closes #86 ## What changes are included in this PR? ## Are these changes tested? --- crates/iceberg/src/spec/values.rs | 303 +++++++++++++++++++++++++++++- 1 file changed, 297 insertions(+), 6 deletions(-) diff --git a/crates/iceberg/src/spec/values.rs b/crates/iceberg/src/spec/values.rs index 4c763a3941..2b4ac927c3 100644 --- a/crates/iceberg/src/spec/values.rs +++ b/crates/iceberg/src/spec/values.rs @@ -2682,12 +2682,71 @@ mod _serde { Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))), _ => Err(invalid_err("string")), }, - // # TODO:https://github.com/apache/iceberg-rust/issues/86 - // rust avro don't support deserialize any bytes representation now. - RawLiteralEnum::Bytes(_) => Err(invalid_err_with_reason( - "bytes", - "todo: rust avro doesn't support deserialize any bytes representation now", - )), + RawLiteralEnum::Bytes(v) => match ty { + Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))), + Type::Primitive(PrimitiveType::Fixed(expected_len)) => { + if v.len() == *expected_len as usize { + Ok(Some(Literal::fixed(v.to_vec()))) + } else { + Err(invalid_err_with_reason( + "bytes", + &format!( + "Fixed type must be exactly {} bytes, got {}", + expected_len, + v.len() + ), + )) + } + } + Type::Primitive(PrimitiveType::Uuid) => { + if v.len() == 16 { + let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| { + invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes") + })?; + Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes)))) + } else { + Err(invalid_err_with_reason( + "bytes", + "UUID must be exactly 16 bytes", + )) + } + } + Type::Primitive(PrimitiveType::Decimal { precision, .. }) => { + let required_bytes = Type::decimal_required_bytes(*precision)? as usize; + + if v.len() == required_bytes { + // Pad the bytes to 16 bytes (i128 size) with sign extension + let mut padded_bytes = [0u8; 16]; + let start_idx = 16 - v.len(); + + // Copy the input bytes to the end of the array + padded_bytes[start_idx..].copy_from_slice(&v); + + // Sign extend if the number is negative (MSB is 1) + if !v.is_empty() && (v[0] & 0x80) != 0 { + // Fill the padding with 0xFF for negative numbers + for byte in &mut padded_bytes[..start_idx] { + *byte = 0xFF; + } + } + + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( + i128::from_be_bytes(padded_bytes), + )))) + } else { + Err(invalid_err_with_reason( + "bytes", + &format!( + "Decimal with precision {} must be exactly {} bytes, got {}", + precision, + required_bytes, + v.len() + ), + )) + } + } + _ => Err(invalid_err("bytes")), + }, RawLiteralEnum::List(v) => match ty { Type::List(ty) => Ok(Some(Literal::List( v.list @@ -3309,6 +3368,238 @@ mod tests { } } + fn check_raw_literal_bytes_serde_via_avro( + input_bytes: Vec, + expected_literal: Literal, + expected_type: &Type, + ) { + use apache_avro::types::Value; + + // Create an Avro bytes value and deserialize it through the RawLiteral path + let avro_value = Value::Bytes(input_bytes); + let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); + let result = raw_literal.try_into(expected_type).unwrap(); + assert_eq!(result, Some(expected_literal)); + } + + fn check_raw_literal_bytes_error_via_avro(input_bytes: Vec, expected_type: &Type) { + use apache_avro::types::Value; + + let avro_value = Value::Bytes(input_bytes); + let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); + let result = raw_literal.try_into(expected_type); + assert!(result.is_err(), "Expected error but got: {:?}", result); + } + + #[test] + fn test_raw_literal_bytes_binary() { + let bytes = vec![1u8, 2u8, 3u8, 4u8, 5u8]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::binary(bytes), + &Type::Primitive(PrimitiveType::Binary), + ); + } + + #[test] + fn test_raw_literal_bytes_binary_empty() { + let bytes = vec![]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::binary(bytes), + &Type::Primitive(PrimitiveType::Binary), + ); + } + + #[test] + fn test_raw_literal_bytes_fixed_correct_length() { + let bytes = vec![1u8, 2u8, 3u8, 4u8]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::fixed(bytes), + &Type::Primitive(PrimitiveType::Fixed(4)), + ); + } + + #[test] + fn test_raw_literal_bytes_fixed_wrong_length() { + let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but expecting 4 + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Fixed(4))); + } + + #[test] + fn test_raw_literal_bytes_fixed_empty_correct_length() { + let bytes = vec![]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::fixed(bytes), + &Type::Primitive(PrimitiveType::Fixed(0)), + ); + } + + #[test] + fn test_raw_literal_bytes_uuid_correct_length() { + let uuid_bytes = vec![ + 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, + 0xcd, 0xef, + ]; + let expected_uuid = u128::from_be_bytes([ + 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, + 0xcd, 0xef, + ]); + check_raw_literal_bytes_serde_via_avro( + uuid_bytes, + Literal::Primitive(PrimitiveLiteral::UInt128(expected_uuid)), + &Type::Primitive(PrimitiveType::Uuid), + ); + } + + #[test] + fn test_raw_literal_bytes_uuid_wrong_length() { + let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but UUID needs 16 + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Uuid)); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_4_scale_2() { + // Precision 4 requires 2 bytes + let decimal_bytes = vec![0x04, 0xd2]; // 1234 in 2 bytes + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_4_negative() { + // Precision 4 requires 2 bytes, negative number + let decimal_bytes = vec![0xfb, 0x2e]; // -1234 in 2 bytes + let expected_decimal = -1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_9_scale_2() { + // Precision 9 requires 4 bytes + let decimal_bytes = vec![0x00, 0x12, 0xd6, 0x87]; // 1234567 in 4 bytes + let expected_decimal = 1234567i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 9, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_18_scale_2() { + // Precision 18 requires 8 bytes + let decimal_bytes = vec![0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0xd2]; // 1234 in 8 bytes + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 18, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_38_scale_2() { + // Precision 38 requires 16 bytes (maximum precision) + let decimal_bytes = vec![ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x04, 0xd2, // 1234 in 16 bytes + ]; + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 38, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_1_scale_0() { + // Precision 1 requires 1 byte + let decimal_bytes = vec![0x07]; // 7 in 1 byte + let expected_decimal = 7i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 1, + scale: 0, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_precision_1_negative() { + // Precision 1 requires 1 byte, negative number + let decimal_bytes = vec![0xf9]; // -7 in 1 byte (two's complement) + let expected_decimal = -7i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 1, + scale: 0, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_wrong_length() { + // 3 bytes provided, but precision 4 requires 2 bytes + let bytes = vec![1u8, 2u8, 3u8]; + check_raw_literal_bytes_error_via_avro( + bytes, + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_decimal_wrong_length_too_few() { + // 1 byte provided, but precision 9 requires 4 bytes + let bytes = vec![0x42]; + check_raw_literal_bytes_error_via_avro( + bytes, + &Type::Primitive(PrimitiveType::Decimal { + precision: 9, + scale: 2, + }), + ); + } + + #[test] + fn test_raw_literal_bytes_unsupported_type() { + let bytes = vec![1u8, 2u8, 3u8, 4u8]; + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Int)); + } + #[test] fn avro_convert_test_int() { check_convert_with_avro( From 5c46965842b716cbef95ea9b47ff11da6edd0ccb Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Mon, 10 Nov 2025 14:54:11 -0500 Subject: [PATCH 063/891] fix: Bump CI Spark version to 3.5.7 (#1832) ## Which issue does this PR close? - N/A. ## What changes are included in this PR? The integration tests are failing CI because Spark 3.5.6 is not available. We should use 3.5.7 now. ## Are these changes tested? Existing CI tests. --- crates/integration_tests/testdata/spark/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/integration_tests/testdata/spark/Dockerfile b/crates/integration_tests/testdata/spark/Dockerfile index 339051bfc1..e31e2273d9 100644 --- a/crates/integration_tests/testdata/spark/Dockerfile +++ b/crates/integration_tests/testdata/spark/Dockerfile @@ -27,7 +27,7 @@ ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$ RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME} && mkdir -p /home/iceberg/spark-events WORKDIR ${SPARK_HOME} -ENV SPARK_VERSION=3.5.6 +ENV SPARK_VERSION=3.5.7 ENV ICEBERG_SPARK_RUNTIME_VERSION=3.5_2.12 ENV ICEBERG_VERSION=1.10.0 From 5c4a7834b91db77d7d63ac4bb993daabe1853e57 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 11:34:03 -0800 Subject: [PATCH 064/891] infra: use python 3.12 for release (#1836) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? #1796 removed python 3.9 support Lets use python 3.12 for release ## Are these changes tested? --- .github/workflows/release_python.yml | 2 +- .github/workflows/release_python_nightly.yml | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 8208373cf0..bc2aae1290 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -146,7 +146,7 @@ jobs: - uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: 3.12 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder with: diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 821f95ac4e..9c27554f9a 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -86,7 +86,7 @@ jobs: - uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: 3.12 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder @@ -133,3 +133,4 @@ jobs: repository-url: https://test.pypi.org/legacy/ skip-existing: true packages-dir: bindings/python/dist + verbose: true From d4c6f484b57966267c6f60de498b70801e4a05c0 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 16:50:51 -0800 Subject: [PATCH 065/891] pyiceberg-core: create smaller artifacts (#1841) ## Which issue does this PR close? - Closes #1839 ## What changes are included in this PR? Add release profile configs to create smaller python artifacts for pyiceberg-core Inspired by [opendal](https://github.com/apache/opendal/blob/e03f40a3f33856506c5f04c149f52bcc48253872/bindings/python/Cargo.toml#L219-L224) ### Size Reduction [This branch](https://github.com/apache/iceberg-rust/actions/runs/19280546898/attempts/1) vs [main branch](https://github.com/apache/iceberg-rust/actions/runs/19274277069) | Name | Old Size | New Size | % Decrease | |-----------------------------------------------|----------|----------|------------| | wheels-macos-latest-universal2-apple-darwin | 63.2 MB | 44.4 MB | 29.75% | | wheels-ubuntu-latest-aarch64 | 101 MB | 66.1 MB | 34.55% | | wheels-ubuntu-latest-armv7l | 104 MB | 70.5 MB | 32.21% | | wheels-ubuntu-latest-x86_64 | 34.8 MB | 23.8 MB | 31.61% | | wheels-windows-latest | 30.3 MB | 25.9 MB | 14.52% | | wheels-sdist | 574 KB | 575 KB ## Are these changes tested? Yes manually built and verified https://github.com/apache/iceberg-rust/actions/runs/19280546898 --- bindings/python/Cargo.toml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index b8c1efe694..9d4e94ab5c 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -37,3 +37,9 @@ pyo3 = { version = "0.25", features = ["extension-module", "abi3-py39"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } datafusion-ffi = { version = "50" } tokio = { version = "1.46.1", default-features = false } + +[profile.release] +codegen-units = 1 +debug = false +lto = "fat" +strip = true From 71977be5114513c3141bf8f438af8c2a087875ae Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 17:33:21 -0800 Subject: [PATCH 066/891] infra: add collaborators to .asf.yaml (#1842) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Add @mbutrovich and @CTTY as collaborator. From https://github.com/apache/infrastructure-asfyaml?tab=readme-ov-file#assigning-the-github-triage-role-to-external-collaborators > Projects may assign external (non-committer) collaborators the triage role for their repository. This allows them to assign, edit, and close issues and pull requests, without giving them write-access to the code. ## Are these changes tested? --- .asf.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.asf.yaml b/.asf.yaml index 842ffc39a3..a5c0f93084 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -52,6 +52,8 @@ github: collaborators: - JanKaul - c-thiel + - mbutrovich + - CTTY ghp_branch: gh-pages ghp_path: / From d7b2f1f6d0874000122929b7aacb150dbb5c9fc1 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 20:16:13 -0800 Subject: [PATCH 067/891] pyiceberg-core: use pyo3 abi3-py310 (#1843) --- bindings/python/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index 9d4e94ab5c..f94b5695bf 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -33,7 +33,7 @@ crate-type = ["cdylib"] [dependencies] arrow = { version = "56", features = ["pyarrow", "chrono-tz"] } iceberg = { path = "../../crates/iceberg" } -pyo3 = { version = "0.25", features = ["extension-module", "abi3-py39"] } +pyo3 = { version = "0.25", features = ["extension-module", "abi3-py310"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } datafusion-ffi = { version = "50" } tokio = { version = "1.46.1", default-features = false } From a7bd0105b1313600d508ef7b427e039b323da006 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 20:17:24 -0800 Subject: [PATCH 068/891] ci: parallelize unit test with matrix (#1833) --- .github/workflows/ci.yml | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d547397808..c0f360b982 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -139,6 +139,13 @@ jobs: unit: runs-on: ubuntu-latest + strategy: + matrix: + test-suite: + - { name: "default", args: "--all-targets --all-features --workspace" } + - { name: "smol", args: "--all-targets --no-default-features --features smol --features storage-all --workspace" } + - { name: "doc", args: "--doc --all-features --workspace" } + name: Unit Tests (${{ matrix.test-suite.name }}) steps: - name: Maximize build space uses: easimon/maximize-build-space@master @@ -163,15 +170,11 @@ jobs: - name: Cache Rust artifacts uses: Swatinem/rust-cache@v2 + with: + key: ${{ matrix.test-suite.name }} - name: Test - run: cargo test --no-fail-fast --all-targets --all-features --workspace - - - name: Smol Test - run: cargo test --no-fail-fast --all-targets --no-default-features --features "smol" --features "storage-all" --workspace - - - name: Doc Test - run: cargo test --no-fail-fast --doc --all-features --workspace + run: cargo test --no-fail-fast ${{ matrix.test-suite.args }} msrv: name: Verify MSRV From cb8826027007b206719461745768516aef55c063 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 11 Nov 2025 22:45:57 -0800 Subject: [PATCH 069/891] pyiceberg-core: create even smaller artifacts (#1844) ## Which issue does this PR close? - Closes #1839 ## What changes are included in this PR? use `lto = "thin", opt-level = "z"` Tested building the wheel locally on mac. And this is the winning combination. Change | Wheel Size | Build Time | Notes ------------------------|------------|------------|------------------ fat LTO, opt-level=3 (default) | 21 MB | ~3 min | Original fat LTO, opt-level=z | 12 MB | ~3 min | 43% smaller thin LTO, opt-level=z | 12 MB | ~1.5 min | Same size, 2x faster build! no LTO, opt-level=z | 13 MB | ~1 min | Slightly bigger ### Size Reduction [This branch](https://github.com/apache/iceberg-rust/actions/runs/19285828822) vs [main branch w/ #1841](https://github.com/apache/iceberg-rust/actions/runs/19280546898/attempts/1) | Name | Old Size | New Size | % Decrease | |-----------------------------------------------|----------|----------|------------| | wheels-macos-latest-universal2-apple-darwin | 44.4 MB | 23.5 MB | 47.07% | | wheels-ubuntu-latest-aarch64 | 66.1 MB | 40.9 MB | 38.12% | | wheels-ubuntu-latest-armv7l | 70.5 MB | 33.1 MB | 53.05% | | wheels-ubuntu-latest-x86_64 | 23.8 MB| 13 MB | 45.38% | | wheels-windows-latest | 25.9 MB | 12.7 MB | 45.38% | | wheels-sdist | 575 KB | 574 KB ## Are these changes tested? Yes https://github.com/apache/iceberg-rust/actions/runs/19285828822 --- bindings/python/Cargo.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index f94b5695bf..953d4a98fc 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -41,5 +41,6 @@ tokio = { version = "1.46.1", default-features = false } [profile.release] codegen-units = 1 debug = false -lto = "fat" +lto = "thin" +opt-level = "z" strip = true From 776701ced56091fe39ef306f3cbdf0edba6878c8 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Wed, 12 Nov 2025 03:23:33 -0500 Subject: [PATCH 070/891] chore: Split values.rs into separate files (#1840) ## Which issue does this PR close? - Closes #1807. ## What changes are included in this PR? Split up values.rs into component files in a values folder. Happy to discuss organization. Here are the details to account for line differences: 1. Copyright headers duplicated - Each of the new files got its own Apache license header 2. Module structure added - Created mod.rs with module declarations and re-exports 3. Module-level docs added - Changed from /*! Value in iceberg */ to individual //! style doc comments for each module 4. Visibility changes - Changed 5 constants from `private const` to `pub(crate) const`. This was necessary so these constants could be accessed across the split files: - `MAX_TIME_VALUE` - `INT_MAX` - `INT_MIN` - `LONG_MAX` - `LONG_MIN` ## Are these changes tested? Existing tests. --------- Co-authored-by: Renjie Liu --- crates/iceberg/src/spec/values.rs | 4287 ----------------- crates/iceberg/src/spec/values/datum.rs | 1225 +++++ crates/iceberg/src/spec/values/literal.rs | 747 +++ crates/iceberg/src/spec/values/map.rs | 145 + crates/iceberg/src/spec/values/mod.rs | 37 + crates/iceberg/src/spec/values/primitive.rs | 59 + crates/iceberg/src/spec/values/serde.rs | 719 +++ .../iceberg/src/spec/values/struct_value.rs | 79 + crates/iceberg/src/spec/values/temporal.rs | 105 + crates/iceberg/src/spec/values/tests.rs | 1334 +++++ 10 files changed, 4450 insertions(+), 4287 deletions(-) delete mode 100644 crates/iceberg/src/spec/values.rs create mode 100644 crates/iceberg/src/spec/values/datum.rs create mode 100644 crates/iceberg/src/spec/values/literal.rs create mode 100644 crates/iceberg/src/spec/values/map.rs create mode 100644 crates/iceberg/src/spec/values/mod.rs create mode 100644 crates/iceberg/src/spec/values/primitive.rs create mode 100644 crates/iceberg/src/spec/values/serde.rs create mode 100644 crates/iceberg/src/spec/values/struct_value.rs create mode 100644 crates/iceberg/src/spec/values/temporal.rs create mode 100644 crates/iceberg/src/spec/values/tests.rs diff --git a/crates/iceberg/src/spec/values.rs b/crates/iceberg/src/spec/values.rs deleted file mode 100644 index 2b4ac927c3..0000000000 --- a/crates/iceberg/src/spec/values.rs +++ /dev/null @@ -1,4287 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -/*! - * Value in iceberg - */ - -use std::any::Any; -use std::cmp::Ordering; -use std::collections::HashMap; -use std::fmt::{Display, Formatter}; -use std::hash::Hash; -use std::ops::Index; -use std::str::FromStr; - -pub use _serde::RawLiteral; -use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeZone, Utc}; -use num_bigint::BigInt; -use ordered_float::{Float, OrderedFloat}; -use rust_decimal::Decimal; -use rust_decimal::prelude::ToPrimitive; -use serde::de::{ - MapAccess, {self}, -}; -use serde::ser::SerializeStruct; -use serde::{Deserialize, Serialize}; -use serde_bytes::ByteBuf; -use serde_json::{Map as JsonMap, Number, Value as JsonValue}; -use timestamp::nanoseconds_to_datetime; -use uuid::Uuid; - -use super::datatypes::{PrimitiveType, Type}; -use crate::error::Result; -use crate::spec::MAX_DECIMAL_PRECISION; -use crate::spec::values::date::{date_from_naive_date, days_to_date, unix_epoch}; -use crate::spec::values::time::microseconds_to_time; -use crate::spec::values::timestamp::microseconds_to_datetime; -use crate::spec::values::timestamptz::{microseconds_to_datetimetz, nanoseconds_to_datetimetz}; -use crate::{Error, ErrorKind, ensure_data_valid}; - -/// Maximum value for [`PrimitiveType::Time`] type in microseconds, e.g. 23 hours 59 minutes 59 seconds 999999 microseconds. -const MAX_TIME_VALUE: i64 = 24 * 60 * 60 * 1_000_000i64 - 1; - -const INT_MAX: i32 = 2147483647; -const INT_MIN: i32 = -2147483648; -const LONG_MAX: i64 = 9223372036854775807; -const LONG_MIN: i64 = -9223372036854775808; - -/// Values present in iceberg type -#[derive(Clone, Debug, PartialOrd, PartialEq, Hash, Eq)] -pub enum PrimitiveLiteral { - /// 0x00 for false, non-zero byte for true - Boolean(bool), - /// Stored as 4-byte little-endian - Int(i32), - /// Stored as 8-byte little-endian - Long(i64), - /// Stored as 4-byte little-endian - Float(OrderedFloat), - /// Stored as 8-byte little-endian - Double(OrderedFloat), - /// UTF-8 bytes (without length) - String(String), - /// Binary value (without length) - Binary(Vec), - /// Stored as 16-byte little-endian - Int128(i128), - /// Stored as 16-byte little-endian - UInt128(u128), - /// When a number is larger than it can hold - AboveMax, - /// When a number is smaller than it can hold - BelowMin, -} - -impl PrimitiveLiteral { - /// Returns true if the Literal represents a primitive type - /// that can be a NaN, and that it's value is NaN - pub fn is_nan(&self) -> bool { - match self { - PrimitiveLiteral::Double(val) => val.is_nan(), - PrimitiveLiteral::Float(val) => val.is_nan(), - _ => false, - } - } -} - -/// Literal associated with its type. The value and type pair is checked when construction, so the type and value is -/// guaranteed to be correct when used. -/// -/// By default, we decouple the type and value of a literal, so we can use avoid the cost of storing extra type info -/// for each literal. But associate type with literal can be useful in some cases, for example, in unbound expression. -#[derive(Clone, Debug, PartialEq, Hash, Eq)] -pub struct Datum { - r#type: PrimitiveType, - literal: PrimitiveLiteral, -} - -impl Serialize for Datum { - fn serialize( - &self, - serializer: S, - ) -> std::result::Result { - let mut struct_ser = serializer - .serialize_struct("Datum", 2) - .map_err(serde::ser::Error::custom)?; - struct_ser - .serialize_field("type", &self.r#type) - .map_err(serde::ser::Error::custom)?; - struct_ser - .serialize_field( - "literal", - &RawLiteral::try_from( - Literal::Primitive(self.literal.clone()), - &Type::Primitive(self.r#type.clone()), - ) - .map_err(serde::ser::Error::custom)?, - ) - .map_err(serde::ser::Error::custom)?; - struct_ser.end() - } -} - -impl<'de> Deserialize<'de> for Datum { - fn deserialize>( - deserializer: D, - ) -> std::result::Result { - #[derive(Deserialize)] - #[serde(field_identifier, rename_all = "lowercase")] - enum Field { - Type, - Literal, - } - - struct DatumVisitor; - - impl<'de> serde::de::Visitor<'de> for DatumVisitor { - type Value = Datum; - - fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { - formatter.write_str("struct Datum") - } - - fn visit_seq(self, mut seq: A) -> std::result::Result - where A: serde::de::SeqAccess<'de> { - let r#type = seq - .next_element::()? - .ok_or_else(|| serde::de::Error::invalid_length(0, &self))?; - let value = seq - .next_element::()? - .ok_or_else(|| serde::de::Error::invalid_length(1, &self))?; - let Literal::Primitive(primitive) = value - .try_into(&Type::Primitive(r#type.clone())) - .map_err(serde::de::Error::custom)? - .ok_or_else(|| serde::de::Error::custom("None value"))? - else { - return Err(serde::de::Error::custom("Invalid value")); - }; - - Ok(Datum::new(r#type, primitive)) - } - - fn visit_map(self, mut map: V) -> std::result::Result - where V: MapAccess<'de> { - let mut raw_primitive: Option = None; - let mut r#type: Option = None; - while let Some(key) = map.next_key()? { - match key { - Field::Type => { - if r#type.is_some() { - return Err(de::Error::duplicate_field("type")); - } - r#type = Some(map.next_value()?); - } - Field::Literal => { - if raw_primitive.is_some() { - return Err(de::Error::duplicate_field("literal")); - } - raw_primitive = Some(map.next_value()?); - } - } - } - let Some(r#type) = r#type else { - return Err(serde::de::Error::missing_field("type")); - }; - let Some(raw_primitive) = raw_primitive else { - return Err(serde::de::Error::missing_field("literal")); - }; - let Literal::Primitive(primitive) = raw_primitive - .try_into(&Type::Primitive(r#type.clone())) - .map_err(serde::de::Error::custom)? - .ok_or_else(|| serde::de::Error::custom("None value"))? - else { - return Err(serde::de::Error::custom("Invalid value")); - }; - Ok(Datum::new(r#type, primitive)) - } - } - const FIELDS: &[&str] = &["type", "literal"]; - deserializer.deserialize_struct("Datum", FIELDS, DatumVisitor) - } -} - -// Compare following iceberg float ordering rules: -// -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN -fn iceberg_float_cmp(a: T, b: T) -> Option { - if a.is_nan() && b.is_nan() { - return match (a.is_sign_negative(), b.is_sign_negative()) { - (true, false) => Some(Ordering::Less), - (false, true) => Some(Ordering::Greater), - _ => Some(Ordering::Equal), - }; - } - - if a.is_nan() { - return Some(if a.is_sign_negative() { - Ordering::Less - } else { - Ordering::Greater - }); - } - - if b.is_nan() { - return Some(if b.is_sign_negative() { - Ordering::Greater - } else { - Ordering::Less - }); - } - - a.partial_cmp(&b) -} - -impl PartialOrd for Datum { - fn partial_cmp(&self, other: &Self) -> Option { - match (&self.literal, &other.literal, &self.r#type, &other.r#type) { - // generate the arm with same type and same literal - ( - PrimitiveLiteral::Boolean(val), - PrimitiveLiteral::Boolean(other_val), - PrimitiveType::Boolean, - PrimitiveType::Boolean, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Int(val), - PrimitiveLiteral::Int(other_val), - PrimitiveType::Int, - PrimitiveType::Int, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Long(val), - PrimitiveLiteral::Long(other_val), - PrimitiveType::Long, - PrimitiveType::Long, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Float(val), - PrimitiveLiteral::Float(other_val), - PrimitiveType::Float, - PrimitiveType::Float, - ) => iceberg_float_cmp(*val, *other_val), - ( - PrimitiveLiteral::Double(val), - PrimitiveLiteral::Double(other_val), - PrimitiveType::Double, - PrimitiveType::Double, - ) => iceberg_float_cmp(*val, *other_val), - ( - PrimitiveLiteral::Int(val), - PrimitiveLiteral::Int(other_val), - PrimitiveType::Date, - PrimitiveType::Date, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Long(val), - PrimitiveLiteral::Long(other_val), - PrimitiveType::Time, - PrimitiveType::Time, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Long(val), - PrimitiveLiteral::Long(other_val), - PrimitiveType::Timestamp, - PrimitiveType::Timestamp, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Long(val), - PrimitiveLiteral::Long(other_val), - PrimitiveType::Timestamptz, - PrimitiveType::Timestamptz, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::String(val), - PrimitiveLiteral::String(other_val), - PrimitiveType::String, - PrimitiveType::String, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::UInt128(val), - PrimitiveLiteral::UInt128(other_val), - PrimitiveType::Uuid, - PrimitiveType::Uuid, - ) => Uuid::from_u128(*val).partial_cmp(&Uuid::from_u128(*other_val)), - ( - PrimitiveLiteral::Binary(val), - PrimitiveLiteral::Binary(other_val), - PrimitiveType::Fixed(_), - PrimitiveType::Fixed(_), - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Binary(val), - PrimitiveLiteral::Binary(other_val), - PrimitiveType::Binary, - PrimitiveType::Binary, - ) => val.partial_cmp(other_val), - ( - PrimitiveLiteral::Int128(val), - PrimitiveLiteral::Int128(other_val), - PrimitiveType::Decimal { - precision: _, - scale, - }, - PrimitiveType::Decimal { - precision: _, - scale: other_scale, - }, - ) => { - let val = Decimal::from_i128_with_scale(*val, *scale); - let other_val = Decimal::from_i128_with_scale(*other_val, *other_scale); - val.partial_cmp(&other_val) - } - _ => None, - } - } -} - -impl Display for Datum { - fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - match (&self.r#type, &self.literal) { - (_, PrimitiveLiteral::Boolean(val)) => write!(f, "{val}"), - (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => write!(f, "{val}"), - (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => write!(f, "{val}"), - (_, PrimitiveLiteral::Float(val)) => write!(f, "{val}"), - (_, PrimitiveLiteral::Double(val)) => write!(f, "{val}"), - (PrimitiveType::Date, PrimitiveLiteral::Int(val)) => { - write!(f, "{}", days_to_date(*val)) - } - (PrimitiveType::Time, PrimitiveLiteral::Long(val)) => { - write!(f, "{}", microseconds_to_time(*val)) - } - (PrimitiveType::Timestamp, PrimitiveLiteral::Long(val)) => { - write!(f, "{}", microseconds_to_datetime(*val)) - } - (PrimitiveType::Timestamptz, PrimitiveLiteral::Long(val)) => { - write!(f, "{}", microseconds_to_datetimetz(*val)) - } - (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(val)) => { - write!(f, "{}", nanoseconds_to_datetime(*val)) - } - (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(val)) => { - write!(f, "{}", nanoseconds_to_datetimetz(*val)) - } - (_, PrimitiveLiteral::String(val)) => write!(f, r#""{val}""#), - (PrimitiveType::Uuid, PrimitiveLiteral::UInt128(val)) => { - write!(f, "{}", Uuid::from_u128(*val)) - } - (_, PrimitiveLiteral::Binary(val)) => display_bytes(val, f), - ( - PrimitiveType::Decimal { - precision: _, - scale, - }, - PrimitiveLiteral::Int128(val), - ) => { - write!(f, "{}", Decimal::from_i128_with_scale(*val, *scale)) - } - (_, _) => { - unreachable!() - } - } - } -} - -fn display_bytes(bytes: &[u8], f: &mut Formatter<'_>) -> std::fmt::Result { - let mut s = String::with_capacity(bytes.len() * 2); - for b in bytes { - s.push_str(&format!("{b:02X}")); - } - f.write_str(&s) -} - -impl From for Literal { - fn from(value: Datum) -> Self { - Literal::Primitive(value.literal) - } -} - -impl From for PrimitiveLiteral { - fn from(value: Datum) -> Self { - value.literal - } -} - -impl Datum { - /// Creates a `Datum` from a `PrimitiveType` and a `PrimitiveLiteral` - pub(crate) fn new(r#type: PrimitiveType, literal: PrimitiveLiteral) -> Self { - Datum { r#type, literal } - } - - /// Create iceberg value from bytes. - /// - /// See [this spec](https://iceberg.apache.org/spec/#binary-single-value-serialization) for reference. - pub fn try_from_bytes(bytes: &[u8], data_type: PrimitiveType) -> Result { - let literal = match data_type { - PrimitiveType::Boolean => { - if bytes.len() == 1 && bytes[0] == 0u8 { - PrimitiveLiteral::Boolean(false) - } else { - PrimitiveLiteral::Boolean(true) - } - } - PrimitiveType::Int => PrimitiveLiteral::Int(i32::from_le_bytes(bytes.try_into()?)), - PrimitiveType::Long => { - if bytes.len() == 4 { - // In the case of an evolved field - PrimitiveLiteral::Long(i32::from_le_bytes(bytes.try_into()?) as i64) - } else { - PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) - } - } - PrimitiveType::Float => { - PrimitiveLiteral::Float(OrderedFloat(f32::from_le_bytes(bytes.try_into()?))) - } - PrimitiveType::Double => { - if bytes.len() == 4 { - // In the case of an evolved field - PrimitiveLiteral::Double(OrderedFloat( - f32::from_le_bytes(bytes.try_into()?) as f64 - )) - } else { - PrimitiveLiteral::Double(OrderedFloat(f64::from_le_bytes(bytes.try_into()?))) - } - } - PrimitiveType::Date => PrimitiveLiteral::Int(i32::from_le_bytes(bytes.try_into()?)), - PrimitiveType::Time => PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)), - PrimitiveType::Timestamp => { - PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) - } - PrimitiveType::Timestamptz => { - PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) - } - PrimitiveType::TimestampNs => { - PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) - } - PrimitiveType::TimestamptzNs => { - PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) - } - PrimitiveType::String => { - PrimitiveLiteral::String(std::str::from_utf8(bytes)?.to_string()) - } - PrimitiveType::Uuid => { - PrimitiveLiteral::UInt128(u128::from_be_bytes(bytes.try_into()?)) - } - PrimitiveType::Fixed(_) => PrimitiveLiteral::Binary(Vec::from(bytes)), - PrimitiveType::Binary => PrimitiveLiteral::Binary(Vec::from(bytes)), - PrimitiveType::Decimal { .. } => { - let unscaled_value = BigInt::from_signed_bytes_be(bytes); - PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't convert bytes to i128: {bytes:?}"), - ) - })?) - } - }; - Ok(Datum::new(data_type, literal)) - } - - /// Convert the value to bytes - /// - /// See [this spec](https://iceberg.apache.org/spec/#binary-single-value-serialization) for reference. - pub fn to_bytes(&self) -> Result { - let buf = match &self.literal { - PrimitiveLiteral::Boolean(val) => { - if *val { - ByteBuf::from([1u8]) - } else { - ByteBuf::from([0u8]) - } - } - PrimitiveLiteral::Int(val) => ByteBuf::from(val.to_le_bytes()), - PrimitiveLiteral::Long(val) => ByteBuf::from(val.to_le_bytes()), - PrimitiveLiteral::Float(val) => ByteBuf::from(val.to_le_bytes()), - PrimitiveLiteral::Double(val) => ByteBuf::from(val.to_le_bytes()), - PrimitiveLiteral::String(val) => ByteBuf::from(val.as_bytes()), - PrimitiveLiteral::UInt128(val) => ByteBuf::from(val.to_be_bytes()), - PrimitiveLiteral::Binary(val) => ByteBuf::from(val.as_slice()), - PrimitiveLiteral::Int128(val) => { - let PrimitiveType::Decimal { precision, .. } = self.r#type else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "PrimitiveLiteral Int128 must be PrimitiveType Decimal but got {}", - &self.r#type - ), - )); - }; - - // It's required by iceberg spec that we must keep the minimum - // number of bytes for the value - let Ok(required_bytes) = Type::decimal_required_bytes(precision) else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "PrimitiveType Decimal must has valid precision but got {precision}" - ), - )); - }; - - // The primitive literal is unscaled value. - let unscaled_value = BigInt::from(*val); - // Convert into two's-complement byte representation of the BigInt - // in big-endian byte order. - let mut bytes = unscaled_value.to_signed_bytes_be(); - // Truncate with required bytes to make sure. - bytes.truncate(required_bytes as usize); - - ByteBuf::from(bytes) - } - PrimitiveLiteral::AboveMax | PrimitiveLiteral::BelowMin => { - return Err(Error::new( - ErrorKind::DataInvalid, - "Cannot convert AboveMax or BelowMin to bytes".to_string(), - )); - } - }; - - Ok(buf) - } - - /// Creates a boolean value. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// let t = Datum::bool(true); - /// - /// assert_eq!(format!("{}", t), "true".to_string()); - /// assert_eq!( - /// Literal::from(t), - /// Literal::Primitive(PrimitiveLiteral::Boolean(true)) - /// ); - /// ``` - pub fn bool>(t: T) -> Self { - Self { - r#type: PrimitiveType::Boolean, - literal: PrimitiveLiteral::Boolean(t.into()), - } - } - - /// Creates a boolean value from string. - /// See [Parse bool from str](https://doc.rust-lang.org/stable/std/primitive.bool.html#impl-FromStr-for-bool) for reference. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// let t = Datum::bool_from_str("false").unwrap(); - /// - /// assert_eq!(&format!("{}", t), "false"); - /// assert_eq!( - /// Literal::Primitive(PrimitiveLiteral::Boolean(false)), - /// t.into() - /// ); - /// ``` - pub fn bool_from_str>(s: S) -> Result { - let v = s.as_ref().parse::().map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse string to bool.").with_source(e) - })?; - Ok(Self::bool(v)) - } - - /// Creates an 32bit integer. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// let t = Datum::int(23i8); - /// - /// assert_eq!(&format!("{}", t), "23"); - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(23)), t.into()); - /// ``` - pub fn int>(t: T) -> Self { - Self { - r#type: PrimitiveType::Int, - literal: PrimitiveLiteral::Int(t.into()), - } - } - - /// Creates an 64bit integer. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// let t = Datum::long(24i8); - /// - /// assert_eq!(&format!("{t}"), "24"); - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Long(24)), t.into()); - /// ``` - pub fn long>(t: T) -> Self { - Self { - r#type: PrimitiveType::Long, - literal: PrimitiveLiteral::Long(t.into()), - } - } - - /// Creates an 32bit floating point number. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// use ordered_float::OrderedFloat; - /// let t = Datum::float(32.1f32); - /// - /// assert_eq!(&format!("{t}"), "32.1"); - /// assert_eq!( - /// Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(32.1))), - /// t.into() - /// ); - /// ``` - pub fn float>(t: T) -> Self { - Self { - r#type: PrimitiveType::Float, - literal: PrimitiveLiteral::Float(OrderedFloat(t.into())), - } - } - - /// Creates an 64bit floating point number. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// use ordered_float::OrderedFloat; - /// let t = Datum::double(32.1f64); - /// - /// assert_eq!(&format!("{t}"), "32.1"); - /// assert_eq!( - /// Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(32.1))), - /// t.into() - /// ); - /// ``` - pub fn double>(t: T) -> Self { - Self { - r#type: PrimitiveType::Double, - literal: PrimitiveLiteral::Double(OrderedFloat(t.into())), - } - } - - /// Creates date literal from number of days from unix epoch directly. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// // 2 days after 1970-01-01 - /// let t = Datum::date(2); - /// - /// assert_eq!(&format!("{t}"), "1970-01-03"); - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(2)), t.into()); - /// ``` - pub fn date(days: i32) -> Self { - Self { - r#type: PrimitiveType::Date, - literal: PrimitiveLiteral::Int(days), - } - } - - /// Creates date literal in `%Y-%m-%d` format, assume in utc timezone. - /// - /// See [`NaiveDate::from_str`]. - /// - /// Example - /// ```rust - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::date_from_str("1970-01-05").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1970-01-05"); - /// assert_eq!(Literal::date(4), t.into()); - /// ``` - pub fn date_from_str>(s: S) -> Result { - let t = s.as_ref().parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse date from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - - Ok(Self::date(date_from_naive_date(t))) - } - - /// Create date literal from calendar date (year, month and day). - /// - /// See [`NaiveDate::from_ymd_opt`]. - /// - /// Example: - /// - ///```rust - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::date_from_ymd(1970, 1, 5).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1970-01-05"); - /// assert_eq!(Literal::date(4), t.into()); - /// ``` - pub fn date_from_ymd(year: i32, month: u32, day: u32) -> Result { - let t = NaiveDate::from_ymd_opt(year, month, day).ok_or_else(|| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't create date from year: {year}, month: {month}, day: {day}"), - ) - })?; - - Ok(Self::date(date_from_naive_date(t))) - } - - /// Creates time literal in microseconds directly. - /// - /// It will return error when it's negative or too large to fit in 24 hours. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::{Datum, Literal}; - /// let micro_secs = { - /// 1 * 3600 * 1_000_000 + // 1 hour - /// 2 * 60 * 1_000_000 + // 2 minutes - /// 1 * 1_000_000 + // 1 second - /// 888999 // microseconds - /// }; - /// - /// let t = Datum::time_micros(micro_secs).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "01:02:01.888999"); - /// assert_eq!(Literal::time(micro_secs), t.into()); - /// - /// let negative_value = -100; - /// assert!(Datum::time_micros(negative_value).is_err()); - /// - /// let too_large_value = 36 * 60 * 60 * 1_000_000; // Too large to fit in 24 hours. - /// assert!(Datum::time_micros(too_large_value).is_err()); - /// ``` - pub fn time_micros(value: i64) -> Result { - ensure_data_valid!( - (0..=MAX_TIME_VALUE).contains(&value), - "Invalid value for Time type: {}", - value - ); - - Ok(Self { - r#type: PrimitiveType::Time, - literal: PrimitiveLiteral::Long(value), - }) - } - - /// Creates time literal from [`chrono::NaiveTime`]. - fn time_from_naive_time(t: NaiveTime) -> Self { - let duration = t - unix_epoch().time(); - // It's safe to unwrap here since less than 24 hours will never overflow. - let micro_secs = duration.num_microseconds().unwrap(); - - Self { - r#type: PrimitiveType::Time, - literal: PrimitiveLiteral::Long(micro_secs), - } - } - - /// Creates time literal in microseconds in `%H:%M:%S:.f` format. - /// - /// See [`NaiveTime::from_str`] for details. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::time_from_str("01:02:01.888999777").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "01:02:01.888999"); - /// ``` - pub fn time_from_str>(s: S) -> Result { - let t = s.as_ref().parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse time from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - - Ok(Self::time_from_naive_time(t)) - } - - /// Creates time literal from hour, minute, second, and microseconds. - /// - /// See [`NaiveTime::from_hms_micro_opt`]. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::time_from_hms_micro(22, 15, 33, 111).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "22:15:33.000111"); - /// ``` - pub fn time_from_hms_micro(hour: u32, min: u32, sec: u32, micro: u32) -> Result { - let t = NaiveTime::from_hms_micro_opt(hour, min, sec, micro) - .ok_or_else(|| Error::new( - ErrorKind::DataInvalid, - format!("Can't create time from hour: {hour}, min: {min}, second: {sec}, microsecond: {micro}"), - ))?; - Ok(Self::time_from_naive_time(t)) - } - - /// Creates a timestamp from unix epoch in microseconds. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::timestamp_micros(1000); - /// - /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.001"); - /// ``` - pub fn timestamp_micros(value: i64) -> Self { - Self { - r#type: PrimitiveType::Timestamp, - literal: PrimitiveLiteral::Long(value), - } - } - - /// Creates a timestamp from unix epoch in nanoseconds. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::timestamp_nanos(1000); - /// - /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.000001"); - /// ``` - pub fn timestamp_nanos(value: i64) -> Self { - Self { - r#type: PrimitiveType::TimestampNs, - literal: PrimitiveLiteral::Long(value), - } - } - - /// Creates a timestamp from [`DateTime`]. - /// - /// Example: - /// - /// ```rust - /// use chrono::{NaiveDate, NaiveDateTime, TimeZone, Utc}; - /// use iceberg::spec::Datum; - /// let t = Datum::timestamp_from_datetime( - /// NaiveDate::from_ymd_opt(1992, 3, 1) - /// .unwrap() - /// .and_hms_micro_opt(1, 2, 3, 88) - /// .unwrap(), - /// ); - /// - /// assert_eq!(&format!("{t}"), "1992-03-01 01:02:03.000088"); - /// ``` - pub fn timestamp_from_datetime(dt: NaiveDateTime) -> Self { - Self::timestamp_micros(dt.and_utc().timestamp_micros()) - } - - /// Parse a timestamp in [`%Y-%m-%dT%H:%M:%S%.f`] format. - /// - /// See [`NaiveDateTime::from_str`]. - /// - /// Example: - /// - /// ```rust - /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::timestamp_from_str("1992-03-01T01:02:03.000088").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1992-03-01 01:02:03.000088"); - /// ``` - pub fn timestamp_from_str>(s: S) -> Result { - let dt = s.as_ref().parse::().map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse timestamp.").with_source(e) - })?; - - Ok(Self::timestamp_from_datetime(dt)) - } - - /// Creates a timestamp with timezone from unix epoch in microseconds. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::timestamptz_micros(1000); - /// - /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.001 UTC"); - /// ``` - pub fn timestamptz_micros(value: i64) -> Self { - Self { - r#type: PrimitiveType::Timestamptz, - literal: PrimitiveLiteral::Long(value), - } - } - - /// Creates a timestamp with timezone from unix epoch in nanoseconds. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::timestamptz_nanos(1000); - /// - /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.000001 UTC"); - /// ``` - pub fn timestamptz_nanos(value: i64) -> Self { - Self { - r#type: PrimitiveType::TimestamptzNs, - literal: PrimitiveLiteral::Long(value), - } - } - - /// Creates a timestamp with timezone from [`DateTime`]. - /// Example: - /// - /// ```rust - /// use chrono::{TimeZone, Utc}; - /// use iceberg::spec::Datum; - /// let t = Datum::timestamptz_from_datetime(Utc.timestamp_opt(1000, 0).unwrap()); - /// - /// assert_eq!(&format!("{t}"), "1970-01-01 00:16:40 UTC"); - /// ``` - pub fn timestamptz_from_datetime(dt: DateTime) -> Self { - Self::timestamptz_micros(dt.with_timezone(&Utc).timestamp_micros()) - } - - /// Parse timestamp with timezone in RFC3339 format. - /// - /// See [`DateTime::from_str`]. - /// - /// Example: - /// - /// ```rust - /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; - /// use iceberg::spec::{Datum, Literal}; - /// let t = Datum::timestamptz_from_str("1992-03-01T01:02:03.000088+08:00").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1992-02-29 17:02:03.000088 UTC"); - /// ``` - pub fn timestamptz_from_str>(s: S) -> Result { - let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) - })?; - - Ok(Self::timestamptz_from_datetime(dt)) - } - - /// Creates a string literal. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::string("ss"); - /// - /// assert_eq!(&format!("{t}"), r#""ss""#); - /// ``` - pub fn string(s: S) -> Self { - Self { - r#type: PrimitiveType::String, - literal: PrimitiveLiteral::String(s.to_string()), - } - } - - /// Creates uuid literal. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// use uuid::uuid; - /// let t = Datum::uuid(uuid!("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8")); - /// - /// assert_eq!(&format!("{t}"), "a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8"); - /// ``` - pub fn uuid(uuid: Uuid) -> Self { - Self { - r#type: PrimitiveType::Uuid, - literal: PrimitiveLiteral::UInt128(uuid.as_u128()), - } - } - - /// Creates uuid from str. See [`Uuid::parse_str`]. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::uuid_from_str("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8"); - /// ``` - pub fn uuid_from_str>(s: S) -> Result { - let uuid = Uuid::parse_str(s.as_ref()).map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse uuid from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - Ok(Self::uuid(uuid)) - } - - /// Creates a fixed literal from bytes. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; - /// let t = Datum::fixed(vec![1u8, 2u8]); - /// - /// assert_eq!(&format!("{t}"), "0102"); - /// ``` - pub fn fixed>(input: I) -> Self { - let value: Vec = input.into_iter().collect(); - Self { - r#type: PrimitiveType::Fixed(value.len() as u64), - literal: PrimitiveLiteral::Binary(value), - } - } - - /// Creates a binary literal from bytes. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// let t = Datum::binary(vec![1u8, 100u8]); - /// - /// assert_eq!(&format!("{t}"), "0164"); - /// ``` - pub fn binary>(input: I) -> Self { - Self { - r#type: PrimitiveType::Binary, - literal: PrimitiveLiteral::Binary(input.into_iter().collect()), - } - } - - /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// use itertools::assert_equal; - /// use rust_decimal::Decimal; - /// let t = Datum::decimal_from_str("123.45").unwrap(); - /// - /// assert_eq!(&format!("{t}"), "123.45"); - /// ``` - pub fn decimal_from_str>(s: S) -> Result { - let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) - })?; - - Self::decimal(decimal) - } - - /// Try to create a decimal literal from [`Decimal`]. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// use rust_decimal::Decimal; - /// - /// let t = Datum::decimal(Decimal::new(123, 2)).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1.23"); - /// ``` - pub fn decimal(value: impl Into) -> Result { - let decimal = value.into(); - let scale = decimal.scale(); - - let r#type = Type::decimal(MAX_DECIMAL_PRECISION, scale)?; - if let Type::Primitive(p) = r#type { - Ok(Self { - r#type: p, - literal: PrimitiveLiteral::Int128(decimal.mantissa()), - }) - } else { - unreachable!("Decimal type must be primitive.") - } - } - - /// Try to create a decimal literal from [`Decimal`] with precision. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// use rust_decimal::Decimal; - /// - /// let t = Datum::decimal_with_precision(Decimal::new(123, 2), 30).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1.23"); - /// ``` - pub fn decimal_with_precision(value: impl Into, precision: u32) -> Result { - let decimal = value.into(); - let scale = decimal.scale(); - - let available_bytes = Type::decimal_required_bytes(precision)? as usize; - let unscaled_value = BigInt::from(decimal.mantissa()); - let actual_bytes = unscaled_value.to_signed_bytes_be(); - if actual_bytes.len() > available_bytes { - return Err(Error::new( - ErrorKind::DataInvalid, - format!("Decimal value {decimal} is too large for precision {precision}"), - )); - } - - let r#type = Type::decimal(precision, scale)?; - if let Type::Primitive(p) = r#type { - Ok(Self { - r#type: p, - literal: PrimitiveLiteral::Int128(decimal.mantissa()), - }) - } else { - unreachable!("Decimal type must be primitive.") - } - } - - fn i64_to_i32 + PartialOrd>(val: T) -> Datum { - if val > INT_MAX as i64 { - Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax) - } else if val < INT_MIN as i64 { - Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin) - } else { - Datum::int(val.into() as i32) - } - } - - fn i128_to_i32 + PartialOrd>(val: T) -> Datum { - if val > INT_MAX as i128 { - Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax) - } else if val < INT_MIN as i128 { - Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin) - } else { - Datum::int(val.into() as i32) - } - } - - fn i128_to_i64 + PartialOrd>(val: T) -> Datum { - if val > LONG_MAX as i128 { - Datum::new(PrimitiveType::Long, PrimitiveLiteral::AboveMax) - } else if val < LONG_MIN as i128 { - Datum::new(PrimitiveType::Long, PrimitiveLiteral::BelowMin) - } else { - Datum::long(val.into() as i64) - } - } - - fn string_to_i128>(s: S) -> Result { - s.as_ref().parse::().map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse string to i128.").with_source(e) - }) - } - - /// Convert the datum to `target_type`. - pub fn to(self, target_type: &Type) -> Result { - match target_type { - Type::Primitive(target_primitive_type) => { - match (&self.literal, &self.r#type, target_primitive_type) { - (PrimitiveLiteral::Int(val), _, PrimitiveType::Int) => Ok(Datum::int(*val)), - (PrimitiveLiteral::Int(val), _, PrimitiveType::Date) => Ok(Datum::date(*val)), - (PrimitiveLiteral::Int(val), _, PrimitiveType::Long) => Ok(Datum::long(*val)), - (PrimitiveLiteral::Long(val), _, PrimitiveType::Int) => { - Ok(Datum::i64_to_i32(*val)) - } - (PrimitiveLiteral::Long(val), _, PrimitiveType::Timestamp) => { - Ok(Datum::timestamp_micros(*val)) - } - (PrimitiveLiteral::Long(val), _, PrimitiveType::Timestamptz) => { - Ok(Datum::timestamptz_micros(*val)) - } - // Let's wait with nano's until this clears up: https://github.com/apache/iceberg/pull/11775 - (PrimitiveLiteral::Int128(val), _, PrimitiveType::Long) => { - Ok(Datum::i128_to_i64(*val)) - } - - (PrimitiveLiteral::String(val), _, PrimitiveType::Boolean) => { - Datum::bool_from_str(val) - } - (PrimitiveLiteral::String(val), _, PrimitiveType::Int) => { - Datum::string_to_i128(val).map(Datum::i128_to_i32) - } - (PrimitiveLiteral::String(val), _, PrimitiveType::Long) => { - Datum::string_to_i128(val).map(Datum::i128_to_i64) - } - (PrimitiveLiteral::String(val), _, PrimitiveType::Timestamp) => { - Datum::timestamp_from_str(val) - } - (PrimitiveLiteral::String(val), _, PrimitiveType::Timestamptz) => { - Datum::timestamptz_from_str(val) - } - - // TODO: implement more type conversions - (_, self_type, target_type) if self_type == target_type => Ok(self), - _ => Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Can't convert datum from {} type to {} type.", - self.r#type, target_primitive_type - ), - )), - } - } - _ => Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Can't convert datum from {} type to {} type.", - self.r#type, target_type - ), - )), - } - } - - /// Get the primitive literal from datum. - pub fn literal(&self) -> &PrimitiveLiteral { - &self.literal - } - - /// Get the primitive type from datum. - pub fn data_type(&self) -> &PrimitiveType { - &self.r#type - } - - /// Returns true if the Literal represents a primitive type - /// that can be a NaN, and that it's value is NaN - pub fn is_nan(&self) -> bool { - match self.literal { - PrimitiveLiteral::Double(val) => val.is_nan(), - PrimitiveLiteral::Float(val) => val.is_nan(), - _ => false, - } - } - - /// Returns a human-readable string representation of this literal. - /// - /// For string literals, this returns the raw string value without quotes. - /// For all other literals, it falls back to [`to_string()`]. - pub fn to_human_string(&self) -> String { - match self.literal() { - PrimitiveLiteral::String(s) => s.to_string(), - _ => self.to_string(), - } - } -} - -/// Map is a collection of key-value pairs with a key type and a value type. -/// It used in Literal::Map, to make it hashable, the order of key-value pairs is stored in a separate vector -/// so that we can hash the map in a deterministic way. But it also means that the order of key-value pairs is matter -/// for the hash value. -/// -/// When converting to Arrow (e.g., for Iceberg), the map should be represented using -/// the default field name "key_value". -/// -/// Example: -/// -/// ```text -/// let key_value_field = Field::new( -/// DEFAULT_MAP_FIELD_NAME, -/// arrow_schema::DataType::Struct(vec![ -/// Arc::new(key_field.clone()), -/// Arc::new(value_field.clone()), -/// ].into()), -/// false -/// ); -/// ''' -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Map { - index: HashMap, - pair: Vec<(Literal, Option)>, -} - -impl Map { - /// Creates a new empty map. - pub fn new() -> Self { - Self { - index: HashMap::new(), - pair: Vec::new(), - } - } - - /// Return the number of key-value pairs in the map. - pub fn len(&self) -> usize { - self.pair.len() - } - - /// Returns true if the map contains no elements. - pub fn is_empty(&self) -> bool { - self.pair.is_empty() - } - - /// Inserts a key-value pair into the map. - /// If the map did not have this key present, None is returned. - /// If the map did have this key present, the value is updated, and the old value is returned. - pub fn insert(&mut self, key: Literal, value: Option) -> Option> { - if let Some(index) = self.index.get(&key) { - let old_value = std::mem::replace(&mut self.pair[*index].1, value); - Some(old_value) - } else { - self.pair.push((key.clone(), value)); - self.index.insert(key, self.pair.len() - 1); - None - } - } - - /// Returns a reference to the value corresponding to the key. - /// If the key is not present in the map, None is returned. - pub fn get(&self, key: &Literal) -> Option<&Option> { - self.index.get(key).map(|index| &self.pair[*index].1) - } - - /// The order of map is matter, so this method used to compare two maps has same key-value pairs without considering the order. - pub fn has_same_content(&self, other: &Map) -> bool { - if self.len() != other.len() { - return false; - } - - for (key, value) in &self.pair { - match other.get(key) { - Some(other_value) if value == other_value => (), - _ => return false, - } - } - - true - } -} - -impl Default for Map { - fn default() -> Self { - Self::new() - } -} - -impl Hash for Map { - fn hash(&self, state: &mut H) { - for (key, value) in &self.pair { - key.hash(state); - value.hash(state); - } - } -} - -impl FromIterator<(Literal, Option)> for Map { - fn from_iter)>>(iter: T) -> Self { - let mut map = Map::new(); - for (key, value) in iter { - map.insert(key, value); - } - map - } -} - -impl IntoIterator for Map { - type Item = (Literal, Option); - type IntoIter = std::vec::IntoIter; - - fn into_iter(self) -> Self::IntoIter { - self.pair.into_iter() - } -} - -impl From<[(Literal, Option); N]> for Map { - fn from(value: [(Literal, Option); N]) -> Self { - value.iter().cloned().collect() - } -} - -/// Values present in iceberg type -#[derive(Clone, Debug, PartialEq, Eq, Hash)] -pub enum Literal { - /// A primitive value - Primitive(PrimitiveLiteral), - /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. - /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. - /// Fields may have an optional comment or doc string. Fields can have default values. - Struct(Struct), - /// A list is a collection of values with some element type. - /// The element field has an integer id that is unique in the table schema. - /// Elements can be either optional or required. Element types may be any type. - List(Vec>), - /// A map is a collection of key-value pairs with a key type and a value type. - /// Both the key field and value field each have an integer id that is unique in the table schema. - /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. - Map(Map), -} - -impl Literal { - /// Creates a boolean value. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t = Literal::bool(true); - /// - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Boolean(true)), t); - /// ``` - pub fn bool>(t: T) -> Self { - Self::Primitive(PrimitiveLiteral::Boolean(t.into())) - } - - /// Creates a boolean value from string. - /// See [Parse bool from str](https://doc.rust-lang.org/stable/std/primitive.bool.html#impl-FromStr-for-bool) for reference. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t = Literal::bool_from_str("false").unwrap(); - /// - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Boolean(false)), t); - /// ``` - pub fn bool_from_str>(s: S) -> Result { - let v = s.as_ref().parse::().map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse string to bool.").with_source(e) - })?; - Ok(Self::Primitive(PrimitiveLiteral::Boolean(v))) - } - - /// Creates an 32bit integer. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t = Literal::int(23i8); - /// - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(23)), t); - /// ``` - pub fn int>(t: T) -> Self { - Self::Primitive(PrimitiveLiteral::Int(t.into())) - } - - /// Creates an 64bit integer. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t = Literal::long(24i8); - /// - /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Long(24)), t); - /// ``` - pub fn long>(t: T) -> Self { - Self::Primitive(PrimitiveLiteral::Long(t.into())) - } - - /// Creates an 32bit floating point number. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// use ordered_float::OrderedFloat; - /// let t = Literal::float(32.1f32); - /// - /// assert_eq!( - /// Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(32.1))), - /// t - /// ); - /// ``` - pub fn float>(t: T) -> Self { - Self::Primitive(PrimitiveLiteral::Float(OrderedFloat(t.into()))) - } - - /// Creates an 32bit floating point number. - /// - /// Example: - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// use ordered_float::OrderedFloat; - /// let t = Literal::double(32.1f64); - /// - /// assert_eq!( - /// Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(32.1))), - /// t - /// ); - /// ``` - pub fn double>(t: T) -> Self { - Self::Primitive(PrimitiveLiteral::Double(OrderedFloat(t.into()))) - } - - /// Creates date literal from number of days from unix epoch directly. - pub fn date(days: i32) -> Self { - Self::Primitive(PrimitiveLiteral::Int(days)) - } - - /// Creates a date in `%Y-%m-%d` format, assume in utc timezone. - /// - /// See [`NaiveDate::from_str`]. - /// - /// Example - /// ```rust - /// use iceberg::spec::Literal; - /// let t = Literal::date_from_str("1970-01-03").unwrap(); - /// - /// assert_eq!(Literal::date(2), t); - /// ``` - pub fn date_from_str>(s: S) -> Result { - let t = s.as_ref().parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse date from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - - Ok(Self::date(date_from_naive_date(t))) - } - - /// Create a date from calendar date (year, month and day). - /// - /// See [`NaiveDate::from_ymd_opt`]. - /// - /// Example: - /// - ///```rust - /// use iceberg::spec::Literal; - /// let t = Literal::date_from_ymd(1970, 1, 5).unwrap(); - /// - /// assert_eq!(Literal::date(4), t); - /// ``` - pub fn date_from_ymd(year: i32, month: u32, day: u32) -> Result { - let t = NaiveDate::from_ymd_opt(year, month, day).ok_or_else(|| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't create date from year: {year}, month: {month}, day: {day}"), - ) - })?; - - Ok(Self::date(date_from_naive_date(t))) - } - - /// Creates time in microseconds directly - pub fn time(value: i64) -> Self { - Self::Primitive(PrimitiveLiteral::Long(value)) - } - - /// Creates time literal from [`chrono::NaiveTime`]. - fn time_from_naive_time(t: NaiveTime) -> Self { - let duration = t - unix_epoch().time(); - // It's safe to unwrap here since less than 24 hours will never overflow. - let micro_secs = duration.num_microseconds().unwrap(); - - Literal::time(micro_secs) - } - - /// Creates time in microseconds in `%H:%M:%S:.f` format. - /// - /// See [`NaiveTime::from_str`] for details. - /// - /// Example: - /// ```rust - /// use iceberg::spec::Literal; - /// let t = Literal::time_from_str("01:02:01.888999777").unwrap(); - /// - /// let micro_secs = { - /// 1 * 3600 * 1_000_000 + // 1 hour - /// 2 * 60 * 1_000_000 + // 2 minutes - /// 1 * 1_000_000 + // 1 second - /// 888999 // microseconds - /// }; - /// assert_eq!(Literal::time(micro_secs), t); - /// ``` - pub fn time_from_str>(s: S) -> Result { - let t = s.as_ref().parse::().map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse time from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - - Ok(Self::time_from_naive_time(t)) - } - - /// Creates time literal from hour, minute, second, and microseconds. - /// - /// See [`NaiveTime::from_hms_micro_opt`]. - /// - /// Example: - /// ```rust - /// use iceberg::spec::Literal; - /// let t = Literal::time_from_hms_micro(22, 15, 33, 111).unwrap(); - /// - /// assert_eq!(Literal::time_from_str("22:15:33.000111").unwrap(), t); - /// ``` - pub fn time_from_hms_micro(hour: u32, min: u32, sec: u32, micro: u32) -> Result { - let t = NaiveTime::from_hms_micro_opt(hour, min, sec, micro) - .ok_or_else(|| Error::new( - ErrorKind::DataInvalid, - format!("Can't create time from hour: {hour}, min: {min}, second: {sec}, microsecond: {micro}"), - ))?; - Ok(Self::time_from_naive_time(t)) - } - - /// Creates a timestamp from unix epoch in microseconds. - pub fn timestamp(value: i64) -> Self { - Self::Primitive(PrimitiveLiteral::Long(value)) - } - - /// Creates a timestamp with timezone from unix epoch in microseconds. - pub fn timestamptz(value: i64) -> Self { - Self::Primitive(PrimitiveLiteral::Long(value)) - } - - /// Creates a timestamp from unix epoch in nanoseconds. - pub(crate) fn timestamp_nano(value: i64) -> Self { - Self::Primitive(PrimitiveLiteral::Long(value)) - } - - /// Creates a timestamp with timezone from unix epoch in nanoseconds. - pub(crate) fn timestamptz_nano(value: i64) -> Self { - Self::Primitive(PrimitiveLiteral::Long(value)) - } - - /// Creates a timestamp from [`DateTime`]. - pub fn timestamp_from_datetime(dt: DateTime) -> Self { - Self::timestamp(dt.with_timezone(&Utc).timestamp_micros()) - } - - /// Creates a timestamp with timezone from [`DateTime`]. - pub fn timestamptz_from_datetime(dt: DateTime) -> Self { - Self::timestamptz(dt.with_timezone(&Utc).timestamp_micros()) - } - - /// Parse a timestamp in RFC3339 format. - /// - /// See [`DateTime::from_str`]. - /// - /// Example: - /// - /// ```rust - /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; - /// use iceberg::spec::Literal; - /// let t = Literal::timestamp_from_str("2012-12-12 12:12:12.8899-04:00").unwrap(); - /// - /// let t2 = { - /// let date = NaiveDate::from_ymd_opt(2012, 12, 12).unwrap(); - /// let time = NaiveTime::from_hms_micro_opt(12, 12, 12, 889900).unwrap(); - /// let dt = NaiveDateTime::new(date, time); - /// Literal::timestamp_from_datetime(DateTime::::from_local( - /// dt, - /// FixedOffset::west_opt(4 * 3600).unwrap(), - /// )) - /// }; - /// - /// assert_eq!(t, t2); - /// ``` - pub fn timestamp_from_str>(s: S) -> Result { - let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) - })?; - - Ok(Self::timestamp_from_datetime(dt)) - } - - /// Similar to [`Literal::timestamp_from_str`], but return timestamp with timezone literal. - pub fn timestamptz_from_str>(s: S) -> Result { - let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) - })?; - - Ok(Self::timestamptz_from_datetime(dt)) - } - - /// Creates a string literal. - pub fn string(s: S) -> Self { - Self::Primitive(PrimitiveLiteral::String(s.to_string())) - } - - /// Creates uuid literal. - pub fn uuid(uuid: Uuid) -> Self { - Self::Primitive(PrimitiveLiteral::UInt128(uuid.as_u128())) - } - - /// Creates uuid from str. See [`Uuid::parse_str`]. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Literal; - /// use uuid::Uuid; - /// let t1 = Literal::uuid_from_str("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8").unwrap(); - /// let t2 = Literal::uuid(Uuid::from_u128_le(0xd8d7d6d5d4d3d2d1c2c1b2b1a4a3a2a1)); - /// - /// assert_eq!(t1, t2); - /// ``` - pub fn uuid_from_str>(s: S) -> Result { - let uuid = Uuid::parse_str(s.as_ref()).map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("Can't parse uuid from string: {}", s.as_ref()), - ) - .with_source(e) - })?; - Ok(Self::uuid(uuid)) - } - - /// Creates a fixed literal from bytes. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t1 = Literal::fixed(vec![1u8, 2u8]); - /// let t2 = Literal::Primitive(PrimitiveLiteral::Binary(vec![1u8, 2u8])); - /// - /// assert_eq!(t1, t2); - /// ``` - pub fn fixed>(input: I) -> Self { - Literal::Primitive(PrimitiveLiteral::Binary(input.into_iter().collect())) - } - - /// Creates a binary literal from bytes. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::{Literal, PrimitiveLiteral}; - /// let t1 = Literal::binary(vec![1u8, 2u8]); - /// let t2 = Literal::Primitive(PrimitiveLiteral::Binary(vec![1u8, 2u8])); - /// - /// assert_eq!(t1, t2); - /// ``` - pub fn binary>(input: I) -> Self { - Literal::Primitive(PrimitiveLiteral::Binary(input.into_iter().collect())) - } - - /// Creates a decimal literal. - pub fn decimal(decimal: i128) -> Self { - Self::Primitive(PrimitiveLiteral::Int128(decimal)) - } - - /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. - /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Literal; - /// use rust_decimal::Decimal; - /// let t1 = Literal::decimal(12345); - /// let t2 = Literal::decimal_from_str("123.45").unwrap(); - /// - /// assert_eq!(t1, t2); - /// ``` - pub fn decimal_from_str>(s: S) -> Result { - let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) - })?; - Ok(Self::decimal(decimal.mantissa())) - } - - /// Attempts to convert the Literal to a PrimitiveLiteral - pub fn as_primitive_literal(&self) -> Option { - match self { - Literal::Primitive(primitive) => Some(primitive.clone()), - _ => None, - } - } -} - -/// The partition struct stores the tuple of partition values for each file. -/// Its type is derived from the partition fields of the partition spec used to write the manifest file. -/// In v2, the partition struct’s field ids must match the ids from the partition spec. -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -pub struct Struct { - /// Vector to store the field values - fields: Vec>, -} - -impl Struct { - /// Create a empty struct. - pub fn empty() -> Self { - Self { fields: Vec::new() } - } - - /// Create a iterator to read the field in order of field_value. - pub fn iter(&self) -> impl ExactSizeIterator> { - self.fields.iter().map(|field| field.as_ref()) - } - - /// returns true if the field at position `index` is null - pub fn is_null_at_index(&self, index: usize) -> bool { - self.fields[index].is_none() - } - - /// Return fields in the struct. - pub fn fields(&self) -> &[Option] { - &self.fields - } -} - -impl Index for Struct { - type Output = Option; - - fn index(&self, idx: usize) -> &Self::Output { - &self.fields[idx] - } -} - -impl IntoIterator for Struct { - type Item = Option; - - type IntoIter = std::vec::IntoIter>; - - fn into_iter(self) -> Self::IntoIter { - self.fields.into_iter() - } -} - -impl FromIterator> for Struct { - fn from_iter>>(iter: I) -> Self { - Struct { - fields: iter.into_iter().collect(), - } - } -} - -impl Literal { - /// Create iceberg value from a json value - /// - /// See [this spec](https://iceberg.apache.org/spec/#json-single-value-serialization) for reference. - pub fn try_from_json(value: JsonValue, data_type: &Type) -> Result> { - match data_type { - Type::Primitive(primitive) => match (primitive, value) { - (PrimitiveType::Boolean, JsonValue::Bool(bool)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Boolean(bool)))) - } - (PrimitiveType::Int, JsonValue::Number(number)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( - number - .as_i64() - .ok_or(Error::new( - crate::ErrorKind::DataInvalid, - "Failed to convert json number to int", - ))? - .try_into()?, - )))) - } - (PrimitiveType::Long, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( - PrimitiveLiteral::Long(number.as_i64().ok_or(Error::new( - crate::ErrorKind::DataInvalid, - "Failed to convert json number to long", - ))?), - ))), - (PrimitiveType::Float, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( - PrimitiveLiteral::Float(OrderedFloat(number.as_f64().ok_or(Error::new( - crate::ErrorKind::DataInvalid, - "Failed to convert json number to float", - ))? as f32)), - ))), - (PrimitiveType::Double, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( - PrimitiveLiteral::Double(OrderedFloat(number.as_f64().ok_or(Error::new( - crate::ErrorKind::DataInvalid, - "Failed to convert json number to double", - ))?)), - ))), - (PrimitiveType::Date, JsonValue::String(s)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( - date::date_to_days(&NaiveDate::parse_from_str(&s, "%Y-%m-%d")?), - )))) - } - (PrimitiveType::Date, JsonValue::Number(number)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( - number - .as_i64() - .ok_or(Error::new( - crate::ErrorKind::DataInvalid, - "Failed to convert json number to date (days since epoch)", - ))? - .try_into()?, - )))) - } - (PrimitiveType::Time, JsonValue::String(s)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Long( - time::time_to_microseconds(&NaiveTime::parse_from_str(&s, "%H:%M:%S%.f")?), - )))) - } - (PrimitiveType::Timestamp, JsonValue::String(s)) => Ok(Some(Literal::Primitive( - PrimitiveLiteral::Long(timestamp::datetime_to_microseconds( - &NaiveDateTime::parse_from_str(&s, "%Y-%m-%dT%H:%M:%S%.f")?, - )), - ))), - (PrimitiveType::Timestamptz, JsonValue::String(s)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::Long( - timestamptz::datetimetz_to_microseconds(&Utc.from_utc_datetime( - &NaiveDateTime::parse_from_str(&s, "%Y-%m-%dT%H:%M:%S%.f+00:00")?, - )), - )))) - } - (PrimitiveType::String, JsonValue::String(s)) => { - Ok(Some(Literal::Primitive(PrimitiveLiteral::String(s)))) - } - (PrimitiveType::Uuid, JsonValue::String(s)) => Ok(Some(Literal::Primitive( - PrimitiveLiteral::UInt128(Uuid::parse_str(&s)?.as_u128()), - ))), - (PrimitiveType::Fixed(_), JsonValue::String(_)) => todo!(), - (PrimitiveType::Binary, JsonValue::String(_)) => todo!(), - ( - PrimitiveType::Decimal { - precision: _, - scale, - }, - JsonValue::String(s), - ) => { - let mut decimal = Decimal::from_str_exact(&s)?; - decimal.rescale(*scale); - Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( - decimal.mantissa(), - )))) - } - (_, JsonValue::Null) => Ok(None), - (i, j) => Err(Error::new( - crate::ErrorKind::DataInvalid, - format!("The json value {j} doesn't fit to the iceberg type {i}."), - )), - }, - Type::Struct(schema) => { - if let JsonValue::Object(mut object) = value { - Ok(Some(Literal::Struct(Struct::from_iter( - schema.fields().iter().map(|field| { - object.remove(&field.id.to_string()).and_then(|value| { - Literal::try_from_json(value, &field.field_type) - .and_then(|value| { - value.ok_or(Error::new( - ErrorKind::DataInvalid, - "Key of map cannot be null", - )) - }) - .ok() - }) - }), - )))) - } else { - Err(Error::new( - crate::ErrorKind::DataInvalid, - "The json value for a struct type must be an object.", - )) - } - } - Type::List(list) => { - if let JsonValue::Array(array) = value { - Ok(Some(Literal::List( - array - .into_iter() - .map(|value| { - Literal::try_from_json(value, &list.element_field.field_type) - }) - .collect::>>()?, - ))) - } else { - Err(Error::new( - crate::ErrorKind::DataInvalid, - "The json value for a list type must be an array.", - )) - } - } - Type::Map(map) => { - if let JsonValue::Object(mut object) = value { - if let (Some(JsonValue::Array(keys)), Some(JsonValue::Array(values))) = - (object.remove("keys"), object.remove("values")) - { - Ok(Some(Literal::Map(Map::from_iter( - keys.into_iter() - .zip(values.into_iter()) - .map(|(key, value)| { - Ok(( - Literal::try_from_json(key, &map.key_field.field_type) - .and_then(|value| { - value.ok_or(Error::new( - ErrorKind::DataInvalid, - "Key of map cannot be null", - )) - })?, - Literal::try_from_json(value, &map.value_field.field_type)?, - )) - }) - .collect::>>()?, - )))) - } else { - Err(Error::new( - crate::ErrorKind::DataInvalid, - "The json value for a list type must be an array.", - )) - } - } else { - Err(Error::new( - crate::ErrorKind::DataInvalid, - "The json value for a list type must be an array.", - )) - } - } - } - } - - /// Converting iceberg value to json value. - /// - /// See [this spec](https://iceberg.apache.org/spec/#json-single-value-serialization) for reference. - pub fn try_into_json(self, r#type: &Type) -> Result { - match (self, r#type) { - (Literal::Primitive(prim), Type::Primitive(prim_type)) => match (prim_type, prim) { - (PrimitiveType::Boolean, PrimitiveLiteral::Boolean(val)) => { - Ok(JsonValue::Bool(val)) - } - (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => { - Ok(JsonValue::Number((val).into())) - } - (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => { - Ok(JsonValue::Number((val).into())) - } - (PrimitiveType::Float, PrimitiveLiteral::Float(val)) => { - match Number::from_f64(val.0 as f64) { - Some(number) => Ok(JsonValue::Number(number)), - None => Ok(JsonValue::Null), - } - } - (PrimitiveType::Double, PrimitiveLiteral::Double(val)) => { - match Number::from_f64(val.0) { - Some(number) => Ok(JsonValue::Number(number)), - None => Ok(JsonValue::Null), - } - } - (PrimitiveType::Date, PrimitiveLiteral::Int(val)) => { - Ok(JsonValue::String(date::days_to_date(val).to_string())) - } - (PrimitiveType::Time, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( - time::microseconds_to_time(val).to_string(), - )), - (PrimitiveType::Timestamp, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( - timestamp::microseconds_to_datetime(val) - .format("%Y-%m-%dT%H:%M:%S%.f") - .to_string(), - )), - (PrimitiveType::Timestamptz, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( - timestamptz::microseconds_to_datetimetz(val) - .format("%Y-%m-%dT%H:%M:%S%.f+00:00") - .to_string(), - )), - (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( - timestamp::nanoseconds_to_datetime(val) - .format("%Y-%m-%dT%H:%M:%S%.f") - .to_string(), - )), - (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(val)) => { - Ok(JsonValue::String( - timestamptz::nanoseconds_to_datetimetz(val) - .format("%Y-%m-%dT%H:%M:%S%.f+00:00") - .to_string(), - )) - } - (PrimitiveType::String, PrimitiveLiteral::String(val)) => { - Ok(JsonValue::String(val.clone())) - } - (_, PrimitiveLiteral::UInt128(val)) => { - Ok(JsonValue::String(Uuid::from_u128(val).to_string())) - } - (_, PrimitiveLiteral::Binary(val)) => Ok(JsonValue::String(val.iter().fold( - String::new(), - |mut acc, x| { - acc.push_str(&format!("{x:x}")); - acc - }, - ))), - (_, PrimitiveLiteral::Int128(val)) => match r#type { - Type::Primitive(PrimitiveType::Decimal { - precision: _precision, - scale, - }) => { - let decimal = Decimal::try_from_i128_with_scale(val, *scale)?; - Ok(JsonValue::String(decimal.to_string())) - } - _ => Err(Error::new( - ErrorKind::DataInvalid, - "The iceberg type for decimal literal must be decimal.", - ))?, - }, - _ => Err(Error::new( - ErrorKind::DataInvalid, - "The iceberg value doesn't fit to the iceberg type.", - )), - }, - (Literal::Struct(s), Type::Struct(struct_type)) => { - let mut id_and_value = Vec::with_capacity(struct_type.fields().len()); - for (value, field) in s.into_iter().zip(struct_type.fields()) { - let json = match value { - Some(val) => val.try_into_json(&field.field_type)?, - None => JsonValue::Null, - }; - id_and_value.push((field.id.to_string(), json)); - } - Ok(JsonValue::Object(JsonMap::from_iter(id_and_value))) - } - (Literal::List(list), Type::List(list_type)) => Ok(JsonValue::Array( - list.into_iter() - .map(|opt| match opt { - Some(literal) => literal.try_into_json(&list_type.element_field.field_type), - None => Ok(JsonValue::Null), - }) - .collect::>>()?, - )), - (Literal::Map(map), Type::Map(map_type)) => { - let mut object = JsonMap::with_capacity(2); - let mut json_keys = Vec::with_capacity(map.len()); - let mut json_values = Vec::with_capacity(map.len()); - for (key, value) in map.into_iter() { - json_keys.push(key.try_into_json(&map_type.key_field.field_type)?); - json_values.push(match value { - Some(literal) => literal.try_into_json(&map_type.value_field.field_type)?, - None => JsonValue::Null, - }); - } - object.insert("keys".to_string(), JsonValue::Array(json_keys)); - object.insert("values".to_string(), JsonValue::Array(json_values)); - Ok(JsonValue::Object(object)) - } - (value, r#type) => Err(Error::new( - ErrorKind::DataInvalid, - format!("The iceberg value {value:?} doesn't fit to the iceberg type {type}."), - )), - } - } - - /// Convert Value to the any type - pub fn into_any(self) -> Box { - match self { - Literal::Primitive(prim) => match prim { - PrimitiveLiteral::Boolean(any) => Box::new(any), - PrimitiveLiteral::Int(any) => Box::new(any), - PrimitiveLiteral::Long(any) => Box::new(any), - PrimitiveLiteral::Float(any) => Box::new(any), - PrimitiveLiteral::Double(any) => Box::new(any), - PrimitiveLiteral::Binary(any) => Box::new(any), - PrimitiveLiteral::String(any) => Box::new(any), - PrimitiveLiteral::UInt128(any) => Box::new(any), - PrimitiveLiteral::Int128(any) => Box::new(any), - PrimitiveLiteral::AboveMax | PrimitiveLiteral::BelowMin => unimplemented!(), - }, - _ => unimplemented!(), - } - } -} - -mod date { - use chrono::{DateTime, NaiveDate, TimeDelta, TimeZone, Utc}; - - pub(crate) fn date_to_days(date: &NaiveDate) -> i32 { - date.signed_duration_since( - // This is always the same and shouldn't fail - NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(), - ) - .num_days() as i32 - } - - pub(crate) fn days_to_date(days: i32) -> NaiveDate { - // This shouldn't fail until the year 262000 - (chrono::DateTime::UNIX_EPOCH + TimeDelta::try_days(days as i64).unwrap()) - .naive_utc() - .date() - } - - /// Returns unix epoch. - pub(crate) fn unix_epoch() -> DateTime { - Utc.timestamp_nanos(0) - } - - /// Creates date literal from `NaiveDate`, assuming it's utc timezone. - pub(crate) fn date_from_naive_date(date: NaiveDate) -> i32 { - (date - unix_epoch().date_naive()).num_days() as i32 - } -} - -mod time { - use chrono::NaiveTime; - - pub(crate) fn time_to_microseconds(time: &NaiveTime) -> i64 { - time.signed_duration_since( - // This is always the same and shouldn't fail - NaiveTime::from_num_seconds_from_midnight_opt(0, 0).unwrap(), - ) - .num_microseconds() - .unwrap() - } - - pub(crate) fn microseconds_to_time(micros: i64) -> NaiveTime { - let (secs, rem) = (micros / 1_000_000, micros % 1_000_000); - - NaiveTime::from_num_seconds_from_midnight_opt(secs as u32, rem as u32 * 1_000).unwrap() - } -} - -mod timestamp { - use chrono::{DateTime, NaiveDateTime}; - - pub(crate) fn datetime_to_microseconds(time: &NaiveDateTime) -> i64 { - time.and_utc().timestamp_micros() - } - - pub(crate) fn microseconds_to_datetime(micros: i64) -> NaiveDateTime { - // This shouldn't fail until the year 262000 - DateTime::from_timestamp_micros(micros).unwrap().naive_utc() - } - - pub(crate) fn nanoseconds_to_datetime(nanos: i64) -> NaiveDateTime { - DateTime::from_timestamp_nanos(nanos).naive_utc() - } -} - -mod timestamptz { - use chrono::{DateTime, Utc}; - - pub(crate) fn datetimetz_to_microseconds(time: &DateTime) -> i64 { - time.timestamp_micros() - } - - pub(crate) fn microseconds_to_datetimetz(micros: i64) -> DateTime { - let (secs, rem) = (micros / 1_000_000, micros % 1_000_000); - - DateTime::from_timestamp(secs, rem as u32 * 1_000).unwrap() - } - - pub(crate) fn nanoseconds_to_datetimetz(nanos: i64) -> DateTime { - let (secs, rem) = (nanos / 1_000_000_000, nanos % 1_000_000_000); - - DateTime::from_timestamp(secs, rem as u32).unwrap() - } -} - -mod _serde { - use serde::de::Visitor; - use serde::ser::{SerializeMap, SerializeSeq, SerializeStruct}; - use serde::{Deserialize, Serialize}; - use serde_bytes::ByteBuf; - use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive}; - - use super::{Literal, Map, PrimitiveLiteral}; - use crate::spec::{MAP_KEY_FIELD_NAME, MAP_VALUE_FIELD_NAME, PrimitiveType, Type}; - use crate::{Error, ErrorKind}; - - #[derive(SerializeDerive, DeserializeDerive, Debug)] - #[serde(transparent)] - /// Raw literal representation used for serde. The serialize way is used for Avro serializer. - pub struct RawLiteral(RawLiteralEnum); - - impl RawLiteral { - /// Covert literal to raw literal. - pub fn try_from(literal: Literal, ty: &Type) -> Result { - Ok(Self(RawLiteralEnum::try_from(literal, ty)?)) - } - - /// Convert raw literal to literal. - pub fn try_into(self, ty: &Type) -> Result, Error> { - self.0.try_into(ty) - } - } - - #[derive(SerializeDerive, Clone, Debug)] - #[serde(untagged)] - enum RawLiteralEnum { - Null, - Boolean(bool), - Int(i32), - Long(i64), - Float(f32), - Double(f64), - String(String), - Bytes(ByteBuf), - List(List), - StringMap(StringMap), - Record(Record), - } - - #[derive(Clone, Debug)] - struct Record { - required: Vec<(String, RawLiteralEnum)>, - optional: Vec<(String, Option)>, - } - - impl Serialize for Record { - fn serialize(&self, serializer: S) -> Result - where S: serde::Serializer { - let len = self.required.len() + self.optional.len(); - let mut record = serializer.serialize_struct("", len)?; - for (k, v) in &self.required { - record.serialize_field(Box::leak(k.clone().into_boxed_str()), &v)?; - } - for (k, v) in &self.optional { - record.serialize_field(Box::leak(k.clone().into_boxed_str()), &v)?; - } - record.end() - } - } - - #[derive(Clone, Debug)] - struct List { - list: Vec>, - required: bool, - } - - impl Serialize for List { - fn serialize(&self, serializer: S) -> Result - where S: serde::Serializer { - let mut seq = serializer.serialize_seq(Some(self.list.len()))?; - for value in &self.list { - if self.required { - seq.serialize_element(value.as_ref().ok_or_else(|| { - serde::ser::Error::custom( - "List element is required, element cannot be null", - ) - })?)?; - } else { - seq.serialize_element(&value)?; - } - } - seq.end() - } - } - - #[derive(Clone, Debug)] - struct StringMap { - raw: Vec<(String, Option)>, - required: bool, - } - - impl Serialize for StringMap { - fn serialize(&self, serializer: S) -> Result - where S: serde::Serializer { - let mut map = serializer.serialize_map(Some(self.raw.len()))?; - for (k, v) in &self.raw { - if self.required { - map.serialize_entry( - k, - v.as_ref().ok_or_else(|| { - serde::ser::Error::custom( - "Map element is required, element cannot be null", - ) - })?, - )?; - } else { - map.serialize_entry(k, v)?; - } - } - map.end() - } - } - - impl<'de> Deserialize<'de> for RawLiteralEnum { - fn deserialize(deserializer: D) -> Result - where D: serde::Deserializer<'de> { - struct RawLiteralVisitor; - impl<'de> Visitor<'de> for RawLiteralVisitor { - type Value = RawLiteralEnum; - - fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { - formatter.write_str("expect") - } - - fn visit_bool(self, v: bool) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Boolean(v)) - } - - fn visit_i32(self, v: i32) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Int(v)) - } - - fn visit_i64(self, v: i64) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Long(v)) - } - - /// Used in json - fn visit_u64(self, v: u64) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Long(v as i64)) - } - - fn visit_f32(self, v: f32) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Float(v)) - } - - fn visit_f64(self, v: f64) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Double(v)) - } - - fn visit_str(self, v: &str) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::String(v.to_string())) - } - - fn visit_bytes(self, v: &[u8]) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Bytes(ByteBuf::from(v))) - } - - fn visit_borrowed_str(self, v: &'de str) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::String(v.to_string())) - } - - fn visit_unit(self) -> Result - where E: serde::de::Error { - Ok(RawLiteralEnum::Null) - } - - fn visit_map(self, mut map: A) -> Result - where A: serde::de::MapAccess<'de> { - let mut required = Vec::new(); - while let Some(key) = map.next_key::()? { - let value = map.next_value::()?; - required.push((key, value)); - } - Ok(RawLiteralEnum::Record(Record { - required, - optional: Vec::new(), - })) - } - - fn visit_seq(self, mut seq: A) -> Result - where A: serde::de::SeqAccess<'de> { - let mut list = Vec::new(); - while let Some(value) = seq.next_element::()? { - list.push(Some(value)); - } - Ok(RawLiteralEnum::List(List { - list, - // `required` only used in serialize, just set default in deserialize. - required: false, - })) - } - } - deserializer.deserialize_any(RawLiteralVisitor) - } - } - - impl RawLiteralEnum { - pub fn try_from(literal: Literal, ty: &Type) -> Result { - let raw = match literal { - Literal::Primitive(prim) => match prim { - super::PrimitiveLiteral::Boolean(v) => RawLiteralEnum::Boolean(v), - super::PrimitiveLiteral::Int(v) => RawLiteralEnum::Int(v), - super::PrimitiveLiteral::Long(v) => RawLiteralEnum::Long(v), - super::PrimitiveLiteral::Float(v) => RawLiteralEnum::Float(v.0), - super::PrimitiveLiteral::Double(v) => RawLiteralEnum::Double(v.0), - super::PrimitiveLiteral::String(v) => RawLiteralEnum::String(v), - super::PrimitiveLiteral::UInt128(v) => { - RawLiteralEnum::Bytes(ByteBuf::from(v.to_be_bytes())) - } - super::PrimitiveLiteral::Binary(v) => RawLiteralEnum::Bytes(ByteBuf::from(v)), - super::PrimitiveLiteral::Int128(v) => { - RawLiteralEnum::Bytes(ByteBuf::from(v.to_be_bytes())) - } - super::PrimitiveLiteral::AboveMax | super::PrimitiveLiteral::BelowMin => { - return Err(Error::new( - ErrorKind::DataInvalid, - "Can't convert AboveMax or BelowMax", - )); - } - }, - Literal::Struct(r#struct) => { - let mut required = Vec::new(); - let mut optional = Vec::new(); - if let Type::Struct(struct_ty) = ty { - for (value, field) in r#struct.into_iter().zip(struct_ty.fields()) { - if field.required { - if let Some(value) = value { - required.push(( - field.name.clone(), - RawLiteralEnum::try_from(value, &field.field_type)?, - )); - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - "Can't convert null to required field", - )); - } - } else if let Some(value) = value { - optional.push(( - field.name.clone(), - Some(RawLiteralEnum::try_from(value, &field.field_type)?), - )); - } else { - optional.push((field.name.clone(), None)); - } - } - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!("Type {ty} should be a struct"), - )); - } - RawLiteralEnum::Record(Record { required, optional }) - } - Literal::List(list) => { - if let Type::List(list_ty) = ty { - let list = list - .into_iter() - .map(|v| { - v.map(|v| { - RawLiteralEnum::try_from(v, &list_ty.element_field.field_type) - }) - .transpose() - }) - .collect::>()?; - RawLiteralEnum::List(List { - list, - required: list_ty.element_field.required, - }) - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!("Type {ty} should be a list"), - )); - } - } - Literal::Map(map) => { - if let Type::Map(map_ty) = ty { - if let Type::Primitive(PrimitiveType::String) = *map_ty.key_field.field_type - { - let mut raw = Vec::with_capacity(map.len()); - for (k, v) in map { - if let Literal::Primitive(PrimitiveLiteral::String(k)) = k { - raw.push(( - k, - v.map(|v| { - RawLiteralEnum::try_from( - v, - &map_ty.value_field.field_type, - ) - }) - .transpose()?, - )); - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - "literal type is inconsistent with type", - )); - } - } - RawLiteralEnum::StringMap(StringMap { - raw, - required: map_ty.value_field.required, - }) - } else { - let list = map.into_iter().map(|(k,v)| { - let raw_k = - RawLiteralEnum::try_from(k, &map_ty.key_field.field_type)?; - let raw_v = v - .map(|v| { - RawLiteralEnum::try_from(v, &map_ty.value_field.field_type) - }) - .transpose()?; - if map_ty.value_field.required { - Ok(Some(RawLiteralEnum::Record(Record { - required: vec![ - (MAP_KEY_FIELD_NAME.to_string(), raw_k), - (MAP_VALUE_FIELD_NAME.to_string(), raw_v.ok_or_else(||Error::new(ErrorKind::DataInvalid, "Map value is required, value cannot be null"))?), - ], - optional: vec![], - }))) - } else { - Ok(Some(RawLiteralEnum::Record(Record { - required: vec![ - (MAP_KEY_FIELD_NAME.to_string(), raw_k), - ], - optional: vec![ - (MAP_VALUE_FIELD_NAME.to_string(), raw_v) - ], - }))) - } - }).collect::>()?; - RawLiteralEnum::List(List { - list, - required: true, - }) - } - } else { - return Err(Error::new( - ErrorKind::DataInvalid, - format!("Type {ty} should be a map"), - )); - } - } - }; - Ok(raw) - } - - pub fn try_into(self, ty: &Type) -> Result, Error> { - let invalid_err = |v: &str| { - Error::new( - ErrorKind::DataInvalid, - format!( - "Unable to convert raw literal ({v}) fail convert to type {ty} for: type mismatch" - ), - ) - }; - let invalid_err_with_reason = |v: &str, reason: &str| { - Error::new( - ErrorKind::DataInvalid, - format!( - "Unable to convert raw literal ({v}) fail convert to type {ty} for: {reason}" - ), - ) - }; - match self { - RawLiteralEnum::Null => Ok(None), - RawLiteralEnum::Boolean(v) => Ok(Some(Literal::bool(v))), - RawLiteralEnum::Int(v) => match ty { - Type::Primitive(PrimitiveType::Int) => Ok(Some(Literal::int(v))), - Type::Primitive(PrimitiveType::Long) => Ok(Some(Literal::long(i64::from(v)))), - Type::Primitive(PrimitiveType::Date) => Ok(Some(Literal::date(v))), - _ => Err(invalid_err("int")), - }, - RawLiteralEnum::Long(v) => match ty { - Type::Primitive(PrimitiveType::Int) => Ok(Some(Literal::int( - i32::try_from(v).map_err(|_| invalid_err("long"))?, - ))), - Type::Primitive(PrimitiveType::Date) => Ok(Some(Literal::date( - i32::try_from(v).map_err(|_| invalid_err("long"))?, - ))), - Type::Primitive(PrimitiveType::Long) => Ok(Some(Literal::long(v))), - Type::Primitive(PrimitiveType::Time) => Ok(Some(Literal::time(v))), - Type::Primitive(PrimitiveType::Timestamp) => Ok(Some(Literal::timestamp(v))), - Type::Primitive(PrimitiveType::Timestamptz) => { - Ok(Some(Literal::timestamptz(v))) - } - _ => Err(invalid_err("long")), - }, - RawLiteralEnum::Float(v) => match ty { - Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))), - Type::Primitive(PrimitiveType::Double) => { - Ok(Some(Literal::double(f64::from(v)))) - } - _ => Err(invalid_err("float")), - }, - RawLiteralEnum::Double(v) => match ty { - Type::Primitive(PrimitiveType::Float) => { - let v_32 = v as f32; - if v_32.is_finite() { - let v_64 = f64::from(v_32); - if (v_64 - v).abs() > f32::EPSILON as f64 { - // there is a precision loss - return Err(invalid_err("double")); - } - } - Ok(Some(Literal::float(v_32))) - } - Type::Primitive(PrimitiveType::Double) => Ok(Some(Literal::double(v))), - _ => Err(invalid_err("double")), - }, - RawLiteralEnum::String(v) => match ty { - Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))), - _ => Err(invalid_err("string")), - }, - RawLiteralEnum::Bytes(v) => match ty { - Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))), - Type::Primitive(PrimitiveType::Fixed(expected_len)) => { - if v.len() == *expected_len as usize { - Ok(Some(Literal::fixed(v.to_vec()))) - } else { - Err(invalid_err_with_reason( - "bytes", - &format!( - "Fixed type must be exactly {} bytes, got {}", - expected_len, - v.len() - ), - )) - } - } - Type::Primitive(PrimitiveType::Uuid) => { - if v.len() == 16 { - let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| { - invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes") - })?; - Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes)))) - } else { - Err(invalid_err_with_reason( - "bytes", - "UUID must be exactly 16 bytes", - )) - } - } - Type::Primitive(PrimitiveType::Decimal { precision, .. }) => { - let required_bytes = Type::decimal_required_bytes(*precision)? as usize; - - if v.len() == required_bytes { - // Pad the bytes to 16 bytes (i128 size) with sign extension - let mut padded_bytes = [0u8; 16]; - let start_idx = 16 - v.len(); - - // Copy the input bytes to the end of the array - padded_bytes[start_idx..].copy_from_slice(&v); - - // Sign extend if the number is negative (MSB is 1) - if !v.is_empty() && (v[0] & 0x80) != 0 { - // Fill the padding with 0xFF for negative numbers - for byte in &mut padded_bytes[..start_idx] { - *byte = 0xFF; - } - } - - Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( - i128::from_be_bytes(padded_bytes), - )))) - } else { - Err(invalid_err_with_reason( - "bytes", - &format!( - "Decimal with precision {} must be exactly {} bytes, got {}", - precision, - required_bytes, - v.len() - ), - )) - } - } - _ => Err(invalid_err("bytes")), - }, - RawLiteralEnum::List(v) => match ty { - Type::List(ty) => Ok(Some(Literal::List( - v.list - .into_iter() - .map(|v| { - if let Some(v) = v { - v.try_into(&ty.element_field.field_type) - } else { - Ok(None) - } - }) - .collect::>()?, - ))), - Type::Map(map_ty) => { - let key_ty = map_ty.key_field.field_type.as_ref(); - let value_ty = map_ty.value_field.field_type.as_ref(); - let mut map = Map::new(); - for k_v in v.list { - let k_v = k_v.ok_or_else(|| invalid_err_with_reason("list","In deserialize, None will be represented as Some(RawLiteral::Null), all element in list must be valid"))?; - if let RawLiteralEnum::Record(Record { - required, - optional: _, - }) = k_v - { - if required.len() != 2 { - return Err(invalid_err_with_reason( - "list", - "Record must contains two element(key and value) of array", - )); - } - let mut key = None; - let mut value = None; - required.into_iter().for_each(|(k, v)| { - if k == MAP_KEY_FIELD_NAME { - key = Some(v); - } else if k == MAP_VALUE_FIELD_NAME { - value = Some(v); - } - }); - match (key, value) { - (Some(k), Some(v)) => { - let key = k.try_into(key_ty)?.ok_or_else(|| { - invalid_err_with_reason( - "list", - "Key element in Map must be valid", - ) - })?; - let value = v.try_into(value_ty)?; - if map_ty.value_field.required && value.is_none() { - return Err(invalid_err_with_reason( - "list", - "Value element is required in this Map", - )); - } - map.insert(key, value); - } - _ => { - return Err(invalid_err_with_reason( - "list", - "The elements of record in list are not key and value", - )); - } - } - } else { - return Err(invalid_err_with_reason( - "list", - "Map should represented as record array.", - )); - } - } - Ok(Some(Literal::Map(map))) - } - Type::Primitive(PrimitiveType::Uuid) => { - if v.list.len() != 16 { - return Err(invalid_err_with_reason( - "list", - "The length of list should be 16", - )); - } - let mut bytes = [0u8; 16]; - for (i, v) in v.list.iter().enumerate() { - if let Some(RawLiteralEnum::Long(v)) = v { - bytes[i] = *v as u8; - } else { - return Err(invalid_err_with_reason( - "list", - "The element of list should be int", - )); - } - } - Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes)))) - } - Type::Primitive(PrimitiveType::Decimal { - precision: _, - scale: _, - }) => { - if v.list.len() != 16 { - return Err(invalid_err_with_reason( - "list", - "The length of list should be 16", - )); - } - let mut bytes = [0u8; 16]; - for (i, v) in v.list.iter().enumerate() { - if let Some(RawLiteralEnum::Long(v)) = v { - bytes[i] = *v as u8; - } else { - return Err(invalid_err_with_reason( - "list", - "The element of list should be int", - )); - } - } - Ok(Some(Literal::decimal(i128::from_be_bytes(bytes)))) - } - Type::Primitive(PrimitiveType::Binary) => { - let bytes = v - .list - .into_iter() - .map(|v| { - if let Some(RawLiteralEnum::Long(v)) = v { - Ok(v as u8) - } else { - Err(invalid_err_with_reason( - "list", - "The element of list should be int", - )) - } - }) - .collect::, Error>>()?; - Ok(Some(Literal::binary(bytes))) - } - Type::Primitive(PrimitiveType::Fixed(size)) => { - if v.list.len() != *size as usize { - return Err(invalid_err_with_reason( - "list", - "The length of list should be equal to size", - )); - } - let bytes = v - .list - .into_iter() - .map(|v| { - if let Some(RawLiteralEnum::Long(v)) = v { - Ok(v as u8) - } else { - Err(invalid_err_with_reason( - "list", - "The element of list should be int", - )) - } - }) - .collect::, Error>>()?; - Ok(Some(Literal::fixed(bytes))) - } - _ => Err(invalid_err("list")), - }, - RawLiteralEnum::Record(Record { - required, - optional: _, - }) => match ty { - Type::Struct(struct_ty) => { - let iters: Vec> = required - .into_iter() - .map(|(field_name, value)| { - let field = struct_ty - .field_by_name(field_name.as_str()) - .ok_or_else(|| { - invalid_err_with_reason( - "record", - &format!("field {} is not exist", &field_name), - ) - })?; - let value = value.try_into(&field.field_type)?; - Ok(value) - }) - .collect::>()?; - Ok(Some(Literal::Struct(super::Struct::from_iter(iters)))) - } - Type::Map(map_ty) => { - if *map_ty.key_field.field_type != Type::Primitive(PrimitiveType::String) { - return Err(invalid_err_with_reason( - "record", - "Map key must be string", - )); - } - let mut map = Map::new(); - for (k, v) in required { - let value = v.try_into(&map_ty.value_field.field_type)?; - if map_ty.value_field.required && value.is_none() { - return Err(invalid_err_with_reason( - "record", - "Value element is required in this Map", - )); - } - map.insert(Literal::string(k), value); - } - Ok(Some(Literal::Map(map))) - } - _ => Err(invalid_err("record")), - }, - RawLiteralEnum::StringMap(_) => Err(invalid_err("string map")), - } - } - } -} - -#[cfg(test)] -mod tests { - use apache_avro::to_value; - use apache_avro::types::Value; - - use super::*; - use crate::avro::schema_to_avro_schema; - use crate::spec::Schema; - use crate::spec::Type::Primitive; - use crate::spec::datatypes::{ListType, MapType, NestedField, StructType}; - - fn check_json_serde(json: &str, expected_literal: Literal, expected_type: &Type) { - let raw_json_value = serde_json::from_str::(json).unwrap(); - let desered_literal = - Literal::try_from_json(raw_json_value.clone(), expected_type).unwrap(); - assert_eq!(desered_literal, Some(expected_literal.clone())); - - let expected_json_value: JsonValue = expected_literal.try_into_json(expected_type).unwrap(); - let sered_json = serde_json::to_string(&expected_json_value).unwrap(); - let parsed_json_value = serde_json::from_str::(&sered_json).unwrap(); - - assert_eq!(parsed_json_value, raw_json_value); - } - - fn check_avro_bytes_serde( - input: Vec, - expected_datum: Datum, - expected_type: &PrimitiveType, - ) { - let raw_schema = r#""bytes""#; - let schema = apache_avro::Schema::parse_str(raw_schema).unwrap(); - - let bytes = ByteBuf::from(input); - let datum = Datum::try_from_bytes(&bytes, expected_type.clone()).unwrap(); - assert_eq!(datum, expected_datum); - - let mut writer = apache_avro::Writer::new(&schema, Vec::new()); - writer.append_ser(datum.to_bytes().unwrap()).unwrap(); - let encoded = writer.into_inner().unwrap(); - let reader = apache_avro::Reader::with_schema(&schema, &*encoded).unwrap(); - - for record in reader { - let result = apache_avro::from_value::(&record.unwrap()).unwrap(); - let desered_datum = Datum::try_from_bytes(&result, expected_type.clone()).unwrap(); - assert_eq!(desered_datum, expected_datum); - } - } - - fn check_convert_with_avro(expected_literal: Literal, expected_type: &Type) { - let fields = vec![NestedField::required(1, "col", expected_type.clone()).into()]; - let schema = Schema::builder() - .with_fields(fields.clone()) - .build() - .unwrap(); - let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); - let struct_type = Type::Struct(StructType::new(fields)); - let struct_literal = - Literal::Struct(Struct::from_iter(vec![Some(expected_literal.clone())])); - - let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); - let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); - writer.append_ser(raw_literal).unwrap(); - let encoded = writer.into_inner().unwrap(); - - let reader = apache_avro::Reader::new(&*encoded).unwrap(); - for record in reader { - let result = apache_avro::from_value::(&record.unwrap()).unwrap(); - let desered_literal = result.try_into(&struct_type).unwrap().unwrap(); - assert_eq!(desered_literal, struct_literal); - } - } - - fn check_serialize_avro(literal: Literal, ty: &Type, expect_value: Value) { - let expect_value = Value::Record(vec![("col".to_string(), expect_value)]); - - let fields = vec![NestedField::required(1, "col", ty.clone()).into()]; - let schema = Schema::builder() - .with_fields(fields.clone()) - .build() - .unwrap(); - let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); - let struct_type = Type::Struct(StructType::new(fields)); - let struct_literal = Literal::Struct(Struct::from_iter(vec![Some(literal.clone())])); - let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); - let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); - let value = to_value(raw_literal) - .unwrap() - .resolve(&avro_schema) - .unwrap(); - writer.append_value_ref(&value).unwrap(); - let encoded = writer.into_inner().unwrap(); - - let reader = apache_avro::Reader::new(&*encoded).unwrap(); - for record in reader { - assert_eq!(record.unwrap(), expect_value); - } - } - - #[test] - fn json_boolean() { - let record = r#"true"#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Boolean(true)), - &Type::Primitive(PrimitiveType::Boolean), - ); - } - - #[test] - fn json_int() { - let record = r#"32"#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Int(32)), - &Type::Primitive(PrimitiveType::Int), - ); - } - - #[test] - fn json_long() { - let record = r#"32"#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Long(32)), - &Type::Primitive(PrimitiveType::Long), - ); - } - - #[test] - fn json_float() { - let record = r#"1.0"#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(1.0))), - &Type::Primitive(PrimitiveType::Float), - ); - } - - #[test] - fn json_double() { - let record = r#"1.0"#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(1.0))), - &Type::Primitive(PrimitiveType::Double), - ); - } - - #[test] - fn json_date() { - let record = r#""2017-11-16""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Int(17486)), - &Type::Primitive(PrimitiveType::Date), - ); - } - - #[test] - fn json_time() { - let record = r#""22:31:08.123456""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Long(81068123456)), - &Type::Primitive(PrimitiveType::Time), - ); - } - - #[test] - fn json_timestamp() { - let record = r#""2017-11-16T22:31:08.123456""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), - &Type::Primitive(PrimitiveType::Timestamp), - ); - } - - #[test] - fn json_timestamptz() { - let record = r#""2017-11-16T22:31:08.123456+00:00""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), - &Type::Primitive(PrimitiveType::Timestamptz), - ); - } - - #[test] - fn json_string() { - let record = r#""iceberg""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::String("iceberg".to_string())), - &Type::Primitive(PrimitiveType::String), - ); - } - - #[test] - fn json_uuid() { - let record = r#""f79c3e09-677c-4bbd-a479-3f349cb785e7""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::UInt128( - Uuid::parse_str("f79c3e09-677c-4bbd-a479-3f349cb785e7") - .unwrap() - .as_u128(), - )), - &Type::Primitive(PrimitiveType::Uuid), - ); - } - - #[test] - fn json_decimal() { - let record = r#""14.20""#; - - check_json_serde( - record, - Literal::Primitive(PrimitiveLiteral::Int128(1420)), - &Type::decimal(28, 2).unwrap(), - ); - } - - #[test] - fn json_struct() { - let record = r#"{"1": 1, "2": "bar", "3": null}"#; - - check_json_serde( - record, - Literal::Struct(Struct::from_iter(vec![ - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - Some(Literal::Primitive(PrimitiveLiteral::String( - "bar".to_string(), - ))), - None, - ])), - &Type::Struct(StructType::new(vec![ - NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), - NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), - NestedField::optional(3, "address", Type::Primitive(PrimitiveType::String)).into(), - ])), - ); - } - - #[test] - fn json_list() { - let record = r#"[1, 2, 3, null]"#; - - check_json_serde( - record, - Literal::List(vec![ - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - Some(Literal::Primitive(PrimitiveLiteral::Int(3))), - None, - ]), - &Type::List(ListType { - element_field: NestedField::list_element( - 0, - Type::Primitive(PrimitiveType::Int), - true, - ) - .into(), - }), - ); - } - - #[test] - fn json_map() { - let record = r#"{ "keys": ["a", "b", "c"], "values": [1, 2, null] }"#; - - check_json_serde( - record, - Literal::Map(Map::from([ - ( - Literal::Primitive(PrimitiveLiteral::String("a".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("b".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("c".to_string())), - None, - ), - ])), - &Type::Map(MapType { - key_field: NestedField::map_key_element(0, Type::Primitive(PrimitiveType::String)) - .into(), - value_field: NestedField::map_value_element( - 1, - Type::Primitive(PrimitiveType::Int), - true, - ) - .into(), - }), - ); - } - - #[test] - fn avro_bytes_boolean() { - let bytes = vec![1u8]; - - check_avro_bytes_serde(bytes, Datum::bool(true), &PrimitiveType::Boolean); - } - - #[test] - fn avro_bytes_int() { - let bytes = vec![32u8, 0u8, 0u8, 0u8]; - - check_avro_bytes_serde(bytes, Datum::int(32), &PrimitiveType::Int); - } - - #[test] - fn avro_bytes_long() { - let bytes = vec![32u8, 0u8, 0u8, 0u8, 0u8, 0u8, 0u8, 0u8]; - - check_avro_bytes_serde(bytes, Datum::long(32), &PrimitiveType::Long); - } - - #[test] - fn avro_bytes_long_from_int() { - let bytes = vec![32u8, 0u8, 0u8, 0u8]; - - check_avro_bytes_serde(bytes, Datum::long(32), &PrimitiveType::Long); - } - - #[test] - fn avro_bytes_float() { - let bytes = vec![0u8, 0u8, 128u8, 63u8]; - - check_avro_bytes_serde(bytes, Datum::float(1.0), &PrimitiveType::Float); - } - - #[test] - fn avro_bytes_double() { - let bytes = vec![0u8, 0u8, 0u8, 0u8, 0u8, 0u8, 240u8, 63u8]; - - check_avro_bytes_serde(bytes, Datum::double(1.0), &PrimitiveType::Double); - } - - #[test] - fn avro_bytes_double_from_float() { - let bytes = vec![0u8, 0u8, 128u8, 63u8]; - - check_avro_bytes_serde(bytes, Datum::double(1.0), &PrimitiveType::Double); - } - - #[test] - fn avro_bytes_string() { - let bytes = vec![105u8, 99u8, 101u8, 98u8, 101u8, 114u8, 103u8]; - - check_avro_bytes_serde(bytes, Datum::string("iceberg"), &PrimitiveType::String); - } - - #[test] - fn avro_bytes_decimal() { - // (input_bytes, decimal_num, expect_scale, expect_precision) - let cases = vec![ - (vec![4u8, 210u8], 1234, 2, 38), - (vec![251u8, 46u8], -1234, 2, 38), - (vec![4u8, 210u8], 1234, 3, 38), - (vec![251u8, 46u8], -1234, 3, 38), - (vec![42u8], 42, 2, 1), - (vec![214u8], -42, 2, 1), - ]; - - for (input_bytes, decimal_num, expect_scale, expect_precision) in cases { - check_avro_bytes_serde( - input_bytes, - Datum::decimal_with_precision( - Decimal::new(decimal_num, expect_scale), - expect_precision, - ) - .unwrap(), - &PrimitiveType::Decimal { - precision: expect_precision, - scale: expect_scale, - }, - ); - } - } - - #[test] - fn avro_bytes_decimal_expect_error() { - // (decimal_num, expect_scale, expect_precision) - let cases = vec![(1234, 2, 1)]; - - for (decimal_num, expect_scale, expect_precision) in cases { - let result = Datum::decimal_with_precision( - Decimal::new(decimal_num, expect_scale), - expect_precision, - ); - assert!(result.is_err(), "expect error but got {result:?}"); - assert_eq!( - result.unwrap_err().kind(), - ErrorKind::DataInvalid, - "expect error DataInvalid", - ); - } - } - - fn check_raw_literal_bytes_serde_via_avro( - input_bytes: Vec, - expected_literal: Literal, - expected_type: &Type, - ) { - use apache_avro::types::Value; - - // Create an Avro bytes value and deserialize it through the RawLiteral path - let avro_value = Value::Bytes(input_bytes); - let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); - let result = raw_literal.try_into(expected_type).unwrap(); - assert_eq!(result, Some(expected_literal)); - } - - fn check_raw_literal_bytes_error_via_avro(input_bytes: Vec, expected_type: &Type) { - use apache_avro::types::Value; - - let avro_value = Value::Bytes(input_bytes); - let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); - let result = raw_literal.try_into(expected_type); - assert!(result.is_err(), "Expected error but got: {:?}", result); - } - - #[test] - fn test_raw_literal_bytes_binary() { - let bytes = vec![1u8, 2u8, 3u8, 4u8, 5u8]; - check_raw_literal_bytes_serde_via_avro( - bytes.clone(), - Literal::binary(bytes), - &Type::Primitive(PrimitiveType::Binary), - ); - } - - #[test] - fn test_raw_literal_bytes_binary_empty() { - let bytes = vec![]; - check_raw_literal_bytes_serde_via_avro( - bytes.clone(), - Literal::binary(bytes), - &Type::Primitive(PrimitiveType::Binary), - ); - } - - #[test] - fn test_raw_literal_bytes_fixed_correct_length() { - let bytes = vec![1u8, 2u8, 3u8, 4u8]; - check_raw_literal_bytes_serde_via_avro( - bytes.clone(), - Literal::fixed(bytes), - &Type::Primitive(PrimitiveType::Fixed(4)), - ); - } - - #[test] - fn test_raw_literal_bytes_fixed_wrong_length() { - let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but expecting 4 - check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Fixed(4))); - } - - #[test] - fn test_raw_literal_bytes_fixed_empty_correct_length() { - let bytes = vec![]; - check_raw_literal_bytes_serde_via_avro( - bytes.clone(), - Literal::fixed(bytes), - &Type::Primitive(PrimitiveType::Fixed(0)), - ); - } - - #[test] - fn test_raw_literal_bytes_uuid_correct_length() { - let uuid_bytes = vec![ - 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, - 0xcd, 0xef, - ]; - let expected_uuid = u128::from_be_bytes([ - 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, - 0xcd, 0xef, - ]); - check_raw_literal_bytes_serde_via_avro( - uuid_bytes, - Literal::Primitive(PrimitiveLiteral::UInt128(expected_uuid)), - &Type::Primitive(PrimitiveType::Uuid), - ); - } - - #[test] - fn test_raw_literal_bytes_uuid_wrong_length() { - let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but UUID needs 16 - check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Uuid)); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_4_scale_2() { - // Precision 4 requires 2 bytes - let decimal_bytes = vec![0x04, 0xd2]; // 1234 in 2 bytes - let expected_decimal = 1234i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 4, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_4_negative() { - // Precision 4 requires 2 bytes, negative number - let decimal_bytes = vec![0xfb, 0x2e]; // -1234 in 2 bytes - let expected_decimal = -1234i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 4, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_9_scale_2() { - // Precision 9 requires 4 bytes - let decimal_bytes = vec![0x00, 0x12, 0xd6, 0x87]; // 1234567 in 4 bytes - let expected_decimal = 1234567i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 9, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_18_scale_2() { - // Precision 18 requires 8 bytes - let decimal_bytes = vec![0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0xd2]; // 1234 in 8 bytes - let expected_decimal = 1234i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 18, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_38_scale_2() { - // Precision 38 requires 16 bytes (maximum precision) - let decimal_bytes = vec![ - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x04, 0xd2, // 1234 in 16 bytes - ]; - let expected_decimal = 1234i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 38, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_1_scale_0() { - // Precision 1 requires 1 byte - let decimal_bytes = vec![0x07]; // 7 in 1 byte - let expected_decimal = 7i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 1, - scale: 0, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_precision_1_negative() { - // Precision 1 requires 1 byte, negative number - let decimal_bytes = vec![0xf9]; // -7 in 1 byte (two's complement) - let expected_decimal = -7i128; - check_raw_literal_bytes_serde_via_avro( - decimal_bytes, - Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), - &Type::Primitive(PrimitiveType::Decimal { - precision: 1, - scale: 0, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_wrong_length() { - // 3 bytes provided, but precision 4 requires 2 bytes - let bytes = vec![1u8, 2u8, 3u8]; - check_raw_literal_bytes_error_via_avro( - bytes, - &Type::Primitive(PrimitiveType::Decimal { - precision: 4, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_decimal_wrong_length_too_few() { - // 1 byte provided, but precision 9 requires 4 bytes - let bytes = vec![0x42]; - check_raw_literal_bytes_error_via_avro( - bytes, - &Type::Primitive(PrimitiveType::Decimal { - precision: 9, - scale: 2, - }), - ); - } - - #[test] - fn test_raw_literal_bytes_unsupported_type() { - let bytes = vec![1u8, 2u8, 3u8, 4u8]; - check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Int)); - } - - #[test] - fn avro_convert_test_int() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Int(32)), - &Type::Primitive(PrimitiveType::Int), - ); - } - - #[test] - fn avro_convert_test_long() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Long(32)), - &Type::Primitive(PrimitiveType::Long), - ); - } - - #[test] - fn avro_convert_test_float() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(1.0))), - &Type::Primitive(PrimitiveType::Float), - ); - } - - #[test] - fn avro_convert_test_double() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(1.0))), - &Type::Primitive(PrimitiveType::Double), - ); - } - - #[test] - fn avro_convert_test_string() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::String("iceberg".to_string())), - &Type::Primitive(PrimitiveType::String), - ); - } - - #[test] - fn avro_convert_test_date() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Int(17486)), - &Type::Primitive(PrimitiveType::Date), - ); - } - - #[test] - fn avro_convert_test_time() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Long(81068123456)), - &Type::Primitive(PrimitiveType::Time), - ); - } - - #[test] - fn avro_convert_test_timestamp() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), - &Type::Primitive(PrimitiveType::Timestamp), - ); - } - - #[test] - fn avro_convert_test_timestamptz() { - check_convert_with_avro( - Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), - &Type::Primitive(PrimitiveType::Timestamptz), - ); - } - - #[test] - fn avro_convert_test_list() { - check_convert_with_avro( - Literal::List(vec![ - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - Some(Literal::Primitive(PrimitiveLiteral::Int(3))), - None, - ]), - &Type::List(ListType { - element_field: NestedField::list_element( - 0, - Type::Primitive(PrimitiveType::Int), - false, - ) - .into(), - }), - ); - - check_convert_with_avro( - Literal::List(vec![ - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - Some(Literal::Primitive(PrimitiveLiteral::Int(3))), - ]), - &Type::List(ListType { - element_field: NestedField::list_element( - 0, - Type::Primitive(PrimitiveType::Int), - true, - ) - .into(), - }), - ); - } - - fn check_convert_with_avro_map(expected_literal: Literal, expected_type: &Type) { - let fields = vec![NestedField::required(1, "col", expected_type.clone()).into()]; - let schema = Schema::builder() - .with_fields(fields.clone()) - .build() - .unwrap(); - let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); - let struct_type = Type::Struct(StructType::new(fields)); - let struct_literal = - Literal::Struct(Struct::from_iter(vec![Some(expected_literal.clone())])); - - let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); - let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); - writer.append_ser(raw_literal).unwrap(); - let encoded = writer.into_inner().unwrap(); - - let reader = apache_avro::Reader::new(&*encoded).unwrap(); - for record in reader { - let result = apache_avro::from_value::(&record.unwrap()).unwrap(); - let desered_literal = result.try_into(&struct_type).unwrap().unwrap(); - match (&desered_literal, &struct_literal) { - (Literal::Struct(desered), Literal::Struct(expected)) => { - match (&desered.fields[0], &expected.fields[0]) { - (Some(Literal::Map(desered)), Some(Literal::Map(expected))) => { - assert!(desered.has_same_content(expected)) - } - _ => { - unreachable!() - } - } - } - _ => { - panic!("unexpected literal type"); - } - } - } - } - - #[test] - fn avro_convert_test_map() { - check_convert_with_avro_map( - Literal::Map(Map::from([ - ( - Literal::Primitive(PrimitiveLiteral::Int(1)), - Some(Literal::Primitive(PrimitiveLiteral::Long(1))), - ), - ( - Literal::Primitive(PrimitiveLiteral::Int(2)), - Some(Literal::Primitive(PrimitiveLiteral::Long(2))), - ), - (Literal::Primitive(PrimitiveLiteral::Int(3)), None), - ])), - &Type::Map(MapType { - key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::Int)) - .into(), - value_field: NestedField::map_value_element( - 3, - Type::Primitive(PrimitiveType::Long), - false, - ) - .into(), - }), - ); - - check_convert_with_avro_map( - Literal::Map(Map::from([ - ( - Literal::Primitive(PrimitiveLiteral::Int(1)), - Some(Literal::Primitive(PrimitiveLiteral::Long(1))), - ), - ( - Literal::Primitive(PrimitiveLiteral::Int(2)), - Some(Literal::Primitive(PrimitiveLiteral::Long(2))), - ), - ( - Literal::Primitive(PrimitiveLiteral::Int(3)), - Some(Literal::Primitive(PrimitiveLiteral::Long(3))), - ), - ])), - &Type::Map(MapType { - key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::Int)) - .into(), - value_field: NestedField::map_value_element( - 3, - Type::Primitive(PrimitiveType::Long), - true, - ) - .into(), - }), - ); - } - - #[test] - fn avro_convert_test_string_map() { - check_convert_with_avro_map( - Literal::Map(Map::from([ - ( - Literal::Primitive(PrimitiveLiteral::String("a".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("b".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("c".to_string())), - None, - ), - ])), - &Type::Map(MapType { - key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::String)) - .into(), - value_field: NestedField::map_value_element( - 3, - Type::Primitive(PrimitiveType::Int), - false, - ) - .into(), - }), - ); - - check_convert_with_avro_map( - Literal::Map(Map::from([ - ( - Literal::Primitive(PrimitiveLiteral::String("a".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("b".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(2))), - ), - ( - Literal::Primitive(PrimitiveLiteral::String("c".to_string())), - Some(Literal::Primitive(PrimitiveLiteral::Int(3))), - ), - ])), - &Type::Map(MapType { - key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::String)) - .into(), - value_field: NestedField::map_value_element( - 3, - Type::Primitive(PrimitiveType::Int), - true, - ) - .into(), - }), - ); - } - - #[test] - fn avro_convert_test_record() { - check_convert_with_avro( - Literal::Struct(Struct::from_iter(vec![ - Some(Literal::Primitive(PrimitiveLiteral::Int(1))), - Some(Literal::Primitive(PrimitiveLiteral::String( - "bar".to_string(), - ))), - None, - ])), - &Type::Struct(StructType::new(vec![ - NestedField::required(2, "id", Type::Primitive(PrimitiveType::Int)).into(), - NestedField::optional(3, "name", Type::Primitive(PrimitiveType::String)).into(), - NestedField::optional(4, "address", Type::Primitive(PrimitiveType::String)).into(), - ])), - ); - } - - // # TODO:https://github.com/apache/iceberg-rust/issues/86 - // rust avro don't support deserialize any bytes representation now: - // - binary - // - decimal - #[test] - fn avro_convert_test_binary_ser() { - let literal = Literal::Primitive(PrimitiveLiteral::Binary(vec![1, 2, 3, 4, 5])); - let ty = Type::Primitive(PrimitiveType::Binary); - let expect_value = Value::Bytes(vec![1, 2, 3, 4, 5]); - check_serialize_avro(literal, &ty, expect_value); - } - - #[test] - fn avro_convert_test_decimal_ser() { - let literal = Literal::decimal(12345); - let ty = Type::Primitive(PrimitiveType::Decimal { - precision: 9, - scale: 8, - }); - let expect_value = Value::Decimal(apache_avro::Decimal::from(12345_i128.to_be_bytes())); - check_serialize_avro(literal, &ty, expect_value); - } - - // # TODO:https://github.com/apache/iceberg-rust/issues/86 - // rust avro can't support to convert any byte-like type to fixed in avro now. - // - uuid ser/de - // - fixed ser/de - - #[test] - fn test_parse_timestamp() { - let value = Datum::timestamp_from_str("2021-08-01T01:09:00.0899").unwrap(); - assert_eq!(&format!("{value}"), "2021-08-01 01:09:00.089900"); - - let value = Datum::timestamp_from_str("2023-01-06T00:00:00").unwrap(); - assert_eq!(&format!("{value}"), "2023-01-06 00:00:00"); - - let value = Datum::timestamp_from_str("2021-08-01T01:09:00.0899+0800"); - assert!(value.is_err(), "Parse timestamp with timezone should fail!"); - - let value = Datum::timestamp_from_str("dfa"); - assert!( - value.is_err(), - "Parse timestamp with invalid input should fail!" - ); - } - - #[test] - fn test_parse_timestamptz() { - let value = Datum::timestamptz_from_str("2021-08-01T09:09:00.0899+0800").unwrap(); - assert_eq!(&format!("{value}"), "2021-08-01 01:09:00.089900 UTC"); - - let value = Datum::timestamptz_from_str("2021-08-01T01:09:00.0899"); - assert!( - value.is_err(), - "Parse timestamptz without timezone should fail!" - ); - - let value = Datum::timestamptz_from_str("dfa"); - assert!( - value.is_err(), - "Parse timestamptz with invalid input should fail!" - ); - } - - #[test] - fn test_datum_ser_deser() { - let test_fn = |datum: Datum| { - let json = serde_json::to_value(&datum).unwrap(); - let desered_datum: Datum = serde_json::from_value(json).unwrap(); - assert_eq!(datum, desered_datum); - }; - let datum = Datum::int(1); - test_fn(datum); - let datum = Datum::long(1); - test_fn(datum); - - let datum = Datum::float(1.0); - test_fn(datum); - let datum = Datum::float(0_f32); - test_fn(datum); - let datum = Datum::float(-0_f32); - test_fn(datum); - let datum = Datum::float(f32::MAX); - test_fn(datum); - let datum = Datum::float(f32::MIN); - test_fn(datum); - - // serde_json can't serialize f32::INFINITY, f32::NEG_INFINITY, f32::NAN - let datum = Datum::float(f32::INFINITY); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - let datum = Datum::float(f32::NEG_INFINITY); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - let datum = Datum::float(f32::NAN); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - - let datum = Datum::double(1.0); - test_fn(datum); - let datum = Datum::double(f64::MAX); - test_fn(datum); - let datum = Datum::double(f64::MIN); - test_fn(datum); - - // serde_json can't serialize f32::INFINITY, f32::NEG_INFINITY, f32::NAN - let datum = Datum::double(f64::INFINITY); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - let datum = Datum::double(f64::NEG_INFINITY); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - let datum = Datum::double(f64::NAN); - let json = serde_json::to_string(&datum).unwrap(); - assert!(serde_json::from_str::(&json).is_err()); - - let datum = Datum::string("iceberg"); - test_fn(datum); - let datum = Datum::bool(true); - test_fn(datum); - let datum = Datum::date(17486); - test_fn(datum); - let datum = Datum::time_from_hms_micro(22, 15, 33, 111).unwrap(); - test_fn(datum); - let datum = Datum::timestamp_micros(1510871468123456); - test_fn(datum); - let datum = Datum::timestamptz_micros(1510871468123456); - test_fn(datum); - let datum = Datum::uuid(Uuid::parse_str("f79c3e09-677c-4bbd-a479-3f349cb785e7").unwrap()); - test_fn(datum); - let datum = Datum::decimal(1420).unwrap(); - test_fn(datum); - let datum = Datum::binary(vec![1, 2, 3, 4, 5]); - test_fn(datum); - let datum = Datum::fixed(vec![1, 2, 3, 4, 5]); - test_fn(datum); - } - - #[test] - fn test_datum_date_convert_to_int() { - let datum_date = Datum::date(12345); - - let result = datum_date.to(&Primitive(PrimitiveType::Int)).unwrap(); - - let expected = Datum::int(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_int_convert_to_date() { - let datum_int = Datum::int(12345); - - let result = datum_int.to(&Primitive(PrimitiveType::Date)).unwrap(); - - let expected = Datum::date(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_long_convert_to_int() { - let datum = Datum::long(12345); - - let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); - - let expected = Datum::int(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_long_convert_to_int_above_max() { - let datum = Datum::long(INT_MAX as i64 + 1); - - let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); - - let expected = Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_long_convert_to_int_below_min() { - let datum = Datum::long(INT_MIN as i64 - 1); - - let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); - - let expected = Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_long_convert_to_timestamp() { - let datum = Datum::long(12345); - - let result = datum.to(&Primitive(PrimitiveType::Timestamp)).unwrap(); - - let expected = Datum::timestamp_micros(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_long_convert_to_timestamptz() { - let datum = Datum::long(12345); - - let result = datum.to(&Primitive(PrimitiveType::Timestamptz)).unwrap(); - - let expected = Datum::timestamptz_micros(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_decimal_convert_to_long() { - let datum = Datum::decimal(12345).unwrap(); - - let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); - - let expected = Datum::long(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_decimal_convert_to_long_above_max() { - let datum = Datum::decimal(LONG_MAX as i128 + 1).unwrap(); - - let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); - - let expected = Datum::new(PrimitiveType::Long, PrimitiveLiteral::AboveMax); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_decimal_convert_to_long_below_min() { - let datum = Datum::decimal(LONG_MIN as i128 - 1).unwrap(); - - let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); - - let expected = Datum::new(PrimitiveType::Long, PrimitiveLiteral::BelowMin); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_string_convert_to_boolean() { - let datum = Datum::string("true"); - - let result = datum.to(&Primitive(PrimitiveType::Boolean)).unwrap(); - - let expected = Datum::bool(true); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_string_convert_to_int() { - let datum = Datum::string("12345"); - - let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); - - let expected = Datum::int(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_string_convert_to_long() { - let datum = Datum::string("12345"); - - let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); - - let expected = Datum::long(12345); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_string_convert_to_timestamp() { - let datum = Datum::string("1925-05-20T19:25:00.000"); - - let result = datum.to(&Primitive(PrimitiveType::Timestamp)).unwrap(); - - let expected = Datum::timestamp_micros(-1407990900000000); - - assert_eq!(result, expected); - } - - #[test] - fn test_datum_string_convert_to_timestamptz() { - let datum = Datum::string("1925-05-20T19:25:00.000 UTC"); - - let result = datum.to(&Primitive(PrimitiveType::Timestamptz)).unwrap(); - - let expected = Datum::timestamptz_micros(-1407990900000000); - - assert_eq!(result, expected); - } - - #[test] - fn test_iceberg_float_order() { - // Test float ordering - let float_values = vec![ - Datum::float(f32::NAN), - Datum::float(-f32::NAN), - Datum::float(f32::MAX), - Datum::float(f32::MIN), - Datum::float(f32::INFINITY), - Datum::float(-f32::INFINITY), - Datum::float(1.0), - Datum::float(-1.0), - Datum::float(0.0), - Datum::float(-0.0), - ]; - - let mut float_sorted = float_values.clone(); - float_sorted.sort_by(|a, b| a.partial_cmp(b).unwrap()); - - let float_expected = vec![ - Datum::float(-f32::NAN), - Datum::float(-f32::INFINITY), - Datum::float(f32::MIN), - Datum::float(-1.0), - Datum::float(-0.0), - Datum::float(0.0), - Datum::float(1.0), - Datum::float(f32::MAX), - Datum::float(f32::INFINITY), - Datum::float(f32::NAN), - ]; - - assert_eq!(float_sorted, float_expected); - - // Test double ordering - let double_values = vec![ - Datum::double(f64::NAN), - Datum::double(-f64::NAN), - Datum::double(f64::INFINITY), - Datum::double(-f64::INFINITY), - Datum::double(f64::MAX), - Datum::double(f64::MIN), - Datum::double(1.0), - Datum::double(-1.0), - Datum::double(0.0), - Datum::double(-0.0), - ]; - - let mut double_sorted = double_values.clone(); - double_sorted.sort_by(|a, b| a.partial_cmp(b).unwrap()); - - let double_expected = vec![ - Datum::double(-f64::NAN), - Datum::double(-f64::INFINITY), - Datum::double(f64::MIN), - Datum::double(-1.0), - Datum::double(-0.0), - Datum::double(0.0), - Datum::double(1.0), - Datum::double(f64::MAX), - Datum::double(f64::INFINITY), - Datum::double(f64::NAN), - ]; - - assert_eq!(double_sorted, double_expected); - } - - /// Test Date deserialization from JSON as number (days since epoch). - /// - /// This reproduces the scenario from Iceberg Java's TestAddFilesProcedure where: - /// - Date partition columns have initial_default values in manifests - /// - These values are serialized as days since epoch (e.g., 18628 for 2021-01-01) - /// - The JSON schema includes: {"type":"date","initial-default":18628} - /// - /// Prior to this fix, Date values in JSON were only parsed from String format ("2021-01-01"), - /// causing initial_default values to be lost during schema deserialization. - /// - /// This test ensures both formats are supported: - /// - String format: "2021-01-01" (used in table metadata) - /// - Number format: 18628 (used in initial-default values from add_files) - /// - /// See: Iceberg Java TestAddFilesProcedure.addDataPartitionedByDateToPartitioned() - #[test] - fn test_date_from_json_as_number() { - use serde_json::json; - - // Test Date as number (days since epoch) - used in initial-default from add_files - let date_number = json!(18628); // 2021-01-01 is 18628 days since 1970-01-01 - let result = - Literal::try_from_json(date_number, &Type::Primitive(PrimitiveType::Date)).unwrap(); - assert_eq!( - result, - Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) - ); - - // Test Date as string - traditional format - let date_string = json!("2021-01-01"); - let result = - Literal::try_from_json(date_string, &Type::Primitive(PrimitiveType::Date)).unwrap(); - assert_eq!( - result, - Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) - ); - - // Both formats should produce the same Literal value - } -} diff --git a/crates/iceberg/src/spec/values/datum.rs b/crates/iceberg/src/spec/values/datum.rs new file mode 100644 index 0000000000..cb60fb94e9 --- /dev/null +++ b/crates/iceberg/src/spec/values/datum.rs @@ -0,0 +1,1225 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Typed literals with validation + +use std::cmp::Ordering; +use std::fmt::{Display, Formatter}; +use std::str::FromStr; + +use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeZone, Utc}; +use num_bigint::BigInt; +use ordered_float::{Float, OrderedFloat}; +use rust_decimal::Decimal; +use rust_decimal::prelude::ToPrimitive; +use serde::de::{self, MapAccess}; +use serde::ser::SerializeStruct; +use serde::{Deserialize, Serialize}; +use serde_bytes::ByteBuf; + +use super::literal::Literal; +use super::primitive::PrimitiveLiteral; +use super::serde::_serde::RawLiteral; +use super::temporal::{date, time, timestamp, timestamptz}; +use crate::error::Result; +use crate::spec::MAX_DECIMAL_PRECISION; +use crate::spec::datatypes::{PrimitiveType, Type}; +use crate::{Error, ErrorKind, ensure_data_valid}; + +/// Maximum value for [`PrimitiveType::Time`] type in microseconds, e.g. 23 hours 59 minutes 59 seconds 999999 microseconds. +pub(crate) const MAX_TIME_VALUE: i64 = 24 * 60 * 60 * 1_000_000i64 - 1; + +pub(crate) const INT_MAX: i32 = 2147483647; +pub(crate) const INT_MIN: i32 = -2147483648; +pub(crate) const LONG_MAX: i64 = 9223372036854775807; +pub(crate) const LONG_MIN: i64 = -9223372036854775808; + +/// Literal associated with its type. The value and type pair is checked when construction, so the type and value is +/// guaranteed to be correct when used. +/// +/// By default, we decouple the type and value of a literal, so we can use avoid the cost of storing extra type info +/// for each literal. But associate type with literal can be useful in some cases, for example, in unbound expression. +#[derive(Clone, Debug, PartialEq, Hash, Eq)] +pub struct Datum { + r#type: PrimitiveType, + literal: PrimitiveLiteral, +} + +impl Serialize for Datum { + fn serialize( + &self, + serializer: S, + ) -> std::result::Result { + let mut struct_ser = serializer + .serialize_struct("Datum", 2) + .map_err(serde::ser::Error::custom)?; + struct_ser + .serialize_field("type", &self.r#type) + .map_err(serde::ser::Error::custom)?; + struct_ser + .serialize_field( + "literal", + &RawLiteral::try_from( + Literal::Primitive(self.literal.clone()), + &Type::Primitive(self.r#type.clone()), + ) + .map_err(serde::ser::Error::custom)?, + ) + .map_err(serde::ser::Error::custom)?; + struct_ser.end() + } +} + +impl<'de> Deserialize<'de> for Datum { + fn deserialize>( + deserializer: D, + ) -> std::result::Result { + #[derive(Deserialize)] + #[serde(field_identifier, rename_all = "lowercase")] + enum Field { + Type, + Literal, + } + + struct DatumVisitor; + + impl<'de> serde::de::Visitor<'de> for DatumVisitor { + type Value = Datum; + + fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { + formatter.write_str("struct Datum") + } + + fn visit_seq(self, mut seq: A) -> std::result::Result + where A: serde::de::SeqAccess<'de> { + let r#type = seq + .next_element::()? + .ok_or_else(|| serde::de::Error::invalid_length(0, &self))?; + let value = seq + .next_element::()? + .ok_or_else(|| serde::de::Error::invalid_length(1, &self))?; + let Literal::Primitive(primitive) = value + .try_into(&Type::Primitive(r#type.clone())) + .map_err(serde::de::Error::custom)? + .ok_or_else(|| serde::de::Error::custom("None value"))? + else { + return Err(serde::de::Error::custom("Invalid value")); + }; + + Ok(Datum::new(r#type, primitive)) + } + + fn visit_map(self, mut map: V) -> std::result::Result + where V: MapAccess<'de> { + let mut raw_primitive: Option = None; + let mut r#type: Option = None; + while let Some(key) = map.next_key()? { + match key { + Field::Type => { + if r#type.is_some() { + return Err(de::Error::duplicate_field("type")); + } + r#type = Some(map.next_value()?); + } + Field::Literal => { + if raw_primitive.is_some() { + return Err(de::Error::duplicate_field("literal")); + } + raw_primitive = Some(map.next_value()?); + } + } + } + let Some(r#type) = r#type else { + return Err(serde::de::Error::missing_field("type")); + }; + let Some(raw_primitive) = raw_primitive else { + return Err(serde::de::Error::missing_field("literal")); + }; + let Literal::Primitive(primitive) = raw_primitive + .try_into(&Type::Primitive(r#type.clone())) + .map_err(serde::de::Error::custom)? + .ok_or_else(|| serde::de::Error::custom("None value"))? + else { + return Err(serde::de::Error::custom("Invalid value")); + }; + Ok(Datum::new(r#type, primitive)) + } + } + const FIELDS: &[&str] = &["type", "literal"]; + deserializer.deserialize_struct("Datum", FIELDS, DatumVisitor) + } +} + +// Compare following iceberg float ordering rules: +// -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN +fn iceberg_float_cmp(a: T, b: T) -> Option { + if a.is_nan() && b.is_nan() { + return match (a.is_sign_negative(), b.is_sign_negative()) { + (true, false) => Some(Ordering::Less), + (false, true) => Some(Ordering::Greater), + _ => Some(Ordering::Equal), + }; + } + + if a.is_nan() { + return Some(if a.is_sign_negative() { + Ordering::Less + } else { + Ordering::Greater + }); + } + + if b.is_nan() { + return Some(if b.is_sign_negative() { + Ordering::Greater + } else { + Ordering::Less + }); + } + + a.partial_cmp(&b) +} + +impl PartialOrd for Datum { + fn partial_cmp(&self, other: &Self) -> Option { + match (&self.literal, &other.literal, &self.r#type, &other.r#type) { + // generate the arm with same type and same literal + ( + PrimitiveLiteral::Boolean(val), + PrimitiveLiteral::Boolean(other_val), + PrimitiveType::Boolean, + PrimitiveType::Boolean, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Int(val), + PrimitiveLiteral::Int(other_val), + PrimitiveType::Int, + PrimitiveType::Int, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::Long, + PrimitiveType::Long, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Float(val), + PrimitiveLiteral::Float(other_val), + PrimitiveType::Float, + PrimitiveType::Float, + ) => iceberg_float_cmp(*val, *other_val), + ( + PrimitiveLiteral::Double(val), + PrimitiveLiteral::Double(other_val), + PrimitiveType::Double, + PrimitiveType::Double, + ) => iceberg_float_cmp(*val, *other_val), + ( + PrimitiveLiteral::Int(val), + PrimitiveLiteral::Int(other_val), + PrimitiveType::Date, + PrimitiveType::Date, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::Time, + PrimitiveType::Time, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::Timestamp, + PrimitiveType::Timestamp, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::Timestamptz, + PrimitiveType::Timestamptz, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::String(val), + PrimitiveLiteral::String(other_val), + PrimitiveType::String, + PrimitiveType::String, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::UInt128(val), + PrimitiveLiteral::UInt128(other_val), + PrimitiveType::Uuid, + PrimitiveType::Uuid, + ) => uuid::Uuid::from_u128(*val).partial_cmp(&uuid::Uuid::from_u128(*other_val)), + ( + PrimitiveLiteral::Binary(val), + PrimitiveLiteral::Binary(other_val), + PrimitiveType::Fixed(_), + PrimitiveType::Fixed(_), + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Binary(val), + PrimitiveLiteral::Binary(other_val), + PrimitiveType::Binary, + PrimitiveType::Binary, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Int128(val), + PrimitiveLiteral::Int128(other_val), + PrimitiveType::Decimal { + precision: _, + scale, + }, + PrimitiveType::Decimal { + precision: _, + scale: other_scale, + }, + ) => { + let val = Decimal::from_i128_with_scale(*val, *scale); + let other_val = Decimal::from_i128_with_scale(*other_val, *other_scale); + val.partial_cmp(&other_val) + } + _ => None, + } + } +} + +impl Display for Datum { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + match (&self.r#type, &self.literal) { + (_, PrimitiveLiteral::Boolean(val)) => write!(f, "{val}"), + (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => write!(f, "{val}"), + (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => write!(f, "{val}"), + (_, PrimitiveLiteral::Float(val)) => write!(f, "{val}"), + (_, PrimitiveLiteral::Double(val)) => write!(f, "{val}"), + (PrimitiveType::Date, PrimitiveLiteral::Int(val)) => { + write!(f, "{}", date::days_to_date(*val)) + } + (PrimitiveType::Time, PrimitiveLiteral::Long(val)) => { + write!(f, "{}", time::microseconds_to_time(*val)) + } + (PrimitiveType::Timestamp, PrimitiveLiteral::Long(val)) => { + write!(f, "{}", timestamp::microseconds_to_datetime(*val)) + } + (PrimitiveType::Timestamptz, PrimitiveLiteral::Long(val)) => { + write!(f, "{}", timestamptz::microseconds_to_datetimetz(*val)) + } + (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(val)) => { + write!(f, "{}", timestamp::nanoseconds_to_datetime(*val)) + } + (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(val)) => { + write!(f, "{}", timestamptz::nanoseconds_to_datetimetz(*val)) + } + (_, PrimitiveLiteral::String(val)) => write!(f, r#""{val}""#), + (PrimitiveType::Uuid, PrimitiveLiteral::UInt128(val)) => { + write!(f, "{}", uuid::Uuid::from_u128(*val)) + } + (_, PrimitiveLiteral::Binary(val)) => display_bytes(val, f), + ( + PrimitiveType::Decimal { + precision: _, + scale, + }, + PrimitiveLiteral::Int128(val), + ) => { + write!(f, "{}", Decimal::from_i128_with_scale(*val, *scale)) + } + (_, _) => { + unreachable!() + } + } + } +} + +fn display_bytes(bytes: &[u8], f: &mut Formatter<'_>) -> std::fmt::Result { + let mut s = String::with_capacity(bytes.len() * 2); + for b in bytes { + s.push_str(&format!("{b:02X}")); + } + f.write_str(&s) +} + +impl From for Literal { + fn from(value: Datum) -> Self { + Literal::Primitive(value.literal) + } +} + +impl From for PrimitiveLiteral { + fn from(value: Datum) -> Self { + value.literal + } +} + +impl Datum { + /// Creates a `Datum` from a `PrimitiveType` and a `PrimitiveLiteral` + pub(crate) fn new(r#type: PrimitiveType, literal: PrimitiveLiteral) -> Self { + Datum { r#type, literal } + } + + /// Create iceberg value from bytes. + /// + /// See [this spec](https://iceberg.apache.org/spec/#binary-single-value-serialization) for reference. + pub fn try_from_bytes(bytes: &[u8], data_type: PrimitiveType) -> Result { + let literal = match data_type { + PrimitiveType::Boolean => { + if bytes.len() == 1 && bytes[0] == 0u8 { + PrimitiveLiteral::Boolean(false) + } else { + PrimitiveLiteral::Boolean(true) + } + } + PrimitiveType::Int => PrimitiveLiteral::Int(i32::from_le_bytes(bytes.try_into()?)), + PrimitiveType::Long => { + if bytes.len() == 4 { + // In the case of an evolved field + PrimitiveLiteral::Long(i32::from_le_bytes(bytes.try_into()?) as i64) + } else { + PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) + } + } + PrimitiveType::Float => { + PrimitiveLiteral::Float(OrderedFloat(f32::from_le_bytes(bytes.try_into()?))) + } + PrimitiveType::Double => { + if bytes.len() == 4 { + // In the case of an evolved field + PrimitiveLiteral::Double(OrderedFloat( + f32::from_le_bytes(bytes.try_into()?) as f64 + )) + } else { + PrimitiveLiteral::Double(OrderedFloat(f64::from_le_bytes(bytes.try_into()?))) + } + } + PrimitiveType::Date => PrimitiveLiteral::Int(i32::from_le_bytes(bytes.try_into()?)), + PrimitiveType::Time => PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)), + PrimitiveType::Timestamp => { + PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) + } + PrimitiveType::Timestamptz => { + PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) + } + PrimitiveType::TimestampNs => { + PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) + } + PrimitiveType::TimestamptzNs => { + PrimitiveLiteral::Long(i64::from_le_bytes(bytes.try_into()?)) + } + PrimitiveType::String => { + PrimitiveLiteral::String(std::str::from_utf8(bytes)?.to_string()) + } + PrimitiveType::Uuid => { + PrimitiveLiteral::UInt128(u128::from_be_bytes(bytes.try_into()?)) + } + PrimitiveType::Fixed(_) => PrimitiveLiteral::Binary(Vec::from(bytes)), + PrimitiveType::Binary => PrimitiveLiteral::Binary(Vec::from(bytes)), + PrimitiveType::Decimal { .. } => { + let unscaled_value = BigInt::from_signed_bytes_be(bytes); + PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't convert bytes to i128: {bytes:?}"), + ) + })?) + } + }; + Ok(Datum::new(data_type, literal)) + } + + /// Convert the value to bytes + /// + /// See [this spec](https://iceberg.apache.org/spec/#binary-single-value-serialization) for reference. + pub fn to_bytes(&self) -> Result { + let buf = match &self.literal { + PrimitiveLiteral::Boolean(val) => { + if *val { + ByteBuf::from([1u8]) + } else { + ByteBuf::from([0u8]) + } + } + PrimitiveLiteral::Int(val) => ByteBuf::from(val.to_le_bytes()), + PrimitiveLiteral::Long(val) => ByteBuf::from(val.to_le_bytes()), + PrimitiveLiteral::Float(val) => ByteBuf::from(val.to_le_bytes()), + PrimitiveLiteral::Double(val) => ByteBuf::from(val.to_le_bytes()), + PrimitiveLiteral::String(val) => ByteBuf::from(val.as_bytes()), + PrimitiveLiteral::UInt128(val) => ByteBuf::from(val.to_be_bytes()), + PrimitiveLiteral::Binary(val) => ByteBuf::from(val.as_slice()), + PrimitiveLiteral::Int128(val) => { + let PrimitiveType::Decimal { precision, .. } = self.r#type else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "PrimitiveLiteral Int128 must be PrimitiveType Decimal but got {}", + &self.r#type + ), + )); + }; + + // It's required by iceberg spec that we must keep the minimum + // number of bytes for the value + let Ok(required_bytes) = Type::decimal_required_bytes(precision) else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "PrimitiveType Decimal must has valid precision but got {precision}" + ), + )); + }; + + // The primitive literal is unscaled value. + let unscaled_value = BigInt::from(*val); + // Convert into two's-complement byte representation of the BigInt + // in big-endian byte order. + let mut bytes = unscaled_value.to_signed_bytes_be(); + // Truncate with required bytes to make sure. + bytes.truncate(required_bytes as usize); + + ByteBuf::from(bytes) + } + PrimitiveLiteral::AboveMax | PrimitiveLiteral::BelowMin => { + return Err(Error::new( + ErrorKind::DataInvalid, + "Cannot convert AboveMax or BelowMin to bytes".to_string(), + )); + } + }; + + Ok(buf) + } + + /// Creates a boolean value. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// let t = Datum::bool(true); + /// + /// assert_eq!(format!("{}", t), "true".to_string()); + /// assert_eq!( + /// Literal::from(t), + /// Literal::Primitive(PrimitiveLiteral::Boolean(true)) + /// ); + /// ``` + pub fn bool>(t: T) -> Self { + Self { + r#type: PrimitiveType::Boolean, + literal: PrimitiveLiteral::Boolean(t.into()), + } + } + + /// Creates a boolean value from string. + /// See [Parse bool from str](https://doc.rust-lang.org/stable/std/primitive.bool.html#impl-FromStr-for-bool) for reference. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// let t = Datum::bool_from_str("false").unwrap(); + /// + /// assert_eq!(&format!("{}", t), "false"); + /// assert_eq!( + /// Literal::Primitive(PrimitiveLiteral::Boolean(false)), + /// t.into() + /// ); + /// ``` + pub fn bool_from_str>(s: S) -> Result { + let v = s.as_ref().parse::().map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse string to bool.").with_source(e) + })?; + Ok(Self::bool(v)) + } + + /// Creates an 32bit integer. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// let t = Datum::int(23i8); + /// + /// assert_eq!(&format!("{}", t), "23"); + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(23)), t.into()); + /// ``` + pub fn int>(t: T) -> Self { + Self { + r#type: PrimitiveType::Int, + literal: PrimitiveLiteral::Int(t.into()), + } + } + + /// Creates an 64bit integer. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// let t = Datum::long(24i8); + /// + /// assert_eq!(&format!("{t}"), "24"); + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Long(24)), t.into()); + /// ``` + pub fn long>(t: T) -> Self { + Self { + r#type: PrimitiveType::Long, + literal: PrimitiveLiteral::Long(t.into()), + } + } + + /// Creates an 32bit floating point number. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// use ordered_float::OrderedFloat; + /// let t = Datum::float(32.1f32); + /// + /// assert_eq!(&format!("{t}"), "32.1"); + /// assert_eq!( + /// Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(32.1))), + /// t.into() + /// ); + /// ``` + pub fn float>(t: T) -> Self { + Self { + r#type: PrimitiveType::Float, + literal: PrimitiveLiteral::Float(OrderedFloat(t.into())), + } + } + + /// Creates an 64bit floating point number. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// use ordered_float::OrderedFloat; + /// let t = Datum::double(32.1f64); + /// + /// assert_eq!(&format!("{t}"), "32.1"); + /// assert_eq!( + /// Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(32.1))), + /// t.into() + /// ); + /// ``` + pub fn double>(t: T) -> Self { + Self { + r#type: PrimitiveType::Double, + literal: PrimitiveLiteral::Double(OrderedFloat(t.into())), + } + } + + /// Creates date literal from number of days from unix epoch directly. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// // 2 days after 1970-01-01 + /// let t = Datum::date(2); + /// + /// assert_eq!(&format!("{t}"), "1970-01-03"); + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(2)), t.into()); + /// ``` + pub fn date(days: i32) -> Self { + Self { + r#type: PrimitiveType::Date, + literal: PrimitiveLiteral::Int(days), + } + } + + /// Creates date literal in `%Y-%m-%d` format, assume in utc timezone. + /// + /// See [`NaiveDate::from_str`]. + /// + /// Example + /// ```rust + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::date_from_str("1970-01-05").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1970-01-05"); + /// assert_eq!(Literal::date(4), t.into()); + /// ``` + pub fn date_from_str>(s: S) -> Result { + let t = s.as_ref().parse::().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse date from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + + Ok(Self::date(date::date_from_naive_date(t))) + } + + /// Create date literal from calendar date (year, month and day). + /// + /// See [`NaiveDate::from_ymd_opt`]. + /// + /// Example: + /// + ///```rust + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::date_from_ymd(1970, 1, 5).unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1970-01-05"); + /// assert_eq!(Literal::date(4), t.into()); + /// ``` + pub fn date_from_ymd(year: i32, month: u32, day: u32) -> Result { + let t = NaiveDate::from_ymd_opt(year, month, day).ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't create date from year: {year}, month: {month}, day: {day}"), + ) + })?; + + Ok(Self::date(date::date_from_naive_date(t))) + } + + /// Creates time literal in microseconds directly. + /// + /// It will return error when it's negative or too large to fit in 24 hours. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::{Datum, Literal}; + /// let micro_secs = { + /// 1 * 3600 * 1_000_000 + // 1 hour + /// 2 * 60 * 1_000_000 + // 2 minutes + /// 1 * 1_000_000 + // 1 second + /// 888999 // microseconds + /// }; + /// + /// let t = Datum::time_micros(micro_secs).unwrap(); + /// + /// assert_eq!(&format!("{t}"), "01:02:01.888999"); + /// assert_eq!(Literal::time(micro_secs), t.into()); + /// + /// let negative_value = -100; + /// assert!(Datum::time_micros(negative_value).is_err()); + /// + /// let too_large_value = 36 * 60 * 60 * 1_000_000; // Too large to fit in 24 hours. + /// assert!(Datum::time_micros(too_large_value).is_err()); + /// ``` + pub fn time_micros(value: i64) -> Result { + ensure_data_valid!( + (0..=MAX_TIME_VALUE).contains(&value), + "Invalid value for Time type: {}", + value + ); + + Ok(Self { + r#type: PrimitiveType::Time, + literal: PrimitiveLiteral::Long(value), + }) + } + + /// Creates time literal from [`chrono::NaiveTime`]. + fn time_from_naive_time(t: NaiveTime) -> Self { + let duration = t - date::unix_epoch().time(); + // It's safe to unwrap here since less than 24 hours will never overflow. + let micro_secs = duration.num_microseconds().unwrap(); + + Self { + r#type: PrimitiveType::Time, + literal: PrimitiveLiteral::Long(micro_secs), + } + } + + /// Creates time literal in microseconds in `%H:%M:%S:.f` format. + /// + /// See [`NaiveTime::from_str`] for details. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::time_from_str("01:02:01.888999777").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "01:02:01.888999"); + /// ``` + pub fn time_from_str>(s: S) -> Result { + let t = s.as_ref().parse::().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse time from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + + Ok(Self::time_from_naive_time(t)) + } + + /// Creates time literal from hour, minute, second, and microseconds. + /// + /// See [`NaiveTime::from_hms_micro_opt`]. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::time_from_hms_micro(22, 15, 33, 111).unwrap(); + /// + /// assert_eq!(&format!("{t}"), "22:15:33.000111"); + /// ``` + pub fn time_from_hms_micro(hour: u32, min: u32, sec: u32, micro: u32) -> Result { + let t = NaiveTime::from_hms_micro_opt(hour, min, sec, micro) + .ok_or_else(|| Error::new( + ErrorKind::DataInvalid, + format!("Can't create time from hour: {hour}, min: {min}, second: {sec}, microsecond: {micro}"), + ))?; + Ok(Self::time_from_naive_time(t)) + } + + /// Creates a timestamp from unix epoch in microseconds. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::timestamp_micros(1000); + /// + /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.001"); + /// ``` + pub fn timestamp_micros(value: i64) -> Self { + Self { + r#type: PrimitiveType::Timestamp, + literal: PrimitiveLiteral::Long(value), + } + } + + /// Creates a timestamp from unix epoch in nanoseconds. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::timestamp_nanos(1000); + /// + /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.000001"); + /// ``` + pub fn timestamp_nanos(value: i64) -> Self { + Self { + r#type: PrimitiveType::TimestampNs, + literal: PrimitiveLiteral::Long(value), + } + } + + /// Creates a timestamp from [`DateTime`]. + /// + /// Example: + /// + /// ```rust + /// use chrono::{NaiveDate, NaiveDateTime, TimeZone, Utc}; + /// use iceberg::spec::Datum; + /// let t = Datum::timestamp_from_datetime( + /// NaiveDate::from_ymd_opt(1992, 3, 1) + /// .unwrap() + /// .and_hms_micro_opt(1, 2, 3, 88) + /// .unwrap(), + /// ); + /// + /// assert_eq!(&format!("{t}"), "1992-03-01 01:02:03.000088"); + /// ``` + pub fn timestamp_from_datetime(dt: NaiveDateTime) -> Self { + Self::timestamp_micros(dt.and_utc().timestamp_micros()) + } + + /// Parse a timestamp in [`%Y-%m-%dT%H:%M:%S%.f`] format. + /// + /// See [`NaiveDateTime::from_str`]. + /// + /// Example: + /// + /// ```rust + /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::timestamp_from_str("1992-03-01T01:02:03.000088").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1992-03-01 01:02:03.000088"); + /// ``` + pub fn timestamp_from_str>(s: S) -> Result { + let dt = s.as_ref().parse::().map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse timestamp.").with_source(e) + })?; + + Ok(Self::timestamp_from_datetime(dt)) + } + + /// Creates a timestamp with timezone from unix epoch in microseconds. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::timestamptz_micros(1000); + /// + /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.001 UTC"); + /// ``` + pub fn timestamptz_micros(value: i64) -> Self { + Self { + r#type: PrimitiveType::Timestamptz, + literal: PrimitiveLiteral::Long(value), + } + } + + /// Creates a timestamp with timezone from unix epoch in nanoseconds. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::timestamptz_nanos(1000); + /// + /// assert_eq!(&format!("{t}"), "1970-01-01 00:00:00.000001 UTC"); + /// ``` + pub fn timestamptz_nanos(value: i64) -> Self { + Self { + r#type: PrimitiveType::TimestamptzNs, + literal: PrimitiveLiteral::Long(value), + } + } + + /// Creates a timestamp with timezone from [`DateTime`]. + /// Example: + /// + /// ```rust + /// use chrono::{TimeZone, Utc}; + /// use iceberg::spec::Datum; + /// let t = Datum::timestamptz_from_datetime(Utc.timestamp_opt(1000, 0).unwrap()); + /// + /// assert_eq!(&format!("{t}"), "1970-01-01 00:16:40 UTC"); + /// ``` + pub fn timestamptz_from_datetime(dt: DateTime) -> Self { + Self::timestamptz_micros(dt.with_timezone(&Utc).timestamp_micros()) + } + + /// Parse timestamp with timezone in RFC3339 format. + /// + /// See [`DateTime::from_str`]. + /// + /// Example: + /// + /// ```rust + /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; + /// use iceberg::spec::{Datum, Literal}; + /// let t = Datum::timestamptz_from_str("1992-03-01T01:02:03.000088+08:00").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1992-02-29 17:02:03.000088 UTC"); + /// ``` + pub fn timestamptz_from_str>(s: S) -> Result { + let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) + })?; + + Ok(Self::timestamptz_from_datetime(dt)) + } + + /// Creates a string literal. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::string("ss"); + /// + /// assert_eq!(&format!("{t}"), r#""ss""#); + /// ``` + pub fn string(s: S) -> Self { + Self { + r#type: PrimitiveType::String, + literal: PrimitiveLiteral::String(s.to_string()), + } + } + + /// Creates uuid literal. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// use uuid::uuid; + /// let t = Datum::uuid(uuid!("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8")); + /// + /// assert_eq!(&format!("{t}"), "a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8"); + /// ``` + pub fn uuid(uuid: uuid::Uuid) -> Self { + Self { + r#type: PrimitiveType::Uuid, + literal: PrimitiveLiteral::UInt128(uuid.as_u128()), + } + } + + /// Creates uuid from str. See [`uuid::Uuid::parse_str`]. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::uuid_from_str("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8"); + /// ``` + pub fn uuid_from_str>(s: S) -> Result { + let uuid = uuid::Uuid::parse_str(s.as_ref()).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse uuid from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + Ok(Self::uuid(uuid)) + } + + /// Creates a fixed literal from bytes. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::{Datum, Literal, PrimitiveLiteral}; + /// let t = Datum::fixed(vec![1u8, 2u8]); + /// + /// assert_eq!(&format!("{t}"), "0102"); + /// ``` + pub fn fixed>(input: I) -> Self { + let value: Vec = input.into_iter().collect(); + Self { + r#type: PrimitiveType::Fixed(value.len() as u64), + literal: PrimitiveLiteral::Binary(value), + } + } + + /// Creates a binary literal from bytes. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// let t = Datum::binary(vec![1u8, 100u8]); + /// + /// assert_eq!(&format!("{t}"), "0164"); + /// ``` + pub fn binary>(input: I) -> Self { + Self { + r#type: PrimitiveType::Binary, + literal: PrimitiveLiteral::Binary(input.into_iter().collect()), + } + } + + /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// use itertools::assert_equal; + /// use rust_decimal::Decimal; + /// let t = Datum::decimal_from_str("123.45").unwrap(); + /// + /// assert_eq!(&format!("{t}"), "123.45"); + /// ``` + pub fn decimal_from_str>(s: S) -> Result { + let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) + })?; + + Self::decimal(decimal) + } + + /// Try to create a decimal literal from [`Decimal`]. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// use rust_decimal::Decimal; + /// + /// let t = Datum::decimal(Decimal::new(123, 2)).unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1.23"); + /// ``` + pub fn decimal(value: impl Into) -> Result { + let decimal = value.into(); + let scale = decimal.scale(); + + let r#type = Type::decimal(MAX_DECIMAL_PRECISION, scale)?; + if let Type::Primitive(p) = r#type { + Ok(Self { + r#type: p, + literal: PrimitiveLiteral::Int128(decimal.mantissa()), + }) + } else { + unreachable!("Decimal type must be primitive.") + } + } + + /// Try to create a decimal literal from [`Decimal`] with precision. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Datum; + /// use rust_decimal::Decimal; + /// + /// let t = Datum::decimal_with_precision(Decimal::new(123, 2), 30).unwrap(); + /// + /// assert_eq!(&format!("{t}"), "1.23"); + /// ``` + pub fn decimal_with_precision(value: impl Into, precision: u32) -> Result { + let decimal = value.into(); + let scale = decimal.scale(); + + let available_bytes = Type::decimal_required_bytes(precision)? as usize; + let unscaled_value = BigInt::from(decimal.mantissa()); + let actual_bytes = unscaled_value.to_signed_bytes_be(); + if actual_bytes.len() > available_bytes { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Decimal value {decimal} is too large for precision {precision}"), + )); + } + + let r#type = Type::decimal(precision, scale)?; + if let Type::Primitive(p) = r#type { + Ok(Self { + r#type: p, + literal: PrimitiveLiteral::Int128(decimal.mantissa()), + }) + } else { + unreachable!("Decimal type must be primitive.") + } + } + + fn i64_to_i32 + PartialOrd>(val: T) -> Datum { + if val > INT_MAX as i64 { + Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax) + } else if val < INT_MIN as i64 { + Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin) + } else { + Datum::int(val.into() as i32) + } + } + + fn i128_to_i32 + PartialOrd>(val: T) -> Datum { + if val > INT_MAX as i128 { + Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax) + } else if val < INT_MIN as i128 { + Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin) + } else { + Datum::int(val.into() as i32) + } + } + + fn i128_to_i64 + PartialOrd>(val: T) -> Datum { + if val > LONG_MAX as i128 { + Datum::new(PrimitiveType::Long, PrimitiveLiteral::AboveMax) + } else if val < LONG_MIN as i128 { + Datum::new(PrimitiveType::Long, PrimitiveLiteral::BelowMin) + } else { + Datum::long(val.into() as i64) + } + } + + fn string_to_i128>(s: S) -> Result { + s.as_ref().parse::().map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse string to i128.").with_source(e) + }) + } + + /// Convert the datum to `target_type`. + pub fn to(self, target_type: &Type) -> Result { + match target_type { + Type::Primitive(target_primitive_type) => { + match (&self.literal, &self.r#type, target_primitive_type) { + (PrimitiveLiteral::Int(val), _, PrimitiveType::Int) => Ok(Datum::int(*val)), + (PrimitiveLiteral::Int(val), _, PrimitiveType::Date) => Ok(Datum::date(*val)), + (PrimitiveLiteral::Int(val), _, PrimitiveType::Long) => Ok(Datum::long(*val)), + (PrimitiveLiteral::Long(val), _, PrimitiveType::Int) => { + Ok(Datum::i64_to_i32(*val)) + } + (PrimitiveLiteral::Long(val), _, PrimitiveType::Timestamp) => { + Ok(Datum::timestamp_micros(*val)) + } + (PrimitiveLiteral::Long(val), _, PrimitiveType::Timestamptz) => { + Ok(Datum::timestamptz_micros(*val)) + } + // Let's wait with nano's until this clears up: https://github.com/apache/iceberg/pull/11775 + (PrimitiveLiteral::Int128(val), _, PrimitiveType::Long) => { + Ok(Datum::i128_to_i64(*val)) + } + + (PrimitiveLiteral::String(val), _, PrimitiveType::Boolean) => { + Datum::bool_from_str(val) + } + (PrimitiveLiteral::String(val), _, PrimitiveType::Int) => { + Datum::string_to_i128(val).map(Datum::i128_to_i32) + } + (PrimitiveLiteral::String(val), _, PrimitiveType::Long) => { + Datum::string_to_i128(val).map(Datum::i128_to_i64) + } + (PrimitiveLiteral::String(val), _, PrimitiveType::Timestamp) => { + Datum::timestamp_from_str(val) + } + (PrimitiveLiteral::String(val), _, PrimitiveType::Timestamptz) => { + Datum::timestamptz_from_str(val) + } + + // TODO: implement more type conversions + (_, self_type, target_type) if self_type == target_type => Ok(self), + _ => Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Can't convert datum from {} type to {} type.", + self.r#type, target_primitive_type + ), + )), + } + } + _ => Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Can't convert datum from {} type to {} type.", + self.r#type, target_type + ), + )), + } + } + + /// Get the primitive literal from datum. + pub fn literal(&self) -> &PrimitiveLiteral { + &self.literal + } + + /// Get the primitive type from datum. + pub fn data_type(&self) -> &PrimitiveType { + &self.r#type + } + + /// Returns true if the Literal represents a primitive type + /// that can be a NaN, and that it's value is NaN + pub fn is_nan(&self) -> bool { + match self.literal { + PrimitiveLiteral::Double(val) => val.is_nan(), + PrimitiveLiteral::Float(val) => val.is_nan(), + _ => false, + } + } + + /// Returns a human-readable string representation of this literal. + /// + /// For string literals, this returns the raw string value without quotes. + /// For all other literals, it falls back to [`to_string()`]. + pub fn to_human_string(&self) -> String { + match self.literal() { + PrimitiveLiteral::String(s) => s.to_string(), + _ => self.to_string(), + } + } +} diff --git a/crates/iceberg/src/spec/values/literal.rs b/crates/iceberg/src/spec/values/literal.rs new file mode 100644 index 0000000000..d6e502e8fd --- /dev/null +++ b/crates/iceberg/src/spec/values/literal.rs @@ -0,0 +1,747 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Literal values in Iceberg + +use std::any::Any; +use std::str::FromStr; + +use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeZone, Utc}; +use ordered_float::OrderedFloat; +use rust_decimal::Decimal; +use serde_json::{Map as JsonMap, Number, Value as JsonValue}; +use uuid::Uuid; + +use super::Map; +use super::primitive::PrimitiveLiteral; +use super::struct_value::Struct; +use super::temporal::{date, time, timestamp, timestamptz}; +use crate::error::Result; +use crate::spec::datatypes::{PrimitiveType, Type}; +use crate::{Error, ErrorKind}; + +/// Values present in iceberg type +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum Literal { + /// A primitive value + Primitive(PrimitiveLiteral), + /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. + /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. + /// Fields may have an optional comment or doc string. Fields can have default values. + Struct(Struct), + /// A list is a collection of values with some element type. + /// The element field has an integer id that is unique in the table schema. + /// Elements can be either optional or required. Element types may be any type. + List(Vec>), + /// A map is a collection of key-value pairs with a key type and a value type. + /// Both the key field and value field each have an integer id that is unique in the table schema. + /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. + Map(Map), +} + +impl Literal { + /// Creates a boolean value. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t = Literal::bool(true); + /// + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Boolean(true)), t); + /// ``` + pub fn bool>(t: T) -> Self { + Self::Primitive(PrimitiveLiteral::Boolean(t.into())) + } + + /// Creates a boolean value from string. + /// See [Parse bool from str](https://doc.rust-lang.org/stable/std/primitive.bool.html#impl-FromStr-for-bool) for reference. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t = Literal::bool_from_str("false").unwrap(); + /// + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Boolean(false)), t); + /// ``` + pub fn bool_from_str>(s: S) -> Result { + let v = s.as_ref().parse::().map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse string to bool.").with_source(e) + })?; + Ok(Self::Primitive(PrimitiveLiteral::Boolean(v))) + } + + /// Creates an 32bit integer. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t = Literal::int(23i8); + /// + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Int(23)), t); + /// ``` + pub fn int>(t: T) -> Self { + Self::Primitive(PrimitiveLiteral::Int(t.into())) + } + + /// Creates an 64bit integer. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t = Literal::long(24i8); + /// + /// assert_eq!(Literal::Primitive(PrimitiveLiteral::Long(24)), t); + /// ``` + pub fn long>(t: T) -> Self { + Self::Primitive(PrimitiveLiteral::Long(t.into())) + } + + /// Creates an 32bit floating point number. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// use ordered_float::OrderedFloat; + /// let t = Literal::float(32.1f32); + /// + /// assert_eq!( + /// Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(32.1))), + /// t + /// ); + /// ``` + pub fn float>(t: T) -> Self { + Self::Primitive(PrimitiveLiteral::Float(OrderedFloat(t.into()))) + } + + /// Creates an 32bit floating point number. + /// + /// Example: + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// use ordered_float::OrderedFloat; + /// let t = Literal::double(32.1f64); + /// + /// assert_eq!( + /// Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(32.1))), + /// t + /// ); + /// ``` + pub fn double>(t: T) -> Self { + Self::Primitive(PrimitiveLiteral::Double(OrderedFloat(t.into()))) + } + + /// Creates date literal from number of days from unix epoch directly. + pub fn date(days: i32) -> Self { + Self::Primitive(PrimitiveLiteral::Int(days)) + } + + /// Creates a date in `%Y-%m-%d` format, assume in utc timezone. + /// + /// See [`NaiveDate::from_str`]. + /// + /// Example + /// ```rust + /// use iceberg::spec::Literal; + /// let t = Literal::date_from_str("1970-01-03").unwrap(); + /// + /// assert_eq!(Literal::date(2), t); + /// ``` + pub fn date_from_str>(s: S) -> Result { + let t = s.as_ref().parse::().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse date from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + + Ok(Self::date(date::date_from_naive_date(t))) + } + + /// Create a date from calendar date (year, month and day). + /// + /// See [`NaiveDate::from_ymd_opt`]. + /// + /// Example: + /// + ///```rust + /// use iceberg::spec::Literal; + /// let t = Literal::date_from_ymd(1970, 1, 5).unwrap(); + /// + /// assert_eq!(Literal::date(4), t); + /// ``` + pub fn date_from_ymd(year: i32, month: u32, day: u32) -> Result { + let t = NaiveDate::from_ymd_opt(year, month, day).ok_or_else(|| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't create date from year: {year}, month: {month}, day: {day}"), + ) + })?; + + Ok(Self::date(date::date_from_naive_date(t))) + } + + /// Creates time in microseconds directly + pub fn time(value: i64) -> Self { + Self::Primitive(PrimitiveLiteral::Long(value)) + } + + /// Creates time literal from [`chrono::NaiveTime`]. + fn time_from_naive_time(t: NaiveTime) -> Self { + let duration = t - date::unix_epoch().time(); + // It's safe to unwrap here since less than 24 hours will never overflow. + let micro_secs = duration.num_microseconds().unwrap(); + + Literal::time(micro_secs) + } + + /// Creates time in microseconds in `%H:%M:%S:.f` format. + /// + /// See [`NaiveTime::from_str`] for details. + /// + /// Example: + /// ```rust + /// use iceberg::spec::Literal; + /// let t = Literal::time_from_str("01:02:01.888999777").unwrap(); + /// + /// let micro_secs = { + /// 1 * 3600 * 1_000_000 + // 1 hour + /// 2 * 60 * 1_000_000 + // 2 minutes + /// 1 * 1_000_000 + // 1 second + /// 888999 // microseconds + /// }; + /// assert_eq!(Literal::time(micro_secs), t); + /// ``` + pub fn time_from_str>(s: S) -> Result { + let t = s.as_ref().parse::().map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse time from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + + Ok(Self::time_from_naive_time(t)) + } + + /// Creates time literal from hour, minute, second, and microseconds. + /// + /// See [`NaiveTime::from_hms_micro_opt`]. + /// + /// Example: + /// ```rust + /// use iceberg::spec::Literal; + /// let t = Literal::time_from_hms_micro(22, 15, 33, 111).unwrap(); + /// + /// assert_eq!(Literal::time_from_str("22:15:33.000111").unwrap(), t); + /// ``` + pub fn time_from_hms_micro(hour: u32, min: u32, sec: u32, micro: u32) -> Result { + let t = NaiveTime::from_hms_micro_opt(hour, min, sec, micro) + .ok_or_else(|| Error::new( + ErrorKind::DataInvalid, + format!("Can't create time from hour: {hour}, min: {min}, second: {sec}, microsecond: {micro}"), + ))?; + Ok(Self::time_from_naive_time(t)) + } + + /// Creates a timestamp from unix epoch in microseconds. + pub fn timestamp(value: i64) -> Self { + Self::Primitive(PrimitiveLiteral::Long(value)) + } + + /// Creates a timestamp with timezone from unix epoch in microseconds. + pub fn timestamptz(value: i64) -> Self { + Self::Primitive(PrimitiveLiteral::Long(value)) + } + + /// Creates a timestamp from unix epoch in nanoseconds. + pub(crate) fn timestamp_nano(value: i64) -> Self { + Self::Primitive(PrimitiveLiteral::Long(value)) + } + + /// Creates a timestamp with timezone from unix epoch in nanoseconds. + pub(crate) fn timestamptz_nano(value: i64) -> Self { + Self::Primitive(PrimitiveLiteral::Long(value)) + } + + /// Creates a timestamp from [`DateTime`]. + pub fn timestamp_from_datetime(dt: DateTime) -> Self { + Self::timestamp(dt.with_timezone(&Utc).timestamp_micros()) + } + + /// Creates a timestamp with timezone from [`DateTime`]. + pub fn timestamptz_from_datetime(dt: DateTime) -> Self { + Self::timestamptz(dt.with_timezone(&Utc).timestamp_micros()) + } + + /// Parse a timestamp in RFC3339 format. + /// + /// See [`DateTime::from_str`]. + /// + /// Example: + /// + /// ```rust + /// use chrono::{DateTime, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; + /// use iceberg::spec::Literal; + /// let t = Literal::timestamp_from_str("2012-12-12 12:12:12.8899-04:00").unwrap(); + /// + /// let t2 = { + /// let date = NaiveDate::from_ymd_opt(2012, 12, 12).unwrap(); + /// let time = NaiveTime::from_hms_micro_opt(12, 12, 12, 889900).unwrap(); + /// let dt = NaiveDateTime::new(date, time); + /// Literal::timestamp_from_datetime(DateTime::::from_local( + /// dt, + /// FixedOffset::west_opt(4 * 3600).unwrap(), + /// )) + /// }; + /// + /// assert_eq!(t, t2); + /// ``` + pub fn timestamp_from_str>(s: S) -> Result { + let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) + })?; + + Ok(Self::timestamp_from_datetime(dt)) + } + + /// Similar to [`Literal::timestamp_from_str`], but return timestamp with timezone literal. + pub fn timestamptz_from_str>(s: S) -> Result { + let dt = DateTime::::from_str(s.as_ref()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse datetime.").with_source(e) + })?; + + Ok(Self::timestamptz_from_datetime(dt)) + } + + /// Creates a string literal. + pub fn string(s: S) -> Self { + Self::Primitive(PrimitiveLiteral::String(s.to_string())) + } + + /// Creates uuid literal. + pub fn uuid(uuid: Uuid) -> Self { + Self::Primitive(PrimitiveLiteral::UInt128(uuid.as_u128())) + } + + /// Creates uuid from str. See [`Uuid::parse_str`]. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Literal; + /// use uuid::Uuid; + /// let t1 = Literal::uuid_from_str("a1a2a3a4-b1b2-c1c2-d1d2-d3d4d5d6d7d8").unwrap(); + /// let t2 = Literal::uuid(Uuid::from_u128_le(0xd8d7d6d5d4d3d2d1c2c1b2b1a4a3a2a1)); + /// + /// assert_eq!(t1, t2); + /// ``` + pub fn uuid_from_str>(s: S) -> Result { + let uuid = Uuid::parse_str(s.as_ref()).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Can't parse uuid from string: {}", s.as_ref()), + ) + .with_source(e) + })?; + Ok(Self::uuid(uuid)) + } + + /// Creates a fixed literal from bytes. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t1 = Literal::fixed(vec![1u8, 2u8]); + /// let t2 = Literal::Primitive(PrimitiveLiteral::Binary(vec![1u8, 2u8])); + /// + /// assert_eq!(t1, t2); + /// ``` + pub fn fixed>(input: I) -> Self { + Literal::Primitive(PrimitiveLiteral::Binary(input.into_iter().collect())) + } + + /// Creates a binary literal from bytes. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::{Literal, PrimitiveLiteral}; + /// let t1 = Literal::binary(vec![1u8, 2u8]); + /// let t2 = Literal::Primitive(PrimitiveLiteral::Binary(vec![1u8, 2u8])); + /// + /// assert_eq!(t1, t2); + /// ``` + pub fn binary>(input: I) -> Self { + Literal::Primitive(PrimitiveLiteral::Binary(input.into_iter().collect())) + } + + /// Creates a decimal literal. + pub fn decimal(decimal: i128) -> Self { + Self::Primitive(PrimitiveLiteral::Int128(decimal)) + } + + /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. + /// + /// Example: + /// + /// ```rust + /// use iceberg::spec::Literal; + /// use rust_decimal::Decimal; + /// let t1 = Literal::decimal(12345); + /// let t2 = Literal::decimal_from_str("123.45").unwrap(); + /// + /// assert_eq!(t1, t2); + /// ``` + pub fn decimal_from_str>(s: S) -> Result { + let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) + })?; + Ok(Self::decimal(decimal.mantissa())) + } + + /// Attempts to convert the Literal to a PrimitiveLiteral + pub fn as_primitive_literal(&self) -> Option { + match self { + Literal::Primitive(primitive) => Some(primitive.clone()), + _ => None, + } + } + + /// Create iceberg value from a json value + /// + /// See [this spec](https://iceberg.apache.org/spec/#json-single-value-serialization) for reference. + pub fn try_from_json(value: JsonValue, data_type: &Type) -> Result> { + match data_type { + Type::Primitive(primitive) => match (primitive, value) { + (PrimitiveType::Boolean, JsonValue::Bool(bool)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Boolean(bool)))) + } + (PrimitiveType::Int, JsonValue::Number(number)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( + number + .as_i64() + .ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to int", + ))? + .try_into()?, + )))) + } + (PrimitiveType::Long, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( + PrimitiveLiteral::Long(number.as_i64().ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to long", + ))?), + ))), + (PrimitiveType::Float, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( + PrimitiveLiteral::Float(OrderedFloat(number.as_f64().ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to float", + ))? as f32)), + ))), + (PrimitiveType::Double, JsonValue::Number(number)) => Ok(Some(Literal::Primitive( + PrimitiveLiteral::Double(OrderedFloat(number.as_f64().ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to double", + ))?)), + ))), + (PrimitiveType::Date, JsonValue::String(s)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( + date::date_to_days(&NaiveDate::parse_from_str(&s, "%Y-%m-%d")?), + )))) + } + (PrimitiveType::Date, JsonValue::Number(number)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int( + number + .as_i64() + .ok_or(Error::new( + crate::ErrorKind::DataInvalid, + "Failed to convert json number to date (days since epoch)", + ))? + .try_into()?, + )))) + } + (PrimitiveType::Time, JsonValue::String(s)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Long( + time::time_to_microseconds(&NaiveTime::parse_from_str(&s, "%H:%M:%S%.f")?), + )))) + } + (PrimitiveType::Timestamp, JsonValue::String(s)) => Ok(Some(Literal::Primitive( + PrimitiveLiteral::Long(timestamp::datetime_to_microseconds( + &NaiveDateTime::parse_from_str(&s, "%Y-%m-%dT%H:%M:%S%.f")?, + )), + ))), + (PrimitiveType::Timestamptz, JsonValue::String(s)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::Long( + timestamptz::datetimetz_to_microseconds(&Utc.from_utc_datetime( + &NaiveDateTime::parse_from_str(&s, "%Y-%m-%dT%H:%M:%S%.f+00:00")?, + )), + )))) + } + (PrimitiveType::String, JsonValue::String(s)) => { + Ok(Some(Literal::Primitive(PrimitiveLiteral::String(s)))) + } + (PrimitiveType::Uuid, JsonValue::String(s)) => Ok(Some(Literal::Primitive( + PrimitiveLiteral::UInt128(Uuid::parse_str(&s)?.as_u128()), + ))), + (PrimitiveType::Fixed(_), JsonValue::String(_)) => todo!(), + (PrimitiveType::Binary, JsonValue::String(_)) => todo!(), + ( + PrimitiveType::Decimal { + precision: _, + scale, + }, + JsonValue::String(s), + ) => { + let mut decimal = Decimal::from_str_exact(&s)?; + decimal.rescale(*scale); + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( + decimal.mantissa(), + )))) + } + (_, JsonValue::Null) => Ok(None), + (i, j) => Err(Error::new( + crate::ErrorKind::DataInvalid, + format!("The json value {j} doesn't fit to the iceberg type {i}."), + )), + }, + Type::Struct(schema) => { + if let JsonValue::Object(mut object) = value { + Ok(Some(Literal::Struct(Struct::from_iter( + schema.fields().iter().map(|field| { + object.remove(&field.id.to_string()).and_then(|value| { + Literal::try_from_json(value, &field.field_type) + .and_then(|value| { + value.ok_or(Error::new( + ErrorKind::DataInvalid, + "Key of map cannot be null", + )) + }) + .ok() + }) + }), + )))) + } else { + Err(Error::new( + crate::ErrorKind::DataInvalid, + "The json value for a struct type must be an object.", + )) + } + } + Type::List(list) => { + if let JsonValue::Array(array) = value { + Ok(Some(Literal::List( + array + .into_iter() + .map(|value| { + Literal::try_from_json(value, &list.element_field.field_type) + }) + .collect::>>()?, + ))) + } else { + Err(Error::new( + crate::ErrorKind::DataInvalid, + "The json value for a list type must be an array.", + )) + } + } + Type::Map(map) => { + if let JsonValue::Object(mut object) = value { + if let (Some(JsonValue::Array(keys)), Some(JsonValue::Array(values))) = + (object.remove("keys"), object.remove("values")) + { + Ok(Some(Literal::Map(Map::from_iter( + keys.into_iter() + .zip(values.into_iter()) + .map(|(key, value)| { + Ok(( + Literal::try_from_json(key, &map.key_field.field_type) + .and_then(|value| { + value.ok_or(Error::new( + ErrorKind::DataInvalid, + "Key of map cannot be null", + )) + })?, + Literal::try_from_json(value, &map.value_field.field_type)?, + )) + }) + .collect::>>()?, + )))) + } else { + Err(Error::new( + crate::ErrorKind::DataInvalid, + "The json value for a list type must be an array.", + )) + } + } else { + Err(Error::new( + crate::ErrorKind::DataInvalid, + "The json value for a list type must be an array.", + )) + } + } + } + } + + /// Converting iceberg value to json value. + /// + /// See [this spec](https://iceberg.apache.org/spec/#json-single-value-serialization) for reference. + pub fn try_into_json(self, r#type: &Type) -> Result { + match (self, r#type) { + (Literal::Primitive(prim), Type::Primitive(prim_type)) => match (prim_type, prim) { + (PrimitiveType::Boolean, PrimitiveLiteral::Boolean(val)) => { + Ok(JsonValue::Bool(val)) + } + (PrimitiveType::Int, PrimitiveLiteral::Int(val)) => { + Ok(JsonValue::Number((val).into())) + } + (PrimitiveType::Long, PrimitiveLiteral::Long(val)) => { + Ok(JsonValue::Number((val).into())) + } + (PrimitiveType::Float, PrimitiveLiteral::Float(val)) => { + match Number::from_f64(val.0 as f64) { + Some(number) => Ok(JsonValue::Number(number)), + None => Ok(JsonValue::Null), + } + } + (PrimitiveType::Double, PrimitiveLiteral::Double(val)) => { + match Number::from_f64(val.0) { + Some(number) => Ok(JsonValue::Number(number)), + None => Ok(JsonValue::Null), + } + } + (PrimitiveType::Date, PrimitiveLiteral::Int(val)) => { + Ok(JsonValue::String(date::days_to_date(val).to_string())) + } + (PrimitiveType::Time, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( + time::microseconds_to_time(val).to_string(), + )), + (PrimitiveType::Timestamp, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( + timestamp::microseconds_to_datetime(val) + .format("%Y-%m-%dT%H:%M:%S%.f") + .to_string(), + )), + (PrimitiveType::Timestamptz, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( + timestamptz::microseconds_to_datetimetz(val) + .format("%Y-%m-%dT%H:%M:%S%.f+00:00") + .to_string(), + )), + (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(val)) => Ok(JsonValue::String( + timestamp::nanoseconds_to_datetime(val) + .format("%Y-%m-%dT%H:%M:%S%.f") + .to_string(), + )), + (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(val)) => { + Ok(JsonValue::String( + timestamptz::nanoseconds_to_datetimetz(val) + .format("%Y-%m-%dT%H:%M:%S%.f+00:00") + .to_string(), + )) + } + (PrimitiveType::String, PrimitiveLiteral::String(val)) => { + Ok(JsonValue::String(val.clone())) + } + (_, PrimitiveLiteral::UInt128(val)) => { + Ok(JsonValue::String(Uuid::from_u128(val).to_string())) + } + (_, PrimitiveLiteral::Binary(val)) => Ok(JsonValue::String(val.iter().fold( + String::new(), + |mut acc, x| { + acc.push_str(&format!("{x:x}")); + acc + }, + ))), + (_, PrimitiveLiteral::Int128(val)) => match r#type { + Type::Primitive(PrimitiveType::Decimal { + precision: _precision, + scale, + }) => { + let decimal = Decimal::try_from_i128_with_scale(val, *scale)?; + Ok(JsonValue::String(decimal.to_string())) + } + _ => Err(Error::new( + ErrorKind::DataInvalid, + "The iceberg type for decimal literal must be decimal.", + ))?, + }, + _ => Err(Error::new( + ErrorKind::DataInvalid, + "The iceberg value doesn't fit to the iceberg type.", + )), + }, + (Literal::Struct(s), Type::Struct(struct_type)) => { + let mut id_and_value = Vec::with_capacity(struct_type.fields().len()); + for (value, field) in s.into_iter().zip(struct_type.fields()) { + let json = match value { + Some(val) => val.try_into_json(&field.field_type)?, + None => JsonValue::Null, + }; + id_and_value.push((field.id.to_string(), json)); + } + Ok(JsonValue::Object(JsonMap::from_iter(id_and_value))) + } + (Literal::List(list), Type::List(list_type)) => Ok(JsonValue::Array( + list.into_iter() + .map(|opt| match opt { + Some(literal) => literal.try_into_json(&list_type.element_field.field_type), + None => Ok(JsonValue::Null), + }) + .collect::>>()?, + )), + (Literal::Map(map), Type::Map(map_type)) => { + let mut object = JsonMap::with_capacity(2); + let mut json_keys = Vec::with_capacity(map.len()); + let mut json_values = Vec::with_capacity(map.len()); + for (key, value) in map.into_iter() { + json_keys.push(key.try_into_json(&map_type.key_field.field_type)?); + json_values.push(match value { + Some(literal) => literal.try_into_json(&map_type.value_field.field_type)?, + None => JsonValue::Null, + }); + } + object.insert("keys".to_string(), JsonValue::Array(json_keys)); + object.insert("values".to_string(), JsonValue::Array(json_values)); + Ok(JsonValue::Object(object)) + } + (value, r#type) => Err(Error::new( + ErrorKind::DataInvalid, + format!("The iceberg value {value:?} doesn't fit to the iceberg type {type}."), + )), + } + } + + /// Convert Value to the any type + pub fn into_any(self) -> Box { + match self { + Literal::Primitive(prim) => match prim { + PrimitiveLiteral::Boolean(any) => Box::new(any), + PrimitiveLiteral::Int(any) => Box::new(any), + PrimitiveLiteral::Long(any) => Box::new(any), + PrimitiveLiteral::Float(any) => Box::new(any), + PrimitiveLiteral::Double(any) => Box::new(any), + PrimitiveLiteral::Binary(any) => Box::new(any), + PrimitiveLiteral::String(any) => Box::new(any), + PrimitiveLiteral::UInt128(any) => Box::new(any), + PrimitiveLiteral::Int128(any) => Box::new(any), + PrimitiveLiteral::AboveMax | PrimitiveLiteral::BelowMin => unimplemented!(), + }, + _ => unimplemented!(), + } + } +} diff --git a/crates/iceberg/src/spec/values/map.rs b/crates/iceberg/src/spec/values/map.rs new file mode 100644 index 0000000000..e0f75205f0 --- /dev/null +++ b/crates/iceberg/src/spec/values/map.rs @@ -0,0 +1,145 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Map collection type for Iceberg + +use std::collections::HashMap; +use std::hash::Hash; + +use super::Literal; + +/// Map is a collection of key-value pairs with a key type and a value type. +/// It used in Literal::Map, to make it hashable, the order of key-value pairs is stored in a separate vector +/// so that we can hash the map in a deterministic way. But it also means that the order of key-value pairs is matter +/// for the hash value. +/// +/// When converting to Arrow (e.g., for Iceberg), the map should be represented using +/// the default field name "key_value". +/// +/// Example: +/// +/// ```text +/// let key_value_field = Field::new( +/// DEFAULT_MAP_FIELD_NAME, +/// arrow_schema::DataType::Struct(vec![ +/// Arc::new(key_field.clone()), +/// Arc::new(value_field.clone()), +/// ].into()), +/// false +/// ); +/// ''' +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Map { + index: HashMap, + pair: Vec<(Literal, Option)>, +} + +impl Map { + /// Creates a new empty map. + pub fn new() -> Self { + Self { + index: HashMap::new(), + pair: Vec::new(), + } + } + + /// Return the number of key-value pairs in the map. + pub fn len(&self) -> usize { + self.pair.len() + } + + /// Returns true if the map contains no elements. + pub fn is_empty(&self) -> bool { + self.pair.is_empty() + } + + /// Inserts a key-value pair into the map. + /// If the map did not have this key present, None is returned. + /// If the map did have this key present, the value is updated, and the old value is returned. + pub fn insert(&mut self, key: Literal, value: Option) -> Option> { + if let Some(index) = self.index.get(&key) { + let old_value = std::mem::replace(&mut self.pair[*index].1, value); + Some(old_value) + } else { + self.pair.push((key.clone(), value)); + self.index.insert(key, self.pair.len() - 1); + None + } + } + + /// Returns a reference to the value corresponding to the key. + /// If the key is not present in the map, None is returned. + pub fn get(&self, key: &Literal) -> Option<&Option> { + self.index.get(key).map(|index| &self.pair[*index].1) + } + + /// The order of map is matter, so this method used to compare two maps has same key-value pairs without considering the order. + pub fn has_same_content(&self, other: &Map) -> bool { + if self.len() != other.len() { + return false; + } + + for (key, value) in &self.pair { + match other.get(key) { + Some(other_value) if value == other_value => (), + _ => return false, + } + } + + true + } +} + +impl Default for Map { + fn default() -> Self { + Self::new() + } +} + +impl Hash for Map { + fn hash(&self, state: &mut H) { + for (key, value) in &self.pair { + key.hash(state); + value.hash(state); + } + } +} + +impl FromIterator<(Literal, Option)> for Map { + fn from_iter)>>(iter: T) -> Self { + let mut map = Map::new(); + for (key, value) in iter { + map.insert(key, value); + } + map + } +} + +impl IntoIterator for Map { + type Item = (Literal, Option); + type IntoIter = std::vec::IntoIter; + + fn into_iter(self) -> Self::IntoIter { + self.pair.into_iter() + } +} + +impl From<[(Literal, Option); N]> for Map { + fn from(value: [(Literal, Option); N]) -> Self { + value.iter().cloned().collect() + } +} diff --git a/crates/iceberg/src/spec/values/mod.rs b/crates/iceberg/src/spec/values/mod.rs new file mode 100644 index 0000000000..2bc967191b --- /dev/null +++ b/crates/iceberg/src/spec/values/mod.rs @@ -0,0 +1,37 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! This module contains Iceberg value types + +pub(crate) mod datum; +mod literal; +mod map; +mod primitive; +pub(crate) mod serde; +mod struct_value; +mod temporal; + +#[cfg(test)] +mod tests; + +// Re-export all public types +pub use datum::Datum; +pub use literal::Literal; +pub use map::Map; +pub use primitive::PrimitiveLiteral; +pub(crate) use serde::_serde::RawLiteral; +pub use struct_value::Struct; diff --git a/crates/iceberg/src/spec/values/primitive.rs b/crates/iceberg/src/spec/values/primitive.rs new file mode 100644 index 0000000000..702437d9b5 --- /dev/null +++ b/crates/iceberg/src/spec/values/primitive.rs @@ -0,0 +1,59 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Primitive literal types + +use ordered_float::OrderedFloat; + +/// Values present in iceberg type +#[derive(Clone, Debug, PartialOrd, PartialEq, Hash, Eq)] +pub enum PrimitiveLiteral { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + Long(i64), + /// Stored as 4-byte little-endian + Float(OrderedFloat), + /// Stored as 8-byte little-endian + Double(OrderedFloat), + /// UTF-8 bytes (without length) + String(String), + /// Binary value (without length) + Binary(Vec), + /// Stored as 16-byte little-endian + Int128(i128), + /// Stored as 16-byte little-endian + UInt128(u128), + /// When a number is larger than it can hold + AboveMax, + /// When a number is smaller than it can hold + BelowMin, +} + +impl PrimitiveLiteral { + /// Returns true if the Literal represents a primitive type + /// that can be a NaN, and that it's value is NaN + pub fn is_nan(&self) -> bool { + match self { + PrimitiveLiteral::Double(val) => val.is_nan(), + PrimitiveLiteral::Float(val) => val.is_nan(), + _ => false, + } + } +} diff --git a/crates/iceberg/src/spec/values/serde.rs b/crates/iceberg/src/spec/values/serde.rs new file mode 100644 index 0000000000..053acca8b0 --- /dev/null +++ b/crates/iceberg/src/spec/values/serde.rs @@ -0,0 +1,719 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//\! Serialization and deserialization support for Iceberg values + +pub(crate) mod _serde { + use serde::de::Visitor; + use serde::ser::{SerializeMap, SerializeSeq, SerializeStruct}; + use serde::{Deserialize, Serialize}; + use serde_bytes::ByteBuf; + use serde_derive::{Deserialize as DeserializeDerive, Serialize as SerializeDerive}; + + use crate::spec::values::{Literal, Map, PrimitiveLiteral, Struct}; + use crate::spec::{MAP_KEY_FIELD_NAME, MAP_VALUE_FIELD_NAME, PrimitiveType, Type}; + use crate::{Error, ErrorKind}; + + #[derive(SerializeDerive, DeserializeDerive, Debug)] + #[serde(transparent)] + /// Raw literal representation used for serde. The serialize way is used for Avro serializer. + pub struct RawLiteral(RawLiteralEnum); + + impl RawLiteral { + /// Covert literal to raw literal. + pub fn try_from(literal: Literal, ty: &Type) -> Result { + Ok(Self(RawLiteralEnum::try_from(literal, ty)?)) + } + + /// Convert raw literal to literal. + pub fn try_into(self, ty: &Type) -> Result, Error> { + self.0.try_into(ty) + } + } + + #[derive(SerializeDerive, Clone, Debug)] + #[serde(untagged)] + enum RawLiteralEnum { + Null, + Boolean(bool), + Int(i32), + Long(i64), + Float(f32), + Double(f64), + String(String), + Bytes(ByteBuf), + List(List), + StringMap(StringMap), + Record(Record), + } + + #[derive(Clone, Debug)] + struct Record { + required: Vec<(String, RawLiteralEnum)>, + optional: Vec<(String, Option)>, + } + + impl Serialize for Record { + fn serialize(&self, serializer: S) -> Result + where S: serde::Serializer { + let len = self.required.len() + self.optional.len(); + let mut record = serializer.serialize_struct("", len)?; + for (k, v) in &self.required { + record.serialize_field(Box::leak(k.clone().into_boxed_str()), &v)?; + } + for (k, v) in &self.optional { + record.serialize_field(Box::leak(k.clone().into_boxed_str()), &v)?; + } + record.end() + } + } + + #[derive(Clone, Debug)] + struct List { + list: Vec>, + required: bool, + } + + impl Serialize for List { + fn serialize(&self, serializer: S) -> Result + where S: serde::Serializer { + let mut seq = serializer.serialize_seq(Some(self.list.len()))?; + for value in &self.list { + if self.required { + seq.serialize_element(value.as_ref().ok_or_else(|| { + serde::ser::Error::custom( + "List element is required, element cannot be null", + ) + })?)?; + } else { + seq.serialize_element(&value)?; + } + } + seq.end() + } + } + + #[derive(Clone, Debug)] + struct StringMap { + raw: Vec<(String, Option)>, + required: bool, + } + + impl Serialize for StringMap { + fn serialize(&self, serializer: S) -> Result + where S: serde::Serializer { + let mut map = serializer.serialize_map(Some(self.raw.len()))?; + for (k, v) in &self.raw { + if self.required { + map.serialize_entry( + k, + v.as_ref().ok_or_else(|| { + serde::ser::Error::custom( + "Map element is required, element cannot be null", + ) + })?, + )?; + } else { + map.serialize_entry(k, v)?; + } + } + map.end() + } + } + + impl<'de> Deserialize<'de> for RawLiteralEnum { + fn deserialize(deserializer: D) -> Result + where D: serde::Deserializer<'de> { + struct RawLiteralVisitor; + impl<'de> Visitor<'de> for RawLiteralVisitor { + type Value = RawLiteralEnum; + + fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { + formatter.write_str("expect") + } + + fn visit_bool(self, v: bool) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Boolean(v)) + } + + fn visit_i32(self, v: i32) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Int(v)) + } + + fn visit_i64(self, v: i64) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Long(v)) + } + + /// Used in json + fn visit_u64(self, v: u64) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Long(v as i64)) + } + + fn visit_f32(self, v: f32) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Float(v)) + } + + fn visit_f64(self, v: f64) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Double(v)) + } + + fn visit_str(self, v: &str) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::String(v.to_string())) + } + + fn visit_bytes(self, v: &[u8]) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Bytes(ByteBuf::from(v))) + } + + fn visit_borrowed_str(self, v: &'de str) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::String(v.to_string())) + } + + fn visit_unit(self) -> Result + where E: serde::de::Error { + Ok(RawLiteralEnum::Null) + } + + fn visit_map(self, mut map: A) -> Result + where A: serde::de::MapAccess<'de> { + let mut required = Vec::new(); + while let Some(key) = map.next_key::()? { + let value = map.next_value::()?; + required.push((key, value)); + } + Ok(RawLiteralEnum::Record(Record { + required, + optional: Vec::new(), + })) + } + + fn visit_seq(self, mut seq: A) -> Result + where A: serde::de::SeqAccess<'de> { + let mut list = Vec::new(); + while let Some(value) = seq.next_element::()? { + list.push(Some(value)); + } + Ok(RawLiteralEnum::List(List { + list, + // `required` only used in serialize, just set default in deserialize. + required: false, + })) + } + } + deserializer.deserialize_any(RawLiteralVisitor) + } + } + + impl RawLiteralEnum { + pub fn try_from(literal: Literal, ty: &Type) -> Result { + let raw = match literal { + Literal::Primitive(prim) => match prim { + PrimitiveLiteral::Boolean(v) => RawLiteralEnum::Boolean(v), + PrimitiveLiteral::Int(v) => RawLiteralEnum::Int(v), + PrimitiveLiteral::Long(v) => RawLiteralEnum::Long(v), + PrimitiveLiteral::Float(v) => RawLiteralEnum::Float(v.0), + PrimitiveLiteral::Double(v) => RawLiteralEnum::Double(v.0), + PrimitiveLiteral::String(v) => RawLiteralEnum::String(v), + PrimitiveLiteral::UInt128(v) => { + RawLiteralEnum::Bytes(ByteBuf::from(v.to_be_bytes())) + } + PrimitiveLiteral::Binary(v) => RawLiteralEnum::Bytes(ByteBuf::from(v)), + PrimitiveLiteral::Int128(v) => { + RawLiteralEnum::Bytes(ByteBuf::from(v.to_be_bytes())) + } + PrimitiveLiteral::AboveMax | PrimitiveLiteral::BelowMin => { + return Err(Error::new( + ErrorKind::DataInvalid, + "Can't convert AboveMax or BelowMax", + )); + } + }, + Literal::Struct(r#struct) => { + let mut required = Vec::new(); + let mut optional = Vec::new(); + if let Type::Struct(struct_ty) = ty { + for (value, field) in r#struct.into_iter().zip(struct_ty.fields()) { + if field.required { + if let Some(value) = value { + required.push(( + field.name.clone(), + RawLiteralEnum::try_from(value, &field.field_type)?, + )); + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + "Can't convert null to required field", + )); + } + } else if let Some(value) = value { + optional.push(( + field.name.clone(), + Some(RawLiteralEnum::try_from(value, &field.field_type)?), + )); + } else { + optional.push((field.name.clone(), None)); + } + } + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Type {ty} should be a struct"), + )); + } + RawLiteralEnum::Record(Record { required, optional }) + } + Literal::List(list) => { + if let Type::List(list_ty) = ty { + let list = list + .into_iter() + .map(|v| { + v.map(|v| { + RawLiteralEnum::try_from(v, &list_ty.element_field.field_type) + }) + .transpose() + }) + .collect::>()?; + RawLiteralEnum::List(List { + list, + required: list_ty.element_field.required, + }) + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Type {ty} should be a list"), + )); + } + } + Literal::Map(map) => { + if let Type::Map(map_ty) = ty { + if let Type::Primitive(PrimitiveType::String) = *map_ty.key_field.field_type + { + let mut raw = Vec::with_capacity(map.len()); + for (k, v) in map { + if let Literal::Primitive(PrimitiveLiteral::String(k)) = k { + raw.push(( + k, + v.map(|v| { + RawLiteralEnum::try_from( + v, + &map_ty.value_field.field_type, + ) + }) + .transpose()?, + )); + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + "literal type is inconsistent with type", + )); + } + } + RawLiteralEnum::StringMap(StringMap { + raw, + required: map_ty.value_field.required, + }) + } else { + let list = map.into_iter().map(|(k,v)| { + let raw_k = + RawLiteralEnum::try_from(k, &map_ty.key_field.field_type)?; + let raw_v = v + .map(|v| { + RawLiteralEnum::try_from(v, &map_ty.value_field.field_type) + }) + .transpose()?; + if map_ty.value_field.required { + Ok(Some(RawLiteralEnum::Record(Record { + required: vec![ + (MAP_KEY_FIELD_NAME.to_string(), raw_k), + (MAP_VALUE_FIELD_NAME.to_string(), raw_v.ok_or_else(||Error::new(ErrorKind::DataInvalid, "Map value is required, value cannot be null"))?), + ], + optional: vec![], + }))) + } else { + Ok(Some(RawLiteralEnum::Record(Record { + required: vec![ + (MAP_KEY_FIELD_NAME.to_string(), raw_k), + ], + optional: vec![ + (MAP_VALUE_FIELD_NAME.to_string(), raw_v) + ], + }))) + } + }).collect::>()?; + RawLiteralEnum::List(List { + list, + required: true, + }) + } + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Type {ty} should be a map"), + )); + } + } + }; + Ok(raw) + } + + pub fn try_into(self, ty: &Type) -> Result, Error> { + let invalid_err = |v: &str| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Unable to convert raw literal ({v}) fail convert to type {ty} for: type mismatch" + ), + ) + }; + let invalid_err_with_reason = |v: &str, reason: &str| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Unable to convert raw literal ({v}) fail convert to type {ty} for: {reason}" + ), + ) + }; + match self { + RawLiteralEnum::Null => Ok(None), + RawLiteralEnum::Boolean(v) => Ok(Some(Literal::bool(v))), + RawLiteralEnum::Int(v) => match ty { + Type::Primitive(PrimitiveType::Int) => Ok(Some(Literal::int(v))), + Type::Primitive(PrimitiveType::Long) => Ok(Some(Literal::long(i64::from(v)))), + Type::Primitive(PrimitiveType::Date) => Ok(Some(Literal::date(v))), + _ => Err(invalid_err("int")), + }, + RawLiteralEnum::Long(v) => match ty { + Type::Primitive(PrimitiveType::Int) => Ok(Some(Literal::int( + i32::try_from(v).map_err(|_| invalid_err("long"))?, + ))), + Type::Primitive(PrimitiveType::Date) => Ok(Some(Literal::date( + i32::try_from(v).map_err(|_| invalid_err("long"))?, + ))), + Type::Primitive(PrimitiveType::Long) => Ok(Some(Literal::long(v))), + Type::Primitive(PrimitiveType::Time) => Ok(Some(Literal::time(v))), + Type::Primitive(PrimitiveType::Timestamp) => Ok(Some(Literal::timestamp(v))), + Type::Primitive(PrimitiveType::Timestamptz) => { + Ok(Some(Literal::timestamptz(v))) + } + _ => Err(invalid_err("long")), + }, + RawLiteralEnum::Float(v) => match ty { + Type::Primitive(PrimitiveType::Float) => Ok(Some(Literal::float(v))), + Type::Primitive(PrimitiveType::Double) => { + Ok(Some(Literal::double(f64::from(v)))) + } + _ => Err(invalid_err("float")), + }, + RawLiteralEnum::Double(v) => match ty { + Type::Primitive(PrimitiveType::Float) => { + let v_32 = v as f32; + if v_32.is_finite() { + let v_64 = f64::from(v_32); + if (v_64 - v).abs() > f32::EPSILON as f64 { + // there is a precision loss + return Err(invalid_err("double")); + } + } + Ok(Some(Literal::float(v_32))) + } + Type::Primitive(PrimitiveType::Double) => Ok(Some(Literal::double(v))), + _ => Err(invalid_err("double")), + }, + RawLiteralEnum::String(v) => match ty { + Type::Primitive(PrimitiveType::String) => Ok(Some(Literal::string(v))), + _ => Err(invalid_err("string")), + }, + RawLiteralEnum::Bytes(v) => match ty { + Type::Primitive(PrimitiveType::Binary) => Ok(Some(Literal::binary(v.to_vec()))), + Type::Primitive(PrimitiveType::Fixed(expected_len)) => { + if v.len() == *expected_len as usize { + Ok(Some(Literal::fixed(v.to_vec()))) + } else { + Err(invalid_err_with_reason( + "bytes", + &format!( + "Fixed type must be exactly {} bytes, got {}", + expected_len, + v.len() + ), + )) + } + } + Type::Primitive(PrimitiveType::Uuid) => { + if v.len() == 16 { + let bytes: [u8; 16] = v.as_slice().try_into().map_err(|_| { + invalid_err_with_reason("bytes", "UUID must be exactly 16 bytes") + })?; + Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes)))) + } else { + Err(invalid_err_with_reason( + "bytes", + "UUID must be exactly 16 bytes", + )) + } + } + Type::Primitive(PrimitiveType::Decimal { precision, .. }) => { + let required_bytes = Type::decimal_required_bytes(*precision)? as usize; + + if v.len() == required_bytes { + // Pad the bytes to 16 bytes (i128 size) with sign extension + let mut padded_bytes = [0u8; 16]; + let start_idx = 16 - v.len(); + + // Copy the input bytes to the end of the array + padded_bytes[start_idx..].copy_from_slice(&v); + + // Sign extend if the number is negative (MSB is 1) + if !v.is_empty() && (v[0] & 0x80) != 0 { + // Fill the padding with 0xFF for negative numbers + for byte in &mut padded_bytes[..start_idx] { + *byte = 0xFF; + } + } + + Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( + i128::from_be_bytes(padded_bytes), + )))) + } else { + Err(invalid_err_with_reason( + "bytes", + &format!( + "Decimal with precision {} must be exactly {} bytes, got {}", + precision, + required_bytes, + v.len() + ), + )) + } + } + _ => Err(invalid_err("bytes")), + }, + RawLiteralEnum::List(v) => match ty { + Type::List(ty) => Ok(Some(Literal::List( + v.list + .into_iter() + .map(|v| { + if let Some(v) = v { + v.try_into(&ty.element_field.field_type) + } else { + Ok(None) + } + }) + .collect::>()?, + ))), + Type::Map(map_ty) => { + let key_ty = map_ty.key_field.field_type.as_ref(); + let value_ty = map_ty.value_field.field_type.as_ref(); + let mut map = Map::new(); + for k_v in v.list { + let k_v = k_v.ok_or_else(|| invalid_err_with_reason("list","In deserialize, None will be represented as Some(RawLiteral::Null), all element in list must be valid"))?; + if let RawLiteralEnum::Record(Record { + required, + optional: _, + }) = k_v + { + if required.len() != 2 { + return Err(invalid_err_with_reason( + "list", + "Record must contains two element(key and value) of array", + )); + } + let mut key = None; + let mut value = None; + required.into_iter().for_each(|(k, v)| { + if k == MAP_KEY_FIELD_NAME { + key = Some(v); + } else if k == MAP_VALUE_FIELD_NAME { + value = Some(v); + } + }); + match (key, value) { + (Some(k), Some(v)) => { + let key = k.try_into(key_ty)?.ok_or_else(|| { + invalid_err_with_reason( + "list", + "Key element in Map must be valid", + ) + })?; + let value = v.try_into(value_ty)?; + if map_ty.value_field.required && value.is_none() { + return Err(invalid_err_with_reason( + "list", + "Value element is required in this Map", + )); + } + map.insert(key, value); + } + _ => { + return Err(invalid_err_with_reason( + "list", + "The elements of record in list are not key and value", + )); + } + } + } else { + return Err(invalid_err_with_reason( + "list", + "Map should represented as record array.", + )); + } + } + Ok(Some(Literal::Map(map))) + } + Type::Primitive(PrimitiveType::Uuid) => { + if v.list.len() != 16 { + return Err(invalid_err_with_reason( + "list", + "The length of list should be 16", + )); + } + let mut bytes = [0u8; 16]; + for (i, v) in v.list.iter().enumerate() { + if let Some(RawLiteralEnum::Long(v)) = v { + bytes[i] = *v as u8; + } else { + return Err(invalid_err_with_reason( + "list", + "The element of list should be int", + )); + } + } + Ok(Some(Literal::uuid(uuid::Uuid::from_bytes(bytes)))) + } + Type::Primitive(PrimitiveType::Decimal { + precision: _, + scale: _, + }) => { + if v.list.len() != 16 { + return Err(invalid_err_with_reason( + "list", + "The length of list should be 16", + )); + } + let mut bytes = [0u8; 16]; + for (i, v) in v.list.iter().enumerate() { + if let Some(RawLiteralEnum::Long(v)) = v { + bytes[i] = *v as u8; + } else { + return Err(invalid_err_with_reason( + "list", + "The element of list should be int", + )); + } + } + Ok(Some(Literal::decimal(i128::from_be_bytes(bytes)))) + } + Type::Primitive(PrimitiveType::Binary) => { + let bytes = v + .list + .into_iter() + .map(|v| { + if let Some(RawLiteralEnum::Long(v)) = v { + Ok(v as u8) + } else { + Err(invalid_err_with_reason( + "list", + "The element of list should be int", + )) + } + }) + .collect::, Error>>()?; + Ok(Some(Literal::binary(bytes))) + } + Type::Primitive(PrimitiveType::Fixed(size)) => { + if v.list.len() != *size as usize { + return Err(invalid_err_with_reason( + "list", + "The length of list should be equal to size", + )); + } + let bytes = v + .list + .into_iter() + .map(|v| { + if let Some(RawLiteralEnum::Long(v)) = v { + Ok(v as u8) + } else { + Err(invalid_err_with_reason( + "list", + "The element of list should be int", + )) + } + }) + .collect::, Error>>()?; + Ok(Some(Literal::fixed(bytes))) + } + _ => Err(invalid_err("list")), + }, + RawLiteralEnum::Record(Record { + required, + optional: _, + }) => match ty { + Type::Struct(struct_ty) => { + let iters: Vec> = required + .into_iter() + .map(|(field_name, value)| { + let field = struct_ty + .field_by_name(field_name.as_str()) + .ok_or_else(|| { + invalid_err_with_reason( + "record", + &format!("field {} is not exist", &field_name), + ) + })?; + let value = value.try_into(&field.field_type)?; + Ok(value) + }) + .collect::>()?; + Ok(Some(Literal::Struct(Struct::from_iter(iters)))) + } + Type::Map(map_ty) => { + if *map_ty.key_field.field_type != Type::Primitive(PrimitiveType::String) { + return Err(invalid_err_with_reason( + "record", + "Map key must be string", + )); + } + let mut map = Map::new(); + for (k, v) in required { + let value = v.try_into(&map_ty.value_field.field_type)?; + if map_ty.value_field.required && value.is_none() { + return Err(invalid_err_with_reason( + "record", + "Value element is required in this Map", + )); + } + map.insert(Literal::string(k), value); + } + Ok(Some(Literal::Map(map))) + } + _ => Err(invalid_err("record")), + }, + RawLiteralEnum::StringMap(_) => Err(invalid_err("string map")), + } + } + } +} diff --git a/crates/iceberg/src/spec/values/struct_value.rs b/crates/iceberg/src/spec/values/struct_value.rs new file mode 100644 index 0000000000..2b44961edf --- /dev/null +++ b/crates/iceberg/src/spec/values/struct_value.rs @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Struct type for Iceberg values + +use std::ops::Index; + +use super::Literal; + +/// The partition struct stores the tuple of partition values for each file. +/// Its type is derived from the partition fields of the partition spec used to write the manifest file. +/// In v2, the partition struct's field ids must match the ids from the partition spec. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct Struct { + /// Vector to store the field values + fields: Vec>, +} + +impl Struct { + /// Create a empty struct. + pub fn empty() -> Self { + Self { fields: Vec::new() } + } + + /// Create a iterator to read the field in order of field_value. + pub fn iter(&self) -> impl ExactSizeIterator> { + self.fields.iter().map(|field| field.as_ref()) + } + + /// returns true if the field at position `index` is null + pub fn is_null_at_index(&self, index: usize) -> bool { + self.fields[index].is_none() + } + + /// Return fields in the struct. + pub fn fields(&self) -> &[Option] { + &self.fields + } +} + +impl Index for Struct { + type Output = Option; + + fn index(&self, idx: usize) -> &Self::Output { + &self.fields[idx] + } +} + +impl IntoIterator for Struct { + type Item = Option; + + type IntoIter = std::vec::IntoIter>; + + fn into_iter(self) -> Self::IntoIter { + self.fields.into_iter() + } +} + +impl FromIterator> for Struct { + fn from_iter>>(iter: I) -> Self { + Struct { + fields: iter.into_iter().collect(), + } + } +} diff --git a/crates/iceberg/src/spec/values/temporal.rs b/crates/iceberg/src/spec/values/temporal.rs new file mode 100644 index 0000000000..ad797e127e --- /dev/null +++ b/crates/iceberg/src/spec/values/temporal.rs @@ -0,0 +1,105 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Temporal value conversions for dates, times, and timestamps + +use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeDelta, TimeZone, Utc}; + +pub(crate) mod date { + use super::*; + + pub(crate) fn date_to_days(date: &NaiveDate) -> i32 { + date.signed_duration_since( + // This is always the same and shouldn't fail + NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(), + ) + .num_days() as i32 + } + + pub(crate) fn days_to_date(days: i32) -> NaiveDate { + // This shouldn't fail until the year 262000 + (chrono::DateTime::UNIX_EPOCH + TimeDelta::try_days(days as i64).unwrap()) + .naive_utc() + .date() + } + + /// Returns unix epoch. + pub(crate) fn unix_epoch() -> DateTime { + Utc.timestamp_nanos(0) + } + + /// Creates date literal from `NaiveDate`, assuming it's utc timezone. + pub(crate) fn date_from_naive_date(date: NaiveDate) -> i32 { + (date - unix_epoch().date_naive()).num_days() as i32 + } +} + +pub(crate) mod time { + use super::*; + + pub(crate) fn time_to_microseconds(time: &NaiveTime) -> i64 { + time.signed_duration_since( + // This is always the same and shouldn't fail + NaiveTime::from_num_seconds_from_midnight_opt(0, 0).unwrap(), + ) + .num_microseconds() + .unwrap() + } + + pub(crate) fn microseconds_to_time(micros: i64) -> NaiveTime { + let (secs, rem) = (micros / 1_000_000, micros % 1_000_000); + + NaiveTime::from_num_seconds_from_midnight_opt(secs as u32, rem as u32 * 1_000).unwrap() + } +} + +pub(crate) mod timestamp { + use super::*; + + pub(crate) fn datetime_to_microseconds(time: &NaiveDateTime) -> i64 { + time.and_utc().timestamp_micros() + } + + pub(crate) fn microseconds_to_datetime(micros: i64) -> NaiveDateTime { + // This shouldn't fail until the year 262000 + DateTime::from_timestamp_micros(micros).unwrap().naive_utc() + } + + pub(crate) fn nanoseconds_to_datetime(nanos: i64) -> NaiveDateTime { + DateTime::from_timestamp_nanos(nanos).naive_utc() + } +} + +pub(crate) mod timestamptz { + use super::*; + + pub(crate) fn datetimetz_to_microseconds(time: &DateTime) -> i64 { + time.timestamp_micros() + } + + pub(crate) fn microseconds_to_datetimetz(micros: i64) -> DateTime { + let (secs, rem) = (micros / 1_000_000, micros % 1_000_000); + + DateTime::from_timestamp(secs, rem as u32 * 1_000).unwrap() + } + + pub(crate) fn nanoseconds_to_datetimetz(nanos: i64) -> DateTime { + let (secs, rem) = (nanos / 1_000_000_000, nanos % 1_000_000_000); + + DateTime::from_timestamp(secs, rem as u32).unwrap() + } +} diff --git a/crates/iceberg/src/spec/values/tests.rs b/crates/iceberg/src/spec/values/tests.rs new file mode 100644 index 0000000000..0e99d44dfe --- /dev/null +++ b/crates/iceberg/src/spec/values/tests.rs @@ -0,0 +1,1334 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Tests for Iceberg value types + +use apache_avro::to_value; +use apache_avro::types::Value; +use ordered_float::OrderedFloat; +use rust_decimal::Decimal; +use serde_bytes::ByteBuf; +use serde_json::Value as JsonValue; +use uuid::Uuid; + +use crate::ErrorKind; +use crate::avro::schema_to_avro_schema; +use crate::spec::Schema; +use crate::spec::Type::Primitive; +use crate::spec::datatypes::{ListType, MapType, NestedField, PrimitiveType, StructType, Type}; +use crate::spec::values::datum::{INT_MAX, INT_MIN, LONG_MAX, LONG_MIN}; +use crate::spec::values::serde::_serde; +use crate::spec::values::{Datum, Literal, Map, PrimitiveLiteral, RawLiteral, Struct}; + +fn check_json_serde(json: &str, expected_literal: Literal, expected_type: &Type) { + let raw_json_value = serde_json::from_str::(json).unwrap(); + let desered_literal = Literal::try_from_json(raw_json_value.clone(), expected_type).unwrap(); + assert_eq!(desered_literal, Some(expected_literal.clone())); + + let expected_json_value: JsonValue = expected_literal.try_into_json(expected_type).unwrap(); + let sered_json = serde_json::to_string(&expected_json_value).unwrap(); + let parsed_json_value = serde_json::from_str::(&sered_json).unwrap(); + + assert_eq!(parsed_json_value, raw_json_value); +} + +fn check_avro_bytes_serde(input: Vec, expected_datum: Datum, expected_type: &PrimitiveType) { + let raw_schema = r#""bytes""#; + let schema = apache_avro::Schema::parse_str(raw_schema).unwrap(); + + let bytes = ByteBuf::from(input); + let datum = Datum::try_from_bytes(&bytes, expected_type.clone()).unwrap(); + assert_eq!(datum, expected_datum); + + let mut writer = apache_avro::Writer::new(&schema, Vec::new()); + writer.append_ser(datum.to_bytes().unwrap()).unwrap(); + let encoded = writer.into_inner().unwrap(); + let reader = apache_avro::Reader::with_schema(&schema, &*encoded).unwrap(); + + for record in reader { + let result = apache_avro::from_value::(&record.unwrap()).unwrap(); + let desered_datum = Datum::try_from_bytes(&result, expected_type.clone()).unwrap(); + assert_eq!(desered_datum, expected_datum); + } +} + +fn check_convert_with_avro(expected_literal: Literal, expected_type: &Type) { + let fields = vec![NestedField::required(1, "col", expected_type.clone()).into()]; + let schema = Schema::builder() + .with_fields(fields.clone()) + .build() + .unwrap(); + let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); + let struct_type = Type::Struct(StructType::new(fields)); + let struct_literal = Literal::Struct(Struct::from_iter(vec![Some(expected_literal.clone())])); + + let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); + let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); + writer.append_ser(raw_literal).unwrap(); + let encoded = writer.into_inner().unwrap(); + + let reader = apache_avro::Reader::new(&*encoded).unwrap(); + for record in reader { + let result = apache_avro::from_value::(&record.unwrap()).unwrap(); + let desered_literal = result.try_into(&struct_type).unwrap().unwrap(); + assert_eq!(desered_literal, struct_literal); + } +} + +fn check_serialize_avro(literal: Literal, ty: &Type, expect_value: Value) { + let expect_value = Value::Record(vec![("col".to_string(), expect_value)]); + + let fields = vec![NestedField::required(1, "col", ty.clone()).into()]; + let schema = Schema::builder() + .with_fields(fields.clone()) + .build() + .unwrap(); + let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); + let struct_type = Type::Struct(StructType::new(fields)); + let struct_literal = Literal::Struct(Struct::from_iter(vec![Some(literal.clone())])); + let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); + let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); + let value = to_value(raw_literal) + .unwrap() + .resolve(&avro_schema) + .unwrap(); + writer.append_value_ref(&value).unwrap(); + let encoded = writer.into_inner().unwrap(); + + let reader = apache_avro::Reader::new(&*encoded).unwrap(); + for record in reader { + assert_eq!(record.unwrap(), expect_value); + } +} + +#[test] +fn json_boolean() { + let record = r#"true"#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Boolean(true)), + &Type::Primitive(PrimitiveType::Boolean), + ); +} + +#[test] +fn json_int() { + let record = r#"32"#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Int(32)), + &Type::Primitive(PrimitiveType::Int), + ); +} + +#[test] +fn json_long() { + let record = r#"32"#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Long(32)), + &Type::Primitive(PrimitiveType::Long), + ); +} + +#[test] +fn json_float() { + let record = r#"1.0"#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(1.0))), + &Type::Primitive(PrimitiveType::Float), + ); +} + +#[test] +fn json_double() { + let record = r#"1.0"#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(1.0))), + &Type::Primitive(PrimitiveType::Double), + ); +} + +#[test] +fn json_date() { + let record = r#""2017-11-16""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Int(17486)), + &Type::Primitive(PrimitiveType::Date), + ); +} + +#[test] +fn json_time() { + let record = r#""22:31:08.123456""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Long(81068123456)), + &Type::Primitive(PrimitiveType::Time), + ); +} + +#[test] +fn json_timestamp() { + let record = r#""2017-11-16T22:31:08.123456""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), + &Type::Primitive(PrimitiveType::Timestamp), + ); +} + +#[test] +fn json_timestamptz() { + let record = r#""2017-11-16T22:31:08.123456+00:00""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), + &Type::Primitive(PrimitiveType::Timestamptz), + ); +} + +#[test] +fn json_string() { + let record = r#""iceberg""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::String("iceberg".to_string())), + &Type::Primitive(PrimitiveType::String), + ); +} + +#[test] +fn json_uuid() { + let record = r#""f79c3e09-677c-4bbd-a479-3f349cb785e7""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::UInt128( + Uuid::parse_str("f79c3e09-677c-4bbd-a479-3f349cb785e7") + .unwrap() + .as_u128(), + )), + &Type::Primitive(PrimitiveType::Uuid), + ); +} + +#[test] +fn json_decimal() { + let record = r#""14.20""#; + + check_json_serde( + record, + Literal::Primitive(PrimitiveLiteral::Int128(1420)), + &Type::decimal(28, 2).unwrap(), + ); +} + +#[test] +fn json_struct() { + let record = r#"{"1": 1, "2": "bar", "3": null}"#; + + check_json_serde( + record, + Literal::Struct(Struct::from_iter(vec![ + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + Some(Literal::Primitive(PrimitiveLiteral::String( + "bar".to_string(), + ))), + None, + ])), + &Type::Struct(StructType::new(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "address", Type::Primitive(PrimitiveType::String)).into(), + ])), + ); +} + +#[test] +fn json_list() { + let record = r#"[1, 2, 3, null]"#; + + check_json_serde( + record, + Literal::List(vec![ + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + Some(Literal::Primitive(PrimitiveLiteral::Int(3))), + None, + ]), + &Type::List(ListType { + element_field: NestedField::list_element(0, Type::Primitive(PrimitiveType::Int), true) + .into(), + }), + ); +} + +#[test] +fn json_map() { + let record = r#"{ "keys": ["a", "b", "c"], "values": [1, 2, null] }"#; + + check_json_serde( + record, + Literal::Map(Map::from([ + ( + Literal::Primitive(PrimitiveLiteral::String("a".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("b".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("c".to_string())), + None, + ), + ])), + &Type::Map(MapType { + key_field: NestedField::map_key_element(0, Type::Primitive(PrimitiveType::String)) + .into(), + value_field: NestedField::map_value_element( + 1, + Type::Primitive(PrimitiveType::Int), + true, + ) + .into(), + }), + ); +} + +#[test] +fn avro_bytes_boolean() { + let bytes = vec![1u8]; + + check_avro_bytes_serde(bytes, Datum::bool(true), &PrimitiveType::Boolean); +} + +#[test] +fn avro_bytes_int() { + let bytes = vec![32u8, 0u8, 0u8, 0u8]; + + check_avro_bytes_serde(bytes, Datum::int(32), &PrimitiveType::Int); +} + +#[test] +fn avro_bytes_long() { + let bytes = vec![32u8, 0u8, 0u8, 0u8, 0u8, 0u8, 0u8, 0u8]; + + check_avro_bytes_serde(bytes, Datum::long(32), &PrimitiveType::Long); +} + +#[test] +fn avro_bytes_long_from_int() { + let bytes = vec![32u8, 0u8, 0u8, 0u8]; + + check_avro_bytes_serde(bytes, Datum::long(32), &PrimitiveType::Long); +} + +#[test] +fn avro_bytes_float() { + let bytes = vec![0u8, 0u8, 128u8, 63u8]; + + check_avro_bytes_serde(bytes, Datum::float(1.0), &PrimitiveType::Float); +} + +#[test] +fn avro_bytes_double() { + let bytes = vec![0u8, 0u8, 0u8, 0u8, 0u8, 0u8, 240u8, 63u8]; + + check_avro_bytes_serde(bytes, Datum::double(1.0), &PrimitiveType::Double); +} + +#[test] +fn avro_bytes_double_from_float() { + let bytes = vec![0u8, 0u8, 128u8, 63u8]; + + check_avro_bytes_serde(bytes, Datum::double(1.0), &PrimitiveType::Double); +} + +#[test] +fn avro_bytes_string() { + let bytes = vec![105u8, 99u8, 101u8, 98u8, 101u8, 114u8, 103u8]; + + check_avro_bytes_serde(bytes, Datum::string("iceberg"), &PrimitiveType::String); +} + +#[test] +fn avro_bytes_decimal() { + // (input_bytes, decimal_num, expect_scale, expect_precision) + let cases = vec![ + (vec![4u8, 210u8], 1234, 2, 38), + (vec![251u8, 46u8], -1234, 2, 38), + (vec![4u8, 210u8], 1234, 3, 38), + (vec![251u8, 46u8], -1234, 3, 38), + (vec![42u8], 42, 2, 1), + (vec![214u8], -42, 2, 1), + ]; + + for (input_bytes, decimal_num, expect_scale, expect_precision) in cases { + check_avro_bytes_serde( + input_bytes, + Datum::decimal_with_precision( + Decimal::new(decimal_num, expect_scale), + expect_precision, + ) + .unwrap(), + &PrimitiveType::Decimal { + precision: expect_precision, + scale: expect_scale, + }, + ); + } +} + +#[test] +fn avro_bytes_decimal_expect_error() { + // (decimal_num, expect_scale, expect_precision) + let cases = vec![(1234, 2, 1)]; + + for (decimal_num, expect_scale, expect_precision) in cases { + let result = Datum::decimal_with_precision( + Decimal::new(decimal_num, expect_scale), + expect_precision, + ); + assert!(result.is_err(), "expect error but got {result:?}"); + assert_eq!( + result.unwrap_err().kind(), + ErrorKind::DataInvalid, + "expect error DataInvalid", + ); + } +} + +fn check_raw_literal_bytes_serde_via_avro( + input_bytes: Vec, + expected_literal: Literal, + expected_type: &Type, +) { + use apache_avro::types::Value; + + // Create an Avro bytes value and deserialize it through the RawLiteral path + let avro_value = Value::Bytes(input_bytes); + let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); + let result = raw_literal.try_into(expected_type).unwrap(); + assert_eq!(result, Some(expected_literal)); +} + +fn check_raw_literal_bytes_error_via_avro(input_bytes: Vec, expected_type: &Type) { + use apache_avro::types::Value; + + let avro_value = Value::Bytes(input_bytes); + let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); + let result = raw_literal.try_into(expected_type); + assert!(result.is_err(), "Expected error but got: {:?}", result); +} + +#[test] +fn test_raw_literal_bytes_binary() { + let bytes = vec![1u8, 2u8, 3u8, 4u8, 5u8]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::binary(bytes), + &Type::Primitive(PrimitiveType::Binary), + ); +} + +#[test] +fn test_raw_literal_bytes_binary_empty() { + let bytes = vec![]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::binary(bytes), + &Type::Primitive(PrimitiveType::Binary), + ); +} + +#[test] +fn test_raw_literal_bytes_fixed_correct_length() { + let bytes = vec![1u8, 2u8, 3u8, 4u8]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::fixed(bytes), + &Type::Primitive(PrimitiveType::Fixed(4)), + ); +} + +#[test] +fn test_raw_literal_bytes_fixed_wrong_length() { + let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but expecting 4 + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Fixed(4))); +} + +#[test] +fn test_raw_literal_bytes_fixed_empty_correct_length() { + let bytes = vec![]; + check_raw_literal_bytes_serde_via_avro( + bytes.clone(), + Literal::fixed(bytes), + &Type::Primitive(PrimitiveType::Fixed(0)), + ); +} + +#[test] +fn test_raw_literal_bytes_uuid_correct_length() { + let uuid_bytes = vec![ + 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, + 0xef, + ]; + let expected_uuid = u128::from_be_bytes([ + 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, + 0xef, + ]); + check_raw_literal_bytes_serde_via_avro( + uuid_bytes, + Literal::Primitive(PrimitiveLiteral::UInt128(expected_uuid)), + &Type::Primitive(PrimitiveType::Uuid), + ); +} + +#[test] +fn test_raw_literal_bytes_uuid_wrong_length() { + let bytes = vec![1u8, 2u8, 3u8]; // 3 bytes, but UUID needs 16 + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Uuid)); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_4_scale_2() { + // Precision 4 requires 2 bytes + let decimal_bytes = vec![0x04, 0xd2]; // 1234 in 2 bytes + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_4_negative() { + // Precision 4 requires 2 bytes, negative number + let decimal_bytes = vec![0xfb, 0x2e]; // -1234 in 2 bytes + let expected_decimal = -1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_9_scale_2() { + // Precision 9 requires 4 bytes + let decimal_bytes = vec![0x00, 0x12, 0xd6, 0x87]; // 1234567 in 4 bytes + let expected_decimal = 1234567i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 9, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_18_scale_2() { + // Precision 18 requires 8 bytes + let decimal_bytes = vec![0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0xd2]; // 1234 in 8 bytes + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 18, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_38_scale_2() { + // Precision 38 requires 16 bytes (maximum precision) + let decimal_bytes = vec![ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, + 0xd2, // 1234 in 16 bytes + ]; + let expected_decimal = 1234i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 38, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_1_scale_0() { + // Precision 1 requires 1 byte + let decimal_bytes = vec![0x07]; // 7 in 1 byte + let expected_decimal = 7i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 1, + scale: 0, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_precision_1_negative() { + // Precision 1 requires 1 byte, negative number + let decimal_bytes = vec![0xf9]; // -7 in 1 byte (two's complement) + let expected_decimal = -7i128; + check_raw_literal_bytes_serde_via_avro( + decimal_bytes, + Literal::Primitive(PrimitiveLiteral::Int128(expected_decimal)), + &Type::Primitive(PrimitiveType::Decimal { + precision: 1, + scale: 0, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_wrong_length() { + // 3 bytes provided, but precision 4 requires 2 bytes + let bytes = vec![1u8, 2u8, 3u8]; + check_raw_literal_bytes_error_via_avro( + bytes, + &Type::Primitive(PrimitiveType::Decimal { + precision: 4, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_decimal_wrong_length_too_few() { + // 1 byte provided, but precision 9 requires 4 bytes + let bytes = vec![0x42]; + check_raw_literal_bytes_error_via_avro( + bytes, + &Type::Primitive(PrimitiveType::Decimal { + precision: 9, + scale: 2, + }), + ); +} + +#[test] +fn test_raw_literal_bytes_unsupported_type() { + let bytes = vec![1u8, 2u8, 3u8, 4u8]; + check_raw_literal_bytes_error_via_avro(bytes, &Type::Primitive(PrimitiveType::Int)); +} + +#[test] +fn avro_convert_test_int() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Int(32)), + &Type::Primitive(PrimitiveType::Int), + ); +} + +#[test] +fn avro_convert_test_long() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Long(32)), + &Type::Primitive(PrimitiveType::Long), + ); +} + +#[test] +fn avro_convert_test_float() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Float(OrderedFloat(1.0))), + &Type::Primitive(PrimitiveType::Float), + ); +} + +#[test] +fn avro_convert_test_double() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Double(OrderedFloat(1.0))), + &Type::Primitive(PrimitiveType::Double), + ); +} + +#[test] +fn avro_convert_test_string() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::String("iceberg".to_string())), + &Type::Primitive(PrimitiveType::String), + ); +} + +#[test] +fn avro_convert_test_date() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Int(17486)), + &Type::Primitive(PrimitiveType::Date), + ); +} + +#[test] +fn avro_convert_test_time() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Long(81068123456)), + &Type::Primitive(PrimitiveType::Time), + ); +} + +#[test] +fn avro_convert_test_timestamp() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), + &Type::Primitive(PrimitiveType::Timestamp), + ); +} + +#[test] +fn avro_convert_test_timestamptz() { + check_convert_with_avro( + Literal::Primitive(PrimitiveLiteral::Long(1510871468123456)), + &Type::Primitive(PrimitiveType::Timestamptz), + ); +} + +#[test] +fn avro_convert_test_list() { + check_convert_with_avro( + Literal::List(vec![ + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + Some(Literal::Primitive(PrimitiveLiteral::Int(3))), + None, + ]), + &Type::List(ListType { + element_field: NestedField::list_element(0, Type::Primitive(PrimitiveType::Int), false) + .into(), + }), + ); + + check_convert_with_avro( + Literal::List(vec![ + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + Some(Literal::Primitive(PrimitiveLiteral::Int(3))), + ]), + &Type::List(ListType { + element_field: NestedField::list_element(0, Type::Primitive(PrimitiveType::Int), true) + .into(), + }), + ); +} + +fn check_convert_with_avro_map(expected_literal: Literal, expected_type: &Type) { + let fields = vec![NestedField::required(1, "col", expected_type.clone()).into()]; + let schema = Schema::builder() + .with_fields(fields.clone()) + .build() + .unwrap(); + let avro_schema = schema_to_avro_schema("test", &schema).unwrap(); + let struct_type = Type::Struct(StructType::new(fields)); + let struct_literal = Literal::Struct(Struct::from_iter(vec![Some(expected_literal.clone())])); + + let mut writer = apache_avro::Writer::new(&avro_schema, Vec::new()); + let raw_literal = RawLiteral::try_from(struct_literal.clone(), &struct_type).unwrap(); + writer.append_ser(raw_literal).unwrap(); + let encoded = writer.into_inner().unwrap(); + + let reader = apache_avro::Reader::new(&*encoded).unwrap(); + for record in reader { + let result = apache_avro::from_value::(&record.unwrap()).unwrap(); + let desered_literal = result.try_into(&struct_type).unwrap().unwrap(); + match (&desered_literal, &struct_literal) { + (Literal::Struct(desered), Literal::Struct(expected)) => { + match (&desered.fields()[0], &expected.fields()[0]) { + (Some(Literal::Map(desered)), Some(Literal::Map(expected))) => { + assert!(desered.has_same_content(expected)) + } + _ => { + unreachable!() + } + } + } + _ => { + panic!("unexpected literal type"); + } + } + } +} + +#[test] +fn avro_convert_test_map() { + check_convert_with_avro_map( + Literal::Map(Map::from([ + ( + Literal::Primitive(PrimitiveLiteral::Int(1)), + Some(Literal::Primitive(PrimitiveLiteral::Long(1))), + ), + ( + Literal::Primitive(PrimitiveLiteral::Int(2)), + Some(Literal::Primitive(PrimitiveLiteral::Long(2))), + ), + (Literal::Primitive(PrimitiveLiteral::Int(3)), None), + ])), + &Type::Map(MapType { + key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::Int)).into(), + value_field: NestedField::map_value_element( + 3, + Type::Primitive(PrimitiveType::Long), + false, + ) + .into(), + }), + ); + + check_convert_with_avro_map( + Literal::Map(Map::from([ + ( + Literal::Primitive(PrimitiveLiteral::Int(1)), + Some(Literal::Primitive(PrimitiveLiteral::Long(1))), + ), + ( + Literal::Primitive(PrimitiveLiteral::Int(2)), + Some(Literal::Primitive(PrimitiveLiteral::Long(2))), + ), + ( + Literal::Primitive(PrimitiveLiteral::Int(3)), + Some(Literal::Primitive(PrimitiveLiteral::Long(3))), + ), + ])), + &Type::Map(MapType { + key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::Int)).into(), + value_field: NestedField::map_value_element( + 3, + Type::Primitive(PrimitiveType::Long), + true, + ) + .into(), + }), + ); +} + +#[test] +fn avro_convert_test_string_map() { + check_convert_with_avro_map( + Literal::Map(Map::from([ + ( + Literal::Primitive(PrimitiveLiteral::String("a".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("b".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("c".to_string())), + None, + ), + ])), + &Type::Map(MapType { + key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::String)) + .into(), + value_field: NestedField::map_value_element( + 3, + Type::Primitive(PrimitiveType::Int), + false, + ) + .into(), + }), + ); + + check_convert_with_avro_map( + Literal::Map(Map::from([ + ( + Literal::Primitive(PrimitiveLiteral::String("a".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("b".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(2))), + ), + ( + Literal::Primitive(PrimitiveLiteral::String("c".to_string())), + Some(Literal::Primitive(PrimitiveLiteral::Int(3))), + ), + ])), + &Type::Map(MapType { + key_field: NestedField::map_key_element(2, Type::Primitive(PrimitiveType::String)) + .into(), + value_field: NestedField::map_value_element( + 3, + Type::Primitive(PrimitiveType::Int), + true, + ) + .into(), + }), + ); +} + +#[test] +fn avro_convert_test_record() { + check_convert_with_avro( + Literal::Struct(Struct::from_iter(vec![ + Some(Literal::Primitive(PrimitiveLiteral::Int(1))), + Some(Literal::Primitive(PrimitiveLiteral::String( + "bar".to_string(), + ))), + None, + ])), + &Type::Struct(StructType::new(vec![ + NestedField::required(2, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(3, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(4, "address", Type::Primitive(PrimitiveType::String)).into(), + ])), + ); +} + +// # TODO:https://github.com/apache/iceberg-rust/issues/86 +// rust avro don't support deserialize any bytes representation now: +// - binary +// - decimal +#[test] +fn avro_convert_test_binary_ser() { + let literal = Literal::Primitive(PrimitiveLiteral::Binary(vec![1, 2, 3, 4, 5])); + let ty = Type::Primitive(PrimitiveType::Binary); + let expect_value = Value::Bytes(vec![1, 2, 3, 4, 5]); + check_serialize_avro(literal, &ty, expect_value); +} + +#[test] +fn avro_convert_test_decimal_ser() { + let literal = Literal::decimal(12345); + let ty = Type::Primitive(PrimitiveType::Decimal { + precision: 9, + scale: 8, + }); + let expect_value = Value::Decimal(apache_avro::Decimal::from(12345_i128.to_be_bytes())); + check_serialize_avro(literal, &ty, expect_value); +} + +// # TODO:https://github.com/apache/iceberg-rust/issues/86 +// rust avro can't support to convert any byte-like type to fixed in avro now. +// - uuid ser/de +// - fixed ser/de + +#[test] +fn test_parse_timestamp() { + let value = Datum::timestamp_from_str("2021-08-01T01:09:00.0899").unwrap(); + assert_eq!(&format!("{value}"), "2021-08-01 01:09:00.089900"); + + let value = Datum::timestamp_from_str("2023-01-06T00:00:00").unwrap(); + assert_eq!(&format!("{value}"), "2023-01-06 00:00:00"); + + let value = Datum::timestamp_from_str("2021-08-01T01:09:00.0899+0800"); + assert!(value.is_err(), "Parse timestamp with timezone should fail!"); + + let value = Datum::timestamp_from_str("dfa"); + assert!( + value.is_err(), + "Parse timestamp with invalid input should fail!" + ); +} + +#[test] +fn test_parse_timestamptz() { + let value = Datum::timestamptz_from_str("2021-08-01T09:09:00.0899+0800").unwrap(); + assert_eq!(&format!("{value}"), "2021-08-01 01:09:00.089900 UTC"); + + let value = Datum::timestamptz_from_str("2021-08-01T01:09:00.0899"); + assert!( + value.is_err(), + "Parse timestamptz without timezone should fail!" + ); + + let value = Datum::timestamptz_from_str("dfa"); + assert!( + value.is_err(), + "Parse timestamptz with invalid input should fail!" + ); +} + +#[test] +fn test_datum_ser_deser() { + let test_fn = |datum: Datum| { + let json = serde_json::to_value(&datum).unwrap(); + let desered_datum: Datum = serde_json::from_value(json).unwrap(); + assert_eq!(datum, desered_datum); + }; + let datum = Datum::int(1); + test_fn(datum); + let datum = Datum::long(1); + test_fn(datum); + + let datum = Datum::float(1.0); + test_fn(datum); + let datum = Datum::float(0_f32); + test_fn(datum); + let datum = Datum::float(-0_f32); + test_fn(datum); + let datum = Datum::float(f32::MAX); + test_fn(datum); + let datum = Datum::float(f32::MIN); + test_fn(datum); + + // serde_json can't serialize f32::INFINITY, f32::NEG_INFINITY, f32::NAN + let datum = Datum::float(f32::INFINITY); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + let datum = Datum::float(f32::NEG_INFINITY); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + let datum = Datum::float(f32::NAN); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + + let datum = Datum::double(1.0); + test_fn(datum); + let datum = Datum::double(f64::MAX); + test_fn(datum); + let datum = Datum::double(f64::MIN); + test_fn(datum); + + // serde_json can't serialize f32::INFINITY, f32::NEG_INFINITY, f32::NAN + let datum = Datum::double(f64::INFINITY); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + let datum = Datum::double(f64::NEG_INFINITY); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + let datum = Datum::double(f64::NAN); + let json = serde_json::to_string(&datum).unwrap(); + assert!(serde_json::from_str::(&json).is_err()); + + let datum = Datum::string("iceberg"); + test_fn(datum); + let datum = Datum::bool(true); + test_fn(datum); + let datum = Datum::date(17486); + test_fn(datum); + let datum = Datum::time_from_hms_micro(22, 15, 33, 111).unwrap(); + test_fn(datum); + let datum = Datum::timestamp_micros(1510871468123456); + test_fn(datum); + let datum = Datum::timestamptz_micros(1510871468123456); + test_fn(datum); + let datum = Datum::uuid(Uuid::parse_str("f79c3e09-677c-4bbd-a479-3f349cb785e7").unwrap()); + test_fn(datum); + let datum = Datum::decimal(1420).unwrap(); + test_fn(datum); + let datum = Datum::binary(vec![1, 2, 3, 4, 5]); + test_fn(datum); + let datum = Datum::fixed(vec![1, 2, 3, 4, 5]); + test_fn(datum); +} + +#[test] +fn test_datum_date_convert_to_int() { + let datum_date = Datum::date(12345); + + let result = datum_date.to(&Primitive(PrimitiveType::Int)).unwrap(); + + let expected = Datum::int(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_int_convert_to_date() { + let datum_int = Datum::int(12345); + + let result = datum_int.to(&Primitive(PrimitiveType::Date)).unwrap(); + + let expected = Datum::date(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_long_convert_to_int() { + let datum = Datum::long(12345); + + let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); + + let expected = Datum::int(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_long_convert_to_int_above_max() { + let datum = Datum::long(INT_MAX as i64 + 1); + + let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); + + let expected = Datum::new(PrimitiveType::Int, PrimitiveLiteral::AboveMax); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_long_convert_to_int_below_min() { + let datum = Datum::long(INT_MIN as i64 - 1); + + let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); + + let expected = Datum::new(PrimitiveType::Int, PrimitiveLiteral::BelowMin); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_long_convert_to_timestamp() { + let datum = Datum::long(12345); + + let result = datum.to(&Primitive(PrimitiveType::Timestamp)).unwrap(); + + let expected = Datum::timestamp_micros(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_long_convert_to_timestamptz() { + let datum = Datum::long(12345); + + let result = datum.to(&Primitive(PrimitiveType::Timestamptz)).unwrap(); + + let expected = Datum::timestamptz_micros(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_decimal_convert_to_long() { + let datum = Datum::decimal(12345).unwrap(); + + let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); + + let expected = Datum::long(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_decimal_convert_to_long_above_max() { + let datum = Datum::decimal(LONG_MAX as i128 + 1).unwrap(); + + let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); + + let expected = Datum::new(PrimitiveType::Long, PrimitiveLiteral::AboveMax); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_decimal_convert_to_long_below_min() { + let datum = Datum::decimal(LONG_MIN as i128 - 1).unwrap(); + + let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); + + let expected = Datum::new(PrimitiveType::Long, PrimitiveLiteral::BelowMin); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_string_convert_to_boolean() { + let datum = Datum::string("true"); + + let result = datum.to(&Primitive(PrimitiveType::Boolean)).unwrap(); + + let expected = Datum::bool(true); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_string_convert_to_int() { + let datum = Datum::string("12345"); + + let result = datum.to(&Primitive(PrimitiveType::Int)).unwrap(); + + let expected = Datum::int(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_string_convert_to_long() { + let datum = Datum::string("12345"); + + let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); + + let expected = Datum::long(12345); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_string_convert_to_timestamp() { + let datum = Datum::string("1925-05-20T19:25:00.000"); + + let result = datum.to(&Primitive(PrimitiveType::Timestamp)).unwrap(); + + let expected = Datum::timestamp_micros(-1407990900000000); + + assert_eq!(result, expected); +} + +#[test] +fn test_datum_string_convert_to_timestamptz() { + let datum = Datum::string("1925-05-20T19:25:00.000 UTC"); + + let result = datum.to(&Primitive(PrimitiveType::Timestamptz)).unwrap(); + + let expected = Datum::timestamptz_micros(-1407990900000000); + + assert_eq!(result, expected); +} + +#[test] +fn test_iceberg_float_order() { + // Test float ordering + let float_values = vec![ + Datum::float(f32::NAN), + Datum::float(-f32::NAN), + Datum::float(f32::MAX), + Datum::float(f32::MIN), + Datum::float(f32::INFINITY), + Datum::float(-f32::INFINITY), + Datum::float(1.0), + Datum::float(-1.0), + Datum::float(0.0), + Datum::float(-0.0), + ]; + + let mut float_sorted = float_values.clone(); + float_sorted.sort_by(|a, b| a.partial_cmp(b).unwrap()); + + let float_expected = vec![ + Datum::float(-f32::NAN), + Datum::float(-f32::INFINITY), + Datum::float(f32::MIN), + Datum::float(-1.0), + Datum::float(-0.0), + Datum::float(0.0), + Datum::float(1.0), + Datum::float(f32::MAX), + Datum::float(f32::INFINITY), + Datum::float(f32::NAN), + ]; + + assert_eq!(float_sorted, float_expected); + + // Test double ordering + let double_values = vec![ + Datum::double(f64::NAN), + Datum::double(-f64::NAN), + Datum::double(f64::INFINITY), + Datum::double(-f64::INFINITY), + Datum::double(f64::MAX), + Datum::double(f64::MIN), + Datum::double(1.0), + Datum::double(-1.0), + Datum::double(0.0), + Datum::double(-0.0), + ]; + + let mut double_sorted = double_values.clone(); + double_sorted.sort_by(|a, b| a.partial_cmp(b).unwrap()); + + let double_expected = vec![ + Datum::double(-f64::NAN), + Datum::double(-f64::INFINITY), + Datum::double(f64::MIN), + Datum::double(-1.0), + Datum::double(-0.0), + Datum::double(0.0), + Datum::double(1.0), + Datum::double(f64::MAX), + Datum::double(f64::INFINITY), + Datum::double(f64::NAN), + ]; + + assert_eq!(double_sorted, double_expected); +} + +/// Test Date deserialization from JSON as number (days since epoch). +/// +/// This reproduces the scenario from Iceberg Java's TestAddFilesProcedure where: +/// - Date partition columns have initial_default values in manifests +/// - These values are serialized as days since epoch (e.g., 18628 for 2021-01-01) +/// - The JSON schema includes: {"type":"date","initial-default":18628} +/// +/// Prior to this fix, Date values in JSON were only parsed from String format ("2021-01-01"), +/// causing initial_default values to be lost during schema deserialization. +/// +/// This test ensures both formats are supported: +/// - String format: "2021-01-01" (used in table metadata) +/// - Number format: 18628 (used in initial-default values from add_files) +/// +/// See: Iceberg Java TestAddFilesProcedure.addDataPartitionedByDateToPartitioned() +#[test] +fn test_date_from_json_as_number() { + use serde_json::json; + + // Test Date as number (days since epoch) - used in initial-default from add_files + let date_number = json!(18628); // 2021-01-01 is 18628 days since 1970-01-01 + let result = + Literal::try_from_json(date_number, &Type::Primitive(PrimitiveType::Date)).unwrap(); + assert_eq!( + result, + Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) + ); + + // Test Date as string - traditional format + let date_string = json!("2021-01-01"); + let result = + Literal::try_from_json(date_string, &Type::Primitive(PrimitiveType::Date)).unwrap(); + assert_eq!( + result, + Some(Literal::Primitive(PrimitiveLiteral::Int(18628))) + ); + + // Both formats should produce the same Literal value +} From 1fcad93641603917829cdd6d60f4570062394855 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 12 Nov 2025 02:49:54 -0800 Subject: [PATCH 071/891] feat(datafusion): Support `INSERT INTO` partitioned tables (#1827) ## Which issue does this PR close? - Closes #1828 - Related to #1540 ## What changes are included in this PR? - Use project to calculate partition values for record batches - Repartition inputs for table_provider::insert_into - Initialize partition_splitter in TaskWriter's constructor - Use TaskWriter in `IcebergWriteExec` to support partitioned data ## Are these changes tested? Added an ut --- .../arrow/record_batch_partition_splitter.rs | 20 +-- .../src/physical_plan/repartition.rs | 1 - .../datafusion/src/physical_plan/write.rs | 48 +++--- .../integrations/datafusion/src/table/mod.rs | 43 +++-- .../datafusion/src/task_writer.rs | 71 ++++---- .../tests/integration_datafusion_test.rs | 152 +++++++++++++++++- 6 files changed, 239 insertions(+), 96 deletions(-) diff --git a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs index dcdd9c68ee..7b83621f2d 100644 --- a/crates/iceberg/src/arrow/record_batch_partition_splitter.rs +++ b/crates/iceberg/src/arrow/record_batch_partition_splitter.rs @@ -59,7 +59,7 @@ impl RecordBatchPartitionSplitter { /// # Returns /// /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. - pub fn new( + pub fn try_new( iceberg_schema: SchemaRef, partition_spec: PartitionSpecRef, calculator: Option, @@ -87,12 +87,12 @@ impl RecordBatchPartitionSplitter { /// # Returns /// /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. - pub fn new_with_computed_values( + pub fn try_new_with_computed_values( iceberg_schema: SchemaRef, partition_spec: PartitionSpecRef, ) -> Result { let calculator = PartitionValueCalculator::try_new(&partition_spec, &iceberg_schema)?; - Self::new(iceberg_schema, partition_spec, Some(calculator)) + Self::try_new(iceberg_schema, partition_spec, Some(calculator)) } /// Create a new RecordBatchPartitionSplitter expecting pre-computed partition values. @@ -108,11 +108,11 @@ impl RecordBatchPartitionSplitter { /// # Returns /// /// Returns a new `RecordBatchPartitionSplitter` instance or an error if initialization fails. - pub fn new_with_precomputed_values( + pub fn try_new_with_precomputed_values( iceberg_schema: SchemaRef, partition_spec: PartitionSpecRef, ) -> Result { - Self::new(iceberg_schema, partition_spec, None) + Self::try_new(iceberg_schema, partition_spec, None) } /// Split the record batch into multiple record batches based on the partition spec. @@ -261,9 +261,11 @@ mod tests { .build() .unwrap(), ); - let partition_splitter = - RecordBatchPartitionSplitter::new_with_computed_values(schema.clone(), partition_spec) - .expect("Failed to create splitter"); + let partition_splitter = RecordBatchPartitionSplitter::try_new_with_computed_values( + schema.clone(), + partition_spec, + ) + .expect("Failed to create splitter"); let arrow_schema = Arc::new(schema_to_arrow_schema(&schema).unwrap()); let id_array = Int32Array::from(vec![1, 2, 1, 3, 2, 3, 1]); @@ -392,7 +394,7 @@ mod tests { ])); // Create splitter expecting pre-computed partition column - let partition_splitter = RecordBatchPartitionSplitter::new_with_precomputed_values( + let partition_splitter = RecordBatchPartitionSplitter::try_new_with_precomputed_values( schema.clone(), partition_spec, ) diff --git a/crates/integrations/datafusion/src/physical_plan/repartition.rs b/crates/integrations/datafusion/src/physical_plan/repartition.rs index 95cdc8472e..8ad87fd1cc 100644 --- a/crates/integrations/datafusion/src/physical_plan/repartition.rs +++ b/crates/integrations/datafusion/src/physical_plan/repartition.rs @@ -86,7 +86,6 @@ use iceberg::spec::{TableMetadata, TableMetadataRef, Transform}; /// NonZeroUsize::new(4).unwrap(), /// )?; /// ``` -#[allow(dead_code)] pub(crate) fn repartition( input: Arc, table_metadata: TableMetadataRef, diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index b9d1f02d1f..9eb53c235f 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -35,7 +35,7 @@ use datafusion::physical_plan::{ execute_input_stream, }; use futures::StreamExt; -use iceberg::arrow::{FieldMatchMode, schema_to_arrow_schema}; +use iceberg::arrow::FieldMatchMode; use iceberg::spec::{DataFileFormat, TableProperties, serialize_data_file_to_json}; use iceberg::table::Table; use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; @@ -44,12 +44,12 @@ use iceberg::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, }; use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; -use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Error, ErrorKind}; use parquet::file::properties::WriterProperties; use uuid::Uuid; use crate::physical_plan::DATA_FILES_COL_NAME; +use crate::task_writer::TaskWriter; use crate::to_datafusion_error; /// An execution plan node that writes data to an Iceberg table. @@ -205,18 +205,6 @@ impl ExecutionPlan for IcebergWriteExec { partition: usize, context: Arc, ) -> DFResult { - if !self - .table - .metadata() - .default_partition_spec() - .is_unpartitioned() - { - // TODO add support for partitioned tables - return Err(DataFusionError::NotImplemented( - "IcebergWriteExec does not support partitioned tables yet".to_string(), - )); - } - let partition_type = self.table.metadata().default_partition_type().clone(); let format_version = self.table.metadata().format_version(); @@ -277,31 +265,41 @@ impl ExecutionPlan for IcebergWriteExec { ); let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); + // Create TaskWriter + // TODO: Make fanout_enabled configurable via table properties + let fanout_enabled = true; + let schema = self.table.metadata().current_schema().clone(); + let partition_spec = self.table.metadata().default_partition_spec().clone(); + let task_writer = TaskWriter::try_new( + data_file_writer_builder, + fanout_enabled, + schema.clone(), + partition_spec, + ) + .map_err(to_datafusion_error)?; + // Get input data let data = execute_input_stream( Arc::clone(&self.input), - Arc::new( - schema_to_arrow_schema(self.table.metadata().current_schema()) - .map_err(to_datafusion_error)?, - ), + self.input.schema(), // input schema may have projected column `_partition` partition, Arc::clone(&context), )?; // Create write stream let stream = futures::stream::once(async move { - let mut writer = data_file_writer_builder - // todo specify partition key when partitioning writer is supported - .build(None) - .await - .map_err(to_datafusion_error)?; + let mut task_writer = task_writer; let mut input_stream = data; while let Some(batch) = input_stream.next().await { - writer.write(batch?).await.map_err(to_datafusion_error)?; + let batch = batch?; + task_writer + .write(batch) + .await + .map_err(to_datafusion_error)?; } - let data_files = writer.close().await.map_err(to_datafusion_error)?; + let data_files = task_writer.close().await.map_err(to_datafusion_error)?; // Convert builders to data files and then to JSON strings let data_files_strs: Vec = data_files diff --git a/crates/integrations/datafusion/src/table/mod.rs b/crates/integrations/datafusion/src/table/mod.rs index a8c49837c3..42a3baad3b 100644 --- a/crates/integrations/datafusion/src/table/mod.rs +++ b/crates/integrations/datafusion/src/table/mod.rs @@ -19,6 +19,7 @@ pub mod metadata_table; pub mod table_provider_factory; use std::any::Any; +use std::num::NonZeroUsize; use std::sync::Arc; use async_trait::async_trait; @@ -38,6 +39,8 @@ use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableIdent}; use metadata_table::IcebergMetadataTableProvider; use crate::physical_plan::commit::IcebergCommitExec; +use crate::physical_plan::project::project_with_partition; +use crate::physical_plan::repartition::repartition; use crate::physical_plan::scan::IcebergTableScan; use crate::physical_plan::write::IcebergWriteExec; @@ -170,32 +173,42 @@ impl TableProvider for IcebergTableProvider { async fn insert_into( &self, - _state: &dyn Session, + state: &dyn Session, input: Arc, _insert_op: InsertOp, ) -> DFResult> { - if !self - .table - .metadata() - .default_partition_spec() - .is_unpartitioned() - { - // TODO add insert into support for partitioned tables - return Err(DataFusionError::NotImplemented( - "IcebergTableProvider::insert_into does not support partitioned tables yet" - .to_string(), - )); - } - let Some(catalog) = self.catalog.clone() else { return Err(DataFusionError::Execution( "Catalog cannot be none for insert_into".to_string(), )); }; + let partition_spec = self.table.metadata().default_partition_spec(); + + // Step 1: Project partition values for partitioned tables + let plan_with_partition = if !partition_spec.is_unpartitioned() { + project_with_partition(input, &self.table)? + } else { + input + }; + + // Step 2: Repartition for parallel processing + let target_partitions = + NonZeroUsize::new(state.config().target_partitions()).ok_or_else(|| { + DataFusionError::Configuration( + "target_partitions must be greater than 0".to_string(), + ) + })?; + + let repartitioned_plan = repartition( + plan_with_partition, + self.table.metadata_ref(), + target_partitions, + )?; + let write_plan = Arc::new(IcebergWriteExec::new( self.table.clone(), - input, + repartitioned_plan, self.schema.clone(), )); diff --git a/crates/integrations/datafusion/src/task_writer.rs b/crates/integrations/datafusion/src/task_writer.rs index d27b2e6fbf..5329f26458 100644 --- a/crates/integrations/datafusion/src/task_writer.rs +++ b/crates/integrations/datafusion/src/task_writer.rs @@ -34,7 +34,7 @@ use iceberg::writer::partitioning::unpartitioned_writer::UnpartitionedWriter; /// /// TaskWriter coordinates writing data to Iceberg tables by: /// - Selecting the appropriate partitioning strategy (unpartitioned, fanout, or clustered) -/// - Lazily initializing the partition splitter on first write +/// - Initializing the partition splitter in the constructor for partitioned tables /// - Routing data to the underlying writer /// - Collecting all written data files /// @@ -63,23 +63,17 @@ use iceberg::writer::partitioning::unpartitioned_writer::UnpartitionedWriter; /// // Close and get data files /// let data_files = task_writer.close().await?; /// ``` -#[allow(dead_code)] pub(crate) struct TaskWriter { /// The underlying writer (UnpartitionedWriter, FanoutWriter, or ClusteredWriter) writer: SupportedWriter, - /// Lazily initialized partition splitter for partitioned tables + /// Partition splitter for partitioned tables (initialized in constructor) partition_splitter: Option, - /// Iceberg schema reference - schema: SchemaRef, - /// Partition specification reference - partition_spec: PartitionSpecRef, } /// Internal enum to hold the different writer types. /// /// This enum allows TaskWriter to work with different partitioning strategies /// while maintaining a unified interface. -#[allow(dead_code)] enum SupportedWriter { /// Writer for unpartitioned tables Unpartitioned(UnpartitionedWriter), @@ -89,7 +83,6 @@ enum SupportedWriter { Clustered(ClusteredWriter), } -#[allow(dead_code)] impl TaskWriter { /// Create a new TaskWriter. /// @@ -125,12 +118,12 @@ impl TaskWriter { /// partition_spec, /// ); /// ``` - pub fn new( + pub fn try_new( writer_builder: B, fanout_enabled: bool, schema: SchemaRef, partition_spec: PartitionSpecRef, - ) -> Self { + ) -> Result { let writer = if partition_spec.is_unpartitioned() { SupportedWriter::Unpartitioned(UnpartitionedWriter::new(writer_builder)) } else if fanout_enabled { @@ -139,17 +132,28 @@ impl TaskWriter { SupportedWriter::Clustered(ClusteredWriter::new(writer_builder)) }; - Self { + // Initialize partition splitter in constructor for partitioned tables + let partition_splitter = if !partition_spec.is_unpartitioned() { + Some( + RecordBatchPartitionSplitter::try_new_with_precomputed_values( + schema.clone(), + partition_spec.clone(), + )?, + ) + } else { + None + }; + + Ok(Self { writer, - partition_splitter: None, - schema, - partition_spec, - } + partition_splitter, + }) } /// Write a RecordBatch to the TaskWriter. /// - /// For the first write to a partitioned table, this method initializes the partition splitter. + /// For partitioned tables, uses the partition splitter to split + /// the batch by partition key and route each partition to the underlying writer. /// For unpartitioned tables, data is written directly without splitting. /// /// # Parameters @@ -163,7 +167,6 @@ impl TaskWriter { /// # Errors /// /// This method will return an error if: - /// - Partition splitter initialization fails /// - Splitting the batch by partition fails /// - Writing to the underlying writer fails /// @@ -183,29 +186,9 @@ impl TaskWriter { writer.write(batch).await } SupportedWriter::Fanout(writer) => { - // Initialize splitter on first write if needed - if self.partition_splitter.is_none() { - self.partition_splitter = - Some(RecordBatchPartitionSplitter::new_with_precomputed_values( - self.schema.clone(), - self.partition_spec.clone(), - )?); - } - - // Split and write partitioned data Self::write_partitioned_batches(writer, &self.partition_splitter, &batch).await } SupportedWriter::Clustered(writer) => { - // Initialize splitter on first write if needed - if self.partition_splitter.is_none() { - self.partition_splitter = - Some(RecordBatchPartitionSplitter::new_with_precomputed_values( - self.schema.clone(), - self.partition_spec.clone(), - )?); - } - - // Split and write partitioned data Self::write_partitioned_batches(writer, &self.partition_splitter, &batch).await } } @@ -214,13 +197,13 @@ impl TaskWriter { /// Helper method to split and write partitioned data. /// /// This method handles the common logic for both FanoutWriter and ClusteredWriter: - /// - Splits the batch by partition key using the provided splitter - /// - Writes each partition to the underlying writer + /// - Splits the batch by partition key using the partition splitter + /// - Writes each partition to the underlying writer with its corresponding partition key /// /// # Parameters /// /// * `writer` - The underlying PartitioningWriter (FanoutWriter or ClusteredWriter) - /// * `partition_splitter` - The partition splitter (must be initialized) + /// * `partition_splitter` - The partition splitter /// * `batch` - The RecordBatch to write /// /// # Returns @@ -393,7 +376,7 @@ mod tests { let partition_spec = Arc::new(PartitionSpec::builder(schema.clone()).build()?); let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; - let mut task_writer = TaskWriter::new(writer_builder, false, schema, partition_spec); + let mut task_writer = TaskWriter::try_new(writer_builder, false, schema, partition_spec)?; // Write data let batch = RecordBatch::try_new(arrow_schema, vec![ @@ -459,7 +442,7 @@ mod tests { ); let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; - let mut task_writer = TaskWriter::new(writer_builder, true, schema, partition_spec); + let mut task_writer = TaskWriter::try_new(writer_builder, true, schema, partition_spec)?; // Create partition column let partition_field = Field::new("region", DataType::Utf8, false).with_metadata( @@ -502,7 +485,7 @@ mod tests { ); let writer_builder = create_writer_builder(&temp_dir, schema.clone())?; - let mut task_writer = TaskWriter::new(writer_builder, false, schema, partition_spec); + let mut task_writer = TaskWriter::try_new(writer_builder, false, schema, partition_spec)?; // Create partition column let partition_field = Field::new("region", DataType::Utf8, false).with_metadata( diff --git a/crates/integrations/datafusion/tests/integration_datafusion_test.rs b/crates/integrations/datafusion/tests/integration_datafusion_test.rs index cb4987a973..fdf5b17d18 100644 --- a/crates/integrations/datafusion/tests/integration_datafusion_test.rs +++ b/crates/integrations/datafusion/tests/integration_datafusion_test.rs @@ -27,9 +27,13 @@ use datafusion::execution::context::SessionContext; use datafusion::parquet::arrow::PARQUET_FIELD_ID_META_KEY; use expect_test::expect; use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; -use iceberg::spec::{NestedField, PrimitiveType, Schema, StructType, Type}; +use iceberg::spec::{ + NestedField, PrimitiveType, Schema, StructType, Transform, Type, UnboundPartitionSpec, +}; use iceberg::test_utils::check_record_batches; -use iceberg::{Catalog, CatalogBuilder, MemoryCatalog, NamespaceIdent, Result, TableCreation}; +use iceberg::{ + Catalog, CatalogBuilder, MemoryCatalog, NamespaceIdent, Result, TableCreation, TableIdent, +}; use iceberg_datafusion::IcebergCatalogProvider; use tempfile::TempDir; @@ -810,3 +814,147 @@ async fn test_insert_into_nested() -> Result<()> { Ok(()) } + +#[tokio::test] +async fn test_insert_into_partitioned() -> Result<()> { + let iceberg_catalog = get_iceberg_catalog().await; + let namespace = NamespaceIdent::new("test_partitioned_write".to_string()); + set_test_namespace(&iceberg_catalog, &namespace).await?; + + // Create a schema with a partition column + let schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "category", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "value", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build()?; + + // Create partition spec with identity transform on category + let partition_spec = UnboundPartitionSpec::builder() + .with_spec_id(0) + .add_partition_field(2, "category", Transform::Identity)? + .build(); + + // Create the partitioned table + let creation = TableCreation::builder() + .name("partitioned_table".to_string()) + .location(temp_path()) + .schema(schema) + .partition_spec(partition_spec) + .properties(HashMap::new()) + .build(); + + iceberg_catalog.create_table(&namespace, creation).await?; + + let client = Arc::new(iceberg_catalog); + let catalog = Arc::new(IcebergCatalogProvider::try_new(client.clone()).await?); + + let ctx = SessionContext::new(); + ctx.register_catalog("catalog", catalog); + + // Insert data with multiple partition values in a single batch + let df = ctx + .sql( + r#" + INSERT INTO catalog.test_partitioned_write.partitioned_table + VALUES + (1, 'electronics', 'laptop'), + (2, 'electronics', 'phone'), + (3, 'books', 'novel'), + (4, 'books', 'textbook'), + (5, 'clothing', 'shirt') + "#, + ) + .await + .unwrap(); + + let batches = df.collect().await.unwrap(); + assert_eq!(batches.len(), 1); + let batch = &batches[0]; + let rows_inserted = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(rows_inserted.value(0), 5); + + // Refresh catalog to get updated table + let catalog = Arc::new(IcebergCatalogProvider::try_new(client.clone()).await?); + ctx.register_catalog("catalog", catalog); + + // Query the table to verify data + let df = ctx + .sql("SELECT * FROM catalog.test_partitioned_write.partitioned_table ORDER BY id") + .await + .unwrap(); + + let batches = df.collect().await.unwrap(); + + // Verify the data - note that _partition column should NOT be present + check_record_batches( + batches, + expect![[r#" + Field { name: "id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, + Field { name: "category", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, + Field { name: "value", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }"#]], + expect![[r#" + id: PrimitiveArray + [ + 1, + 2, + 3, + 4, + 5, + ], + category: StringArray + [ + "electronics", + "electronics", + "books", + "books", + "clothing", + ], + value: StringArray + [ + "laptop", + "phone", + "novel", + "textbook", + "shirt", + ]"#]], + &[], + Some("id"), + ); + + // Verify that data files exist under correct partition paths + let table_ident = TableIdent::new(namespace.clone(), "partitioned_table".to_string()); + let table = client.load_table(&table_ident).await?; + let table_location = table.metadata().location(); + let file_io = table.file_io(); + + // List files under each expected partition path + let electronics_path = format!("{}/data/category=electronics", table_location); + let books_path = format!("{}/data/category=books", table_location); + let clothing_path = format!("{}/data/category=clothing", table_location); + + // Verify partition directories exist and contain data files + assert!( + file_io.exists(&electronics_path).await?, + "Expected partition directory: {}", + electronics_path + ); + assert!( + file_io.exists(&books_path).await?, + "Expected partition directory: {}", + books_path + ); + assert!( + file_io.exists(&clothing_path).await?, + "Expected partition directory: {}", + clothing_path + ); + + Ok(()) +} From d1927dbac595b42b0d45541b06949f949d4d8147 Mon Sep 17 00:00:00 2001 From: Bo Lu Date: Thu, 13 Nov 2025 13:21:42 +1100 Subject: [PATCH 072/891] docs: Add Wrappers project to README (#1852) ## Which issue does this PR close? ## What changes are included in this PR? This PR is to add [Wrappers](https://github.com/supabase/wrappers) to the Readme doc. ## Are these changes tested? --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 88b8714cc5..27cce7664e 100644 --- a/README.md +++ b/README.md @@ -97,6 +97,7 @@ The Apache Iceberg community is built on the principles described in the [Apache - [Lakekeeper](https://github.com/lakekeeper/lakekeeper/): An Apache-licensed Iceberg REST Catalog with data access controls. - [Moonlink](https://github.com/Mooncake-Labs/moonlink): A Rust library that enables sub-second mirroring (CDC) of Postgres tables into Iceberg. - [RisingWave](https://github.com/risingwavelabs/risingwave): A Postgres-compatible SQL database designed for real-time event streaming data processing, analysis, and management. +- [Wrappers](https://github.com/supabase/wrappers): Postgres Foreign Data Wrapper development framework in Rust. ## License From 8d4851f0db824050c0419bd20d939e5dd3a5a80b Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Thu, 13 Nov 2025 04:35:12 -0500 Subject: [PATCH 073/891] feat(reader): Add PartitionSpec support to FileScanTask and RecordBatchTransformer (#1821) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? Partially address #1749. ## What changes are included in this PR? This PR adds partition spec handling to `FileScanTask` and `RecordBatchTransformer` to correctly implement the Iceberg spec's "Column Projection" rules for fields "not present" in data files. ### Problem Statement Prior to this PR, `iceberg-rust`'s `FileScanTask` had no mechanism to pass partition information to `RecordBatchTransformer`, causing two issues: 1. **Incorrect handling of bucket partitioning**: Couldn't distinguish identity transforms (which should use partition metadata constants) from non-identity transforms like bucket/truncate/year/month (which must read from data file). For example, `bucket(4, id)` stores `id_bucket = 2` (bucket number) in partition metadata, but actual `id` values (100, 200, 300) are only in the data file. iceberg-rust was incorrectly treating bucket-partitioned source columns as constants, breaking runtime filtering and returning incorrect query results. 2. **Field ID conflicts in add_files scenarios**: When importing Hive tables via `add_files`, partition columns could have field IDs conflicting with Parquet data columns. Example: Parquet has field_id=1→"name", but Iceberg expects field_id=1→"id" (partition). Per spec, the correct field is "not present" and requires name mapping fallback. ### Iceberg Specification Requirements Per the Iceberg spec (https://iceberg.apache.org/spec/#column-projection), when a field ID is "not present" in a data file, it must be resolved using these rules: 1. Return the value from partition metadata if an **Identity Transform** exists 2. Use `schema.name-mapping.default` metadata to map field id to columns without field id 3. Return the default value if it has a defined `initial-default` 4. Return null in all other cases **Why this matters:** - **Identity transforms** (e.g., `identity(dept)`) store actual column values in partition metadata that can be used as constants without reading the data file - **Non-identity transforms** (e.g., `bucket(4, id)`, `day(timestamp)`) store transformed values in partition metadata (e.g., bucket number 2, not the actual `id` values 100, 200, 300) and must read source columns from the data file ### Changes Made 1. **Added partition fields to `FileScanTask`** (`scan/task.rs`): - `partition: Option` - Partition data from manifest entry - `partition_spec: Option>` - For transform-aware constant detection - `name_mapping: Option>` - Name mapping from table metadata 2. **Implemented `constants_map()` function** (`arrow/record_batch_transformer.rs`): - Replicates Java's `PartitionUtil.constantsMap()` behavior - Only includes fields where transform is `Transform::Identity` - Used to determine which fields use partition metadata constants vs. reading from data files 3. **Enhanced `RecordBatchTransformer`** (`arrow/record_batch_transformer.rs`): - Added `build_with_partition_data()` method to accept partition spec, partition data, and name mapping - Implements all 4 spec rules for column resolution with identity-transform awareness - Detects field ID conflicts by verifying both field ID AND name match - Falls back to name mapping when field IDs are missing/conflicting (spec rule #2) 4. **Updated `ArrowReader`** (`arrow/reader.rs`): - Uses `build_with_partition_data()` when partition information is available - Falls back to `build()` when not available 5. **Updated manifest entry processing** (`scan/context.rs`): - Populates partition fields in `FileScanTask` from manifest entry data ### Tests Added 1. **`bucket_partitioning_reads_source_column_from_file`** - Verifies that bucket-partitioned source columns are read from data files (not treated as constants from partition metadata) 2. **`identity_partition_uses_constant_from_metadata`** - Verifies that identity-transformed fields correctly use partition metadata constants 3. **`test_bucket_partitioning_with_renamed_source_column`** - Verifies field-ID-based mapping works despite column rename 4. **`add_files_partition_columns_without_field_ids`** - Verifies name mapping resolution for Hive table imports without field IDs (spec rule #2) 5. **`add_files_with_true_field_id_conflict`** - Verifies correct field ID conflict detection with name mapping fallback (spec rule #2) 6. **`test_all_four_spec_rules`** - Integration test verifying all 4 spec rules work together ## Are these changes tested? Yes, there are 6 new unit tests covering all 4 Iceberg spec rules. This also resolved approximately 50 Iceberg Java tests when running with DataFusion Comet's experimental https://github.com/apache/datafusion-comet/pull/2528 PR. --------- Co-authored-by: Renjie Liu --- .../src/arrow/caching_delete_file_loader.rs | 3 + .../iceberg/src/arrow/delete_file_loader.rs | 4 +- crates/iceberg/src/arrow/delete_filter.rs | 6 + crates/iceberg/src/arrow/reader.rs | 339 ++++++- .../src/arrow/record_batch_transformer.rs | 894 ++++++++++++++++-- crates/iceberg/src/scan/context.rs | 7 + crates/iceberg/src/scan/mod.rs | 6 + crates/iceberg/src/scan/task.rs | 54 +- 8 files changed, 1222 insertions(+), 91 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 8a3ab3a955..f1c4f86f23 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -881,6 +881,9 @@ mod tests { project_field_ids: vec![2, 3], predicate: None, deletes: vec![pos_del, eq_del], + partition: None, + partition_spec: None, + name_mapping: None, }; // Load the deletes - should handle both types without error diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index c0b1392dc6..e12daf5324 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -20,7 +20,7 @@ use std::sync::Arc; use futures::{StreamExt, TryStreamExt}; use crate::arrow::ArrowReader; -use crate::arrow::record_batch_transformer::RecordBatchTransformer; +use crate::arrow::record_batch_transformer::RecordBatchTransformerBuilder; use crate::io::FileIO; use crate::scan::{ArrowRecordBatchStream, FileScanTaskDeleteFile}; use crate::spec::{Schema, SchemaRef}; @@ -82,7 +82,7 @@ impl BasicDeleteFileLoader { equality_ids: &[i32], ) -> Result { let mut record_batch_transformer = - RecordBatchTransformer::build(target_schema.clone(), equality_ids); + RecordBatchTransformerBuilder::new(target_schema.clone(), equality_ids).build(); let record_batch_stream = record_batch_stream.map(move |record_batch| { record_batch.and_then(|record_batch| { diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index 4250974bcd..14b5124ee6 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -341,6 +341,9 @@ pub(crate) mod tests { project_field_ids: vec![], predicate: None, deletes: vec![pos_del_1, pos_del_2.clone()], + partition: None, + partition_spec: None, + name_mapping: None, }, FileScanTask { start: 0, @@ -352,6 +355,9 @@ pub(crate) mod tests { project_field_ids: vec![], predicate: None, deletes: vec![pos_del_3], + partition: None, + partition_spec: None, + name_mapping: None, }, ]; diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index fed8f19c05..ab5a96f751 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -45,7 +45,7 @@ use parquet::file::metadata::{ use parquet::schema::types::{SchemaDescriptor, Type as ParquetType}; use crate::arrow::caching_delete_file_loader::CachingDeleteFileLoader; -use crate::arrow::record_batch_transformer::RecordBatchTransformer; +use crate::arrow::record_batch_transformer::RecordBatchTransformerBuilder; use crate::arrow::{arrow_schema_to_schema, get_arrow_datum}; use crate::delete_vector::DeleteVector; use crate::error::Result; @@ -55,7 +55,7 @@ use crate::expr::visitors::row_group_metrics_evaluator::RowGroupMetricsEvaluator use crate::expr::{BoundPredicate, BoundReference}; use crate::io::{FileIO, FileMetadata, FileRead}; use crate::scan::{ArrowRecordBatchStream, FileScanTask, FileScanTaskStream}; -use crate::spec::{Datum, NestedField, PrimitiveType, Schema, Type}; +use crate::spec::{Datum, NameMapping, NestedField, PrimitiveType, Schema, Type}; use crate::utils::available_parallelism; use crate::{Error, ErrorKind}; @@ -181,7 +181,8 @@ impl ArrowReader { let should_load_page_index = (row_selection_enabled && task.predicate.is_some()) || !task.deletes.is_empty(); - let delete_filter_rx = delete_file_loader.load_deletes(&task.deletes, task.schema.clone()); + let delete_filter_rx = + delete_file_loader.load_deletes(&task.deletes, Arc::clone(&task.schema)); // Migrated tables lack field IDs, requiring us to inspect the schema to choose // between field-ID-based or position-based projection @@ -193,7 +194,9 @@ impl ArrowReader { ) .await?; - // Parquet files from Hive/Spark migrations lack field IDs in their metadata + // Check if Parquet file has embedded field IDs + // Corresponds to Java's ParquetSchemaUtil.hasIds() + // Reference: parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java:118 let missing_field_ids = initial_stream_builder .schema() .fields() @@ -201,11 +204,38 @@ impl ArrowReader { .next() .is_some_and(|f| f.metadata().get(PARQUET_FIELD_ID_META_KEY).is_none()); - // Adding position-based fallback IDs at schema level (not per-batch) enables projection - // on files that lack embedded field IDs. We recreate the builder to apply the modified schema. + // Three-branch schema resolution strategy matching Java's ReadConf constructor + // + // Per Iceberg spec Column Projection rules: + // "Columns in Iceberg data files are selected by field id. The table schema's column + // names and order may change after a data file is written, and projection must be done + // using field ids." + // https://iceberg.apache.org/spec/#column-projection + // + // When Parquet files lack field IDs (e.g., Hive/Spark migrations via add_files), + // we must assign field IDs BEFORE reading data to enable correct projection. + // + // Java's ReadConf determines field ID strategy: + // - Branch 1: hasIds(fileSchema) → trust embedded field IDs, use pruneColumns() + // - Branch 2: nameMapping present → applyNameMapping(), then pruneColumns() + // - Branch 3: fallback → addFallbackIds(), then pruneColumnsFallback() let mut record_batch_stream_builder = if missing_field_ids { - let arrow_schema = - add_fallback_field_ids_to_arrow_schema(initial_stream_builder.schema()); + // Parquet file lacks field IDs - must assign them before reading + let arrow_schema = if let Some(name_mapping) = &task.name_mapping { + // Branch 2: Apply name mapping to assign correct Iceberg field IDs + // Per spec rule #2: "Use schema.name-mapping.default metadata to map field id + // to columns without field id" + // Corresponds to Java's ParquetSchemaUtil.applyNameMapping() + apply_name_mapping_to_arrow_schema( + Arc::clone(initial_stream_builder.schema()), + name_mapping, + )? + } else { + // Branch 3: No name mapping - use position-based fallback IDs + // Corresponds to Java's ParquetSchemaUtil.addFallbackIds() + add_fallback_field_ids_to_arrow_schema(initial_stream_builder.schema()) + }; + let options = ArrowReaderOptions::new().with_schema(arrow_schema); Self::create_parquet_record_batch_stream_builder( @@ -216,11 +246,14 @@ impl ArrowReader { ) .await? } else { + // Branch 1: File has embedded field IDs - trust them initial_stream_builder }; - // Fallback IDs don't match Parquet's embedded field IDs (since they don't exist), - // so we must use position-based projection instead of field-ID matching + // Create projection mask based on field IDs + // - If file has embedded IDs: field-ID-based projection (missing_field_ids=false) + // - If name mapping applied: field-ID-based projection (missing_field_ids=true but IDs now match) + // - If fallback IDs: position-based projection (missing_field_ids=true) let projection_mask = Self::get_arrow_projection_mask( &task.project_field_ids, &task.schema, @@ -234,9 +267,18 @@ impl ArrowReader { // RecordBatchTransformer performs any transformations required on the RecordBatches // that come back from the file, such as type promotion, default column insertion - // and column re-ordering - let mut record_batch_transformer = - RecordBatchTransformer::build(task.schema_ref(), task.project_field_ids()); + // and column re-ordering. + let mut record_batch_transformer_builder = + RecordBatchTransformerBuilder::new(task.schema_ref(), task.project_field_ids()); + + if let (Some(partition_spec), Some(partition_data)) = + (task.partition_spec.clone(), task.partition.clone()) + { + record_batch_transformer_builder = + record_batch_transformer_builder.with_partition(partition_spec, partition_data); + } + + let mut record_batch_transformer = record_batch_transformer_builder.build(); if let Some(batch_size) = batch_size { record_batch_stream_builder = record_batch_stream_builder.with_batch_size(batch_size); @@ -919,6 +961,77 @@ fn build_fallback_field_id_map(parquet_schema: &SchemaDescriptor) -> HashMap Result> { + debug_assert!( + arrow_schema + .fields() + .iter() + .next() + .is_none_or(|f| f.metadata().get(PARQUET_FIELD_ID_META_KEY).is_none()), + "Schema already has field IDs - name mapping should not be applied" + ); + + use arrow_schema::Field; + + let fields_with_mapped_ids: Vec<_> = arrow_schema + .fields() + .iter() + .map(|field| { + // Look up this column name in name mapping to get the Iceberg field ID. + // Corresponds to Java's ApplyNameMapping visitor which calls + // nameMapping.find(currentPath()) and returns field.withId() if found. + // + // If the field isn't in the mapping, leave it WITHOUT assigning an ID + // (matching Java's behavior of returning the field unchanged). + // Later, during projection, fields without IDs are filtered out. + let mapped_field_opt = name_mapping + .fields() + .iter() + .find(|f| f.names().contains(&field.name().to_string())); + + let mut metadata = field.metadata().clone(); + + if let Some(mapped_field) = mapped_field_opt { + if let Some(field_id) = mapped_field.field_id() { + // Field found in mapping with a field_id → assign it + metadata.insert(PARQUET_FIELD_ID_META_KEY.to_string(), field_id.to_string()); + } + // If field_id is None, leave the field without an ID (will be filtered by projection) + } + // If field not found in mapping, leave it without an ID (will be filtered by projection) + + Field::new(field.name(), field.data_type().clone(), field.is_nullable()) + .with_metadata(metadata) + }) + .collect(); + + Ok(Arc::new(ArrowSchema::new_with_metadata( + fields_with_mapped_ids, + arrow_schema.metadata().clone(), + ))) +} + /// Add position-based fallback field IDs to Arrow schema for Parquet files lacking them. /// Enables projection on migrated files (e.g., from Hive/Spark). /// @@ -1948,6 +2061,9 @@ message schema { project_field_ids: vec![1], predicate: Some(predicate.bind(schema, true).unwrap()), deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -2266,6 +2382,9 @@ message schema { project_field_ids: vec![1], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, }; // Task 2: read the second and third row groups @@ -2279,6 +2398,9 @@ message schema { project_field_ids: vec![1], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, }; let tasks1 = Box::pin(futures::stream::iter(vec![Ok(task1)])) as FileScanTaskStream; @@ -2403,6 +2525,9 @@ message schema { project_field_ids: vec![1, 2], // Request both columns 'a' and 'b' predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -2571,6 +2696,9 @@ message schema { partition_spec_id: 0, equality_ids: None, }], + partition: None, + partition_spec: None, + name_mapping: None, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -2786,6 +2914,9 @@ message schema { partition_spec_id: 0, equality_ids: None, }], + partition: None, + partition_spec: None, + name_mapping: None, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -2994,6 +3125,9 @@ message schema { partition_spec_id: 0, equality_ids: None, }], + partition: None, + partition_spec: None, + name_mapping: None, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -3094,6 +3228,9 @@ message schema { project_field_ids: vec![1, 2], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3188,6 +3325,9 @@ message schema { project_field_ids: vec![1, 3], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3271,6 +3411,9 @@ message schema { project_field_ids: vec![1, 2, 3], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3368,6 +3511,9 @@ message schema { project_field_ids: vec![1, 2], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3494,6 +3640,9 @@ message schema { project_field_ids: vec![1, 2], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3587,6 +3736,9 @@ message schema { project_field_ids: vec![1, 5, 2], predicate: None, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3693,6 +3845,9 @@ message schema { project_field_ids: vec![1, 2, 3], predicate: Some(predicate.bind(schema, true).unwrap()), deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, })] .into_iter(), )) as FileScanTaskStream; @@ -3708,4 +3863,162 @@ message schema { // Should return empty results assert!(result.is_empty() || result.iter().all(|batch| batch.num_rows() == 0)); } + + /// Test bucket partitioning reads source column from data file (not partition metadata). + /// + /// This is an integration test verifying the complete ArrowReader pipeline with bucket partitioning. + /// It corresponds to TestRuntimeFiltering tests in Iceberg Java (e.g., testRenamedSourceColumnTable). + /// + /// # Iceberg Spec Requirements + /// + /// Per the Iceberg spec "Column Projection" section: + /// > "Return the value from partition metadata if an **Identity Transform** exists for the field" + /// + /// This means: + /// - Identity transforms (e.g., `identity(dept)`) use constants from partition metadata + /// - Non-identity transforms (e.g., `bucket(4, id)`) must read source columns from data files + /// - Partition metadata for bucket transforms stores bucket numbers (0-3), NOT source values + /// + /// Java's PartitionUtil.constantsMap() implements this via: + /// ```java + /// if (field.transform().isIdentity()) { + /// idToConstant.put(field.sourceId(), converted); + /// } + /// ``` + /// + /// # What This Test Verifies + /// + /// This test ensures the full ArrowReader → RecordBatchTransformer pipeline correctly handles + /// bucket partitioning when FileScanTask provides partition_spec and partition_data: + /// + /// - Parquet file has field_id=1 named "id" with actual data [1, 5, 9, 13] + /// - FileScanTask specifies partition_spec with bucket(4, id) and partition_data with bucket=1 + /// - RecordBatchTransformer.constants_map() excludes bucket-partitioned field from constants + /// - ArrowReader correctly reads [1, 5, 9, 13] from the data file + /// - Values are NOT replaced with constant 1 from partition metadata + /// + /// # Why This Matters + /// + /// Without correct handling: + /// - Runtime filtering would break (e.g., `WHERE id = 5` would fail) + /// - Query results would be incorrect (all rows would have id=1) + /// - Bucket partitioning would be unusable for query optimization + /// + /// # References + /// - Iceberg spec: format/spec.md "Column Projection" + "Partition Transforms" + /// - Java test: spark/src/test/java/.../TestRuntimeFiltering.java + /// - Java impl: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java + #[tokio::test] + async fn test_bucket_partitioning_reads_source_column_from_file() { + use arrow_array::Int32Array; + + use crate::spec::{Literal, PartitionSpec, Struct, Transform}; + + // Iceberg schema with id and name columns + let schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + // Partition spec: bucket(4, id) + let partition_spec = Arc::new( + PartitionSpec::builder(schema.clone()) + .with_spec_id(0) + .add_partition_field("id", "id_bucket", Transform::Bucket(4)) + .unwrap() + .build() + .unwrap(), + ); + + // Partition data: bucket value is 1 + let partition_data = Struct::from_iter(vec![Some(Literal::int(1))]); + + // Create Arrow schema with field IDs for Parquet file + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + Field::new("name", DataType::Utf8, true).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + ])); + + // Write Parquet file with data + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + let id_data = Arc::new(Int32Array::from(vec![1, 5, 9, 13])) as ArrayRef; + let name_data = + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie", "Dave"])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema.clone(), vec![id_data, name_data]).unwrap(); + + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + let file = File::create(format!("{}/data.parquet", &table_location)).unwrap(); + let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + + // Read the Parquet file with partition spec and data + let reader = ArrowReaderBuilder::new(file_io).build(); + let tasks = Box::pin(futures::stream::iter( + vec![Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{}/data.parquet", table_location), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + partition: Some(partition_data), + partition_spec: Some(partition_spec), + name_mapping: None, + })] + .into_iter(), + )) as FileScanTaskStream; + + let result = reader + .read(tasks) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Verify we got the correct data + assert_eq!(result.len(), 1); + let batch = &result[0]; + + assert_eq!(batch.num_columns(), 2); + assert_eq!(batch.num_rows(), 4); + + // The id column MUST contain actual values from the Parquet file [1, 5, 9, 13], + // NOT the constant partition value 1 + let id_col = batch + .column(0) + .as_primitive::(); + assert_eq!(id_col.value(0), 1); + assert_eq!(id_col.value(1), 5); + assert_eq!(id_col.value(2), 9); + assert_eq!(id_col.value(3), 13); + + let name_col = batch.column(1).as_string::(); + assert_eq!(name_col.value(0), "Alice"); + assert_eq!(name_col.value(1), "Bob"); + assert_eq!(name_col.value(2), "Charlie"); + assert_eq!(name_col.value(3), "Dave"); + } } diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index 5fbbbb106a..e7d8b8f0fb 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -29,9 +29,44 @@ use arrow_schema::{ use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use crate::arrow::schema_to_arrow_schema; -use crate::spec::{Literal, PrimitiveLiteral, Schema as IcebergSchema}; +use crate::spec::{ + Literal, PartitionSpec, PrimitiveLiteral, Schema as IcebergSchema, Struct, Transform, +}; use crate::{Error, ErrorKind, Result}; +/// Build a map of field ID to constant value for identity-partitioned fields. +/// +/// Implements Iceberg spec "Column Projection" rule #1: use partition metadata constants +/// only for identity-transformed fields. Non-identity transforms (bucket, truncate, year, etc.) +/// store derived values in partition metadata, so source columns must be read from data files. +/// +/// Example: For `bucket(4, id)`, partition metadata has `id_bucket = 2` (bucket number), +/// but the actual `id` values (100, 200, 300) are only in the data file. +/// +/// Matches Java's `PartitionUtil.constantsMap()` which filters `if (field.transform().isIdentity())`. +/// +/// # References +/// - Spec: https://iceberg.apache.org/spec/#column-projection +/// - Java: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java:constantsMap() +fn constants_map( + partition_spec: &PartitionSpec, + partition_data: &Struct, +) -> HashMap { + let mut constants = HashMap::new(); + + for (pos, field) in partition_spec.fields().iter().enumerate() { + // Only identity transforms should use constant values from partition metadata + if matches!(field.transform, Transform::Identity) { + // Get the partition value for this field + if let Some(Literal::Primitive(value)) = &partition_data[pos] { + constants.insert(field.source_id, value.clone()); + } + } + } + + constants +} + /// Indicates how a particular column in a processed RecordBatch should /// be sourced. #[derive(Debug)] @@ -107,32 +142,107 @@ enum SchemaComparison { Different, } +/// Builder for RecordBatchTransformer to improve ergonomics when constructing with optional parameters. +/// +/// See [`RecordBatchTransformer`] for details on partition spec and partition data. #[derive(Debug)] -pub(crate) struct RecordBatchTransformer { +pub(crate) struct RecordBatchTransformerBuilder { snapshot_schema: Arc, projected_iceberg_field_ids: Vec, - - // BatchTransform gets lazily constructed based on the schema of - // the first RecordBatch we receive from the file - batch_transform: Option, + partition_spec: Option>, + partition_data: Option, } -impl RecordBatchTransformer { - /// Build a RecordBatchTransformer for a given - /// Iceberg snapshot schema and list of projected field ids. - pub(crate) fn build( +impl RecordBatchTransformerBuilder { + pub(crate) fn new( snapshot_schema: Arc, projected_iceberg_field_ids: &[i32], ) -> Self { - let projected_iceberg_field_ids = projected_iceberg_field_ids.to_vec(); - Self { snapshot_schema, - projected_iceberg_field_ids, + projected_iceberg_field_ids: projected_iceberg_field_ids.to_vec(), + partition_spec: None, + partition_data: None, + } + } + + /// Set partition spec and data together for identifying identity-transformed partition columns. + /// + /// Both partition_spec and partition_data must be provided together since the spec defines + /// which fields are identity-partitioned, and the data provides their constant values. + /// One without the other cannot produce a valid constants map. + pub(crate) fn with_partition( + mut self, + partition_spec: Arc, + partition_data: Struct, + ) -> Self { + self.partition_spec = Some(partition_spec); + self.partition_data = Some(partition_data); + self + } + + pub(crate) fn build(self) -> RecordBatchTransformer { + RecordBatchTransformer { + snapshot_schema: self.snapshot_schema, + projected_iceberg_field_ids: self.projected_iceberg_field_ids, + partition_spec: self.partition_spec, + partition_data: self.partition_data, batch_transform: None, } } +} + +/// Transforms RecordBatches from Parquet files to match the Iceberg table schema. +/// +/// Handles schema evolution, column reordering, type promotion, and implements the Iceberg spec's +/// "Column Projection" rules for resolving field IDs "not present" in data files: +/// 1. Return the value from partition metadata if an Identity Transform exists +/// 2. Use schema.name-mapping.default metadata to map field id to columns without field id (applied in ArrowReader) +/// 3. Return the default value if it has a defined initial-default +/// 4. Return null in all other cases +/// +/// # Field ID Resolution +/// +/// Field ID resolution happens in ArrowReader before data is read (matching Java's ReadConf): +/// - If file has embedded field IDs: trust them (ParquetSchemaUtil.hasIds() = true) +/// - If file lacks IDs and name_mapping exists: apply name mapping (ParquetSchemaUtil.applyNameMapping()) +/// - If file lacks IDs and no name_mapping: use position-based fallback (ParquetSchemaUtil.addFallbackIds()) +/// +/// By the time RecordBatchTransformer processes data, all field IDs are trustworthy. +/// This transformer only handles remaining projection rules (#1, #3, #4) for fields still "not present". +/// +/// # Partition Spec and Data +/// +/// **Bucket partitioning**: Distinguish identity transforms (use partition metadata constants) +/// from non-identity transforms like bucket (read from data file) to enable runtime filtering on +/// bucket-partitioned columns. For example, `bucket(4, id)` stores only the bucket number in +/// partition metadata, so actual `id` values must be read from the data file. +/// +/// # References +/// - Spec: https://iceberg.apache.org/spec/#column-projection +/// - Java: parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java (field ID resolution) +/// - Java: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java (partition constants) +#[derive(Debug)] +pub(crate) struct RecordBatchTransformer { + snapshot_schema: Arc, + projected_iceberg_field_ids: Vec, + + /// Partition spec for identifying identity-transformed partition columns (spec rule #1). + /// Only fields with identity transforms use partition data constants; non-identity transforms + /// (bucket, truncate, etc.) must read source columns from data files. + partition_spec: Option>, + /// Partition data providing constant values for identity-transformed partition columns (spec rule #1). + /// For example, in a file at path `dept=engineering/file.parquet`, this would contain + /// the value "engineering" for the dept field. + partition_data: Option, + + // BatchTransform gets lazily constructed based on the schema of + // the first RecordBatch we receive from the file + batch_transform: Option, +} + +impl RecordBatchTransformer { pub(crate) fn process_record_batch( &mut self, record_batch: RecordBatch, @@ -147,7 +257,7 @@ impl RecordBatchTransformer { .with_match_field_names(false) .with_row_count(Some(record_batch.num_rows())); RecordBatch::try_new_with_options( - target_schema.clone(), + Arc::clone(target_schema), self.transform_columns(record_batch.columns(), operations)?, &options, )? @@ -157,7 +267,7 @@ impl RecordBatchTransformer { .with_match_field_names(false) .with_row_count(Some(record_batch.num_rows())); RecordBatch::try_new_with_options( - target_schema.clone(), + Arc::clone(target_schema), record_batch.columns().to_vec(), &options, )? @@ -167,6 +277,8 @@ impl RecordBatchTransformer { record_batch.schema_ref(), self.snapshot_schema.as_ref(), &self.projected_iceberg_field_ids, + self.partition_spec.as_ref().map(|s| s.as_ref()), + self.partition_data.as_ref(), )?); self.process_record_batch(record_batch)? @@ -185,6 +297,8 @@ impl RecordBatchTransformer { source_schema: &ArrowSchemaRef, snapshot_schema: &IcebergSchema, projected_iceberg_field_ids: &[i32], + partition_spec: Option<&PartitionSpec>, + partition_data: Option<&Struct>, ) -> Result { let mapped_unprojected_arrow_schema = Arc::new(schema_to_arrow_schema(snapshot_schema)?); let field_id_to_mapped_schema_map = @@ -205,6 +319,12 @@ impl RecordBatchTransformer { let target_schema = Arc::new(ArrowSchema::new(fields?)); + let constants_map = if let (Some(spec), Some(data)) = (partition_spec, partition_data) { + constants_map(spec, data) + } else { + HashMap::new() + }; + match Self::compare_schemas(source_schema, &target_schema) { SchemaComparison::Equivalent => Ok(BatchTransform::PassThrough), SchemaComparison::NameChangesOnly => Ok(BatchTransform::ModifySchema { target_schema }), @@ -214,6 +334,8 @@ impl RecordBatchTransformer { snapshot_schema, projected_iceberg_field_ids, field_id_to_mapped_schema_map, + constants_map, + partition_spec, )?, target_schema, }), @@ -270,57 +392,92 @@ impl RecordBatchTransformer { snapshot_schema: &IcebergSchema, projected_iceberg_field_ids: &[i32], field_id_to_mapped_schema_map: HashMap, + constants_map: HashMap, + _partition_spec: Option<&PartitionSpec>, ) -> Result> { let field_id_to_source_schema_map = Self::build_field_id_to_arrow_schema_map(source_schema)?; - projected_iceberg_field_ids.iter().map(|field_id|{ - let (target_field, _) = field_id_to_mapped_schema_map.get(field_id).ok_or( - Error::new(ErrorKind::Unexpected, "could not find field in schema") - )?; - let target_type = target_field.data_type(); - - Ok(if let Some((source_field, source_index)) = field_id_to_source_schema_map.get(field_id) { - // column present in source + projected_iceberg_field_ids + .iter() + .map(|field_id| { + let (target_field, _) = + field_id_to_mapped_schema_map + .get(field_id) + .ok_or(Error::new( + ErrorKind::Unexpected, + "could not find field in schema", + ))?; + let target_type = target_field.data_type(); - if source_field.data_type().equals_datatype(target_type) { - // no promotion required - ColumnSource::PassThrough { - source_index: *source_index + let iceberg_field = snapshot_schema.field_by_id(*field_id).ok_or(Error::new( + ErrorKind::Unexpected, + "Field not found in snapshot schema", + ))?; + + // Iceberg spec's "Column Projection" rules (https://iceberg.apache.org/spec/#column-projection). + // For fields "not present" in data files: + // 1. Use partition metadata (identity transforms only) + // 2. Use name mapping + // 3. Use initial_default + // 4. Return null + // + // Why check partition constants before Parquet field IDs (Java: BaseParquetReaders.java:299): + // In add_files scenarios, partition columns may exist in BOTH Parquet AND partition metadata. + // Partition metadata is authoritative - it defines which partition this file belongs to. + + // Field ID resolution now happens in ArrowReader via: + // 1. Embedded field IDs (ParquetSchemaUtil.hasIds() = true) - trust them + // 2. Name mapping (ParquetSchemaUtil.applyNameMapping()) - applied upfront + // 3. Position-based fallback (ParquetSchemaUtil.addFallbackIds()) - applied upfront + // + // At this point, all field IDs in the source schema are trustworthy. + // No conflict detection needed - schema resolution happened in reader.rs. + let field_by_id = field_id_to_source_schema_map.get(field_id).map( + |(source_field, source_index)| { + if source_field.data_type().equals_datatype(target_type) { + ColumnSource::PassThrough { + source_index: *source_index, + } + } else { + ColumnSource::Promote { + target_type: target_type.clone(), + source_index: *source_index, + } + } + }, + ); + + // Apply spec's fallback steps for "not present" fields. + let column_source = if let Some(constant_value) = constants_map.get(field_id) { + // Rule #1: Identity partition constant + ColumnSource::Add { + value: Some(constant_value.clone()), + target_type: target_type.clone(), } + } else if let Some(source) = field_by_id { + source } else { - // promotion required - ColumnSource::Promote { + // Rules #2, #3 and #4: + // Rule #2 (name mapping) was already applied in reader.rs if needed. + // If field_id is still not found, the column doesn't exist in the Parquet file. + // Fall through to rule #3 (initial_default) or rule #4 (null). + let default_value = iceberg_field.initial_default.as_ref().and_then(|lit| { + if let Literal::Primitive(prim) = lit { + Some(prim.clone()) + } else { + None + } + }); + ColumnSource::Add { + value: default_value, target_type: target_type.clone(), - source_index: *source_index, } - } - } else { - // column must be added - let iceberg_field = snapshot_schema.field_by_id(*field_id).ok_or( - Error::new(ErrorKind::Unexpected, "Field not found in snapshot schema") - )?; - - let default_value = if let Some(iceberg_default_value) = - &iceberg_field.initial_default - { - let Literal::Primitive(primitive_literal) = iceberg_default_value else { - return Err(Error::new( - ErrorKind::Unexpected, - format!("Default value for column must be primitive type, but encountered {iceberg_default_value:?}") - )); - }; - Some(primitive_literal.clone()) - } else { - None }; - ColumnSource::Add { - value: default_value, - target_type: target_type.clone(), - } + Ok(column_source) }) - }).collect() + .collect() } fn build_field_id_to_arrow_schema_map( @@ -328,25 +485,19 @@ impl RecordBatchTransformer { ) -> Result> { let mut field_id_to_source_schema = HashMap::new(); for (source_field_idx, source_field) in source_schema.fields.iter().enumerate() { - let this_field_id = source_field - .metadata() - .get(PARQUET_FIELD_ID_META_KEY) - .ok_or_else(|| { + // Check if field has a field ID in metadata + if let Some(field_id_str) = source_field.metadata().get(PARQUET_FIELD_ID_META_KEY) { + let this_field_id = field_id_str.parse().map_err(|e| { Error::new( ErrorKind::DataInvalid, - "field ID not present in parquet metadata", - ) - })? - .parse() - .map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!("field id not parseable as an i32: {e}"), + format!("field id not parseable as an i32: {}", e), ) })?; - field_id_to_source_schema - .insert(this_field_id, (source_field.clone(), source_field_idx)); + field_id_to_source_schema + .insert(this_field_id, (source_field.clone(), source_field_idx)); + } + // If field doesn't have a field ID, skip it - name mapping will handle it } Ok(field_id_to_source_schema) @@ -447,7 +598,7 @@ impl RecordBatchTransformer { (dt, _) => { return Err(Error::new( ErrorKind::Unexpected, - format!("unexpected target column type {dt}"), + format!("unexpected target column type {}", dt), )); } }) @@ -466,8 +617,10 @@ mod test { use arrow_schema::{DataType, Field, Schema as ArrowSchema}; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; - use crate::arrow::record_batch_transformer::RecordBatchTransformer; - use crate::spec::{Literal, NestedField, PrimitiveType, Schema, Type}; + use crate::arrow::record_batch_transformer::{ + RecordBatchTransformer, RecordBatchTransformerBuilder, + }; + use crate::spec::{Literal, NestedField, PrimitiveType, Schema, Struct, Type}; #[test] fn build_field_id_to_source_schema_map_works() { @@ -492,7 +645,9 @@ mod test { let snapshot_schema = Arc::new(iceberg_table_schema()); let projected_iceberg_field_ids = [13, 14]; - let mut inst = RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + let mut inst = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_iceberg_field_ids) + .build(); let result = inst .process_record_batch(source_record_batch_no_migration_required()) @@ -508,7 +663,9 @@ mod test { let snapshot_schema = Arc::new(iceberg_table_schema()); let projected_iceberg_field_ids = [10, 11, 12, 14, 15]; // a, b, c, e, f - let mut inst = RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + let mut inst = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_iceberg_field_ids) + .build(); let result = inst.process_record_batch(source_record_batch()).unwrap(); @@ -537,7 +694,8 @@ mod test { let projected_iceberg_field_ids = [1, 2, 3]; let mut transformer = - RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_iceberg_field_ids) + .build(); let file_schema = Arc::new(ArrowSchema::new(vec![ simple_field("id", DataType::Int32, false, "1"), @@ -696,4 +854,592 @@ mod test { value.to_string(), )])) } + + /// Test for add_files with Parquet files that have NO field IDs (Hive tables). + /// + /// This reproduces the scenario from Iceberg spec where: + /// - Hive-style partitioned Parquet files are imported via add_files procedure + /// - Parquet files originally DO NOT have field IDs (typical for Hive tables) + /// - ArrowReader applies name mapping to assign correct Iceberg field IDs + /// - Iceberg schema assigns field IDs: id (1), name (2), dept (3), subdept (4) + /// - Partition columns (id, dept) have initial_default values + /// + /// Per the Iceberg spec (https://iceberg.apache.org/spec/#column-projection), + /// this scenario requires `schema.name-mapping.default` from table metadata + /// to correctly map Parquet columns by name to Iceberg field IDs. + /// This mapping is now applied in ArrowReader before data is processed. + /// + /// Expected behavior: + /// 1. id=1 (from initial_default) - spec rule #3 + /// 2. name="John Doe" (from Parquet with field_id=2 assigned by reader) - found by field ID + /// 3. dept="hr" (from initial_default) - spec rule #3 + /// 4. subdept="communications" (from Parquet with field_id=4 assigned by reader) - found by field ID + #[test] + fn add_files_with_name_mapping_applied_in_reader() { + // Iceberg schema after add_files: id (partition), name, dept (partition), subdept + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::optional(1, "id", Type::Primitive(PrimitiveType::Int)) + .with_initial_default(Literal::int(1)) + .into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "dept", Type::Primitive(PrimitiveType::String)) + .with_initial_default(Literal::string("hr")) + .into(), + NestedField::optional(4, "subdept", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build() + .unwrap(), + ); + + // Simulate ArrowReader having applied name mapping: + // Original Parquet: name, subdept (NO field IDs) + // After reader.rs applies name mapping: name (field_id=2), subdept (field_id=4) + // + // Note: Partition columns (id, dept) are NOT in the Parquet file - they're in directory paths + use std::collections::HashMap; + let parquet_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("name", DataType::Utf8, true).with_metadata(HashMap::from([( + "PARQUET:field_id".to_string(), + "2".to_string(), + )])), + Field::new("subdept", DataType::Utf8, true).with_metadata(HashMap::from([( + "PARQUET:field_id".to_string(), + "4".to_string(), + )])), + ])); + + let projected_field_ids = [1, 2, 3, 4]; // id, name, dept, subdept + + let mut transformer = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids).build(); + + // Create a Parquet RecordBatch with data for: name="John Doe", subdept="communications" + let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ + Arc::new(StringArray::from(vec!["John Doe"])), + Arc::new(StringArray::from(vec!["communications"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(parquet_batch).unwrap(); + + // Verify the transformed RecordBatch has: + // - id=1 (from initial_default, not from Parquet) + // - name="John Doe" (from Parquet with correct field_id=2) + // - dept="hr" (from initial_default, not from Parquet) + // - subdept="communications" (from Parquet with correct field_id=4) + assert_eq!(result.num_columns(), 4); + assert_eq!(result.num_rows(), 1); + + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_column.value(0), 1); + + let name_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_column.value(0), "John Doe"); + + let dept_column = result + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(dept_column.value(0), "hr"); + + let subdept_column = result + .column(3) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(subdept_column.value(0), "communications"); + } + + /// Test for bucket partitioning where source columns must be read from data files. + /// + /// This test verifies correct implementation of the Iceberg spec's "Column Projection" rules: + /// > "Return the value from partition metadata if an **Identity Transform** exists for the field" + /// + /// # Why this test is critical + /// + /// The key insight is that partition metadata stores TRANSFORMED values, not source values: + /// - For `bucket(4, id)`, partition metadata has `id_bucket = 2` (the bucket number) + /// - The actual `id` column values (100, 200, 300) are ONLY in the data file + /// + /// If iceberg-rust incorrectly treated bucket-partitioned fields as constants, it would: + /// 1. Replace all `id` values with the constant `2` from partition metadata + /// 2. Break runtime filtering (e.g., `WHERE id = 100` would match no rows) + /// 3. Return incorrect query results + /// + /// # What this test verifies + /// + /// - Bucket-partitioned fields (e.g., `bucket(4, id)`) are read from the data file + /// - The source column `id` contains actual values (100, 200, 300), not constants + /// - Java's `PartitionUtil.constantsMap()` behavior is correctly replicated: + /// ```java + /// if (field.transform().isIdentity()) { // FALSE for bucket transforms + /// idToConstant.put(field.sourceId(), converted); + /// } + /// ``` + /// + /// # Real-world impact + /// + /// This reproduces the failure scenario from Iceberg Java's TestRuntimeFiltering: + /// - Tables partitioned by `bucket(N, col)` are common for load balancing + /// - Queries filter on the source column: `SELECT * FROM tbl WHERE col = value` + /// - Runtime filtering pushes predicates down to Iceberg file scans + /// - Without this fix, the filter would match against constant partition values instead of data + /// + /// # References + /// - Iceberg spec: format/spec.md "Column Projection" + "Partition Transforms" + /// - Java impl: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java + /// - Java test: spark/src/test/java/.../TestRuntimeFiltering.java + #[test] + fn bucket_partitioning_reads_source_column_from_file() { + use crate::spec::{Struct, Transform}; + + // Table schema: id (data column), name (data column), id_bucket (partition column) + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + // Partition spec: bucket(4, id) - the id field is bucketed + let partition_spec = Arc::new( + crate::spec::PartitionSpec::builder(snapshot_schema.clone()) + .with_spec_id(0) + .add_partition_field("id", "id_bucket", Transform::Bucket(4)) + .unwrap() + .build() + .unwrap(), + ); + + // Partition data: bucket value is 2 + // In Iceberg, partition data is a Struct where each field corresponds to a partition field + let partition_data = Struct::from_iter(vec![Some(Literal::int(2))]); + + // Parquet file contains both id and name columns + let parquet_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("name", DataType::Utf8, true, "2"), + ])); + + let projected_field_ids = [1, 2]; // id, name + + let mut transformer = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) + .with_partition(partition_spec, partition_data) + .build(); + + // Create a Parquet RecordBatch with actual data + // The id column MUST be read from here, not treated as a constant + let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ + Arc::new(Int32Array::from(vec![100, 200, 300])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(parquet_batch).unwrap(); + + // Verify the transformed RecordBatch correctly reads id from the file + // (NOT as a constant from partition metadata) + assert_eq!(result.num_columns(), 2); + assert_eq!(result.num_rows(), 3); + + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + // These values MUST come from the Parquet file, not be replaced by constants + assert_eq!(id_column.value(0), 100); + assert_eq!(id_column.value(1), 200); + assert_eq!(id_column.value(2), 300); + + let name_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_column.value(0), "Alice"); + assert_eq!(name_column.value(1), "Bob"); + assert_eq!(name_column.value(2), "Charlie"); + } + + /// Test that identity-transformed partition fields ARE treated as constants. + /// + /// This is the complement to `bucket_partitioning_reads_source_column_from_file`, + /// verifying that constants_map() correctly identifies identity-transformed + /// partition fields per the Iceberg spec. + /// + /// # Spec requirement (format/spec.md "Column Projection") + /// + /// > "Return the value from partition metadata if an Identity Transform exists for the field + /// > and the partition value is present in the `partition` struct on `data_file` object + /// > in the manifest. This allows for metadata only migrations of Hive tables." + /// + /// # Why identity transforms use constants + /// + /// Unlike bucket/truncate/year/etc., identity transforms don't modify the value: + /// - `identity(dept)` stores the actual `dept` value in partition metadata + /// - Partition metadata has `dept = "engineering"` (the real value, not a hash/bucket) + /// - This value can be used directly without reading the data file + /// + /// # Performance benefit + /// + /// For Hive migrations where partition columns aren't in data files: + /// - Partition metadata provides the column values + /// - No need to read from data files (metadata-only query optimization) + /// - Common pattern: `dept=engineering/subdept=backend/file.parquet` + /// - `dept` and `subdept` are in directory structure, not in `file.parquet` + /// - Iceberg populates these from partition metadata as constants + /// + /// # What this test verifies + /// + /// - Identity-partitioned fields use constants from partition metadata + /// - The `dept` column is populated with `"engineering"` (not read from file) + /// - Java's `PartitionUtil.constantsMap()` behavior is matched: + /// ```java + /// if (field.transform().isIdentity()) { // TRUE for identity + /// idToConstant.put(field.sourceId(), converted); + /// } + /// ``` + /// + /// # References + /// - Iceberg spec: format/spec.md "Column Projection" + /// - Java impl: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java + #[test] + fn identity_partition_uses_constant_from_metadata() { + use crate::spec::{Struct, Transform}; + + // Table schema: id (data column), dept (partition column), name (data column) + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "dept", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + // Partition spec: identity(dept) - the dept field uses identity transform + let partition_spec = Arc::new( + crate::spec::PartitionSpec::builder(snapshot_schema.clone()) + .with_spec_id(0) + .add_partition_field("dept", "dept", Transform::Identity) + .unwrap() + .build() + .unwrap(), + ); + + // Partition data: dept="engineering" + let partition_data = Struct::from_iter(vec![Some(Literal::string("engineering"))]); + + // Parquet file contains only id and name (dept is in partition path) + let parquet_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("name", DataType::Utf8, true, "3"), + ])); + + let projected_field_ids = [1, 2, 3]; // id, dept, name + + let mut transformer = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) + .with_partition(partition_spec, partition_data) + .build(); + + let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ + Arc::new(Int32Array::from(vec![100, 200])), + Arc::new(StringArray::from(vec!["Alice", "Bob"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(parquet_batch).unwrap(); + + // Verify the dept column is populated with the constant from partition metadata + assert_eq!(result.num_columns(), 3); + assert_eq!(result.num_rows(), 2); + + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_column.value(0), 100); + assert_eq!(id_column.value(1), 200); + + let dept_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + // This value MUST come from partition metadata (constant) + assert_eq!(dept_column.value(0), "engineering"); + assert_eq!(dept_column.value(1), "engineering"); + + let name_column = result + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_column.value(0), "Alice"); + assert_eq!(name_column.value(1), "Bob"); + } + + /// Test bucket partitioning with renamed source column. + /// + /// This verifies correct behavior for TestRuntimeFiltering.testRenamedSourceColumnTable() in Iceberg Java. + /// When a source column is renamed after partitioning is established, field-ID-based mapping + /// must still correctly identify the column in Parquet files. + /// + /// # Scenario + /// + /// 1. Table created with `bucket(4, id)` partitioning + /// 2. Data written to Parquet files (field_id=1, name="id") + /// 3. Column renamed: `ALTER TABLE ... RENAME COLUMN id TO row_id` + /// 4. Iceberg schema now has: field_id=1, name="row_id" + /// 5. Parquet files still have: field_id=1, name="id" + /// + /// # Expected Behavior Per Iceberg Spec + /// + /// Per the Iceberg spec "Column Projection" section and Java's PartitionUtil.constantsMap(): + /// - Bucket transforms are NON-identity, so partition metadata stores bucket numbers (0-3), not source values + /// - Source columns for non-identity transforms MUST be read from data files + /// - Field-ID-based mapping should find the column by field_id=1 (ignoring name mismatch) + /// - Runtime filtering on `row_id` should work correctly + /// + /// # What This Tests + /// + /// This test ensures that when FileScanTask provides partition_spec and partition_data: + /// - constants_map() correctly identifies that bucket(4, row_id) is NOT an identity transform + /// - The source column (field_id=1) is NOT added to constants_map + /// - Field-ID-based mapping reads actual values from the Parquet file + /// - Values [100, 200, 300] are read, not replaced with bucket constant 2 + /// + /// # References + /// - Java test: spark/src/test/java/.../TestRuntimeFiltering.java::testRenamedSourceColumnTable + /// - Java impl: core/src/main/java/org/apache/iceberg/util/PartitionUtil.java::constantsMap() + /// - Iceberg spec: format/spec.md "Column Projection" section + #[test] + fn test_bucket_partitioning_with_renamed_source_column() { + use crate::spec::{Struct, Transform}; + + // Iceberg schema after rename: row_id (was id), name + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "row_id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + // Partition spec: bucket(4, row_id) - but source_id still points to field_id=1 + let partition_spec = Arc::new( + crate::spec::PartitionSpec::builder(snapshot_schema.clone()) + .with_spec_id(0) + .add_partition_field("row_id", "row_id_bucket", Transform::Bucket(4)) + .unwrap() + .build() + .unwrap(), + ); + + // Partition data: bucket value is 2 + let partition_data = Struct::from_iter(vec![Some(Literal::int(2))]); + + // Parquet file has OLD column name "id" but SAME field_id=1 + // Field-ID-based mapping should find this despite name mismatch + let parquet_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("name", DataType::Utf8, true, "2"), + ])); + + let projected_field_ids = [1, 2]; // row_id (field_id=1), name (field_id=2) + + let mut transformer = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) + .with_partition(partition_spec, partition_data) + .build(); + + // Create a Parquet RecordBatch with actual data + // Despite column rename, data should be read via field_id=1 + let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ + Arc::new(Int32Array::from(vec![100, 200, 300])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(parquet_batch).unwrap(); + + // Verify the transformed RecordBatch correctly reads data despite name mismatch + assert_eq!(result.num_columns(), 2); + assert_eq!(result.num_rows(), 3); + + let row_id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + // These values MUST come from the Parquet file via field_id=1, + // not be replaced by the bucket constant (2) + assert_eq!(row_id_column.value(0), 100); + assert_eq!(row_id_column.value(1), 200); + assert_eq!(row_id_column.value(2), 300); + + let name_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_column.value(0), "Alice"); + assert_eq!(name_column.value(1), "Bob"); + assert_eq!(name_column.value(2), "Charlie"); + } + + /// Comprehensive integration test that verifies all 4 Iceberg spec rules work correctly. + /// + /// Per the Iceberg spec (https://iceberg.apache.org/spec/#column-projection), + /// "Values for field ids which are not present in a data file must be resolved + /// according the following rules:" + /// + /// This test creates a scenario where each rule is exercised: + /// - Rule #1: dept (identity-partitioned) -> constant from partition metadata + /// - Rule #2: data (via name mapping) -> read from Parquet file by name + /// - Rule #3: category (initial_default) -> use default value + /// - Rule #4: notes (no default) -> return null + /// + /// # References + /// - Iceberg spec: format/spec.md "Column Projection" section + #[test] + fn test_all_four_spec_rules() { + use crate::spec::Transform; + + // Iceberg schema with columns designed to exercise each spec rule + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + // Field in Parquet by field ID (normal case) + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + // Rule #1: Identity-partitioned field - should use partition metadata + NestedField::required(2, "dept", Type::Primitive(PrimitiveType::String)).into(), + // Rule #2: Field resolved by name mapping (ArrowReader already applied) + NestedField::required(3, "data", Type::Primitive(PrimitiveType::String)).into(), + // Rule #3: Field with initial_default + NestedField::optional(4, "category", Type::Primitive(PrimitiveType::String)) + .with_initial_default(Literal::string("default_category")) + .into(), + // Rule #4: Field with no default - should be null + NestedField::optional(5, "notes", Type::Primitive(PrimitiveType::String)) + .into(), + ]) + .build() + .unwrap(), + ); + + // Partition spec: identity transform on dept + let partition_spec = Arc::new( + crate::spec::PartitionSpec::builder(snapshot_schema.clone()) + .with_spec_id(0) + .add_partition_field("dept", "dept", Transform::Identity) + .unwrap() + .build() + .unwrap(), + ); + + // Partition data: dept="engineering" + let partition_data = Struct::from_iter(vec![Some(Literal::string("engineering"))]); + + // Parquet schema: simulates post-ArrowReader state where name mapping already applied + // Has id (field_id=1) and data (field_id=3, assigned by ArrowReader via name mapping) + // Missing: dept (in partition), category (has default), notes (no default) + let parquet_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("data", DataType::Utf8, false, "3"), + ])); + + let projected_field_ids = [1, 2, 3, 4, 5]; // id, dept, data, category, notes + + let mut transformer = + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) + .with_partition(partition_spec, partition_data) + .build(); + + let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ + Arc::new(Int32Array::from(vec![100, 200])), + Arc::new(StringArray::from(vec!["value1", "value2"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(parquet_batch).unwrap(); + + assert_eq!(result.num_columns(), 5); + assert_eq!(result.num_rows(), 2); + + // Verify each column demonstrates the correct spec rule: + + // Normal case: id from Parquet by field ID + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_column.value(0), 100); + assert_eq!(id_column.value(1), 200); + + // Rule #1: dept from partition metadata (identity transform) + let dept_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(dept_column.value(0), "engineering"); + assert_eq!(dept_column.value(1), "engineering"); + + // Rule #2: data from Parquet via name mapping + let data_column = result + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(data_column.value(0), "value1"); + assert_eq!(data_column.value(1), "value2"); + + // Rule #3: category from initial_default + let category_column = result + .column(3) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(category_column.value(0), "default_category"); + assert_eq!(category_column.value(1), "default_category"); + + // Rule #4: notes is null (no default, not in Parquet, not in partition) + let notes_column = result + .column(4) + .as_any() + .downcast_ref::() + .unwrap(); + assert!(notes_column.is_null(0)); + assert!(notes_column.is_null(1)); + } } diff --git a/crates/iceberg/src/scan/context.rs b/crates/iceberg/src/scan/context.rs index 3f7c29dbf4..fe3f5c8f7e 100644 --- a/crates/iceberg/src/scan/context.rs +++ b/crates/iceberg/src/scan/context.rs @@ -128,6 +128,13 @@ impl ManifestEntryContext { .map(|x| x.as_ref().snapshot_bound_predicate.clone()), deletes, + + // Include partition data and spec from manifest entry + partition: Some(self.manifest_entry.data_file.partition.clone()), + // TODO: Pass actual PartitionSpec through context chain for native flow + partition_spec: None, + // TODO: Extract name_mapping from table metadata property "schema.name-mapping.default" + name_mapping: None, }) } } diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 6884e00b9b..3e319ca062 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -1777,6 +1777,9 @@ pub mod tests { record_count: Some(100), data_file_format: DataFileFormat::Parquet, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, }; test_fn(task); @@ -1791,6 +1794,9 @@ pub mod tests { record_count: None, data_file_format: DataFileFormat::Avro, deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, }; test_fn(task); } diff --git a/crates/iceberg/src/scan/task.rs b/crates/iceberg/src/scan/task.rs index 32fe3ae309..e1ef241a57 100644 --- a/crates/iceberg/src/scan/task.rs +++ b/crates/iceberg/src/scan/task.rs @@ -15,16 +15,39 @@ // specific language governing permissions and limitations // under the License. +use std::sync::Arc; + use futures::stream::BoxStream; -use serde::{Deserialize, Serialize}; +use serde::{Deserialize, Serialize, Serializer}; use crate::Result; use crate::expr::BoundPredicate; -use crate::spec::{DataContentType, DataFileFormat, ManifestEntryRef, Schema, SchemaRef}; +use crate::spec::{ + DataContentType, DataFileFormat, ManifestEntryRef, NameMapping, PartitionSpec, Schema, + SchemaRef, Struct, +}; /// A stream of [`FileScanTask`]. pub type FileScanTaskStream = BoxStream<'static, Result>; +/// Serialization helper that always returns NotImplementedError. +/// Used for fields that should not be serialized but we want to be explicit about it. +fn serialize_not_implemented(_: &T, _: S) -> std::result::Result +where S: Serializer { + Err(serde::ser::Error::custom( + "Serialization not implemented for this field", + )) +} + +/// Deserialization helper that always returns NotImplementedError. +/// Used for fields that should not be deserialized but we want to be explicit about it. +fn deserialize_not_implemented<'de, D, T>(_: D) -> std::result::Result +where D: serde::Deserializer<'de> { + Err(serde::de::Error::custom( + "Deserialization not implemented for this field", + )) +} + /// A task to scan part of file. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct FileScanTask { @@ -54,6 +77,33 @@ pub struct FileScanTask { /// The list of delete files that may need to be applied to this data file pub deletes: Vec, + + /// Partition data from the manifest entry, used to identify which columns can use + /// constant values from partition metadata vs. reading from the data file. + /// Per the Iceberg spec, only identity-transformed partition fields should use constants. + #[serde(default)] + #[serde(skip_serializing_if = "Option::is_none")] + #[serde(serialize_with = "serialize_not_implemented")] + #[serde(deserialize_with = "deserialize_not_implemented")] + pub partition: Option, + + /// The partition spec for this file, used to distinguish identity transforms + /// (which use partition metadata constants) from non-identity transforms like + /// bucket/truncate (which must read source columns from the data file). + #[serde(default)] + #[serde(skip_serializing_if = "Option::is_none")] + #[serde(serialize_with = "serialize_not_implemented")] + #[serde(deserialize_with = "deserialize_not_implemented")] + pub partition_spec: Option>, + + /// Name mapping from table metadata (property: schema.name-mapping.default), + /// used to resolve field IDs from column names when Parquet files lack field IDs + /// or have field ID conflicts. + #[serde(default)] + #[serde(skip_serializing_if = "Option::is_none")] + #[serde(serialize_with = "serialize_not_implemented")] + #[serde(deserialize_with = "deserialize_not_implemented")] + pub name_mapping: Option>, } impl FileScanTask { From 12c4c2110174b76eb82e584cb7933efdc9afcecc Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Thu, 13 Nov 2025 05:21:02 -0500 Subject: [PATCH 074/891] feat(reader): null struct default values in create_column (#1847) Fixes `TestSparkReaderDeletes.testPosDeletesOnParquetFileWithMultipleRowGroups` in Iceberg Java 1.10 with DataFusion Comet. ## Which issue does this PR close? - Partially address #1749. ## What changes are included in this PR? - While `RecordBatchTransformer` does not have exhaustive nested type support yet, this adds logic to `create_column` in the specific scenario for a schema evolution with a new struct column that uses the default NULL value. - If the column has a default value other than NULL defined, it will fall into the existing match arm and say it is unsupported. ## Are these changes tested? New test to reflect what happens with Iceberg Java 1.10's `TestSparkReaderDeletes.testPosDeletesOnParquetFileWithMultipleRowGroups`. The test is misleading, since I figured testing positional deletes would just be a delete vector and be schema agnostic, but [it includes schema change with binary and struct types so we need default NULL values](https://github.com/apache/iceberg/blob/53c046efda5d6c6ac67caf7de29849ab7ac6d406/data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java#L65). --- .../src/arrow/record_batch_transformer.rs | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index e7d8b8f0fb..07ec43918f 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -21,7 +21,9 @@ use std::sync::Arc; use arrow_array::{ Array as ArrowArray, ArrayRef, BinaryArray, BooleanArray, Date32Array, Float32Array, Float64Array, Int32Array, Int64Array, NullArray, RecordBatch, RecordBatchOptions, StringArray, + StructArray, }; +use arrow_buffer::NullBuffer; use arrow_cast::cast; use arrow_schema::{ DataType, FieldRef, Schema as ArrowSchema, SchemaRef as ArrowSchemaRef, SchemaRef, @@ -594,6 +596,21 @@ impl RecordBatchTransformer { let vals: Vec> = vec![None; num_rows]; Arc::new(BinaryArray::from_opt_vec(vals)) } + (DataType::Struct(fields), None) => { + // Create a StructArray filled with nulls. Per Iceberg spec, optional struct fields + // default to null when added to the schema. We defer non-null default struct values + // and leave them as not implemented yet. + let null_arrays: Vec = fields + .iter() + .map(|field| Self::create_column(field.data_type(), &None, num_rows)) + .collect::>>()?; + + Arc::new(StructArray::new( + fields.clone(), + null_arrays, + Some(NullBuffer::new_null(num_rows)), + )) + } (DataType::Null, _) => Arc::new(NullArray::new(num_rows)), (dt, _) => { return Err(Error::new( @@ -743,6 +760,82 @@ mod test { assert!(date_column.is_null(2)); } + #[test] + fn schema_evolution_adds_struct_column_with_nulls() { + // Test that when a struct column is added after data files are written, + // the transformer can materialize the missing struct column with null values. + // This reproduces the scenario from Iceberg 1.10.0 TestSparkReaderDeletes tests + // where binaryData and structData columns were added to the schema. + let snapshot_schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "data", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional( + 3, + "struct_col", + Type::Struct(crate::spec::StructType::new(vec![ + NestedField::optional( + 100, + "inner_field", + Type::Primitive(PrimitiveType::String), + ) + .into(), + ])), + ) + .into(), + ]) + .build() + .unwrap(), + ); + let projected_iceberg_field_ids = [1, 2, 3]; + + let mut transformer = + RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + + let file_schema = Arc::new(ArrowSchema::new(vec![ + simple_field("id", DataType::Int32, false, "1"), + simple_field("data", DataType::Utf8, false, "2"), + ])); + + let file_batch = RecordBatch::try_new(file_schema, vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ]) + .unwrap(); + + let result = transformer.process_record_batch(file_batch).unwrap(); + + assert_eq!(result.num_columns(), 3); + assert_eq!(result.num_rows(), 3); + + let id_column = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_column.values(), &[1, 2, 3]); + + let data_column = result + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(data_column.value(0), "a"); + assert_eq!(data_column.value(1), "b"); + assert_eq!(data_column.value(2), "c"); + + let struct_column = result + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + assert!(struct_column.is_null(0)); + assert!(struct_column.is_null(1)); + assert!(struct_column.is_null(2)); + } + pub fn source_record_batch() -> RecordBatch { RecordBatch::try_new( arrow_schema_promotion_addition_and_renaming_required(), From ff408ff3e0dfff24a6e6d39090dbd145afecf2f6 Mon Sep 17 00:00:00 2001 From: Li0k Date: Thu, 13 Nov 2025 18:36:57 +0800 Subject: [PATCH 075/891] refactor: Remove redundant parameters from SnapshotProducer validation methods (#1853) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Summary Refactor `SnapshotProducer` validation methods to use internal state instead of requiring redundant parameters. ## Problem I've noticed that while the current **SnapshotProducer** API design already equips SnapshotProducer with all necessary state, the current invocations still redundantly pass parameters externally. I believe this could lead to some issues. 1. **Data mismatch risk**: Callers could pass different data than what's stored in `SnapshotProducer`, leading to validating one set of files but committing another 2. **API complexity**: As more validations are added (e.g., delete files, file existence checks), each method would require additional parameters, making the API harder to use 3. **Redundant passing**: The same data that was already provided during construction has to be passed again ## Changes - Modified `validate_added_data_files()` and `validate_duplicate_files()` to operate on `self.added_data_files` directly - Updated `FastAppendAction::commit()` to call validation methods without passing `added_data_files` parameter ## Motivation Previously, `added_data_files` was passed as a parameter to validation methods even though it was already stored in `SnapshotProducer`: ```rust // Before snapshot_producer.validate_added_data_files(&self.added_data_files)?; // After snapshot_producer.validate_added_data_files()?; ``` ## Benefits 1. Better encapsulation - validation operates on object's own state 2. Safer API - eliminates possibility of data mismatch 3. Simpler interface - no redundant parameters needed ## Discussion Since **SnapshotProducer** already holds all necessary state, can we further refine validation by performing it during the **new** function's execution to improve data consistency and encapsulation? ## Are these changes tested? --- crates/iceberg/src/transaction/append.rs | 6 ++---- crates/iceberg/src/transaction/snapshot.rs | 12 +++++------- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/crates/iceberg/src/transaction/append.rs b/crates/iceberg/src/transaction/append.rs index f248543df2..08d4032409 100644 --- a/crates/iceberg/src/transaction/append.rs +++ b/crates/iceberg/src/transaction/append.rs @@ -93,13 +93,11 @@ impl TransactionAction for FastAppendAction { ); // validate added files - snapshot_producer.validate_added_data_files(&self.added_data_files)?; + snapshot_producer.validate_added_data_files()?; // Checks duplicate files if self.check_duplicate { - snapshot_producer - .validate_duplicate_files(&self.added_data_files) - .await?; + snapshot_producer.validate_duplicate_files().await?; } snapshot_producer diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index 4f85962ff1..6b3d0e4ff4 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -99,8 +99,8 @@ impl<'a> SnapshotProducer<'a> { } } - pub(crate) fn validate_added_data_files(&self, added_data_files: &[DataFile]) -> Result<()> { - for data_file in added_data_files { + pub(crate) fn validate_added_data_files(&self) -> Result<()> { + for data_file in &self.added_data_files { if data_file.content_type() != crate::spec::DataContentType::Data { return Err(Error::new( ErrorKind::DataInvalid, @@ -123,11 +123,9 @@ impl<'a> SnapshotProducer<'a> { Ok(()) } - pub(crate) async fn validate_duplicate_files( - &self, - added_data_files: &[DataFile], - ) -> Result<()> { - let new_files: HashSet<&str> = added_data_files + pub(crate) async fn validate_duplicate_files(&self) -> Result<()> { + let new_files: HashSet<&str> = self + .added_data_files .iter() .map(|df| df.file_path.as_str()) .collect(); From 0107f17f568c770d22c13aab9f38f1726df38745 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Thu, 13 Nov 2025 02:38:24 -0800 Subject: [PATCH 076/891] infra: add verbose=true to pypa/gh-action-pypi-publish (#1846) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Follow up to #1836 Also add `verbose: true` to `pypa/gh-action-pypi-publish` in `.github/workflows/release_python.yml` ## Are these changes tested? --- .github/workflows/release_python.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index bc2aae1290..e6b7021c9b 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -185,7 +185,7 @@ jobs: path: bindings/python/dist - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@release/v1 - with: skip-existing: true packages-dir: bindings/python/dist + verbose: true From 641772ca30efd93b12b109efcddd514e9458f081 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sun, 16 Nov 2025 08:59:37 -0800 Subject: [PATCH 077/891] use RecordBatchTransformerBuilder instead of RecordBatchTransformer (#1857) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This should fix CI, currently broken with ``` 795 | RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); | ^^^^^ function or associated item not found in `RecordBatchTransformer` ``` Likely due to merge conflict between https://github.com/apache/iceberg-rust/pull/1847/files#diff-640415fbb0b37eac6f774eff524fb364c4e54c02da3a3c0e355af5caa941007cR637 and https://github.com/apache/iceberg-rust/pull/1821/files#diff-640415fbb0b37eac6f774eff524fb364c4e54c02da3a3c0e355af5caa941007cR149 ## Are these changes tested? --- crates/iceberg/src/arrow/record_batch_transformer.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index 07ec43918f..a20adb6a5a 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -792,7 +792,8 @@ mod test { let projected_iceberg_field_ids = [1, 2, 3]; let mut transformer = - RecordBatchTransformer::build(snapshot_schema, &projected_iceberg_field_ids); + RecordBatchTransformerBuilder::new(snapshot_schema, &projected_iceberg_field_ids) + .build(); let file_schema = Arc::new(ArrowSchema::new(vec![ simple_field("id", DataType::Int32, false, "1"), From ff91de2994f7901a71fb4bbe0cd37dbbfc917cb5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Nov 2025 09:29:52 -0800 Subject: [PATCH 078/891] chore(deps): Bump bytes from 1.10.1 to 1.11.0 (#1861) Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.10.1 to 1.11.0.
Changelog

Sourced from bytes's changelog.

1.11.0 (November 14th, 2025)

  • Bump MSRV to 1.57 (#788)

Fixed

  • fix: BytesMut only reuse if src has remaining (#803)
  • Specialize BytesMut::put::<Bytes> (#793)
  • Reserve capacity in BytesMut::put (#794)
  • Change BytesMut::remaining_mut to use isize::MAX instead of usize::MAX (#795)

Internal changes

  • Guarantee address in slice() for empty slices. (#780)
  • Rename Vtable::to_* -> Vtable::into_* (#776)
  • Fix latest clippy warnings (#787)
  • Ignore BytesMut::freeze doctest on wasm (#790)
  • Move drop_fn of from_owner into vtable (#801)
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=bytes&package-manager=cargo&previous-version=1.10.1&new-version=1.11.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Liu --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 2edc51a4ea..b6efcd9c1a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1284,9 +1284,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.10.1" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d71b6127be86fdcfddb610f7182ac57211d4b18a3e9c82eb2d17662f2227ad6a" +checksum = "b35204fbdc0b3f4446b89fc1ac2cf84a8a68971995d0bf2e925ec7cd960f9cb3" dependencies = [ "serde", ] From 419b1008c103b45f719d002a1f8a0da2498d061d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Nov 2025 09:30:02 -0800 Subject: [PATCH 079/891] chore(deps): Bump serde_with from 3.15.1 to 3.16.0 (#1859) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [serde_with](https://github.com/jonasbb/serde_with) from 3.15.1 to 3.16.0.
Release notes

Sourced from serde_with's releases.

serde_with v3.16.0

Added

Commits
  • 066b9d4 Bump version to 3.16.0 (#903)
  • 961a0b9 Bump version to 3.16.0
  • ac1f5d0 Add support for smallvec v1 (#895)
  • c83c4fc Sort the crate features and remove alloc feature
  • af734ff Add support for smallvec v1
  • ec0ffa3 Add JsonSchemaAs implementation for json::JsonString (#901)
  • b0c119c Add JsonSchemaAs implementation for json::JsonString
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=serde_with&package-manager=cargo&previous-version=3.15.1&new-version=3.16.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Liu --- Cargo.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index b6efcd9c1a..1bad202c8c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6287,9 +6287,9 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.15.1" +version = "3.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa66c845eee442168b2c8134fec70ac50dc20e760769c8ba0ad1319ca1959b04" +checksum = "10574371d41b0d9b2cff89418eda27da52bcaff2cc8741db26382a77c29131f1" dependencies = [ "base64 0.22.1", "chrono", @@ -6306,9 +6306,9 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.15.1" +version = "3.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b91a903660542fced4e99881aa481bdbaec1634568ee02e0b8bd57c64cb38955" +checksum = "08a72d8216842fdd57820dc78d840bef99248e35fb2554ff923319e60f2d686b" dependencies = [ "darling 0.21.3", "proc-macro2", From 08e28cdbc83dd0d60cef0fb80fa8d07392f29e7c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Nov 2025 10:33:47 -0800 Subject: [PATCH 080/891] chore(deps): Bump fs-err from 3.1.3 to 3.2.0 (#1860) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [fs-err](https://github.com/andrewhickman/fs-err) from 3.1.3 to 3.2.0.
Changelog

Sourced from fs-err's changelog.

fs-err Changelog

  • Introduce debug and debug_tokio feature. Debug filesystem errors faster by exposing more information (#81). Without this feature on, errors might look like this:

    failed to open file `file.txt`: The system cannot find the
    file specified. (os error 2)
    

    With this feature on, it will include additional information. For example:

    failed to open file `file.txt`: The system cannot find the
    file specified. (os error 2)
    

    Path does not exist file.txt

    • Absolute path /path/to/dir/file.txt
    • Missing file.txt from parent directory:
      /path/to/dir
      └── file.md
      └── different.txt

It's suggested to enable this feature in dev-dependencies for security and performance reasons.

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fs-err&package-manager=cargo&previous-version=3.1.3&new-version=3.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Liu --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1bad202c8c..62478f32a0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2949,9 +2949,9 @@ checksum = "28dd6caf6059519a65843af8fe2a3ae298b14b80179855aeb4adc2c1934ee619" [[package]] name = "fs-err" -version = "3.1.3" +version = "3.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ad492b2cf1d89d568a43508ab24f98501fe03f2f31c01e1d0fe7366a71745d2" +checksum = "62d91fd049c123429b018c47887d3f75a265540dd3c30ba9cb7bae9197edb03a" dependencies = [ "autocfg", ] From 75be9c6a3b970079fdccc3d638b673aaaec68cb8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Nov 2025 10:33:55 -0800 Subject: [PATCH 081/891] chore(deps): Bump crate-ci/typos from 1.39.0 to 1.39.2 (#1858) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.39.0 to 1.39.2.
Release notes

Sourced from crate-ci/typos's releases.

v1.39.2

[1.39.2] - 2025-11-13

Fixes

  • Don't offer entry as a correction for entrys

v1.39.1

[1.39.1] - 2025-11-12

Features

  • Make --help more vibrant
Changelog

Sourced from crate-ci/typos's changelog.

[1.39.2] - 2025-11-13

Fixes

  • Don't offer entry as a correction for entrys

[1.39.1] - 2025-11-12

Features

  • Make --help more vibrant
Commits
  • 626c4be chore: Release
  • c6b458d docs: Update changelog
  • eed0419 Merge pull request #1423 from epage/entrys
  • 40383f4 fix(dict): Don't offer 'entry' as a correction for 'entrys'
  • 1af53e3 chore: Release
  • e5d291b docs: Update changelog
  • 55474f5 Merge pull request #1417 from starsep/colorful_help
  • 78b9375 feat: Enable colors for typos --help
  • 308f8f5 Merge pull request #1409 from crate-ci/renovate/actions-download-artifact-6.x
  • cf03418 Merge pull request #1410 from crate-ci/renovate/actions-setup-python-6.x
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.39.0&new-version=1.39.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Liu --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 693199eda6..4c60369482 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v5 - name: Check typos - uses: crate-ci/typos@v1.39.0 + uses: crate-ci/typos@v1.39.2 From e519c3aed12ec762b360e7ee9b53530b6214015e Mon Sep 17 00:00:00 2001 From: Jonathan Chen Date: Mon, 17 Nov 2025 04:02:12 -0500 Subject: [PATCH 082/891] chore: Remove deprecated `remove_all` in FileIO (#1863) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? --- crates/iceberg/src/io/file_io.rs | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/crates/iceberg/src/io/file_io.rs b/crates/iceberg/src/io/file_io.rs index 6e2d152ed7..24e91ca8a4 100644 --- a/crates/iceberg/src/io/file_io.rs +++ b/crates/iceberg/src/io/file_io.rs @@ -93,17 +93,6 @@ impl FileIO { Ok(op.delete(relative_path).await?) } - /// Remove the path and all nested dirs and files recursively. - /// - /// # Arguments - /// - /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. - #[deprecated(note = "use remove_dir_all instead", since = "0.4.0")] - pub async fn remove_all(&self, path: impl AsRef) -> Result<()> { - let (op, relative_path) = self.inner.create_operator(&path)?; - Ok(op.remove_all(relative_path).await?) - } - /// Remove the path and all nested dirs and files recursively. /// /// # Arguments @@ -292,7 +281,6 @@ pub struct FileMetadata { /// Trait for reading file. /// /// # TODO -/// /// It's possible for us to remove the async_trait, but we need to figure /// out how to handle the object safety. #[async_trait::async_trait] From 46fa95cc0b8c8b94059684fa44fa5eca0d501032 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 18 Nov 2025 01:30:54 -0800 Subject: [PATCH 083/891] infra: notify on github workflow failure (#1870) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Notify ci-jobs@iceberg.apache.org when Github Workflow fails For more context, see https://lists.apache.org/thread/sx7hz7tcs39tcfgywdh97gvbrtpdftxl ## Are these changes tested? --- .asf.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.asf.yaml b/.asf.yaml index a5c0f93084..f46c437a79 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -61,5 +61,6 @@ notifications: commits: commits@iceberg.apache.org issues: issues@iceberg.apache.org pullrequests: issues@iceberg.apache.org + jobs: ci-jobs@iceberg.apache.org jira_options: link label link label discussions: issues@iceberg.apache.org From a6675394c0b15f90e70b907b0742d5be6ac39a3e Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 18 Nov 2025 06:00:25 -0500 Subject: [PATCH 084/891] feat(reader): Add binary support to `get_arrow_datum` for equality deletes with binary type (#1848) --- .../src/arrow/caching_delete_file_loader.rs | 16 ++++++++++++---- crates/iceberg/src/arrow/schema.rs | 15 +++++++++++++-- 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index f1c4f86f23..192ca390a8 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -543,7 +543,9 @@ mod tests { use std::sync::Arc; use arrow_array::cast::AsArray; - use arrow_array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringArray, StructArray}; + use arrow_array::{ + ArrayRef, BinaryArray, Int32Array, Int64Array, RecordBatch, StringArray, StructArray, + }; use arrow_schema::{DataType, Field, Fields}; use parquet::arrow::{ArrowWriter, PARQUET_FIELD_ID_META_KEY}; use parquet::basic::Compression; @@ -552,6 +554,8 @@ mod tests { use super::*; use crate::arrow::delete_filter::tests::setup; + use crate::scan::FileScanTaskDeleteFile; + use crate::spec::{DataContentType, Schema}; #[tokio::test] async fn test_delete_file_loader_parse_equality_deletes() { @@ -567,7 +571,7 @@ mod tests { .await .expect("could not get batch stream"); - let eq_ids = HashSet::from_iter(vec![2, 3, 4, 6]); + let eq_ids = HashSet::from_iter(vec![2, 3, 4, 6, 8]); let parsed_eq_delete = CachingDeleteFileLoader::parse_equality_deletes_record_batch_stream( record_batch_stream, @@ -577,7 +581,7 @@ mod tests { .expect("error parsing batch stream"); println!("{parsed_eq_delete}"); - let expected = "((((y != 1) OR (z != 100)) OR (a != \"HELP\")) OR (sa != 4)) AND ((((y != 2) OR (z IS NOT NULL)) OR (a IS NOT NULL)) OR (sa != 5))".to_string(); + let expected = "(((((y != 1) OR (z != 100)) OR (a != \"HELP\")) OR (sa != 4)) OR (b != 62696E6172795F64617461)) AND (((((y != 2) OR (z IS NOT NULL)) OR (a IS NOT NULL)) OR (sa != 5)) OR (b IS NOT NULL))".to_string(); assert_eq!(parsed_eq_delete.to_string(), expected); } @@ -611,6 +615,9 @@ mod tests { ), ])); + let col_b_vals = vec![Some(&b"binary_data"[..]), None]; + let col_b = Arc::new(BinaryArray::from(col_b_vals)) as ArrayRef; + let equality_delete_schema = { let struct_field = DataType::Struct(Fields::from(vec![ simple_field("sa", DataType::Int32, false, "6"), @@ -628,12 +635,13 @@ mod tests { (PARQUET_FIELD_ID_META_KEY.to_string(), "4".to_string()), ])), simple_field("s", struct_field, false, "5"), + simple_field("b", DataType::Binary, true, "8"), ]; Arc::new(arrow_schema::Schema::new(fields)) }; let equality_deletes_to_write = RecordBatch::try_new(equality_delete_schema.clone(), vec![ - col_y, col_z, col_a, col_s, + col_y, col_z, col_a, col_s, col_b, ]) .unwrap(); diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index e10db3a5aa..ec0135bd77 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -22,8 +22,8 @@ use std::sync::Arc; use arrow_array::types::{Decimal128Type, validate_decimal_precision_and_scale}; use arrow_array::{ - BooleanArray, Date32Array, Datum as ArrowDatum, Decimal128Array, FixedSizeBinaryArray, - Float32Array, Float64Array, Int32Array, Int64Array, Scalar, StringArray, + BinaryArray, BooleanArray, Date32Array, Datum as ArrowDatum, Decimal128Array, + FixedSizeBinaryArray, Float32Array, Float64Array, Int32Array, Int64Array, Scalar, StringArray, TimestampMicrosecondArray, }; use arrow_schema::{DataType, Field, Fields, Schema as ArrowSchema, TimeUnit}; @@ -678,6 +678,9 @@ pub(crate) fn get_arrow_datum(datum: &Datum) -> Result { Ok(Arc::new(StringArray::new_scalar(value.as_str()))) } + (PrimitiveType::Binary, PrimitiveLiteral::Binary(value)) => { + Ok(Arc::new(BinaryArray::new_scalar(value.as_slice()))) + } (PrimitiveType::Date, PrimitiveLiteral::Int(value)) => { Ok(Arc::new(Date32Array::new_scalar(*value))) } @@ -1814,6 +1817,14 @@ mod tests { assert!(is_scalar); assert_eq!(array.value(0), "abc"); } + { + let datum = Datum::binary(vec![1, 2, 3, 4]); + let arrow_datum = get_arrow_datum(&datum).unwrap(); + let (array, is_scalar) = arrow_datum.get(); + let array = array.as_any().downcast_ref::().unwrap(); + assert!(is_scalar); + assert_eq!(array.value(0), &[1, 2, 3, 4]); + } { let datum = Datum::date(42); let arrow_datum = get_arrow_datum(&datum).unwrap(); From 760b752aeda1884dc429252e6cf6405d8fb51fa1 Mon Sep 17 00:00:00 2001 From: Jem Bishop <40360024+jembishop@users.noreply.github.com> Date: Wed, 19 Nov 2025 12:16:43 +0000 Subject: [PATCH 085/891] Raise concurrency errors properly for glue tables (#1875) ## Which issue does this PR close? #1868 ## What changes are included in this PR? As referenced in the issue, the optimistic concurrency was not working correctly as we need to check the glue table version has not incremented before we make our update, so we can get back the error if there is a concurrent modification. This changes the `update_table` to properly handle this case. Also changed the `load_table` to `load_table_with_version_id`, and `load_table` now just uses that. ## Are these changes tested? I have tested them using my specific setup with concurrent writers to iceberg tables in aws glue, but not generically. Writing a test case doesn't seem feasible as one needs a glue table. --------- Co-authored-by: Jem Bishop --- crates/catalog/glue/src/catalog.rs | 108 ++++++++++++++++++----------- 1 file changed, 69 insertions(+), 39 deletions(-) diff --git a/crates/catalog/glue/src/catalog.rs b/crates/catalog/glue/src/catalog.rs index 4514f2d7ab..dce287ed6e 100644 --- a/crates/catalog/glue/src/catalog.rs +++ b/crates/catalog/glue/src/catalog.rs @@ -196,6 +196,62 @@ impl GlueCatalog { pub fn file_io(&self) -> FileIO { self.file_io.clone() } + + /// Loads a table from the Glue Catalog along with its version_id for optimistic locking. + /// + /// # Returns + /// A `Result` wrapping a tuple of (`Table`, `Option`) where the String is the version_id + /// from Glue that should be used for optimistic concurrency control when updating the table. + /// + /// # Errors + /// This function may return an error in several scenarios, including: + /// - Failure to validate the namespace. + /// - Failure to retrieve the table from the Glue Catalog. + /// - Absence of metadata location information in the table's properties. + /// - Issues reading or deserializing the table's metadata file. + async fn load_table_with_version_id( + &self, + table: &TableIdent, + ) -> Result<(Table, Option)> { + let db_name = validate_namespace(table.namespace())?; + let table_name = table.name(); + + let builder = self + .client + .0 + .get_table() + .database_name(&db_name) + .name(table_name); + let builder = with_catalog_id!(builder, self.config); + + let glue_table_output = builder.send().await.map_err(from_aws_sdk_error)?; + + let glue_table = glue_table_output.table().ok_or_else(|| { + Error::new( + ErrorKind::TableNotFound, + format!( + "Table object for database: {db_name} and table: {table_name} does not exist" + ), + ) + })?; + + let version_id = glue_table.version_id.clone(); + let metadata_location = get_metadata_location(&glue_table.parameters)?; + + let metadata = TableMetadata::read_from(&self.file_io, &metadata_location).await?; + + let table = Table::builder() + .file_io(self.file_io()) + .metadata_location(metadata_location) + .metadata(metadata) + .identifier(TableIdent::new( + NamespaceIdent::new(db_name), + table_name.to_owned(), + )) + .build()?; + + Ok((table, version_id)) + } } #[async_trait] @@ -514,42 +570,8 @@ impl Catalog for GlueCatalog { /// - Absence of metadata location information in the table's properties. /// - Issues reading or deserializing the table's metadata file. async fn load_table(&self, table: &TableIdent) -> Result
Release notes

Sourced from bytes's releases.

Bytes v1.11.0

1.11.0 (November 14th, 2025)

  • Bump MSRV to 1.57 (#788)

Fixed

  • fix: BytesMut only reuse if src has remaining (#803)
  • Specialize BytesMut::put::<Bytes> (#793)
  • Reserve capacity in BytesMut::put (#794)
  • Change BytesMut::remaining_mut to use isize::MAX instead of usize::MAX (#795)

Internal changes

  • Guarantee address in slice() for empty slices. (#780)
  • Rename Vtable::to_* -> Vtable::into_* (#776)
  • Fix latest clippy warnings (#787)
  • Ignore BytesMut::freeze doctest on wasm (#790)
  • Move drop_fn of from_owner into vtable (#801)
{ - let db_name = validate_namespace(table.namespace())?; - let table_name = table.name(); - - let builder = self - .client - .0 - .get_table() - .database_name(&db_name) - .name(table_name); - let builder = with_catalog_id!(builder, self.config); - - let glue_table_output = builder.send().await.map_err(from_aws_sdk_error)?; - - match glue_table_output.table() { - None => Err(Error::new( - ErrorKind::TableNotFound, - format!( - "Table object for database: {db_name} and table: {table_name} does not exist" - ), - )), - Some(table) => { - let metadata_location = get_metadata_location(&table.parameters)?; - - let metadata = TableMetadata::read_from(&self.file_io, &metadata_location).await?; - - Table::builder() - .file_io(self.file_io()) - .metadata_location(metadata_location) - .metadata(metadata) - .identifier(TableIdent::new( - NamespaceIdent::new(db_name), - table_name.to_owned(), - )) - .build() - } - } + let (table, _) = self.load_table_with_version_id(table).await?; + Ok(table) } /// Asynchronously drops a table from the database. @@ -761,7 +783,9 @@ impl Catalog for GlueCatalog { async fn update_table(&self, commit: TableCommit) -> Result
{ let table_ident = commit.identifier().clone(); let table_namespace = validate_namespace(table_ident.namespace())?; - let current_table = self.load_table(&table_ident).await?; + + let (current_table, current_version_id) = + self.load_table_with_version_id(&table_ident).await?; let current_metadata_location = current_table.metadata_location_result()?.to_string(); let staged_table = commit.apply(current_table)?; @@ -773,8 +797,8 @@ impl Catalog for GlueCatalog { .write_to(staged_table.file_io(), staged_metadata_location) .await?; - // Persist staged table to Glue - let builder = self + // Persist staged table to Glue with optimistic locking + let mut builder = self .client .0 .update_table() @@ -787,6 +811,12 @@ impl Catalog for GlueCatalog { staged_table.metadata().properties(), Some(current_metadata_location), )?); + + // Add VersionId for optimistic locking + if let Some(version_id) = current_version_id { + builder = builder.version_id(version_id); + } + let builder = with_catalog_id!(builder, self.config); let _ = builder.send().await.map_err(|e| { let error = e.into_service_error(); From 3fe5030948796196e97786f87a97e46eab1b5cbf Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Thu, 20 Nov 2025 09:23:19 -0800 Subject: [PATCH 086/891] infra: add docs for cleaning up testpypi artifacts (#1855) ## Which issue does this PR close? - Closes #1837 ## What changes are included in this PR? Used the [`pypi-cleanup`](https://github.com/arcivanov/pypi-cleanup) utility to bulk cleanup old artifacts of pyiceberg-core from testpypi for #1837. The alternative is to do this manually in the UI for each artifact. Adding this as reminder / instruction when the nightly pipeline fails again ## Are these changes tested? Co-authored-by: Fokko Driesprong --- .github/workflows/release_python_nightly.yml | 27 ++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 9c27554f9a..5ed6b1ed5f 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -128,9 +128,36 @@ jobs: - name: List downloaded artifacts run: ls -R bindings/python/dist - name: Publish to TestPyPI + id: publish-testpypi + continue-on-error: true uses: pypa/gh-action-pypi-publish@release/v1 with: repository-url: https://test.pypi.org/legacy/ skip-existing: true packages-dir: bindings/python/dist verbose: true + - name: Display error message on publish failure + if: steps.publish-testpypi.outcome == 'failure' + run: | + echo "::error::Failed to publish to TestPyPI" + echo "" + echo "⚠️ TestPyPI Publish Failed" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "This may be due to TestPyPI storage limits." + echo "See: https://docs.pypi.org/project-management/storage-limits" + echo "" + echo "To resolve this issue, use the pypi-cleanup utility to clean up old TestPyPI artifacts:" + echo "https://pypi.org/project/pypi-cleanup/" + echo "" + echo " uvx pypi-cleanup --package pyiceberg-core --host https://test.pypi.org/ \\" + echo " --verbose -d 10 --do-it --username " + echo "" + echo "Requirements:" + echo " • Must be a maintainer for pyiceberg-core on TestPyPI" + echo " (https://test.pypi.org/project/pyiceberg-core)" + echo " • Requires TestPyPI password and 2FA" + echo " • ⚠️ ONLY do this for TestPyPI, NOT for production PyPI!" + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + exit 1 From 24932ba6cf3f72973a566e87c973e0ca92663a25 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 24 Nov 2025 14:54:20 +0800 Subject: [PATCH 087/891] chore(deps): Bump actions/checkout from 5 to 6 (#1883) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
Release notes

Sourced from actions/checkout's releases.

v6.0.0

What's Changed

Full Changelog: https://github.com/actions/checkout/compare/v5.0.0...v6.0.0

v6-beta

What's Changed

Updated persist-credentials to store the credentials under $RUNNER_TEMP instead of directly in the local git config.

This requires a minimum Actions Runner version of v2.329.0 to access the persisted credentials for Docker container action scenarios.

v5.0.1

What's Changed

Full Changelog: https://github.com/actions/checkout/compare/v5...v5.0.1

Changelog

Sourced from actions/checkout's changelog.

Changelog

V6.0.0

V5.0.1

V5.0.0

V4.3.1

V4.3.0

v4.2.2

v4.2.1

v4.2.0

v4.1.7

v4.1.6

v4.1.5

... (truncated)

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/checkout&package-manager=github_actions&previous-version=5&new-version=6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/audit.yml | 2 +- .github/workflows/bindings_python_ci.yml | 6 +++--- .github/workflows/ci.yml | 10 +++++----- .github/workflows/ci_typos.yml | 2 +- .github/workflows/publish.yml | 2 +- .github/workflows/release_python.yml | 4 ++-- .github/workflows/release_python_nightly.yml | 4 ++-- .github/workflows/website.yml | 2 +- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index 47b0c47874..28c8a3db75 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -40,7 +40,7 @@ jobs: security_audit: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder with: diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index e9eabda2cb..ed2c03b763 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -44,7 +44,7 @@ jobs: check-rust: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Check format working-directory: "bindings/python" run: cargo fmt --all -- --check @@ -55,7 +55,7 @@ jobs: check-python: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@v7 with: version: "0.9.3" @@ -81,7 +81,7 @@ jobs: - macos-latest - windows-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - uses: actions/setup-python@v6 with: python-version: 3.12 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c0f360b982..7b7e81837e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -52,7 +52,7 @@ jobs: - ubuntu-latest - macos-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder @@ -101,7 +101,7 @@ jobs: root-reserve-mb: 10240 temp-reserve-mb: 10240 - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder @@ -126,7 +126,7 @@ jobs: - macos-latest - windows-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder @@ -158,7 +158,7 @@ jobs: root-reserve-mb: 10240 temp-reserve-mb: 10240 - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder @@ -180,7 +180,7 @@ jobs: name: Verify MSRV runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Install protoc uses: arduino/setup-protoc@v3 with: diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 4c60369482..3e63a6926c 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -40,6 +40,6 @@ jobs: env: FORCE_COLOR: 1 steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Check typos uses: crate-ci/typos@v1.39.2 diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 71d35001da..634cd73519 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -45,7 +45,7 @@ jobs: - "crates/catalog/sql" - "crates/integrations/datafusion" steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup Rust toolchain uses: ./.github/actions/setup-builder diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index e6b7021c9b..9646085c7e 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -85,7 +85,7 @@ jobs: runs-on: ubuntu-latest needs: [validate-release-tag] steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Install toml-cli if: ${{ needs.validate-release-tag.outputs.is-rc == 'true' }} @@ -128,7 +128,7 @@ jobs: } - { os: ubuntu-latest, target: "armv7l" } steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Install toml-cli if: ${{ needs.validate-release-tag.outputs.is-rc == 'true' }} diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 5ed6b1ed5f..1d6faaeee9 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -43,7 +43,7 @@ jobs: if: github.repository == 'apache/iceberg-rust' # Only run for apache repo runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - uses: ./.github/actions/overwrite-package-version # Overwrite package version with timestamp with: @@ -78,7 +78,7 @@ jobs: } - { os: ubuntu-latest, target: "armv7l" } steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - uses: ./.github/actions/overwrite-package-version # Overwrite package version with timestamp with: diff --git a/.github/workflows/website.yml b/.github/workflows/website.yml index c5925da6ae..1a52482b08 100644 --- a/.github/workflows/website.yml +++ b/.github/workflows/website.yml @@ -36,7 +36,7 @@ jobs: permissions: contents: write steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v6 - name: Setup mdBook uses: peaceiris/actions-mdbook@v2 From 18a757768b38c2f2b017a0eb2cbcb03e7e79748a Mon Sep 17 00:00:00 2001 From: Najib Boutaib <104867447+N-Boutaib@users.noreply.github.com> Date: Tue, 25 Nov 2025 11:56:00 +0100 Subject: [PATCH 088/891] Update apache-avro to v0.21.0 (#1881) ## Which issue does this PR close? None ## What changes are included in this PR? Upgrades `apache-avro` from `0.20.0` to `0.21.0` in order to get rid of the deprecated `xz2` crate. ## Are these changes tested? Build & tests succeeds --- Cargo.lock | 45 +++++++++++++++++++++++++++++++++++---------- Cargo.toml | 2 +- 2 files changed, 36 insertions(+), 11 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 62478f32a0..4c82df79ff 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -169,6 +169,31 @@ dependencies = [ "zstd", ] +[[package]] +name = "apache-avro" +version = "0.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf" +dependencies = [ + "bigdecimal", + "bon", + "digest", + "log", + "miniz_oxide", + "num-bigint", + "quad-rand", + "rand 0.9.2", + "regex-lite", + "serde", + "serde_bytes", + "serde_json", + "strum 0.27.2", + "strum_macros 0.27.2", + "thiserror 2.0.17", + "uuid", + "zstd", +] + [[package]] name = "ar_archive_writer" version = "0.2.0" @@ -1925,7 +1950,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a83760d9a13122d025fbdb1d5d5aaf93dd9ada5e90ea229add92aa30898b2d1" dependencies = [ "ahash 0.8.12", - "apache-avro", + "apache-avro 0.20.0", "arrow", "arrow-ipc", "base64 0.22.1", @@ -1999,7 +2024,7 @@ version = "50.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "10d40b6953ebc9099b37adfd12fde97eb73ff0cee44355c6dea64b8a4537d561" dependencies = [ - "apache-avro", + "apache-avro 0.20.0", "arrow", "async-trait", "bytes", @@ -2642,7 +2667,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2784,7 +2809,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3496,7 +3521,7 @@ name = "iceberg" version = "0.7.0" dependencies = [ "anyhow", - "apache-avro", + "apache-avro 0.21.0", "array-init", "arrow-arith", "arrow-array", @@ -4524,7 +4549,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -5422,7 +5447,7 @@ dependencies = [ "once_cell", "socket2 0.6.1", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -5879,7 +5904,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6951,7 +6976,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -7777,7 +7802,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index c10c01d94a..7ca365dce2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -40,7 +40,7 @@ rust-version = "1.87" [workspace.dependencies] anyhow = "1.0.72" -apache-avro = { version = "0.20", features = ["zstandard"] } +apache-avro = { version = "0.21", features = ["zstandard"] } array-init = "2" arrow-arith = "56.2" arrow-array = "56.2" From 7c2d4c3abd6c5496c7ecb76854b6ee0d7295369c Mon Sep 17 00:00:00 2001 From: Jonathan Chen Date: Tue, 25 Nov 2025 20:32:33 -0500 Subject: [PATCH 089/891] docs: Clarify functionality of `SnapshotProduceOperation` (#1874) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? While refreshing myself on the internals of iceberg-rust, I felt that `SnapshotproduceOperation` should have documentation to be more clear to anybody who wants to work on Iceberg-rust ## Use of LLM Clarification I did use LLM to generate these docs however I reviewed it myself. --- crates/iceberg/src/transaction/snapshot.rs | 40 ++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index 6b3d0e4ff4..d59828ce31 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -34,13 +34,53 @@ use crate::{Error, ErrorKind, TableRequirement, TableUpdate}; const META_ROOT_PATH: &str = "metadata"; +/// A trait that defines how different table operations produce new snapshots. +/// +/// `SnapshotProduceOperation` is used by [`SnapshotProducer`] to customize snapshot creation +/// based on the type of operation being performed (e.g., `Append`, `Overwrite`, `Delete`, etc.). +/// Each operation type implements this trait to specify: +/// - Which operation type to record in the snapshot summary +/// - Which existing manifest files should be included in the new snapshot +/// - Which manifest entries should be marked as deleted +/// +/// # When it accomplishes +/// +/// This trait is used during the snapshot creation process in [`SnapshotProducer::commit()`]: +/// +/// 1. **Operation Type Recording**: The `operation()` method determines which operation type +/// (e.g., `Operation::Append`, `Operation::Overwrite`) is recorded in the snapshot summary. +/// This metadata helps track what kind of change was made to the table. +/// +/// 2. **Manifest File Selection**: The `existing_manifest()` method determines which existing +/// manifest files from the current snapshot should be carried forward to the new snapshot. +/// For example: +/// - An `Append` operation typically includes all existing manifests plus new ones +/// - An `Overwrite` operation might exclude manifests for partitions being overwritten +/// +/// 3. **Delete Entry Processing**: The `delete_entries()` method is intended for future delete +/// operations to specify which manifest entries should be marked as deleted. pub(crate) trait SnapshotProduceOperation: Send + Sync { + /// Returns the operation type that will be recorded in the snapshot summary. + /// + /// This determines what kind of operation is being performed (e.g., `Append`, `Overwrite`), + /// which is stored in the snapshot metadata for tracking and auditing purposes. fn operation(&self) -> Operation; + + /// Returns manifest entries that should be marked as deleted in the new snapshot. #[allow(unused)] fn delete_entries( &self, snapshot_produce: &SnapshotProducer, ) -> impl Future>> + Send; + + /// Returns existing manifest files that should be included in the new snapshot. + /// + /// This method determines which manifest files from the current snapshot should be + /// carried forward to the new snapshot. The selection depends on the operation type: + /// + /// - **Append operations**: Typically include all existing manifests + /// - **Overwrite operations**: May exclude manifests for partitions being overwritten + /// - **Delete operations**: May exclude manifests for partitions being deleted fn existing_manifest( &self, snapshot_produce: &SnapshotProducer<'_>, From bf984c754630cd4b536853c81528b9b69a3dab4d Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Thu, 27 Nov 2025 04:50:40 -0800 Subject: [PATCH 090/891] feat(datafusion): Split IcebergTableProvider into static and non-static table provider (#1879) --- bindings/python/Cargo.lock | 1 + .../python/src/datafusion_table_provider.rs | 6 +- .../tests/shared_tests/datafusion.rs | 4 +- crates/integrations/datafusion/src/schema.rs | 6 +- .../integrations/datafusion/src/table/mod.rs | 468 ++++++++++++++---- .../src/table/table_provider_factory.rs | 8 +- .../tests/integration_datafusion_test.rs | 12 - 7 files changed, 390 insertions(+), 115 deletions(-) diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 8249414b8d..814c9afb35 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -2313,6 +2313,7 @@ dependencies = [ "chrono", "derive_builder", "expect-test", + "flate2", "fnv", "futures", "itertools 0.13.0", diff --git a/bindings/python/src/datafusion_table_provider.rs b/bindings/python/src/datafusion_table_provider.rs index b5e1bf952e..8db7223b34 100644 --- a/bindings/python/src/datafusion_table_provider.rs +++ b/bindings/python/src/datafusion_table_provider.rs @@ -23,7 +23,7 @@ use datafusion_ffi::table_provider::FFI_TableProvider; use iceberg::TableIdent; use iceberg::io::FileIO; use iceberg::table::StaticTable; -use iceberg_datafusion::table::IcebergTableProvider; +use iceberg_datafusion::table::IcebergStaticTableProvider; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyCapsule; @@ -32,7 +32,7 @@ use crate::runtime::runtime; #[pyclass(name = "IcebergDataFusionTable")] pub struct PyIcebergDataFusionTable { - inner: Arc, + inner: Arc, } #[pymethods] @@ -69,7 +69,7 @@ impl PyIcebergDataFusionTable { let table = static_table.into_table(); - IcebergTableProvider::try_new_from_table(table) + IcebergStaticTableProvider::try_new_from_table(table) .await .map_err(|e| { PyRuntimeError::new_err(format!("Failed to create table provider: {e}")) diff --git a/crates/integration_tests/tests/shared_tests/datafusion.rs b/crates/integration_tests/tests/shared_tests/datafusion.rs index 81bbb5f54c..60dd9f36c8 100644 --- a/crates/integration_tests/tests/shared_tests/datafusion.rs +++ b/crates/integration_tests/tests/shared_tests/datafusion.rs @@ -26,7 +26,7 @@ use datafusion::error::DataFusionError; use datafusion::prelude::SessionContext; use iceberg::{Catalog, CatalogBuilder, TableIdent}; use iceberg_catalog_rest::RestCatalogBuilder; -use iceberg_datafusion::IcebergTableProvider; +use iceberg_datafusion::IcebergStaticTableProvider; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use crate::get_shared_containers; @@ -47,7 +47,7 @@ async fn test_basic_queries() -> Result<(), DataFusionError> { let ctx = SessionContext::new(); let table_provider = Arc::new( - IcebergTableProvider::try_new_from_table(table) + IcebergStaticTableProvider::try_new_from_table(table) .await .unwrap(), ); diff --git a/crates/integrations/datafusion/src/schema.rs b/crates/integrations/datafusion/src/schema.rs index 3920ee73ca..31bbdbd67f 100644 --- a/crates/integrations/datafusion/src/schema.rs +++ b/crates/integrations/datafusion/src/schema.rs @@ -28,6 +28,7 @@ use iceberg::inspect::MetadataTableType; use iceberg::{Catalog, NamespaceIdent, Result}; use crate::table::IcebergTableProvider; +use crate::to_datafusion_error; /// Represents a [`SchemaProvider`] for the Iceberg [`Catalog`], managing /// access to table providers within a specific namespace. @@ -113,7 +114,10 @@ impl SchemaProvider for IcebergSchemaProvider { let metadata_table_type = MetadataTableType::try_from(metadata_table_name).map_err(DataFusionError::Plan)?; if let Some(table) = self.tables.get(table_name) { - let metadata_table = table.metadata_table(metadata_table_type); + let metadata_table = table + .metadata_table(metadata_table_type) + .await + .map_err(to_datafusion_error)?; return Ok(Some(Arc::new(metadata_table))); } else { return Ok(None); diff --git a/crates/integrations/datafusion/src/table/mod.rs b/crates/integrations/datafusion/src/table/mod.rs index 42a3baad3b..8527668d6c 100644 --- a/crates/integrations/datafusion/src/table/mod.rs +++ b/crates/integrations/datafusion/src/table/mod.rs @@ -15,6 +15,16 @@ // specific language governing permissions and limitations // under the License. +//! Iceberg table providers for DataFusion. +//! +//! This module provides two table provider implementations: +//! +//! - [`IcebergTableProvider`]: Catalog-backed provider with automatic metadata refresh. +//! Use for write operations and when you need to see the latest table state. +//! +//! - [`IcebergStaticTableProvider`]: Static provider for read-only access to a specific +//! table snapshot. Use for consistent analytical queries or time-travel scenarios. + pub mod metadata_table; pub mod table_provider_factory; @@ -38,98 +48,61 @@ use iceberg::table::Table; use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableIdent}; use metadata_table::IcebergMetadataTableProvider; +use crate::error::to_datafusion_error; use crate::physical_plan::commit::IcebergCommitExec; use crate::physical_plan::project::project_with_partition; use crate::physical_plan::repartition::repartition; use crate::physical_plan::scan::IcebergTableScan; use crate::physical_plan::write::IcebergWriteExec; -/// Represents a [`TableProvider`] for the Iceberg [`Catalog`], -/// managing access to a [`Table`]. +/// Catalog-backed table provider with automatic metadata refresh. +/// +/// This provider loads fresh table metadata from the catalog on every scan and write +/// operation, ensuring you always see the latest table state. Use this when you need +/// write operations or want to see the most up-to-date data. +/// +/// For read-only access to a specific snapshot without catalog overhead, use +/// [`IcebergStaticTableProvider`] instead. #[derive(Debug, Clone)] pub struct IcebergTableProvider { - /// A table in the catalog. - table: Table, - /// Table snapshot id that will be queried via this provider. - snapshot_id: Option, - /// A reference-counted arrow `Schema`. + /// The catalog that manages this table + catalog: Arc, + /// The table identifier (namespace + name) + table_ident: TableIdent, + /// A reference-counted arrow `Schema` (cached at construction) schema: ArrowSchemaRef, - /// The catalog that the table belongs to. - catalog: Option>, } impl IcebergTableProvider { - pub(crate) fn new(table: Table, schema: ArrowSchemaRef) -> Self { - IcebergTableProvider { - table, - snapshot_id: None, - schema, - catalog: None, - } - } - /// Asynchronously tries to construct a new [`IcebergTableProvider`] - /// using the given client and table name to fetch an actual [`Table`] - /// in the provided namespace. + /// Creates a new catalog-backed table provider. + /// + /// Loads the table once to get the initial schema, then stores the catalog + /// reference for future metadata refreshes on each operation. pub(crate) async fn try_new( - client: Arc, + catalog: Arc, namespace: NamespaceIdent, name: impl Into, ) -> Result { - let ident = TableIdent::new(namespace, name.into()); - let table = client.load_table(&ident).await?; + let table_ident = TableIdent::new(namespace, name.into()); + // Load table once to get initial schema + let table = catalog.load_table(&table_ident).await?; let schema = Arc::new(schema_to_arrow_schema(table.metadata().current_schema())?); Ok(IcebergTableProvider { - table, - snapshot_id: None, - schema, - catalog: Some(client), - }) - } - - /// Asynchronously tries to construct a new [`IcebergTableProvider`] - /// using the given table. Can be used to create a table provider from an existing table regardless of the catalog implementation. - pub async fn try_new_from_table(table: Table) -> Result { - let schema = Arc::new(schema_to_arrow_schema(table.metadata().current_schema())?); - Ok(IcebergTableProvider { - table, - snapshot_id: None, - schema, - catalog: None, - }) - } - - /// Asynchronously tries to construct a new [`IcebergTableProvider`] - /// using a specific snapshot of the given table. Can be used to create a table provider from an existing table regardless of the catalog implementation. - pub async fn try_new_from_table_snapshot(table: Table, snapshot_id: i64) -> Result { - let snapshot = table - .metadata() - .snapshot_by_id(snapshot_id) - .ok_or_else(|| { - Error::new( - ErrorKind::Unexpected, - format!( - "snapshot id {snapshot_id} not found in table {}", - table.identifier().name() - ), - ) - })?; - let schema = snapshot.schema(table.metadata())?; - let schema = Arc::new(schema_to_arrow_schema(&schema)?); - Ok(IcebergTableProvider { - table, - snapshot_id: Some(snapshot_id), + catalog, + table_ident, schema, - catalog: None, }) } - pub(crate) fn metadata_table(&self, r#type: MetadataTableType) -> IcebergMetadataTableProvider { - IcebergMetadataTableProvider { - table: self.table.clone(), - r#type, - } + pub(crate) async fn metadata_table( + &self, + r#type: MetadataTableType, + ) -> Result { + // Load fresh table metadata for metadata table access + let table = self.catalog.load_table(&self.table_ident).await?; + Ok(IcebergMetadataTableProvider { table, r#type }) } } @@ -154,9 +127,17 @@ impl TableProvider for IcebergTableProvider { filters: &[Expr], _limit: Option, ) -> DFResult> { + // Load fresh table metadata from catalog + let table = self + .catalog + .load_table(&self.table_ident) + .await + .map_err(to_datafusion_error)?; + + // Create scan with fresh metadata (always use current snapshot) Ok(Arc::new(IcebergTableScan::new( - self.table.clone(), - self.snapshot_id, + table, + None, // Always use current snapshot for catalog-backed provider self.schema.clone(), projection, filters, @@ -177,17 +158,18 @@ impl TableProvider for IcebergTableProvider { input: Arc, _insert_op: InsertOp, ) -> DFResult> { - let Some(catalog) = self.catalog.clone() else { - return Err(DataFusionError::Execution( - "Catalog cannot be none for insert_into".to_string(), - )); - }; + // Load fresh table metadata from catalog + let table = self + .catalog + .load_table(&self.table_ident) + .await + .map_err(to_datafusion_error)?; - let partition_spec = self.table.metadata().default_partition_spec(); + let partition_spec = table.metadata().default_partition_spec(); // Step 1: Project partition values for partitioned tables let plan_with_partition = if !partition_spec.is_unpartitioned() { - project_with_partition(input, &self.table)? + project_with_partition(input, &table)? } else { input }; @@ -200,14 +182,11 @@ impl TableProvider for IcebergTableProvider { ) })?; - let repartitioned_plan = repartition( - plan_with_partition, - self.table.metadata_ref(), - target_partitions, - )?; + let repartitioned_plan = + repartition(plan_with_partition, table.metadata_ref(), target_partitions)?; let write_plan = Arc::new(IcebergWriteExec::new( - self.table.clone(), + table.clone(), repartitioned_plan, self.schema.clone(), )); @@ -216,21 +195,139 @@ impl TableProvider for IcebergTableProvider { let coalesce_partitions = Arc::new(CoalescePartitionsExec::new(write_plan)); Ok(Arc::new(IcebergCommitExec::new( - self.table.clone(), - catalog, + table, + self.catalog.clone(), coalesce_partitions, self.schema.clone(), ))) } } +/// Static table provider for read-only snapshot access. +/// +/// This provider holds a cached table instance and does not refresh metadata or support +/// write operations. Use this for consistent analytical queries, time-travel scenarios, +/// or when you want to avoid catalog overhead. +/// +/// For catalog-backed tables with write support and automatic refresh, use +/// [`IcebergTableProvider`] instead. +#[derive(Debug, Clone)] +pub struct IcebergStaticTableProvider { + /// The static table instance (never refreshed) + table: Table, + /// Optional snapshot ID for this static view + snapshot_id: Option, + /// A reference-counted arrow `Schema` + schema: ArrowSchemaRef, +} + +impl IcebergStaticTableProvider { + /// Creates a static provider from a table instance. + /// + /// Uses the table's current snapshot for all queries. Does not support write operations. + pub async fn try_new_from_table(table: Table) -> Result { + let schema = Arc::new(schema_to_arrow_schema(table.metadata().current_schema())?); + Ok(IcebergStaticTableProvider { + table, + snapshot_id: None, + schema, + }) + } + + /// Creates a static provider for a specific table snapshot. + /// + /// Queries the specified snapshot for all operations. Useful for time-travel queries. + /// Does not support write operations. + pub async fn try_new_from_table_snapshot(table: Table, snapshot_id: i64) -> Result { + let snapshot = table + .metadata() + .snapshot_by_id(snapshot_id) + .ok_or_else(|| { + Error::new( + ErrorKind::Unexpected, + format!( + "snapshot id {snapshot_id} not found in table {}", + table.identifier().name() + ), + ) + })?; + let table_schema = snapshot.schema(table.metadata())?; + let schema = Arc::new(schema_to_arrow_schema(&table_schema)?); + Ok(IcebergStaticTableProvider { + table, + snapshot_id: Some(snapshot_id), + schema, + }) + } +} + +#[async_trait] +impl TableProvider for IcebergStaticTableProvider { + fn as_any(&self) -> &dyn Any { + self + } + + fn schema(&self) -> ArrowSchemaRef { + self.schema.clone() + } + + fn table_type(&self) -> TableType { + TableType::Base + } + + async fn scan( + &self, + _state: &dyn Session, + projection: Option<&Vec>, + filters: &[Expr], + _limit: Option, + ) -> DFResult> { + // Use cached table (no refresh) + Ok(Arc::new(IcebergTableScan::new( + self.table.clone(), + self.snapshot_id, + self.schema.clone(), + projection, + filters, + ))) + } + + fn supports_filters_pushdown( + &self, + filters: &[&Expr], + ) -> DFResult> { + // Push down all filters, as a single source of truth, the scanner will drop the filters which couldn't be push down + Ok(vec![TableProviderFilterPushDown::Inexact; filters.len()]) + } + + async fn insert_into( + &self, + _state: &dyn Session, + _input: Arc, + _insert_op: InsertOp, + ) -> DFResult> { + Err(to_datafusion_error(Error::new( + ErrorKind::FeatureUnsupported, + "Write operations are not supported on IcebergStaticTableProvider. \ + Use IcebergTableProvider with a catalog for write support." + .to_string(), + ))) + } +} + #[cfg(test)] mod tests { + use std::collections::HashMap; + use std::sync::Arc; + use datafusion::common::Column; use datafusion::prelude::SessionContext; - use iceberg::TableIdent; use iceberg::io::FileIO; + use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; + use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; use iceberg::table::{StaticTable, Table}; + use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation, TableIdent}; + use tempfile::TempDir; use super::*; @@ -253,10 +350,59 @@ mod tests { static_table.into_table() } + async fn get_test_catalog_and_table() -> (Arc, NamespaceIdent, String, TempDir) { + let temp_dir = TempDir::new().unwrap(); + let warehouse_path = temp_dir.path().to_str().unwrap().to_string(); + + let catalog = MemoryCatalogBuilder::default() + .load( + "memory", + HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), warehouse_path.clone())]), + ) + .await + .unwrap(); + + let namespace = NamespaceIdent::new("test_ns".to_string()); + catalog + .create_namespace(&namespace, HashMap::new()) + .await + .unwrap(); + + let schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let table_creation = TableCreation::builder() + .name("test_table".to_string()) + .location(format!("{}/test_table", warehouse_path)) + .schema(schema) + .properties(HashMap::new()) + .build(); + + catalog + .create_table(&namespace, table_creation) + .await + .unwrap(); + + ( + Arc::new(catalog), + namespace, + "test_table".to_string(), + temp_dir, + ) + } + + // Tests for IcebergStaticTableProvider + #[tokio::test] - async fn test_try_new_from_table() { + async fn test_static_provider_from_table() { let table = get_test_table_from_metadata_file().await; - let table_provider = IcebergTableProvider::try_new_from_table(table.clone()) + let table_provider = IcebergStaticTableProvider::try_new_from_table(table.clone()) .await .unwrap(); let ctx = SessionContext::new(); @@ -278,11 +424,11 @@ mod tests { } #[tokio::test] - async fn test_try_new_from_table_snapshot() { + async fn test_static_provider_from_snapshot() { let table = get_test_table_from_metadata_file().await; let snapshot_id = table.metadata().snapshots().next().unwrap().snapshot_id(); let table_provider = - IcebergTableProvider::try_new_from_table_snapshot(table.clone(), snapshot_id) + IcebergStaticTableProvider::try_new_from_table_snapshot(table.clone(), snapshot_id) .await .unwrap(); let ctx = SessionContext::new(); @@ -304,16 +450,152 @@ mod tests { } #[tokio::test] - async fn test_physical_input_schema_consistent_with_logical_input_schema() { + async fn test_static_provider_rejects_writes() { + let table = get_test_table_from_metadata_file().await; + let table_provider = IcebergStaticTableProvider::try_new_from_table(table.clone()) + .await + .unwrap(); + let ctx = SessionContext::new(); + ctx.register_table("mytable", Arc::new(table_provider)) + .unwrap(); + + // Attempt to insert into the static provider should fail + let result = ctx.sql("INSERT INTO mytable VALUES (1, 2, 3)").await; + + // The error should occur during planning or execution + // We expect an error indicating write operations are not supported + assert!( + result.is_err() || { + let df = result.unwrap(); + df.collect().await.is_err() + } + ); + } + + #[tokio::test] + async fn test_static_provider_scan() { let table = get_test_table_from_metadata_file().await; - let table_provider = IcebergTableProvider::try_new_from_table(table.clone()) + let table_provider = IcebergStaticTableProvider::try_new_from_table(table.clone()) .await .unwrap(); let ctx = SessionContext::new(); ctx.register_table("mytable", Arc::new(table_provider)) .unwrap(); + + // Test that scan operations work correctly let df = ctx.sql("SELECT count(*) FROM mytable").await.unwrap(); let physical_plan = df.create_physical_plan().await; - assert!(physical_plan.is_ok()) + assert!(physical_plan.is_ok()); + } + + // Tests for IcebergTableProvider + + #[tokio::test] + async fn test_catalog_backed_provider_creation() { + let (catalog, namespace, table_name, _temp_dir) = get_test_catalog_and_table().await; + + // Test creating a catalog-backed provider + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + // Verify the schema is loaded correctly + let schema = provider.schema(); + assert_eq!(schema.fields().len(), 2); + assert_eq!(schema.field(0).name(), "id"); + assert_eq!(schema.field(1).name(), "name"); + } + + #[tokio::test] + async fn test_catalog_backed_provider_scan() { + let (catalog, namespace, table_name, _temp_dir) = get_test_catalog_and_table().await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + ctx.register_table("test_table", Arc::new(provider)) + .unwrap(); + + // Test that scan operations work correctly + let df = ctx.sql("SELECT * FROM test_table").await.unwrap(); + + // Verify the schema in the query result + let df_schema = df.schema(); + assert_eq!(df_schema.fields().len(), 2); + assert_eq!(df_schema.field(0).name(), "id"); + assert_eq!(df_schema.field(1).name(), "name"); + + let physical_plan = df.create_physical_plan().await; + assert!(physical_plan.is_ok()); + } + + #[tokio::test] + async fn test_catalog_backed_provider_insert() { + let (catalog, namespace, table_name, _temp_dir) = get_test_catalog_and_table().await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + ctx.register_table("test_table", Arc::new(provider)) + .unwrap(); + + // Test that insert operations work correctly + let result = ctx.sql("INSERT INTO test_table VALUES (1, 'test')").await; + + // Insert should succeed (or at least not fail during planning) + assert!(result.is_ok()); + + // Try to execute the insert plan + let df = result.unwrap(); + let execution_result = df.collect().await; + + // The execution should succeed + assert!(execution_result.is_ok()); + } + + #[tokio::test] + async fn test_physical_input_schema_consistent_with_logical_input_schema() { + let (catalog, namespace, table_name, _temp_dir) = get_test_catalog_and_table().await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + ctx.register_table("test_table", Arc::new(provider)) + .unwrap(); + + // Create a query plan + let df = ctx.sql("SELECT id, name FROM test_table").await.unwrap(); + + // Get logical schema before consuming df + let logical_schema = df.schema().clone(); + + // Get physical plan (this consumes df) + let physical_plan = df.create_physical_plan().await.unwrap(); + let physical_schema = physical_plan.schema(); + + // Verify that logical and physical schemas are consistent + assert_eq!( + logical_schema.fields().len(), + physical_schema.fields().len() + ); + + for (logical_field, physical_field) in logical_schema + .fields() + .iter() + .zip(physical_schema.fields().iter()) + { + assert_eq!(logical_field.name(), physical_field.name()); + assert_eq!(logical_field.data_type(), physical_field.data_type()); + } } } diff --git a/crates/integrations/datafusion/src/table/table_provider_factory.rs b/crates/integrations/datafusion/src/table/table_provider_factory.rs index e8e87dd318..8c0c8e90de 100644 --- a/crates/integrations/datafusion/src/table/table_provider_factory.rs +++ b/crates/integrations/datafusion/src/table/table_provider_factory.rs @@ -24,12 +24,11 @@ use datafusion::catalog::{Session, TableProvider, TableProviderFactory}; use datafusion::error::Result as DFResult; use datafusion::logical_expr::CreateExternalTable; use datafusion::sql::TableReference; -use iceberg::arrow::schema_to_arrow_schema; use iceberg::io::FileIO; use iceberg::table::StaticTable; use iceberg::{Error, ErrorKind, Result, TableIdent}; -use super::IcebergTableProvider; +use super::IcebergStaticTableProvider; use crate::to_datafusion_error; /// A factory that implements DataFusion's `TableProviderFactory` to create `IcebergTableProvider` instances. @@ -126,10 +125,11 @@ impl TableProviderFactory for IcebergTableProviderFactory { .map_err(to_datafusion_error)? .into_table(); - let schema = schema_to_arrow_schema(table.metadata().current_schema()) + let provider = IcebergStaticTableProvider::try_new_from_table(table) + .await .map_err(to_datafusion_error)?; - Ok(Arc::new(IcebergTableProvider::new(table, Arc::new(schema)))) + Ok(Arc::new(provider)) } } diff --git a/crates/integrations/datafusion/tests/integration_datafusion_test.rs b/crates/integrations/datafusion/tests/integration_datafusion_test.rs index fdf5b17d18..3ad84f383e 100644 --- a/crates/integrations/datafusion/tests/integration_datafusion_test.rs +++ b/crates/integrations/datafusion/tests/integration_datafusion_test.rs @@ -492,10 +492,6 @@ async fn test_insert_into() -> Result<()> { .unwrap(); assert_eq!(rows_inserted.value(0), 2); - // Refresh context to avoid getting stale table - let catalog = Arc::new(IcebergCatalogProvider::try_new(client).await?); - ctx.register_catalog("catalog", catalog); - // Query the table to verify the inserted data let df = ctx .sql("SELECT * FROM catalog.test_insert_into.my_table") @@ -650,10 +646,6 @@ async fn test_insert_into_nested() -> Result<()> { .unwrap(); assert_eq!(rows_inserted.value(0), 2); - // Refresh context to avoid getting stale table - let catalog = Arc::new(IcebergCatalogProvider::try_new(client).await?); - ctx.register_catalog("catalog", catalog); - // Query the table to verify the inserted data let df = ctx .sql("SELECT * FROM catalog.test_insert_nested.nested_table ORDER BY id") @@ -880,10 +872,6 @@ async fn test_insert_into_partitioned() -> Result<()> { .unwrap(); assert_eq!(rows_inserted.value(0), 5); - // Refresh catalog to get updated table - let catalog = Arc::new(IcebergCatalogProvider::try_new(client.clone()).await?); - ctx.register_catalog("catalog", catalog); - // Query the table to verify data let df = ctx .sql("SELECT * FROM catalog.test_partitioned_write.partitioned_table ORDER BY id") From 617be094eed3adf04b77b36225bbda15a33527d9 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Fri, 28 Nov 2025 19:36:44 -0800 Subject: [PATCH 091/891] infra: use new `del_branch_on_merge` in .asf.yaml (#1888) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Previous use of `del_branch_on_merge` ``` github: del_branch_on_merge: true ``` is deprecated, https://github.com/apache/infrastructure-asfyaml?tab=readme-ov-file#delete-branch-on-merge New way is to ``` github: pull_requests: del_branch_on_merge: true ``` https://github.com/apache/infrastructure-asfyaml?tab=readme-ov-file#pull_requests ## Are these changes tested? --- .asf.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.asf.yaml b/.asf.yaml index f46c437a79..b217fd7807 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -43,7 +43,9 @@ github: dismiss_stale_reviews: false required_linear_history: true - del_branch_on_merge: true + pull_requests: + # auto-delete head branches after being merged + del_branch_on_merge: true features: wiki: false issues: true From 2871b4760a42e5d8d4261c025793f6cf02bd1628 Mon Sep 17 00:00:00 2001 From: dentiny Date: Sun, 30 Nov 2025 22:49:53 -0800 Subject: [PATCH 092/891] Upgrade opendal to v0.55 (#1895) ## What changes are included in this PR? Dependency upgrade is supposed to perform automatically via dependabot, but somehow I didn't see it this weekend. ## Are these changes tested? Checked with local iceberg unit tests and seem fine. --- Cargo.lock | 24 +++++++++++++++++++++--- Cargo.toml | 4 ++-- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 4c82df79ff..d29f6e97f4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4015,10 +4015,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be1f93b8b1eb69c77f24bbb0afdf66f54b632ee39af40ca21c4365a1d7347e49" dependencies = [ "jiff-static", + "jiff-tzdb-platform", "log", "portable-atomic", "portable-atomic-util", "serde", + "windows-sys 0.59.0", ] [[package]] @@ -4032,6 +4034,21 @@ dependencies = [ "syn 2.0.108", ] +[[package]] +name = "jiff-tzdb" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c1283705eb0a21404d2bfd6eef2a7593d240bc42a0bdb39db0ad6fa2ec026524" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -4742,20 +4759,20 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "opendal" -version = "0.54.1" +version = "0.55.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42afda58fa2cf50914402d132cc1caacff116a85d10c72ab2082bb7c50021754" +checksum = "d075ab8a203a6ab4bc1bce0a4b9fe486a72bf8b939037f4b78d95386384bc80a" dependencies = [ "anyhow", "backon", "base64 0.22.1", "bytes", - "chrono", "crc32c", "futures", "getrandom 0.2.16", "http 1.3.1", "http-body 1.0.1", + "jiff", "log", "md-5", "percent-encoding", @@ -4765,6 +4782,7 @@ dependencies = [ "serde", "serde_json", "tokio", + "url", "uuid", ] diff --git a/Cargo.toml b/Cargo.toml index 7ca365dce2..0512155534 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -81,9 +81,9 @@ http = "1.2" iceberg = { version = "0.7.0", path = "./crates/iceberg" } iceberg-catalog-glue = { version = "0.7.0", path = "./crates/catalog/glue" } iceberg-catalog-hms = { version = "0.7.0", path = "./crates/catalog/hms" } -iceberg-catalog-sql = { version = "0.7.0", path = "./crates/catalog/sql" } iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } +iceberg-catalog-sql = { version = "0.7.0", path = "./crates/catalog/sql" } iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } indicatif = "0.18" itertools = "0.13" @@ -99,7 +99,7 @@ motore-macros = "0.4.3" murmur3 = "0.5.2" num-bigint = "0.4.6" once_cell = "1.20" -opendal = "0.54.0" +opendal = "0.55.0" ordered-float = "4" parquet = "56.2" pilota = "0.11.10" From e20910ed71fa63d4b9459a29add8698656530160 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 Dec 2025 16:51:05 +0800 Subject: [PATCH 093/891] chore(deps): Bump http from 1.3.1 to 1.4.0 (#1892) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [http](https://github.com/hyperium/http) from 1.3.1 to 1.4.0.
Release notes

Sourced from http's releases.

v1.4.0

Highlights

  • Add StatusCode::EARLY_HINTS constant for 103 Early Hints.
  • Make StatusCode::from_u16 now a const fn.
  • Make Authority::from_static now a const fn.
  • Make PathAndQuery::from_static now a const fn.
  • MSRV increased to 1.57 (allows legible const fn panic messages).

What's Changed

New Contributors

Full Changelog: https://github.com/hyperium/http/compare/v1.3.1...v1.4.0

Changelog

Sourced from http's changelog.

1.4.0 (November 24, 2025)

  • Add StatusCode::EARLY_HINTS constant for 103 Early Hints.
  • Make StatusCode::from_u16 now a const fn.
  • Make Authority::from_static now a const fn.
  • Make PathAndQuery::from_static now a const fn.
  • MSRV increased to 1.57 (allows legible const fn panic messages).
Commits
  • b9625d8 v1.4.0
  • 50b009c refactor(header): inline FNV hasher to reduce dependencies (#796)
  • b370d36 feat(uri): make Authority/PathAndQuery::from_static const (#786)
  • 0d74251 chore(ci): update to actions/checkout@v5 (#800)
  • a760767 docs: remove unnecessary extern crate sentence (#799)
  • fb1d457 refactor(header): use better panic message in const HeaderName and HeaderValu...
  • 20dbd6e feat(status): Add 103 EARLY_HINTS status code (#758)
  • e7a7337 chore: bump MSRV to 1.57
  • 1888e28 tests: downgrade rand back to 0.8 for now
  • 918bbc3 chore: minor improvement for docs (#790)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=http&package-manager=cargo&previous-version=1.3.1&new-version=1.4.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 45 ++++++++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d29f6e97f4..736f2b1461 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -665,7 +665,7 @@ dependencies = [ "bytes", "fastrand", "hex", - "http 1.3.1", + "http 1.4.0", "ring", "time", "tokio", @@ -859,7 +859,7 @@ dependencies = [ "hex", "hmac", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "percent-encoding", "sha2", "time", @@ -889,7 +889,7 @@ dependencies = [ "bytes-utils", "futures-core", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "http-body 0.4.6", "percent-encoding", "pin-project-lite", @@ -909,7 +909,7 @@ dependencies = [ "h2 0.3.27", "h2 0.4.12", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "http-body 0.4.6", "hyper 0.14.32", "hyper 1.7.0", @@ -970,7 +970,7 @@ dependencies = [ "bytes", "fastrand", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "http-body 0.4.6", "http-body 1.0.1", "pin-project-lite", @@ -989,7 +989,7 @@ dependencies = [ "aws-smithy-types", "bytes", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "pin-project-lite", "tokio", "tracing", @@ -1007,7 +1007,7 @@ dependencies = [ "bytes-utils", "futures-core", "http 0.2.12", - "http 1.3.1", + "http 1.4.0", "http-body 0.4.6", "http-body 1.0.1", "http-body-util", @@ -3191,7 +3191,7 @@ dependencies = [ "fnv", "futures-core", "futures-sink", - "http 1.3.1", + "http 1.4.0", "indexmap 2.12.0", "slab", "tokio", @@ -3326,12 +3326,11 @@ dependencies = [ [[package]] name = "http" -version = "1.3.1" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4a85d31aea989eead29a3aaf9e1115a180df8282431156e533de47660892565" +checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" dependencies = [ "bytes", - "fnv", "itoa", ] @@ -3353,7 +3352,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" dependencies = [ "bytes", - "http 1.3.1", + "http 1.4.0", ] [[package]] @@ -3364,7 +3363,7 @@ checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" dependencies = [ "bytes", "futures-core", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "pin-project-lite", ] @@ -3422,7 +3421,7 @@ dependencies = [ "futures-channel", "futures-core", "h2 0.4.12", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "httparse", "httpdate", @@ -3456,7 +3455,7 @@ version = "0.27.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" dependencies = [ - "http 1.3.1", + "http 1.4.0", "hyper 1.7.0", "hyper-util", "rustls 0.23.34", @@ -3479,7 +3478,7 @@ dependencies = [ "futures-channel", "futures-core", "futures-util", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "hyper 1.7.0", "ipnet", @@ -3651,7 +3650,7 @@ dependencies = [ "async-trait", "chrono", "ctor", - "http 1.3.1", + "http 1.4.0", "iceberg", "iceberg_test_utils", "itertools 0.13.0", @@ -4420,7 +4419,7 @@ dependencies = [ "bytes", "colored", "futures-util", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "http-body-util", "hyper 1.7.0", @@ -4720,7 +4719,7 @@ dependencies = [ "chrono", "form_urlencoded", "futures", - "http 1.3.1", + "http 1.4.0", "http-body-util", "humantime", "hyper 1.7.0", @@ -4770,7 +4769,7 @@ dependencies = [ "crc32c", "futures", "getrandom 0.2.16", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "jiff", "log", @@ -5704,7 +5703,7 @@ dependencies = [ "hex", "hmac", "home", - "http 1.3.1", + "http 1.4.0", "jsonwebtoken", "log", "once_cell", @@ -5732,7 +5731,7 @@ dependencies = [ "futures-core", "futures-util", "h2 0.4.12", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "http-body-util", "hyper 1.7.0", @@ -7306,7 +7305,7 @@ dependencies = [ "bitflags", "bytes", "futures-util", - "http 1.3.1", + "http 1.4.0", "http-body 1.0.1", "iri-string", "pin-project-lite", From a371d82dd537bb266b52f6daa4b552f3c6f28b41 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 1 Dec 2025 19:15:27 +0800 Subject: [PATCH 094/891] chore(deps): Bump crate-ci/typos from 1.39.2 to 1.40.0 (#1891) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.39.2 to 1.40.0.
Release notes

Sourced from crate-ci/typos's releases.

v1.40.0

[1.40.0] - 2025-11-26

Features

Changelog

Sourced from crate-ci/typos's changelog.

[1.40.0] - 2025-11-26

Features

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.39.2&new-version=1.40.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 3e63a6926c..6406b82142 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.39.2 + uses: crate-ci/typos@v1.40.0 From 5724fc556ed8699dfdba5fb657ea5dd9a733cbf1 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Mon, 1 Dec 2025 03:42:10 -0800 Subject: [PATCH 095/891] feat(datafusion): Add `sort_by_partition` to sort the input partitioned data (#1618) --- .../datafusion/src/physical_plan/mod.rs | 1 + .../datafusion/src/physical_plan/sort.rs | 244 ++++++++++++++++++ 2 files changed, 245 insertions(+) create mode 100644 crates/integrations/datafusion/src/physical_plan/sort.rs diff --git a/crates/integrations/datafusion/src/physical_plan/mod.rs b/crates/integrations/datafusion/src/physical_plan/mod.rs index eb58082fe5..5a9845cde0 100644 --- a/crates/integrations/datafusion/src/physical_plan/mod.rs +++ b/crates/integrations/datafusion/src/physical_plan/mod.rs @@ -21,6 +21,7 @@ pub(crate) mod metadata_scan; pub(crate) mod project; pub(crate) mod repartition; pub(crate) mod scan; +pub(crate) mod sort; pub(crate) mod write; pub(crate) const DATA_FILES_COL_NAME: &str = "data_files"; diff --git a/crates/integrations/datafusion/src/physical_plan/sort.rs b/crates/integrations/datafusion/src/physical_plan/sort.rs new file mode 100644 index 0000000000..2a57e16e43 --- /dev/null +++ b/crates/integrations/datafusion/src/physical_plan/sort.rs @@ -0,0 +1,244 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Partition-based sorting for Iceberg tables. + +use std::sync::Arc; + +use datafusion::arrow::compute::SortOptions; +use datafusion::common::Result as DFResult; +use datafusion::error::DataFusionError; +use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr}; +use datafusion::physical_plan::ExecutionPlan; +use datafusion::physical_plan::expressions::Column; +use datafusion::physical_plan::sorts::sort::SortExec; +use iceberg::arrow::PROJECTED_PARTITION_VALUE_COLUMN; + +/// Sorts an ExecutionPlan by partition values for Iceberg tables. +/// +/// This function takes an input ExecutionPlan that has been extended with partition values +/// (via `project_with_partition`) and returns a SortExec that sorts by the partition column. +/// The partition values are expected to be in a struct column named `PROJECTED_PARTITION_VALUE_COLUMN`. +/// +/// For unpartitioned tables or plans without the partition column, returns an error. +/// +/// # Arguments +/// * `input` - The input ExecutionPlan with projected partition values +/// +/// # Returns +/// * `Ok(Arc)` - A SortExec that sorts by partition values +/// * `Err` - If the partition column is not found +/// +/// TODO remove dead_code mark when integrating with insert_into +#[allow(dead_code)] +pub(crate) fn sort_by_partition(input: Arc) -> DFResult> { + let schema = input.schema(); + + // Find the partition column in the schema + let (partition_column_index, _partition_field) = schema + .column_with_name(PROJECTED_PARTITION_VALUE_COLUMN) + .ok_or_else(|| { + DataFusionError::Plan(format!( + "Partition column '{}' not found in schema. Ensure the plan has been extended with partition values using project_with_partition.", + PROJECTED_PARTITION_VALUE_COLUMN + )) + })?; + + // Create a single sort expression for the partition column + let column_expr = Arc::new(Column::new( + PROJECTED_PARTITION_VALUE_COLUMN, + partition_column_index, + )); + + let sort_expr = PhysicalSortExpr { + expr: column_expr, + options: SortOptions::default(), // Ascending, nulls last + }; + + // Create a SortExec with preserve_partitioning=true to ensure the output partitioning + // is the same as the input partitioning, and the data is sorted within each partition + let lex_ordering = LexOrdering::new(vec![sort_expr]).ok_or_else(|| { + DataFusionError::Plan("Failed to create LexOrdering from sort expression".to_string()) + })?; + + let sort_exec = SortExec::new(lex_ordering, input).with_preserve_partitioning(true); + + Ok(Arc::new(sort_exec)) +} + +#[cfg(test)] +mod tests { + use datafusion::arrow::array::{Int32Array, RecordBatch, StringArray, StructArray}; + use datafusion::arrow::datatypes::{DataType, Field, Fields, Schema as ArrowSchema}; + use datafusion::datasource::{MemTable, TableProvider}; + use datafusion::prelude::SessionContext; + + use super::*; + + #[tokio::test] + async fn test_sort_by_partition_basic() { + // Create a schema with a partition column + let partition_fields = + Fields::from(vec![Field::new("id_partition", DataType::Int32, false)]); + + let schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + Field::new( + PROJECTED_PARTITION_VALUE_COLUMN, + DataType::Struct(partition_fields.clone()), + false, + ), + ])); + + // Create test data with partition values + let id_array = Arc::new(Int32Array::from(vec![3, 1, 2])); + let name_array = Arc::new(StringArray::from(vec!["c", "a", "b"])); + let partition_array = Arc::new(StructArray::from(vec![( + Arc::new(Field::new("id_partition", DataType::Int32, false)), + Arc::new(Int32Array::from(vec![3, 1, 2])) as _, + )])); + + let batch = + RecordBatch::try_new(schema.clone(), vec![id_array, name_array, partition_array]) + .unwrap(); + + let ctx = SessionContext::new(); + let mem_table = MemTable::try_new(schema.clone(), vec![vec![batch]]).unwrap(); + let input = mem_table.scan(&ctx.state(), None, &[], None).await.unwrap(); + + // Apply sort + let sorted_plan = sort_by_partition(input).unwrap(); + + // Execute and verify + let result = datafusion::physical_plan::collect(sorted_plan, ctx.task_ctx()) + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let result_batch = &result[0]; + + let id_col = result_batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + + // Verify data is sorted by partition value + assert_eq!(id_col.value(0), 1); + assert_eq!(id_col.value(1), 2); + assert_eq!(id_col.value(2), 3); + } + + #[tokio::test] + async fn test_sort_by_partition_missing_column() { + let schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let batch = RecordBatch::try_new(schema.clone(), vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ]) + .unwrap(); + + let ctx = SessionContext::new(); + let mem_table = MemTable::try_new(schema.clone(), vec![vec![batch]]).unwrap(); + let input = mem_table.scan(&ctx.state(), None, &[], None).await.unwrap(); + + let result = sort_by_partition(input); + assert!(result.is_err()); + assert!( + result + .unwrap_err() + .to_string() + .contains("Partition column '_partition' not found") + ); + } + + #[tokio::test] + async fn test_sort_by_partition_multi_field() { + // Test with multiple partition fields in the struct + let partition_fields = Fields::from(vec![ + Field::new("year", DataType::Int32, false), + Field::new("month", DataType::Int32, false), + ]); + + let schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("data", DataType::Utf8, false), + Field::new( + PROJECTED_PARTITION_VALUE_COLUMN, + DataType::Struct(partition_fields.clone()), + false, + ), + ])); + + // Create test data with partition values (year, month) + let id_array = Arc::new(Int32Array::from(vec![1, 2, 3, 4])); + let data_array = Arc::new(StringArray::from(vec!["a", "b", "c", "d"])); + + // Partition values: (2024, 2), (2024, 1), (2023, 12), (2024, 1) + let year_array = Arc::new(Int32Array::from(vec![2024, 2024, 2023, 2024])); + let month_array = Arc::new(Int32Array::from(vec![2, 1, 12, 1])); + + let partition_array = Arc::new(StructArray::from(vec![ + ( + Arc::new(Field::new("year", DataType::Int32, false)), + year_array as _, + ), + ( + Arc::new(Field::new("month", DataType::Int32, false)), + month_array as _, + ), + ])); + + let batch = + RecordBatch::try_new(schema.clone(), vec![id_array, data_array, partition_array]) + .unwrap(); + + let ctx = SessionContext::new(); + let mem_table = MemTable::try_new(schema.clone(), vec![vec![batch]]).unwrap(); + let input = mem_table.scan(&ctx.state(), None, &[], None).await.unwrap(); + + // Apply sort + let sorted_plan = sort_by_partition(input).unwrap(); + + // Execute and verify + let result = datafusion::physical_plan::collect(sorted_plan, ctx.task_ctx()) + .await + .unwrap(); + + assert_eq!(result.len(), 1); + let result_batch = &result[0]; + + let id_col = result_batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + + // Verify data is sorted by partition value (struct comparison) + // Expected order: (2023, 12), (2024, 1), (2024, 1), (2024, 2) + // Which corresponds to ids: 3, 2, 4, 1 + assert_eq!(id_col.value(0), 3); + assert_eq!(id_col.value(1), 2); + assert_eq!(id_col.value(2), 4); + assert_eq!(id_col.value(3), 1); + } +} From 052feaf3b6c6cd0e3310b1b8211ab3fed7d0e520 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Tue, 2 Dec 2025 18:01:47 +0800 Subject: [PATCH 096/891] rfc: Modularize `iceberg` Implementations (#1854) ## Which issue does this PR close? - Part of https://github.com/apache/iceberg-rust/issues/1819 ## What changes are included in this PR? Add RFC for iceberg-kernel ## Are these changes tested? --------- Signed-off-by: Xuanwo Co-authored-by: Kevin Liu Co-authored-by: Andrew Lamb Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- ...0001_modularize_iceberg_implementations.md | 120 ++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/rfcs/0001_modularize_iceberg_implementations.md diff --git a/docs/rfcs/0001_modularize_iceberg_implementations.md b/docs/rfcs/0001_modularize_iceberg_implementations.md new file mode 100644 index 0000000000..14bd478270 --- /dev/null +++ b/docs/rfcs/0001_modularize_iceberg_implementations.md @@ -0,0 +1,120 @@ + + +# RFC: Modularize `iceberg` Implementations + +## Background + +Issue #1819 highlighted that the current `iceberg` crate mixes the Iceberg protocol abstractions (catalog/table/plan/transaction) with concrete runtime, storage, and execution code (Tokio runtime wrappers, opendal-based `FileIO`, Arrow helpers, DataFusion glue, etc.). This coupling makes the crate heavy and blocks users from composing their own storage or execution stacks. + +Two principles have been agreed: +1. The `iceberg` crate remains the single source of truth for all protocol traits and data structures. We will not create a separate “kernel” crate or facade layer. +2. Concrete integrations (Tokio runtime, opendal `FileIO`, Arrow/DataFusion glue, catalog adapters, etc.) move out into dedicated companion crates. Users needing a ready path can depend on those crates (e.g., `iceberg-datafusion` or `integrations/local`), while custom stacks depend only on `iceberg`. + +This RFC focuses on modularizing implementations; detailed trait signatures (e.g., `FileIO`, `Runtime`) will be handled in separate RFCs. + +## Goals and Scope + +- Keep `iceberg` as the protocol crate (traits + metadata + planning), without bundling runtimes, storage adapters, or execution glue. +- Relocate concrete code into companion crates under `crates/fileio/*`, `crates/runtime/*`, and `crates/integrations/*`. +- Provide a staged plan for extracting Arrow-dependent APIs to avoid destabilizing file-format code. +- Minimize breaking surfaces: traits stay in `iceberg`; downstream crates mainly adjust dependencies. + +Out of scope: changes to the Iceberg table specification or catalog adapter external behavior; detailed trait method design (covered by follow-up RFCs). + +## Architecture Overview + +### Workspace Layout (target) + +``` +crates/ + iceberg/ # core traits, metadata, planning, transactions + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + catalog/* # catalog adapters (REST, HMS, Glue, etc.) + integrations/ + local/ # simple local/arrow-based helper crate + datafusion/ # combines core + implementations for DF + cache-moka/ + playground/ +``` + +- `crates/iceberg` drops direct deps on opendal, Tokio, Arrow, and DataFusion. +- Implementation crates depend on `iceberg` to implement the traits. +- Higher-level crates (`integrations/local`, `iceberg-datafusion`) assemble the pieces for ready-to-use scenarios. + +### Core Trait Surfaces + +`FileIO`, `Runtime`, `Catalog`, `Table`, `Transaction`, `TableScan` (plan descriptors) all remain hosted in `iceberg`. Precise method signatures are deferred to dedicated RFCs to avoid locking details prematurely. + +### Usage Modes + +- **Custom stacks**: depend on `iceberg` and provide your own implementations. +- **Pre-built stacks**: depend on `integrations/local` or `iceberg-datafusion`, which bundle `iceberg` with selected runtime/FileIO/Arrow helpers. +- `iceberg` does not re-export companion crates; users compose explicitly. + +## Migration Plan (staged, with Arrow extraction phased) + +1. **Phase 1 – Confirm trait hosting, defer details** + - Keep all protocol traits in `iceberg`; move detailed API design (FileIO, Runtime, etc.) to separate RFCs. + - Add temporary shims/deprecations only when traits are finalized. + +2. **Phase 2 – First Arrow step: move `to_arrow()` out** + - Relocate the public `to_arrow()` API to `integrations/local` (or another higher-level crate). Core no longer exposes Arrow entry points. + - Keep internal Arrow-dependent helpers (e.g., `ArrowFileReader`) temporarily in `iceberg` to avoid breaking file-format flows. + +3. **Phase 3 – Gradual Arrow dependency removal** + - Incrementally migrate/replace Arrow-dependent internals (`ArrowFileReader`, format-specific readers) into `integrations/local` or other helper crates. + - Adjust file-format APIs as needed; expect this to be multi-release work. + +4. **Phase 4 – Dependency cleanup** + - Ensure catalog and integration crates depend only on `iceberg` plus the specific runtime/FileIO/helper crates they need. + - Verify build/test pipelines against the new dependency graph. + +5. **Phase 5 – Docs & release** + - Publish migration guides: where `to_arrow()` moved, how to assemble local/DataFusion stacks. + - Schedule deprecation windows for remaining Arrow helpers; target a breaking release once Arrow is fully removed from `iceberg`. + +## Compatibility + +- Short term: users of `Table::scan().to_arrow()` must switch to `integrations/local` (or another crate that rehosts that API). Other Arrow types stay temporarily but will migrate in later phases. +- Long term: `iceberg` will be Arrow-free; companion crates provide Arrow-based helpers. +- Tests/examples move alongside the implementations they exercise. + +## Risks and Mitigations + +| Risk | Description | Mitigation | +| ---- | ----------- | ---------- | +| Arrow dependency unwinding is complex | File-format readers may rely on Arrow types | Phase the work; move `to_arrow()` first, then refactor readers; document interim state | +| Discoverability | Users may not know where Arrow helpers went | Clear docs pointing to `integrations/local` and `iceberg-datafusion`; migration guide | +| Trait churn | Future trait RFCs may break early adopters | Use deprecation shims and communicate timelines | +| Duplicate impls | Multiple helper crates could overlap | Provide recommended combinations and feature guidance | + +## Open Questions + +1. Versioning: align companion crate versions with `iceberg`, or allow independent versions plus compatibility matrix? +2. Deprecation schedule: how long do we keep interim Arrow helpers before full removal from `iceberg`? + +## Conclusion + +We will keep `iceberg` as the protocol crate while modularizing concrete implementations. Arrow removal will be phased: first relocating `to_arrow()` to `integrations/local`, then gradually moving Arrow-dependent readers and helpers. This keeps the core lean, lets users compose their preferred runtime/FileIO stacks, and still offers ready-to-use combinations via companion crates. From 3d47be53fbb2486055d83524a0804079cf973c27 Mon Sep 17 00:00:00 2001 From: Leonz Date: Thu, 4 Dec 2025 13:11:54 +0800 Subject: [PATCH 097/891] refactor(writer): Make writer builders non-consuming in build (#1889) ## Which issue does this PR close? - Closes #1753. - Related https://github.com/apache/iceberg-rust/pull/1735#discussion_r2428605135 ## What changes are included in this PR? This change allows users to reuse builder instances without cloning when creating multiple writers with the same configuration. Modification non-consuming self in build function: - `IcebergWriterBuilder` - `RollingFileWriterBuilder` - `FileWriterBuilder` ## Are these changes tested? --- .../src/writer/base_writer/data_file_writer.rs | 6 +++--- .../src/writer/base_writer/equality_delete_writer.rs | 12 ++++++------ .../src/writer/file_writer/location_generator.rs | 4 ++-- crates/iceberg/src/writer/file_writer/mod.rs | 4 ++-- .../iceberg/src/writer/file_writer/parquet_writer.rs | 4 ++-- .../iceberg/src/writer/file_writer/rolling_writer.rs | 12 +++++------- crates/iceberg/src/writer/mod.rs | 8 +++----- .../src/writer/partitioning/clustered_writer.rs | 1 - .../iceberg/src/writer/partitioning/fanout_writer.rs | 1 - .../src/writer/partitioning/unpartitioned_writer.rs | 2 +- 10 files changed, 24 insertions(+), 30 deletions(-) diff --git a/crates/iceberg/src/writer/base_writer/data_file_writer.rs b/crates/iceberg/src/writer/base_writer/data_file_writer.rs index dcaa56cc97..cb7bd172ea 100644 --- a/crates/iceberg/src/writer/base_writer/data_file_writer.rs +++ b/crates/iceberg/src/writer/base_writer/data_file_writer.rs @@ -27,7 +27,7 @@ use crate::writer::{CurrentFileStatus, IcebergWriter, IcebergWriterBuilder}; use crate::{Error, ErrorKind, Result}; /// Builder for `DataFileWriter`. -#[derive(Clone, Debug)] +#[derive(Debug)] pub struct DataFileWriterBuilder { inner: RollingFileWriterBuilder, } @@ -53,9 +53,9 @@ where { type R = DataFileWriter; - async fn build(self, partition_key: Option) -> Result { + async fn build(&self, partition_key: Option) -> Result { Ok(DataFileWriter { - inner: Some(self.inner.clone().build()), + inner: Some(self.inner.build()), partition_key, }) } diff --git a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs index 664ea84334..cd0b19148d 100644 --- a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs +++ b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs @@ -34,7 +34,7 @@ use crate::writer::{IcebergWriter, IcebergWriterBuilder}; use crate::{Error, ErrorKind, Result}; /// Builder for `EqualityDeleteWriter`. -#[derive(Clone, Debug)] +#[derive(Debug)] pub struct EqualityDeleteFileWriterBuilder< B: FileWriterBuilder, L: LocationGenerator, @@ -60,7 +60,7 @@ where } /// Config for `EqualityDeleteWriter`. -#[derive(Clone, Debug)] +#[derive(Debug)] pub struct EqualityDeleteWriterConfig { // Field ids used to determine row equality in equality delete files. equality_ids: Vec, @@ -123,11 +123,11 @@ where { type R = EqualityDeleteFileWriter; - async fn build(self, partition_key: Option) -> Result { + async fn build(&self, partition_key: Option) -> Result { Ok(EqualityDeleteFileWriter { - inner: Some(self.inner.clone().build()), - projector: self.config.projector, - equality_ids: self.config.equality_ids, + inner: Some(self.inner.build()), + projector: self.config.projector.clone(), + equality_ids: self.config.equality_ids.clone(), partition_key, }) } diff --git a/crates/iceberg/src/writer/file_writer/location_generator.rs b/crates/iceberg/src/writer/file_writer/location_generator.rs index a5cfc28292..0ad4d91ac6 100644 --- a/crates/iceberg/src/writer/file_writer/location_generator.rs +++ b/crates/iceberg/src/writer/file_writer/location_generator.rs @@ -24,7 +24,7 @@ use crate::Result; use crate::spec::{DataFileFormat, PartitionKey, TableMetadata}; /// `LocationGenerator` used to generate the location of data file. -pub trait LocationGenerator: Clone + Send + 'static { +pub trait LocationGenerator: Clone + Send + Sync + 'static { /// Generate an absolute path for the given file name that includes the partition path. /// /// # Arguments @@ -94,7 +94,7 @@ impl LocationGenerator for DefaultLocationGenerator { } /// `FileNameGeneratorTrait` used to generate file name for data file. The file name can be passed to `LocationGenerator` to generate the location of the file. -pub trait FileNameGenerator: Clone + Send + 'static { +pub trait FileNameGenerator: Clone + Send + Sync + 'static { /// Generate a file name. fn generate_file_name(&self) -> String; } diff --git a/crates/iceberg/src/writer/file_writer/mod.rs b/crates/iceberg/src/writer/file_writer/mod.rs index 2ed6414ce8..101919f5b3 100644 --- a/crates/iceberg/src/writer/file_writer/mod.rs +++ b/crates/iceberg/src/writer/file_writer/mod.rs @@ -36,11 +36,11 @@ pub mod rolling_writer; type DefaultOutput = Vec; /// File writer builder trait. -pub trait FileWriterBuilder: Send + Clone + 'static { +pub trait FileWriterBuilder: Clone + Send + Sync + 'static { /// The associated file writer type. type R: FileWriter; /// Build file writer. - fn build(self, output_file: OutputFile) -> impl Future> + Send; + fn build(&self, output_file: OutputFile) -> impl Future> + Send; } /// File writer focus on writing record batch to different physical file format.(Such as parquet. orc) diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 3e9d1715c9..5cf031a9fb 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -81,11 +81,11 @@ impl ParquetWriterBuilder { impl FileWriterBuilder for ParquetWriterBuilder { type R = ParquetWriter; - async fn build(self, output_file: OutputFile) -> Result { + async fn build(&self, output_file: OutputFile) -> Result { Ok(ParquetWriter { schema: self.schema.clone(), inner_writer: None, - writer_properties: self.props, + writer_properties: self.props.clone(), current_row_num: 0, output_file, nan_value_count_visitor: NanValueCountVisitor::new_with_match_mode(self.match_mode), diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index 8f03654786..06246ab660 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -103,15 +103,15 @@ where } /// Build a new [`RollingFileWriter`]. - pub fn build(self) -> RollingFileWriter { + pub fn build(&self) -> RollingFileWriter { RollingFileWriter { inner: None, - inner_builder: self.inner_builder, + inner_builder: self.inner_builder.clone(), target_file_size: self.target_file_size, data_file_builders: vec![], - file_io: self.file_io, - location_generator: self.location_generator, - file_name_generator: self.file_name_generator, + file_io: self.file_io.clone(), + location_generator: self.location_generator.clone(), + file_name_generator: self.file_name_generator.clone(), } } } @@ -192,7 +192,6 @@ where // initialize inner writer self.inner = Some( self.inner_builder - .clone() .build(self.new_output_file(partition_key)?) .await?, ); @@ -206,7 +205,6 @@ where // start a new writer self.inner = Some( self.inner_builder - .clone() .build(self.new_output_file(partition_key)?) .await?, ); diff --git a/crates/iceberg/src/writer/mod.rs b/crates/iceberg/src/writer/mod.rs index a7892d49e1..d475230685 100644 --- a/crates/iceberg/src/writer/mod.rs +++ b/crates/iceberg/src/writer/mod.rs @@ -148,7 +148,7 @@ //! impl IcebergWriterBuilder for LatencyRecordWriterBuilder { //! type R = LatencyRecordWriter; //! -//! async fn build(self, partition_key: Option) -> Result { +//! async fn build(&self, partition_key: Option) -> Result { //! Ok(LatencyRecordWriter { //! inner_writer: self.inner_writer_builder.build(partition_key).await?, //! }) @@ -398,13 +398,11 @@ type DefaultOutput = Vec; /// The builder for iceberg writer. #[async_trait::async_trait] -pub trait IcebergWriterBuilder: - Send + Clone + 'static -{ +pub trait IcebergWriterBuilder: Send + Sync + 'static { /// The associated writer type. type R: IcebergWriter; /// Build the iceberg writer with an optional partition key. - async fn build(self, partition_key: Option) -> Result; + async fn build(&self, partition_key: Option) -> Result; } /// The iceberg writer used to write data to iceberg table. diff --git a/crates/iceberg/src/writer/partitioning/clustered_writer.rs b/crates/iceberg/src/writer/partitioning/clustered_writer.rs index 3587723965..01eb452083 100644 --- a/crates/iceberg/src/writer/partitioning/clustered_writer.rs +++ b/crates/iceberg/src/writer/partitioning/clustered_writer.rs @@ -118,7 +118,6 @@ where // Create a new writer for the new partition self.current_writer = Some( self.inner_builder - .clone() .build(Some(partition_key.clone())) .await?, ); diff --git a/crates/iceberg/src/writer/partitioning/fanout_writer.rs b/crates/iceberg/src/writer/partitioning/fanout_writer.rs index 796c1a4888..21a174b0d0 100644 --- a/crates/iceberg/src/writer/partitioning/fanout_writer.rs +++ b/crates/iceberg/src/writer/partitioning/fanout_writer.rs @@ -73,7 +73,6 @@ where if !self.partition_writers.contains_key(partition_key.data()) { let writer = self .inner_builder - .clone() .build(Some(partition_key.clone())) .await?; self.partition_writers diff --git a/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs b/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs index 0fb9cba3f1..29825a5416 100644 --- a/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs +++ b/crates/iceberg/src/writer/partitioning/unpartitioned_writer.rs @@ -75,7 +75,7 @@ where pub async fn write(&mut self, input: I) -> Result<()> { // Lazily create writer on first write if self.writer.is_none() { - self.writer = Some(self.inner_builder.clone().build(None).await?); + self.writer = Some(self.inner_builder.build(None).await?); } // Write directly to inner writer From 9fa3776cbd73159809e1c91ea1904ff9449d7a3d Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 5 Dec 2025 10:27:54 +0100 Subject: [PATCH 098/891] fix: Keep snapshot log on replace (#1896) ## Which issue does this PR close? Fixes remove_ref() to preserve snapshot log when removing MainBranch reference during CREATE OR REPLACE TABLE operations. Previously cleared entire snapshot history, causing testReplaceTableKeepsSnapshotLog RCK test to fail. Related Go Issue: https://github.com/apache/iceberg-go/pull/638 Java also does not clear the log: https://github.com/apache/iceberg/blob/16e84356dae1975fa04d8c3ecce30a90df18ca9f/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1342-L1352 ## What changes are included in this PR? - Do not clear `snapshot_log` if ref to `main` branch is removed ## Are these changes tested? Yes --------- Signed-off-by: Xuanwo Co-authored-by: Xuanwo --- .../src/spec/table_metadata_builder.rs | 68 ++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 6b8ce1e6a5..eee4fec345 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -572,7 +572,6 @@ impl TableMetadataBuilder { pub fn remove_ref(mut self, ref_name: &str) -> Self { if ref_name == MAIN_BRANCH { self.metadata.current_snapshot_id = None; - self.metadata.snapshot_log.clear(); } if self.metadata.refs.remove(ref_name).is_some() || ref_name == MAIN_BRANCH { @@ -2237,6 +2236,73 @@ mod tests { assert_eq!(result.metadata.current_snapshot().unwrap().snapshot_id(), 2); } + #[test] + fn test_remove_main_ref_keeps_snapshot_log() { + let builder = builder_without_changes(FormatVersion::V2); + + let snapshot = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(builder.metadata.last_updated_ms + 1) + .with_sequence_number(0) + .with_schema_id(0) + .with_manifest_list("/snap-1.avro") + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::from_iter(vec![ + ( + "spark.app.id".to_string(), + "local-1662532784305".to_string(), + ), + ("added-data-files".to_string(), "4".to_string()), + ("added-records".to_string(), "4".to_string()), + ("added-files-size".to_string(), "6001".to_string()), + ]), + }) + .build(); + + let result = builder + .add_snapshot(snapshot.clone()) + .unwrap() + .set_ref(MAIN_BRANCH, SnapshotReference { + snapshot_id: 1, + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: Some(10), + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + }) + .unwrap() + .build() + .unwrap(); + + // Verify snapshot log was created + assert_eq!(result.metadata.snapshot_log.len(), 1); + assert_eq!(result.metadata.snapshot_log[0].snapshot_id, 1); + assert_eq!(result.metadata.current_snapshot_id, Some(1)); + + // Remove the main ref + let result_after_remove = result + .metadata + .into_builder(Some( + "s3://bucket/test/location/metadata/metadata2.json".to_string(), + )) + .remove_ref(MAIN_BRANCH) + .build() + .unwrap(); + + // Verify snapshot log is kept even after removing main ref + assert_eq!(result_after_remove.metadata.snapshot_log.len(), 1); + assert_eq!(result_after_remove.metadata.snapshot_log[0].snapshot_id, 1); + assert_eq!(result_after_remove.metadata.current_snapshot_id, None); + assert_eq!(result_after_remove.changes.len(), 1); + assert_eq!( + result_after_remove.changes[0], + TableUpdate::RemoveSnapshotRef { + ref_name: MAIN_BRANCH.to_string() + } + ); + } + #[test] fn test_set_branch_snapshot_creates_branch_if_not_exists() { let builder = builder_without_changes(FormatVersion::V2); From 7e66ded6555a8c445f95b0e4837fed22c75fe46c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Dec 2025 01:53:06 +0800 Subject: [PATCH 099/891] chore(deps): Bump actions/stale from 10.1.0 to 10.1.1 (#1908) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/stale](https://github.com/actions/stale) from 10.1.0 to 10.1.1.
Release notes

Sourced from actions/stale's releases.

v10.1.1

What's Changed

Bug Fix

Improvement

Dependency Upgrades

New Contributors

Full Changelog: https://github.com/actions/stale/compare/v10...v10.1.1

Changelog

Sourced from actions/stale's changelog.

Changelog

Commits
  • 9971854 build(deps): bump actions/checkout from 4 to 6 (#1306)
  • 5611b9d build(deps): bump actions/publish-action from 0.3.0 to 0.4.0 (#1291)
  • fad0de8 Improves error handling when rate limiting is disabled on GHES. (#1300)
  • 39bea7d Add Missing Input Reading for only-issue-types (#1298)
  • e46bbab build(deps-dev): bump @​types/node from 20.10.3 to 24.2.0 and document breakin...
  • 65d1d48 build(deps-dev): bump eslint-config-prettier from 8.10.0 to 10.1.8 (#1276)
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/stale&package-manager=github_actions&previous-version=10.1.0&new-version=10.1.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/stale.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 5e653cffe4..95a4fdc256 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -31,7 +31,7 @@ jobs: if: github.repository_owner == 'apache' runs-on: ubuntu-22.04 steps: - - uses: actions/stale@v10.1.0 + - uses: actions/stale@v10.1.1 with: stale-issue-label: 'stale' exempt-issue-labels: 'not-stale' From c0f9fdcd283ec650c64df7f367cd1ae473c24e62 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Mon, 8 Dec 2025 02:14:42 -0800 Subject: [PATCH 100/891] feat(datafusion): Add sqllogictest for DataFusion INSERT INTO (#1887) ## Which issue does this PR close? - Closes #1835 ## What changes are included in this PR? - Added a new schedule to run insert into sqllogic test ## Are these changes tested? It's a test itself --- crates/sqllogictest/src/engine/datafusion.rs | 71 ++++++++++- .../testdata/schedules/df_test.toml | 6 +- .../testdata/slts/df_test/insert_into.slt | 119 ++++++++++++++++++ .../testdata/slts/df_test/show_tables.slt | 6 + 4 files changed, 200 insertions(+), 2 deletions(-) create mode 100644 crates/sqllogictest/testdata/slts/df_test/insert_into.slt diff --git a/crates/sqllogictest/src/engine/datafusion.rs b/crates/sqllogictest/src/engine/datafusion.rs index b3e37d9206..e3402dfa97 100644 --- a/crates/sqllogictest/src/engine/datafusion.rs +++ b/crates/sqllogictest/src/engine/datafusion.rs @@ -22,8 +22,9 @@ use std::sync::Arc; use datafusion::catalog::CatalogProvider; use datafusion::prelude::{SessionConfig, SessionContext}; use datafusion_sqllogictest::DataFusion; -use iceberg::CatalogBuilder; use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; +use iceberg::spec::{NestedField, PrimitiveType, Schema, Transform, Type, UnboundPartitionSpec}; +use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation}; use iceberg_datafusion::IcebergCatalogProvider; use indicatif::ProgressBar; use toml::Table as TomlTable; @@ -84,8 +85,76 @@ impl DataFusionEngine { ) .await?; + // Create a test namespace for INSERT INTO tests + let namespace = NamespaceIdent::new("default".to_string()); + catalog.create_namespace(&namespace, HashMap::new()).await?; + + // Create test tables + Self::create_unpartitioned_table(&catalog, &namespace).await?; + Self::create_partitioned_table(&catalog, &namespace).await?; + Ok(Arc::new( IcebergCatalogProvider::try_new(Arc::new(catalog)).await?, )) } + + /// Create an unpartitioned test table with id and name columns + /// TODO: this can be removed when we support CREATE TABLE + async fn create_unpartitioned_table( + catalog: &impl Catalog, + namespace: &NamespaceIdent, + ) -> anyhow::Result<()> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build()?; + + catalog + .create_table( + namespace, + TableCreation::builder() + .name("test_unpartitioned_table".to_string()) + .schema(schema) + .build(), + ) + .await?; + + Ok(()) + } + + /// Create a partitioned test table with id, category, and value columns + /// Partitioned by category using identity transform + /// TODO: this can be removed when we support CREATE TABLE + async fn create_partitioned_table( + catalog: &impl Catalog, + namespace: &NamespaceIdent, + ) -> anyhow::Result<()> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "category", Type::Primitive(PrimitiveType::String)).into(), + NestedField::optional(3, "value", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build()?; + + let partition_spec = UnboundPartitionSpec::builder() + .with_spec_id(0) + .add_partition_field(2, "category", Transform::Identity)? + .build(); + + catalog + .create_table( + namespace, + TableCreation::builder() + .name("test_partitioned_table".to_string()) + .schema(schema) + .partition_spec(partition_spec) + .build(), + ) + .await?; + + Ok(()) + } } diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index 0733744951..df5e638d5a 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -20,4 +20,8 @@ df = { type = "datafusion" } [[steps]] engine = "df" -slt = "df_test/show_tables.slt" \ No newline at end of file +slt = "df_test/show_tables.slt" + +[[steps]] +engine = "df" +slt = "df_test/insert_into.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/insert_into.slt b/crates/sqllogictest/testdata/slts/df_test/insert_into.slt new file mode 100644 index 0000000000..2ba33afcd1 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/insert_into.slt @@ -0,0 +1,119 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Verify the table is initially empty +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table +---- + +# Insert a single row and verify the count +query I +INSERT INTO default.default.test_unpartitioned_table VALUES (1, 'Alice') +---- +1 + +# Verify the inserted row +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table +---- +1 Alice + +# Insert multiple rows and verify the count +query I +INSERT INTO default.default.test_unpartitioned_table VALUES (2, 'Bob'), (3, 'Charlie') +---- +2 + +# Verify all rows +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table +---- +1 Alice +2 Bob +3 Charlie + +# Insert with NULL value and verify the count +query I +INSERT INTO default.default.test_unpartitioned_table VALUES (4, NULL) +---- +1 + +# Verify NULL handling +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table +---- +1 Alice +2 Bob +3 Charlie +4 NULL + +# Test partitioned table - verify initially empty +query ITT rowsort +SELECT * FROM default.default.test_partitioned_table +---- + +# Insert single row into partitioned table +query I +INSERT INTO default.default.test_partitioned_table VALUES (1, 'electronics', 'laptop') +---- +1 + +# Verify the inserted row in partitioned table +query ITT rowsort +SELECT * FROM default.default.test_partitioned_table +---- +1 electronics laptop + +# Insert multiple rows with different partition values +query I +INSERT INTO default.default.test_partitioned_table VALUES (2, 'electronics', 'phone'), (3, 'books', 'novel'), (4, 'books', 'textbook'), (5, 'clothing', 'shirt') +---- +4 + +# Verify all rows in partitioned table +query ITT rowsort +SELECT * FROM default.default.test_partitioned_table +---- +1 electronics laptop +2 electronics phone +3 books novel +4 books textbook +5 clothing shirt + +# Insert with NULL value in optional column +query I +INSERT INTO default.default.test_partitioned_table VALUES (6, 'electronics', NULL) +---- +1 + +# Verify NULL handling in partitioned table +query ITT rowsort +SELECT * FROM default.default.test_partitioned_table +---- +1 electronics laptop +2 electronics phone +3 books novel +4 books textbook +5 clothing shirt +6 electronics NULL + +# Verify partition filtering works +query ITT rowsort +SELECT * FROM default.default.test_partitioned_table WHERE category = 'books' +---- +3 books novel +4 books textbook diff --git a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt index 34709d7359..c5da5f6276 100644 --- a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt +++ b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt @@ -25,6 +25,12 @@ datafusion information_schema routines VIEW datafusion information_schema schemata VIEW datafusion information_schema tables VIEW datafusion information_schema views VIEW +default default test_partitioned_table BASE TABLE +default default test_partitioned_table$manifests BASE TABLE +default default test_partitioned_table$snapshots BASE TABLE +default default test_unpartitioned_table BASE TABLE +default default test_unpartitioned_table$manifests BASE TABLE +default default test_unpartitioned_table$snapshots BASE TABLE default information_schema columns VIEW default information_schema df_settings VIEW default information_schema parameters VIEW From 84b2ba86765b96b3d4afc8b30ab29b1ef61be510 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Mon, 8 Dec 2025 18:45:42 +0800 Subject: [PATCH 101/891] refactor: Drop smol runtime support (#1900) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1866 ## What changes are included in this PR? ## Are these changes tested? --------- Signed-off-by: Xuanwo Co-authored-by: Renjie Liu --- .github/workflows/ci.yml | 1 - Cargo.lock | 189 +----------------------------- Cargo.toml | 1 - crates/iceberg/Cargo.toml | 5 +- crates/iceberg/src/io/storage.rs | 8 ++ crates/iceberg/src/runtime/mod.rs | 58 ++------- 6 files changed, 25 insertions(+), 237 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7b7e81837e..f393309bcb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -143,7 +143,6 @@ jobs: matrix: test-suite: - { name: "default", args: "--all-targets --all-features --workspace" } - - { name: "smol", args: "--all-targets --no-default-features --features smol --features storage-all --workspace" } - { name: "doc", args: "--doc --all-features --workspace" } name: Unit Tests (${{ matrix.test-suite.name }}) steps: diff --git a/Cargo.lock b/Cargo.lock index 736f2b1461..4d9f2b99c3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -465,18 +465,6 @@ dependencies = [ "pin-project-lite", ] -[[package]] -name = "async-channel" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" -dependencies = [ - "concurrent-queue", - "event-listener-strategy", - "futures-core", - "pin-project-lite", -] - [[package]] name = "async-compression" version = "0.4.19" @@ -494,49 +482,6 @@ dependencies = [ "zstd-safe", ] -[[package]] -name = "async-executor" -version = "1.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "497c00e0fd83a72a79a39fcbd8e3e2f055d6f6c7e025f3b3d91f4f8e76527fb8" -dependencies = [ - "async-task", - "concurrent-queue", - "fastrand", - "futures-lite", - "pin-project-lite", - "slab", -] - -[[package]] -name = "async-fs" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8034a681df4aed8b8edbd7fbe472401ecf009251c8b40556b304567052e294c5" -dependencies = [ - "async-lock", - "blocking", - "futures-lite", -] - -[[package]] -name = "async-io" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "456b8a8feb6f42d237746d4b3e9a178494627745c3c56c6ea55d92ba50d026fc" -dependencies = [ - "autocfg", - "cfg-if", - "concurrent-queue", - "futures-io", - "futures-lite", - "parking", - "polling", - "rustix", - "slab", - "windows-sys 0.61.2", -] - [[package]] name = "async-lock" version = "3.4.1" @@ -548,35 +493,6 @@ dependencies = [ "pin-project-lite", ] -[[package]] -name = "async-net" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b948000fad4873c1c9339d60f2623323a0cfd3816e5181033c6a5cb68b2accf7" -dependencies = [ - "async-io", - "blocking", - "futures-lite", -] - -[[package]] -name = "async-process" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc50921ec0055cdd8a16de48773bfeec5c972598674347252c0399676be7da75" -dependencies = [ - "async-channel", - "async-io", - "async-lock", - "async-signal", - "async-task", - "blocking", - "cfg-if", - "event-listener", - "futures-lite", - "rustix", -] - [[package]] name = "async-recursion" version = "1.1.1" @@ -588,30 +504,6 @@ dependencies = [ "syn 2.0.108", ] -[[package]] -name = "async-signal" -version = "0.2.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43c070bbf59cd3570b6b2dd54cd772527c7c3620fce8be898406dd3ed6adc64c" -dependencies = [ - "async-io", - "async-lock", - "atomic-waker", - "cfg-if", - "futures-core", - "futures-io", - "rustix", - "signal-hook-registry", - "slab", - "windows-sys 0.61.2", -] - -[[package]] -name = "async-task" -version = "4.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" - [[package]] name = "async-trait" version = "0.1.89" @@ -1185,19 +1077,6 @@ dependencies = [ "generic-array", ] -[[package]] -name = "blocking" -version = "1.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e83f8d02be6967315521be875afa792a316e28d57b5a2d401897e2a7921b7f21" -dependencies = [ - "async-channel", - "async-task", - "futures-io", - "futures-lite", - "piper", -] - [[package]] name = "bon" version = "3.8.1" @@ -2667,7 +2546,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2809,7 +2688,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -3052,19 +2931,6 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6" -[[package]] -name = "futures-lite" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f78e10609fe0e0b3f4157ffab1876319b5b0db102a2c60dc4626306dc46b44ad" -dependencies = [ - "fastrand", - "futures-core", - "futures-io", - "parking", - "pin-project-lite", -] - [[package]] name = "futures-macro" version = "0.3.31" @@ -3567,7 +3433,6 @@ dependencies = [ "serde_json", "serde_repr", "serde_with", - "smol", "strum 0.27.2", "tempfile", "thrift", @@ -4565,7 +4430,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5077,17 +4942,6 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" -[[package]] -name = "piper" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96c8c490f422ef9a4efd2cb5b42b76c8613d7e7dfc1caf667b8a3350a5acc066" -dependencies = [ - "atomic-waker", - "fastrand", - "futures-io", -] - [[package]] name = "pkcs1" version = "0.7.5" @@ -5132,20 +4986,6 @@ version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" -[[package]] -name = "polling" -version = "3.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d0e4f59085d47d8241c88ead0f274e8a0cb551f3625263c05eb8dd897c34218" -dependencies = [ - "cfg-if", - "concurrent-queue", - "hermit-abi", - "pin-project-lite", - "rustix", - "windows-sys 0.61.2", -] - [[package]] name = "port_scanner" version = "0.1.5" @@ -5921,7 +5761,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -6478,23 +6318,6 @@ dependencies = [ "serde", ] -[[package]] -name = "smol" -version = "2.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a33bd3e260892199c3ccfc487c88b2da2265080acb316cd920da72fdfd7c599f" -dependencies = [ - "async-channel", - "async-executor", - "async-fs", - "async-io", - "async-lock", - "async-net", - "async-process", - "blocking", - "futures-lite", -] - [[package]] name = "snap" version = "1.1.1" @@ -6993,7 +6816,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -7819,7 +7642,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 0512155534..36093d92aa 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -116,7 +116,6 @@ serde_derive = "1.0.219" serde_json = "1.0.142" serde_repr = "0.1.16" serde_with = "3.4" -smol = "2.0.2" sqllogictest = "0.28.3" sqlx = { version = "0.8.1", default-features = false } stacker = "0.1.20" diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index 895a5cf5e4..84c90d466f 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -29,7 +29,7 @@ license = { workspace = true } repository = { workspace = true } [features] -default = ["storage-memory", "storage-fs", "storage-s3", "tokio"] +default = ["storage-memory", "storage-fs", "storage-s3"] storage-all = ["storage-memory", "storage-fs", "storage-s3", "storage-gcs"] storage-azdls = ["opendal/services-azdls"] @@ -39,8 +39,6 @@ storage-memory = ["opendal/services-memory"] storage-oss = ["opendal/services-oss"] storage-s3 = ["opendal/services-s3", "reqsign"] -smol = ["dep:smol"] -tokio = ["tokio/rt-multi-thread"] [dependencies] anyhow = { workspace = true } @@ -85,7 +83,6 @@ serde_derive = { workspace = true } serde_json = { workspace = true } serde_repr = { workspace = true } serde_with = { workspace = true } -smol = { workspace = true, optional = true } strum = { workspace = true, features = ["derive"] } thrift = { workspace = true } tokio = { workspace = true, optional = false, features = ["sync"] } diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index d5f2ad8fab..5880ccca59 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -15,6 +15,12 @@ // specific language governing permissions and limitations // under the License. +#[cfg(any( + feature = "storage-s3", + feature = "storage-gcs", + feature = "storage-oss", + feature = "storage-azdls", +))] use std::sync::Arc; use opendal::layers::RetryLayer; @@ -71,6 +77,7 @@ impl Storage { /// Convert iceberg config to opendal config. pub(crate) fn build(file_io_builder: FileIOBuilder) -> crate::Result { let (scheme_str, props, extensions) = file_io_builder.into_parts(); + let _ = (&props, &extensions); let scheme = Self::parse_scheme(&scheme_str)?; match scheme { @@ -127,6 +134,7 @@ impl Storage { path: &'a impl AsRef, ) -> crate::Result<(Operator, &'a str)> { let path = path.as_ref(); + let _ = path; let (operator, relative_path): (Operator, &str) = match self { #[cfg(feature = "storage-memory")] Storage::Memory(op) => { diff --git a/crates/iceberg/src/runtime/mod.rs b/crates/iceberg/src/runtime/mod.rs index d0a3ce6602..61aa623f58 100644 --- a/crates/iceberg/src/runtime/mod.rs +++ b/crates/iceberg/src/runtime/mod.rs @@ -21,28 +21,20 @@ use std::future::Future; use std::pin::Pin; use std::task::{Context, Poll}; -pub enum JoinHandle { - #[cfg(feature = "tokio")] - Tokio(tokio::task::JoinHandle), - #[cfg(all(feature = "smol", not(feature = "tokio")))] - Smol(smol::Task), - #[cfg(all(not(feature = "smol"), not(feature = "tokio")))] - Unimplemented(Box), -} +use tokio::task; + +pub struct JoinHandle(task::JoinHandle); + +impl Unpin for JoinHandle {} impl Future for JoinHandle { type Output = T; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { match self.get_mut() { - #[cfg(feature = "tokio")] - JoinHandle::Tokio(handle) => Pin::new(handle) + JoinHandle(handle) => Pin::new(handle) .poll(cx) - .map(|h| h.expect("tokio spawned task failed")), - #[cfg(all(feature = "smol", not(feature = "tokio")))] - JoinHandle::Smol(handle) => Pin::new(handle).poll(cx), - #[cfg(all(not(feature = "smol"), not(feature = "tokio")))] - JoinHandle::Unimplemented(_) => unimplemented!("no runtime has been enabled"), + .map(|r| r.expect("tokio spawned task failed")), } } } @@ -50,17 +42,10 @@ impl Future for JoinHandle { #[allow(dead_code)] pub fn spawn(f: F) -> JoinHandle where - F: Future + Send + 'static, + F: std::future::Future + Send + 'static, F::Output: Send + 'static, { - #[cfg(feature = "tokio")] - return JoinHandle::Tokio(tokio::task::spawn(f)); - - #[cfg(all(feature = "smol", not(feature = "tokio")))] - return JoinHandle::Smol(smol::spawn(f)); - - #[cfg(all(not(feature = "smol"), not(feature = "tokio")))] - unimplemented!("no runtime has been enabled") + JoinHandle(task::spawn(f)) } #[allow(dead_code)] @@ -69,45 +54,22 @@ where F: FnOnce() -> T + Send + 'static, T: Send + 'static, { - #[cfg(feature = "tokio")] - return JoinHandle::Tokio(tokio::task::spawn_blocking(f)); - - #[cfg(all(feature = "smol", not(feature = "tokio")))] - return JoinHandle::Smol(smol::unblock(f)); - - #[cfg(all(not(feature = "smol"), not(feature = "tokio")))] - unimplemented!("no runtime has been enabled") + JoinHandle(task::spawn_blocking(f)) } #[cfg(test)] mod tests { use super::*; - #[cfg(feature = "tokio")] #[tokio::test] async fn test_tokio_spawn() { let handle = spawn(async { 1 + 1 }); assert_eq!(handle.await, 2); } - #[cfg(feature = "tokio")] #[tokio::test] async fn test_tokio_spawn_blocking() { let handle = spawn_blocking(|| 1 + 1); assert_eq!(handle.await, 2); } - - #[cfg(all(feature = "smol", not(feature = "tokio")))] - #[smol::test] - async fn test_smol_spawn() { - let handle = spawn(async { 1 + 1 }); - assert_eq!(handle.await, 2); - } - - #[cfg(all(feature = "smol", not(feature = "tokio")))] - #[smo::test] - async fn test_smol_spawn_blocking() { - let handle = spawn_blocking(|| 1 + 1); - assert_eq!(handle.await, 2); - } } From 5c3c5a2800f2c1f61b6a306b66b32be35e96dd11 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Dec 2025 21:02:28 +0800 Subject: [PATCH 102/891] chore(deps): Bump minijinja from 2.12.0 to 2.13.0 (#1909) --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 4d9f2b99c3..7d23305002 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4214,9 +4214,9 @@ dependencies = [ [[package]] name = "minijinja" -version = "2.12.0" +version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9f264d75233323f4b7d2f03aefe8a990690cdebfbfe26ea86bcbaec5e9ac990" +checksum = "0adbe6e92a6ce0fd6c4aac593fdfd3e3950b0f61b1a63aa9731eb6fd85776fa3" dependencies = [ "serde", ] From 26b98394015c57631566aa3ff36cd4ccd7792b1a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 9 Dec 2025 13:20:39 +0800 Subject: [PATCH 103/891] chore(deps): Bump uuid from 1.18.1 to 1.19.0 (#1910) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.18.1 to 1.19.0.
Release notes

Sourced from uuid's releases.

v1.19.0

What's Changed

Full Changelog: https://github.com/uuid-rs/uuid/compare/v1.18.1...v1.19.0

Commits
  • 7527cef Merge pull request #849 from uuid-rs/cargo/v1.19.0
  • d0422fa prepare for 1.19.0 release
  • f9a36e2 Merge pull request #848 from paolobarbolini/maintenance
  • 029a57e Fix most clippy warnings
  • e73bb27 Upgrade to 2021 edition
  • c597622 Merge pull request #843 from paolobarbolini/serde_core-migration
  • 9835bd6 Switch serde dependency to serde_core
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=uuid&package-manager=cargo&previous-version=1.18.1&new-version=1.19.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- Cargo.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 7d23305002..c7c103d279 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7387,13 +7387,13 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.18.1" +version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f87b8aa10b915a06587d0dec516c282ff295b475d94abf425d62b57710070a2" +checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" dependencies = [ "getrandom 0.3.4", "js-sys", - "serde", + "serde_core", "wasm-bindgen", ] From 1384a4f2d71ed16b73f3b1f139d5dbd4e5035428 Mon Sep 17 00:00:00 2001 From: Gerald Berger <59661379+gbrgr@users.noreply.github.com> Date: Tue, 9 Dec 2025 11:46:49 +0100 Subject: [PATCH 104/891] feat(core): Add support for `_file` column (#1824) ## Which issue does this PR close? - Closes #1766. ## What changes are included in this PR? Integrates virtual field handling for the `_file` metadata column into `RecordBatchTransformer` using a pre-computed constants map, eliminating post-processing and duplicate lookups. ## Key Changes **New `metadata_columns.rs` module**: Centralized utilities for metadata columns - Constants: `RESERVED_FIELD_ID_FILE`, `RESERVED_COL_NAME_FILE` - Helper functions: `get_metadata_column_name()`, `get_metadata_field_id()`, `is_metadata_field()`, `is_metadata_column_name()` **Enhanced `RecordBatchTransformer`**: - Added `constant_fields: HashMap` - pre-computed during initialization - New `with_constant()` method - computes Arrow type once during setup - Updated to use pre-computed types and values (avoids duplicate lookups) - Handles `DataType::RunEndEncoded` for constant strings (memory efficient) **Simplified `reader.rs`**: - Pass full `project_field_ids` (including virtual) to RecordBatchTransformer - Single `with_constant()` call to register `_file` column - Removed post-processing loop **Updated `scan/mod.rs`**: - Use `is_metadata_column_name()` and `get_metadata_field_id()` instead of hardcoded checks ## Are these changes tested? Yes, comprehensive tests have been added to verify the functionality: ### New Tests (7 tests added) #### Table Scan API Tests (7 tests) 1. **`test_select_with_file_column`** - Verifies basic functionality of selecting `_file` with regular columns 2. **`test_select_file_column_position`** - Verifies column ordering is preserved 3. **`test_select_file_column_only`** - Tests selecting only the `_file` column 4. **`test_file_column_with_multiple_files`** - Tests multiple data files scenario 5. **`test_file_column_at_start`** - Tests `_file` at position 0 6. **`test_file_column_at_end`** - Tests `_file` at the last position 7. **`test_select_with_repeated_column_names`** - Tests repeated column selection --- crates/iceberg/src/arrow/reader.rs | 29 +- .../src/arrow/record_batch_transformer.rs | 476 ++++++++++-------- crates/iceberg/src/arrow/schema.rs | 54 ++ crates/iceberg/src/arrow/value.rs | 216 +++++++- crates/iceberg/src/lib.rs | 1 + crates/iceberg/src/metadata_columns.rs | 127 +++++ crates/iceberg/src/scan/mod.rs | 341 ++++++++++++- 7 files changed, 1031 insertions(+), 213 deletions(-) create mode 100644 crates/iceberg/src/metadata_columns.rs diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index ab5a96f751..de8a1420e4 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -54,6 +54,7 @@ use crate::expr::visitors::page_index_evaluator::PageIndexEvaluator; use crate::expr::visitors::row_group_metrics_evaluator::RowGroupMetricsEvaluator; use crate::expr::{BoundPredicate, BoundReference}; use crate::io::{FileIO, FileMetadata, FileRead}; +use crate::metadata_columns::{RESERVED_FIELD_ID_FILE, is_metadata_field}; use crate::scan::{ArrowRecordBatchStream, FileScanTask, FileScanTaskStream}; use crate::spec::{Datum, NameMapping, NestedField, PrimitiveType, Schema, Type}; use crate::utils::available_parallelism; @@ -250,12 +251,20 @@ impl ArrowReader { initial_stream_builder }; + // Filter out metadata fields for Parquet projection (they don't exist in files) + let project_field_ids_without_metadata: Vec = task + .project_field_ids + .iter() + .filter(|&&id| !is_metadata_field(id)) + .copied() + .collect(); + // Create projection mask based on field IDs // - If file has embedded IDs: field-ID-based projection (missing_field_ids=false) // - If name mapping applied: field-ID-based projection (missing_field_ids=true but IDs now match) // - If fallback IDs: position-based projection (missing_field_ids=true) let projection_mask = Self::get_arrow_projection_mask( - &task.project_field_ids, + &project_field_ids_without_metadata, &task.schema, record_batch_stream_builder.parquet_schema(), record_batch_stream_builder.schema(), @@ -266,16 +275,23 @@ impl ArrowReader { record_batch_stream_builder.with_projection(projection_mask.clone()); // RecordBatchTransformer performs any transformations required on the RecordBatches - // that come back from the file, such as type promotion, default column insertion - // and column re-ordering. + // that come back from the file, such as type promotion, default column insertion, + // column re-ordering, partition constants, and virtual field addition (like _file) let mut record_batch_transformer_builder = RecordBatchTransformerBuilder::new(task.schema_ref(), task.project_field_ids()); + // Add the _file metadata column if it's in the projected fields + if task.project_field_ids().contains(&RESERVED_FIELD_ID_FILE) { + let file_datum = Datum::string(task.data_file_path.clone()); + record_batch_transformer_builder = + record_batch_transformer_builder.with_constant(RESERVED_FIELD_ID_FILE, file_datum); + } + if let (Some(partition_spec), Some(partition_data)) = (task.partition_spec.clone(), task.partition.clone()) { record_batch_transformer_builder = - record_batch_transformer_builder.with_partition(partition_spec, partition_data); + record_batch_transformer_builder.with_partition(partition_spec, partition_data)?; } let mut record_batch_transformer = record_batch_transformer_builder.build(); @@ -416,7 +432,10 @@ impl ArrowReader { record_batch_stream_builder .build()? .map(move |batch| match batch { - Ok(batch) => record_batch_transformer.process_record_batch(batch), + Ok(batch) => { + // Process the record batch (type promotion, column reordering, virtual fields, etc.) + record_batch_transformer.process_record_batch(batch) + } Err(err) => Err(err.into()), }); diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index a20adb6a5a..f30d4a09c3 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -19,24 +19,23 @@ use std::collections::HashMap; use std::sync::Arc; use arrow_array::{ - Array as ArrowArray, ArrayRef, BinaryArray, BooleanArray, Date32Array, Float32Array, - Float64Array, Int32Array, Int64Array, NullArray, RecordBatch, RecordBatchOptions, StringArray, - StructArray, + Array as ArrowArray, ArrayRef, Int32Array, RecordBatch, RecordBatchOptions, RunArray, }; -use arrow_buffer::NullBuffer; use arrow_cast::cast; use arrow_schema::{ - DataType, FieldRef, Schema as ArrowSchema, SchemaRef as ArrowSchemaRef, SchemaRef, + DataType, Field, FieldRef, Schema as ArrowSchema, SchemaRef as ArrowSchemaRef, SchemaRef, }; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; -use crate::arrow::schema_to_arrow_schema; +use crate::arrow::value::{create_primitive_array_repeated, create_primitive_array_single_element}; +use crate::arrow::{datum_to_arrow_type_with_ree, schema_to_arrow_schema}; +use crate::metadata_columns::get_metadata_field; use crate::spec::{ - Literal, PartitionSpec, PrimitiveLiteral, Schema as IcebergSchema, Struct, Transform, + Datum, Literal, PartitionSpec, PrimitiveLiteral, Schema as IcebergSchema, Struct, Transform, }; use crate::{Error, ErrorKind, Result}; -/// Build a map of field ID to constant value for identity-partitioned fields. +/// Build a map of field ID to constant value (as Datum) for identity-partitioned fields. /// /// Implements Iceberg spec "Column Projection" rule #1: use partition metadata constants /// only for identity-transformed fields. Non-identity transforms (bucket, truncate, year, etc.) @@ -53,20 +52,65 @@ use crate::{Error, ErrorKind, Result}; fn constants_map( partition_spec: &PartitionSpec, partition_data: &Struct, -) -> HashMap { + schema: &IcebergSchema, +) -> Result> { let mut constants = HashMap::new(); for (pos, field) in partition_spec.fields().iter().enumerate() { // Only identity transforms should use constant values from partition metadata if matches!(field.transform, Transform::Identity) { + // Get the field from schema to extract its type + let iceberg_field = schema.field_by_id(field.source_id).ok_or(Error::new( + ErrorKind::Unexpected, + format!("Field {} not found in schema", field.source_id), + ))?; + + // Ensure the field type is primitive + let prim_type = match &*iceberg_field.field_type { + crate::spec::Type::Primitive(prim_type) => prim_type, + _ => { + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "Partition field {} has non-primitive type {:?}", + field.source_id, iceberg_field.field_type + ), + )); + } + }; + // Get the partition value for this field - if let Some(Literal::Primitive(value)) = &partition_data[pos] { - constants.insert(field.source_id, value.clone()); + // Handle both None (null) and Some(Literal::Primitive) cases + match &partition_data[pos] { + None => { + // TODO (https://github.com/apache/iceberg-rust/issues/1914): Add support for null datum values. + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "Partition field {} has null value for identity transform", + field.source_id + ), + )); + } + Some(Literal::Primitive(value)) => { + // Create a Datum from the primitive type and value + let datum = Datum::new(prim_type.clone(), value.clone()); + constants.insert(field.source_id, datum); + } + Some(literal) => { + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "Partition field {} has non-primitive value: {:?}", + field.source_id, literal + ), + )); + } } } } - constants + Ok(constants) } /// Indicates how a particular column in a processed RecordBatch should @@ -146,13 +190,13 @@ enum SchemaComparison { /// Builder for RecordBatchTransformer to improve ergonomics when constructing with optional parameters. /// -/// See [`RecordBatchTransformer`] for details on partition spec and partition data. +/// Constant fields are pre-computed for both virtual/metadata fields (like _file) and +/// identity-partitioned fields to avoid duplicate work during batch processing. #[derive(Debug)] pub(crate) struct RecordBatchTransformerBuilder { snapshot_schema: Arc, projected_iceberg_field_ids: Vec, - partition_spec: Option>, - partition_data: Option, + constant_fields: HashMap, } impl RecordBatchTransformerBuilder { @@ -163,32 +207,48 @@ impl RecordBatchTransformerBuilder { Self { snapshot_schema, projected_iceberg_field_ids: projected_iceberg_field_ids.to_vec(), - partition_spec: None, - partition_data: None, + constant_fields: HashMap::new(), } } + /// Add a constant value for a specific field ID. + /// This is used for virtual/metadata fields like _file that have constant values per batch. + /// + /// # Arguments + /// * `field_id` - The field ID to associate with the constant + /// * `datum` - The constant value (with type) for this field + pub(crate) fn with_constant(mut self, field_id: i32, datum: Datum) -> Self { + self.constant_fields.insert(field_id, datum); + self + } + /// Set partition spec and data together for identifying identity-transformed partition columns. /// /// Both partition_spec and partition_data must be provided together since the spec defines /// which fields are identity-partitioned, and the data provides their constant values. - /// One without the other cannot produce a valid constants map. + /// This method computes the partition constants and merges them into constant_fields. pub(crate) fn with_partition( mut self, partition_spec: Arc, partition_data: Struct, - ) -> Self { - self.partition_spec = Some(partition_spec); - self.partition_data = Some(partition_data); - self + ) -> Result { + // Compute partition constants for identity-transformed fields (already returns Datum) + let partition_constants = + constants_map(&partition_spec, &partition_data, &self.snapshot_schema)?; + + // Add partition constants to constant_fields + for (field_id, datum) in partition_constants { + self.constant_fields.insert(field_id, datum); + } + + Ok(self) } pub(crate) fn build(self) -> RecordBatchTransformer { RecordBatchTransformer { snapshot_schema: self.snapshot_schema, projected_iceberg_field_ids: self.projected_iceberg_field_ids, - partition_spec: self.partition_spec, - partition_data: self.partition_data, + constant_fields: self.constant_fields, batch_transform: None, } } @@ -228,16 +288,10 @@ impl RecordBatchTransformerBuilder { pub(crate) struct RecordBatchTransformer { snapshot_schema: Arc, projected_iceberg_field_ids: Vec, - - /// Partition spec for identifying identity-transformed partition columns (spec rule #1). - /// Only fields with identity transforms use partition data constants; non-identity transforms - /// (bucket, truncate, etc.) must read source columns from data files. - partition_spec: Option>, - - /// Partition data providing constant values for identity-transformed partition columns (spec rule #1). - /// For example, in a file at path `dept=engineering/file.parquet`, this would contain - /// the value "engineering" for the dept field. - partition_data: Option, + // Pre-computed constant field information: field_id -> Datum + // Includes both virtual/metadata fields (like _file) and identity-partitioned fields + // Datum holds both the Iceberg type and the value + constant_fields: HashMap, // BatchTransform gets lazily constructed based on the schema of // the first RecordBatch we receive from the file @@ -279,8 +333,7 @@ impl RecordBatchTransformer { record_batch.schema_ref(), self.snapshot_schema.as_ref(), &self.projected_iceberg_field_ids, - self.partition_spec.as_ref().map(|s| s.as_ref()), - self.partition_data.as_ref(), + &self.constant_fields, )?); self.process_record_batch(record_batch)? @@ -299,8 +352,7 @@ impl RecordBatchTransformer { source_schema: &ArrowSchemaRef, snapshot_schema: &IcebergSchema, projected_iceberg_field_ids: &[i32], - partition_spec: Option<&PartitionSpec>, - partition_data: Option<&Struct>, + constant_fields: &HashMap, ) -> Result { let mapped_unprojected_arrow_schema = Arc::new(schema_to_arrow_schema(snapshot_schema)?); let field_id_to_mapped_schema_map = @@ -311,22 +363,54 @@ impl RecordBatchTransformer { let fields: Result> = projected_iceberg_field_ids .iter() .map(|field_id| { - Ok(field_id_to_mapped_schema_map - .get(field_id) - .ok_or(Error::new(ErrorKind::Unexpected, "field not found"))? - .0 - .clone()) + // Check if this is a constant field + if constant_fields.contains_key(field_id) { + // For metadata/virtual fields (like _file), get name from metadata_columns + // For partition fields, get name from schema (they exist in schema) + if let Ok(iceberg_field) = get_metadata_field(*field_id) { + // This is a metadata/virtual field - convert Iceberg field to Arrow + let datum = constant_fields.get(field_id).ok_or(Error::new( + ErrorKind::Unexpected, + "constant field not found", + ))?; + let arrow_type = datum_to_arrow_type_with_ree(datum); + let arrow_field = + Field::new(&iceberg_field.name, arrow_type, !iceberg_field.required) + .with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + iceberg_field.id.to_string(), + )])); + Ok(Arc::new(arrow_field)) + } else { + // This is a partition constant field (exists in schema but uses constant value) + let field = &field_id_to_mapped_schema_map + .get(field_id) + .ok_or(Error::new(ErrorKind::Unexpected, "field not found"))? + .0; + let datum = constant_fields.get(field_id).ok_or(Error::new( + ErrorKind::Unexpected, + "constant field not found", + ))?; + let arrow_type = datum_to_arrow_type_with_ree(datum); + // Use the type from constant_fields (REE for constants) + let constant_field = + Field::new(field.name(), arrow_type, field.is_nullable()) + .with_metadata(field.metadata().clone()); + Ok(Arc::new(constant_field)) + } + } else { + // Regular field - use schema as-is + Ok(field_id_to_mapped_schema_map + .get(field_id) + .ok_or(Error::new(ErrorKind::Unexpected, "field not found"))? + .0 + .clone()) + } }) .collect(); let target_schema = Arc::new(ArrowSchema::new(fields?)); - let constants_map = if let (Some(spec), Some(data)) = (partition_spec, partition_data) { - constants_map(spec, data) - } else { - HashMap::new() - }; - match Self::compare_schemas(source_schema, &target_schema) { SchemaComparison::Equivalent => Ok(BatchTransform::PassThrough), SchemaComparison::NameChangesOnly => Ok(BatchTransform::ModifySchema { target_schema }), @@ -336,8 +420,7 @@ impl RecordBatchTransformer { snapshot_schema, projected_iceberg_field_ids, field_id_to_mapped_schema_map, - constants_map, - partition_spec, + constant_fields, )?, target_schema, }), @@ -394,8 +477,7 @@ impl RecordBatchTransformer { snapshot_schema: &IcebergSchema, projected_iceberg_field_ids: &[i32], field_id_to_mapped_schema_map: HashMap, - constants_map: HashMap, - _partition_spec: Option<&PartitionSpec>, + constant_fields: &HashMap, ) -> Result> { let field_id_to_source_schema_map = Self::build_field_id_to_arrow_schema_map(source_schema)?; @@ -403,6 +485,18 @@ impl RecordBatchTransformer { projected_iceberg_field_ids .iter() .map(|field_id| { + // Check if this is a constant field (metadata/virtual or identity-partitioned) + // Constant fields always use their pre-computed constant values, regardless of whether + // they exist in the Parquet file. This is per Iceberg spec rule #1: partition metadata + // is authoritative and should be preferred over file data. + if let Some(datum) = constant_fields.get(field_id) { + let arrow_type = datum_to_arrow_type_with_ree(datum); + return Ok(ColumnSource::Add { + value: Some(datum.literal().clone()), + target_type: arrow_type, + }); + } + let (target_field, _) = field_id_to_mapped_schema_map .get(field_id) @@ -451,13 +545,8 @@ impl RecordBatchTransformer { ); // Apply spec's fallback steps for "not present" fields. - let column_source = if let Some(constant_value) = constants_map.get(field_id) { - // Rule #1: Identity partition constant - ColumnSource::Add { - value: Some(constant_value.clone()), - target_type: target_type.clone(), - } - } else if let Some(source) = field_by_id { + // Rule #1 (constants) is handled at the beginning of this function + let column_source = if let Some(source) = field_by_id { source } else { // Rules #2, #3 and #4: @@ -471,6 +560,7 @@ impl RecordBatchTransformer { None } }); + ColumnSource::Add { value: default_value, target_type: target_type.clone(), @@ -539,86 +629,36 @@ impl RecordBatchTransformer { prim_lit: &Option, num_rows: usize, ) -> Result { - Ok(match (target_type, prim_lit) { - (DataType::Boolean, Some(PrimitiveLiteral::Boolean(value))) => { - Arc::new(BooleanArray::from(vec![*value; num_rows])) - } - (DataType::Boolean, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(BooleanArray::from(vals)) - } - (DataType::Int32, Some(PrimitiveLiteral::Int(value))) => { - Arc::new(Int32Array::from(vec![*value; num_rows])) - } - (DataType::Int32, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(Int32Array::from(vals)) - } - (DataType::Date32, Some(PrimitiveLiteral::Int(value))) => { - Arc::new(Date32Array::from(vec![*value; num_rows])) - } - (DataType::Date32, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(Date32Array::from(vals)) - } - (DataType::Int64, Some(PrimitiveLiteral::Long(value))) => { - Arc::new(Int64Array::from(vec![*value; num_rows])) - } - (DataType::Int64, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(Int64Array::from(vals)) - } - (DataType::Float32, Some(PrimitiveLiteral::Float(value))) => { - Arc::new(Float32Array::from(vec![value.0; num_rows])) - } - (DataType::Float32, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(Float32Array::from(vals)) - } - (DataType::Float64, Some(PrimitiveLiteral::Double(value))) => { - Arc::new(Float64Array::from(vec![value.0; num_rows])) - } - (DataType::Float64, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(Float64Array::from(vals)) - } - (DataType::Utf8, Some(PrimitiveLiteral::String(value))) => { - Arc::new(StringArray::from(vec![value.clone(); num_rows])) - } - (DataType::Utf8, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(StringArray::from(vals)) - } - (DataType::Binary, Some(PrimitiveLiteral::Binary(value))) => { - Arc::new(BinaryArray::from_vec(vec![value; num_rows])) - } - (DataType::Binary, None) => { - let vals: Vec> = vec![None; num_rows]; - Arc::new(BinaryArray::from_opt_vec(vals)) - } - (DataType::Struct(fields), None) => { - // Create a StructArray filled with nulls. Per Iceberg spec, optional struct fields - // default to null when added to the schema. We defer non-null default struct values - // and leave them as not implemented yet. - let null_arrays: Vec = fields - .iter() - .map(|field| Self::create_column(field.data_type(), &None, num_rows)) - .collect::>>()?; - - Arc::new(StructArray::new( - fields.clone(), - null_arrays, - Some(NullBuffer::new_null(num_rows)), + // Check if this is a RunEndEncoded type (for constant fields) + if let DataType::RunEndEncoded(_, values_field) = target_type { + // Helper to create a Run-End Encoded array + let create_ree_array = |values_array: ArrayRef| -> Result { + let run_ends = if num_rows == 0 { + Int32Array::from(Vec::::new()) + } else { + Int32Array::from(vec![num_rows as i32]) + }; + Ok(Arc::new( + RunArray::try_new(&run_ends, &values_array).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + "Failed to create RunArray for constant value", + ) + .with_source(e) + })?, )) - } - (DataType::Null, _) => Arc::new(NullArray::new(num_rows)), - (dt, _) => { - return Err(Error::new( - ErrorKind::Unexpected, - format!("unexpected target column type {}", dt), - )); - } - }) + }; + + // Create the values array using the helper function + let values_array = + create_primitive_array_single_element(values_field.data_type(), prim_lit)?; + + // Wrap in Run-End Encoding + create_ree_array(values_array) + } else { + // Non-REE type (simple arrays for non-constant fields) + create_primitive_array_repeated(target_type, prim_lit, num_rows) + } } } @@ -639,6 +679,54 @@ mod test { }; use crate::spec::{Literal, NestedField, PrimitiveType, Schema, Struct, Type}; + /// Helper to extract string values from either StringArray or RunEndEncoded + /// Returns empty string for null values + fn get_string_value(array: &dyn Array, index: usize) -> String { + if let Some(string_array) = array.as_any().downcast_ref::() { + if string_array.is_null(index) { + String::new() + } else { + string_array.value(index).to_string() + } + } else if let Some(run_array) = array + .as_any() + .downcast_ref::>() + { + let values = run_array.values(); + let string_values = values + .as_any() + .downcast_ref::() + .expect("REE values should be StringArray"); + // For REE, all rows have the same value (index 0 in the values array) + if string_values.is_null(0) { + String::new() + } else { + string_values.value(0).to_string() + } + } else { + panic!("Expected StringArray or RunEndEncoded"); + } + } + + /// Helper to extract int values from either Int32Array or RunEndEncoded + fn get_int_value(array: &dyn Array, index: usize) -> i32 { + if let Some(int_array) = array.as_any().downcast_ref::() { + int_array.value(index) + } else if let Some(run_array) = array + .as_any() + .downcast_ref::>() + { + let values = run_array.values(); + let int_values = values + .as_any() + .downcast_ref::() + .expect("REE values should be Int32Array"); + int_values.value(0) + } else { + panic!("Expected Int32Array or RunEndEncoded"); + } + } + #[test] fn build_field_id_to_source_schema_map_works() { let arrow_schema = arrow_schema_already_same_as_target(); @@ -1137,6 +1225,7 @@ mod test { let mut transformer = RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) .with_partition(partition_spec, partition_data) + .expect("Failed to add partition constants") .build(); // Create a Parquet RecordBatch with actual data @@ -1257,6 +1346,7 @@ mod test { let mut transformer = RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) .with_partition(partition_spec, partition_data) + .expect("Failed to add partition constants") .build(); let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ @@ -1271,30 +1361,23 @@ mod test { assert_eq!(result.num_columns(), 3); assert_eq!(result.num_rows(), 2); - let id_column = result - .column(0) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(id_column.value(0), 100); - assert_eq!(id_column.value(1), 200); + // Use helpers to handle both simple and REE arrays + assert_eq!(get_int_value(result.column(0).as_ref(), 0), 100); + assert_eq!(get_int_value(result.column(0).as_ref(), 1), 200); - let dept_column = result - .column(1) - .as_any() - .downcast_ref::() - .unwrap(); - // This value MUST come from partition metadata (constant) - assert_eq!(dept_column.value(0), "engineering"); - assert_eq!(dept_column.value(1), "engineering"); + // dept column comes from partition metadata (constant) - will be REE + assert_eq!( + get_string_value(result.column(1).as_ref(), 0), + "engineering" + ); + assert_eq!( + get_string_value(result.column(1).as_ref(), 1), + "engineering" + ); - let name_column = result - .column(2) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(name_column.value(0), "Alice"); - assert_eq!(name_column.value(1), "Bob"); + // name column comes from file + assert_eq!(get_string_value(result.column(2).as_ref(), 0), "Alice"); + assert_eq!(get_string_value(result.column(2).as_ref(), 1), "Bob"); } /// Test bucket partitioning with renamed source column. @@ -1372,6 +1455,7 @@ mod test { let mut transformer = RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) .with_partition(partition_spec, partition_data) + .expect("Failed to add partition constants") .build(); // Create a Parquet RecordBatch with actual data @@ -1476,6 +1560,7 @@ mod test { let mut transformer = RecordBatchTransformerBuilder::new(snapshot_schema, &projected_field_ids) .with_partition(partition_spec, partition_data) + .expect("Failed to add partition constants") .build(); let parquet_batch = RecordBatch::try_new(parquet_schema, vec![ @@ -1492,48 +1577,37 @@ mod test { // Verify each column demonstrates the correct spec rule: // Normal case: id from Parquet by field ID - let id_column = result - .column(0) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(id_column.value(0), 100); - assert_eq!(id_column.value(1), 200); + // Use helpers to handle both simple and REE arrays + assert_eq!(get_int_value(result.column(0).as_ref(), 0), 100); + assert_eq!(get_int_value(result.column(0).as_ref(), 1), 200); + + // Rule #1: dept from partition metadata (identity transform) - will be REE + assert_eq!( + get_string_value(result.column(1).as_ref(), 0), + "engineering" + ); + assert_eq!( + get_string_value(result.column(1).as_ref(), 1), + "engineering" + ); - // Rule #1: dept from partition metadata (identity transform) - let dept_column = result - .column(1) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(dept_column.value(0), "engineering"); - assert_eq!(dept_column.value(1), "engineering"); + // Rule #2: data from Parquet via name mapping - will be regular array + assert_eq!(get_string_value(result.column(2).as_ref(), 0), "value1"); + assert_eq!(get_string_value(result.column(2).as_ref(), 1), "value2"); - // Rule #2: data from Parquet via name mapping - let data_column = result - .column(2) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(data_column.value(0), "value1"); - assert_eq!(data_column.value(1), "value2"); - - // Rule #3: category from initial_default - let category_column = result - .column(3) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(category_column.value(0), "default_category"); - assert_eq!(category_column.value(1), "default_category"); + // Rule #3: category from initial_default - will be REE + assert_eq!( + get_string_value(result.column(3).as_ref(), 0), + "default_category" + ); + assert_eq!( + get_string_value(result.column(3).as_ref(), 1), + "default_category" + ); - // Rule #4: notes is null (no default, not in Parquet, not in partition) - let notes_column = result - .column(4) - .as_any() - .downcast_ref::() - .unwrap(); - assert!(notes_column.is_null(0)); - assert!(notes_column.is_null(1)); + // Rule #4: notes is null (no default, not in Parquet, not in partition) - will be REE with null + // For null REE arrays, we still use the helper which handles extraction + assert_eq!(get_string_value(result.column(4).as_ref(), 0), ""); + assert_eq!(get_string_value(result.column(4).as_ref(), 1), ""); } } diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index ec0135bd77..4f4f083c73 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -1019,6 +1019,60 @@ impl TryFrom<&crate::spec::Schema> for ArrowSchema { } } +/// Converts a Datum (Iceberg type + primitive literal) to its corresponding Arrow DataType +/// with Run-End Encoding (REE). +/// +/// This function is used for constant fields in record batches, where all values are the same. +/// Run-End Encoding provides efficient storage for such constant columns. +/// +/// # Arguments +/// * `datum` - The Datum to convert, which contains both type and value information +/// +/// # Returns +/// Arrow DataType with Run-End Encoding applied +/// +/// # Example +/// ``` +/// use iceberg::arrow::datum_to_arrow_type_with_ree; +/// use iceberg::spec::Datum; +/// +/// let datum = Datum::string("test_file.parquet"); +/// let ree_type = datum_to_arrow_type_with_ree(&datum); +/// // Returns: RunEndEncoded(Int32, Utf8) +/// ``` +pub fn datum_to_arrow_type_with_ree(datum: &Datum) -> DataType { + // Helper to create REE type with the given values type. + // Note: values field is nullable as Arrow expects this when building the + // final Arrow schema with `RunArray::try_new`. + let make_ree = |values_type: DataType| -> DataType { + let run_ends_field = Arc::new(Field::new("run_ends", DataType::Int32, false)); + let values_field = Arc::new(Field::new("values", values_type, true)); + DataType::RunEndEncoded(run_ends_field, values_field) + }; + + // Match on the PrimitiveType from the Datum to determine the Arrow type + match datum.data_type() { + PrimitiveType::Boolean => make_ree(DataType::Boolean), + PrimitiveType::Int => make_ree(DataType::Int32), + PrimitiveType::Long => make_ree(DataType::Int64), + PrimitiveType::Float => make_ree(DataType::Float32), + PrimitiveType::Double => make_ree(DataType::Float64), + PrimitiveType::Date => make_ree(DataType::Date32), + PrimitiveType::Time => make_ree(DataType::Int64), + PrimitiveType::Timestamp => make_ree(DataType::Int64), + PrimitiveType::Timestamptz => make_ree(DataType::Int64), + PrimitiveType::TimestampNs => make_ree(DataType::Int64), + PrimitiveType::TimestamptzNs => make_ree(DataType::Int64), + PrimitiveType::String => make_ree(DataType::Utf8), + PrimitiveType::Uuid => make_ree(DataType::Binary), + PrimitiveType::Fixed(_) => make_ree(DataType::Binary), + PrimitiveType::Binary => make_ree(DataType::Binary), + PrimitiveType::Decimal { precision, scale } => { + make_ree(DataType::Decimal128(*precision as u8, *scale as i8)) + } + } +} + #[cfg(test)] mod tests { use std::collections::HashMap; diff --git a/crates/iceberg/src/arrow/value.rs b/crates/iceberg/src/arrow/value.rs index f1cf225bb4..0e0b85f073 100644 --- a/crates/iceberg/src/arrow/value.rs +++ b/crates/iceberg/src/arrow/value.rs @@ -15,18 +15,21 @@ // specific language governing permissions and limitations // under the License. +use std::sync::Arc; + use arrow_array::{ Array, ArrayRef, BinaryArray, BooleanArray, Date32Array, Decimal128Array, FixedSizeBinaryArray, FixedSizeListArray, Float32Array, Float64Array, Int32Array, Int64Array, LargeBinaryArray, LargeListArray, LargeStringArray, ListArray, MapArray, StringArray, StructArray, Time64MicrosecondArray, TimestampMicrosecondArray, TimestampNanosecondArray, }; +use arrow_buffer::NullBuffer; use arrow_schema::{DataType, FieldRef}; use uuid::Uuid; use super::get_field_id; use crate::spec::{ - ListType, Literal, Map, MapType, NestedField, PartnerAccessor, PrimitiveType, + ListType, Literal, Map, MapType, NestedField, PartnerAccessor, PrimitiveLiteral, PrimitiveType, SchemaWithPartnerVisitor, Struct, StructType, Type, visit_struct_with_partner, visit_type_with_partner, }; @@ -617,6 +620,217 @@ pub fn arrow_primitive_to_literal( ) } +/// Create a single-element array from a primitive literal. +/// +/// This is used for creating constant arrays (Run-End Encoded arrays) where we need +/// a single value that represents all rows. +pub(crate) fn create_primitive_array_single_element( + data_type: &DataType, + prim_lit: &Option, +) -> Result { + match (data_type, prim_lit) { + (DataType::Boolean, Some(PrimitiveLiteral::Boolean(v))) => { + Ok(Arc::new(BooleanArray::from(vec![*v]))) + } + (DataType::Boolean, None) => Ok(Arc::new(BooleanArray::from(vec![Option::::None]))), + (DataType::Int32, Some(PrimitiveLiteral::Int(v))) => { + Ok(Arc::new(Int32Array::from(vec![*v]))) + } + (DataType::Int32, None) => Ok(Arc::new(Int32Array::from(vec![Option::::None]))), + (DataType::Date32, Some(PrimitiveLiteral::Int(v))) => { + Ok(Arc::new(Date32Array::from(vec![*v]))) + } + (DataType::Date32, None) => Ok(Arc::new(Date32Array::from(vec![Option::::None]))), + (DataType::Int64, Some(PrimitiveLiteral::Long(v))) => { + Ok(Arc::new(Int64Array::from(vec![*v]))) + } + (DataType::Int64, None) => Ok(Arc::new(Int64Array::from(vec![Option::::None]))), + (DataType::Float32, Some(PrimitiveLiteral::Float(v))) => { + Ok(Arc::new(Float32Array::from(vec![v.0]))) + } + (DataType::Float32, None) => Ok(Arc::new(Float32Array::from(vec![Option::::None]))), + (DataType::Float64, Some(PrimitiveLiteral::Double(v))) => { + Ok(Arc::new(Float64Array::from(vec![v.0]))) + } + (DataType::Float64, None) => Ok(Arc::new(Float64Array::from(vec![Option::::None]))), + (DataType::Utf8, Some(PrimitiveLiteral::String(v))) => { + Ok(Arc::new(StringArray::from(vec![v.as_str()]))) + } + (DataType::Utf8, None) => Ok(Arc::new(StringArray::from(vec![Option::<&str>::None]))), + (DataType::Binary, Some(PrimitiveLiteral::Binary(v))) => { + Ok(Arc::new(BinaryArray::from_vec(vec![v.as_slice()]))) + } + (DataType::Binary, None) => Ok(Arc::new(BinaryArray::from_opt_vec(vec![ + Option::<&[u8]>::None, + ]))), + (DataType::Decimal128(_, _), Some(PrimitiveLiteral::Int128(v))) => { + Ok(Arc::new(arrow_array::Decimal128Array::from(vec![{ *v }]))) + } + (DataType::Decimal128(_, _), Some(PrimitiveLiteral::UInt128(v))) => { + Ok(Arc::new(arrow_array::Decimal128Array::from(vec![ + *v as i128, + ]))) + } + (DataType::Decimal128(_, _), None) => { + Ok(Arc::new(arrow_array::Decimal128Array::from(vec![ + Option::::None, + ]))) + } + (DataType::Struct(fields), None) => { + // Create a single-element StructArray with nulls + let null_arrays: Vec = fields + .iter() + .map(|f| { + // Recursively create null arrays for struct fields + // For primitive fields in structs, use simple null arrays (not REE within struct) + match f.data_type() { + DataType::Boolean => { + Ok(Arc::new(BooleanArray::from(vec![Option::::None])) + as ArrayRef) + } + DataType::Int32 | DataType::Date32 => { + Ok(Arc::new(Int32Array::from(vec![Option::::None])) as ArrayRef) + } + DataType::Int64 => { + Ok(Arc::new(Int64Array::from(vec![Option::::None])) as ArrayRef) + } + DataType::Float32 => { + Ok(Arc::new(Float32Array::from(vec![Option::::None])) as ArrayRef) + } + DataType::Float64 => { + Ok(Arc::new(Float64Array::from(vec![Option::::None])) as ArrayRef) + } + DataType::Utf8 => { + Ok(Arc::new(StringArray::from(vec![Option::<&str>::None])) as ArrayRef) + } + DataType::Binary => { + Ok( + Arc::new(BinaryArray::from_opt_vec(vec![Option::<&[u8]>::None])) + as ArrayRef, + ) + } + _ => Err(Error::new( + ErrorKind::Unexpected, + format!("Unsupported struct field type: {:?}", f.data_type()), + )), + } + }) + .collect::>>()?; + Ok(Arc::new(arrow_array::StructArray::new( + fields.clone(), + null_arrays, + Some(arrow_buffer::NullBuffer::new_null(1)), + ))) + } + _ => Err(Error::new( + ErrorKind::Unexpected, + format!( + "Unsupported constant type combination: {:?} with {:?}", + data_type, prim_lit + ), + )), + } +} + +/// Create a repeated array from a primitive literal for a given number of rows. +/// +/// This is used for creating non-constant arrays where we need the same value +/// repeated for each row. +pub(crate) fn create_primitive_array_repeated( + data_type: &DataType, + prim_lit: &Option, + num_rows: usize, +) -> Result { + Ok(match (data_type, prim_lit) { + (DataType::Boolean, Some(PrimitiveLiteral::Boolean(value))) => { + Arc::new(BooleanArray::from(vec![*value; num_rows])) + } + (DataType::Boolean, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(BooleanArray::from(vals)) + } + (DataType::Int32, Some(PrimitiveLiteral::Int(value))) => { + Arc::new(Int32Array::from(vec![*value; num_rows])) + } + (DataType::Int32, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Int32Array::from(vals)) + } + (DataType::Date32, Some(PrimitiveLiteral::Int(value))) => { + Arc::new(Date32Array::from(vec![*value; num_rows])) + } + (DataType::Date32, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Date32Array::from(vals)) + } + (DataType::Int64, Some(PrimitiveLiteral::Long(value))) => { + Arc::new(Int64Array::from(vec![*value; num_rows])) + } + (DataType::Int64, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Int64Array::from(vals)) + } + (DataType::Float32, Some(PrimitiveLiteral::Float(value))) => { + Arc::new(Float32Array::from(vec![value.0; num_rows])) + } + (DataType::Float32, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Float32Array::from(vals)) + } + (DataType::Float64, Some(PrimitiveLiteral::Double(value))) => { + Arc::new(Float64Array::from(vec![value.0; num_rows])) + } + (DataType::Float64, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Float64Array::from(vals)) + } + (DataType::Utf8, Some(PrimitiveLiteral::String(value))) => { + Arc::new(StringArray::from(vec![value.clone(); num_rows])) + } + (DataType::Utf8, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(StringArray::from(vals)) + } + (DataType::Binary, Some(PrimitiveLiteral::Binary(value))) => { + Arc::new(BinaryArray::from_vec(vec![value; num_rows])) + } + (DataType::Binary, None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(BinaryArray::from_opt_vec(vals)) + } + (DataType::Decimal128(_, _), Some(PrimitiveLiteral::Int128(value))) => { + Arc::new(Decimal128Array::from(vec![*value; num_rows])) + } + (DataType::Decimal128(_, _), Some(PrimitiveLiteral::UInt128(value))) => { + Arc::new(Decimal128Array::from(vec![*value as i128; num_rows])) + } + (DataType::Decimal128(_, _), None) => { + let vals: Vec> = vec![None; num_rows]; + Arc::new(Decimal128Array::from(vals)) + } + (DataType::Struct(fields), None) => { + // Create a StructArray filled with nulls + let null_arrays: Vec = fields + .iter() + .map(|field| create_primitive_array_repeated(field.data_type(), &None, num_rows)) + .collect::>>()?; + + Arc::new(StructArray::new( + fields.clone(), + null_arrays, + Some(NullBuffer::new_null(num_rows)), + )) + } + (DataType::Null, _) => Arc::new(arrow_array::NullArray::new(num_rows)), + (dt, _) => { + return Err(Error::new( + ErrorKind::Unexpected, + format!("unexpected target column type {}", dt), + )); + } + }) +} + #[cfg(test)] mod test { use std::collections::HashMap; diff --git a/crates/iceberg/src/lib.rs b/crates/iceberg/src/lib.rs index aae8efed74..8d8f40f72d 100644 --- a/crates/iceberg/src/lib.rs +++ b/crates/iceberg/src/lib.rs @@ -96,4 +96,5 @@ mod utils; pub mod writer; mod delete_vector; +pub mod metadata_columns; pub mod puffin; diff --git a/crates/iceberg/src/metadata_columns.rs b/crates/iceberg/src/metadata_columns.rs new file mode 100644 index 0000000000..b11b5cadb2 --- /dev/null +++ b/crates/iceberg/src/metadata_columns.rs @@ -0,0 +1,127 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Metadata columns (virtual/reserved fields) for Iceberg tables. +//! +//! This module defines metadata columns that can be requested in projections +//! but are not stored in data files. Instead, they are computed on-the-fly +//! during reading. Examples include the _file column (file path) and future +//! columns like partition values or row numbers. + +use std::sync::Arc; + +use once_cell::sync::Lazy; + +use crate::spec::{NestedField, NestedFieldRef, PrimitiveType, Type}; +use crate::{Error, ErrorKind, Result}; + +/// Reserved field ID for the file path (_file) column per Iceberg spec +pub const RESERVED_FIELD_ID_FILE: i32 = i32::MAX - 1; + +/// Reserved column name for the file path metadata column +pub const RESERVED_COL_NAME_FILE: &str = "_file"; + +/// Documentation for the _file metadata column +pub const RESERVED_COL_DOC_FILE: &str = "Path of the file in which a row is stored"; + +/// Lazy-initialized Iceberg field definition for the _file metadata column. +/// This field represents the file path as a required string field. +static FILE_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_FILE, + RESERVED_COL_NAME_FILE, + Type::Primitive(PrimitiveType::String), + ) + .with_doc(RESERVED_COL_DOC_FILE), + ) +}); + +/// Returns the Iceberg field definition for the _file metadata column. +/// +/// # Returns +/// A reference to the _file field definition as an Iceberg NestedField +pub fn file_field() -> &'static NestedFieldRef { + &FILE_FIELD +} + +/// Returns the Iceberg field definition for a metadata field ID. +/// +/// # Arguments +/// * `field_id` - The metadata field ID +/// +/// # Returns +/// The Iceberg field definition for the metadata column, or an error if not a metadata field +pub fn get_metadata_field(field_id: i32) -> Result { + match field_id { + RESERVED_FIELD_ID_FILE => Ok(Arc::clone(file_field())), + _ if is_metadata_field(field_id) => { + // Future metadata fields can be added here + Err(Error::new( + ErrorKind::Unexpected, + format!( + "Metadata field ID {} recognized but field definition not implemented", + field_id + ), + )) + } + _ => Err(Error::new( + ErrorKind::Unexpected, + format!("Field ID {} is not a metadata field", field_id), + )), + } +} + +/// Returns the field ID for a metadata column name. +/// +/// # Arguments +/// * `column_name` - The metadata column name +/// +/// # Returns +/// The field ID of the metadata column, or an error if the column name is not recognized +pub fn get_metadata_field_id(column_name: &str) -> Result { + match column_name { + RESERVED_COL_NAME_FILE => Ok(RESERVED_FIELD_ID_FILE), + _ => Err(Error::new( + ErrorKind::Unexpected, + format!("Unknown/unsupported metadata column name: {column_name}"), + )), + } +} + +/// Checks if a field ID is a metadata field. +/// +/// # Arguments +/// * `field_id` - The field ID to check +/// +/// # Returns +/// `true` if the field ID is a (currently supported) metadata field, `false` otherwise +pub fn is_metadata_field(field_id: i32) -> bool { + field_id == RESERVED_FIELD_ID_FILE + // Additional metadata fields can be checked here in the future +} + +/// Checks if a column name is a metadata column. +/// +/// # Arguments +/// * `column_name` - The column name to check +/// +/// # Returns +/// `true` if the column name is a metadata column, `false` otherwise +pub fn is_metadata_column_name(column_name: &str) -> bool { + get_metadata_field_id(column_name).is_ok() +} diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 3e319ca062..24c03b0b2c 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -36,6 +36,7 @@ use crate::delete_file_index::DeleteFileIndex; use crate::expr::visitors::inclusive_metrics_evaluator::InclusiveMetricsEvaluator; use crate::expr::{Bind, BoundPredicate, Predicate}; use crate::io::FileIO; +use crate::metadata_columns::{get_metadata_field_id, is_metadata_column_name}; use crate::runtime::spawn; use crate::spec::{DataContentType, SnapshotRef}; use crate::table::Table; @@ -217,9 +218,13 @@ impl<'a> TableScanBuilder<'a> { let schema = snapshot.schema(self.table.metadata())?; - // Check that all column names exist in the schema. + // Check that all column names exist in the schema (skip reserved columns). if let Some(column_names) = self.column_names.as_ref() { for column_name in column_names { + // Skip reserved columns that don't exist in the schema + if is_metadata_column_name(column_name) { + continue; + } if schema.field_by_name(column_name).is_none() { return Err(Error::new( ErrorKind::DataInvalid, @@ -240,6 +245,12 @@ impl<'a> TableScanBuilder<'a> { }); for column_name in column_names.iter() { + // Handle metadata columns (like "_file") + if is_metadata_column_name(column_name) { + field_ids.push(get_metadata_field_id(column_name)?); + continue; + } + let field_id = schema.field_id_by_name(column_name).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, @@ -254,10 +265,10 @@ impl<'a> TableScanBuilder<'a> { Error::new( ErrorKind::FeatureUnsupported, format!( - "Column {column_name} is not a direct child of schema but a nested field, which is not supported now. Schema: {schema}" - ), - ) - })?; + "Column {column_name} is not a direct child of schema but a nested field, which is not supported now. Schema: {schema}" + ), + ) + })?; field_ids.push(field_id); } @@ -559,8 +570,10 @@ pub mod tests { use std::fs::File; use std::sync::Arc; + use arrow_array::cast::AsArray; use arrow_array::{ - ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray, + Array, ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, + StringArray, }; use futures::{TryStreamExt, stream}; use minijinja::value::Value; @@ -575,6 +588,7 @@ pub mod tests { use crate::arrow::ArrowReaderBuilder; use crate::expr::{BoundPredicate, Reference}; use crate::io::{FileIO, OutputFile}; + use crate::metadata_columns::RESERVED_COL_NAME_FILE; use crate::scan::FileScanTask; use crate::spec::{ DataContentType, DataFileBuilder, DataFileFormat, Datum, Literal, ManifestEntry, @@ -1800,4 +1814,319 @@ pub mod tests { }; test_fn(task); } + + #[tokio::test] + async fn test_select_with_file_column() { + use arrow_array::cast::AsArray; + + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select regular columns plus the _file column + let table_scan = fixture + .table + .scan() + .select(["x", RESERVED_COL_NAME_FILE]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + // Verify we have 2 columns: x and _file + assert_eq!(batches[0].num_columns(), 2); + + // Verify the x column exists and has correct data + let x_col = batches[0].column_by_name("x").unwrap(); + let x_arr = x_col.as_primitive::(); + assert_eq!(x_arr.value(0), 1); + + // Verify the _file column exists + let file_col = batches[0].column_by_name(RESERVED_COL_NAME_FILE); + assert!( + file_col.is_some(), + "_file column should be present in the batch" + ); + + // Verify the _file column contains a file path + let file_col = file_col.unwrap(); + assert!( + matches!( + file_col.data_type(), + arrow_schema::DataType::RunEndEncoded(_, _) + ), + "_file column should use RunEndEncoded type" + ); + + // Decode the RunArray to verify it contains the file path + let run_array = file_col + .as_any() + .downcast_ref::>() + .expect("_file column should be a RunArray"); + + let values = run_array.values(); + let string_values = values.as_string::(); + assert_eq!(string_values.len(), 1, "Should have a single file path"); + + let file_path = string_values.value(0); + assert!( + file_path.ends_with(".parquet"), + "File path should end with .parquet, got: {}", + file_path + ); + } + + #[tokio::test] + async fn test_select_file_column_position() { + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select columns in specific order: x, _file, z + let table_scan = fixture + .table + .scan() + .select(["x", RESERVED_COL_NAME_FILE, "z"]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + assert_eq!(batches[0].num_columns(), 3); + + // Verify column order: x at position 0, _file at position 1, z at position 2 + let schema = batches[0].schema(); + assert_eq!(schema.field(0).name(), "x"); + assert_eq!(schema.field(1).name(), RESERVED_COL_NAME_FILE); + assert_eq!(schema.field(2).name(), "z"); + + // Verify columns by name also works + assert!(batches[0].column_by_name("x").is_some()); + assert!(batches[0].column_by_name(RESERVED_COL_NAME_FILE).is_some()); + assert!(batches[0].column_by_name("z").is_some()); + } + + #[tokio::test] + async fn test_select_file_column_only() { + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select only the _file column + let table_scan = fixture + .table + .scan() + .select([RESERVED_COL_NAME_FILE]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + // Should have exactly 1 column + assert_eq!(batches[0].num_columns(), 1); + + // Verify it's the _file column + let schema = batches[0].schema(); + assert_eq!(schema.field(0).name(), RESERVED_COL_NAME_FILE); + + // Verify the batch has the correct number of rows + // The scan reads files 1.parquet and 3.parquet (2.parquet is deleted) + // Each file has 1024 rows, so total is 2048 rows + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 2048); + } + + #[tokio::test] + async fn test_file_column_with_multiple_files() { + use std::collections::HashSet; + + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select x and _file columns + let table_scan = fixture + .table + .scan() + .select(["x", RESERVED_COL_NAME_FILE]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + // Collect all unique file paths from the batches + let mut file_paths = HashSet::new(); + for batch in &batches { + let file_col = batch.column_by_name(RESERVED_COL_NAME_FILE).unwrap(); + let run_array = file_col + .as_any() + .downcast_ref::>() + .expect("_file column should be a RunArray"); + + let values = run_array.values(); + let string_values = values.as_string::(); + for i in 0..string_values.len() { + file_paths.insert(string_values.value(i).to_string()); + } + } + + // We should have multiple files (the test creates 1.parquet and 3.parquet) + assert!(!file_paths.is_empty(), "Should have at least one file path"); + + // All paths should end with .parquet + for path in &file_paths { + assert!( + path.ends_with(".parquet"), + "All file paths should end with .parquet, got: {}", + path + ); + } + } + + #[tokio::test] + async fn test_file_column_at_start() { + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select _file at the start + let table_scan = fixture + .table + .scan() + .select([RESERVED_COL_NAME_FILE, "x", "y"]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + assert_eq!(batches[0].num_columns(), 3); + + // Verify _file is at position 0 + let schema = batches[0].schema(); + assert_eq!(schema.field(0).name(), RESERVED_COL_NAME_FILE); + assert_eq!(schema.field(1).name(), "x"); + assert_eq!(schema.field(2).name(), "y"); + } + + #[tokio::test] + async fn test_file_column_at_end() { + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select _file at the end + let table_scan = fixture + .table + .scan() + .select(["x", "y", RESERVED_COL_NAME_FILE]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + assert_eq!(batches[0].num_columns(), 3); + + // Verify _file is at position 2 (the end) + let schema = batches[0].schema(); + assert_eq!(schema.field(0).name(), "x"); + assert_eq!(schema.field(1).name(), "y"); + assert_eq!(schema.field(2).name(), RESERVED_COL_NAME_FILE); + } + + #[tokio::test] + async fn test_select_with_repeated_column_names() { + let mut fixture = TableTestFixture::new(); + fixture.setup_manifest_files().await; + + // Select with repeated column names - both regular columns and virtual columns + // Repeated columns should appear multiple times in the result (duplicates are allowed) + let table_scan = fixture + .table + .scan() + .select([ + "x", + RESERVED_COL_NAME_FILE, + "x", // x repeated + "y", + RESERVED_COL_NAME_FILE, // _file repeated + "y", // y repeated + ]) + .with_row_selection_enabled(true) + .build() + .unwrap(); + + let batch_stream = table_scan.to_arrow().await.unwrap(); + let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); + + // Verify we have exactly 6 columns (duplicates are allowed and preserved) + assert_eq!( + batches[0].num_columns(), + 6, + "Should have exactly 6 columns with duplicates" + ); + + let schema = batches[0].schema(); + + // Verify columns appear in the exact order requested: x, _file, x, y, _file, y + assert_eq!(schema.field(0).name(), "x", "Column 0 should be x"); + assert_eq!( + schema.field(1).name(), + RESERVED_COL_NAME_FILE, + "Column 1 should be _file" + ); + assert_eq!( + schema.field(2).name(), + "x", + "Column 2 should be x (duplicate)" + ); + assert_eq!(schema.field(3).name(), "y", "Column 3 should be y"); + assert_eq!( + schema.field(4).name(), + RESERVED_COL_NAME_FILE, + "Column 4 should be _file (duplicate)" + ); + assert_eq!( + schema.field(5).name(), + "y", + "Column 5 should be y (duplicate)" + ); + + // Verify all columns have correct data types + assert!( + matches!(schema.field(0).data_type(), arrow_schema::DataType::Int64), + "Column x should be Int64" + ); + assert!( + matches!(schema.field(2).data_type(), arrow_schema::DataType::Int64), + "Column x (duplicate) should be Int64" + ); + assert!( + matches!(schema.field(3).data_type(), arrow_schema::DataType::Int64), + "Column y should be Int64" + ); + assert!( + matches!(schema.field(5).data_type(), arrow_schema::DataType::Int64), + "Column y (duplicate) should be Int64" + ); + assert!( + matches!( + schema.field(1).data_type(), + arrow_schema::DataType::RunEndEncoded(_, _) + ), + "_file column should use RunEndEncoded type" + ); + assert!( + matches!( + schema.field(4).data_type(), + arrow_schema::DataType::RunEndEncoded(_, _) + ), + "_file column (duplicate) should use RunEndEncoded type" + ); + } } From d973fefb06bfb3555890492fad853dabd85fb186 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 9 Dec 2025 14:15:01 +0100 Subject: [PATCH 105/891] feat: Make `rest` types public, add documentation (#1901) --- crates/catalog/rest/src/catalog.rs | 52 ++--- crates/catalog/rest/src/lib.rs | 1 + crates/catalog/rest/src/types.rs | 307 ++++++++++++++++++++++------- 3 files changed, 253 insertions(+), 107 deletions(-) diff --git a/crates/catalog/rest/src/catalog.rs b/crates/catalog/rest/src/catalog.rs index 39553f7554..ddbf6a4e01 100644 --- a/crates/catalog/rest/src/catalog.rs +++ b/crates/catalog/rest/src/catalog.rs @@ -41,9 +41,9 @@ use crate::client::{ HttpClient, deserialize_catalog_response, deserialize_unexpected_catalog_error, }; use crate::types::{ - CatalogConfig, CommitTableRequest, CommitTableResponse, CreateTableRequest, - ListNamespaceResponse, ListTableResponse, LoadTableResponse, NamespaceSerde, - RegisterTableRequest, RenameTableRequest, + CatalogConfig, CommitTableRequest, CommitTableResponse, CreateNamespaceRequest, + CreateTableRequest, ListNamespaceResponse, ListTablesResponse, LoadTableResult, + NamespaceResponse, RegisterTableRequest, RenameTableRequest, }; /// REST catalog URI @@ -466,13 +466,7 @@ impl Catalog for RestCatalog { deserialize_catalog_response::(http_response) .await?; - let ns_identifiers = response - .namespaces - .into_iter() - .map(NamespaceIdent::from_vec) - .collect::>>()?; - - namespaces.extend(ns_identifiers); + namespaces.extend(response.namespaces); match response.next_page_token { Some(token) => next_token = Some(token), @@ -502,9 +496,9 @@ impl Catalog for RestCatalog { let request = context .client .request(Method::POST, context.config.namespaces_endpoint()) - .json(&NamespaceSerde { - namespace: namespace.as_ref().clone(), - properties: Some(properties), + .json(&CreateNamespaceRequest { + namespace: namespace.clone(), + properties, }) .build()?; @@ -513,8 +507,8 @@ impl Catalog for RestCatalog { match http_response.status() { StatusCode::OK => { let response = - deserialize_catalog_response::(http_response).await?; - Namespace::try_from(response) + deserialize_catalog_response::(http_response).await?; + Ok(Namespace::from(response)) } StatusCode::CONFLICT => Err(Error::new( ErrorKind::Unexpected, @@ -537,8 +531,8 @@ impl Catalog for RestCatalog { match http_response.status() { StatusCode::OK => { let response = - deserialize_catalog_response::(http_response).await?; - Namespace::try_from(response) + deserialize_catalog_response::(http_response).await?; + Ok(Namespace::from(response)) } StatusCode::NOT_FOUND => Err(Error::new( ErrorKind::Unexpected, @@ -614,7 +608,7 @@ impl Catalog for RestCatalog { match http_response.status() { StatusCode::OK => { let response = - deserialize_catalog_response::(http_response).await?; + deserialize_catalog_response::(http_response).await?; identifiers.extend(response.identifiers); @@ -661,11 +655,7 @@ impl Catalog for RestCatalog { partition_spec: creation.partition_spec, write_order: creation.sort_order, stage_create: Some(false), - properties: if creation.properties.is_empty() { - None - } else { - Some(creation.properties) - }, + properties: creation.properties, }) .build()?; @@ -673,7 +663,7 @@ impl Catalog for RestCatalog { let response = match http_response.status() { StatusCode::OK => { - deserialize_catalog_response::(http_response).await? + deserialize_catalog_response::(http_response).await? } StatusCode::NOT_FOUND => { return Err(Error::new( @@ -697,7 +687,6 @@ impl Catalog for RestCatalog { let config = response .config - .unwrap_or_default() .into_iter() .chain(self.user_config.props.clone()) .collect(); @@ -735,7 +724,7 @@ impl Catalog for RestCatalog { let response = match http_response.status() { StatusCode::OK | StatusCode::NOT_MODIFIED => { - deserialize_catalog_response::(http_response).await? + deserialize_catalog_response::(http_response).await? } StatusCode::NOT_FOUND => { return Err(Error::new( @@ -748,7 +737,6 @@ impl Catalog for RestCatalog { let config = response .config - .unwrap_or_default() .into_iter() .chain(self.user_config.props.clone()) .collect(); @@ -861,9 +849,9 @@ impl Catalog for RestCatalog { let http_response = context.client.query_catalog(request).await?; - let response: LoadTableResponse = match http_response.status() { + let response: LoadTableResult = match http_response.status() { StatusCode::OK => { - deserialize_catalog_response::(http_response).await? + deserialize_catalog_response::(http_response).await? } StatusCode::NOT_FOUND => { return Err(Error::new( @@ -905,7 +893,7 @@ impl Catalog for RestCatalog { context.config.table_endpoint(commit.identifier()), ) .json(&CommitTableRequest { - identifier: commit.identifier().clone(), + identifier: Some(commit.identifier().clone()), requirements: commit.take_requirements(), updates: commit.take_updates(), }) @@ -2428,7 +2416,7 @@ mod tests { )) .unwrap(); let reader = BufReader::new(file); - let resp = serde_json::from_reader::<_, LoadTableResponse>(reader).unwrap(); + let resp = serde_json::from_reader::<_, LoadTableResult>(reader).unwrap(); Table::builder() .metadata(resp.metadata) @@ -2568,7 +2556,7 @@ mod tests { )) .unwrap(); let reader = BufReader::new(file); - let resp = serde_json::from_reader::<_, LoadTableResponse>(reader).unwrap(); + let resp = serde_json::from_reader::<_, LoadTableResult>(reader).unwrap(); Table::builder() .metadata(resp.metadata) diff --git a/crates/catalog/rest/src/lib.rs b/crates/catalog/rest/src/lib.rs index 70cdeaabd0..6bee950970 100644 --- a/crates/catalog/rest/src/lib.rs +++ b/crates/catalog/rest/src/lib.rs @@ -56,3 +56,4 @@ mod client; mod types; pub use catalog::*; +pub use types::*; diff --git a/crates/catalog/rest/src/types.rs b/crates/catalog/rest/src/types.rs index 70ed72051a..ab44c40ee3 100644 --- a/crates/catalog/rest/src/types.rs +++ b/crates/catalog/rest/src/types.rs @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +//! Request and response types for the Iceberg REST API. + use std::collections::HashMap; use iceberg::spec::{Schema, SortOrder, TableMetadata, UnboundPartitionSpec}; @@ -30,7 +32,8 @@ pub(super) struct CatalogConfig { } #[derive(Debug, Serialize, Deserialize)] -pub(super) struct ErrorResponse { +/// Wrapper for all non-2xx error responses from the REST API +pub struct ErrorResponse { error: ErrorModel, } @@ -41,11 +44,16 @@ impl From for Error { } #[derive(Debug, Serialize, Deserialize)] -pub(super) struct ErrorModel { - pub(super) message: String, - pub(super) r#type: String, - pub(super) code: u16, - pub(super) stack: Option>, +/// Error payload returned in a response with further details on the error +pub struct ErrorModel { + /// Human-readable error message + pub message: String, + /// Internal type definition of the error + pub r#type: String, + /// HTTP response code + pub code: u16, + /// Optional error stack / context + pub stack: Option>, } impl From for Error { @@ -96,106 +104,255 @@ pub(super) struct TokenResponse { pub(super) issued_token_type: Option, } -#[derive(Debug, Serialize, Deserialize)] -pub(super) struct NamespaceSerde { - pub(super) namespace: Vec, - pub(super) properties: Option>, -} - -impl TryFrom for Namespace { - type Error = Error; - fn try_from(value: NamespaceSerde) -> std::result::Result { - Ok(Namespace::with_properties( - NamespaceIdent::from_vec(value.namespace)?, - value.properties.unwrap_or_default(), - )) - } +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Namespace response +pub struct NamespaceResponse { + /// Namespace identifier + pub namespace: NamespaceIdent, + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + /// Properties stored on the namespace, if supported by the server. + pub properties: HashMap, } -impl From<&Namespace> for NamespaceSerde { +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Create namespace request +pub struct CreateNamespaceRequest { + /// Name of the namespace to create + pub namespace: NamespaceIdent, + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + /// Properties to set on the namespace + pub properties: HashMap, +} + +impl From<&Namespace> for NamespaceResponse { fn from(value: &Namespace) -> Self { Self { - namespace: value.name().as_ref().clone(), - properties: Some(value.properties().clone()), + namespace: value.name().clone(), + properties: value.properties().clone(), } } } -#[derive(Debug, Serialize, Deserialize)] +impl From for Namespace { + fn from(value: NamespaceResponse) -> Self { + Namespace::with_properties(value.namespace, value.properties) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct ListNamespaceResponse { - pub(super) namespaces: Vec>, - #[serde(default)] - pub(super) next_page_token: Option, +/// Response containing a list of namespace identifiers, with optional pagination support. +pub struct ListNamespaceResponse { + /// List of namespace identifiers returned by the server + pub namespaces: Vec, + /// Opaque token for pagination. If present, indicates there are more results available. + /// Use this value in subsequent requests to retrieve the next page. + pub next_page_token: Option, } -#[allow(dead_code)] -#[derive(Debug, Serialize, Deserialize)] -pub(super) struct UpdateNamespacePropsRequest { - removals: Option>, - updates: Option>, +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Request to update properties on a namespace. +/// +/// Properties that are not in the request are not modified or removed by this call. +/// Server implementations are not required to support namespace properties. +pub struct UpdateNamespacePropertiesRequest { + /// List of property keys to remove from the namespace + pub removals: Option>, + /// Map of property keys to values to set or update on the namespace + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub updates: HashMap, } -#[allow(dead_code)] -#[derive(Debug, Serialize, Deserialize)] -pub(super) struct UpdateNamespacePropsResponse { - updated: Vec, - removed: Vec, - missing: Option>, +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Response from updating namespace properties, indicating which properties were changed. +pub struct UpdateNamespacePropertiesResponse { + /// List of property keys that were added or updated + pub updated: Vec, + /// List of properties that were removed + pub removed: Vec, + /// List of properties requested for removal that were not found in the namespace's properties. + /// Represents a partial success response. Servers do not need to implement this. + #[serde(skip_serializing_if = "Option::is_none")] + pub missing: Option>, } -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct ListTableResponse { - pub(super) identifiers: Vec, +/// Response containing a list of table identifiers, with optional pagination support. +pub struct ListTablesResponse { + /// List of table identifiers under the requested namespace + pub identifiers: Vec, + /// Opaque token for pagination. If present, indicates there are more results available. + /// Use this value in subsequent requests to retrieve the next page. #[serde(default)] - pub(super) next_page_token: Option, + pub next_page_token: Option, } -#[derive(Debug, Serialize, Deserialize)] -pub(super) struct RenameTableRequest { - pub(super) source: TableIdent, - pub(super) destination: TableIdent, +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Request to rename a table from one identifier to another. +/// +/// It's valid to move a table across namespaces, but the server implementation +/// is not required to support it. +pub struct RenameTableRequest { + /// Current table identifier to rename + pub source: TableIdent, + /// New table identifier to rename to + pub destination: TableIdent, } -#[derive(Debug, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct LoadTableResponse { - pub(super) metadata_location: Option, - pub(super) metadata: TableMetadata, - pub(super) config: Option>, +/// Result returned when a table is successfully loaded or created. +/// +/// The table metadata JSON is returned in the `metadata` field. The corresponding file location +/// of table metadata should be returned in the `metadata_location` field, unless the metadata +/// is not yet committed. For example, a create transaction may return metadata that is staged +/// but not committed. +/// +/// The `config` map returns table-specific configuration for the table's resources, including +/// its HTTP client and FileIO. For example, config may contain a specific FileIO implementation +/// class for the table depending on its underlying storage. +pub struct LoadTableResult { + /// May be null if the table is staged as part of a transaction + pub metadata_location: Option, + /// The table's full metadata + pub metadata: TableMetadata, + /// Table-specific configuration overriding catalog configuration + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub config: HashMap, + /// Storage credentials for accessing table data. Clients should check this field + /// before falling back to credentials in the `config` field. + #[serde(skip_serializing_if = "Option::is_none")] + pub storage_credentials: Option>, } -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +/// Storage credential for a specific location prefix. +/// +/// Indicates a storage location prefix where the credential is relevant. Clients should +/// choose the most specific prefix (by selecting the longest prefix) if several credentials +/// of the same type are available. +pub struct StorageCredential { + /// Storage location prefix where this credential is relevant + pub prefix: String, + /// Configuration map containing credential information + pub config: HashMap, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct CreateTableRequest { - pub(super) name: String, - pub(super) location: Option, - pub(super) schema: Schema, - pub(super) partition_spec: Option, - pub(super) write_order: Option, - pub(super) stage_create: Option, - pub(super) properties: Option>, +/// Request to create a new table in a namespace. +/// +/// If `stage_create` is false, the table is created immediately. +/// If `stage_create` is true, the table is not created, but table metadata is initialized +/// and returned. The service should prepare as needed for a commit to the table commit +/// endpoint to complete the create transaction. +pub struct CreateTableRequest { + /// Name of the table to create + pub name: String, + /// Optional table location. If not provided, the server will choose a location. + pub location: Option, + /// Table schema + pub schema: Schema, + /// Optional partition specification. If not provided, the table will be unpartitioned. + pub partition_spec: Option, + /// Optional sort order for the table + pub write_order: Option, + /// Whether to stage the create for a transaction (true) or create immediately (false) + pub stage_create: Option, + /// Optional properties to set on the table + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub properties: HashMap, } -#[derive(Debug, Serialize, Deserialize)] -pub(super) struct CommitTableRequest { - pub(super) identifier: TableIdent, - pub(super) requirements: Vec, - pub(super) updates: Vec, +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +/// Request to commit updates to a table. +/// +/// Commits have two parts: requirements and updates. Requirements are assertions that will +/// be validated before attempting to make and commit changes. Updates are changes to make +/// to table metadata. +/// +/// Create table transactions that are started by createTable with `stage-create` set to true +/// are committed using this request. Transactions should include all changes to the table, +/// including table initialization, like AddSchemaUpdate and SetCurrentSchemaUpdate. +pub struct CommitTableRequest { + /// Table identifier to update; must be present for CommitTransactionRequest + #[serde(skip_serializing_if = "Option::is_none")] + pub identifier: Option, + /// List of requirements that must be satisfied before committing changes + pub requirements: Vec, + /// List of updates to apply to the table metadata + pub updates: Vec, } -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct CommitTableResponse { - pub(super) metadata_location: String, - pub(super) metadata: TableMetadata, +/// Response returned when a table is successfully updated. +/// +/// The table metadata JSON is returned in the metadata field. The corresponding file location +/// of table metadata must be returned in the metadata-location field. Clients can check whether +/// metadata has changed by comparing metadata locations. +pub struct CommitTableResponse { + /// Location of the updated table metadata file + pub metadata_location: String, + /// The table's updated metadata + pub metadata: TableMetadata, } -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] -pub(super) struct RegisterTableRequest { - pub(super) name: String, - pub(super) metadata_location: String, - pub(super) overwrite: Option, +/// Request to register a table using an existing metadata file location. +pub struct RegisterTableRequest { + /// Name of the table to register + pub name: String, + /// Location of the metadata file for the table + pub metadata_location: String, + /// Whether to overwrite table metadata if the table already exists + pub overwrite: Option, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_namespace_response_serde() { + let json = serde_json::json!({ + "namespace": ["nested", "ns"], + "properties": { + "key1": "value1", + "key2": "value2" + } + }); + let ns_response: NamespaceResponse = + serde_json::from_value(json.clone()).expect("Deserialization failed"); + assert_eq!(ns_response, NamespaceResponse { + namespace: NamespaceIdent::from_vec(vec!["nested".to_string(), "ns".to_string()]) + .unwrap(), + properties: HashMap::from([ + ("key1".to_string(), "value1".to_string()), + ("key2".to_string(), "value2".to_string()), + ]), + }); + assert_eq!( + serde_json::to_value(&ns_response).expect("Serialization failed"), + json + ); + + // Without properties + let json_no_props = serde_json::json!({ + "namespace": ["db", "schema"] + }); + let ns_response_no_props: NamespaceResponse = + serde_json::from_value(json_no_props.clone()).expect("Deserialization failed"); + assert_eq!(ns_response_no_props, NamespaceResponse { + namespace: NamespaceIdent::from_vec(vec!["db".to_string(), "schema".to_string()]) + .unwrap(), + properties: HashMap::new(), + }); + assert_eq!( + serde_json::to_value(&ns_response_no_props).expect("Serialization failed"), + json_no_props + ); + } } From d78f6567cabe917d75ae9884144a5a0f70e08ace Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 9 Dec 2025 09:57:36 -0500 Subject: [PATCH 106/891] chore: bump MSRV to 1.88, fix warnings and clippy errors (#1902) https://github.com/apache/iceberg-rust/pull/1899 requires a bump to MSRV 1.88. This version is within the policy of this project, and since the README mentions `MSRV is updated when we release iceberg-rust` and we're preparing 0.8, here's a PR for just MSRV 1.88. ## Which issue does this PR close? N/A. ## What changes are included in this PR? - Bump MSRV to 1.88 - Fix warnings - Fix errors found by `make check-clippy` - Format ## Are these changes tested? Existing tests --- Cargo.toml | 2 +- crates/catalog/glue/src/catalog.rs | 46 +++++------ crates/iceberg/src/arrow/reader.rs | 68 ++++++++-------- .../src/arrow/record_batch_projector.rs | 35 ++++----- .../src/arrow/record_batch_transformer.rs | 2 +- crates/iceberg/src/arrow/value.rs | 33 ++++---- crates/iceberg/src/catalog/mod.rs | 14 ++-- crates/iceberg/src/delete_vector.rs | 10 +-- .../src/expr/visitors/manifest_evaluator.rs | 26 +++---- .../src/expr/visitors/page_index_evaluator.rs | 16 ++-- .../expr/visitors/strict_metrics_evaluator.rs | 24 +++--- crates/iceberg/src/inspect/metadata_table.rs | 4 +- crates/iceberg/src/io/storage.rs | 4 +- crates/iceberg/src/io/storage_azdls.rs | 20 ++--- crates/iceberg/src/io/storage_gcs.rs | 26 +++---- crates/iceberg/src/io/storage_oss.rs | 2 +- crates/iceberg/src/io/storage_s3.rs | 24 +++--- crates/iceberg/src/metadata_columns.rs | 5 +- crates/iceberg/src/scan/mod.rs | 6 +- crates/iceberg/src/spec/datatypes.rs | 3 +- crates/iceberg/src/spec/manifest/writer.rs | 8 +- .../iceberg/src/spec/schema/prune_columns.rs | 34 ++++---- crates/iceberg/src/spec/table_metadata.rs | 71 +++++++++-------- crates/iceberg/src/spec/transform.rs | 78 +++++++++---------- crates/iceberg/src/spec/values/tests.rs | 2 +- .../iceberg/src/spec/view_metadata_builder.rs | 8 +- crates/iceberg/src/transaction/mod.rs | 2 +- crates/iceberg/src/transaction/snapshot.rs | 14 ++-- .../src/writer/file_writer/rolling_writer.rs | 24 +++--- crates/iceberg/tests/file_io_gcs_test.rs | 6 +- .../src/physical_plan/repartition.rs | 11 +-- .../datafusion/src/physical_plan/sort.rs | 3 +- .../integrations/datafusion/src/table/mod.rs | 2 +- .../tests/integration_datafusion_test.rs | 15 ++-- rust-toolchain.toml | 2 +- 35 files changed, 315 insertions(+), 335 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index 36093d92aa..9904820dea 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -36,7 +36,7 @@ version = "0.7.0" license = "Apache-2.0" repository = "https://github.com/apache/iceberg-rust" # Check the MSRV policy in README.md before changing this -rust-version = "1.87" +rust-version = "1.88" [workspace.dependencies] anyhow = "1.0.72" diff --git a/crates/catalog/glue/src/catalog.rs b/crates/catalog/glue/src/catalog.rs index dce287ed6e..37a7996f80 100644 --- a/crates/catalog/glue/src/catalog.rs +++ b/crates/catalog/glue/src/catalog.rs @@ -151,33 +151,33 @@ impl GlueCatalog { async fn new(config: GlueCatalogConfig) -> Result { let sdk_config = create_sdk_config(&config.props, config.uri.as_ref()).await; let mut file_io_props = config.props.clone(); - if !file_io_props.contains_key(S3_ACCESS_KEY_ID) { - if let Some(access_key_id) = file_io_props.get(AWS_ACCESS_KEY_ID) { - file_io_props.insert(S3_ACCESS_KEY_ID.to_string(), access_key_id.to_string()); - } + if !file_io_props.contains_key(S3_ACCESS_KEY_ID) + && let Some(access_key_id) = file_io_props.get(AWS_ACCESS_KEY_ID) + { + file_io_props.insert(S3_ACCESS_KEY_ID.to_string(), access_key_id.to_string()); } - if !file_io_props.contains_key(S3_SECRET_ACCESS_KEY) { - if let Some(secret_access_key) = file_io_props.get(AWS_SECRET_ACCESS_KEY) { - file_io_props.insert( - S3_SECRET_ACCESS_KEY.to_string(), - secret_access_key.to_string(), - ); - } + if !file_io_props.contains_key(S3_SECRET_ACCESS_KEY) + && let Some(secret_access_key) = file_io_props.get(AWS_SECRET_ACCESS_KEY) + { + file_io_props.insert( + S3_SECRET_ACCESS_KEY.to_string(), + secret_access_key.to_string(), + ); } - if !file_io_props.contains_key(S3_REGION) { - if let Some(region) = file_io_props.get(AWS_REGION_NAME) { - file_io_props.insert(S3_REGION.to_string(), region.to_string()); - } + if !file_io_props.contains_key(S3_REGION) + && let Some(region) = file_io_props.get(AWS_REGION_NAME) + { + file_io_props.insert(S3_REGION.to_string(), region.to_string()); } - if !file_io_props.contains_key(S3_SESSION_TOKEN) { - if let Some(session_token) = file_io_props.get(AWS_SESSION_TOKEN) { - file_io_props.insert(S3_SESSION_TOKEN.to_string(), session_token.to_string()); - } + if !file_io_props.contains_key(S3_SESSION_TOKEN) + && let Some(session_token) = file_io_props.get(AWS_SESSION_TOKEN) + { + file_io_props.insert(S3_SESSION_TOKEN.to_string(), session_token.to_string()); } - if !file_io_props.contains_key(S3_ENDPOINT) { - if let Some(aws_endpoint) = config.uri.as_ref() { - file_io_props.insert(S3_ENDPOINT.to_string(), aws_endpoint.to_string()); - } + if !file_io_props.contains_key(S3_ENDPOINT) + && let Some(aws_endpoint) = config.uri.as_ref() + { + file_io_props.insert(S3_ENDPOINT.to_string(), aws_endpoint.to_string()); } let client = aws_sdk_glue::Client::new(&sdk_config); diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index de8a1420e4..380d48530c 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -504,10 +504,10 @@ impl ArrowReader { // we need to call next() to update the cache with the newly positioned value. delete_vector_iter.advance_to(next_row_group_base_idx); // Only update the cache if the cached value is stale (in the skipped range) - if let Some(cached_idx) = next_deleted_row_idx_opt { - if cached_idx < next_row_group_base_idx { - next_deleted_row_idx_opt = delete_vector_iter.next(); - } + if let Some(cached_idx) = next_deleted_row_idx_opt + && cached_idx < next_row_group_base_idx + { + next_deleted_row_idx_opt = delete_vector_iter.next(); } // still increment the current page base index but then skip to the next row group @@ -861,10 +861,10 @@ impl ArrowReader { }; // If all row groups were filtered out, return an empty RowSelection (select no rows) - if let Some(selected_row_groups) = selected_row_groups { - if selected_row_groups.is_empty() { - return Ok(RowSelection::from(Vec::new())); - } + if let Some(selected_row_groups) = selected_row_groups + && selected_row_groups.is_empty() + { + return Ok(RowSelection::from(Vec::new())); } let mut selected_row_groups_idx = 0; @@ -897,10 +897,10 @@ impl ArrowReader { results.push(selections_for_page); - if let Some(selected_row_groups) = selected_row_groups { - if selected_row_groups_idx == selected_row_groups.len() { - break; - } + if let Some(selected_row_groups) = selected_row_groups + && selected_row_groups_idx == selected_row_groups.len() + { + break; } } @@ -1031,13 +1031,13 @@ fn apply_name_mapping_to_arrow_schema( let mut metadata = field.metadata().clone(); - if let Some(mapped_field) = mapped_field_opt { - if let Some(field_id) = mapped_field.field_id() { - // Field found in mapping with a field_id → assign it - metadata.insert(PARQUET_FIELD_ID_META_KEY.to_string(), field_id.to_string()); - } - // If field_id is None, leave the field without an ID (will be filtered by projection) + if let Some(mapped_field) = mapped_field_opt + && let Some(field_id) = mapped_field.field_id() + { + // Field found in mapping with a field_id → assign it + metadata.insert(PARQUET_FIELD_ID_META_KEY.to_string(), field_id.to_string()); } + // If field_id is None, leave the field without an ID (will be filtered by projection) // If field not found in mapping, leave it without an ID (will be filtered by projection) Field::new(field.name(), field.data_type().clone(), field.is_nullable()) @@ -2731,15 +2731,14 @@ message schema { // Step 4: Verify we got 199 rows (not 200) let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); - println!("Total rows read: {}", total_rows); + println!("Total rows read: {total_rows}"); println!("Expected: 199 rows (deleted row 199 which had id=200)"); // This assertion will FAIL before the fix and PASS after the fix assert_eq!( total_rows, 199, - "Expected 199 rows after deleting row 199, but got {} rows. \ - The bug causes position deletes in later row groups to be ignored.", - total_rows + "Expected 199 rows after deleting row 199, but got {total_rows} rows. \ + The bug causes position deletes in later row groups to be ignored." ); // Verify the deleted row (id=200) is not present @@ -2950,16 +2949,15 @@ message schema { // Row group 1 has 100 rows (ids 101-200), minus 1 delete (id=200) = 99 rows let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); - println!("Total rows read from row group 1: {}", total_rows); + println!("Total rows read from row group 1: {total_rows}"); println!("Expected: 99 rows (row group 1 has 100 rows, 1 delete at position 199)"); // This assertion will FAIL before the fix and PASS after the fix assert_eq!( total_rows, 99, - "Expected 99 rows from row group 1 after deleting position 199, but got {} rows. \ + "Expected 99 rows from row group 1 after deleting position 199, but got {total_rows} rows. \ The bug causes position deletes to be lost when advance_to() is followed by next() \ - when skipping unselected row groups.", - total_rows + when skipping unselected row groups." ); // Verify the deleted row (id=200) is not present @@ -3241,7 +3239,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2], @@ -3338,7 +3336,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 3], @@ -3424,7 +3422,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2, 3], @@ -3524,7 +3522,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2], @@ -3565,7 +3563,7 @@ message schema { assert_eq!(all_values.len(), 6); for i in 0..6 { - assert_eq!(all_names[i], format!("name_{}", i)); + assert_eq!(all_names[i], format!("name_{i}")); assert_eq!(all_values[i], i as i32); } } @@ -3653,7 +3651,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2], @@ -3749,7 +3747,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 5, 2], @@ -3858,7 +3856,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/1.parquet", table_location), + data_file_path: format!("{table_location}/1.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2, 3], @@ -3997,7 +3995,7 @@ message schema { start: 0, length: 0, record_count: None, - data_file_path: format!("{}/data.parquet", table_location), + data_file_path: format!("{table_location}/data.parquet"), data_file_format: DataFileFormat::Parquet, schema: schema.clone(), project_field_ids: vec![1, 2], diff --git a/crates/iceberg/src/arrow/record_batch_projector.rs b/crates/iceberg/src/arrow/record_batch_projector.rs index 45de0212e8..7028eee961 100644 --- a/crates/iceberg/src/arrow/record_batch_projector.rs +++ b/crates/iceberg/src/arrow/record_batch_projector.rs @@ -133,25 +133,24 @@ impl RecordBatchProjector { { for (pos, field) in fields.iter().enumerate() { let id = field_id_fetch_func(field)?; - if let Some(id) = id { - if target_field_id == id { - index_vec.push(pos); - return Ok(Some(field.clone())); - } + if let Some(id) = id + && target_field_id == id + { + index_vec.push(pos); + return Ok(Some(field.clone())); } - if let DataType::Struct(inner) = field.data_type() { - if searchable_field_func(field) { - if let Some(res) = Self::fetch_field_index( - inner, - index_vec, - target_field_id, - field_id_fetch_func, - searchable_field_func, - )? { - index_vec.push(pos); - return Ok(Some(res)); - } - } + if let DataType::Struct(inner) = field.data_type() + && searchable_field_func(field) + && let Some(res) = Self::fetch_field_index( + inner, + index_vec, + target_field_id, + field_id_fetch_func, + searchable_field_func, + )? + { + index_vec.push(pos); + return Ok(Some(res)); } } Ok(None) diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index f30d4a09c3..c4782464c1 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -582,7 +582,7 @@ impl RecordBatchTransformer { let this_field_id = field_id_str.parse().map_err(|e| { Error::new( ErrorKind::DataInvalid, - format!("field id not parseable as an i32: {}", e), + format!("field id not parseable as an i32: {e}"), ) })?; diff --git a/crates/iceberg/src/arrow/value.rs b/crates/iceberg/src/arrow/value.rs index 0e0b85f073..bc123d99e8 100644 --- a/crates/iceberg/src/arrow/value.rs +++ b/crates/iceberg/src/arrow/value.rs @@ -261,15 +261,15 @@ impl SchemaWithPartnerVisitor for ArrowArrayToIcebergStructConverter { "The partner is not a decimal128 array", ) })?; - if let DataType::Decimal128(arrow_precision, arrow_scale) = array.data_type() { - if *arrow_precision as u32 != *precision || *arrow_scale as u32 != *scale { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "The precision or scale ({arrow_precision},{arrow_scale}) of arrow decimal128 array is not compatible with iceberg decimal type ({precision},{scale})" - ), - )); - } + if let DataType::Decimal128(arrow_precision, arrow_scale) = array.data_type() + && (*arrow_precision as u32 != *precision || *arrow_scale as u32 != *scale) + { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "The precision or scale ({arrow_precision},{arrow_scale}) of arrow decimal128 array is not compatible with iceberg decimal type ({precision},{scale})" + ), + )); } Ok(array.iter().map(|v| v.map(Literal::decimal)).collect()) } @@ -351,10 +351,10 @@ impl SchemaWithPartnerVisitor for ArrowArrayToIcebergStructConverter { } else if let Some(array) = partner.as_any().downcast_ref::() { Ok(array.iter().map(|v| v.map(Literal::string)).collect()) } else { - return Err(Error::new( + Err(Error::new( ErrorKind::DataInvalid, "The partner is not a string array", - )); + )) } } PrimitiveType::Uuid => { @@ -418,10 +418,10 @@ impl SchemaWithPartnerVisitor for ArrowArrayToIcebergStructConverter { .map(|v| v.map(|v| Literal::binary(v.to_vec()))) .collect()) } else { - return Err(Error::new( + Err(Error::new( ErrorKind::DataInvalid, "The partner is not a binary array", - )); + )) } } } @@ -724,10 +724,7 @@ pub(crate) fn create_primitive_array_single_element( } _ => Err(Error::new( ErrorKind::Unexpected, - format!( - "Unsupported constant type combination: {:?} with {:?}", - data_type, prim_lit - ), + format!("Unsupported constant type combination: {data_type:?} with {prim_lit:?}"), )), } } @@ -825,7 +822,7 @@ pub(crate) fn create_primitive_array_repeated( (dt, _) => { return Err(Error::new( ErrorKind::Unexpected, - format!("unexpected target column type {}", dt), + format!("unexpected target column type {dt}"), )); } }) diff --git a/crates/iceberg/src/catalog/mod.rs b/crates/iceberg/src/catalog/mod.rs index 27d5edaedb..f3a521379e 100644 --- a/crates/iceberg/src/catalog/mod.rs +++ b/crates/iceberg/src/catalog/mod.rs @@ -1000,13 +1000,13 @@ mod _serde_set_statistics { snapshot_id, statistics, } = SetStatistics::deserialize(deserializer)?; - if let Some(snapshot_id) = snapshot_id { - if snapshot_id != statistics.snapshot_id { - return Err(serde::de::Error::custom(format!( - "Snapshot id to set {snapshot_id} does not match the statistics file snapshot id {}", - statistics.snapshot_id - ))); - } + if let Some(snapshot_id) = snapshot_id + && snapshot_id != statistics.snapshot_id + { + return Err(serde::de::Error::custom(format!( + "Snapshot id to set {snapshot_id} does not match the statistics file snapshot id {}", + statistics.snapshot_id + ))); } Ok(statistics) diff --git a/crates/iceberg/src/delete_vector.rs b/crates/iceberg/src/delete_vector.rs index f382bf079e..df8a10193c 100644 --- a/crates/iceberg/src/delete_vector.rs +++ b/crates/iceberg/src/delete_vector.rs @@ -36,7 +36,7 @@ impl DeleteVector { } } - pub fn iter(&self) -> DeleteVectorIterator { + pub fn iter(&self) -> DeleteVectorIterator<'_> { let outer = self.inner.bitmaps(); DeleteVectorIterator { outer, inner: None } } @@ -93,10 +93,10 @@ impl Iterator for DeleteVectorIterator<'_> { type Item = u64; fn next(&mut self) -> Option { - if let Some(inner) = &mut self.inner { - if let Some(inner_next) = inner.bitmap_iter.next() { - return Some(u64::from(inner.high_bits) << 32 | u64::from(inner_next)); - } + if let Some(inner) = &mut self.inner + && let Some(inner_next) = inner.bitmap_iter.next() + { + return Some(u64::from(inner.high_bits) << 32 | u64::from(inner_next)); } if let Some((high_bits, next_bitmap)) = self.outer.next() { diff --git a/crates/iceberg/src/expr/visitors/manifest_evaluator.rs b/crates/iceberg/src/expr/visitors/manifest_evaluator.rs index abbd136cb1..770163ae95 100644 --- a/crates/iceberg/src/expr/visitors/manifest_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/manifest_evaluator.rs @@ -161,10 +161,10 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { _predicate: &BoundPredicate, ) -> crate::Result { let field = self.field_summary_for_reference(reference); - if let Some(contains_nan) = field.contains_nan { - if !contains_nan { - return ROWS_CANNOT_MATCH; - } + if let Some(contains_nan) = field.contains_nan + && !contains_nan + { + return ROWS_CANNOT_MATCH; } if ManifestFilterVisitor::are_all_null(field, &reference.field().field_type) { @@ -389,16 +389,16 @@ impl BoundPredicateVisitor for ManifestFilterVisitor<'_> { return ROWS_MIGHT_MATCH; } - if prefix.as_bytes().eq(&lower_bound[..prefix_len]) { - if let Some(upper_bound) = &field.upper_bound { - // if upper is shorter than the prefix then upper can't start with the prefix - if prefix_len > upper_bound.len() { - return ROWS_MIGHT_MATCH; - } + if prefix.as_bytes().eq(&lower_bound[..prefix_len]) + && let Some(upper_bound) = &field.upper_bound + { + // if upper is shorter than the prefix then upper can't start with the prefix + if prefix_len > upper_bound.len() { + return ROWS_MIGHT_MATCH; + } - if prefix.as_bytes().eq(&upper_bound[..prefix_len]) { - return ROWS_CANNOT_MATCH; - } + if prefix.as_bytes().eq(&upper_bound[..prefix_len]) { + return ROWS_CANNOT_MATCH; } } } diff --git a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs index 3745d94d18..ae3a906275 100644 --- a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs @@ -547,16 +547,16 @@ impl BoundPredicateVisitor for PageIndexEvaluator<'_> { return Ok(false); } - if let Some(min) = min { - if min.gt(datum) { - return Ok(false); - } + if let Some(min) = min + && min.gt(datum) + { + return Ok(false); } - if let Some(max) = max { - if max.lt(datum) { - return Ok(false); - } + if let Some(max) = max + && max.lt(datum) + { + return Ok(false); } Ok(true) diff --git a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs index e9bed775ef..7c652e2068 100644 --- a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs @@ -129,10 +129,10 @@ impl<'a> StrictMetricsEvaluator<'a> { self.upper_bound(field_id) }; - if let Some(bound) = bound { - if cmp_fn(bound, datum) { - return ROWS_MUST_MATCH; - } + if let Some(bound) = bound + && cmp_fn(bound, datum) + { + return ROWS_MUST_MATCH; } ROWS_MIGHT_NOT_MATCH @@ -219,10 +219,10 @@ impl BoundPredicateVisitor for StrictMetricsEvaluator<'_> { ) -> crate::Result { let field_id = reference.field().id; - if let Some(&nan_count) = self.nan_count(field_id) { - if nan_count == 0 { - return ROWS_MUST_MATCH; - } + if let Some(&nan_count) = self.nan_count(field_id) + && nan_count == 0 + { + return ROWS_MUST_MATCH; } if self.contains_nulls_only(field_id) { @@ -258,10 +258,10 @@ impl BoundPredicateVisitor for StrictMetricsEvaluator<'_> { ) -> crate::Result { let field_id = reference.field().id; - if let Some(lower) = self.lower_bound(field_id) { - if lower.is_nan() { - return ROWS_MIGHT_NOT_MATCH; - } + if let Some(lower) = self.lower_bound(field_id) + && lower.is_nan() + { + return ROWS_MIGHT_NOT_MATCH; } self.visit_inequality(reference, datum, PartialOrd::gt, true) diff --git a/crates/iceberg/src/inspect/metadata_table.rs b/crates/iceberg/src/inspect/metadata_table.rs index 92571db181..d5e9d60869 100644 --- a/crates/iceberg/src/inspect/metadata_table.rs +++ b/crates/iceberg/src/inspect/metadata_table.rs @@ -71,12 +71,12 @@ impl<'a> MetadataTable<'a> { } /// Get the snapshots table. - pub fn snapshots(&self) -> SnapshotsTable { + pub fn snapshots(&self) -> SnapshotsTable<'_> { SnapshotsTable::new(self.0) } /// Get the manifests table. - pub fn manifests(&self) -> ManifestsTable { + pub fn manifests(&self) -> ManifestsTable<'_> { ManifestsTable::new(self.0) } } diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index 5880ccca59..03e43600dd 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -183,7 +183,7 @@ impl Storage { } else { Err(Error::new( ErrorKind::DataInvalid, - format!("Invalid gcs url: {}, should start with {}", path, prefix), + format!("Invalid gcs url: {path}, should start with {prefix}"), )) } } @@ -198,7 +198,7 @@ impl Storage { } else { Err(Error::new( ErrorKind::DataInvalid, - format!("Invalid oss url: {}, should start with {}", path, prefix), + format!("Invalid oss url: {path}, should start with {prefix}"), )) } } diff --git a/crates/iceberg/src/io/storage_azdls.rs b/crates/iceberg/src/io/storage_azdls.rs index fe12167f6f..5abb0cd6e0 100644 --- a/crates/iceberg/src/io/storage_azdls.rs +++ b/crates/iceberg/src/io/storage_azdls.rs @@ -165,7 +165,7 @@ impl FromStr for AzureStorageScheme { "wasbs" => Ok(AzureStorageScheme::Wasbs), _ => Err(Error::new( ErrorKind::DataInvalid, - format!("Unexpected Azure Storage scheme: {}", s), + format!("Unexpected Azure Storage scheme: {s}"), )), } } @@ -397,11 +397,11 @@ mod tests { let config = azdls_config_parse(properties); match expected { Some(expected_config) => { - assert!(config.is_ok(), "Test case {} failed: {:?}", name, config); - assert_eq!(config.unwrap(), expected_config, "Test case: {}", name); + assert!(config.is_ok(), "Test case {name} failed: {config:?}"); + assert_eq!(config.unwrap(), expected_config, "Test case: {name}"); } None => { - assert!(config.is_err(), "Test case {} expected error.", name); + assert!(config.is_err(), "Test case {name} expected error."); } } } @@ -495,14 +495,14 @@ mod tests { let result = azdls_create_operator(input.0, &input.1, &input.2); match expected { Some((expected_filesystem, expected_path)) => { - assert!(result.is_ok(), "Test case {} failed: {:?}", name, result); + assert!(result.is_ok(), "Test case {name} failed: {result:?}"); let (op, relative_path) = result.unwrap(); assert_eq!(op.info().name(), expected_filesystem); assert_eq!(relative_path, expected_path); } None => { - assert!(result.is_err(), "Test case {} expected error.", name); + assert!(result.is_err(), "Test case {name} expected error."); } } } @@ -543,11 +543,11 @@ mod tests { let result = input.parse::(); match expected { Some(expected_path) => { - assert!(result.is_ok(), "Test case {} failed: {:?}", name, result); - assert_eq!(result.unwrap(), expected_path, "Test case: {}", name); + assert!(result.is_ok(), "Test case {name} failed: {result:?}"); + assert_eq!(result.unwrap(), expected_path, "Test case: {name}"); } None => { - assert!(result.is_err(), "Test case {} expected error.", name); + assert!(result.is_err(), "Test case {name} expected error."); } } } @@ -593,7 +593,7 @@ mod tests { for (name, path, expected) in test_cases { let endpoint = path.as_endpoint(); - assert_eq!(endpoint, expected, "Test case: {}", name); + assert_eq!(endpoint, expected, "Test case: {name}"); } } } diff --git a/crates/iceberg/src/io/storage_gcs.rs b/crates/iceberg/src/io/storage_gcs.rs index 8c3d914c86..7718df603f 100644 --- a/crates/iceberg/src/io/storage_gcs.rs +++ b/crates/iceberg/src/io/storage_gcs.rs @@ -71,20 +71,20 @@ pub(crate) fn gcs_config_parse(mut m: HashMap) -> Result Result let bucket = url.host_str().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Invalid gcs url: {}, bucket is required", path), + format!("Invalid gcs url: {path}, bucket is required"), ) })?; diff --git a/crates/iceberg/src/io/storage_oss.rs b/crates/iceberg/src/io/storage_oss.rs index 8bfffc6ca8..e82dda23a5 100644 --- a/crates/iceberg/src/io/storage_oss.rs +++ b/crates/iceberg/src/io/storage_oss.rs @@ -56,7 +56,7 @@ pub(crate) fn oss_config_build(cfg: &OssConfig, path: &str) -> Result let bucket = url.host_str().ok_or_else(|| { Error::new( ErrorKind::DataInvalid, - format!("Invalid oss url: {}, missing bucket", path), + format!("Invalid oss url: {path}, missing bucket"), ) })?; diff --git a/crates/iceberg/src/io/storage_s3.rs b/crates/iceberg/src/io/storage_s3.rs index fcf9afed1f..f069e0e2f9 100644 --- a/crates/iceberg/src/io/storage_s3.rs +++ b/crates/iceberg/src/io/storage_s3.rs @@ -134,20 +134,20 @@ pub(crate) fn s3_config_parse(mut m: HashMap) -> Result Result { Err(Error::new( ErrorKind::Unexpected, format!( - "Metadata field ID {} recognized but field definition not implemented", - field_id + "Metadata field ID {field_id} recognized but field definition not implemented" ), )) } _ => Err(Error::new( ErrorKind::Unexpected, - format!("Field ID {} is not a metadata field", field_id), + format!("Field ID {field_id} is not a metadata field"), )), } } diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 24c03b0b2c..d83da8a879 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -1872,8 +1872,7 @@ pub mod tests { let file_path = string_values.value(0); assert!( file_path.ends_with(".parquet"), - "File path should end with .parquet, got: {}", - file_path + "File path should end with .parquet, got: {file_path}" ); } @@ -1981,8 +1980,7 @@ pub mod tests { for path in &file_paths { assert!( path.ends_with(".parquet"), - "All file paths should end with .parquet, got: {}", - path + "All file paths should end with .parquet, got: {path}" ); } } diff --git a/crates/iceberg/src/spec/datatypes.rs b/crates/iceberg/src/spec/datatypes.rs index 456b754408..0379465584 100644 --- a/crates/iceberg/src/spec/datatypes.rs +++ b/crates/iceberg/src/spec/datatypes.rs @@ -427,8 +427,7 @@ impl<'de> Deserialize<'de> for StructType { let type_val: String = map.next_value()?; if type_val != "struct" { return Err(serde::de::Error::custom(format!( - "expected type 'struct', got '{}'", - type_val + "expected type 'struct', got '{type_val}'" ))); } } diff --git a/crates/iceberg/src/spec/manifest/writer.rs b/crates/iceberg/src/spec/manifest/writer.rs index ebb0590bcf..389ac7a1fd 100644 --- a/crates/iceberg/src/spec/manifest/writer.rs +++ b/crates/iceberg/src/spec/manifest/writer.rs @@ -388,10 +388,10 @@ impl ManifestWriter { self.existing_rows += entry.data_file.record_count; } } - if entry.is_alive() { - if let Some(seq_num) = entry.sequence_number { - self.min_seq_num = Some(self.min_seq_num.map_or(seq_num, |v| min(v, seq_num))); - } + if entry.is_alive() + && let Some(seq_num) = entry.sequence_number + { + self.min_seq_num = Some(self.min_seq_num.map_or(seq_num, |v| min(v, seq_num))); } self.manifest_entries.push(entry); Ok(()) diff --git a/crates/iceberg/src/spec/schema/prune_columns.rs b/crates/iceberg/src/spec/schema/prune_columns.rs index 5a2f0b50fc..14f1bfd25f 100644 --- a/crates/iceberg/src/spec/schema/prune_columns.rs +++ b/crates/iceberg/src/spec/schema/prune_columns.rs @@ -110,19 +110,19 @@ impl SchemaVisitor for PruneColumn { if self.select_full_types { Ok(Some(*field.field_type.clone())) } else if field.field_type.is_struct() { - return Ok(Some(Type::Struct(PruneColumn::project_selected_struct( + Ok(Some(Type::Struct(PruneColumn::project_selected_struct( value, - )?))); + )?))) } else if !field.field_type.is_nested() { - return Ok(Some(*field.field_type.clone())); + Ok(Some(*field.field_type.clone())) } else { - return Err(Error::new( + Err(Error::new( ErrorKind::DataInvalid, "Can't project list or map field directly when not selecting full type." .to_string(), ) .with_context("field_id", field.id.to_string()) - .with_context("field_type", field.field_type.to_string())); + .with_context("field_type", field.field_type.to_string())) } } else { Ok(value) @@ -174,20 +174,20 @@ impl SchemaVisitor for PruneColumn { Ok(Some(Type::List(list.clone()))) } else if list.element_field.field_type.is_struct() { let projected_struct = PruneColumn::project_selected_struct(value).unwrap(); - return Ok(Some(Type::List(PruneColumn::project_list( + Ok(Some(Type::List(PruneColumn::project_list( list, Type::Struct(projected_struct), - )?))); + )?))) } else if list.element_field.field_type.is_primitive() { - return Ok(Some(Type::List(list.clone()))); + Ok(Some(Type::List(list.clone()))) } else { - return Err(Error::new( + Err(Error::new( ErrorKind::DataInvalid, format!( "Cannot explicitly project List or Map types, List element {} of type {} was selected", list.element_field.id, list.element_field.field_type ), - )); + )) } } else if let Some(result) = value { Ok(Some(Type::List(PruneColumn::project_list(list, result)?))) @@ -208,26 +208,26 @@ impl SchemaVisitor for PruneColumn { } else if map.value_field.field_type.is_struct() { let projected_struct = PruneColumn::project_selected_struct(Some(value.unwrap())).unwrap(); - return Ok(Some(Type::Map(PruneColumn::project_map( + Ok(Some(Type::Map(PruneColumn::project_map( map, Type::Struct(projected_struct), - )?))); + )?))) } else if map.value_field.field_type.is_primitive() { - return Ok(Some(Type::Map(map.clone()))); + Ok(Some(Type::Map(map.clone()))) } else { - return Err(Error::new( + Err(Error::new( ErrorKind::DataInvalid, format!( "Cannot explicitly project List or Map types, Map value {} of type {} was selected", map.value_field.id, map.value_field.field_type ), - )); + )) } } else if let Some(value_result) = value { - return Ok(Some(Type::Map(PruneColumn::project_map( + Ok(Some(Type::Map(PruneColumn::project_map( map, value_result, - )?))); + )?))) } else if self.selected.contains(&map.key_field.id) { Ok(Some(Type::Map(map.clone()))) } else { diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index 06b32cc847..48b715da59 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -390,18 +390,18 @@ impl TableMetadata { } fn construct_refs(&mut self) { - if let Some(current_snapshot_id) = self.current_snapshot_id { - if !self.refs.contains_key(MAIN_BRANCH) { - self.refs - .insert(MAIN_BRANCH.to_string(), SnapshotReference { - snapshot_id: current_snapshot_id, - retention: SnapshotRetention::Branch { - min_snapshots_to_keep: None, - max_snapshot_age_ms: None, - max_ref_age_ms: None, - }, - }); - } + if let Some(current_snapshot_id) = self.current_snapshot_id + && !self.refs.contains_key(MAIN_BRANCH) + { + self.refs + .insert(MAIN_BRANCH.to_string(), SnapshotReference { + snapshot_id: current_snapshot_id, + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: None, + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + }); } } @@ -572,17 +572,17 @@ impl TableMetadata { let main_ref = self.refs.get(MAIN_BRANCH); if self.current_snapshot_id.is_some() { - if let Some(main_ref) = main_ref { - if main_ref.snapshot_id != self.current_snapshot_id.unwrap_or_default() { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Current snapshot id does not match main branch ({:?} != {:?})", - self.current_snapshot_id.unwrap_or_default(), - main_ref.snapshot_id - ), - )); - } + if let Some(main_ref) = main_ref + && main_ref.snapshot_id != self.current_snapshot_id.unwrap_or_default() + { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Current snapshot id does not match main branch ({:?} != {:?})", + self.current_snapshot_id.unwrap_or_default(), + main_ref.snapshot_id + ), + )); } } else if main_ref.is_some() { return Err(Error::new( @@ -606,22 +606,21 @@ impl TableMetadata { )); } - if self.format_version >= FormatVersion::V2 { - if let Some(snapshot) = self + if self.format_version >= FormatVersion::V2 + && let Some(snapshot) = self .snapshots .values() .find(|snapshot| snapshot.sequence_number() > self.last_sequence_number) - { - return Err(Error::new( - ErrorKind::DataInvalid, - format!( - "Invalid snapshot with id {} and sequence number {} greater than last sequence number {}", - snapshot.snapshot_id(), - snapshot.sequence_number(), - self.last_sequence_number - ), - )); - } + { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Invalid snapshot with id {} and sequence number {} greater than last sequence number {}", + snapshot.snapshot_id(), + snapshot.sequence_number(), + self.last_sequence_number + ), + )); } Ok(()) diff --git a/crates/iceberg/src/spec/transform.rs b/crates/iceberg/src/spec/transform.rs index 6068716eff..354dc1889c 100644 --- a/crates/iceberg/src/spec/transform.rs +++ b/crates/iceberg/src/spec/transform.rs @@ -711,10 +711,10 @@ impl Transform { PredicateOperator::GreaterThan => Some(PredicateOperator::GreaterThanOrEq), PredicateOperator::StartsWith => match datum.literal() { PrimitiveLiteral::String(s) => { - if let Some(w) = width { - if s.len() == w as usize { - return Some(PredicateOperator::Eq); - }; + if let Some(w) = width + && s.len() == w as usize + { + return Some(PredicateOperator::Eq); }; Some(*op) } @@ -757,47 +757,45 @@ impl Transform { _ => false, }; - if should_adjust { - if let &PrimitiveLiteral::Int(v) = transformed.literal() { - match op { - PredicateOperator::LessThan - | PredicateOperator::LessThanOrEq - | PredicateOperator::In => { - if v < 0 { + if should_adjust && let &PrimitiveLiteral::Int(v) = transformed.literal() { + match op { + PredicateOperator::LessThan + | PredicateOperator::LessThanOrEq + | PredicateOperator::In => { + if v < 0 { + // # TODO + // An ugly hack to fix. Refine the increment and decrement logic later. + match self { + Transform::Day => { + return Some(AdjustedProjection::Single(Datum::date(v + 1))); + } + _ => { + return Some(AdjustedProjection::Single(Datum::int(v + 1))); + } + } + }; + } + PredicateOperator::Eq => { + if v < 0 { + let new_set = FnvHashSet::from_iter(vec![ + transformed.to_owned(), // # TODO // An ugly hack to fix. Refine the increment and decrement logic later. - match self { - Transform::Day => { - return Some(AdjustedProjection::Single(Datum::date(v + 1))); + { + match self { + Transform::Day => Datum::date(v + 1), + _ => Datum::int(v + 1), } - _ => { - return Some(AdjustedProjection::Single(Datum::int(v + 1))); - } - } - }; - } - PredicateOperator::Eq => { - if v < 0 { - let new_set = FnvHashSet::from_iter(vec![ - transformed.to_owned(), - // # TODO - // An ugly hack to fix. Refine the increment and decrement logic later. - { - match self { - Transform::Day => Datum::date(v + 1), - _ => Datum::int(v + 1), - } - }, - ]); - return Some(AdjustedProjection::Set(new_set)); - } - } - _ => { - return None; + }, + ]); + return Some(AdjustedProjection::Set(new_set)); } } - }; - } + _ => { + return None; + } + } + }; None } diff --git a/crates/iceberg/src/spec/values/tests.rs b/crates/iceberg/src/spec/values/tests.rs index 0e99d44dfe..73343a9a1a 100644 --- a/crates/iceberg/src/spec/values/tests.rs +++ b/crates/iceberg/src/spec/values/tests.rs @@ -447,7 +447,7 @@ fn check_raw_literal_bytes_error_via_avro(input_bytes: Vec, expected_type: & let avro_value = Value::Bytes(input_bytes); let raw_literal: _serde::RawLiteral = apache_avro::from_value(&avro_value).unwrap(); let result = raw_literal.try_into(expected_type); - assert!(result.is_err(), "Expected error but got: {:?}", result); + assert!(result.is_err(), "Expected error but got: {result:?}"); } #[test] diff --git a/crates/iceberg/src/spec/view_metadata_builder.rs b/crates/iceberg/src/spec/view_metadata_builder.rs index 9f542a7c61..38041ca625 100644 --- a/crates/iceberg/src/spec/view_metadata_builder.rs +++ b/crates/iceberg/src/spec/view_metadata_builder.rs @@ -478,10 +478,10 @@ impl ViewMetadataBuilder { // as it might panic if the metadata is invalid. self.metadata.validate()?; - if let Some(previous) = self.previous_view_version.take() { - if !allow_replace_drop_dialects(&self.metadata.properties) { - require_no_dialect_dropped(&previous, self.metadata.current_version())?; - } + if let Some(previous) = self.previous_view_version.take() + && !allow_replace_drop_dialects(&self.metadata.properties) + { + require_no_dialect_dropped(&previous, self.metadata.current_version())?; } let _expired_versions = self.expire_versions(); diff --git a/crates/iceberg/src/transaction/mod.rs b/crates/iceberg/src/transaction/mod.rs index 4116264a14..8ddaa26698 100644 --- a/crates/iceberg/src/transaction/mod.rs +++ b/crates/iceberg/src/transaction/mod.rs @@ -518,7 +518,7 @@ mod test_row_lineage { fn file_with_rows(record_count: u64) -> DataFile { DataFileBuilder::default() .content(DataContentType::Data) - .file_path(format!("test/{}.parquet", record_count)) + .file_path(format!("test/{record_count}.parquet")) .file_format(DataFileFormat::Parquet) .file_size_in_bytes(100) .record_count(record_count) diff --git a/crates/iceberg/src/transaction/snapshot.rs b/crates/iceberg/src/transaction/snapshot.rs index d59828ce31..c8bf26a174 100644 --- a/crates/iceberg/src/transaction/snapshot.rs +++ b/crates/iceberg/src/transaction/snapshot.rs @@ -276,13 +276,13 @@ impl<'a> SnapshotProducer<'a> { "Partition field should only be primitive type.", ) })?; - if let Some(value) = value { - if !field.compatible(&value.as_primitive_literal().unwrap()) { - return Err(Error::new( - ErrorKind::DataInvalid, - "Partition value is not compatible partition type", - )); - } + if let Some(value) = value + && !field.compatible(&value.as_primitive_literal().unwrap()) + { + return Err(Error::new( + ErrorKind::DataInvalid, + "Partition value is not compatible partition type", + )); } } Ok(()) diff --git a/crates/iceberg/src/writer/file_writer/rolling_writer.rs b/crates/iceberg/src/writer/file_writer/rolling_writer.rs index 06246ab660..a93e494d48 100644 --- a/crates/iceberg/src/writer/file_writer/rolling_writer.rs +++ b/crates/iceberg/src/writer/file_writer/rolling_writer.rs @@ -197,18 +197,18 @@ where ); } - if self.should_roll() { - if let Some(inner) = self.inner.take() { - // close the current writer, roll to a new file - self.data_file_builders.extend(inner.close().await?); - - // start a new writer - self.inner = Some( - self.inner_builder - .build(self.new_output_file(partition_key)?) - .await?, - ); - } + if self.should_roll() + && let Some(inner) = self.inner.take() + { + // close the current writer, roll to a new file + self.data_file_builders.extend(inner.close().await?); + + // start a new writer + self.inner = Some( + self.inner_builder + .build(self.new_output_file(partition_key)?) + .await?, + ); } // write the input diff --git a/crates/iceberg/tests/file_io_gcs_test.rs b/crates/iceberg/tests/file_io_gcs_test.rs index 161285ae6f..9fbcdadd0e 100644 --- a/crates/iceberg/tests/file_io_gcs_test.rs +++ b/crates/iceberg/tests/file_io_gcs_test.rs @@ -68,7 +68,7 @@ mod tests { FileIOBuilder::new("gcs") .with_props(vec![ - (GCS_SERVICE_PATH, format!("http://{}", addr)), + (GCS_SERVICE_PATH, format!("http://{addr}")), (GCS_NO_AUTH, "true".to_string()), ]) .build() @@ -81,13 +81,13 @@ mod tests { bucket_data.insert("name", name); let client = reqwest::Client::new(); - let endpoint = format!("http://{}/storage/v1/b", server_addr); + let endpoint = format!("http://{server_addr}/storage/v1/b"); client.post(endpoint).json(&bucket_data).send().await?; Ok(()) } fn get_gs_path() -> String { - format!("gs://{}", FAKE_GCS_BUCKET) + format!("gs://{FAKE_GCS_BUCKET}") } #[tokio::test] diff --git a/crates/integrations/datafusion/src/physical_plan/repartition.rs b/crates/integrations/datafusion/src/physical_plan/repartition.rs index 8ad87fd1cc..2d1d7f862c 100644 --- a/crates/integrations/datafusion/src/physical_plan/repartition.rs +++ b/crates/integrations/datafusion/src/physical_plan/repartition.rs @@ -159,9 +159,8 @@ fn determine_partitioning_strategy( // Case 2: Partitioned table missing _partition column (normally this should not happen) (true, Err(_)) => Err(DataFusionError::Plan(format!( - "Partitioned table input missing {} column. \ - Ensure projection happens before repartitioning.", - PROJECTED_PARTITION_VALUE_COLUMN + "Partitioned table input missing {PROJECTED_PARTITION_VALUE_COLUMN} column. \ + Ensure projection happens before repartitioning." ))), // Case 3: Unpartitioned table, always use RoundRobinBatch @@ -508,8 +507,7 @@ mod tests { assert!( column_names.contains(&PROJECTED_PARTITION_VALUE_COLUMN.to_string()), - "Should use _partition column, got: {:?}", - column_names + "Should use _partition column, got: {column_names:?}" ); } _ => panic!("Expected Hash partitioning with Identity transform"), @@ -733,8 +731,7 @@ mod tests { .collect(); assert!( column_names.contains(&PROJECTED_PARTITION_VALUE_COLUMN.to_string()), - "Should use _partition column for mixed transforms with Identity, got: {:?}", - column_names + "Should use _partition column for mixed transforms with Identity, got: {column_names:?}" ); } _ => panic!("Expected Hash partitioning for table with identity transforms"), diff --git a/crates/integrations/datafusion/src/physical_plan/sort.rs b/crates/integrations/datafusion/src/physical_plan/sort.rs index 2a57e16e43..ede2547535 100644 --- a/crates/integrations/datafusion/src/physical_plan/sort.rs +++ b/crates/integrations/datafusion/src/physical_plan/sort.rs @@ -53,8 +53,7 @@ pub(crate) fn sort_by_partition(input: Arc) -> DFResult Result<()> { let file_io = table.file_io(); // List files under each expected partition path - let electronics_path = format!("{}/data/category=electronics", table_location); - let books_path = format!("{}/data/category=books", table_location); - let clothing_path = format!("{}/data/category=clothing", table_location); + let electronics_path = format!("{table_location}/data/category=electronics"); + let books_path = format!("{table_location}/data/category=books"); + let clothing_path = format!("{table_location}/data/category=clothing"); // Verify partition directories exist and contain data files assert!( file_io.exists(&electronics_path).await?, - "Expected partition directory: {}", - electronics_path + "Expected partition directory: {electronics_path}" ); assert!( file_io.exists(&books_path).await?, - "Expected partition directory: {}", - books_path + "Expected partition directory: {books_path}" ); assert!( file_io.exists(&clothing_path).await?, - "Expected partition directory: {}", - clothing_path + "Expected partition directory: {clothing_path}" ); Ok(()) diff --git a/rust-toolchain.toml b/rust-toolchain.toml index ff7d1f7fbb..4b20d68e44 100644 --- a/rust-toolchain.toml +++ b/rust-toolchain.toml @@ -20,5 +20,5 @@ # # The channel is exactly same day for our MSRV. [toolchain] -channel = "nightly-2025-03-28" +channel = "nightly-2025-06-23" components = ["rustfmt", "clippy"] From 5f631a5e2830eaa22457495f77db612068dc587c Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 10 Dec 2025 14:34:11 +0800 Subject: [PATCH 107/891] ci: Make s3tables ready for publish (#1916) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1886 ## What changes are included in this PR? Add missing files needed for s3tables. ## Are these changes tested? Signed-off-by: Xuanwo --- .github/workflows/publish.yml | 1 + crates/catalog/s3tables/Cargo.toml | 1 + crates/catalog/s3tables/LICENSE | 201 +++++++++++++++++++++++++++++ crates/catalog/s3tables/NOTICE | 5 + crates/catalog/s3tables/README.md | 56 ++++++++ 5 files changed, 264 insertions(+) create mode 100644 crates/catalog/s3tables/LICENSE create mode 100644 crates/catalog/s3tables/NOTICE create mode 100644 crates/catalog/s3tables/README.md diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 634cd73519..66c17a668d 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -42,6 +42,7 @@ jobs: - "crates/catalog/glue" - "crates/catalog/hms" - "crates/catalog/rest" + - "crates/catalog/s3tables" - "crates/catalog/sql" - "crates/integrations/datafusion" steps: diff --git a/crates/catalog/s3tables/Cargo.toml b/crates/catalog/s3tables/Cargo.toml index 66fb70fefc..fde08b9a49 100644 --- a/crates/catalog/s3tables/Cargo.toml +++ b/crates/catalog/s3tables/Cargo.toml @@ -21,6 +21,7 @@ homepage = { workspace = true } name = "iceberg-catalog-s3tables" rust-version = { workspace = true } version = { workspace = true } +readme = "README.md" categories = ["database"] description = "Apache Iceberg Rust S3Tables Catalog" diff --git a/crates/catalog/s3tables/LICENSE b/crates/catalog/s3tables/LICENSE new file mode 100644 index 0000000000..261eeb9e9f --- /dev/null +++ b/crates/catalog/s3tables/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/crates/catalog/s3tables/NOTICE b/crates/catalog/s3tables/NOTICE new file mode 100644 index 0000000000..9340680cbd --- /dev/null +++ b/crates/catalog/s3tables/NOTICE @@ -0,0 +1,5 @@ +Apache Iceberg Rust +Copyright 2023-2024 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). diff --git a/crates/catalog/s3tables/README.md b/crates/catalog/s3tables/README.md new file mode 100644 index 0000000000..d9784d5231 --- /dev/null +++ b/crates/catalog/s3tables/README.md @@ -0,0 +1,56 @@ + + +# Apache Iceberg S3Tables Catalog (Rust) + +[![crates.io](https://img.shields.io/crates/v/iceberg.svg)](https://crates.io/crates/iceberg-catalog-s3tables) +[![docs.rs](https://img.shields.io/docsrs/iceberg.svg)](https://docs.rs/iceberg/latest/iceberg_catalog_s3tables/) + +Official Native Rust implementation of the Apache Iceberg S3Tables catalog. + +## Quick start + +```rust,no_run +use std::collections::HashMap; + +use iceberg::CatalogBuilder; +use iceberg_catalog_s3tables::{ + S3TABLES_CATALOG_PROP_ENDPOINT_URL, S3TABLES_CATALOG_PROP_TABLE_BUCKET_ARN, + S3TablesCatalogBuilder, +}; + +#[tokio::main] +async fn main() { + let catalog = S3TablesCatalogBuilder::default() + .with_endpoint_url("http://localhost:4566") + .load( + "s3tables", + HashMap::from([( + S3TABLES_CATALOG_PROP_TABLE_BUCKET_ARN.to_string(), + "arn:aws:s3tables:us-east-1:123456789012:bucket/my-bucket".to_string(), + )]), + ) + .await + .unwrap(); + + // use `catalog` as any Iceberg Catalog +} +``` + +See the [API documentation](https://docs.rs/iceberg_catalog_s3tables/latest) for the full API surface. From aaa700d4af6c9e86a0a7169000b3788c6d54430e Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Wed, 10 Dec 2025 02:07:35 -0500 Subject: [PATCH 108/891] deps: bump DataFusion to 51, Arrow to 57, pyo to 0.26 (#1899) --- Cargo.lock | 1239 ++++++++--------- Cargo.toml | 24 +- bindings/python/Cargo.lock | 379 +++-- bindings/python/Cargo.toml | 8 +- bindings/python/src/transform.rs | 20 +- crates/iceberg/Cargo.toml | 1 - crates/iceberg/src/arrow/reader.rs | 35 +- .../src/expr/visitors/page_index_evaluator.rs | 613 ++++---- crates/iceberg/src/inspect/manifests.rs | 24 +- crates/iceberg/src/inspect/snapshots.rs | 14 +- .../src/writer/file_writer/parquet_writer.rs | 37 +- .../src/table/table_provider_factory.rs | 1 + .../tests/integration_datafusion_test.rs | 72 +- crates/integrations/playground/src/main.rs | 2 + 14 files changed, 1124 insertions(+), 1345 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c7c103d279..2d464709fd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -116,22 +116,22 @@ dependencies = [ [[package]] name = "anstyle-query" -version = "1.1.4" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e231f6134f61b71076a3eab506c379d4f36122f2af15a9ff04415ea4c3339e2" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] name = "anstyle-wincon" -version = "3.0.10" +version = "3.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3e0633414522a32ffaac8ac6cc8f748e090c5717661fddeea04219e2344f5f2a" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -161,8 +161,8 @@ dependencies = [ "serde_bytes", "serde_json", "snap", - "strum 0.27.2", - "strum_macros 0.27.2", + "strum", + "strum_macros", "thiserror 2.0.17", "uuid", "xz2", @@ -187,8 +187,8 @@ dependencies = [ "serde", "serde_bytes", "serde_json", - "strum 0.27.2", - "strum_macros 0.27.2", + "strum", + "strum_macros", "thiserror 2.0.17", "uuid", "zstd", @@ -223,9 +223,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e833808ff2d94ed40d9379848a950d995043c7fb3e81a30b383f4c6033821cc" +checksum = "cb372a7cbcac02a35d3fb7b3fc1f969ec078e871f9bb899bf00a2e1809bec8a3" dependencies = [ "arrow-arith", "arrow-array", @@ -244,23 +244,23 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad08897b81588f60ba983e3ca39bda2b179bdd84dced378e7df81a5313802ef8" +checksum = "0f377dcd19e440174596d83deb49cd724886d91060c07fec4f67014ef9d54049" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "chrono", - "num", + "num-traits", ] [[package]] name = "arrow-array" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8548ca7c070d8db9ce7aa43f37393e4bfcf3f2d3681df278490772fd1673d08d" +checksum = "a23eaff85a44e9fa914660fb0d0bb00b79c4a3d888b5334adb3ea4330c84f002" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -269,47 +269,51 @@ dependencies = [ "chrono", "chrono-tz", "half", - "hashbrown 0.16.0", - "num", + "hashbrown 0.16.1", + "num-complex", + "num-integer", + "num-traits", ] [[package]] name = "arrow-buffer" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e003216336f70446457e280807a73899dd822feaf02087d31febca1363e2fccc" +checksum = "a2819d893750cb3380ab31ebdc8c68874dd4429f90fd09180f3c93538bd21626" dependencies = [ "bytes", "half", - "num", + "num-bigint", + "num-traits", ] [[package]] name = "arrow-cast" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "919418a0681298d3a77d1a315f625916cb5678ad0d74b9c60108eb15fd083023" +checksum = "e3d131abb183f80c450d4591dc784f8d7750c50c6e2bc3fcaad148afc8361271" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", + "arrow-ord", "arrow-schema", "arrow-select", "atoi", - "base64 0.22.1", + "base64", "chrono", "comfy-table", "half", "lexical-core", - "num", + "num-traits", "ryu", ] [[package]] name = "arrow-csv" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa9bf02705b5cf762b6f764c65f04ae9082c7cfc4e96e0c33548ee3f67012eb" +checksum = "2275877a0e5e7e7c76954669366c2aa1a829e340ab1f612e647507860906fb6b" dependencies = [ "arrow-array", "arrow-cast", @@ -322,21 +326,22 @@ dependencies = [ [[package]] name = "arrow-data" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5c64fff1d142f833d78897a772f2e5b55b36cb3e6320376f0961ab0db7bd6d0" +checksum = "05738f3d42cb922b9096f7786f606fcb8669260c2640df8490533bb2fa38c9d3" dependencies = [ "arrow-buffer", "arrow-schema", "half", - "num", + "num-integer", + "num-traits", ] [[package]] name = "arrow-ipc" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d3594dcddccc7f20fd069bc8e9828ce37220372680ff638c5e00dea427d88f5" +checksum = "3d09446e8076c4b3f235603d9ea7c5494e73d441b01cd61fb33d7254c11964b3" dependencies = [ "arrow-array", "arrow-buffer", @@ -350,9 +355,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88cf36502b64a127dc659e3b305f1d993a544eab0d48cce704424e62074dc04b" +checksum = "371ffd66fa77f71d7628c63f209c9ca5341081051aa32f9c8020feb0def787c0" dependencies = [ "arrow-array", "arrow-buffer", @@ -361,20 +366,22 @@ dependencies = [ "arrow-schema", "chrono", "half", - "indexmap 2.12.0", + "indexmap 2.12.1", + "itoa", "lexical-core", "memchr", - "num", - "serde", + "num-traits", + "ryu", + "serde_core", "serde_json", "simdutf8", ] [[package]] name = "arrow-ord" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c8f82583eb4f8d84d4ee55fd1cb306720cddead7596edce95b50ee418edf66f" +checksum = "cbc94fc7adec5d1ba9e8cd1b1e8d6f72423b33fe978bf1f46d970fafab787521" dependencies = [ "arrow-array", "arrow-buffer", @@ -385,9 +392,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d07ba24522229d9085031df6b94605e0f4b26e099fb7cdeec37abd941a73753" +checksum = "169676f317157dc079cc5def6354d16db63d8861d61046d2f3883268ced6f99f" dependencies = [ "arrow-array", "arrow-buffer", @@ -398,33 +405,33 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b3aa9e59c611ebc291c28582077ef25c97f1975383f1479b12f3b9ffee2ffabe" +checksum = "d27609cd7dd45f006abae27995c2729ef6f4b9361cde1ddd019dc31a5aa017e0" dependencies = [ - "serde", + "serde_core", "serde_json", ] [[package]] name = "arrow-select" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c41dbbd1e97bfcaee4fcb30e29105fb2c75e4d82ae4de70b792a5d3f66b2e7a" +checksum = "ae980d021879ea119dd6e2a13912d81e64abed372d53163e804dfe84639d8010" dependencies = [ "ahash 0.8.12", "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", - "num", + "num-traits", ] [[package]] name = "arrow-string" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53f5183c150fbc619eede22b861ea7c0eebed8eaac0333eaa7f6da5205fd504d" +checksum = "cf35e8ef49dcf0c5f6d175edee6b8af7b45611805333129c541a8b89a0fc0534" dependencies = [ "arrow-array", "arrow-buffer", @@ -432,7 +439,7 @@ dependencies = [ "arrow-schema", "arrow-select", "memchr", - "num", + "num-traits", "regex", "regex-syntax", ] @@ -501,7 +508,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -512,7 +519,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -538,9 +545,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-config" -version = "1.8.8" +version = "1.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37cf2b6af2a95a20e266782b4f76f1a5e12bf412a9db2de9c1e9123b9d8c0ad8" +checksum = "a0149602eeaf915158e14029ba0c78dedb8c08d554b024d54c8f239aab46511d" dependencies = [ "aws-credential-types", "aws-runtime", @@ -568,9 +575,9 @@ dependencies = [ [[package]] name = "aws-credential-types" -version = "1.2.8" +version = "1.2.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "faf26925f4a5b59eb76722b63c2892b1d70d06fa053c72e4a100ec308c1d47bc" +checksum = "b01c9521fa01558f750d183c8c68c81b0155b9d193a4ba7f84c36bd1b6d04a06" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -580,9 +587,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.14.1" +version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879b6c89592deb404ba4dc0ae6b58ffd1795c78991cbb5b8bc441c48a070440d" +checksum = "6b5ce75405893cd713f9ab8e297d8e438f624dde7d706108285f7e17a25a180f" dependencies = [ "aws-lc-sys", "zeroize", @@ -590,11 +597,10 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.32.3" +version = "0.34.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "107a4e9d9cab9963e04e84bb8dee0e25f2a987f9a8bad5ed054abd439caa8f8c" +checksum = "179c3777a8b5e70e90ea426114ffc565b2c1a9f82f6c4a0c5a34aa6ef5e781b6" dependencies = [ - "bindgen", "cc", "cmake", "dunce", @@ -603,9 +609,9 @@ dependencies = [ [[package]] name = "aws-runtime" -version = "1.5.12" +version = "1.5.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa006bb32360ed90ac51203feafb9d02e3d21046e1fd3a450a404b90ea73e5d" +checksum = "7ce527fb7e53ba9626fc47824f25e256250556c40d8f81d27dd92aa38239d632" dependencies = [ "aws-credential-types", "aws-sigv4", @@ -627,9 +633,9 @@ dependencies = [ [[package]] name = "aws-sdk-glue" -version = "1.126.0" +version = "1.132.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fd9c10a11584c0b619c9e478143072c4028c39017f98534e206156a7e94188be" +checksum = "35638d8e6ef97adb7f3154ffc618bbe1d631a503b6d8328b94af77b7615fbeb2" dependencies = [ "aws-credential-types", "aws-runtime", @@ -649,9 +655,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3tables" -version = "1.41.0" +version = "1.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "761f176da526badb4c3dbd67ee1da2faf3dc1e537ed229355f7590d80595ae35" +checksum = "ce68b5d4652e6248827e472c67df8773ae6ab3946ff176de8d3ee7c295299efd" dependencies = [ "aws-credential-types", "aws-runtime", @@ -671,9 +677,9 @@ dependencies = [ [[package]] name = "aws-sdk-sso" -version = "1.86.0" +version = "1.90.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a0abbfab841446cce6e87af853a3ba2cc1bc9afcd3f3550dd556c43d434c86d" +checksum = "4f18e53542c522459e757f81e274783a78f8c81acdfc8d1522ee8a18b5fb1c66" dependencies = [ "aws-credential-types", "aws-runtime", @@ -693,9 +699,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.89.0" +version = "1.92.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "695dc67bb861ccb8426c9129b91c30e266a0e3d85650cafdf62fcca14c8fd338" +checksum = "532f4d866012ffa724a4385c82e8dd0e59f0ca0e600f3f22d4c03b6824b34e4a" dependencies = [ "aws-credential-types", "aws-runtime", @@ -715,9 +721,9 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.88.0" +version = "1.94.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d30990923f4f675523c51eb1c0dec9b752fb267b36a61e83cbc219c9d86da715" +checksum = "1be6fbbfa1a57724788853a623378223fe828fc4c09b146c992f0c95b6256174" dependencies = [ "aws-credential-types", "aws-runtime", @@ -738,9 +744,9 @@ dependencies = [ [[package]] name = "aws-sigv4" -version = "1.3.5" +version = "1.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bffc03068fbb9c8dd5ce1c6fb240678a5cffb86fb2b7b1985c999c4b83c8df68" +checksum = "c35452ec3f001e1f2f6db107b6373f1f48f05ec63ba2c5c9fa91f07dad32af11" dependencies = [ "aws-credential-types", "aws-smithy-http", @@ -760,9 +766,9 @@ dependencies = [ [[package]] name = "aws-smithy-async" -version = "1.2.6" +version = "1.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "127fcfad33b7dfc531141fda7e1c402ac65f88aca5511a4d31e2e3d2cd01ce9c" +checksum = "9ee19095c7c4dda59f1697d028ce704c24b2d33c6718790c7f1d5a3015b4107c" dependencies = [ "futures-util", "pin-project-lite", @@ -771,15 +777,16 @@ dependencies = [ [[package]] name = "aws-smithy-http" -version = "0.62.4" +version = "0.62.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3feafd437c763db26aa04e0cc7591185d0961e64c61885bece0fb9d50ceac671" +checksum = "826141069295752372f8203c17f28e30c464d22899a43a0c9fd9c458d469c88b" dependencies = [ "aws-smithy-runtime-api", "aws-smithy-types", "bytes", "bytes-utils", "futures-core", + "futures-util", "http 0.2.12", "http 1.4.0", "http-body 0.4.6", @@ -791,9 +798,9 @@ dependencies = [ [[package]] name = "aws-smithy-http-client" -version = "1.1.3" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1053b5e587e6fa40ce5a79ea27957b04ba660baa02b28b7436f64850152234f1" +checksum = "59e62db736db19c488966c8d787f52e6270be565727236fd5579eaa301e7bc4a" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -804,14 +811,14 @@ dependencies = [ "http 1.4.0", "http-body 0.4.6", "hyper 0.14.32", - "hyper 1.7.0", + "hyper 1.8.1", "hyper-rustls 0.24.2", "hyper-rustls 0.27.7", "hyper-util", "pin-project-lite", "rustls 0.21.12", - "rustls 0.23.34", - "rustls-native-certs 0.8.2", + "rustls 0.23.35", + "rustls-native-certs", "rustls-pki-types", "tokio", "tokio-rustls 0.26.4", @@ -821,27 +828,27 @@ dependencies = [ [[package]] name = "aws-smithy-json" -version = "0.61.6" +version = "0.61.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cff418fc8ec5cadf8173b10125f05c2e7e1d46771406187b2c878557d4503390" +checksum = "a6864c190cbb8e30cf4b77b2c8f3b6dfffa697a09b7218d2f7cd3d4c4065a9f7" dependencies = [ "aws-smithy-types", ] [[package]] name = "aws-smithy-observability" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d1881b1ea6d313f9890710d65c158bdab6fb08c91ea825f74c1c8c357baf4cc" +checksum = "17f616c3f2260612fe44cede278bafa18e73e6479c4e393e2c4518cf2a9a228a" dependencies = [ "aws-smithy-runtime-api", ] [[package]] name = "aws-smithy-query" -version = "0.60.8" +version = "0.60.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d28a63441360c477465f80c7abac3b9c4d075ca638f982e605b7dc2a2c7156c9" +checksum = "ae5d689cf437eae90460e944a58b5668530d433b4ff85789e69d2f2a556e057d" dependencies = [ "aws-smithy-types", "urlencoding", @@ -849,9 +856,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime" -version = "1.9.3" +version = "1.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40ab99739082da5347660c556689256438defae3bcefd66c52b095905730e404" +checksum = "a392db6c583ea4a912538afb86b7be7c5d8887d91604f50eb55c262ee1b4a5f5" dependencies = [ "aws-smithy-async", "aws-smithy-http", @@ -873,9 +880,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime-api" -version = "1.9.1" +version = "1.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3683c5b152d2ad753607179ed71988e8cfd52964443b4f74fd8e552d0bbfeb46" +checksum = "ab0d43d899f9e508300e587bf582ba54c27a452dd0a9ea294690669138ae14a2" dependencies = [ "aws-smithy-async", "aws-smithy-types", @@ -890,9 +897,9 @@ dependencies = [ [[package]] name = "aws-smithy-types" -version = "1.3.3" +version = "1.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f5b3a7486f6690ba25952cabf1e7d75e34d69eaff5081904a47bc79074d6457" +checksum = "905cb13a9895626d49cf2ced759b062d913834c7482c38e49557eac4e6193f01" dependencies = [ "base64-simd", "bytes", @@ -916,18 +923,18 @@ dependencies = [ [[package]] name = "aws-smithy-xml" -version = "0.60.11" +version = "0.60.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9c34127e8c624bc2999f3b657e749c1393bedc9cd97b92a804db8ced4d2e163" +checksum = "11b2f670422ff42bf7065031e72b45bc52a3508bd089f743ea90731ca2b6ea57" dependencies = [ "xmlparser", ] [[package]] name = "aws-types" -version = "1.3.9" +version = "1.3.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2fd329bf0e901ff3f60425691410c69094dc2a1f34b331f37bfc4e9ac1565a1" +checksum = "d79fb68e3d7fe5d4833ea34dc87d2e97d26d3086cb3da660bb6b1f76d98680b6" dependencies = [ "aws-credential-types", "aws-smithy-async", @@ -948,12 +955,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "base64" -version = "0.21.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" - [[package]] name = "base64" version = "0.22.1" @@ -972,9 +973,9 @@ dependencies = [ [[package]] name = "base64ct" -version = "1.8.0" +version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55248b47b0caf0546f7988906588779981c43bb1bc9d0c44087278f80cdb44ba" +checksum = "0e050f626429857a27ddccb31e0aca21356bfa709c04041aefddac081a8f068a" [[package]] name = "bigdecimal" @@ -996,26 +997,6 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "230c5f1ca6a325a32553f8640d31ac9b49f2411e901e427570154868b46da4f7" -[[package]] -name = "bindgen" -version = "0.72.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" -dependencies = [ - "bitflags", - "cexpr", - "clang-sys", - "itertools 0.13.0", - "log", - "prettyplease", - "proc-macro2", - "quote", - "regex", - "rustc-hash", - "shlex", - "syn 2.0.108", -] - [[package]] name = "bitflags" version = "2.10.0" @@ -1099,14 +1080,14 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "borsh" -version = "1.5.7" +version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8646f98db542e39fc66e68a20b2144f6a732636df7c2354e74645faaa433ce" +checksum = "d1da5ab77c1437701eeff7c88d968729e7766172279eab0676857b3d63af7a6f" dependencies = [ "borsh-derive", "cfg_aliases", @@ -1114,15 +1095,15 @@ dependencies = [ [[package]] name = "borsh-derive" -version = "1.5.7" +version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdd1d3c0c2f5833f22386f252fe8ed005c7f59fdcddeef025c01b4c3b9fd9ac3" +checksum = "0686c856aa6aac0c4498f936d7d6a02df690f614c03e4d906d1018062b5c5e2c" dependencies = [ "once_cell", "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1244,9 +1225,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.43" +version = "1.2.49" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "739eb0f94557554b3ca9a86d2d37bebd49c5e6d0c1d2bda35ba5bdac830befc2" +checksum = "90583009037521a116abf44494efecd645ba48b6622457080f080b85544e2215" dependencies = [ "find-msvc-tools", "jobserver", @@ -1254,15 +1235,6 @@ dependencies = [ "shlex", ] -[[package]] -name = "cexpr" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" -dependencies = [ - "nom", -] - [[package]] name = "cfg-if" version = "1.0.4" @@ -1309,22 +1281,11 @@ dependencies = [ "inout", ] -[[package]] -name = "clang-sys" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" -dependencies = [ - "glob", - "libc", - "libloading", -] - [[package]] name = "clap" -version = "4.5.50" +version = "4.5.53" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c2cfd7bf8a6017ddaa4e32ffe7403d547790db06bd171c1c53926faab501623" +checksum = "c9e340e012a1bf4935f5282ed1436d1489548e8f72308207ea5df0e23d2d03f8" dependencies = [ "clap_builder", "clap_derive", @@ -1332,9 +1293,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.50" +version = "4.5.53" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a4c05b9e80c5ccd3a7ef080ad7b6ba7d6fc00a985b8b157197075677c82c7a0" +checksum = "d76b5d13eaa18c901fd2f7fca939fefe3a0727a953561fefdf3b2922b8569d00" dependencies = [ "anstream", "anstyle", @@ -1351,7 +1312,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1395,12 +1356,11 @@ dependencies = [ [[package]] name = "comfy-table" -version = "7.1.2" +version = "7.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56" +checksum = "b03b7db8e0b4b2fdad6c551e634134e99ec000e5c8c3b6856c65e8bbaded7a3b" dependencies = [ - "strum 0.26.3", - "strum_macros 0.26.4", + "unicode-segmentation", "unicode-width 0.2.2", ] @@ -1458,16 +1418,6 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" -[[package]] -name = "core-foundation" -version = "0.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" -dependencies = [ - "core-foundation-sys", - "libc", -] - [[package]] name = "core-foundation" version = "0.10.1" @@ -1495,9 +1445,9 @@ dependencies = [ [[package]] name = "crc" -version = "3.3.0" +version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9710d3b3739c2e349eb44fe848ad0b7c8cb1e42bd87ee49371df2f7acaf3e675" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" dependencies = [ "crc-catalog", ] @@ -1567,9 +1517,9 @@ checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" [[package]] name = "crypto-common" -version = "0.1.6" +version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" dependencies = [ "generic-array", "typenum", @@ -1603,7 +1553,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" dependencies = [ "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1637,7 +1587,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1651,7 +1601,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1662,7 +1612,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1673,7 +1623,7 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -1692,12 +1642,11 @@ dependencies = [ [[package]] name = "datafusion" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af15bb3c6ffa33011ef579f6b0bcbe7c26584688bd6c994f548e44df67f011a" +checksum = "8ba7cb113e9c0bedf9e9765926031e132fa05a1b09ba6e93a6d1a4d7044457b8" dependencies = [ "arrow", - "arrow-ipc", "arrow-schema", "async-trait", "bytes", @@ -1708,6 +1657,7 @@ dependencies = [ "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", + "datafusion-datasource-arrow", "datafusion-datasource-avro", "datafusion-datasource-csv", "datafusion-datasource-json", @@ -1730,7 +1680,6 @@ dependencies = [ "datafusion-sql", "flate2", "futures", - "hex", "itertools 0.14.0", "log", "object_store", @@ -1738,6 +1687,7 @@ dependencies = [ "parquet", "rand 0.9.2", "regex", + "rstest", "sqlparser", "tempfile", "tokio", @@ -1749,9 +1699,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "187622262ad8f7d16d3be9202b4c1e0116f1c9aa387e5074245538b755261621" +checksum = "66a3a799f914a59b1ea343906a0486f17061f39509af74e874a866428951130d" dependencies = [ "arrow", "async-trait", @@ -1764,7 +1714,6 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-plan", "datafusion-session", - "datafusion-sql", "futures", "itertools 0.14.0", "log", @@ -1775,9 +1724,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9657314f0a32efd0382b9a46fdeb2d233273ece64baa68a7c45f5a192daf0f83" +checksum = "6db1b113c80d7a0febcd901476a57aef378e717c54517a163ed51417d87621b0" dependencies = [ "arrow", "async-trait", @@ -1787,10 +1736,11 @@ dependencies = [ "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", - "datafusion-session", "futures", + "itertools 0.14.0", "log", "object_store", "tokio", @@ -1798,16 +1748,18 @@ dependencies = [ [[package]] name = "datafusion-cli" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a0b9c821d14e79070f42ea3a6d6618ced04d94277f0a32301918d7a022c250f" +checksum = "fab982df44f818a749cb5200504ccb919f4608cb9808daf8b3fb98aa7955fd1e" dependencies = [ "arrow", "async-trait", "aws-config", "aws-credential-types", + "chrono", "clap", "datafusion", + "datafusion-common", "dirs", "env_logger", "futures", @@ -1824,20 +1776,19 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a83760d9a13122d025fbdb1d5d5aaf93dd9ada5e90ea229add92aa30898b2d1" +checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" dependencies = [ "ahash 0.8.12", "apache-avro 0.20.0", "arrow", "arrow-ipc", - "base64 0.22.1", "chrono", "half", "hashbrown 0.14.5", "hex", - "indexmap 2.12.0", + "indexmap 2.12.1", "libc", "log", "object_store", @@ -1851,9 +1802,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b6234a6c7173fe5db1c6c35c01a12b2aa0f803a3007feee53483218817f8b1e" +checksum = "b92065bbc6532c6651e2f7dd30b55cba0c7a14f860c7e1d15f165c41a1868d95" dependencies = [ "futures", "log", @@ -1862,9 +1813,9 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7256c9cb27a78709dd42d0c80f0178494637209cac6e29d5c93edd09b6721b86" +checksum = "fde13794244bc7581cd82f6fff217068ed79cdc344cafe4ab2c3a1c3510b38d6" dependencies = [ "arrow", "async-compression", @@ -1887,9 +1838,7 @@ dependencies = [ "itertools 0.14.0", "log", "object_store", - "parquet", "rand 0.9.2", - "tempfile", "tokio", "tokio-util", "url", @@ -1897,47 +1846,64 @@ dependencies = [ "zstd", ] +[[package]] +name = "datafusion-datasource-arrow" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "804fa9b4ecf3157982021770617200ef7c1b2979d57bec9044748314775a9aea" +dependencies = [ + "arrow", + "arrow-ipc", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools 0.14.0", + "object_store", + "tokio", +] + [[package]] name = "datafusion-datasource-avro" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d40b6953ebc9099b37adfd12fde97eb73ff0cee44355c6dea64b8a4537d561" +checksum = "388ed8be535f562cc655b9c3d22edbfb0f1a50a25c242647a98b6d92a75b55a1" dependencies = [ "apache-avro 0.20.0", "arrow", "async-trait", "bytes", - "chrono", - "datafusion-catalog", "datafusion-common", "datafusion-datasource", - "datafusion-execution", - "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "num-traits", "object_store", - "tokio", ] [[package]] name = "datafusion-datasource-csv" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64533a90f78e1684bfb113d200b540f18f268134622d7c96bbebc91354d04825" +checksum = "61a1641a40b259bab38131c5e6f48fac0717bedb7dc93690e604142a849e0568" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", @@ -1949,74 +1915,67 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d7ebeb12c77df0aacad26f21b0d033aeede423a64b2b352f53048a75bf1d6e6" +checksum = "adeacdb00c1d37271176f8fb6a1d8ce096baba16ea7a4b2671840c5c9c64fe85" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "object_store", - "serde_json", "tokio", ] [[package]] name = "datafusion-datasource-parquet" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09e783c4c7d7faa1199af2df4761c68530634521b176a8d1331ddbc5a5c75133" +checksum = "43d0b60ffd66f28bfb026565d62b0a6cbc416da09814766a3797bba7d85a3cd9" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", - "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-pruning", "datafusion-session", "futures", - "hex", "itertools 0.14.0", "log", "object_store", "parking_lot", "parquet", - "rand 0.9.2", "tokio", ] [[package]] name = "datafusion-doc" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99ee6b1d9a80d13f9deb2291f45c07044b8e62fb540dbde2453a18be17a36429" +checksum = "2b99e13947667b36ad713549237362afb054b2d8f8cc447751e23ec61202db07" [[package]] name = "datafusion-execution" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4cec0a57653bec7b933fb248d3ffa3fa3ab3bd33bd140dc917f714ac036f531" +checksum = "63695643190679037bc946ad46a263b62016931547bf119859c511f7ff2f5178" dependencies = [ "arrow", "async-trait", @@ -2035,9 +1994,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef76910bdca909722586389156d0aa4da4020e1631994d50fadd8ad4b1aa05fe" +checksum = "f9a4787cbf5feb1ab351f789063398f67654a6df75c4d37d7f637dc96f951a91" dependencies = [ "arrow", "async-trait", @@ -2048,7 +2007,8 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", - "indexmap 2.12.0", + "indexmap 2.12.1", + "itertools 0.14.0", "paste", "recursive", "serde_json", @@ -2057,26 +2017,26 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d155ccbda29591ca71a1344dd6bed26c65a4438072b400df9db59447f590bb6" +checksum = "5ce2fb1b8c15c9ac45b0863c30b268c69dc9ee7a1ee13ecf5d067738338173dc" dependencies = [ "arrow", "datafusion-common", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "paste", ] [[package]] name = "datafusion-functions" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7de2782136bd6014670fd84fe3b0ca3b3e4106c96403c3ae05c0598577139977" +checksum = "794a9db7f7b96b3346fc007ff25e994f09b8f0511b4cf7dff651fadfe3ebb28f" dependencies = [ "arrow", "arrow-buffer", - "base64 0.22.1", + "base64", "blake2", "blake3", "chrono", @@ -2090,6 +2050,7 @@ dependencies = [ "itertools 0.14.0", "log", "md-5", + "num-traits", "rand 0.9.2", "regex", "sha2", @@ -2099,9 +2060,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07331fc13603a9da97b74fd8a273f4238222943dffdbbed1c4c6f862a30105bf" +checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" dependencies = [ "ahash 0.8.12", "arrow", @@ -2120,9 +2081,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5951e572a8610b89968a09b5420515a121fbc305c0258651f318dc07c97ab17" +checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" dependencies = [ "ahash 0.8.12", "arrow", @@ -2133,9 +2094,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdacca9302c3d8fc03f3e94f338767e786a88a33f5ebad6ffc0e7b50364b9ea3" +checksum = "ae5c06eed03918dc7fe7a9f082a284050f0e9ecf95d72f57712d1496da03b8c4" dependencies = [ "arrow", "arrow-ord", @@ -2143,6 +2104,7 @@ dependencies = [ "datafusion-doc", "datafusion-execution", "datafusion-expr", + "datafusion-expr-common", "datafusion-functions", "datafusion-functions-aggregate", "datafusion-functions-aggregate-common", @@ -2155,9 +2117,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c37ff8a99434fbbad604a7e0669717c58c7c4f14c472d45067c4b016621d981" +checksum = "db4fed1d71738fbe22e2712d71396db04c25de4111f1ec252b8f4c6d3b25d7f5" dependencies = [ "arrow", "async-trait", @@ -2171,9 +2133,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48e2aea7c79c926cffabb13dc27309d4eaeb130f4a21c8ba91cdd241c813652b" +checksum = "1d92206aa5ae21892f1552b4d61758a862a70956e6fd7a95cb85db1de74bc6d1" dependencies = [ "arrow", "datafusion-common", @@ -2189,9 +2151,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fead257ab5fd2ffc3b40fda64da307e20de0040fe43d49197241d9de82a487f" +checksum = "53ae9bcc39800820d53a22d758b3b8726ff84a5a3e24cecef04ef4e5fdf1c7cc" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -2199,20 +2161,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec6f637bce95efac05cdfb9b6c19579ed4aa5f6b94d951cfa5bb054b7bb4f730" +checksum = "1063ad4c9e094b3f798acee16d9a47bd7372d9699be2de21b05c3bd3f34ab848" dependencies = [ - "datafusion-expr", + "datafusion-doc", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "datafusion-optimizer" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6583ef666ae000a613a837e69e456681a9faa96347bf3877661e9e89e141d8a" +checksum = "9f35f9ec5d08b87fd1893a30c2929f2559c2f9806ca072d8fefca5009dc0f06a" dependencies = [ "arrow", "chrono", @@ -2220,7 +2182,7 @@ dependencies = [ "datafusion-expr", "datafusion-expr-common", "datafusion-physical-expr", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "log", "recursive", @@ -2230,9 +2192,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8668103361a272cbbe3a61f72eca60c9b7c706e87cc3565bcf21e2b277b84f6" +checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" dependencies = [ "ahash 0.8.12", "arrow", @@ -2243,9 +2205,8 @@ dependencies = [ "datafusion-physical-expr-common", "half", "hashbrown 0.14.5", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", - "log", "parking_lot", "paste", "petgraph 0.8.3", @@ -2253,9 +2214,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "815acced725d30601b397e39958e0e55630e0a10d66ef7769c14ae6597298bb0" +checksum = "7f9ff2dbd476221b1f67337699eff432781c4e6e1713d2aefdaa517dfbf79768" dependencies = [ "arrow", "datafusion-common", @@ -2268,9 +2229,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6652fe7b5bf87e85ed175f571745305565da2c0b599d98e697bcbedc7baa47c3" +checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" dependencies = [ "ahash 0.8.12", "arrow", @@ -2282,9 +2243,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49b7d623eb6162a3332b564a0907ba00895c505d101b99af78345f1acf929b5c" +checksum = "ce9804f799acd7daef3be7aaffe77c0033768ed8fdbf5fb82fc4c5f2e6bc14e6" dependencies = [ "arrow", "datafusion-common", @@ -2296,15 +2257,14 @@ dependencies = [ "datafusion-physical-plan", "datafusion-pruning", "itertools 0.14.0", - "log", "recursive", ] [[package]] name = "datafusion-physical-plan" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2f7f778a1a838dec124efb96eae6144237d546945587557c9e6936b3414558c" +checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" dependencies = [ "ahash 0.8.12", "arrow", @@ -2323,7 +2283,7 @@ dependencies = [ "futures", "half", "hashbrown 0.14.5", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "log", "parking_lot", @@ -2333,12 +2293,11 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd1e59e2ca14fe3c30f141600b10ad8815e2856caa59ebbd0e3e07cd3d127a65" +checksum = "ac2c2498a1f134a9e11a9f5ed202a2a7d7e9774bd9249295593053ea3be999db" dependencies = [ "arrow", - "arrow-schema", "datafusion-common", "datafusion-datasource", "datafusion-expr-common", @@ -2351,35 +2310,26 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21ef8e2745583619bd7a49474e8f45fbe98ebb31a133f27802217125a7b3d58d" +checksum = "8f96eebd17555386f459037c65ab73aae8df09f464524c709d6a3134ad4f4776" dependencies = [ - "arrow", "async-trait", - "dashmap", "datafusion-common", - "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-plan", - "datafusion-sql", - "futures", - "itertools 0.14.0", - "log", - "object_store", "parking_lot", - "tokio", ] [[package]] name = "datafusion-spark" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613efb6666a7d42fcb922b90cd0daa2b25ea486d141350e5d3e86e46df28309a" +checksum = "97a8d6fed24c80dd403dcc6afec33766a599d1b72575f222237f01429b2e58ba" dependencies = [ "arrow", + "bigdecimal", "chrono", "crc32fast", "datafusion-catalog", @@ -2387,24 +2337,24 @@ dependencies = [ "datafusion-execution", "datafusion-expr", "datafusion-functions", - "datafusion-macros", "log", + "rand 0.9.2", "sha1", "url", - "xxhash-rust", ] [[package]] name = "datafusion-sql" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89abd9868770386fede29e5a4b14f49c0bf48d652c3b9d7a8a0332329b87d50b" +checksum = "3fc195fe60634b2c6ccfd131b487de46dc30eccae8a3c35a13f136e7f440414f" dependencies = [ "arrow", "bigdecimal", + "chrono", "datafusion-common", "datafusion-expr", - "indexmap 2.12.0", + "indexmap 2.12.1", "log", "recursive", "regex", @@ -2413,9 +2363,9 @@ dependencies = [ [[package]] name = "datafusion-sqllogictest" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17598193dd875ca895400c51ccab1c30fceb1855220dc60aa415a4db7c95a2d7" +checksum = "a6830e357705e0d54fda6e3ce70a87c2b255197563c6463d668520cbfc1e0b7c" dependencies = [ "arrow", "async-trait", @@ -2440,14 +2390,15 @@ dependencies = [ [[package]] name = "datafusion-substrait" -version = "50.3.0" +version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eaa011a3814d91a03ab655ad41bbe5e57b203b2859281af8fe2c30aebbbcc5d9" +checksum = "2505af06d103a55b4e8ded0c6aeb6c72a771948da939c0bd3f8eee67af475a9c" dependencies = [ "async-recursion", "async-trait", "chrono", "datafusion", + "half", "itertools 0.14.0", "object_store", "pbjson-types", @@ -2497,7 +2448,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2507,7 +2458,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2546,7 +2497,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2557,7 +2508,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2608,7 +2559,7 @@ dependencies = [ "enum-ordinalize", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2649,7 +2600,7 @@ checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2688,7 +2639,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -2782,9 +2733,9 @@ dependencies = [ [[package]] name = "find-msvc-tools" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52051878f80a721bb68ebfbc930e07b65ba72f2da88968ea5c06fd6ca3d3a127" +checksum = "3a3076410a55c90011c298b04d0cfa770b00fa04e1e3c97d3f6c9de105a03844" [[package]] name = "fixedbitset" @@ -2836,6 +2787,12 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + [[package]] name = "form_urlencoded" version = "1.2.2" @@ -2939,7 +2896,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -2954,6 +2911,12 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" +[[package]] +name = "futures-timer" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" + [[package]] name = "futures-util" version = "0.3.31" @@ -2974,9 +2937,9 @@ dependencies = [ [[package]] name = "generic-array" -version = "0.14.9" +version = "0.14.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", @@ -3039,7 +3002,7 @@ dependencies = [ "futures-sink", "futures-util", "http 0.2.12", - "indexmap 2.12.0", + "indexmap 2.12.1", "slab", "tokio", "tokio-util", @@ -3058,7 +3021,7 @@ dependencies = [ "futures-core", "futures-sink", "http 1.4.0", - "indexmap 2.12.0", + "indexmap 2.12.1", "slab", "tokio", "tokio-util", @@ -3104,14 +3067,19 @@ checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ "allocator-api2", "equivalent", - "foldhash", + "foldhash 0.1.5", ] [[package]] name = "hashbrown" -version = "0.16.0" +version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] [[package]] name = "hashlink" @@ -3128,12 +3096,6 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" -[[package]] -name = "hermit-abi" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" - [[package]] name = "hex" version = "0.4.3" @@ -3278,9 +3240,9 @@ dependencies = [ [[package]] name = "hyper" -version = "1.7.0" +version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb3aa54a13a0dfe7fbe3a59e0c76093041720fdc77b110cc0fc260fafb4dc51e" +checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11" dependencies = [ "atomic-waker", "bytes", @@ -3310,7 +3272,6 @@ dependencies = [ "hyper 0.14.32", "log", "rustls 0.21.12", - "rustls-native-certs 0.6.3", "tokio", "tokio-rustls 0.24.1", ] @@ -3322,31 +3283,31 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" dependencies = [ "http 1.4.0", - "hyper 1.7.0", + "hyper 1.8.1", "hyper-util", - "rustls 0.23.34", - "rustls-native-certs 0.8.2", + "rustls 0.23.35", + "rustls-native-certs", "rustls-pki-types", "tokio", "tokio-rustls 0.26.4", "tower-service", - "webpki-roots 1.0.3", + "webpki-roots 1.0.4", ] [[package]] name = "hyper-util" -version = "0.1.17" +version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c6995591a8f1380fcb4ba966a252a4b29188d51d2b89e3a252f5305be65aea8" +checksum = "727805d60e7938b76b826a6ef209eb70eaa1812794f9424d4a4e2d740662df5f" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-channel", "futures-core", "futures-util", "http 1.4.0", "http-body 1.0.1", - "hyper 1.7.0", + "hyper 1.8.1", "ipnet", "libc", "percent-encoding", @@ -3399,7 +3360,7 @@ dependencies = [ "as-any", "async-trait", "backon", - "base64 0.22.1", + "base64", "bimap", "bytes", "chrono", @@ -3433,9 +3394,8 @@ dependencies = [ "serde_json", "serde_repr", "serde_with", - "strum 0.27.2", + "strum", "tempfile", - "thrift", "tokio", "typed-builder", "url", @@ -3554,7 +3514,7 @@ dependencies = [ "itertools 0.13.0", "regex", "sqlx", - "strum 0.27.2", + "strum", "tempfile", "tokio", ] @@ -3777,21 +3737,21 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.12.0" +version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6717a8d2a5a929a1a2eb43a12812498ed141a0bcfb7e8f7844fbdbe4303bba9f" +checksum = "0ad4bb2b565bca0645f4d68c5c9af97fba094e9791da685bf83cb5f3ce74acf2" dependencies = [ "equivalent", - "hashbrown 0.16.0", + "hashbrown 0.16.1", "serde", "serde_core", ] [[package]] name = "indicatif" -version = "0.18.2" +version = "0.18.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ade6dfcba0dfb62ad59e59e7241ec8912af34fd29e0e743e3db992bd278e8b65" +checksum = "9375e112e4b463ec1b1c6c011953545c65a30164fbab5b581df32b3abf0dcb88" dependencies = [ "console", "portable-atomic", @@ -3834,9 +3794,9 @@ checksum = "469fb0b9cefa57e3ef31275ee7cacb78f2fdca44e4765491884a2b119d4eb130" [[package]] name = "iri-string" -version = "0.7.8" +version = "0.7.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbc5ebe9c3a1a7a5127f920a418f7585e9e758e911d0466ed004f393b0e380b2" +checksum = "4f867b9d1d896b67beb18518eda36fdb77a32ea590de864f1325b294a6d14397" dependencies = [ "memchr", "serde", @@ -3874,28 +3834,28 @@ checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c" [[package]] name = "jiff" -version = "0.2.15" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be1f93b8b1eb69c77f24bbb0afdf66f54b632ee39af40ca21c4365a1d7347e49" +checksum = "49cce2b81f2098e7e3efc35bc2e0a6b7abec9d34128283d7a26fa8f32a6dbb35" dependencies = [ "jiff-static", "jiff-tzdb-platform", "log", "portable-atomic", "portable-atomic-util", - "serde", - "windows-sys 0.59.0", + "serde_core", + "windows-sys 0.61.2", ] [[package]] name = "jiff-static" -version = "0.2.15" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03343451ff899767262ec32146f6d559dd759fdadf42ff0e227c7c48f72594b4" +checksum = "980af8b43c3ad5d8d349ace167ec8170839f753a42d233ba19e08afe1850fa69" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -3925,9 +3885,9 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.82" +version = "0.3.83" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b011eec8cc36da2aab2d5cff675ec18454fad408585853910a202391cf9f8e65" +checksum = "464a3709c7f55f1f721e5389aa6ea4e3bc6aba669353300af094b29ffbdde1d8" dependencies = [ "once_cell", "wasm-bindgen", @@ -3939,7 +3899,7 @@ version = "9.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a87cc7a48537badeae96744432de36f4be2b4a34a05a5ef32e9dd8a1c169dde" dependencies = [ - "base64 0.22.1", + "base64", "js-sys", "pem", "ring", @@ -4022,19 +3982,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" [[package]] name = "libc" -version = "0.2.177" +version = "0.2.178" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2874a2af47a2325c2001a6e6fad9b16a53b802102b528163885171cf92b15976" - -[[package]] -name = "libloading" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" -dependencies = [ - "cfg-if", - "windows-link", -] +checksum = "37c93d8daa9d8a012fd8ab92f088405fb202ea0b6ab73ee2482ae66af4f42091" [[package]] name = "libm" @@ -4088,9 +4038,9 @@ dependencies = [ [[package]] name = "libz-rs-sys" -version = "0.5.2" +version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "840db8cf39d9ec4dd794376f38acc40d0fc65eec2a8f484f7fd375b84602becd" +checksum = "8b484ba8d4f775eeca644c452a56650e544bf7e617f1d170fe7298122ead5222" dependencies = [ "zlib-rs", ] @@ -4135,9 +4085,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.28" +version = "0.4.29" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432" +checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" [[package]] name = "lru-slab" @@ -4147,9 +4097,9 @@ checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" [[package]] name = "lz4_flex" -version = "0.11.5" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08ab2867e3eeeca90e844d1940eab391c9dc5228783db2ed999acbc0a9ed375a" +checksum = "ab6473172471198271ff72e9379150e9dfd70d8e533e0752a27e515b48dd375e" dependencies = [ "twox-hash", ] @@ -4221,12 +4171,6 @@ dependencies = [ "serde", ] -[[package]] -name = "minimal-lexical" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" - [[package]] name = "miniz_oxide" version = "0.8.9" @@ -4239,9 +4183,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.1.0" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69d83b0086dc8ecf3ce9ae2874b2d1290252e2a30720bea58a5c6639b0092873" +checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" dependencies = [ "libc", "wasi", @@ -4271,25 +4215,26 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "mockito" -version = "1.7.0" +version = "1.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7760e0e418d9b7e5777c0374009ca4c93861b9066f18cb334a20ce50ab63aa48" +checksum = "7e0603425789b4a70fcc4ac4f5a46a566c116ee3e2a6b768dc623f7719c611de" dependencies = [ "assert-json-diff", "bytes", "colored", - "futures-util", + "futures-core", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.7.0", + "hyper 1.8.1", "hyper-util", "log", + "pin-project-lite", "rand 0.9.2", "regex", "serde_json", @@ -4339,7 +4284,7 @@ checksum = "b40e46c845ac234bcba19db7ab252bc2778cbadd516a466d2f12b1580852d136" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -4365,7 +4310,7 @@ checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -4414,37 +4359,13 @@ dependencies = [ "libc", ] -[[package]] -name = "nom" -version = "7.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" -dependencies = [ - "memchr", - "minimal-lexical", -] - [[package]] name = "nu-ansi-term" version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", -] - -[[package]] -name = "num" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" -dependencies = [ - "num-bigint", - "num-complex", - "num-integer", - "num-iter", - "num-rational", - "num-traits", + "windows-sys 0.61.2", ] [[package]] @@ -4460,11 +4381,10 @@ dependencies = [ [[package]] name = "num-bigint-dig" -version = "0.8.4" +version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc84195820f291c7697304f3cbdadd1cb7199c0efc917ff5eafd71225c136151" +checksum = "e661dda6640fad38e827a6d4a310ff4763082116fe217f279885c97f511bb0b7" dependencies = [ - "byteorder", "lazy_static", "libm", "num-integer", @@ -4510,17 +4430,6 @@ dependencies = [ "num-traits", ] -[[package]] -name = "num-rational" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" -dependencies = [ - "num-bigint", - "num-integer", - "num-traits", -] - [[package]] name = "num-traits" version = "0.2.19" @@ -4531,16 +4440,6 @@ dependencies = [ "libm", ] -[[package]] -name = "num_cpus" -version = "1.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" -dependencies = [ - "hermit-abi", - "libc", -] - [[package]] name = "num_enum" version = "0.7.5" @@ -4560,7 +4459,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -4579,7 +4478,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c1be0c6c22ec0817cdc77d3842f721a17fd30ab6965001415b5402a74e6b740" dependencies = [ "async-trait", - "base64 0.22.1", + "base64", "bytes", "chrono", "form_urlencoded", @@ -4587,16 +4486,16 @@ dependencies = [ "http 1.4.0", "http-body-util", "humantime", - "hyper 1.7.0", + "hyper 1.8.1", "itertools 0.14.0", "md-5", "parking_lot", "percent-encoding", - "quick-xml 0.38.3", + "quick-xml 0.38.4", "rand 0.9.2", "reqwest", "ring", - "rustls-pemfile 2.2.0", + "rustls-pemfile", "serde", "serde_json", "serde_urlencoded", @@ -4629,7 +4528,7 @@ checksum = "d075ab8a203a6ab4bc1bce0a4b9fe486a72bf8b939037f4b78d95386384bc80a" dependencies = [ "anyhow", "backon", - "base64 0.22.1", + "base64", "bytes", "crc32c", "futures", @@ -4640,7 +4539,7 @@ dependencies = [ "log", "md-5", "percent-encoding", - "quick-xml 0.38.3", + "quick-xml 0.38.4", "reqsign", "reqwest", "serde", @@ -4735,9 +4634,9 @@ dependencies = [ [[package]] name = "parquet" -version = "56.2.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0dbd48ad52d7dccf8ea1b90a3ddbfaea4f69878dd7683e51c507d4bc52b5b27" +checksum = "be3e4f6d320dd92bfa7d612e265d7d08bba0a240bab86af3425e1d255a511d89" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -4747,17 +4646,18 @@ dependencies = [ "arrow-ipc", "arrow-schema", "arrow-select", - "base64 0.22.1", + "base64", "brotli", "bytes", "chrono", "flate2", "futures", "half", - "hashbrown 0.16.0", + "hashbrown 0.16.1", "lz4_flex", - "num", "num-bigint", + "num-integer", + "num-traits", "object_store", "paste", "ring", @@ -4778,31 +4678,31 @@ checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" [[package]] name = "pbjson" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7e6349fa080353f4a597daffd05cb81572a9c031a6d4fff7e504947496fcc68" +checksum = "898bac3fa00d0ba57a4e8289837e965baa2dee8c3749f3b11d45a64b4223d9c3" dependencies = [ - "base64 0.21.7", + "base64", "serde", ] [[package]] name = "pbjson-build" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6eea3058763d6e656105d1403cb04e0a41b7bbac6362d413e7c33be0c32279c9" +checksum = "af22d08a625a2213a78dbb0ffa253318c5c79ce3133d32d296655a7bdfb02095" dependencies = [ "heck", - "itertools 0.13.0", + "itertools 0.14.0", "prost", "prost-types", ] [[package]] name = "pbjson-types" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e54e5e7bfb1652f95bc361d76f3c780d8e526b134b85417e774166ee941f0887" +checksum = "8e748e28374f10a330ee3bb9f29b828c0ac79831a32bab65015ad9b661ead526" dependencies = [ "bytes", "chrono", @@ -4829,7 +4729,7 @@ version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" dependencies = [ - "base64 0.22.1", + "base64", "serde_core", ] @@ -4855,7 +4755,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" dependencies = [ "fixedbitset", - "indexmap 2.12.0", + "indexmap 2.12.1", ] [[package]] @@ -4866,7 +4766,7 @@ checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset", "hashbrown 0.15.5", - "indexmap 2.12.0", + "indexmap 2.12.1", "serde", ] @@ -4927,7 +4827,7 @@ checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5013,7 +4913,7 @@ version = "0.6.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fbef655056b916eb868048276cfd5d6a7dea4f81560dfd047f97c8c6fe3fcfd4" dependencies = [ - "base64 0.22.1", + "base64", "byteorder", "bytes", "fallible-iterator", @@ -5103,7 +5003,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5112,7 +5012,7 @@ version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" dependencies = [ - "toml_edit 0.23.7", + "toml_edit 0.23.9", ] [[package]] @@ -5126,9 +5026,9 @@ dependencies = [ [[package]] name = "prost" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +checksum = "7231bd9b3d3d33c86b58adbac74b5ec0ad9f496b19d22801d773636feaa95f3d" dependencies = [ "bytes", "prost-derive", @@ -5136,9 +5036,9 @@ dependencies = [ [[package]] name = "prost-build" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +checksum = "ac6c3320f9abac597dcbc668774ef006702672474aad53c6d596b62e487b40b1" dependencies = [ "heck", "itertools 0.14.0", @@ -5150,28 +5050,28 @@ dependencies = [ "prost", "prost-types", "regex", - "syn 2.0.108", + "syn 2.0.111", "tempfile", ] [[package]] name = "prost-derive" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +checksum = "9120690fafc389a67ba3803df527d0ec9cbbc9cc45e4cc20b332996dfb672425" dependencies = [ "anyhow", "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "prost-types" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" +checksum = "b9b4db3d6da204ed77bb26ba83b6122a73aeb2e87e25fbf7ad2e84c4ccbf8f72" dependencies = [ "prost", ] @@ -5223,7 +5123,7 @@ checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5244,9 +5144,9 @@ dependencies = [ [[package]] name = "quick-xml" -version = "0.38.3" +version = "0.38.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42a232e7487fc2ef313d96dde7948e7a3c05101870d8985e4fd8d26aedd27b89" +checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c" dependencies = [ "memchr", "serde", @@ -5264,7 +5164,7 @@ dependencies = [ "quinn-proto", "quinn-udp", "rustc-hash", - "rustls 0.23.34", + "rustls 0.23.35", "socket2 0.6.1", "thiserror 2.0.17", "tokio", @@ -5284,7 +5184,7 @@ dependencies = [ "rand 0.9.2", "ring", "rustc-hash", - "rustls 0.23.34", + "rustls 0.23.35", "rustls-pki-types", "slab", "thiserror 2.0.17", @@ -5309,9 +5209,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.41" +version = "1.0.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1" +checksum = "a338cc41d27e6cc6dce6cefc13a0729dfbb81c262b1f519331575dd80ef3067f" dependencies = [ "proc-macro2", ] @@ -5425,7 +5325,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5465,7 +5365,7 @@ checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5505,14 +5405,20 @@ checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58" [[package]] name = "regress" -version = "0.10.4" +version = "0.10.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "145bb27393fe455dd64d6cbc8d059adfa392590a45eadf079c01b11857e7b010" +checksum = "2057b2325e68a893284d1538021ab90279adac1139957ca2a74426c6f118fb48" dependencies = [ - "hashbrown 0.15.5", + "hashbrown 0.16.1", "memchr", ] +[[package]] +name = "relative-path" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" + [[package]] name = "rend" version = "0.4.2" @@ -5536,7 +5442,7 @@ checksum = "43451dbf3590a7590684c25fb8d12ecdcc90ed3ac123433e500447c7d77ed701" dependencies = [ "anyhow", "async-trait", - "base64 0.22.1", + "base64", "chrono", "form_urlencoded", "getrandom 0.2.16", @@ -5562,11 +5468,11 @@ dependencies = [ [[package]] name = "reqwest" -version = "0.12.24" +version = "0.12.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d0946410b9f7b082a427e4ef5c8ff541a88b357bc6c637c40db3a68ac70a36f" +checksum = "b6eff9328d40131d43bd911d42d79eb6a47312002a4daefc9e37f17e74a7701a" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-core", "futures-util", @@ -5574,7 +5480,7 @@ dependencies = [ "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.7.0", + "hyper 1.8.1", "hyper-rustls 0.27.7", "hyper-util", "js-sys", @@ -5582,8 +5488,8 @@ dependencies = [ "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.34", - "rustls-native-certs 0.8.2", + "rustls 0.23.35", + "rustls-native-certs", "rustls-pki-types", "serde", "serde_json", @@ -5600,7 +5506,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams", "web-sys", - "webpki-roots 1.0.3", + "webpki-roots 1.0.4", ] [[package]] @@ -5643,7 +5549,7 @@ checksum = "35a640b26f007713818e9a9b65d34da1cf58538207b052916a83d80e43f3ffa4" dependencies = [ "bytes", "hashbrown 0.15.5", - "indexmap 2.12.0", + "indexmap 2.12.1", "munge", "ptr_meta 0.3.1", "rancor", @@ -5672,7 +5578,7 @@ checksum = "bd83f5f173ff41e00337d97f6572e416d022ef8a19f371817259ae960324c482" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -5687,9 +5593,9 @@ dependencies = [ [[package]] name = "rsa" -version = "0.9.8" +version = "0.9.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78928ac1ed176a5ca1d17e578a1825f3d81ca54cf41053a592584b020cfd691b" +checksum = "40a0376c50d0358279d9d643e4bf7b7be212f1f4ff1da9070a7b54d22ef75c88" dependencies = [ "const-oid", "digest", @@ -5706,6 +5612,35 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rstest" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5a3193c063baaa2a95a33f03035c8a72b83d97a54916055ba22d35ed3839d49" +dependencies = [ + "futures-timer", + "futures-util", + "rstest_macros", +] + +[[package]] +name = "rstest_macros" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0" +dependencies = [ + "cfg-if", + "glob", + "proc-macro-crate", + "proc-macro2", + "quote", + "regex", + "relative-path", + "rustc_version", + "syn 2.0.111", + "unicode-ident", +] + [[package]] name = "rust-ini" version = "0.21.3" @@ -5761,7 +5696,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -5778,9 +5713,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.34" +version = "0.23.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a9586e9ee2b4f8fab52a0048ca7334d7024eef48e2cb9407e3497bb7cab7fa7" +checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f" dependencies = [ "aws-lc-rs", "once_cell", @@ -5791,18 +5726,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-native-certs" -version = "0.6.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9aace74cb666635c918e9c12bc0d348266037aa8eb599b5cba565709a8dff00" -dependencies = [ - "openssl-probe", - "rustls-pemfile 1.0.4", - "schannel", - "security-framework 2.11.1", -] - [[package]] name = "rustls-native-certs" version = "0.8.2" @@ -5812,16 +5735,7 @@ dependencies = [ "openssl-probe", "rustls-pki-types", "schannel", - "security-framework 3.5.1", -] - -[[package]] -name = "rustls-pemfile" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c74cae0a4cf6ccbbf5f359f08efdf8ee7e1dc532573bf0db71968cb56b1448c" -dependencies = [ - "base64 0.21.7", + "security-framework", ] [[package]] @@ -5835,9 +5749,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.13.0" +version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94182ad936a0c91c324cd46c6511b9510ed16af436d7b5bab34beab0afd55f7a" +checksum = "708c0f9d5f54ba0272468c1d306a52c495b31fa155e91bc25371e6df7996908c" dependencies = [ "web-time", "zeroize", @@ -5952,9 +5866,9 @@ dependencies = [ [[package]] name = "schemars" -version = "1.0.4" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82d20c4491bc164fa2f6c5d44565947a52ad80b9505d8e36f8d54c27c739fcd0" +checksum = "9558e172d4e8533736ba97870c4b2cd63f84b382a3d6eb063da41b91cce17289" dependencies = [ "dyn-clone", "ref-cast", @@ -5971,7 +5885,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6007,19 +5921,6 @@ version = "4.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b" -[[package]] -name = "security-framework" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" -dependencies = [ - "bitflags", - "core-foundation 0.9.4", - "core-foundation-sys", - "libc", - "security-framework-sys", -] - [[package]] name = "security-framework" version = "3.5.1" @@ -6027,7 +5928,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b3297343eaf830f66ede390ea39da1d462b6b0c1b000f420d0a83f898bbbe6ef" dependencies = [ "bitflags", - "core-foundation 0.10.1", + "core-foundation", "core-foundation-sys", "libc", "security-framework-sys", @@ -6096,7 +5997,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6107,7 +6008,7 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6131,7 +6032,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6152,7 +6053,7 @@ dependencies = [ "proc-macro2", "quote", "serde", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6169,17 +6070,17 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.16.0" +version = "3.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10574371d41b0d9b2cff89418eda27da52bcaff2cc8741db26382a77c29131f1" +checksum = "4fa237f2807440d238e0364a218270b98f767a00d3dada77b1c53ae88940e2e7" dependencies = [ - "base64 0.22.1", + "base64", "chrono", "hex", "indexmap 1.9.3", - "indexmap 2.12.0", + "indexmap 2.12.1", "schemars 0.9.0", - "schemars 1.0.4", + "schemars 1.1.0", "serde_core", "serde_json", "serde_with_macros", @@ -6188,14 +6089,14 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.16.0" +version = "3.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08a72d8216842fdd57820dc78d840bef99248e35fb2554ff923319e60f2d686b" +checksum = "52a8e3ca0ca629121f70ab50f95249e5a6f925cc0f6ffe8256c45b728875706c" dependencies = [ "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6204,7 +6105,7 @@ version = "0.9.34+deprecated" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" dependencies = [ - "indexmap 2.12.0", + "indexmap 2.12.1", "itoa", "ryu", "serde", @@ -6250,9 +6151,9 @@ checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" [[package]] name = "signal-hook-registry" -version = "1.4.6" +version = "1.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2a4719bff48cee6b39d12c020eeb490953ad2443b7055bd0b21fca26bd8c28b" +checksum = "7664a098b8e616bdfcc2dc0e9ac44eb231eedf41db4e9fe95d8d32ec728dedad" dependencies = [ "libc", ] @@ -6269,9 +6170,9 @@ dependencies = [ [[package]] name = "simd-adler32" -version = "0.3.7" +version = "0.3.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d66dc143e6b11c1eddc06d5c423cfc97062865baf299914ab64caa38182078fe" +checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" [[package]] name = "simdutf8" @@ -6375,9 +6276,9 @@ dependencies = [ [[package]] name = "sonic-simd" -version = "0.1.1" +version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b421f7b6aa4a5de8f685aaf398dfaa828346ee639d2b1c1061ab43d40baa6223" +checksum = "5707edbfb34a40c9f2a55fa09a49101d9fec4e0cc171ce386086bd9616f34257" dependencies = [ "cfg-if", ] @@ -6428,9 +6329,9 @@ dependencies = [ [[package]] name = "sqlparser" -version = "0.58.0" +version = "0.59.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec4b661c54b1e4b603b37873a18c59920e4c51ea8ea2cf527d925424dbd4437c" +checksum = "4591acadbcf52f0af60eafbb2c003232b2b4cd8de5f0e9437cb8b1b59046cc0f" dependencies = [ "log", "recursive", @@ -6445,7 +6346,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6467,7 +6368,7 @@ version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "crc", "crossbeam-queue", @@ -6479,12 +6380,12 @@ dependencies = [ "futures-util", "hashbrown 0.15.5", "hashlink", - "indexmap 2.12.0", + "indexmap 2.12.1", "log", "memchr", "once_cell", "percent-encoding", - "rustls 0.23.34", + "rustls 0.23.35", "serde", "serde_json", "sha2", @@ -6507,7 +6408,7 @@ dependencies = [ "quote", "sqlx-core", "sqlx-macros-core", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6528,7 +6429,7 @@ dependencies = [ "sha2", "sqlx-core", "sqlx-sqlite", - "syn 2.0.108", + "syn 2.0.111", "tokio", "url", ] @@ -6540,7 +6441,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526" dependencies = [ "atoi", - "base64 0.22.1", + "base64", "bitflags", "byteorder", "bytes", @@ -6581,7 +6482,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46" dependencies = [ "atoi", - "base64 0.22.1", + "base64", "bitflags", "byteorder", "crc", @@ -6671,32 +6572,13 @@ version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" -[[package]] -name = "strum" -version = "0.26.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" - [[package]] name = "strum" version = "0.27.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" dependencies = [ - "strum_macros 0.27.2", -] - -[[package]] -name = "strum_macros" -version = "0.26.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "rustversion", - "syn 2.0.108", + "strum_macros", ] [[package]] @@ -6708,7 +6590,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6723,9 +6605,9 @@ dependencies = [ [[package]] name = "substrait" -version = "0.58.0" +version = "0.62.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de6d24c270c6c672a86c183c3a8439ba46c1936f93cf7296aa692de3b0ff0228" +checksum = "21f1cb6d0bcd097a39fc25f7236236be29881fe122e282e4173d6d007a929927" dependencies = [ "heck", "pbjson", @@ -6741,7 +6623,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.108", + "syn 2.0.111", "typify", "walkdir", ] @@ -6765,9 +6647,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.108" +version = "2.0.111" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da58917d35242480a05c2897064da0a80589a2a0476c9a3f2fdc83b53502e917" +checksum = "390cc9a294ab71bdb1aa2e99d13be9c753cd2d7bd6560c77118597410c4d2e87" dependencies = [ "proc-macro2", "quote", @@ -6791,7 +6673,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6816,7 +6698,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -6851,7 +6733,7 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6862,7 +6744,7 @@ checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -6874,15 +6756,6 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "threadpool" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d050e60b33d41c19108b32cea32164033a9013fe3b46cbd4457559bfbf77afaa" -dependencies = [ - "num_cpus", -] - [[package]] name = "thrift" version = "0.17.0" @@ -6891,9 +6764,7 @@ checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" dependencies = [ "byteorder", "integer-encoding 3.0.4", - "log", "ordered-float 2.10.1", - "threadpool", ] [[package]] @@ -6986,7 +6857,7 @@ checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -7005,7 +6876,7 @@ version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.34", + "rustls 0.23.35", "tokio", ] @@ -7022,9 +6893,9 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.16" +version = "0.7.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14307c986784f72ef81c89db7d9e28d6ac26d16213b109ea501696195e6e3ce5" +checksum = "2efa149fe76073d6e8fd97ef4f4eca7b67f599660115591483572e406e165594" dependencies = [ "bytes", "futures-core", @@ -7069,7 +6940,7 @@ version = "0.22.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" dependencies = [ - "indexmap 2.12.0", + "indexmap 2.12.1", "serde", "serde_spanned", "toml_datetime 0.6.11", @@ -7079,11 +6950,11 @@ dependencies = [ [[package]] name = "toml_edit" -version = "0.23.7" +version = "0.23.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6485ef6d0d9b5d0ec17244ff7eb05310113c3f316f2d14200d4de56b3cb98f8d" +checksum = "5d7cbc3b4b49633d57a0509303158ca50de80ae32c265093b24c414705807832" dependencies = [ - "indexmap 2.12.0", + "indexmap 2.12.1", "toml_datetime 0.7.3", "toml_parser", "winnow", @@ -7121,9 +6992,9 @@ dependencies = [ [[package]] name = "tower-http" -version = "0.6.6" +version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adc82fd73de2a9722ac5da747f12383d2bfdb93591ee6c58486e0097890f05f2" +checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8" dependencies = [ "bitflags", "bytes", @@ -7151,9 +7022,9 @@ checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" [[package]] name = "tracing" -version = "0.1.41" +version = "0.1.43" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0" +checksum = "2d15d90a0b5c19378952d479dc858407149d7bb45a14de0142f6c534b16fc647" dependencies = [ "log", "pin-project-lite", @@ -7163,20 +7034,20 @@ dependencies = [ [[package]] name = "tracing-attributes" -version = "0.1.30" +version = "0.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "tracing-core" -version = "0.1.34" +version = "0.1.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d12581f227e93f094d3af2ae690a574abb8a2b9b7a96e7cfe9647b2b617678" +checksum = "7a04e24fab5c89c6a36eb8558c9656f30d81de51dfa4d3b45f26b21d61fa0a6c" dependencies = [ "once_cell", "valuable", @@ -7195,9 +7066,9 @@ dependencies = [ [[package]] name = "tracing-subscriber" -version = "0.3.20" +version = "0.3.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2054a14f5307d601f88daf0553e1cbf472acc4f2c51afab632431cdcd72124d5" +checksum = "2f30143827ddab0d256fd843b7a66d164e9f271cfa0dde49142c5ca0ca291f1e" dependencies = [ "nu-ansi-term", "sharded-slab", @@ -7236,7 +7107,7 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -7247,9 +7118,9 @@ checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" [[package]] name = "typify" -version = "0.4.3" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7144144e97e987c94758a3017c920a027feac0799df325d6df4fc8f08d02068e" +checksum = "e6d5bcc6f62eb1fa8aa4098f39b29f93dcb914e17158b76c50360911257aa629" dependencies = [ "typify-impl", "typify-macro", @@ -7257,9 +7128,9 @@ dependencies = [ [[package]] name = "typify-impl" -version = "0.4.3" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "062879d46aa4c9dfe0d33b035bbaf512da192131645d05deacb7033ec8581a09" +checksum = "a1eb359f7ffa4f9ebe947fa11a1b2da054564502968db5f317b7e37693cb2240" dependencies = [ "heck", "log", @@ -7270,16 +7141,16 @@ dependencies = [ "semver", "serde", "serde_json", - "syn 2.0.108", + "syn 2.0.111", "thiserror 2.0.17", "unicode-ident", ] [[package]] name = "typify-macro" -version = "0.4.3" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9708a3ceb6660ba3f8d2b8f0567e7d4b8b198e2b94d093b8a6077a751425de9e" +checksum = "911c32f3c8514b048c1b228361bebb5e6d73aeec01696e8cc0e82e2ffef8ab7a" dependencies = [ "proc-macro2", "quote", @@ -7288,7 +7159,7 @@ dependencies = [ "serde", "serde_json", "serde_tokenstream", - "syn 2.0.108", + "syn 2.0.111", "typify-impl", ] @@ -7300,24 +7171,24 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" -version = "1.0.20" +version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "462eeb75aeb73aea900253ce739c8e18a67423fadf006037cd3ff27e82748a06" +checksum = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5" [[package]] name = "unicode-normalization" -version = "0.1.24" +version = "0.1.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5033c97c4262335cded6d6fc3e5c18ab755e1a3dc96376350f3d8e9f009ad956" +checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" dependencies = [ "tinyvec", ] [[package]] name = "unicode-properties" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e70f2a8b45122e719eb623c01822704c4e0907e7e426a05927e1a1cfff5b75d0" +checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" [[package]] name = "unicode-segmentation" @@ -7339,9 +7210,9 @@ checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" [[package]] name = "unit-prefix" -version = "0.5.1" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "323402cff2dd658f39ca17c789b502021b3f18707c91cdf22e3838e1b4023817" +checksum = "81e544489bf3d8ef66c953931f56617f423cd4b5494be343d9b9d3dda037b9a3" [[package]] name = "unsafe-libyaml" @@ -7519,9 +7390,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasm-bindgen" -version = "0.2.105" +version = "0.2.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da95793dfc411fbbd93f5be7715b0578ec61fe87cb1a42b12eb625caa5c5ea60" +checksum = "0d759f433fa64a2d763d1340820e46e111a7a5ab75f993d1852d70b03dbb80fd" dependencies = [ "cfg-if", "once_cell", @@ -7532,9 +7403,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-futures" -version = "0.4.55" +version = "0.4.56" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "551f88106c6d5e7ccc7cd9a16f312dd3b5d36ea8b4954304657d5dfba115d4a0" +checksum = "836d9622d604feee9e5de25ac10e3ea5f2d65b41eac0d9ce72eb5deae707ce7c" dependencies = [ "cfg-if", "js-sys", @@ -7545,9 +7416,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.105" +version = "0.2.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04264334509e04a7bf8690f2384ef5265f05143a4bff3889ab7a3269adab59c2" +checksum = "48cb0d2638f8baedbc542ed444afc0644a29166f1595371af4fecf8ce1e7eeb3" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -7555,22 +7426,22 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.105" +version = "0.2.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "420bc339d9f322e562942d52e115d57e950d12d88983a14c79b86859ee6c7ebc" +checksum = "cefb59d5cd5f92d9dcf80e4683949f15ca4b511f4ac0a6e14d4e1ac60c6ecd40" dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.105" +version = "0.2.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76f218a38c84bcb33c25ec7059b07847d465ce0e0a76b995e134a45adcb6af76" +checksum = "cbc538057e648b67f72a982e708d485b2efa771e1ac05fec311f9f63e5800db4" dependencies = [ "unicode-ident", ] @@ -7590,9 +7461,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.82" +version = "0.3.83" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a1f95c0d03a47f4ae1f7a64643a6bb97465d9b740f0fa8f90ea33915c99a9a1" +checksum = "9b32828d774c412041098d182a8b38b16ea816958e07cf40eec2bc080ae137ac" dependencies = [ "js-sys", "wasm-bindgen", @@ -7614,14 +7485,14 @@ version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.3", + "webpki-roots 1.0.4", ] [[package]] name = "webpki-roots" -version = "1.0.3" +version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32b130c0d2d49f8b6889abc456e795e82525204f27c42cf767cf0d7734e089b8" +checksum = "b2878ef029c47c6e8cf779119f20fcf52bde7ad42a731b2a304bc221df17571e" dependencies = [ "rustls-pki-types", ] @@ -7642,7 +7513,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -7666,7 +7537,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -7677,7 +7548,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -7937,9 +7808,9 @@ checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" [[package]] name = "winnow" -version = "0.7.13" +version = "0.7.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21a0236b59786fed61e2a80582dd500fe61f18b5dca67a4a067d0bc9039339cf" +checksum = "5a5364e9d77fcdeeaa6062ced926ee3381faa2ee02d3eb83a5c27a8825540829" dependencies = [ "memchr", ] @@ -7971,12 +7842,6 @@ version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" -[[package]] -name = "xxhash-rust" -version = "0.8.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3" - [[package]] name = "xz2" version = "0.1.7" @@ -8011,28 +7876,28 @@ checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", "synstructure", ] [[package]] name = "zerocopy" -version = "0.8.27" +version = "0.8.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0894878a5fa3edfd6da3f88c4805f4c8558e2b996227a3d864f47fe11e38282c" +checksum = "fd74ec98b9250adb3ca554bdde269adf631549f51d8a8f8f0a10b50f1cb298c3" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.27" +version = "0.8.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831" +checksum = "d8a8d209fdf45cf5138cbb5a506f6b52522a25afccc534d1475dad8e31105c6a" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] @@ -8052,7 +7917,7 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", "synstructure", ] @@ -8092,14 +7957,14 @@ checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.111", ] [[package]] name = "zlib-rs" -version = "0.5.2" +version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f06ae92f42f5e5c42443fd094f245eb656abf56dd7cce9b8b263236565e00f2" +checksum = "36134c44663532e6519d7a6dfdbbe06f6f8192bde8ae9ed076e9b213f0e31df7" [[package]] name = "zstd" diff --git a/Cargo.toml b/Cargo.toml index 9904820dea..ded3aedecb 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -42,14 +42,14 @@ rust-version = "1.88" anyhow = "1.0.72" apache-avro = { version = "0.21", features = ["zstandard"] } array-init = "2" -arrow-arith = "56.2" -arrow-array = "56.2" -arrow-buffer = "56.2" -arrow-cast = "56.2" -arrow-ord = "56.2" -arrow-schema = "56.2" -arrow-select = "56.2" -arrow-string = "56.2" +arrow-arith = "57.0" +arrow-array = "57.0" +arrow-buffer = "57.0" +arrow-cast = "57.0" +arrow-ord = "57.0" +arrow-schema = "57.0" +arrow-select = "57.0" +arrow-string = "57.0" as-any = "0.3.2" async-trait = "0.1.89" aws-config = "1.8.7" @@ -62,9 +62,9 @@ bytes = "1.10" chrono = "0.4.41" clap = { version = "4.5.48", features = ["derive", "cargo"] } ctor = "0.2.8" -datafusion = "50" -datafusion-cli = "50" -datafusion-sqllogictest = "50" +datafusion = "51.0" +datafusion-cli = "51.0" +datafusion-sqllogictest = "51.0" derive_builder = "0.20" dirs = "6" enum-ordinalize = "4.3.0" @@ -101,7 +101,7 @@ num-bigint = "0.4.6" once_cell = "1.20" opendal = "0.55.0" ordered-float = "4" -parquet = "56.2" +parquet = "57.0" pilota = "0.11.10" port_scanner = "0.1.5" pretty_assertions = "1.4" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 814c9afb35..a7244cc8dd 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -180,9 +180,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e833808ff2d94ed40d9379848a950d995043c7fb3e81a30b383f4c6033821cc" +checksum = "4df8bb5b0bd64c0b9bc61317fcc480bad0f00e56d3bc32c69a4c8dada4786bae" dependencies = [ "arrow-arith", "arrow-array", @@ -202,23 +202,23 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad08897b81588f60ba983e3ca39bda2b179bdd84dced378e7df81a5313802ef8" +checksum = "a1a640186d3bd30a24cb42264c2dafb30e236a6f50d510e56d40b708c9582491" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "chrono", - "num", + "num-traits", ] [[package]] name = "arrow-array" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8548ca7c070d8db9ce7aa43f37393e4bfcf3f2d3681df278490772fd1673d08d" +checksum = "219fe420e6800979744c8393b687afb0252b3f8a89b91027d27887b72aa36d31" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -228,25 +228,28 @@ dependencies = [ "chrono-tz", "half", "hashbrown 0.16.0", - "num", + "num-complex", + "num-integer", + "num-traits", ] [[package]] name = "arrow-buffer" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e003216336f70446457e280807a73899dd822feaf02087d31febca1363e2fccc" +checksum = "76885a2697a7edf6b59577f568b456afc94ce0e2edc15b784ce3685b6c3c5c27" dependencies = [ "bytes", "half", - "num", + "num-bigint", + "num-traits", ] [[package]] name = "arrow-cast" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "919418a0681298d3a77d1a315f625916cb5678ad0d74b9c60108eb15fd083023" +checksum = "9c9ebb4c987e6b3b236fb4a14b20b34835abfdd80acead3ccf1f9bf399e1f168" dependencies = [ "arrow-array", "arrow-buffer", @@ -259,15 +262,15 @@ dependencies = [ "comfy-table", "half", "lexical-core", - "num", + "num-traits", "ryu", ] [[package]] name = "arrow-csv" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa9bf02705b5cf762b6f764c65f04ae9082c7cfc4e96e0c33548ee3f67012eb" +checksum = "92386159c8d4bce96f8bd396b0642a0d544d471bdc2ef34d631aec80db40a09c" dependencies = [ "arrow-array", "arrow-cast", @@ -280,21 +283,22 @@ dependencies = [ [[package]] name = "arrow-data" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5c64fff1d142f833d78897a772f2e5b55b36cb3e6320376f0961ab0db7bd6d0" +checksum = "727681b95de313b600eddc2a37e736dcb21980a40f640314dcf360e2f36bc89b" dependencies = [ "arrow-buffer", "arrow-schema", "half", - "num", + "num-integer", + "num-traits", ] [[package]] name = "arrow-ipc" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d3594dcddccc7f20fd069bc8e9828ce37220372680ff638c5e00dea427d88f5" +checksum = "da9ba92e3de170295c98a84e5af22e2b037f0c7b32449445e6c493b5fca27f27" dependencies = [ "arrow-array", "arrow-buffer", @@ -308,9 +312,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88cf36502b64a127dc659e3b305f1d993a544eab0d48cce704424e62074dc04b" +checksum = "b969b4a421ae83828591c6bf5450bd52e6d489584142845ad6a861f42fe35df8" dependencies = [ "arrow-array", "arrow-buffer", @@ -320,19 +324,21 @@ dependencies = [ "chrono", "half", "indexmap 2.12.0", + "itoa", "lexical-core", "memchr", - "num", - "serde", + "num-traits", + "ryu", + "serde_core", "serde_json", "simdutf8", ] [[package]] name = "arrow-ord" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c8f82583eb4f8d84d4ee55fd1cb306720cddead7596edce95b50ee418edf66f" +checksum = "141c05298b21d03e88062317a1f1a73f5ba7b6eb041b350015b1cd6aabc0519b" dependencies = [ "arrow-array", "arrow-buffer", @@ -343,9 +349,9 @@ dependencies = [ [[package]] name = "arrow-pyarrow" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d924b32e96f8bb74d94cd82bd97b313c432fcb0ea331689ef9e7c6b8be4b258" +checksum = "cfcfb2be2e9096236f449c11f425cddde18c4cc540f516d90f066f10a29ed515" dependencies = [ "arrow-array", "arrow-data", @@ -355,9 +361,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d07ba24522229d9085031df6b94605e0f4b26e099fb7cdeec37abd941a73753" +checksum = "c5f3c06a6abad6164508ed283c7a02151515cef3de4b4ff2cebbcaeb85533db2" dependencies = [ "arrow-array", "arrow-buffer", @@ -368,34 +374,34 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b3aa9e59c611ebc291c28582077ef25c97f1975383f1479b12f3b9ffee2ffabe" +checksum = "9cfa7a03d1eee2a4d061476e1840ad5c9867a544ca6c4c59256496af5d0a8be5" dependencies = [ "bitflags", - "serde", + "serde_core", "serde_json", ] [[package]] name = "arrow-select" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c41dbbd1e97bfcaee4fcb30e29105fb2c75e4d82ae4de70b792a5d3f66b2e7a" +checksum = "bafa595babaad59f2455f4957d0f26448fb472722c186739f4fac0823a1bdb47" dependencies = [ "ahash 0.8.12", "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", - "num", + "num-traits", ] [[package]] name = "arrow-string" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53f5183c150fbc619eede22b861ea7c0eebed8eaac0333eaa7f6da5205fd504d" +checksum = "32f46457dbbb99f2650ff3ac23e46a929e0ab81db809b02aa5511c258348bef2" dependencies = [ "arrow-array", "arrow-buffer", @@ -403,7 +409,7 @@ dependencies = [ "arrow-schema", "arrow-select", "memchr", - "num", + "num-traits", "regex", "regex-syntax", ] @@ -1029,11 +1035,9 @@ dependencies = [ [[package]] name = "datafusion" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af15bb3c6ffa33011ef579f6b0bcbe7c26584688bd6c994f548e44df67f011a" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", - "arrow-ipc", "arrow-schema", "async-trait", "bytes", @@ -1044,6 +1048,7 @@ dependencies = [ "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", + "datafusion-datasource-arrow", "datafusion-datasource-csv", "datafusion-datasource-json", "datafusion-datasource-parquet", @@ -1072,6 +1077,7 @@ dependencies = [ "parquet", "rand 0.9.2", "regex", + "rstest", "sqlparser", "tempfile", "tokio", @@ -1084,8 +1090,7 @@ dependencies = [ [[package]] name = "datafusion-catalog" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "187622262ad8f7d16d3be9202b4c1e0116f1c9aa387e5074245538b755261621" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", @@ -1098,7 +1103,6 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-plan", "datafusion-session", - "datafusion-sql", "futures", "itertools 0.14.0", "log", @@ -1110,8 +1114,7 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9657314f0a32efd0382b9a46fdeb2d233273ece64baa68a7c45f5a192daf0f83" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", @@ -1121,10 +1124,11 @@ dependencies = [ "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", + "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", - "datafusion-session", "futures", + "itertools 0.14.0", "log", "object_store", "tokio", @@ -1133,13 +1137,11 @@ dependencies = [ [[package]] name = "datafusion-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a83760d9a13122d025fbdb1d5d5aaf93dd9ada5e90ea229add92aa30898b2d1" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", "arrow-ipc", - "base64", "chrono", "half", "hashbrown 0.14.5", @@ -1158,8 +1160,7 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b6234a6c7173fe5db1c6c35c01a12b2aa0f803a3007feee53483218817f8b1e" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "futures", "log", @@ -1169,8 +1170,7 @@ dependencies = [ [[package]] name = "datafusion-datasource" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7256c9cb27a78709dd42d0c80f0178494637209cac6e29d5c93edd09b6721b86" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-compression", @@ -1193,9 +1193,7 @@ dependencies = [ "itertools 0.14.0", "log", "object_store", - "parquet", "rand 0.9.2", - "tempfile", "tokio", "tokio-util", "url", @@ -1203,22 +1201,42 @@ dependencies = [ "zstd", ] +[[package]] +name = "datafusion-datasource-arrow" +version = "50.3.0" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +dependencies = [ + "arrow", + "arrow-ipc", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools 0.14.0", + "object_store", + "tokio", +] + [[package]] name = "datafusion-datasource-csv" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64533a90f78e1684bfb113d200b540f18f268134622d7c96bbebc91354d04825" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", @@ -1231,48 +1249,41 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d7ebeb12c77df0aacad26f21b0d033aeede423a64b2b352f53048a75bf1d6e6" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "object_store", - "serde_json", "tokio", ] [[package]] name = "datafusion-datasource-parquet" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09e783c4c7d7faa1199af2df4761c68530634521b176a8d1331ddbc5a5c75133" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", - "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", - "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-pruning", "datafusion-session", @@ -1282,21 +1293,18 @@ dependencies = [ "object_store", "parking_lot", "parquet", - "rand 0.9.2", "tokio", ] [[package]] name = "datafusion-doc" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99ee6b1d9a80d13f9deb2291f45c07044b8e62fb540dbde2453a18be17a36429" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" [[package]] name = "datafusion-execution" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4cec0a57653bec7b933fb248d3ffa3fa3ab3bd33bd140dc917f714ac036f531" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", @@ -1315,8 +1323,7 @@ dependencies = [ [[package]] name = "datafusion-expr" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef76910bdca909722586389156d0aa4da4020e1631994d50fadd8ad4b1aa05fe" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", @@ -1328,6 +1335,7 @@ dependencies = [ "datafusion-functions-window-common", "datafusion-physical-expr-common", "indexmap 2.12.0", + "itertools 0.14.0", "paste", "recursive", "serde_json", @@ -1337,8 +1345,7 @@ dependencies = [ [[package]] name = "datafusion-expr-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d155ccbda29591ca71a1344dd6bed26c65a4438072b400df9db59447f590bb6" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "datafusion-common", @@ -1350,8 +1357,7 @@ dependencies = [ [[package]] name = "datafusion-ffi" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "25ddb7c4e645df080c27dad13a198d191da328dd1c98e198664a7a0f64b335cc" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "abi_stable", "arrow", @@ -1359,6 +1365,7 @@ dependencies = [ "async-ffi", "async-trait", "datafusion", + "datafusion-common", "datafusion-functions-aggregate-common", "datafusion-proto", "datafusion-proto-common", @@ -1372,8 +1379,7 @@ dependencies = [ [[package]] name = "datafusion-functions" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7de2782136bd6014670fd84fe3b0ca3b3e4106c96403c3ae05c0598577139977" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "arrow-buffer", @@ -1391,6 +1397,7 @@ dependencies = [ "itertools 0.14.0", "log", "md-5", + "num-traits", "rand 0.9.2", "regex", "sha2", @@ -1401,8 +1408,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07331fc13603a9da97b74fd8a273f4238222943dffdbbed1c4c6f862a30105bf" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", @@ -1422,8 +1428,7 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5951e572a8610b89968a09b5420515a121fbc305c0258651f318dc07c97ab17" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", @@ -1435,8 +1440,7 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdacca9302c3d8fc03f3e94f338767e786a88a33f5ebad6ffc0e7b50364b9ea3" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "arrow-ord", @@ -1444,6 +1448,7 @@ dependencies = [ "datafusion-doc", "datafusion-execution", "datafusion-expr", + "datafusion-expr-common", "datafusion-functions", "datafusion-functions-aggregate", "datafusion-functions-aggregate-common", @@ -1457,8 +1462,7 @@ dependencies = [ [[package]] name = "datafusion-functions-table" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c37ff8a99434fbbad604a7e0669717c58c7c4f14c472d45067c4b016621d981" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "async-trait", @@ -1473,8 +1477,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48e2aea7c79c926cffabb13dc27309d4eaeb130f4a21c8ba91cdd241c813652b" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "datafusion-common", @@ -1491,8 +1494,7 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fead257ab5fd2ffc3b40fda64da307e20de0040fe43d49197241d9de82a487f" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1501,10 +1503,9 @@ dependencies = [ [[package]] name = "datafusion-macros" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec6f637bce95efac05cdfb9b6c19579ed4aa5f6b94d951cfa5bb054b7bb4f730" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ - "datafusion-expr", + "datafusion-doc", "quote", "syn 2.0.108", ] @@ -1512,8 +1513,7 @@ dependencies = [ [[package]] name = "datafusion-optimizer" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6583ef666ae000a613a837e69e456681a9faa96347bf3877661e9e89e141d8a" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "chrono", @@ -1532,8 +1532,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8668103361a272cbbe3a61f72eca60c9b7c706e87cc3565bcf21e2b277b84f6" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", @@ -1546,7 +1545,6 @@ dependencies = [ "hashbrown 0.14.5", "indexmap 2.12.0", "itertools 0.14.0", - "log", "parking_lot", "paste", "petgraph", @@ -1555,8 +1553,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "815acced725d30601b397e39958e0e55630e0a10d66ef7769c14ae6597298bb0" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "datafusion-common", @@ -1570,8 +1567,7 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6652fe7b5bf87e85ed175f571745305565da2c0b599d98e697bcbedc7baa47c3" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", @@ -1584,8 +1580,7 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49b7d623eb6162a3332b564a0907ba00895c505d101b99af78345f1acf929b5c" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "datafusion-common", @@ -1597,15 +1592,13 @@ dependencies = [ "datafusion-physical-plan", "datafusion-pruning", "itertools 0.14.0", - "log", "recursive", ] [[package]] name = "datafusion-physical-plan" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2f7f778a1a838dec124efb96eae6144237d546945587557c9e6936b3414558c" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "ahash 0.8.12", "arrow", @@ -1635,14 +1628,24 @@ dependencies = [ [[package]] name = "datafusion-proto" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7df9f606892e6af45763d94d210634eec69b9bb6ced5353381682ff090028a3" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "chrono", - "datafusion", + "datafusion-catalog", + "datafusion-catalog-listing", "datafusion-common", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", "datafusion-expr", + "datafusion-functions-table", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "datafusion-proto-common", "object_store", "prost", @@ -1651,8 +1654,7 @@ dependencies = [ [[package]] name = "datafusion-proto-common" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4b14f288ca4ef77743d9672cafecf3adfffff0b9b04af9af79ecbeaaf736901" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "datafusion-common", @@ -1662,11 +1664,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd1e59e2ca14fe3c30f141600b10ad8815e2856caa59ebbd0e3e07cd3d127a65" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", - "arrow-schema", "datafusion-common", "datafusion-datasource", "datafusion-expr-common", @@ -1680,35 +1680,24 @@ dependencies = [ [[package]] name = "datafusion-session" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21ef8e2745583619bd7a49474e8f45fbe98ebb31a133f27802217125a7b3d58d" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ - "arrow", "async-trait", - "dashmap", "datafusion-common", - "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", "datafusion-physical-plan", - "datafusion-sql", - "futures", - "itertools 0.14.0", - "log", - "object_store", "parking_lot", - "tokio", ] [[package]] name = "datafusion-sql" version = "50.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89abd9868770386fede29e5a4b14f49c0bf48d652c3b9d7a8a0332329b87d50b" +source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" dependencies = [ "arrow", "bigdecimal", + "chrono", "datafusion-common", "datafusion-expr", "indexmap 2.12.0", @@ -1993,6 +1982,12 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" +[[package]] +name = "futures-timer" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" + [[package]] name = "futures-util" version = "0.3.31" @@ -2784,20 +2779,6 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9252111cf132ba0929b6f8e030cac2a24b507f3a4d6db6fb2896f27b354c714b" -[[package]] -name = "num" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" -dependencies = [ - "num-bigint", - "num-complex", - "num-integer", - "num-iter", - "num-rational", - "num-traits", -] - [[package]] name = "num-bigint" version = "0.4.6" @@ -2833,28 +2814,6 @@ dependencies = [ "num-traits", ] -[[package]] -name = "num-iter" -version = "0.1.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" -dependencies = [ - "autocfg", - "num-integer", - "num-traits", -] - -[[package]] -name = "num-rational" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" -dependencies = [ - "num-bigint", - "num-integer", - "num-traits", -] - [[package]] name = "num-traits" version = "0.2.19" @@ -3001,9 +2960,9 @@ dependencies = [ [[package]] name = "parquet" -version = "56.2.0" +version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0dbd48ad52d7dccf8ea1b90a3ddbfaea4f69878dd7683e51c507d4bc52b5b27" +checksum = "7a0f31027ef1af7549f7cec603a9a21dce706d3f8d7c2060a68f43c1773be95a" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -3022,11 +2981,11 @@ dependencies = [ "half", "hashbrown 0.16.0", "lz4_flex", - "num", "num-bigint", + "num-integer", + "num-traits", "object_store", "paste", - "ring", "seq-macro", "simdutf8", "snap", @@ -3156,9 +3115,9 @@ dependencies = [ [[package]] name = "prost" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +checksum = "7231bd9b3d3d33c86b58adbac74b5ec0ad9f496b19d22801d773636feaa95f3d" dependencies = [ "bytes", "prost-derive", @@ -3166,9 +3125,9 @@ dependencies = [ [[package]] name = "prost-derive" -version = "0.13.5" +version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +checksum = "9120690fafc389a67ba3803df527d0ec9cbbc9cc45e4cc20b332996dfb672425" dependencies = [ "anyhow", "itertools 0.14.0", @@ -3221,9 +3180,9 @@ dependencies = [ [[package]] name = "pyo3" -version = "0.25.1" +version = "0.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8970a78afe0628a3e3430376fc5fd76b6b45c4d43360ffd6cdd40bdde72b682a" +checksum = "7ba0117f4212101ee6544044dae45abe1083d30ce7b29c4b5cbdfa2354e07383" dependencies = [ "indoc", "libc", @@ -3238,19 +3197,18 @@ dependencies = [ [[package]] name = "pyo3-build-config" -version = "0.25.1" +version = "0.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "458eb0c55e7ece017adeba38f2248ff3ac615e53660d7c71a238d7d2a01c7598" +checksum = "4fc6ddaf24947d12a9aa31ac65431fb1b851b8f4365426e182901eabfb87df5f" dependencies = [ - "once_cell", "target-lexicon", ] [[package]] name = "pyo3-ffi" -version = "0.25.1" +version = "0.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7114fe5457c61b276ab77c5055f206295b812608083644a5c5b2640c3102565c" +checksum = "025474d3928738efb38ac36d4744a74a400c901c7596199e20e45d98eb194105" dependencies = [ "libc", "pyo3-build-config", @@ -3258,9 +3216,9 @@ dependencies = [ [[package]] name = "pyo3-macros" -version = "0.25.1" +version = "0.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8725c0a622b374d6cb051d11a0983786448f7785336139c3c94f5aa6bef7e50" +checksum = "2e64eb489f22fe1c95911b77c44cc41e7c19f3082fc81cce90f657cdc42ffded" dependencies = [ "proc-macro2", "pyo3-macros-backend", @@ -3270,9 +3228,9 @@ dependencies = [ [[package]] name = "pyo3-macros-backend" -version = "0.25.1" +version = "0.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4109984c22491085343c05b0dbc54ddc405c3cf7b4374fc533f5c3313a572ccc" +checksum = "100246c0ecf400b475341b8455a9213344569af29a3c841d29270e53102e0fcf" dependencies = [ "heck", "proc-macro2", @@ -3526,6 +3484,12 @@ version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58" +[[package]] +name = "relative-path" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" + [[package]] name = "rend" version = "0.4.2" @@ -3667,6 +3631,35 @@ dependencies = [ "byteorder", ] +[[package]] +name = "rstest" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5a3193c063baaa2a95a33f03035c8a72b83d97a54916055ba22d35ed3839d49" +dependencies = [ + "futures-timer", + "futures-util", + "rstest_macros", +] + +[[package]] +name = "rstest_macros" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0" +dependencies = [ + "cfg-if", + "glob", + "proc-macro-crate", + "proc-macro2", + "quote", + "regex", + "relative-path", + "rustc_version", + "syn 2.0.108", + "unicode-ident", +] + [[package]] name = "rust-ini" version = "0.21.3" @@ -4008,9 +4001,9 @@ dependencies = [ [[package]] name = "sqlparser" -version = "0.58.0" +version = "0.59.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec4b661c54b1e4b603b37873a18c59920e4c51ea8ea2cf527d925424dbd4437c" +checksum = "4591acadbcf52f0af60eafbb2c003232b2b4cd8de5f0e9437cb8b1b59046cc0f" dependencies = [ "log", "recursive", diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index 953d4a98fc..d8b8444195 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -19,7 +19,7 @@ edition = "2024" homepage = "https://rust.iceberg.apache.org" name = "pyiceberg_core_rust" -rust-version = "1.87" +rust-version = "1.88" version = "0.7.0" # This crate is used to build python bindings, we don't want to publish it publish = false @@ -31,11 +31,11 @@ license = "Apache-2.0" crate-type = ["cdylib"] [dependencies] -arrow = { version = "56", features = ["pyarrow", "chrono-tz"] } +arrow = { version = "57.0", features = ["pyarrow", "chrono-tz"] } iceberg = { path = "../../crates/iceberg" } -pyo3 = { version = "0.25", features = ["extension-module", "abi3-py310"] } +pyo3 = { version = "0.26", features = ["extension-module", "abi3-py310"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } -datafusion-ffi = { version = "50" } +datafusion-ffi = { version = "51.0" } tokio = { version = "1.46.1", default-features = false } [profile.release] diff --git a/bindings/python/src/transform.rs b/bindings/python/src/transform.rs index 24e9f061dd..c159d573fc 100644 --- a/bindings/python/src/transform.rs +++ b/bindings/python/src/transform.rs @@ -24,46 +24,46 @@ use pyo3::prelude::*; use crate::error::to_py_err; #[pyfunction] -pub fn identity(py: Python, array: PyObject) -> PyResult { +pub fn identity(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Identity) } #[pyfunction] -pub fn void(py: Python, array: PyObject) -> PyResult { +pub fn void(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Void) } #[pyfunction] -pub fn year(py: Python, array: PyObject) -> PyResult { +pub fn year(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Year) } #[pyfunction] -pub fn month(py: Python, array: PyObject) -> PyResult { +pub fn month(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Month) } #[pyfunction] -pub fn day(py: Python, array: PyObject) -> PyResult { +pub fn day(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Day) } #[pyfunction] -pub fn hour(py: Python, array: PyObject) -> PyResult { +pub fn hour(py: Python, array: Py) -> PyResult> { apply(py, array, Transform::Hour) } #[pyfunction] -pub fn bucket(py: Python, array: PyObject, num_buckets: u32) -> PyResult { +pub fn bucket(py: Python, array: Py, num_buckets: u32) -> PyResult> { apply(py, array, Transform::Bucket(num_buckets)) } #[pyfunction] -pub fn truncate(py: Python, array: PyObject, width: u32) -> PyResult { +pub fn truncate(py: Python, array: Py, width: u32) -> PyResult> { apply(py, array, Transform::Truncate(width)) } -fn apply(py: Python, array: PyObject, transform: Transform) -> PyResult { +fn apply(py: Python, array: Py, transform: Transform) -> PyResult> { // import let array = ArrayData::from_pyarrow_bound(array.bind(py))?; let array = make_array(array); @@ -71,7 +71,7 @@ fn apply(py: Python, array: PyObject, transform: Transform) -> PyResult, m: &Bound<'_, PyModule>) -> PyResult<()> { diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index 84c90d466f..6f1332a444 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -84,7 +84,6 @@ serde_json = { workspace = true } serde_repr = { workspace = true } serde_with = { workspace = true } strum = { workspace = true, features = ["derive"] } -thrift = { workspace = true } tokio = { workspace = true, optional = false, features = ["sync"] } typed-builder = { workspace = true } url = { workspace = true } diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 380d48530c..6209c1e261 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -1038,7 +1038,6 @@ fn apply_name_mapping_to_arrow_schema( metadata.insert(PARQUET_FIELD_ID_META_KEY.to_string(), field_id.to_string()); } // If field_id is None, leave the field without an ID (will be filtered by projection) - // If field not found in mapping, leave it without an ID (will be filtered by projection) Field::new(field.name(), field.data_type().clone(), field.is_nullable()) .with_metadata(metadata) @@ -1915,7 +1914,7 @@ message schema { assert_eq!(err.kind(), ErrorKind::DataInvalid); assert_eq!( err.to_string(), - "DataInvalid => Unsupported Arrow data type: Duration(Microsecond)".to_string() + "DataInvalid => Unsupported Arrow data type: Duration(µs)".to_string() ); // Omitting field c2, we still get an error due to c3 being selected @@ -2141,7 +2140,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props.clone())).unwrap(); @@ -2322,7 +2321,7 @@ message schema { let tmp_dir = TempDir::new().unwrap(); let table_location = tmp_dir.path().to_str().unwrap().to_string(); - let file_path = format!("{}/multi_row_group.parquet", &table_location); + let file_path = format!("{table_location}/multi_row_group.parquet"); // Force each batch into its own row group for testing byte range filtering. let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new(Int32Array::from( @@ -2526,7 +2525,7 @@ message schema { let props = WriterProperties::builder() .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/old_file.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/old_file.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); writer.close().unwrap(); @@ -2632,7 +2631,7 @@ message schema { // Step 1: Create data file with 200 rows in 2 row groups // Row group 0: rows 0-99 (ids 1-100) // Row group 1: rows 100-199 (ids 101-200) - let data_file_path = format!("{}/data.parquet", &table_location); + let data_file_path = format!("{table_location}/data.parquet"); let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( Int32Array::from_iter_values(1..=100), @@ -2666,7 +2665,7 @@ message schema { ); // Step 2: Create position delete file that deletes row 199 (id=200, last row in row group 1) - let delete_file_path = format!("{}/deletes.parquet", &table_location); + let delete_file_path = format!("{table_location}/deletes.parquet"); let delete_schema = Arc::new(ArrowSchema::new(vec![ Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( @@ -2825,7 +2824,7 @@ message schema { // Step 1: Create data file with 200 rows in 2 row groups // Row group 0: rows 0-99 (ids 1-100) // Row group 1: rows 100-199 (ids 101-200) - let data_file_path = format!("{}/data.parquet", &table_location); + let data_file_path = format!("{table_location}/data.parquet"); let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( Int32Array::from_iter_values(1..=100), @@ -2859,7 +2858,7 @@ message schema { ); // Step 2: Create position delete file that deletes row 199 (id=200, last row in row group 1) - let delete_file_path = format!("{}/deletes.parquet", &table_location); + let delete_file_path = format!("{table_location}/deletes.parquet"); let delete_schema = Arc::new(ArrowSchema::new(vec![ Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( @@ -3046,7 +3045,7 @@ message schema { // Step 1: Create data file with 200 rows in 2 row groups // Row group 0: rows 0-99 (ids 1-100) // Row group 1: rows 100-199 (ids 101-200) - let data_file_path = format!("{}/data.parquet", &table_location); + let data_file_path = format!("{table_location}/data.parquet"); let batch1 = RecordBatch::try_new(arrow_schema.clone(), vec![Arc::new( Int32Array::from_iter_values(1..=100), @@ -3080,7 +3079,7 @@ message schema { ); // Step 2: Create position delete file that deletes row 0 (id=1, first row in row group 0) - let delete_file_path = format!("{}/deletes.parquet", &table_location); + let delete_file_path = format!("{table_location}/deletes.parquet"); let delete_schema = Arc::new(ArrowSchema::new(vec![ Field::new("file_path", DataType::Utf8, false).with_metadata(HashMap::from([( @@ -3226,7 +3225,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); @@ -3323,7 +3322,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); @@ -3409,7 +3408,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); @@ -3497,7 +3496,7 @@ message schema { .set_max_row_group_size(2) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); // Write 6 rows in 3 batches (will create 3 row groups) @@ -3638,7 +3637,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); @@ -3735,7 +3734,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); writer.close().unwrap(); @@ -3837,7 +3836,7 @@ message schema { .set_compression(Compression::SNAPPY) .build(); - let file = File::create(format!("{}/1.parquet", &table_location)).unwrap(); + let file = File::create(format!("{table_location}/1.parquet")).unwrap(); let mut writer = ArrowWriter::try_new(file, to_write.schema(), Some(props)).unwrap(); writer.write(&to_write).expect("Writing batch"); writer.close().unwrap(); diff --git a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs index ae3a906275..66e2898532 100644 --- a/crates/iceberg/src/expr/visitors/page_index_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/page_index_evaluator.rs @@ -23,7 +23,7 @@ use fnv::FnvHashSet; use ordered_float::OrderedFloat; use parquet::arrow::arrow_reader::{RowSelection, RowSelector}; use parquet::file::metadata::RowGroupMetaData; -use parquet::file::page_index::index::Index; +use parquet::file::page_index::column_index::ColumnIndexMetaData; use parquet::file::page_index::offset_index::OffsetIndexMetaData; use crate::expr::visitors::bound_predicate_visitor::{BoundPredicateVisitor, visit}; @@ -59,7 +59,7 @@ impl PageNullCount { } pub(crate) struct PageIndexEvaluator<'a> { - column_index: &'a [Index], + column_index: &'a [ColumnIndexMetaData], offset_index: &'a OffsetIndex, row_group_metadata: &'a RowGroupMetaData, iceberg_field_id_to_parquet_column_index: &'a HashMap, @@ -69,7 +69,7 @@ pub(crate) struct PageIndexEvaluator<'a> { impl<'a> PageIndexEvaluator<'a> { pub(crate) fn new( - column_index: &'a [Index], + column_index: &'a [ColumnIndexMetaData], offset_index: &'a OffsetIndex, row_group_metadata: &'a RowGroupMetaData, field_id_map: &'a HashMap, @@ -92,7 +92,7 @@ impl<'a> PageIndexEvaluator<'a> { /// matching the filter predicate. pub(crate) fn eval( filter: &'a BoundPredicate, - column_index: &'a [Index], + column_index: &'a [ColumnIndexMetaData], offset_index: &'a OffsetIndex, row_group_metadata: &'a RowGroupMetaData, field_id_map: &'a HashMap, @@ -240,137 +240,135 @@ impl<'a> PageIndexEvaluator<'a> { fn apply_predicate_to_column_index( predicate: F, field_type: &PrimitiveType, - column_index: &Index, + column_index: &ColumnIndexMetaData, row_counts: &[usize], ) -> Result>> where F: Fn(Option, Option, PageNullCount) -> Result, { let result: Result> = match column_index { - Index::NONE => { + ColumnIndexMetaData::NONE => { return Ok(None); } - Index::BOOLEAN(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::BOOLEAN(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min.map(|val| { + min.map(|&val| { Datum::new(field_type.clone(), PrimitiveLiteral::Boolean(val)) }), - item.max.map(|val| { + max.map(|&val| { Datum::new(field_type.clone(), PrimitiveLiteral::Boolean(val)) }), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::INT32(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::INT32(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min - .map(|val| Datum::new(field_type.clone(), PrimitiveLiteral::Int(val))), - item.max - .map(|val| Datum::new(field_type.clone(), PrimitiveLiteral::Int(val))), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + min.map(|&val| Datum::new(field_type.clone(), PrimitiveLiteral::Int(val))), + max.map(|&val| Datum::new(field_type.clone(), PrimitiveLiteral::Int(val))), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::INT64(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::INT64(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min - .map(|val| Datum::new(field_type.clone(), PrimitiveLiteral::Long(val))), - item.max - .map(|val| Datum::new(field_type.clone(), PrimitiveLiteral::Long(val))), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + min.map(|&val| Datum::new(field_type.clone(), PrimitiveLiteral::Long(val))), + max.map(|&val| Datum::new(field_type.clone(), PrimitiveLiteral::Long(val))), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::FLOAT(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::FLOAT(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min.map(|val| { + min.map(|&val| { Datum::new( field_type.clone(), PrimitiveLiteral::Float(OrderedFloat::from(val)), ) }), - item.max.map(|val| { + max.map(|&val| { Datum::new( field_type.clone(), PrimitiveLiteral::Float(OrderedFloat::from(val)), ) }), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::DOUBLE(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::DOUBLE(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min.map(|val| { + min.map(|&val| { Datum::new( field_type.clone(), PrimitiveLiteral::Double(OrderedFloat::from(val)), ) }), - item.max.map(|val| { + max.map(|&val| { Datum::new( field_type.clone(), PrimitiveLiteral::Double(OrderedFloat::from(val)), ) }), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::BYTE_ARRAY(idx) => idx - .indexes - .iter() + ColumnIndexMetaData::BYTE_ARRAY(idx) => idx + .min_values_iter() + .zip(idx.max_values_iter()) + .enumerate() .zip(row_counts.iter()) - .map(|(item, &row_count)| { + .map(|((i, (min, max)), &row_count)| { predicate( - item.min.clone().map(|val| { + min.map(|val| { Datum::new( field_type.clone(), - PrimitiveLiteral::String( - String::from_utf8(val.data().to_vec()).unwrap(), - ), + PrimitiveLiteral::String(String::from_utf8(val.to_vec()).unwrap()), ) }), - item.max.clone().map(|val| { + max.map(|val| { Datum::new( field_type.clone(), - PrimitiveLiteral::String( - String::from_utf8(val.data().to_vec()).unwrap(), - ), + PrimitiveLiteral::String(String::from_utf8(val.to_vec()).unwrap()), ) }), - PageNullCount::from_row_and_null_counts(row_count, item.null_count), + PageNullCount::from_row_and_null_counts(row_count, idx.null_count(i)), ) }) .collect(), - Index::FIXED_LEN_BYTE_ARRAY(_) => { + ColumnIndexMetaData::FIXED_LEN_BYTE_ARRAY(_) => { return Err(Error::new( ErrorKind::FeatureUnsupported, "unsupported 'FIXED_LEN_BYTE_ARRAY' index type in column_index", )); } - Index::INT96(_) => { + ColumnIndexMetaData::INT96(_) => { return Err(Error::new( ErrorKind::FeatureUnsupported, "unsupported 'INT96' index type in column_index", @@ -787,28 +785,164 @@ mod tests { use std::collections::HashMap; use std::sync::Arc; - use parquet::arrow::arrow_reader::RowSelector; - use parquet::basic::{LogicalType as ParquetLogicalType, Type as ParquetPhysicalType}; - use parquet::data_type::ByteArray; - use parquet::file::metadata::{ColumnChunkMetaData, RowGroupMetaData}; - use parquet::file::page_index::index::{Index, NativeIndex, PageIndex}; - use parquet::file::page_index::offset_index::OffsetIndexMetaData; - use parquet::file::statistics::Statistics; - use parquet::format::{BoundaryOrder, PageLocation}; - use parquet::schema::types::{ - ColumnDescriptor, ColumnPath, SchemaDescriptor, Type as parquetSchemaType, + use arrow_array::{ArrayRef, Float32Array, RecordBatch, StringArray}; + use arrow_schema::{DataType, Field, Schema as ArrowSchema}; + use parquet::arrow::ArrowWriter; + use parquet::arrow::arrow_reader::{ + ArrowReaderOptions, ParquetRecordBatchReaderBuilder, RowSelector, }; + use parquet::file::metadata::ParquetMetaData; + use parquet::file::properties::WriterProperties; use rand::{Rng, thread_rng}; + use tempfile::NamedTempFile; use super::PageIndexEvaluator; use crate::expr::{Bind, Reference}; use crate::spec::{Datum, NestedField, PrimitiveType, Schema, Type}; use crate::{ErrorKind, Result}; + /// Helper function to create a test parquet file with page indexes + /// and return the metadata needed for testing + fn create_test_parquet_file() -> Result<(Arc, NamedTempFile)> { + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("col_float", DataType::Float32, true), + Field::new("col_string", DataType::Utf8, true), + ])); + + let temp_file = NamedTempFile::new().unwrap(); + let file = temp_file.reopen().unwrap(); + + let props = WriterProperties::builder() + .set_data_page_row_count_limit(1024) + .set_write_batch_size(512) + .build(); + + let mut writer = ArrowWriter::try_new(file, arrow_schema.clone(), Some(props)).unwrap(); + + let mut batches = vec![]; + + // Batch 1: 1024 rows - strings with AARDVARK, BEAR, BISON + let float_vals: Vec> = vec![None; 1024]; + let mut string_vals = vec![]; + string_vals.push(Some("AARDVARK".to_string())); + for _ in 1..1023 { + string_vals.push(Some("BEAR".to_string())); + } + string_vals.push(Some("BISON".to_string())); + + batches.push( + RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Float32Array::from(float_vals)), + Arc::new(StringArray::from(string_vals)), + ]) + .unwrap(), + ); + + // Batch 2: 1024 rows - all DEER + let float_vals: Vec> = vec![None; 1024]; + let string_vals = vec![Some("DEER".to_string()); 1024]; + + batches.push( + RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Float32Array::from(float_vals)), + Arc::new(StringArray::from(string_vals)), + ]) + .unwrap(), + ); + + // Batch 3: 1024 rows - float 0-10 + let mut float_vals = vec![]; + for i in 0..1024 { + float_vals.push(Some(i as f32 * 10.0 / 1024.0)); + } + let mut string_vals = vec![]; + string_vals.push(Some("GIRAFFE".to_string())); + string_vals.push(None); + for _ in 2..1024 { + string_vals.push(Some("HIPPO".to_string())); + } + + batches.push( + RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Float32Array::from(float_vals)), + Arc::new(StringArray::from(string_vals)), + ]) + .unwrap(), + ); + + // Batch 4: 1024 rows - float 10-20 + let mut float_vals = vec![None]; + for i in 1..1024 { + float_vals.push(Some(10.0 + i as f32 * 10.0 / 1024.0)); + } + let string_vals = vec![Some("HIPPO".to_string()); 1024]; + + batches.push( + RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Float32Array::from(float_vals)), + Arc::new(StringArray::from(string_vals)), + ]) + .unwrap(), + ); + + // Write rows one at a time to give the writer a chance to split into pages + for batch in &batches { + for i in 0..batch.num_rows() { + writer.write(&batch.slice(i, 1)).unwrap(); + } + } + + writer.close().unwrap(); + + let file = temp_file.reopen().unwrap(); + let options = ArrowReaderOptions::new().with_page_index(true); + let reader = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap(); + let metadata = reader.metadata().clone(); + + Ok((metadata, temp_file)) + } + + /// Get the test metadata components for testing + fn get_test_metadata( + metadata: &ParquetMetaData, + ) -> ( + Vec, + Vec, + &parquet::file::metadata::RowGroupMetaData, + ) { + let row_group_metadata = metadata.row_group(0); + let column_index = metadata.column_index().unwrap()[0].to_vec(); + let offset_index = metadata.offset_index().unwrap()[0].to_vec(); + (column_index, offset_index, row_group_metadata) + } + #[test] fn eval_matches_no_rows_for_empty_row_group() -> Result<()> { - let row_group_metadata = create_row_group_metadata(0, 0, None, 0, None)?; - let (column_index, offset_index) = create_page_index()?; + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("col_float", DataType::Float32, true), + Field::new("col_string", DataType::Utf8, true), + ])); + + let empty_float: ArrayRef = Arc::new(Float32Array::from(Vec::>::new())); + let empty_string: ArrayRef = Arc::new(StringArray::from(Vec::>::new())); + let empty_batch = + RecordBatch::try_new(arrow_schema.clone(), vec![empty_float, empty_string]).unwrap(); + + let temp_file = NamedTempFile::new().unwrap(); + let file = temp_file.reopen().unwrap(); + + let mut writer = ArrowWriter::try_new(file, arrow_schema, None).unwrap(); + writer.write(&empty_batch).unwrap(); + writer.close().unwrap(); + + let file = temp_file.reopen().unwrap(); + let options = ArrowReaderOptions::new().with_page_index(true); + let reader = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap(); + let metadata = reader.metadata(); + + if metadata.num_row_groups() == 0 || metadata.row_group(0).num_rows() == 0 { + return Ok(()); + } let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; @@ -816,27 +950,28 @@ mod tests { .greater_than(Datum::float(1.0)) .bind(iceberg_schema_ref.clone(), false)?; + let row_group_metadata = metadata.row_group(0); + let column_index = metadata.column_index().unwrap()[0].to_vec(); + let offset_index = metadata.offset_index().unwrap()[0].to_vec(); + let result = PageIndexEvaluator::eval( &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; - let expected = vec![]; - - assert_eq!(result, expected); + assert_eq!(result.len(), 0); Ok(()) } #[test] fn eval_is_null_select_only_pages_with_nulls() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -847,15 +982,15 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; let expected = vec![ - RowSelector::select(1024), - RowSelector::skip(1024), RowSelector::select(2048), + RowSelector::skip(1024), + RowSelector::select(1024), ]; assert_eq!(result, expected); @@ -865,9 +1000,8 @@ mod tests { #[test] fn eval_is_not_null_dont_select_pages_with_all_nulls() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -878,12 +1012,12 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; - let expected = vec![RowSelector::skip(1024), RowSelector::select(3072)]; + let expected = vec![RowSelector::skip(2048), RowSelector::select(2048)]; assert_eq!(result, expected); @@ -892,9 +1026,8 @@ mod tests { #[test] fn eval_is_nan_select_all() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -905,7 +1038,7 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; @@ -919,9 +1052,8 @@ mod tests { #[test] fn eval_not_nan_select_all() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -932,7 +1064,7 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; @@ -946,9 +1078,8 @@ mod tests { #[test] fn eval_inequality_nan_datum_all_rows_except_all_null_pages() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -959,12 +1090,12 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; - let expected = vec![RowSelector::skip(1024), RowSelector::select(3072)]; + let expected = vec![RowSelector::skip(2048), RowSelector::select(2048)]; assert_eq!(result, expected); @@ -973,9 +1104,8 @@ mod tests { #[test] fn eval_inequality_pages_containing_value_except_all_null_pages() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -986,16 +1116,15 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; let expected = vec![ - RowSelector::skip(1024), + RowSelector::skip(2048), RowSelector::select(1024), RowSelector::skip(1024), - RowSelector::select(1024), ]; assert_eq!(result, expected); @@ -1005,9 +1134,8 @@ mod tests { #[test] fn eval_eq_pages_containing_value_except_all_null_pages() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -1018,16 +1146,18 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; + // Pages 0-1: all null (skip) + // Page 2: 0-10 (select, might contain 5.0) + // Page 3: 10-20 (skip, min > 5.0) let expected = vec![ - RowSelector::skip(1024), + RowSelector::skip(2048), RowSelector::select(1024), RowSelector::skip(1024), - RowSelector::select(1024), ]; assert_eq!(result, expected); @@ -1037,9 +1167,8 @@ mod tests { #[test] fn eval_not_eq_all_rows() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -1050,7 +1179,7 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; @@ -1064,9 +1193,8 @@ mod tests { #[test] fn eval_starts_with_error_float_col() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -1077,7 +1205,7 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), ); @@ -1089,11 +1217,13 @@ mod tests { #[test] fn eval_starts_with_pages_containing_value_except_all_null_pages() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; + // Test starts_with on string column where only some pages match + // Our file has 4 pages: ["AARDVARK".."BISON"], ["DEER"], ["GIRAFFE".."HIPPO"], ["HIPPO"] + // Testing starts_with("B") should select only page 0 let filter = Reference::new("col_string") .starts_with(Datum::string("B")) .bind(iceberg_schema_ref.clone(), false)?; @@ -1102,16 +1232,13 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; - let expected = vec![ - RowSelector::select(512), - RowSelector::skip(3536), - RowSelector::select(48), - ]; + // Page 0 has "BEAR" and "BISON" (starts with B), rest don't + let expected = vec![RowSelector::select(1024), RowSelector::skip(3072)]; assert_eq!(result, expected); @@ -1121,11 +1248,13 @@ mod tests { #[test] fn eval_not_starts_with_pages_containing_value_except_pages_with_min_and_max_equal_to_prefix_and_all_null_pages() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; + // Test not_starts_with where one page has ALL values starting with prefix + // Our file has page 1 with all "DEER" (min="DEER", max="DEER") + // Testing not_starts_with("DE") should skip page 1 where all values start with "DE" let filter = Reference::new("col_string") .not_starts_with(Datum::string("DE")) .bind(iceberg_schema_ref.clone(), false)?; @@ -1134,15 +1263,18 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; + // Page 0: mixed values (select) + // Page 1: all "DEER" starting with "DE" (skip) + // Pages 2-3: other values not all starting with "DE" (select) let expected = vec![ - RowSelector::select(512), - RowSelector::skip(512), - RowSelector::select(3072), + RowSelector::select(1024), + RowSelector::skip(1024), + RowSelector::select(2048), ]; assert_eq!(result, expected); @@ -1153,10 +1285,8 @@ mod tests { #[test] fn eval_in_length_of_set_above_limit_all_rows() -> Result<()> { let mut rng = thread_rng(); - - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; let filter = Reference::new("col_float") @@ -1167,7 +1297,7 @@ mod tests { &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; @@ -1181,30 +1311,32 @@ mod tests { #[test] fn eval_in_valid_set_size_some_rows() -> Result<()> { - let row_group_metadata = create_row_group_metadata(4096, 1000, None, 1000, None)?; - let (column_index, offset_index) = create_page_index()?; - + let (metadata, _temp_file) = create_test_parquet_file()?; + let (column_index, offset_index, row_group_metadata) = get_test_metadata(&metadata); let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?; + // Test is_in with multiple values using min/max bounds + // Our file has 4 pages: ["AARDVARK".."BISON"], ["DEER"], ["GIRAFFE".."HIPPO"], ["HIPPO"] + // Testing is_in(["AARDVARK", "GIRAFFE"]) - both are in different pages let filter = Reference::new("col_string") - .is_in([Datum::string("AARDVARK"), Datum::string("ICEBERG")]) + .is_in([Datum::string("AARDVARK"), Datum::string("GIRAFFE")]) .bind(iceberg_schema_ref.clone(), false)?; let result = PageIndexEvaluator::eval( &filter, &column_index, &offset_index, - &row_group_metadata, + row_group_metadata, &field_id_map, iceberg_schema_ref.as_ref(), )?; + // Page 0 contains "AARDVARK", page 1 doesn't contain either, page 2 contains "GIRAFFE", page 3 doesn't let expected = vec![ - RowSelector::select(512), - RowSelector::skip(512), - RowSelector::select(2976), - RowSelector::skip(48), - RowSelector::select(48), + RowSelector::select(1024), + RowSelector::skip(1024), + RowSelector::select(1024), + RowSelector::skip(1024), ]; assert_eq!(result, expected); @@ -1235,183 +1367,4 @@ mod tests { Ok((iceberg_schema_ref, field_id_map)) } - - fn build_parquet_schema_descriptor() -> Result> { - let field_1 = Arc::new( - parquetSchemaType::primitive_type_builder("col_float", ParquetPhysicalType::FLOAT) - .with_id(Some(1)) - .build()?, - ); - - let field_2 = Arc::new( - parquetSchemaType::primitive_type_builder( - "col_string", - ParquetPhysicalType::BYTE_ARRAY, - ) - .with_id(Some(2)) - .with_logical_type(Some(ParquetLogicalType::String)) - .build()?, - ); - - let group_type = Arc::new( - parquetSchemaType::group_type_builder("all") - .with_id(Some(1000)) - .with_fields(vec![field_1, field_2]) - .build()?, - ); - - let schema_descriptor = SchemaDescriptor::new(group_type); - let schema_descriptor_arc = Arc::new(schema_descriptor); - Ok(schema_descriptor_arc) - } - - fn create_row_group_metadata( - num_rows: i64, - col_1_num_vals: i64, - col_1_stats: Option, - col_2_num_vals: i64, - col_2_stats: Option, - ) -> Result { - let schema_descriptor_arc = build_parquet_schema_descriptor()?; - - let column_1_desc_ptr = Arc::new(ColumnDescriptor::new( - schema_descriptor_arc.column(0).self_type_ptr(), - 1, - 1, - ColumnPath::new(vec!["col_float".to_string()]), - )); - - let column_2_desc_ptr = Arc::new(ColumnDescriptor::new( - schema_descriptor_arc.column(1).self_type_ptr(), - 1, - 1, - ColumnPath::new(vec!["col_string".to_string()]), - )); - - let mut col_1_meta = - ColumnChunkMetaData::builder(column_1_desc_ptr).set_num_values(col_1_num_vals); - if let Some(stats1) = col_1_stats { - col_1_meta = col_1_meta.set_statistics(stats1) - } - - let mut col_2_meta = - ColumnChunkMetaData::builder(column_2_desc_ptr).set_num_values(col_2_num_vals); - if let Some(stats2) = col_2_stats { - col_2_meta = col_2_meta.set_statistics(stats2) - } - - let row_group_metadata = RowGroupMetaData::builder(schema_descriptor_arc) - .set_num_rows(num_rows) - .set_column_metadata(vec![ - col_1_meta.build()?, - // .set_statistics(Statistics::float(None, None, None, 1, false)) - col_2_meta.build()?, - ]) - .build(); - - Ok(row_group_metadata?) - } - - fn create_page_index() -> Result<(Vec, Vec)> { - let idx_float = Index::FLOAT(NativeIndex:: { - indexes: vec![ - PageIndex { - min: None, - max: None, - null_count: Some(1024), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: Some(0.0), - max: Some(10.0), - null_count: Some(0), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: Some(10.0), - max: Some(20.0), - null_count: Some(1), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: None, - max: None, - null_count: None, - repetition_level_histogram: None, - definition_level_histogram: None, - }, - ], - boundary_order: BoundaryOrder(0), // UNORDERED - }); - - let idx_string = Index::BYTE_ARRAY(NativeIndex:: { - indexes: vec![ - PageIndex { - min: Some("AA".into()), - max: Some("DD".into()), - null_count: Some(0), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: Some("DE".into()), - max: Some("DE".into()), - null_count: Some(0), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: Some("DF".into()), - max: Some("UJ".into()), - null_count: Some(1), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: None, - max: None, - null_count: Some(48), - repetition_level_histogram: None, - definition_level_histogram: None, - }, - PageIndex { - min: None, - max: None, - null_count: None, - repetition_level_histogram: None, - definition_level_histogram: None, - }, - ], - boundary_order: BoundaryOrder(0), // UNORDERED - }); - - let page_locs_float = vec![ - PageLocation::new(0, 1024, 0), - PageLocation::new(1024, 1024, 1024), - PageLocation::new(2048, 1024, 2048), - PageLocation::new(3072, 1024, 3072), - ]; - - let page_locs_string = vec![ - PageLocation::new(0, 512, 0), - PageLocation::new(512, 512, 512), - PageLocation::new(1024, 2976, 1024), - PageLocation::new(4000, 48, 4000), - PageLocation::new(4048, 48, 4048), - ]; - - Ok((vec![idx_float, idx_string], vec![ - OffsetIndexMetaData { - page_locations: page_locs_float, - unencoded_byte_array_data_bytes: None, - }, - OffsetIndexMetaData { - page_locations: page_locs_string, - unencoded_byte_array_data_bytes: None, - }, - ])) - } } diff --git a/crates/iceberg/src/inspect/manifests.rs b/crates/iceberg/src/inspect/manifests.rs index 60854b8bae..4c30ca2ec5 100644 --- a/crates/iceberg/src/inspect/manifests.rs +++ b/crates/iceberg/src/inspect/manifests.rs @@ -296,18 +296,18 @@ mod tests { check_record_batches( record_batch.try_collect::>().await.unwrap(), expect![[r#" - Field { name: "content", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "14"} }, - Field { name: "path", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "length", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "partition_spec_id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }, - Field { name: "added_snapshot_id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "4"} }, - Field { name: "added_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "5"} }, - Field { name: "existing_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }, - Field { name: "deleted_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, - Field { name: "added_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "15"} }, - Field { name: "existing_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "16"} }, - Field { name: "deleted_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "17"} }, - Field { name: "partition_summaries", data_type: List(Field { name: "item", data_type: Struct([Field { name: "contains_null", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "10"} }, Field { name: "contains_nan", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "11"} }, Field { name: "lower_bound", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "12"} }, Field { name: "upper_bound", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "13"} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "9"} }), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }"#]], + Field { "content": Int32, metadata: {"PARQUET:field_id": "14"} }, + Field { "path": Utf8, metadata: {"PARQUET:field_id": "1"} }, + Field { "length": Int64, metadata: {"PARQUET:field_id": "2"} }, + Field { "partition_spec_id": Int32, metadata: {"PARQUET:field_id": "3"} }, + Field { "added_snapshot_id": Int64, metadata: {"PARQUET:field_id": "4"} }, + Field { "added_data_files_count": Int32, metadata: {"PARQUET:field_id": "5"} }, + Field { "existing_data_files_count": Int32, metadata: {"PARQUET:field_id": "6"} }, + Field { "deleted_data_files_count": Int32, metadata: {"PARQUET:field_id": "7"} }, + Field { "added_delete_files_count": Int32, metadata: {"PARQUET:field_id": "15"} }, + Field { "existing_delete_files_count": Int32, metadata: {"PARQUET:field_id": "16"} }, + Field { "deleted_delete_files_count": Int32, metadata: {"PARQUET:field_id": "17"} }, + Field { "partition_summaries": List(non-null Struct("contains_null": non-null Boolean, metadata: {"PARQUET:field_id": "10"}, "contains_nan": Boolean, metadata: {"PARQUET:field_id": "11"}, "lower_bound": Utf8, metadata: {"PARQUET:field_id": "12"}, "upper_bound": Utf8, metadata: {"PARQUET:field_id": "13"}), metadata: {"PARQUET:field_id": "9"}), metadata: {"PARQUET:field_id": "8"} }"#]], expect![[r#" content: PrimitiveArray [ diff --git a/crates/iceberg/src/inspect/snapshots.rs b/crates/iceberg/src/inspect/snapshots.rs index 6081ec165b..fbed7ec11e 100644 --- a/crates/iceberg/src/inspect/snapshots.rs +++ b/crates/iceberg/src/inspect/snapshots.rs @@ -151,14 +151,14 @@ mod tests { check_record_batches( batch_stream.try_collect::>().await.unwrap(), expect![[r#" - Field { name: "committed_at", data_type: Timestamp(Microsecond, Some("+00:00")), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "snapshot_id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "parent_id", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }, - Field { name: "operation", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "4"} }, - Field { name: "manifest_list", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "5"} }, - Field { name: "summary", data_type: Map(Field { name: "key_value", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }"#]], + Field { "committed_at": Timestamp(µs, "+00:00"), metadata: {"PARQUET:field_id": "1"} }, + Field { "snapshot_id": Int64, metadata: {"PARQUET:field_id": "2"} }, + Field { "parent_id": nullable Int64, metadata: {"PARQUET:field_id": "3"} }, + Field { "operation": nullable Utf8, metadata: {"PARQUET:field_id": "4"} }, + Field { "manifest_list": nullable Utf8, metadata: {"PARQUET:field_id": "5"} }, + Field { "summary": nullable Map("key_value": non-null Struct("key": non-null Utf8, metadata: {"PARQUET:field_id": "7"}, "value": Utf8, metadata: {"PARQUET:field_id": "8"}), unsorted), metadata: {"PARQUET:field_id": "6"} }"#]], expect![[r#" - committed_at: PrimitiveArray + committed_at: PrimitiveArray [ 2018-01-04T21:22:35.770+00:00, 2019-04-12T20:29:15.770+00:00, diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 5cf031a9fb..356c2cb43d 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -27,12 +27,9 @@ use itertools::Itertools; use parquet::arrow::AsyncArrowWriter; use parquet::arrow::async_reader::AsyncFileReader; use parquet::arrow::async_writer::AsyncFileWriter as ArrowAsyncFileWriter; -use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader}; +use parquet::file::metadata::ParquetMetaData; use parquet::file::properties::WriterProperties; use parquet::file::statistics::Statistics; -use parquet::format::FileMetaData; -use parquet::thrift::{TCompactOutputProtocol, TSerializable}; -use thrift::protocol::TOutputProtocol; use super::{FileWriter, FileWriterBuilder}; use crate::arrow::{ @@ -349,29 +346,6 @@ impl ParquetWriter { Ok(data_files) } - fn thrift_to_parquet_metadata(&self, file_metadata: FileMetaData) -> Result { - let mut buffer = Vec::new(); - { - let mut protocol = TCompactOutputProtocol::new(&mut buffer); - file_metadata - .write_to_out_protocol(&mut protocol) - .map_err(|err| { - Error::new(ErrorKind::Unexpected, "Failed to write parquet metadata") - .with_source(err) - })?; - - protocol.flush().map_err(|err| { - Error::new(ErrorKind::Unexpected, "Failed to flush protocol").with_source(err) - })?; - } - - let parquet_metadata = ParquetMetaDataReader::decode_metadata(&buffer).map_err(|err| { - Error::new(ErrorKind::Unexpected, "Failed to decode parquet metadata").with_source(err) - })?; - - Ok(parquet_metadata) - } - /// `ParquetMetadata` to data file builder pub(crate) fn parquet_to_data_file_builder( schema: SchemaRef, @@ -564,14 +538,7 @@ impl FileWriter for ParquetWriter { })?; Ok(vec![]) } else { - let parquet_metadata = - Arc::new(self.thrift_to_parquet_metadata(metadata).map_err(|err| { - Error::new( - ErrorKind::Unexpected, - "Failed to convert metadata from thrift to parquet.", - ) - .with_source(err) - })?); + let parquet_metadata = Arc::new(metadata); Ok(vec![Self::parquet_to_data_file_builder( self.schema, diff --git a/crates/integrations/datafusion/src/table/table_provider_factory.rs b/crates/integrations/datafusion/src/table/table_provider_factory.rs index 8c0c8e90de..8cae597b7b 100644 --- a/crates/integrations/datafusion/src/table/table_provider_factory.rs +++ b/crates/integrations/datafusion/src/table/table_provider_factory.rs @@ -244,6 +244,7 @@ mod tests { constraints: Constraints::default(), column_defaults: Default::default(), if_not_exists: Default::default(), + or_replace: false, temporary: false, definition: Default::default(), unbounded: Default::default(), diff --git a/crates/integrations/datafusion/tests/integration_datafusion_test.rs b/crates/integrations/datafusion/tests/integration_datafusion_test.rs index 06d9cab03a..6f8898abb8 100644 --- a/crates/integrations/datafusion/tests/integration_datafusion_test.rs +++ b/crates/integrations/datafusion/tests/integration_datafusion_test.rs @@ -347,14 +347,14 @@ async fn test_metadata_table() -> Result<()> { check_record_batches( snapshots, expect![[r#" - Field { name: "committed_at", data_type: Timestamp(Microsecond, Some("+00:00")), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "snapshot_id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "parent_id", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }, - Field { name: "operation", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "4"} }, - Field { name: "manifest_list", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "5"} }, - Field { name: "summary", data_type: Map(Field { name: "key_value", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }"#]], + Field { "committed_at": Timestamp(µs, "+00:00"), metadata: {"PARQUET:field_id": "1"} }, + Field { "snapshot_id": Int64, metadata: {"PARQUET:field_id": "2"} }, + Field { "parent_id": nullable Int64, metadata: {"PARQUET:field_id": "3"} }, + Field { "operation": nullable Utf8, metadata: {"PARQUET:field_id": "4"} }, + Field { "manifest_list": nullable Utf8, metadata: {"PARQUET:field_id": "5"} }, + Field { "summary": nullable Map("key_value": non-null Struct("key": non-null Utf8, metadata: {"PARQUET:field_id": "7"}, "value": Utf8, metadata: {"PARQUET:field_id": "8"}), unsorted), metadata: {"PARQUET:field_id": "6"} }"#]], expect![[r#" - committed_at: PrimitiveArray + committed_at: PrimitiveArray [ ], snapshot_id: PrimitiveArray @@ -386,18 +386,18 @@ async fn test_metadata_table() -> Result<()> { check_record_batches( manifests, expect![[r#" - Field { name: "content", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "14"} }, - Field { name: "path", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "length", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "partition_spec_id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }, - Field { name: "added_snapshot_id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "4"} }, - Field { name: "added_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "5"} }, - Field { name: "existing_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }, - Field { name: "deleted_data_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, - Field { name: "added_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "15"} }, - Field { name: "existing_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "16"} }, - Field { name: "deleted_delete_files_count", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "17"} }, - Field { name: "partition_summaries", data_type: List(Field { name: "item", data_type: Struct([Field { name: "contains_null", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "10"} }, Field { name: "contains_nan", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "11"} }, Field { name: "lower_bound", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "12"} }, Field { name: "upper_bound", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "13"} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "9"} }), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }"#]], + Field { "content": Int32, metadata: {"PARQUET:field_id": "14"} }, + Field { "path": Utf8, metadata: {"PARQUET:field_id": "1"} }, + Field { "length": Int64, metadata: {"PARQUET:field_id": "2"} }, + Field { "partition_spec_id": Int32, metadata: {"PARQUET:field_id": "3"} }, + Field { "added_snapshot_id": Int64, metadata: {"PARQUET:field_id": "4"} }, + Field { "added_data_files_count": Int32, metadata: {"PARQUET:field_id": "5"} }, + Field { "existing_data_files_count": Int32, metadata: {"PARQUET:field_id": "6"} }, + Field { "deleted_data_files_count": Int32, metadata: {"PARQUET:field_id": "7"} }, + Field { "added_delete_files_count": Int32, metadata: {"PARQUET:field_id": "15"} }, + Field { "existing_delete_files_count": Int32, metadata: {"PARQUET:field_id": "16"} }, + Field { "deleted_delete_files_count": Int32, metadata: {"PARQUET:field_id": "17"} }, + Field { "partition_summaries": List(non-null Struct("contains_null": non-null Boolean, metadata: {"PARQUET:field_id": "10"}, "contains_nan": Boolean, metadata: {"PARQUET:field_id": "11"}, "lower_bound": Utf8, metadata: {"PARQUET:field_id": "12"}, "upper_bound": Utf8, metadata: {"PARQUET:field_id": "13"}), metadata: {"PARQUET:field_id": "9"}), metadata: {"PARQUET:field_id": "8"} }"#]], expect![[r#" content: PrimitiveArray [ @@ -504,8 +504,8 @@ async fn test_insert_into() -> Result<()> { check_record_batches( batches, expect![[r#" - Field { name: "foo1", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "foo2", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }"#]], + Field { "foo1": Int32, metadata: {"PARQUET:field_id": "1"} }, + Field { "foo2": Utf8, metadata: {"PARQUET:field_id": "2"} }"#]], expect![[r#" foo1: PrimitiveArray [ @@ -658,9 +658,9 @@ async fn test_insert_into_nested() -> Result<()> { check_record_batches( batches, expect![[r#" - Field { name: "id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "name", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "profile", data_type: Struct([Field { name: "address", data_type: Struct([Field { name: "street", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }, Field { name: "city", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, Field { name: "zip", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "4"} }, Field { name: "contact", data_type: Struct([Field { name: "email", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "9"} }, Field { name: "phone", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "10"} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "5"} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }"#]], + Field { "id": Int32, metadata: {"PARQUET:field_id": "1"} }, + Field { "name": Utf8, metadata: {"PARQUET:field_id": "2"} }, + Field { "profile": nullable Struct("address": Struct("street": non-null Utf8, metadata: {"PARQUET:field_id": "6"}, "city": non-null Utf8, metadata: {"PARQUET:field_id": "7"}, "zip": non-null Int32, metadata: {"PARQUET:field_id": "8"}), metadata: {"PARQUET:field_id": "4"}, "contact": Struct("email": Utf8, metadata: {"PARQUET:field_id": "9"}, "phone": Utf8, metadata: {"PARQUET:field_id": "10"}), metadata: {"PARQUET:field_id": "5"}), metadata: {"PARQUET:field_id": "3"} }"#]], expect![[r#" id: PrimitiveArray [ @@ -679,7 +679,7 @@ async fn test_insert_into_nested() -> Result<()> { valid, ] [ - -- child 0: "address" (Struct([Field { name: "street", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }, Field { name: "city", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, Field { name: "zip", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }])) + -- child 0: "address" (Struct([Field { name: "street", data_type: Utf8, metadata: {"PARQUET:field_id": "6"} }, Field { name: "city", data_type: Utf8, metadata: {"PARQUET:field_id": "7"} }, Field { name: "zip", data_type: Int32, metadata: {"PARQUET:field_id": "8"} }])) StructArray -- validity: [ @@ -706,7 +706,7 @@ async fn test_insert_into_nested() -> Result<()> { 95113, ] ] - -- child 1: "contact" (Struct([Field { name: "email", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "9"} }, Field { name: "phone", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "10"} }])) + -- child 1: "contact" (Struct([Field { name: "email", data_type: Utf8, nullable: true, metadata: {"PARQUET:field_id": "9"} }, Field { name: "phone", data_type: Utf8, nullable: true, metadata: {"PARQUET:field_id": "10"} }])) StructArray -- validity: [ @@ -757,13 +757,13 @@ async fn test_insert_into_nested() -> Result<()> { check_record_batches( batches, expect![[r#" - Field { name: "id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "name", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "catalog.test_insert_nested.nested_table.profile[address][street]", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "6"} }, - Field { name: "catalog.test_insert_nested.nested_table.profile[address][city]", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "7"} }, - Field { name: "catalog.test_insert_nested.nested_table.profile[address][zip]", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "8"} }, - Field { name: "catalog.test_insert_nested.nested_table.profile[contact][email]", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "9"} }, - Field { name: "catalog.test_insert_nested.nested_table.profile[contact][phone]", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "10"} }"#]], + Field { "id": Int32, metadata: {"PARQUET:field_id": "1"} }, + Field { "name": Utf8, metadata: {"PARQUET:field_id": "2"} }, + Field { "catalog.test_insert_nested.nested_table.profile[address][street]": nullable Utf8, metadata: {"PARQUET:field_id": "6"} }, + Field { "catalog.test_insert_nested.nested_table.profile[address][city]": nullable Utf8, metadata: {"PARQUET:field_id": "7"} }, + Field { "catalog.test_insert_nested.nested_table.profile[address][zip]": nullable Int32, metadata: {"PARQUET:field_id": "8"} }, + Field { "catalog.test_insert_nested.nested_table.profile[contact][email]": nullable Utf8, metadata: {"PARQUET:field_id": "9"} }, + Field { "catalog.test_insert_nested.nested_table.profile[contact][phone]": nullable Utf8, metadata: {"PARQUET:field_id": "10"} }"#]], expect![[r#" id: PrimitiveArray [ @@ -884,9 +884,9 @@ async fn test_insert_into_partitioned() -> Result<()> { check_record_batches( batches, expect![[r#" - Field { name: "id", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "1"} }, - Field { name: "category", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "2"} }, - Field { name: "value", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {"PARQUET:field_id": "3"} }"#]], + Field { "id": Int32, metadata: {"PARQUET:field_id": "1"} }, + Field { "category": Utf8, metadata: {"PARQUET:field_id": "2"} }, + Field { "value": Utf8, metadata: {"PARQUET:field_id": "3"} }"#]], expect![[r#" id: PrimitiveArray [ diff --git a/crates/integrations/playground/src/main.rs b/crates/integrations/playground/src/main.rs index c522209957..94068bb558 100644 --- a/crates/integrations/playground/src/main.rs +++ b/crates/integrations/playground/src/main.rs @@ -24,6 +24,7 @@ use clap::Parser; use datafusion::execution::runtime_env::RuntimeEnvBuilder; use datafusion::prelude::{SessionConfig, SessionContext}; use datafusion_cli::exec; +use datafusion_cli::object_storage::instrumented::InstrumentedObjectStoreRegistry; use datafusion_cli::print_format::PrintFormat; use datafusion_cli::print_options::{MaxRows, PrintOptions}; use iceberg_playground::{ICEBERG_PLAYGROUND_VERSION, IcebergCatalogList}; @@ -94,6 +95,7 @@ async fn main_inner() -> anyhow::Result<()> { quiet: args.quiet, maxrows: args.maxrows, color: args.color, + instrumented_registry: Arc::new(InstrumentedObjectStoreRegistry::new()), }; let rc = match args.rc { From 2944ccb551ec7c07ccc802a8edc01d5eb2ab6b87 Mon Sep 17 00:00:00 2001 From: Andrea Bozzo Date: Wed, 10 Dec 2025 08:11:33 +0100 Subject: [PATCH 109/891] fix: Serialize `split_offsets` as null when empty (#1906) - Change `split_offsets` in `DataFile` from `Vec` to `Option>` - Empty values now serialize as `null` instead of `[]` - Aligns with Iceberg spec (field is optional) Closes #1897 --------- Co-authored-by: Renjie Liu --- bindings/python/src/data_file.rs | 2 +- .../src/expr/visitors/expression_evaluator.rs | 4 ++-- .../visitors/inclusive_metrics_evaluator.rs | 12 +++++----- .../expr/visitors/strict_metrics_evaluator.rs | 8 +++---- crates/iceberg/src/spec/manifest/_serde.rs | 6 ++--- crates/iceberg/src/spec/manifest/data_file.rs | 10 ++++---- crates/iceberg/src/spec/manifest/mod.rs | 24 +++++++++---------- crates/iceberg/src/spec/manifest/writer.rs | 6 ++--- crates/iceberg/src/spec/snapshot_summary.rs | 10 ++++---- .../base_writer/equality_delete_writer.rs | 16 ++++++------- .../src/writer/file_writer/parquet_writer.rs | 4 ++-- 11 files changed, 52 insertions(+), 50 deletions(-) diff --git a/bindings/python/src/data_file.rs b/bindings/python/src/data_file.rs index 900d6c6014..b0e42e7d73 100644 --- a/bindings/python/src/data_file.rs +++ b/bindings/python/src/data_file.rs @@ -143,7 +143,7 @@ impl PyDataFile { } #[getter] - fn split_offsets(&self) -> &[i64] { + fn split_offsets(&self) -> Option<&[i64]> { self.inner.split_offsets() } diff --git a/crates/iceberg/src/expr/visitors/expression_evaluator.rs b/crates/iceberg/src/expr/visitors/expression_evaluator.rs index 3675ce355f..570c409502 100644 --- a/crates/iceberg/src/expr/visitors/expression_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/expression_evaluator.rs @@ -346,7 +346,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -374,7 +374,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, diff --git a/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs b/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs index 2b65cf12aa..06c92ab3e8 100644 --- a/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs @@ -1995,7 +1995,7 @@ mod test { lower_bounds: Default::default(), upper_bounds: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -2021,7 +2021,7 @@ mod test { lower_bounds: Default::default(), upper_bounds: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -2083,7 +2083,7 @@ mod test { column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -2114,7 +2114,7 @@ mod test { column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -2146,7 +2146,7 @@ mod test { column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -2178,7 +2178,7 @@ mod test { column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, diff --git a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs index 7c652e2068..a6af2990c8 100644 --- a/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs +++ b/crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs @@ -578,7 +578,7 @@ mod test { ]), column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -604,7 +604,7 @@ mod test { lower_bounds: Default::default(), upper_bounds: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -630,7 +630,7 @@ mod test { upper_bounds: HashMap::from([(1, Datum::int(42))]), column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -657,7 +657,7 @@ mod test { upper_bounds: HashMap::from([(3, Datum::string("dC"))]), column_sizes: Default::default(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, diff --git a/crates/iceberg/src/spec/manifest/_serde.rs b/crates/iceberg/src/spec/manifest/_serde.rs index 7738af46d4..07306be2b9 100644 --- a/crates/iceberg/src/spec/manifest/_serde.rs +++ b/crates/iceberg/src/spec/manifest/_serde.rs @@ -153,7 +153,7 @@ impl DataFileSerde { lower_bounds: Some(to_bytes_entry(value.lower_bounds)?), upper_bounds: Some(to_bytes_entry(value.upper_bounds)?), key_metadata: value.key_metadata.map(serde_bytes::ByteBuf::from), - split_offsets: Some(value.split_offsets), + split_offsets: value.split_offsets, equality_ids: value.equality_ids, sort_order_id: value.sort_order_id, first_row_id: value.first_row_id, @@ -222,7 +222,7 @@ impl DataFileSerde { .transpose()? .unwrap_or_default(), key_metadata: self.key_metadata.map(|v| v.to_vec()), - split_offsets: self.split_offsets.unwrap_or_default(), + split_offsets: self.split_offsets, equality_ids: self.equality_ids, sort_order_id: self.sort_order_id, partition_spec_id, @@ -380,7 +380,7 @@ mod tests { lower_bounds: HashMap::from([(1,Datum::int(1)),(2,Datum::string("a")),(3,Datum::string("AC/DC"))]), upper_bounds: HashMap::from([(1,Datum::int(1)),(2,Datum::string("a")),(3,Datum::string("AC/DC"))]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: Some(0), partition_spec_id: 0, diff --git a/crates/iceberg/src/spec/manifest/data_file.rs b/crates/iceberg/src/spec/manifest/data_file.rs index a9c041f540..77bd046f8a 100644 --- a/crates/iceberg/src/spec/manifest/data_file.rs +++ b/crates/iceberg/src/spec/manifest/data_file.rs @@ -127,9 +127,10 @@ pub struct DataFile { /// element field id: 133 /// /// Split offsets for the data file. For example, all row group offsets - /// in a Parquet file. Must be sorted ascending + /// in a Parquet file. Must be sorted ascending. Optional field that + /// should be serialized as null when not present. #[builder(default)] - pub(crate) split_offsets: Vec, + pub(crate) split_offsets: Option>, /// field id: 135 /// element field id: 136 /// @@ -247,8 +248,9 @@ impl DataFile { } /// Get the split offsets of the data file. /// For example, all row group offsets in a Parquet file. - pub fn split_offsets(&self) -> &[i64] { - &self.split_offsets + /// Returns `None` if no split offsets are present. + pub fn split_offsets(&self) -> Option<&[i64]> { + self.split_offsets.as_deref() } /// Get the equality ids of the data file. /// Field ids used to determine row equality in equality delete files. diff --git a/crates/iceberg/src/spec/manifest/mod.rs b/crates/iceberg/src/spec/manifest/mod.rs index 51219bfdb7..b126396e3c 100644 --- a/crates/iceberg/src/spec/manifest/mod.rs +++ b/crates/iceberg/src/spec/manifest/mod.rs @@ -257,7 +257,7 @@ mod tests { snapshot_id: None, sequence_number: None, file_sequence_number: None, - data_file: DataFile {content:DataContentType::Data,file_path:"s3a://icebergdata/demo/s1/t1/data/00000-0-ba56fbfa-f2ff-40c9-bb27-565ad6dc2be8-00000.parquet".to_string(),file_format:DataFileFormat::Parquet,partition:Struct::empty(),record_count:1,file_size_in_bytes:5442,column_sizes:HashMap::from([(0,73),(6,34),(2,73),(7,61),(3,61),(5,62),(9,79),(10,73),(1,61),(4,73),(8,73)]),value_counts:HashMap::from([(4,1),(5,1),(2,1),(0,1),(3,1),(6,1),(8,1),(1,1),(10,1),(7,1),(9,1)]),null_value_counts:HashMap::from([(1,0),(6,0),(2,0),(8,0),(0,0),(3,0),(5,0),(9,0),(7,0),(4,0),(10,0)]),nan_value_counts:HashMap::new(),lower_bounds:HashMap::new(),upper_bounds:HashMap::new(),key_metadata:None,split_offsets:vec![4],equality_ids:Some(Vec::new()),sort_order_id:None, partition_spec_id: 0,first_row_id: None,referenced_data_file: None,content_offset: None,content_size_in_bytes: None } + data_file: DataFile {content:DataContentType::Data,file_path:"s3a://icebergdata/demo/s1/t1/data/00000-0-ba56fbfa-f2ff-40c9-bb27-565ad6dc2be8-00000.parquet".to_string(),file_format:DataFileFormat::Parquet,partition:Struct::empty(),record_count:1,file_size_in_bytes:5442,column_sizes:HashMap::from([(0,73),(6,34),(2,73),(7,61),(3,61),(5,62),(9,79),(10,73),(1,61),(4,73),(8,73)]),value_counts:HashMap::from([(4,1),(5,1),(2,1),(0,1),(3,1),(6,1),(8,1),(1,1),(10,1),(7,1),(9,1)]),null_value_counts:HashMap::from([(1,0),(6,0),(2,0),(8,0),(0,0),(3,0),(5,0),(9,0),(7,0),(4,0),(10,0)]),nan_value_counts:HashMap::new(),lower_bounds:HashMap::new(),upper_bounds:HashMap::new(),key_metadata:None,split_offsets:Some(vec![4]),equality_ids:Some(Vec::new()),sort_order_id:None, partition_spec_id: 0,first_row_id: None,referenced_data_file: None,content_offset: None,content_size_in_bytes: None } } ]; @@ -435,7 +435,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: Some(Vec::new()), sort_order_id: None, partition_spec_id: 0, @@ -532,7 +532,7 @@ mod tests { lower_bounds: HashMap::from([(1,Datum::int(1)),(2,Datum::string("a")),(3,Datum::string("AC/DC"))]), upper_bounds: HashMap::from([(1,Datum::int(1)),(2,Datum::string("a")),(3,Datum::string("AC/DC"))]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: Some(0), partition_spec_id: 0, @@ -640,7 +640,7 @@ mod tests { (3, Datum::string("x")) ]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: Some(0), partition_spec_id: 0, @@ -749,7 +749,7 @@ mod tests { (3, Datum::string("x")) ]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -840,7 +840,7 @@ mod tests { (2, Datum::int(2)), ]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -922,7 +922,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -957,7 +957,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -992,7 +992,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -1027,7 +1027,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -1182,7 +1182,7 @@ mod tests { "lower_bounds": [], "upper_bounds": [], "key_metadata": null, - "split_offsets": [], + "split_offsets": null, "equality_ids": null, "sort_order_id": null, "first_row_id": null, @@ -1213,7 +1213,7 @@ mod tests { "lower_bounds": [], "upper_bounds": [], "key_metadata": null, - "split_offsets": [], + "split_offsets": null, "equality_ids": null, "sort_order_id": null, "first_row_id": null, diff --git a/crates/iceberg/src/spec/manifest/writer.rs b/crates/iceberg/src/spec/manifest/writer.rs index 389ac7a1fd..2fb6a42062 100644 --- a/crates/iceberg/src/spec/manifest/writer.rs +++ b/crates/iceberg/src/spec/manifest/writer.rs @@ -608,7 +608,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: Some(Vec::new()), - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -637,7 +637,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: Some(Vec::new()), - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -666,7 +666,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: Some(Vec::new()), - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: None, partition_spec_id: 0, diff --git a/crates/iceberg/src/spec/snapshot_summary.rs b/crates/iceberg/src/spec/snapshot_summary.rs index 4cd3715e06..c67ee37d3e 100644 --- a/crates/iceberg/src/spec/snapshot_summary.rs +++ b/crates/iceberg/src/spec/snapshot_summary.rs @@ -767,7 +767,7 @@ mod tests { (3, Datum::string("x")), ]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: Some(0), partition_spec_id: 0, @@ -799,7 +799,7 @@ mod tests { (3, Datum::string("x")), ]), key_metadata: None, - split_offsets: vec![4], + split_offsets: Some(vec![4]), equality_ids: None, sort_order_id: Some(0), partition_spec_id: 0, @@ -910,7 +910,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -938,7 +938,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, @@ -993,7 +993,7 @@ mod tests { lower_bounds: HashMap::new(), upper_bounds: HashMap::new(), key_metadata: None, - split_offsets: vec![], + split_offsets: None, equality_ids: None, sort_order_id: None, partition_spec_id: 0, diff --git a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs index cd0b19148d..dd8487f9cc 100644 --- a/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs +++ b/crates/iceberg/src/writer/base_writer/equality_delete_writer.rs @@ -293,15 +293,15 @@ mod test { assert_eq!(*data_file.null_value_counts.get(id).unwrap(), expect); } - assert_eq!(data_file.split_offsets.len(), metadata.num_row_groups()); - data_file + let split_offsets = data_file .split_offsets - .iter() - .enumerate() - .for_each(|(i, &v)| { - let expect = metadata.row_groups()[i].file_offset().unwrap(); - assert_eq!(v, expect); - }); + .as_ref() + .expect("split_offsets should be set"); + assert_eq!(split_offsets.len(), metadata.num_row_groups()); + split_offsets.iter().enumerate().for_each(|(i, &v)| { + let expect = metadata.row_groups()[i].file_offset().unwrap(); + assert_eq!(v, expect); + }); } #[tokio::test] diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 356c2cb43d..8fe40df71c 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -412,13 +412,13 @@ impl ParquetWriter { // - We can ignore implementing distinct_counts due to this: https://lists.apache.org/thread/j52tsojv0x4bopxyzsp7m7bqt23n5fnd .lower_bounds(lower_bounds) .upper_bounds(upper_bounds) - .split_offsets( + .split_offsets(Some( metadata .row_groups() .iter() .filter_map(|group| group.file_offset()) .collect(), - ); + )); Ok(builder) } From bdb44ae58a45ec12a34af28037901900d6020e08 Mon Sep 17 00:00:00 2001 From: Landon Gingerich Date: Wed, 10 Dec 2025 04:21:38 -0600 Subject: [PATCH 110/891] feat(catalog): Implement update_table for SqlCatalog (#1911) ## Which issue does this PR close? - Closes the SQL catalog part of https://github.com/apache/iceberg-rust/issues/1389 ## What changes are included in this PR? - Implement `update_table()` for SQL catalog - Add corresponding `test_update_table` test ## Are these changes tested? Yes. Covered by new `test_update_table` test. --- crates/catalog/sql/src/catalog.rs | 107 ++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 5 deletions(-) diff --git a/crates/catalog/sql/src/catalog.rs b/crates/catalog/sql/src/catalog.rs index 77b35a228f..8209cd04c1 100644 --- a/crates/catalog/sql/src/catalog.rs +++ b/crates/catalog/sql/src/catalog.rs @@ -917,11 +917,55 @@ impl Catalog for SqlCatalog { .build()?) } - async fn update_table(&self, _commit: TableCommit) -> Result
{ - Err(Error::new( - ErrorKind::FeatureUnsupported, - "Updating a table is not supported yet", - )) + /// Updates an existing table within the SQL catalog. + async fn update_table(&self, commit: TableCommit) -> Result
{ + let table_ident = commit.identifier().clone(); + let current_table = self.load_table(&table_ident).await?; + let current_metadata_location = current_table.metadata_location_result()?.to_string(); + + let staged_table = commit.apply(current_table)?; + let staged_metadata_location = staged_table.metadata_location_result()?; + + staged_table + .metadata() + .write_to(staged_table.file_io(), &staged_metadata_location) + .await?; + + let update_result = self + .execute( + &format!( + "UPDATE {CATALOG_TABLE_NAME} + SET {CATALOG_FIELD_METADATA_LOCATION_PROP} = ?, {CATALOG_FIELD_PREVIOUS_METADATA_LOCATION_PROP} = ? + WHERE {CATALOG_FIELD_CATALOG_NAME} = ? + AND {CATALOG_FIELD_TABLE_NAME} = ? + AND {CATALOG_FIELD_TABLE_NAMESPACE} = ? + AND ( + {CATALOG_FIELD_RECORD_TYPE} = '{CATALOG_FIELD_TABLE_RECORD_TYPE}' + OR {CATALOG_FIELD_RECORD_TYPE} IS NULL + ) + AND {CATALOG_FIELD_METADATA_LOCATION_PROP} = ?" + ), + vec![ + Some(staged_metadata_location), + Some(current_metadata_location.as_str()), + Some(&self.name), + Some(table_ident.name()), + Some(&table_ident.namespace().join(".")), + Some(current_metadata_location.as_str()), + ], + None, + ) + .await?; + + if update_result.rows_affected() == 0 { + return Err(Error::new( + ErrorKind::CatalogCommitConflicts, + format!("Commit conflicted for table: {table_ident}"), + ) + .with_retryable(true)); + } + + Ok(staged_table) } } @@ -932,6 +976,7 @@ mod tests { use iceberg::spec::{NestedField, PartitionSpec, PrimitiveType, Schema, SortOrder, Type}; use iceberg::table::Table; + use iceberg::transaction::{ApplyTransactionAction, Transaction}; use iceberg::{Catalog, CatalogBuilder, Namespace, NamespaceIdent, TableCreation, TableIdent}; use itertools::Itertools; use regex::Regex; @@ -2293,4 +2338,56 @@ mod tests { assert_eq!(table.identifier(), expected_table.identifier()); assert_eq!(table.metadata_location(), Some(metadata_location.as_str())); } + + #[tokio::test] + async fn test_update_table() { + let warehouse_loc = temp_path(); + let catalog = new_sql_catalog(warehouse_loc).await; + + // Create a test namespace and table + let namespace_ident = NamespaceIdent::new("ns1".into()); + create_namespace(&catalog, &namespace_ident).await; + let table_ident = TableIdent::new(namespace_ident.clone(), "tbl1".into()); + create_table(&catalog, &table_ident).await; + + let table = catalog.load_table(&table_ident).await.unwrap(); + + // Store the original metadata location for comparison + let original_metadata_location = table.metadata_location().unwrap().to_string(); + + // Create a transaction to update the table + let tx = Transaction::new(&table); + let tx = tx + .update_table_properties() + .set("test_property".to_string(), "test_value".to_string()) + .apply(tx) + .unwrap(); + + // Commit the transaction to the catalog + let updated_table = tx.commit(&catalog).await.unwrap(); + + // Verify the update was successful + assert_eq!( + updated_table.metadata().properties().get("test_property"), + Some(&"test_value".to_string()) + ); + // Verify the metadata location has been updated + assert_ne!( + updated_table.metadata_location().unwrap(), + original_metadata_location.as_str() + ); + + // Load the table again from the catalog to verify changes were persisted + let reloaded = catalog.load_table(&table_ident).await.unwrap(); + + // Verify the reloaded table matches the updated table + assert_eq!( + reloaded.metadata().properties().get("test_property"), + Some(&"test_value".to_string()) + ); + assert_eq!( + reloaded.metadata_location(), + updated_table.metadata_location() + ); + } } From 2ed0a6f509a5c6e7bdb143a74fa79c59a9df1bf2 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Wed, 10 Dec 2025 13:54:49 -0500 Subject: [PATCH 111/891] fix: Respect precision and scale for Decimal128 in value.rs (#1921) --- crates/iceberg/src/arrow/value.rs | 135 ++++++++++++++++++++++++++---- 1 file changed, 119 insertions(+), 16 deletions(-) diff --git a/crates/iceberg/src/arrow/value.rs b/crates/iceberg/src/arrow/value.rs index bc123d99e8..190aba08e8 100644 --- a/crates/iceberg/src/arrow/value.rs +++ b/crates/iceberg/src/arrow/value.rs @@ -663,18 +663,44 @@ pub(crate) fn create_primitive_array_single_element( (DataType::Binary, None) => Ok(Arc::new(BinaryArray::from_opt_vec(vec![ Option::<&[u8]>::None, ]))), - (DataType::Decimal128(_, _), Some(PrimitiveLiteral::Int128(v))) => { - Ok(Arc::new(arrow_array::Decimal128Array::from(vec![{ *v }]))) + (DataType::Decimal128(precision, scale), Some(PrimitiveLiteral::Int128(v))) => { + let array = Decimal128Array::from(vec![{ *v }]) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?; + Ok(Arc::new(array)) } - (DataType::Decimal128(_, _), Some(PrimitiveLiteral::UInt128(v))) => { - Ok(Arc::new(arrow_array::Decimal128Array::from(vec![ - *v as i128, - ]))) + (DataType::Decimal128(precision, scale), Some(PrimitiveLiteral::UInt128(v))) => { + let array = Decimal128Array::from(vec![*v as i128]) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?; + Ok(Arc::new(array)) } - (DataType::Decimal128(_, _), None) => { - Ok(Arc::new(arrow_array::Decimal128Array::from(vec![ - Option::::None, - ]))) + (DataType::Decimal128(precision, scale), None) => { + let array = Decimal128Array::from(vec![Option::::None]) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?; + Ok(Arc::new(array)) } (DataType::Struct(fields), None) => { // Create a single-element StructArray with nulls @@ -795,15 +821,48 @@ pub(crate) fn create_primitive_array_repeated( let vals: Vec> = vec![None; num_rows]; Arc::new(BinaryArray::from_opt_vec(vals)) } - (DataType::Decimal128(_, _), Some(PrimitiveLiteral::Int128(value))) => { - Arc::new(Decimal128Array::from(vec![*value; num_rows])) + (DataType::Decimal128(precision, scale), Some(PrimitiveLiteral::Int128(value))) => { + Arc::new( + Decimal128Array::from(vec![*value; num_rows]) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?, + ) } - (DataType::Decimal128(_, _), Some(PrimitiveLiteral::UInt128(value))) => { - Arc::new(Decimal128Array::from(vec![*value as i128; num_rows])) + (DataType::Decimal128(precision, scale), Some(PrimitiveLiteral::UInt128(value))) => { + Arc::new( + Decimal128Array::from(vec![*value as i128; num_rows]) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?, + ) } - (DataType::Decimal128(_, _), None) => { + (DataType::Decimal128(precision, scale), None) => { let vals: Vec> = vec![None; num_rows]; - Arc::new(Decimal128Array::from(vals)) + Arc::new( + Decimal128Array::from(vals) + .with_precision_and_scale(*precision, *scale) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Failed to create Decimal128Array with precision {precision} and scale {scale}: {e}" + ), + ) + })?, + ) } (DataType::Struct(fields), None) => { // Create a StructArray filled with nulls @@ -1678,4 +1737,48 @@ mod test { ]))), ]); } + + #[test] + fn test_create_decimal_array_respects_precision() { + // Decimal128Array::from() uses Arrow's default precision (38) instead of the + // target precision, causing RecordBatch construction to fail when schemas don't match. + let target_precision = 18u8; + let target_scale = 10i8; + let target_type = DataType::Decimal128(target_precision, target_scale); + let value = PrimitiveLiteral::Int128(10000000000); + + let array = create_primitive_array_single_element(&target_type, &Some(value)) + .expect("Failed to create decimal array"); + + match array.data_type() { + DataType::Decimal128(precision, scale) => { + assert_eq!(*precision, target_precision); + assert_eq!(*scale, target_scale); + } + other => panic!("Expected Decimal128, got {other:?}"), + } + } + + #[test] + fn test_create_decimal_array_repeated_respects_precision() { + // Ensure repeated arrays also respect target precision, not Arrow's default. + let target_precision = 18u8; + let target_scale = 10i8; + let target_type = DataType::Decimal128(target_precision, target_scale); + let value = PrimitiveLiteral::Int128(10000000000); + let num_rows = 5; + + let array = create_primitive_array_repeated(&target_type, &Some(value), num_rows) + .expect("Failed to create repeated decimal array"); + + match array.data_type() { + DataType::Decimal128(precision, scale) => { + assert_eq!(*precision, target_precision); + assert_eq!(*scale, target_scale); + } + other => panic!("Expected Decimal128, got {other:?}"), + } + + assert_eq!(array.len(), num_rows); + } } From 58bdb9fd94c9c5d847abffbdcb36e2273bb8bf14 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Thu, 11 Dec 2025 05:30:07 -0500 Subject: [PATCH 112/891] fix: restore no-op logic in constants_map for NULL identity-partitioned columns (#1922) ## Which issue does this PR close? See https://github.com/apache/iceberg-rust/pull/1824#discussion_r2584486989 and https://github.com/apache/iceberg-rust/issues/1914#issuecomment-3634315005 ## What changes are included in this PR? This restores the behavior in `record_batch_transformer.rs`'s `constants_map` function to pre-#1824 behavior where `NULL`s are not inserted into the constants map, and instead are just skipped. This allows the column projection rules for missing partition values to default to `NULL`. ## Are these changes tested? New test, and running the entire Iceberg Java suite via DataFusion Comet in https://github.com/apache/datafusion-comet/pull/2729. --- .../src/arrow/record_batch_transformer.rs | 81 +++++++++++++++++-- 1 file changed, 73 insertions(+), 8 deletions(-) diff --git a/crates/iceberg/src/arrow/record_batch_transformer.rs b/crates/iceberg/src/arrow/record_batch_transformer.rs index c4782464c1..439358435c 100644 --- a/crates/iceberg/src/arrow/record_batch_transformer.rs +++ b/crates/iceberg/src/arrow/record_batch_transformer.rs @@ -83,14 +83,10 @@ fn constants_map( // Handle both None (null) and Some(Literal::Primitive) cases match &partition_data[pos] { None => { - // TODO (https://github.com/apache/iceberg-rust/issues/1914): Add support for null datum values. - return Err(Error::new( - ErrorKind::Unexpected, - format!( - "Partition field {} has null value for identity transform", - field.source_id - ), - )); + // Skip null partition values - they will be resolved as null per Iceberg spec rule #4. + // When a partition value is null, we don't add it to the constants map, + // allowing downstream column resolution to handle it correctly. + continue; } Some(Literal::Primitive(value)) => { // Create a Datum from the primitive type and value @@ -1610,4 +1606,73 @@ mod test { assert_eq!(get_string_value(result.column(4).as_ref(), 0), ""); assert_eq!(get_string_value(result.column(4).as_ref(), 1), ""); } + + /// Test handling of null values in identity-partitioned columns. + /// + /// Reproduces TestPartitionValues.testNullPartitionValue() from iceberg-java, which + /// writes records where the partition column has null values. Before the fix in #1922, + /// this would error with "Partition field X has null value for identity transform". + #[test] + fn null_identity_partition_value() { + use crate::spec::{Struct, Transform}; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::optional(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "data", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + let partition_spec = Arc::new( + crate::spec::PartitionSpec::builder(schema.clone()) + .with_spec_id(0) + .add_partition_field("data", "data", Transform::Identity) + .unwrap() + .build() + .unwrap(), + ); + + // Partition has null value for the data column + let partition_data = Struct::from_iter(vec![None]); + + let file_schema = Arc::new(ArrowSchema::new(vec![simple_field( + "id", + DataType::Int32, + true, + "1", + )])); + + let projected_field_ids = [1, 2]; + + let mut transformer = RecordBatchTransformerBuilder::new(schema, &projected_field_ids) + .with_partition(partition_spec, partition_data) + .expect("Should handle null partition values") + .build(); + + let file_batch = + RecordBatch::try_new(file_schema, vec![Arc::new(Int32Array::from(vec![1, 2, 3]))]) + .unwrap(); + + let result = transformer.process_record_batch(file_batch).unwrap(); + + assert_eq!(result.num_columns(), 2); + assert_eq!(result.num_rows(), 3); + + let id_col = result + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_col.values(), &[1, 2, 3]); + + // Partition column with null value should produce nulls + let data_col = result.column(1); + assert!(data_col.is_null(0)); + assert!(data_col.is_null(1)); + assert!(data_col.is_null(2)); + } } From 16906c127d521395a789a9019350e467cc34d063 Mon Sep 17 00:00:00 2001 From: Lo <980321+dojiong@users.noreply.github.com> Date: Thu, 11 Dec 2025 18:32:34 +0800 Subject: [PATCH 113/891] fix: stack overflow when loading large equality deletes (#1915) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? A stack overflow occurs when processing data files containing a large number of equality deletes (e.g., > 6000 rows). This happens because parse_equality_deletes_record_batch_stream previously constructed the final predicate by linearly calling .and() in a loop: ```rust result_predicate = result_predicate.and(row_predicate.not()); ``` This resulted in a deeply nested, left-skewed tree structure with a depth equal to the number of rows (N). When rewrite_not() (which uses a recursive visitor pattern) was subsequently called on this structure, or when the structure was dropped, the call stack limit was exceeded. Changes 1. Balanced Tree Construction: Refactored the predicate combination logic. Instead of linear accumulation, row predicates are collected and combined using a pairwise combination approach to build a balanced tree. This reduces the tree depth from O(N) to O(log N). 2. Early Rewrite: rewrite_not() is now called immediately on each individual row predicate before they are combined. This ensures we are combining simplified predicates and avoids traversing a massive unoptimized tree later. 3. Regression Test: Added test_large_equality_delete_batch_stack_overflow, which processes 20,000 equality delete rows to verify the fix. ## Are these changes tested? - [x] New regression test test_large_equality_delete_batch_stack_overflow passed. - [x] All existing tests in arrow::caching_delete_file_loader passed. Co-authored-by: Renjie Liu --- .../src/arrow/caching_delete_file_loader.rs | 72 ++++++++++++++++++- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 192ca390a8..250fc5e8d9 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -330,7 +330,7 @@ impl CachingDeleteFileLoader { mut stream: ArrowRecordBatchStream, equality_ids: HashSet, ) -> Result { - let mut result_predicate = AlwaysTrue; + let mut row_predicates = Vec::new(); let mut batch_schema_iceberg: Option = None; let accessor = EqDelRecordBatchPartnerAccessor; @@ -374,10 +374,29 @@ impl CachingDeleteFileLoader { row_predicate = row_predicate.and(cell_predicate) } } - result_predicate = result_predicate.and(row_predicate.not()); + row_predicates.push(row_predicate.not().rewrite_not()); } } - Ok(result_predicate.rewrite_not()) + + // All row predicates are combined to a single predicate by creating a balanced binary tree. + // Using a simple fold would result in a deeply nested predicate that can cause a stack overflow. + while row_predicates.len() > 1 { + let mut next_level = Vec::with_capacity(row_predicates.len().div_ceil(2)); + let mut iter = row_predicates.into_iter(); + while let Some(p1) = iter.next() { + if let Some(p2) = iter.next() { + next_level.push(p1.and(p2)); + } else { + next_level.push(p1); + } + } + row_predicates = next_level; + } + + match row_predicates.pop() { + Some(p) => Ok(p), + None => Ok(AlwaysTrue), + } } } @@ -912,4 +931,51 @@ mod tests { result.err() ); } + + #[tokio::test] + async fn test_large_equality_delete_batch_stack_overflow() { + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().as_os_str().to_str().unwrap(); + let file_io = FileIO::from_path(table_location).unwrap().build().unwrap(); + + // Create a large batch of equality deletes + let num_rows = 20_000; + let col_y_vals: Vec = (0..num_rows).collect(); + let col_y = Arc::new(Int64Array::from(col_y_vals)) as ArrayRef; + + let schema = Arc::new(arrow_schema::Schema::new(vec![ + Field::new("y", arrow_schema::DataType::Int64, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + ])); + + let record_batch = RecordBatch::try_new(schema.clone(), vec![col_y]).unwrap(); + + // Write to file + let path = format!("{}/large-eq-deletes.parquet", &table_location); + let file = File::create(&path).unwrap(); + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + let mut writer = ArrowWriter::try_new(file, schema, Some(props)).unwrap(); + writer.write(&record_batch).unwrap(); + writer.close().unwrap(); + + let basic_delete_file_loader = BasicDeleteFileLoader::new(file_io.clone()); + let record_batch_stream = basic_delete_file_loader + .parquet_to_batch_stream(&path) + .await + .expect("could not get batch stream"); + + let eq_ids = HashSet::from_iter(vec![2]); + + let result = CachingDeleteFileLoader::parse_equality_deletes_record_batch_stream( + record_batch_stream, + eq_ids, + ) + .await; + + assert!(result.is_ok()); + } } From c25619df3f8f24f729d75d635b53efbead48a3ad Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 15 Dec 2025 09:53:21 +0800 Subject: [PATCH 114/891] chore(deps): Bump actions/upload-artifact from 5 to 6 (#1931) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6.
Release notes

Sourced from actions/upload-artifact's releases.

v6.0.0

v6 - What's new

[!IMPORTANT] actions/upload-artifact@v6 now runs on Node.js 24 (runs.using: node24) and requires a minimum Actions Runner version of 2.327.1. If you are using self-hosted runners, ensure they are updated before upgrading.

Node.js 24

This release updates the runtime to Node.js 24. v5 had preliminary support for Node.js 24, however this action was by default still running on Node.js 20. Now this action by default will run on Node.js 24.

What's Changed

Full Changelog: https://github.com/actions/upload-artifact/compare/v5.0.0...v6.0.0

Commits
  • b7c566a Merge pull request #745 from actions/upload-artifact-v6-release
  • e516bc8 docs: correct description of Node.js 24 support in README
  • ddc45ed docs: update README to correct action name for Node.js 24 support
  • 615b319 chore: release v6.0.0 for Node.js 24 support
  • 017748b Merge pull request #744 from actions/fix-storage-blob
  • 38d4c79 chore: rebuild dist
  • 7d27270 chore: add missing license cache files for @​actions/core, @​actions/io, and mi...
  • 5f643d3 chore: update license files for @​actions/artifact@​5.0.1 dependencies
  • 1df1684 chore: update package-lock.json with @​actions/artifact@​5.0.1
  • b5b1a91 fix: update @​actions/artifact to ^5.0.0 for Node.js 24 punycode fix
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/upload-artifact&package-manager=github_actions&previous-version=5&new-version=6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release_python.yml | 4 ++-- .github/workflows/release_python_nightly.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 9646085c7e..48f5a4d47a 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -107,7 +107,7 @@ jobs: command: sdist args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v5 + uses: actions/upload-artifact@v6 with: name: wheels-sdist path: bindings/python/dist @@ -159,7 +159,7 @@ jobs: command: build args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v5 + uses: actions/upload-artifact@v6 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 1d6faaeee9..cda2fd6b2c 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -56,7 +56,7 @@ jobs: args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v5 + uses: actions/upload-artifact@v6 with: name: wheels-sdist path: bindings/python/dist @@ -102,7 +102,7 @@ jobs: args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v5 + uses: actions/upload-artifact@v6 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist From 939617116aeeccd77b8705e5d0cfbb356491d4ea Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 15 Dec 2025 10:09:25 +0800 Subject: [PATCH 115/891] chore(deps): Bump actions/download-artifact from 6 to 7 (#1932) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 6 to 7.
Release notes

Sourced from actions/download-artifact's releases.

v7.0.0

v7 - What's new

[!IMPORTANT] actions/download-artifact@v7 now runs on Node.js 24 (runs.using: node24) and requires a minimum Actions Runner version of 2.327.1. If you are using self-hosted runners, ensure they are updated before upgrading.

Node.js 24

This release updates the runtime to Node.js 24. v6 had preliminary support for Node 24, however this action was by default still running on Node.js 20. Now this action by default will run on Node.js 24.

What's Changed

New Contributors

Full Changelog: https://github.com/actions/download-artifact/compare/v6.0.0...v7.0.0

Commits
  • 37930b1 Merge pull request #452 from actions/download-artifact-v7-release
  • 72582b9 doc: update readme
  • 0d2ec9d chore: release v7.0.0 for Node.js 24 support
  • fd7ae8f Merge pull request #451 from actions/fix-storage-blob
  • d484700 chore: restore minimatch.dep.yml license file
  • 03a8080 chore: remove obsolete dependency license files
  • 56fe6d9 chore: update @​actions/artifact license file to 5.0.1
  • 8e3ebc4 chore: update package-lock.json with @​actions/artifact@​5.0.1
  • 1e3c4b4 fix: update @​actions/artifact to ^5.0.0 for Node.js 24 punycode fix
  • 458627d chore: use local @​actions/artifact package for Node.js 24 testing
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/download-artifact&package-manager=github_actions&previous-version=6&new-version=7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release_python.yml | 2 +- .github/workflows/release_python_nightly.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 48f5a4d47a..85663fc75f 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -178,7 +178,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v6 + uses: actions/download-artifact@v7 with: pattern: wheels-* merge-multiple: true diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index cda2fd6b2c..833b8ee6a6 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -120,7 +120,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v6 + uses: actions/download-artifact@v7 with: pattern: wheels-* merge-multiple: true From 36aedc6fbccec5a25d649d1792923d140ed83cbd Mon Sep 17 00:00:00 2001 From: Landon Gingerich Date: Mon, 15 Dec 2025 04:16:15 -0600 Subject: [PATCH 116/891] Remove wildcard pattern in exhaustive enums (#1925) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1924. ## What changes are included in this PR? - For match expression on `PrimitiveType`, removes wildcard and explicitly handles all enum variants. - Removes support of `PrimititveType::TimestampzNs` for HMS and Glue. - Hive/HMS does not support timezone aware timestamps, per [Hive docs](https://cwiki.apache.org/confluence/display/hive/languagemanual+types#LanguageManualTypes-TimestampstimestampTimestamps). - Glue uses the Hive type system so it should also be disabled for Glue. ## Are these changes tested? Tested with existing test suite --- crates/catalog/glue/src/schema.rs | 13 ++++++------- crates/catalog/hms/src/schema.rs | 13 ++++++------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/crates/catalog/glue/src/schema.rs b/crates/catalog/glue/src/schema.rs index cfd7487973..864320dae4 100644 --- a/crates/catalog/glue/src/schema.rs +++ b/crates/catalog/glue/src/schema.rs @@ -165,7 +165,12 @@ impl SchemaVisitor for GlueSchemaBuilder { PrimitiveType::Date => "date".to_string(), PrimitiveType::Timestamp => "timestamp".to_string(), PrimitiveType::TimestampNs => "timestamp_ns".to_string(), - PrimitiveType::TimestamptzNs => "timestamptz_ns".to_string(), + PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs => { + return Err(Error::new( + ErrorKind::FeatureUnsupported, + format!("Conversion from {p:?} is not supported"), + )); + } PrimitiveType::Time | PrimitiveType::String | PrimitiveType::Uuid => { "string".to_string() } @@ -173,12 +178,6 @@ impl SchemaVisitor for GlueSchemaBuilder { PrimitiveType::Decimal { precision, scale } => { format!("decimal({precision},{scale})") } - _ => { - return Err(Error::new( - ErrorKind::FeatureUnsupported, - "Conversion from 'Timestamptz' is not supported", - )); - } }; Ok(glue_type) diff --git a/crates/catalog/hms/src/schema.rs b/crates/catalog/hms/src/schema.rs index 8893a80521..c23b48719d 100644 --- a/crates/catalog/hms/src/schema.rs +++ b/crates/catalog/hms/src/schema.rs @@ -122,7 +122,12 @@ impl SchemaVisitor for HiveSchemaBuilder { PrimitiveType::Date => "date".to_string(), PrimitiveType::Timestamp => "timestamp".to_string(), PrimitiveType::TimestampNs => "timestamp_ns".to_string(), - PrimitiveType::TimestamptzNs => "timestamptz_ns".to_string(), + PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs => { + return Err(Error::new( + ErrorKind::FeatureUnsupported, + format!("Conversion from {p:?} is not supported"), + )); + } PrimitiveType::Time | PrimitiveType::String | PrimitiveType::Uuid => { "string".to_string() } @@ -130,12 +135,6 @@ impl SchemaVisitor for HiveSchemaBuilder { PrimitiveType::Decimal { precision, scale } => { format!("decimal({precision},{scale})") } - _ => { - return Err(Error::new( - ErrorKind::FeatureUnsupported, - "Conversion from 'Timestamptz' is not supported", - )); - } }; Ok(hive_type) From 26a7606e0f04d5175e452a809c7bdceb47ba2c07 Mon Sep 17 00:00:00 2001 From: Lo <980321+dojiong@users.noreply.github.com> Date: Mon, 15 Dec 2025 18:30:10 +0800 Subject: [PATCH 117/891] fix: prioritize delete manifests to prevent scan deadlock (#1937) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This change ensures that delete manifests are processed before data manifests during the table scan planning phase. Previously, if data manifests were processed first and produced enough entries to fill the channel, the producer would block. Since the delete manifest consumer might still be waiting for its entries (which hadn't been produced yet), this could lead to a deadlock. Prioritizing delete manifests ensures the delete consumer can proceed, allowing the data consumer to eventually drain the channel. ## Are these changes tested? Added a reproduction test case `test_scan_deadlock` to verify the fix. --- crates/iceberg/src/scan/context.rs | 12 ++- crates/iceberg/src/scan/mod.rs | 125 +++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/scan/context.rs b/crates/iceberg/src/scan/context.rs index fe3f5c8f7e..f28b6b0901 100644 --- a/crates/iceberg/src/scan/context.rs +++ b/crates/iceberg/src/scan/context.rs @@ -194,7 +194,17 @@ impl PlanContext { delete_file_idx: DeleteFileIndex, delete_file_tx: Sender, ) -> Result> + 'static>> { - let manifest_files = manifest_list.entries().iter(); + let mut manifest_files = manifest_list.entries().iter().collect::>(); + // Sort manifest files to process delete manifests first. + // This avoids a deadlock where the producer blocks on sending data manifest entries + // (because the data channel is full) while the delete manifest consumer is waiting + // for delete manifest entries (which haven't been produced yet). + // By processing delete manifests first, we ensure the delete consumer can finish, + // which then allows the data consumer to start draining the data channel. + manifest_files.sort_by_key(|m| match m.content { + ManifestContentType::Deletes => 0, + ManifestContentType::Data => 1, + }); // TODO: Ideally we could ditch this intermediate Vec as we return an iterator. let mut filtered_mfcs = vec![]; diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index d83da8a879..1f7fa50df8 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -1170,6 +1170,97 @@ pub mod tests { writer.close().unwrap(); } } + + pub async fn setup_deadlock_manifests(&mut self) { + let current_snapshot = self.table.metadata().current_snapshot().unwrap(); + let _parent_snapshot = current_snapshot + .parent_snapshot(self.table.metadata()) + .unwrap(); + let current_schema = current_snapshot.schema(self.table.metadata()).unwrap(); + let current_partition_spec = self.table.metadata().default_partition_spec(); + + // 1. Write DATA manifest with MULTIPLE entries to fill buffer + let mut writer = ManifestWriterBuilder::new( + self.next_manifest_file(), + Some(current_snapshot.snapshot_id()), + None, + current_schema.clone(), + current_partition_spec.as_ref().clone(), + ) + .build_v2_data(); + + // Add 10 data entries + for i in 0..10 { + writer + .add_entry( + ManifestEntry::builder() + .status(ManifestStatus::Added) + .data_file( + DataFileBuilder::default() + .partition_spec_id(0) + .content(DataContentType::Data) + .file_path(format!("{}/{}.parquet", &self.table_location, i)) + .file_format(DataFileFormat::Parquet) + .file_size_in_bytes(100) + .record_count(1) + .partition(Struct::from_iter([Some(Literal::long(100))])) + .key_metadata(None) + .build() + .unwrap(), + ) + .build(), + ) + .unwrap(); + } + let data_manifest = writer.write_manifest_file().await.unwrap(); + + // 2. Write DELETE manifest + let mut writer = ManifestWriterBuilder::new( + self.next_manifest_file(), + Some(current_snapshot.snapshot_id()), + None, + current_schema.clone(), + current_partition_spec.as_ref().clone(), + ) + .build_v2_deletes(); + + writer + .add_entry( + ManifestEntry::builder() + .status(ManifestStatus::Added) + .data_file( + DataFileBuilder::default() + .partition_spec_id(0) + .content(DataContentType::PositionDeletes) + .file_path(format!("{}/del.parquet", &self.table_location)) + .file_format(DataFileFormat::Parquet) + .file_size_in_bytes(100) + .record_count(1) + .partition(Struct::from_iter([Some(Literal::long(100))])) + .build() + .unwrap(), + ) + .build(), + ) + .unwrap(); + let delete_manifest = writer.write_manifest_file().await.unwrap(); + + // Write to manifest list - DATA FIRST then DELETE + // This order is crucial for reproduction + let mut manifest_list_write = ManifestListWriter::v2( + self.table + .file_io() + .new_output(current_snapshot.manifest_list()) + .unwrap(), + current_snapshot.snapshot_id(), + current_snapshot.parent_snapshot_id(), + current_snapshot.sequence_number(), + ); + manifest_list_write + .add_manifests(vec![data_manifest, delete_manifest].into_iter()) + .unwrap(); + manifest_list_write.close().await.unwrap(); + } } #[test] @@ -2127,4 +2218,38 @@ pub mod tests { "_file column (duplicate) should use RunEndEncoded type" ); } + + #[tokio::test] + async fn test_scan_deadlock() { + let mut fixture = TableTestFixture::new(); + fixture.setup_deadlock_manifests().await; + + // Create table scan with concurrency limit 1 + // This sets channel size to 1. + // Data manifest has 10 entries -> will block producer. + // Delete manifest is 2nd in list -> won't be processed. + // Consumer 2 (Data) not started -> blocked. + // Consumer 1 (Delete) waiting -> blocked. + let table_scan = fixture + .table + .scan() + .with_concurrency_limit(1) + .build() + .unwrap(); + + // This should timeout/hang if deadlock exists + // We can use tokio::time::timeout + let result = tokio::time::timeout(std::time::Duration::from_secs(5), async { + table_scan + .plan_files() + .await + .unwrap() + .try_collect::>() + .await + }) + .await; + + // Assert it finished (didn't timeout) + assert!(result.is_ok(), "Scan timed out - deadlock detected"); + } } From b047baa476a46dd83e8d5fcd2b586a8b95f46d92 Mon Sep 17 00:00:00 2001 From: Fokko Driesprong Date: Mon, 15 Dec 2025 12:01:56 +0100 Subject: [PATCH 118/891] feat: Include statistics for Reserved Fields (#1849) This is a behavioral change. In Iceberg-Rust we require upper/lower bounds to be part of the schema. But in some cases, this isn't the case, for example when you use reserved fields. In PyIceberg we expect these values in some tests: ``` FAILED tests/integration/test_inspect_table.py::test_inspect_files[2] - AssertionError: Difference in column lower_bounds: {} != {2147483546: b's3://warehouse/default/table_metadata_files/data/00000-0-8d621c18-079b-4217-afd8-559ce216e875.parquet', 2147483545: b'\x00\x00\x00\x00\x00\x00\x00\x00'} assert {} == {2147483545: ...e875.parquet'} Right contains 2 more items: {2147483545: b'\x00\x00\x00\x00\x00\x00\x00\x00', 2147483546: b's3://warehouse/default/table_metadata_files/data/00000-0-8d621c1' b'8-079b-4217-afd8-559ce216e875.parquet'} Full diff: { + , - 2147483545: b'\x00\x00\x00\x00\x00\x00\x00\x00', - 2147483546: b's3://warehouse/default/table_metadata_files/data/00000-0-8d621c1' - b'8-079b-4217-afd8-559ce216e875.parquet', } !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! ==== 1 failed, 238 passed, 32 skipped, 3123 deselected in 61.56s (0:01:01) ===== ``` This is a positional delete where the field-IDs are constant, but never part of a schema (they are reserved). ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? --------- Co-authored-by: Kevin Liu Co-authored-by: Renjie Liu --- crates/iceberg/src/metadata_columns.rs | 460 ++++++++++++++++++++- crates/iceberg/src/spec/manifest/_serde.rs | 42 +- 2 files changed, 492 insertions(+), 10 deletions(-) diff --git a/crates/iceberg/src/metadata_columns.rs b/crates/iceberg/src/metadata_columns.rs index b388e40c4a..b622a76edc 100644 --- a/crates/iceberg/src/metadata_columns.rs +++ b/crates/iceberg/src/metadata_columns.rs @@ -32,11 +32,74 @@ use crate::{Error, ErrorKind, Result}; /// Reserved field ID for the file path (_file) column per Iceberg spec pub const RESERVED_FIELD_ID_FILE: i32 = i32::MAX - 1; +/// Reserved field ID for the position (_pos) column per Iceberg spec +pub const RESERVED_FIELD_ID_POS: i32 = i32::MAX - 2; + +/// Reserved field ID for the deleted (_deleted) column per Iceberg spec +pub const RESERVED_FIELD_ID_DELETED: i32 = i32::MAX - 3; + +/// Reserved field ID for the spec ID (_spec_id) column per Iceberg spec +pub const RESERVED_FIELD_ID_SPEC_ID: i32 = i32::MAX - 4; + +/// Reserved field ID for the partition (_partition) column per Iceberg spec +pub const RESERVED_FIELD_ID_PARTITION: i32 = i32::MAX - 5; + +/// Reserved field ID for the file path in position delete files +pub const RESERVED_FIELD_ID_DELETE_FILE_PATH: i32 = i32::MAX - 101; + +/// Reserved field ID for the position in position delete files +pub const RESERVED_FIELD_ID_DELETE_FILE_POS: i32 = i32::MAX - 102; + +/// Reserved field ID for the change type (_change_type) column per Iceberg spec +pub const RESERVED_FIELD_ID_CHANGE_TYPE: i32 = i32::MAX - 104; + +/// Reserved field ID for the change ordinal (_change_ordinal) column per Iceberg spec +pub const RESERVED_FIELD_ID_CHANGE_ORDINAL: i32 = i32::MAX - 105; + +/// Reserved field ID for the commit snapshot ID (_commit_snapshot_id) column per Iceberg spec +pub const RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID: i32 = i32::MAX - 106; + +/// Reserved field ID for the row ID (_row_id) column per Iceberg spec +pub const RESERVED_FIELD_ID_ROW_ID: i32 = i32::MAX - 107; + +/// Reserved field ID for the last updated sequence number (_last_updated_sequence_number) column per Iceberg spec +pub const RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER: i32 = i32::MAX - 108; + /// Reserved column name for the file path metadata column pub const RESERVED_COL_NAME_FILE: &str = "_file"; -/// Documentation for the _file metadata column -pub const RESERVED_COL_DOC_FILE: &str = "Path of the file in which a row is stored"; +/// Reserved column name for the position metadata column +pub const RESERVED_COL_NAME_POS: &str = "_pos"; + +/// Reserved column name for the deleted metadata column +pub const RESERVED_COL_NAME_DELETED: &str = "_deleted"; + +/// Reserved column name for the spec ID metadata column +pub const RESERVED_COL_NAME_SPEC_ID: &str = "_spec_id"; + +/// Reserved column name for the partition metadata column +pub const RESERVED_COL_NAME_PARTITION: &str = "_partition"; + +/// Reserved column name for the file path in position delete files +pub const RESERVED_COL_NAME_DELETE_FILE_PATH: &str = "file_path"; + +/// Reserved column name for the position in position delete files +pub const RESERVED_COL_NAME_DELETE_FILE_POS: &str = "pos"; + +/// Reserved column name for the change type metadata column +pub const RESERVED_COL_NAME_CHANGE_TYPE: &str = "_change_type"; + +/// Reserved column name for the change ordinal metadata column +pub const RESERVED_COL_NAME_CHANGE_ORDINAL: &str = "_change_ordinal"; + +/// Reserved column name for the commit snapshot ID metadata column +pub const RESERVED_COL_NAME_COMMIT_SNAPSHOT_ID: &str = "_commit_snapshot_id"; + +/// Reserved column name for the row ID metadata column +pub const RESERVED_COL_NAME_ROW_ID: &str = "_row_id"; + +/// Reserved column name for the last updated sequence number metadata column +pub const RESERVED_COL_NAME_LAST_UPDATED_SEQUENCE_NUMBER: &str = "_last_updated_sequence_number"; /// Lazy-initialized Iceberg field definition for the _file metadata column. /// This field represents the file path as a required string field. @@ -47,7 +110,139 @@ static FILE_FIELD: Lazy = Lazy::new(|| { RESERVED_COL_NAME_FILE, Type::Primitive(PrimitiveType::String), ) - .with_doc(RESERVED_COL_DOC_FILE), + .with_doc("Path of the file in which a row is stored"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _pos metadata column. +/// This field represents the ordinal position of a row in the source data file. +static POS_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_POS, + RESERVED_COL_NAME_POS, + Type::Primitive(PrimitiveType::Long), + ) + .with_doc("Ordinal position of a row in the source data file"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _deleted metadata column. +/// This field indicates whether a row has been deleted. +static DELETED_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_DELETED, + RESERVED_COL_NAME_DELETED, + Type::Primitive(PrimitiveType::Boolean), + ) + .with_doc("Whether the row has been deleted"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _spec_id metadata column. +/// This field represents the spec ID used to track the file containing a row. +static SPEC_ID_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_SPEC_ID, + RESERVED_COL_NAME_SPEC_ID, + Type::Primitive(PrimitiveType::Int), + ) + .with_doc("Spec ID used to track the file containing a row"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the file_path column in position delete files. +/// This field represents the path of a file in position-based delete files. +static DELETE_FILE_PATH_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_DELETE_FILE_PATH, + RESERVED_COL_NAME_DELETE_FILE_PATH, + Type::Primitive(PrimitiveType::String), + ) + .with_doc("Path of a file, used in position-based delete files"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the pos column in position delete files. +/// This field represents the ordinal position of a row in position-based delete files. +static DELETE_FILE_POS_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_DELETE_FILE_POS, + RESERVED_COL_NAME_DELETE_FILE_POS, + Type::Primitive(PrimitiveType::Long), + ) + .with_doc("Ordinal position of a row, used in position-based delete files"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _change_type metadata column. +/// This field represents the record type in the changelog. +static CHANGE_TYPE_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_CHANGE_TYPE, + RESERVED_COL_NAME_CHANGE_TYPE, + Type::Primitive(PrimitiveType::String), + ) + .with_doc( + "The record type in the changelog (INSERT, DELETE, UPDATE_BEFORE, or UPDATE_AFTER)", + ), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _change_ordinal metadata column. +/// This field represents the order of the change. +static CHANGE_ORDINAL_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_CHANGE_ORDINAL, + RESERVED_COL_NAME_CHANGE_ORDINAL, + Type::Primitive(PrimitiveType::Int), + ) + .with_doc("The order of the change"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _commit_snapshot_id metadata column. +/// This field represents the snapshot ID in which the change occurred. +static COMMIT_SNAPSHOT_ID_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID, + RESERVED_COL_NAME_COMMIT_SNAPSHOT_ID, + Type::Primitive(PrimitiveType::Long), + ) + .with_doc("The snapshot ID in which the change occurred"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _row_id metadata column. +/// This field represents a unique long assigned for row lineage. +static ROW_ID_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_ROW_ID, + RESERVED_COL_NAME_ROW_ID, + Type::Primitive(PrimitiveType::Long), + ) + .with_doc("A unique long assigned for row lineage"), + ) +}); + +/// Lazy-initialized Iceberg field definition for the _last_updated_sequence_number metadata column. +/// This field represents the sequence number which last updated this row. +static LAST_UPDATED_SEQUENCE_NUMBER_FIELD: Lazy = Lazy::new(|| { + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER, + RESERVED_COL_NAME_LAST_UPDATED_SEQUENCE_NUMBER, + Type::Primitive(PrimitiveType::Long), + ) + .with_doc("The sequence number which last updated this row"), ) }); @@ -59,16 +254,159 @@ pub fn file_field() -> &'static NestedFieldRef { &FILE_FIELD } +/// Returns the Iceberg field definition for the _pos metadata column. +/// +/// # Returns +/// A reference to the _pos field definition as an Iceberg NestedField +pub fn pos_field() -> &'static NestedFieldRef { + &POS_FIELD +} + +/// Returns the Iceberg field definition for the _deleted metadata column. +/// +/// # Returns +/// A reference to the _deleted field definition as an Iceberg NestedField +pub fn deleted_field() -> &'static NestedFieldRef { + &DELETED_FIELD +} + +/// Returns the Iceberg field definition for the _spec_id metadata column. +/// +/// # Returns +/// A reference to the _spec_id field definition as an Iceberg NestedField +pub fn spec_id_field() -> &'static NestedFieldRef { + &SPEC_ID_FIELD +} + +/// Returns the Iceberg field definition for the file_path column in position delete files. +/// +/// # Returns +/// A reference to the file_path field definition as an Iceberg NestedField +pub fn delete_file_path_field() -> &'static NestedFieldRef { + &DELETE_FILE_PATH_FIELD +} + +/// Returns the Iceberg field definition for the pos column in position delete files. +/// +/// # Returns +/// A reference to the pos field definition as an Iceberg NestedField +pub fn delete_file_pos_field() -> &'static NestedFieldRef { + &DELETE_FILE_POS_FIELD +} + +/// Returns the Iceberg field definition for the _change_type metadata column. +/// +/// # Returns +/// A reference to the _change_type field definition as an Iceberg NestedField +pub fn change_type_field() -> &'static NestedFieldRef { + &CHANGE_TYPE_FIELD +} + +/// Returns the Iceberg field definition for the _change_ordinal metadata column. +/// +/// # Returns +/// A reference to the _change_ordinal field definition as an Iceberg NestedField +pub fn change_ordinal_field() -> &'static NestedFieldRef { + &CHANGE_ORDINAL_FIELD +} + +/// Returns the Iceberg field definition for the _commit_snapshot_id metadata column. +/// +/// # Returns +/// A reference to the _commit_snapshot_id field definition as an Iceberg NestedField +pub fn commit_snapshot_id_field() -> &'static NestedFieldRef { + &COMMIT_SNAPSHOT_ID_FIELD +} + +/// Returns the Iceberg field definition for the _row_id metadata column. +/// +/// # Returns +/// A reference to the _row_id field definition as an Iceberg NestedField +pub fn row_id_field() -> &'static NestedFieldRef { + &ROW_ID_FIELD +} + +/// Returns the Iceberg field definition for the _last_updated_sequence_number metadata column. +/// +/// # Returns +/// A reference to the _last_updated_sequence_number field definition as an Iceberg NestedField +pub fn last_updated_sequence_number_field() -> &'static NestedFieldRef { + &LAST_UPDATED_SEQUENCE_NUMBER_FIELD +} + +/// Creates the Iceberg field definition for the _partition metadata column. +/// +/// The _partition field is a struct whose fields depend on the partition spec. +/// This function creates the field dynamically with the provided partition fields. +/// +/// # Arguments +/// * `partition_fields` - The fields that make up the partition struct +/// +/// # Returns +/// A new _partition field definition as an Iceberg NestedField +/// +/// # Example +/// ``` +/// use std::sync::Arc; +/// +/// use iceberg::metadata_columns::partition_field; +/// use iceberg::spec::{NestedField, PrimitiveType, Type}; +/// +/// let fields = vec![ +/// Arc::new(NestedField::required( +/// 1, +/// "year", +/// Type::Primitive(PrimitiveType::Int), +/// )), +/// Arc::new(NestedField::required( +/// 2, +/// "month", +/// Type::Primitive(PrimitiveType::Int), +/// )), +/// ]; +/// let partition_field = partition_field(fields); +/// ``` +pub fn partition_field(partition_fields: Vec) -> NestedFieldRef { + use crate::spec::StructType; + + Arc::new( + NestedField::required( + RESERVED_FIELD_ID_PARTITION, + RESERVED_COL_NAME_PARTITION, + Type::Struct(StructType::new(partition_fields)), + ) + .with_doc("Partition to which a row belongs"), + ) +} + /// Returns the Iceberg field definition for a metadata field ID. /// +/// Note: This function does not support `_partition` (field ID `i32::MAX - 5`) because +/// it's a struct field that requires dynamic partition fields. Use `partition_field()` +/// instead to create the `_partition` field with the appropriate partition fields. +/// /// # Arguments /// * `field_id` - The metadata field ID /// /// # Returns /// The Iceberg field definition for the metadata column, or an error if not a metadata field -pub fn get_metadata_field(field_id: i32) -> Result { +pub fn get_metadata_field(field_id: i32) -> Result<&'static NestedFieldRef> { match field_id { - RESERVED_FIELD_ID_FILE => Ok(Arc::clone(file_field())), + RESERVED_FIELD_ID_FILE => Ok(file_field()), + RESERVED_FIELD_ID_POS => Ok(pos_field()), + RESERVED_FIELD_ID_DELETED => Ok(deleted_field()), + RESERVED_FIELD_ID_SPEC_ID => Ok(spec_id_field()), + RESERVED_FIELD_ID_PARTITION => Err(Error::new( + ErrorKind::Unexpected, + "The _partition field must be created using partition_field() with appropriate partition fields", + )), + RESERVED_FIELD_ID_DELETE_FILE_PATH => Ok(delete_file_path_field()), + RESERVED_FIELD_ID_DELETE_FILE_POS => Ok(delete_file_pos_field()), + RESERVED_FIELD_ID_CHANGE_TYPE => Ok(change_type_field()), + RESERVED_FIELD_ID_CHANGE_ORDINAL => Ok(change_ordinal_field()), + RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID => Ok(commit_snapshot_id_field()), + RESERVED_FIELD_ID_ROW_ID => Ok(row_id_field()), + RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER => Ok(last_updated_sequence_number_field()), _ if is_metadata_field(field_id) => { // Future metadata fields can be added here Err(Error::new( @@ -95,6 +433,19 @@ pub fn get_metadata_field(field_id: i32) -> Result { pub fn get_metadata_field_id(column_name: &str) -> Result { match column_name { RESERVED_COL_NAME_FILE => Ok(RESERVED_FIELD_ID_FILE), + RESERVED_COL_NAME_POS => Ok(RESERVED_FIELD_ID_POS), + RESERVED_COL_NAME_DELETED => Ok(RESERVED_FIELD_ID_DELETED), + RESERVED_COL_NAME_SPEC_ID => Ok(RESERVED_FIELD_ID_SPEC_ID), + RESERVED_COL_NAME_PARTITION => Ok(RESERVED_FIELD_ID_PARTITION), + RESERVED_COL_NAME_DELETE_FILE_PATH => Ok(RESERVED_FIELD_ID_DELETE_FILE_PATH), + RESERVED_COL_NAME_DELETE_FILE_POS => Ok(RESERVED_FIELD_ID_DELETE_FILE_POS), + RESERVED_COL_NAME_CHANGE_TYPE => Ok(RESERVED_FIELD_ID_CHANGE_TYPE), + RESERVED_COL_NAME_CHANGE_ORDINAL => Ok(RESERVED_FIELD_ID_CHANGE_ORDINAL), + RESERVED_COL_NAME_COMMIT_SNAPSHOT_ID => Ok(RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID), + RESERVED_COL_NAME_ROW_ID => Ok(RESERVED_FIELD_ID_ROW_ID), + RESERVED_COL_NAME_LAST_UPDATED_SEQUENCE_NUMBER => { + Ok(RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER) + } _ => Err(Error::new( ErrorKind::Unexpected, format!("Unknown/unsupported metadata column name: {column_name}"), @@ -110,8 +461,21 @@ pub fn get_metadata_field_id(column_name: &str) -> Result { /// # Returns /// `true` if the field ID is a (currently supported) metadata field, `false` otherwise pub fn is_metadata_field(field_id: i32) -> bool { - field_id == RESERVED_FIELD_ID_FILE - // Additional metadata fields can be checked here in the future + matches!( + field_id, + RESERVED_FIELD_ID_FILE + | RESERVED_FIELD_ID_POS + | RESERVED_FIELD_ID_DELETED + | RESERVED_FIELD_ID_SPEC_ID + | RESERVED_FIELD_ID_PARTITION + | RESERVED_FIELD_ID_DELETE_FILE_PATH + | RESERVED_FIELD_ID_DELETE_FILE_POS + | RESERVED_FIELD_ID_CHANGE_TYPE + | RESERVED_FIELD_ID_CHANGE_ORDINAL + | RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID + | RESERVED_FIELD_ID_ROW_ID + | RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER + ) } /// Checks if a column name is a metadata column. @@ -124,3 +488,85 @@ pub fn is_metadata_field(field_id: i32) -> bool { pub fn is_metadata_column_name(column_name: &str) -> bool { get_metadata_field_id(column_name).is_ok() } + +#[cfg(test)] +mod tests { + use super::*; + use crate::spec::PrimitiveType; + + #[test] + fn test_partition_field_creation() { + // Create partition fields for a hypothetical year/month partition + let partition_fields = vec![ + Arc::new(NestedField::required( + 1000, + "year", + Type::Primitive(PrimitiveType::Int), + )), + Arc::new(NestedField::required( + 1001, + "month", + Type::Primitive(PrimitiveType::Int), + )), + ]; + + // Create the _partition metadata field + let partition = partition_field(partition_fields); + + // Verify field properties + assert_eq!(partition.id, RESERVED_FIELD_ID_PARTITION); + assert_eq!(partition.name, RESERVED_COL_NAME_PARTITION); + assert!(partition.required); + + // Verify it's a struct type with correct fields + if let Type::Struct(struct_type) = partition.field_type.as_ref() { + assert_eq!(struct_type.fields().len(), 2); + assert_eq!(struct_type.fields()[0].name, "year"); + assert_eq!(struct_type.fields()[1].name, "month"); + } else { + panic!("Expected struct type for _partition field"); + } + } + + #[test] + fn test_partition_field_id_recognized() { + assert!(is_metadata_field(RESERVED_FIELD_ID_PARTITION)); + } + + #[test] + fn test_partition_field_name_recognized() { + assert_eq!( + get_metadata_field_id(RESERVED_COL_NAME_PARTITION).unwrap(), + RESERVED_FIELD_ID_PARTITION + ); + } + + #[test] + fn test_get_metadata_field_returns_error_for_partition() { + // partition field requires dynamic creation, so get_metadata_field should return an error + let result = get_metadata_field(RESERVED_FIELD_ID_PARTITION); + assert!(result.is_err()); + assert!( + result + .unwrap_err() + .to_string() + .contains("partition_field()") + ); + } + + #[test] + fn test_all_metadata_field_ids() { + // Test that all non-partition metadata fields can be retrieved + assert!(get_metadata_field(RESERVED_FIELD_ID_FILE).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_POS).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_DELETED).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_SPEC_ID).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_DELETE_FILE_PATH).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_DELETE_FILE_POS).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_CHANGE_TYPE).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_CHANGE_ORDINAL).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_COMMIT_SNAPSHOT_ID).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_ROW_ID).is_ok()); + assert!(get_metadata_field(RESERVED_FIELD_ID_LAST_UPDATED_SEQUENCE_NUMBER).is_ok()); + } +} diff --git a/crates/iceberg/src/spec/manifest/_serde.rs b/crates/iceberg/src/spec/manifest/_serde.rs index 07306be2b9..247b6dde5f 100644 --- a/crates/iceberg/src/spec/manifest/_serde.rs +++ b/crates/iceberg/src/spec/manifest/_serde.rs @@ -22,7 +22,7 @@ use serde_with::serde_as; use super::{Datum, ManifestEntry, Schema, Struct}; use crate::spec::{FormatVersion, Literal, RawLiteral, StructType, Type}; -use crate::{Error, ErrorKind}; +use crate::{Error, ErrorKind, metadata_columns}; #[derive(Serialize, Deserialize)] pub(super) struct ManifestEntryV2 { @@ -245,8 +245,12 @@ struct BytesEntry { fn parse_bytes_entry(v: Vec, schema: &Schema) -> Result, Error> { let mut m = HashMap::with_capacity(v.len()); for entry in v { - // We ignore the entry if the field is not found in the schema, due to schema evolution. - if let Some(field) = schema.field_by_id(entry.key) { + // First try to find the field in the schema, or check if it's a reserved metadata field + let field = schema + .field_by_id(entry.key) + .or_else(|| metadata_columns::get_metadata_field(entry.key).ok()); + + if let Some(field) = field { let data_type = field .field_type .as_primitive_type() @@ -259,6 +263,7 @@ fn parse_bytes_entry(v: Vec, schema: &Schema) -> Result Date: Tue, 16 Dec 2025 20:22:07 +0800 Subject: [PATCH 119/891] Add case-sensitive support for equality deletes in DeleteFilter (#1930) --- .../src/arrow/caching_delete_file_loader.rs | 1 + crates/iceberg/src/arrow/delete_filter.rs | 62 ++++++++++++++++++- crates/iceberg/src/arrow/reader.rs | 15 +++++ crates/iceberg/src/scan/context.rs | 5 ++ crates/iceberg/src/scan/mod.rs | 2 + crates/iceberg/src/scan/task.rs | 3 + 6 files changed, 85 insertions(+), 3 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 250fc5e8d9..aceeae49f7 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -911,6 +911,7 @@ mod tests { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; // Load the deletes - should handle both types without error diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index 14b5124ee6..d05e028997 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -141,8 +141,8 @@ impl DeleteFilter { return Ok(None); } - // TODO: handle case-insensitive case - let bound_predicate = combined_predicate.bind(file_scan_task.schema.clone(), false)?; + let bound_predicate = combined_predicate + .bind(file_scan_task.schema.clone(), file_scan_task.case_sensitive)?; Ok(Some(bound_predicate)) } @@ -211,8 +211,9 @@ pub(crate) mod tests { use super::*; use crate::arrow::caching_delete_file_loader::CachingDeleteFileLoader; + use crate::expr::Reference; use crate::io::FileIO; - use crate::spec::{DataFileFormat, Schema}; + use crate::spec::{DataFileFormat, Datum, NestedField, PrimitiveType, Schema, Type}; type ArrowSchemaRef = Arc; @@ -344,6 +345,7 @@ pub(crate) mod tests { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }, FileScanTask { start: 0, @@ -358,6 +360,7 @@ pub(crate) mod tests { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }, ]; @@ -380,4 +383,57 @@ pub(crate) mod tests { ]; Arc::new(arrow_schema::Schema::new(fields)) } + + #[tokio::test] + async fn test_build_equality_delete_predicate_case_sensitive() { + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "Id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(), + ); + + // ---------- fake FileScanTask ---------- + let task = FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: "data.parquet".to_string(), + data_file_format: crate::spec::DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![], + predicate: None, + deletes: vec![FileScanTaskDeleteFile { + file_path: "eq-del.parquet".to_string(), + file_type: DataContentType::EqualityDeletes, + partition_spec_id: 0, + equality_ids: None, + }], + partition: None, + partition_spec: None, + name_mapping: None, + case_sensitive: true, + }; + + let filter = DeleteFilter::default(); + + // ---------- insert equality delete predicate ---------- + let pred = Reference::new("id").equal_to(Datum::long(10)); + + let (tx, rx) = tokio::sync::oneshot::channel(); + filter.insert_equality_delete("eq-del.parquet", rx); + + tx.send(pred).unwrap(); + + // ---------- should FAIL ---------- + let result = filter.build_equality_delete_predicate(&task).await; + + assert!( + result.is_err(), + "case_sensitive=true should fail when column case mismatches" + ); + } } diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 6209c1e261..f7f90663a5 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -2082,6 +2082,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -2403,6 +2404,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; // Task 2: read the second and third row groups @@ -2419,6 +2421,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; let tasks1 = Box::pin(futures::stream::iter(vec![Ok(task1)])) as FileScanTaskStream; @@ -2546,6 +2549,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -2717,6 +2721,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -2934,6 +2939,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -3144,6 +3150,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as FileScanTaskStream; @@ -3247,6 +3254,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3344,6 +3352,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3430,6 +3439,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3530,6 +3540,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3659,6 +3670,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3755,6 +3767,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -3864,6 +3877,7 @@ message schema { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; @@ -4003,6 +4017,7 @@ message schema { partition: Some(partition_data), partition_spec: Some(partition_spec), name_mapping: None, + case_sensitive: false, })] .into_iter(), )) as FileScanTaskStream; diff --git a/crates/iceberg/src/scan/context.rs b/crates/iceberg/src/scan/context.rs index f28b6b0901..169d8e6405 100644 --- a/crates/iceberg/src/scan/context.rs +++ b/crates/iceberg/src/scan/context.rs @@ -46,6 +46,7 @@ pub(crate) struct ManifestFileContext { snapshot_schema: SchemaRef, expression_evaluator_cache: Arc, delete_file_index: DeleteFileIndex, + case_sensitive: bool, } /// Wraps a [`ManifestEntryRef`] alongside the objects that are needed @@ -59,6 +60,7 @@ pub(crate) struct ManifestEntryContext { pub partition_spec_id: i32, pub snapshot_schema: SchemaRef, pub delete_file_index: DeleteFileIndex, + pub case_sensitive: bool, } impl ManifestFileContext { @@ -89,6 +91,7 @@ impl ManifestFileContext { bound_predicates: bound_predicates.clone(), snapshot_schema: snapshot_schema.clone(), delete_file_index: delete_file_index.clone(), + case_sensitive: self.case_sensitive, }; sender @@ -135,6 +138,7 @@ impl ManifestEntryContext { partition_spec: None, // TODO: Extract name_mapping from table metadata property "schema.name-mapping.default" name_mapping: None, + case_sensitive: self.case_sensitive, }) } } @@ -277,6 +281,7 @@ impl PlanContext { field_ids: self.field_ids.clone(), expression_evaluator_cache: self.expression_evaluator_cache.clone(), delete_file_index, + case_sensitive: self.case_sensitive, } } } diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index 1f7fa50df8..c055c12c9a 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -1885,6 +1885,7 @@ pub mod tests { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; test_fn(task); @@ -1902,6 +1903,7 @@ pub mod tests { partition: None, partition_spec: None, name_mapping: None, + case_sensitive: false, }; test_fn(task); } diff --git a/crates/iceberg/src/scan/task.rs b/crates/iceberg/src/scan/task.rs index e1ef241a57..5349a9bdd2 100644 --- a/crates/iceberg/src/scan/task.rs +++ b/crates/iceberg/src/scan/task.rs @@ -104,6 +104,9 @@ pub struct FileScanTask { #[serde(serialize_with = "serialize_not_implemented")] #[serde(deserialize_with = "deserialize_not_implemented")] pub name_mapping: Option>, + + /// Whether this scan task should treat column names as case-sensitive when binding predicates. + pub case_sensitive: bool, } impl FileScanTask { From d4c4bd4ad3375ef012a5e943498e588bf808d14a Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 17 Dec 2025 07:57:34 +0800 Subject: [PATCH 120/891] Bump to version 0.8.0 (#1938) ## Which issue does this PR close? - Part of https://github.com/apache/iceberg-rust/issues/1850 ## What changes are included in this PR? ## Are these changes tested? --- **Parts of this PR were drafted with assistance from Codex (with `gpt-5.2`) and fully reviewed and edited by me. I take full responsibility for all changes.** --- CHANGELOG.md | 144 +++ Cargo.lock | 28 +- Cargo.toml | 16 +- bindings/python/Cargo.lock | 265 +++-- bindings/python/Cargo.toml | 2 +- bindings/python/DEPENDENCIES.rust.tsv | 884 ++++++++------- crates/catalog/glue/DEPENDENCIES.rust.tsv | 357 +++--- crates/catalog/hms/DEPENDENCIES.rust.tsv | 323 +++--- crates/catalog/loader/DEPENDENCIES.rust.tsv | 401 +++---- crates/catalog/rest/DEPENDENCIES.rust.tsv | 302 +++-- crates/catalog/s3tables/DEPENDENCIES.rust.tsv | 357 +++--- crates/catalog/sql/DEPENDENCIES.rust.tsv | 303 +++-- crates/examples/DEPENDENCIES.rust.tsv | 306 +++-- crates/iceberg/DEPENDENCIES.rust.tsv | 299 +++-- .../integration_tests/DEPENDENCIES.rust.tsv | 820 +++++++------- .../cache-moka/DEPENDENCIES.rust.tsv | 301 +++-- .../datafusion/DEPENDENCIES.rust.tsv | 810 ++++++------- .../playground/DEPENDENCIES.rust.tsv | 1001 ++++++++--------- crates/sqllogictest/DEPENDENCIES.rust.tsv | 880 ++++++++------- crates/test_utils/DEPENDENCIES.rust.tsv | 36 +- deny.toml | 3 +- 21 files changed, 4024 insertions(+), 3814 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f66f64f478..50d4576dfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,150 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/) and this project adheres to [Semantic Versioning](https://semver.org/). +## [v0.8.0] - 2025-12-16 + +### Breaking Changes + +* **API Changes:** + * refactor: Remove redundant parameters from SnapshotProducer validation methods by @Li0k in https://github.com/apache/iceberg-rust/pull/1853 + * chore: Remove deprecated `remove_all` in FileIO by @jonathanc-n in https://github.com/apache/iceberg-rust/pull/1863 + * refactor: Drop smol runtime support by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1900 + +* **Compatibility:** + * chore: bump MSRV to 1.88, fix warnings and clippy errors by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1902 + +* **Dependency Updates:** + * Upgrade opendal to v0.55 by @dentiny in https://github.com/apache/iceberg-rust/pull/1895 + * deps: bump DataFusion to 51, Arrow to 57, pyo to 0.26 by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1899 + +* **Other:** + * Remove wildcard pattern in exhaustive enums by @lgingerich in https://github.com/apache/iceberg-rust/pull/1925 + +### All Changes + +* chore(deps): Bump tempfile from 3.22.0 to 3.23.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1717 +* chore(deps): Bump rand from 0.8.5 to 0.9.2 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1716 +* chore(deps): Bump crate-ci/typos from 1.36.2 to 1.36.3 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1715 +* refactor: Improve REST catalog's authenticate method by @imor in https://github.com/apache/iceberg-rust/pull/1712 +* chore(deps): Bump serde_with from 3.14.0 to 3.14.1 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1727 +* refactor(writer): Refactor writers for the future partitioning writers by @CTTY in https://github.com/apache/iceberg-rust/pull/1657 +* Set lock on version of Pydantic by @Fokko in https://github.com/apache/iceberg-rust/pull/1737 +* chore(deps): Bump crate-ci/typos from 1.36.3 to 1.37.2 by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1734 +* feat: support more partition transformations for PartitionSpec::partition_to_path by @mnpw in https://github.com/apache/iceberg-rust/pull/1730 +* chore: Update website for 0.7.0 by @CTTY in https://github.com/apache/iceberg-rust/pull/1738 +* feat(sql-catalog): implement register table for sql catalog by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1724 +* fix: ensure CoalescePartitionsExec is enabled for IcebergCommitExec by @sgrebnov in https://github.com/apache/iceberg-rust/pull/1723 +* chore(deps): Bump regex from 1.11.2 to 1.12.1 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1741 +* chore(deps): Bump crate-ci/typos from 1.37.2 to 1.38.1 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1740 +* Improve `IcebergCommitExec` to correctly populate properties/schema by @sgrebnov in https://github.com/apache/iceberg-rust/pull/1721 +* feat(spec): add `table_properties.rs` to spec by @kaushiksrini in https://github.com/apache/iceberg-rust/pull/1733 +* chore(deps): Bump actions/stale from 10.0.0 to 10.1.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1726 +* docs: remove -src suffix from artifact name by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1743 +* feat(reader): Make ArrowReaderBuilder::new public by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1748 +* feat(writer): Add clustered and fanout writer by @CTTY in https://github.com/apache/iceberg-rust/pull/1735 +* feat(catalog): impl builder for SqlCatalog by @335g in https://github.com/apache/iceberg-rust/pull/1666 +* fix: fix read parquert file when schema change by @chenzl25 in https://github.com/apache/iceberg-rust/pull/1750 +* docs: Fix broken orbstack and podman links in CONTRIBUTING.md by @petern48 in https://github.com/apache/iceberg-rust/pull/1757 +* chore(deps): Bump tokio from 1.47.1 to 1.48.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1763 +* chore(deps): Bump backon from 1.5.2 to 1.6.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1762 +* fix: global eq delete matching should apply to only strictly older files, and fix partition scoped matching to consider spec id by @amogh-jahagirdar in https://github.com/apache/iceberg-rust/pull/1758 +* chore(deps): Bump apache/skywalking-eyes from 0.7.0 to 0.8.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1760 +* chore(deps): Bump rust_decimal from 1.38.0 to 1.39.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1761 +* feat(datafusion): implement the project node to add the partition columns by @fvaleye in https://github.com/apache/iceberg-rust/pull/1602 +* fix: snapshot was producing empty summary by @imor in https://github.com/apache/iceberg-rust/pull/1767 +* docs: Add examples for PartitioningWriter by @CTTY in https://github.com/apache/iceberg-rust/pull/1754 +* feat(sqllogictest): Add support for iceberg datafusion sqllogictest integration by @lliangyu-lin in https://github.com/apache/iceberg-rust/pull/1764 +* fix(build): Pin home version after merging #1764 by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1783 +* minor: Update Cargo.lock to add home by @CTTY in https://github.com/apache/iceberg-rust/pull/1785 +* chore(deps): Bump aws-sdk-s3tables from 1.40.0 to 1.41.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1790 +* chore(deps): Bump rand from 0.8.5 to 0.9.2 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1789 +* chore(deps): Bump actions/download-artifact from 5 to 6 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1788 +* chore(deps): Bump actions/upload-artifact from 4 to 5 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1787 +* fix(reader): filter row groups when FileScanTask contains byte ranges by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1779 +* refactor(arrow,datafusion): Reuse PartitionValueCalculator in RecordBatchPartitionSplitter by @CTTY in https://github.com/apache/iceberg-rust/pull/1781 +* feat: Update Datafusion to v49 by @DerGut in https://github.com/apache/iceberg-rust/pull/1704 +* deps: unpin pydantic by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1793 +* feat(reader): Add Date32 support to RecordBatchTransformer create_column by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1792 +* feat(catalog): Implement update_table for S3TablesCatalog by @CTTY in https://github.com/apache/iceberg-rust/pull/1594 +* feat: Update Datafusion to v50 by @DerGut in https://github.com/apache/iceberg-rust/pull/1728 +* ci: Migrate to uv for python by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1796 +* ci: Relax msrv check thanks to rust 2024 by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1795 +* ci: Don't dismiss stale review to make contribution easier by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1799 +* add Makefile to bindings/python by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1800 +* chore: inline format args by @colinmarc in https://github.com/apache/iceberg-rust/pull/1805 +* refactor: Migrate from tera to minijinja by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1798 +* fix(reader): fix position delete bugs with row group skipping by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1806 +* feat(datafusion): implement the partitioning node for DataFusion to define the partitioning by @fvaleye in https://github.com/apache/iceberg-rust/pull/1620 +* feat(reader): Date32 from days since epoch for Literal:try_from_json by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1803 +* chore(deps): Bump aws-sdk-glue from 1.125.0 to 1.126.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1812 +* chore(deps): Bump astral-sh/setup-uv from 6 to 7 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1811 +* chore(deps): Bump crate-ci/typos from 1.38.1 to 1.39.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1810 +* feat(reader): position-based column projection for Parquet files without field IDs (migrated tables) by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1777 +* fix(reader): Equality delete files with partial schemas (containing only equality columns) by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1782 +* infra: use apache/hive:4.0.0 as hive Dockerfile base image by @geruh in https://github.com/apache/iceberg-rust/pull/1823 +* fix: StructType fails to deserialize JSON with type field by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1822 +* feat: Support for V3 Metadata by @c-thiel in https://github.com/apache/iceberg-rust/pull/1682 +* fix(reader): Support both position and equality delete files on the same FileScanTask by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1778 +* feat(datafusion): Add TaskWriter for DataFusion by @CTTY in https://github.com/apache/iceberg-rust/pull/1769 +* fix: support reading compressed metadata by @colinmarc in https://github.com/apache/iceberg-rust/pull/1802 +* Support deserializing bytes by @Fokko in https://github.com/apache/iceberg-rust/pull/1820 +* fix: Bump CI Spark version to 3.5.7 by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1832 +* infra: use python 3.12 for release by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1836 +* pyiceberg-core: create smaller artifacts by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1841 +* infra: add collaborators to .asf.yaml by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1842 +* pyiceberg-core: use pyo3 abi3-py310 by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1843 +* ci: parallelize unit test with matrix by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1833 +* pyiceberg-core: create even smaller artifacts by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1844 +* chore: Split values.rs into separate files by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1840 +* feat(datafusion): Support `INSERT INTO` partitioned tables by @CTTY in https://github.com/apache/iceberg-rust/pull/1827 +* docs: Add Wrappers project to README by @burmecia in https://github.com/apache/iceberg-rust/pull/1852 +* feat(reader): Add PartitionSpec support to FileScanTask and RecordBatchTransformer by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1821 +* feat(reader): null struct default values in create_column by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1847 +* refactor: Remove redundant parameters from SnapshotProducer validation methods by @Li0k in https://github.com/apache/iceberg-rust/pull/1853 +* infra: add verbose=true to pypa/gh-action-pypi-publish by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1846 +* use RecordBatchTransformerBuilder instead of RecordBatchTransformer by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1857 +* chore(deps): Bump bytes from 1.10.1 to 1.11.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1861 +* chore(deps): Bump serde_with from 3.15.1 to 3.16.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1859 +* chore(deps): Bump fs-err from 3.1.3 to 3.2.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1860 +* chore(deps): Bump crate-ci/typos from 1.39.0 to 1.39.2 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1858 +* chore: Remove deprecated `remove_all` in FileIO by @jonathanc-n in https://github.com/apache/iceberg-rust/pull/1863 +* infra: notify on github workflow failure by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1870 +* feat(reader): Add binary support to `get_arrow_datum` for equality deletes with binary type by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1848 +* Raise concurrency errors properly for glue tables by @jembishop in https://github.com/apache/iceberg-rust/pull/1875 +* infra: add instructions for cleaning up testpypi artifacts by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1855 +* chore(deps): Bump actions/checkout from 5 to 6 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1883 +* Update apache-avro to v0.21.0 by @N-Boutaib in https://github.com/apache/iceberg-rust/pull/1881 +* docs: Clarify functionality of `SnapshotProduceOperation` by @jonathanc-n in https://github.com/apache/iceberg-rust/pull/1874 +* feat(datafusion): Split IcebergTableProvider into static and non-static table provider by @CTTY in https://github.com/apache/iceberg-rust/pull/1879 +* infra: use new `del_branch_on_merge` in .asf.yaml by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1888 +* Upgrade opendal to v0.55 by @dentiny in https://github.com/apache/iceberg-rust/pull/1895 +* chore(deps): Bump http from 1.3.1 to 1.4.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1892 +* chore(deps): Bump crate-ci/typos from 1.39.2 to 1.40.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1891 +* feat(datafusion): Add `sort_by_partition` to sort the input partitioned data by @CTTY in https://github.com/apache/iceberg-rust/pull/1618 +* rfc: Modularize `iceberg` Implementations by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1854 +* refactor(writer): Make writer builders non-consuming in build by @leonzchang in https://github.com/apache/iceberg-rust/pull/1889 +* fix: Keep snapshot log on replace by @c-thiel in https://github.com/apache/iceberg-rust/pull/1896 +* chore(deps): Bump actions/stale from 10.1.0 to 10.1.1 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1908 +* feat(datafusion): Add sqllogictest for DataFusion INSERT INTO by @CTTY in https://github.com/apache/iceberg-rust/pull/1887 +* refactor: Drop smol runtime support by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1900 +* chore(deps): Bump minijinja from 2.12.0 to 2.13.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1909 +* chore(deps): Bump uuid from 1.18.1 to 1.19.0 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1910 +* feat(core): Add support for `_file` column by @gbrgr in https://github.com/apache/iceberg-rust/pull/1824 +* feat: Make `rest` types public, add documentation by @c-thiel in https://github.com/apache/iceberg-rust/pull/1901 +* chore: bump MSRV to 1.88, fix warnings and clippy errors by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1902 +* ci: Make s3tables ready for publish by @Xuanwo in https://github.com/apache/iceberg-rust/pull/1916 +* deps: bump DataFusion to 51, Arrow to 57, pyo to 0.26 by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1899 +* fix: Serialize `split_offsets` as null when empty by @AndreaBozzo in https://github.com/apache/iceberg-rust/pull/1906 +* feat(catalog): Implement update_table for SqlCatalog by @lgingerich in https://github.com/apache/iceberg-rust/pull/1911 +* fix: Respect precision and scale for Decimal128 in value.rs by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1921 +* fix: restore no-op logic in constants_map for NULL identity-partitioned columns by @mbutrovich in https://github.com/apache/iceberg-rust/pull/1922 +* fix: stack overflow when loading large equality deletes by @dojiong in https://github.com/apache/iceberg-rust/pull/1915 +* chore(deps): Bump actions/upload-artifact from 5 to 6 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1931 +* chore(deps): Bump actions/download-artifact from 6 to 7 by @dependabot[bot] in https://github.com/apache/iceberg-rust/pull/1932 +* Remove wildcard pattern in exhaustive enums by @lgingerich in https://github.com/apache/iceberg-rust/pull/1925 +* fix: prioritize delete manifests to prevent scan deadlock by @dojiong in https://github.com/apache/iceberg-rust/pull/1937 +* feat: Include statistics for Reserved Fields by @Fokko in https://github.com/apache/iceberg-rust/pull/1849 + ## [v0.7.0] - 2025-09-23 ### Breaking Changes diff --git a/Cargo.lock b/Cargo.lock index 2d464709fd..97ee25d658 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3344,7 +3344,7 @@ dependencies = [ [[package]] name = "iceberg" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "apache-avro 0.21.0", @@ -3405,7 +3405,7 @@ dependencies = [ [[package]] name = "iceberg-cache-moka" -version = "0.7.0" +version = "0.8.0" dependencies = [ "iceberg", "moka", @@ -3413,7 +3413,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-glue" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -3430,7 +3430,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-hms" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -3454,7 +3454,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-loader" -version = "0.7.0" +version = "0.8.0" dependencies = [ "async-trait", "iceberg", @@ -3470,7 +3470,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-rest" -version = "0.7.0" +version = "0.8.0" dependencies = [ "async-trait", "chrono", @@ -3493,7 +3493,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-s3tables" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -3507,7 +3507,7 @@ dependencies = [ [[package]] name = "iceberg-catalog-sql" -version = "0.7.0" +version = "0.8.0" dependencies = [ "async-trait", "iceberg", @@ -3521,7 +3521,7 @@ dependencies = [ [[package]] name = "iceberg-datafusion" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -3537,7 +3537,7 @@ dependencies = [ [[package]] name = "iceberg-examples" -version = "0.7.0" +version = "0.8.0" dependencies = [ "futures", "iceberg", @@ -3547,7 +3547,7 @@ dependencies = [ [[package]] name = "iceberg-integration-tests" -version = "0.7.0" +version = "0.8.0" dependencies = [ "arrow-array", "arrow-schema", @@ -3566,7 +3566,7 @@ dependencies = [ [[package]] name = "iceberg-playground" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "clap", @@ -3588,7 +3588,7 @@ dependencies = [ [[package]] name = "iceberg-sqllogictest" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -3610,7 +3610,7 @@ dependencies = [ [[package]] name = "iceberg_test_utils" -version = "0.7.0" +version = "0.8.0" dependencies = [ "tracing", "tracing-subscriber", diff --git a/Cargo.toml b/Cargo.toml index ded3aedecb..d099398dbd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -31,7 +31,7 @@ resolver = "2" [workspace.package] edition = "2024" homepage = "https://rust.iceberg.apache.org/" -version = "0.7.0" +version = "0.8.0" license = "Apache-2.0" repository = "https://github.com/apache/iceberg-rust" @@ -78,13 +78,13 @@ futures = "0.3" hive_metastore = "0.2.0" home = "=0.5.11" http = "1.2" -iceberg = { version = "0.7.0", path = "./crates/iceberg" } -iceberg-catalog-glue = { version = "0.7.0", path = "./crates/catalog/glue" } -iceberg-catalog-hms = { version = "0.7.0", path = "./crates/catalog/hms" } -iceberg-catalog-rest = { version = "0.7.0", path = "./crates/catalog/rest" } -iceberg-catalog-s3tables = { version = "0.7.0", path = "./crates/catalog/s3tables" } -iceberg-catalog-sql = { version = "0.7.0", path = "./crates/catalog/sql" } -iceberg-datafusion = { version = "0.7.0", path = "./crates/integrations/datafusion" } +iceberg = { version = "0.8.0", path = "./crates/iceberg" } +iceberg-catalog-glue = { version = "0.8.0", path = "./crates/catalog/glue" } +iceberg-catalog-hms = { version = "0.8.0", path = "./crates/catalog/hms" } +iceberg-catalog-rest = { version = "0.8.0", path = "./crates/catalog/rest" } +iceberg-catalog-s3tables = { version = "0.8.0", path = "./crates/catalog/s3tables" } +iceberg-catalog-sql = { version = "0.8.0", path = "./crates/catalog/sql" } +iceberg-datafusion = { version = "0.8.0", path = "./crates/integrations/datafusion" } indicatif = "0.18" itertools = "0.13" libtest-mimic = "0.8.1" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index a7244cc8dd..4647f9d886 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -128,9 +128,9 @@ checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" [[package]] name = "apache-avro" -version = "0.20.0" +version = "0.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a033b4ced7c585199fb78ef50fca7fe2f444369ec48080c5fd072efa1a03cc7" +checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf" dependencies = [ "bigdecimal", "bon", @@ -1034,8 +1034,9 @@ dependencies = [ [[package]] name = "datafusion" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ba7cb113e9c0bedf9e9765926031e132fa05a1b09ba6e93a6d1a4d7044457b8" dependencies = [ "arrow", "arrow-schema", @@ -1089,8 +1090,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66a3a799f914a59b1ea343906a0486f17061f39509af74e874a866428951130d" dependencies = [ "arrow", "async-trait", @@ -1113,8 +1115,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6db1b113c80d7a0febcd901476a57aef378e717c54517a163ed51417d87621b0" dependencies = [ "arrow", "async-trait", @@ -1136,8 +1139,9 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" dependencies = [ "ahash 0.8.12", "arrow", @@ -1159,8 +1163,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b92065bbc6532c6651e2f7dd30b55cba0c7a14f860c7e1d15f165c41a1868d95" dependencies = [ "futures", "log", @@ -1169,8 +1174,9 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fde13794244bc7581cd82f6fff217068ed79cdc344cafe4ab2c3a1c3510b38d6" dependencies = [ "arrow", "async-compression", @@ -1203,8 +1209,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "804fa9b4ecf3157982021770617200ef7c1b2979d57bec9044748314775a9aea" dependencies = [ "arrow", "arrow-ipc", @@ -1226,8 +1233,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61a1641a40b259bab38131c5e6f48fac0717bedb7dc93690e604142a849e0568" dependencies = [ "arrow", "async-trait", @@ -1248,8 +1256,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adeacdb00c1d37271176f8fb6a1d8ce096baba16ea7a4b2671840c5c9c64fe85" dependencies = [ "arrow", "async-trait", @@ -1269,8 +1278,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43d0b60ffd66f28bfb026565d62b0a6cbc416da09814766a3797bba7d85a3cd9" dependencies = [ "arrow", "async-trait", @@ -1298,13 +1308,15 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b99e13947667b36ad713549237362afb054b2d8f8cc447751e23ec61202db07" [[package]] name = "datafusion-execution" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63695643190679037bc946ad46a263b62016931547bf119859c511f7ff2f5178" dependencies = [ "arrow", "async-trait", @@ -1322,8 +1334,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9a4787cbf5feb1ab351f789063398f67654a6df75c4d37d7f637dc96f951a91" dependencies = [ "arrow", "async-trait", @@ -1344,8 +1357,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ce2fb1b8c15c9ac45b0863c30b268c69dc9ee7a1ee13ecf5d067738338173dc" dependencies = [ "arrow", "datafusion-common", @@ -1356,8 +1370,9 @@ dependencies = [ [[package]] name = "datafusion-ffi" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec510e7787641279b0336e8b79e4b7bd1385d5976875ff9b97f4269ce5231a67" dependencies = [ "abi_stable", "arrow", @@ -1378,8 +1393,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "794a9db7f7b96b3346fc007ff25e994f09b8f0511b4cf7dff651fadfe3ebb28f" dependencies = [ "arrow", "arrow-buffer", @@ -1407,8 +1423,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" dependencies = [ "ahash 0.8.12", "arrow", @@ -1427,8 +1444,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" dependencies = [ "ahash 0.8.12", "arrow", @@ -1439,8 +1457,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae5c06eed03918dc7fe7a9f082a284050f0e9ecf95d72f57712d1496da03b8c4" dependencies = [ "arrow", "arrow-ord", @@ -1461,8 +1480,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db4fed1d71738fbe22e2712d71396db04c25de4111f1ec252b8f4c6d3b25d7f5" dependencies = [ "arrow", "async-trait", @@ -1476,8 +1496,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d92206aa5ae21892f1552b4d61758a862a70956e6fd7a95cb85db1de74bc6d1" dependencies = [ "arrow", "datafusion-common", @@ -1493,8 +1514,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53ae9bcc39800820d53a22d758b3b8726ff84a5a3e24cecef04ef4e5fdf1c7cc" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1502,8 +1524,9 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1063ad4c9e094b3f798acee16d9a47bd7372d9699be2de21b05c3bd3f34ab848" dependencies = [ "datafusion-doc", "quote", @@ -1512,8 +1535,9 @@ dependencies = [ [[package]] name = "datafusion-optimizer" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f35f9ec5d08b87fd1893a30c2929f2559c2f9806ca072d8fefca5009dc0f06a" dependencies = [ "arrow", "chrono", @@ -1531,8 +1555,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" dependencies = [ "ahash 0.8.12", "arrow", @@ -1552,8 +1577,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f9ff2dbd476221b1f67337699eff432781c4e6e1713d2aefdaa517dfbf79768" dependencies = [ "arrow", "datafusion-common", @@ -1566,8 +1592,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" dependencies = [ "ahash 0.8.12", "arrow", @@ -1579,8 +1606,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce9804f799acd7daef3be7aaffe77c0033768ed8fdbf5fb82fc4c5f2e6bc14e6" dependencies = [ "arrow", "datafusion-common", @@ -1597,8 +1625,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" dependencies = [ "ahash 0.8.12", "arrow", @@ -1627,8 +1656,9 @@ dependencies = [ [[package]] name = "datafusion-proto" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d368093a98a17d1449b1083ac22ed16b7128e4c67789991869480d8c4a40ecb9" dependencies = [ "arrow", "chrono", @@ -1653,8 +1683,9 @@ dependencies = [ [[package]] name = "datafusion-proto-common" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b6aef3d5e5c1d2bc3114c4876730cb76a9bdc5a8df31ef1b6db48f0c1671895" dependencies = [ "arrow", "datafusion-common", @@ -1663,8 +1694,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac2c2498a1f134a9e11a9f5ed202a2a7d7e9774bd9249295593053ea3be999db" dependencies = [ "arrow", "datafusion-common", @@ -1679,8 +1711,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f96eebd17555386f459037c65ab73aae8df09f464524c709d6a3134ad4f4776" dependencies = [ "async-trait", "datafusion-common", @@ -1692,8 +1725,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "50.3.0" -source = "git+https://github.com/apache/datafusion.git?rev=f32984b2dbf9e5a193c20643ce624167295fbd61#f32984b2dbf9e5a193c20643ce624167295fbd61" +version = "51.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3fc195fe60634b2c6ccfd131b487de46dc30eccae8a3c35a13f136e7f440414f" dependencies = [ "arrow", "bigdecimal", @@ -2122,12 +2156,6 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" -[[package]] -name = "hermit-abi" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" - [[package]] name = "hex" version = "0.4.3" @@ -2286,7 +2314,7 @@ dependencies = [ [[package]] name = "iceberg" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "apache-avro", @@ -2331,7 +2359,6 @@ dependencies = [ "serde_repr", "serde_with", "strum 0.27.2", - "thrift", "tokio", "typed-builder", "url", @@ -2341,7 +2368,7 @@ dependencies = [ [[package]] name = "iceberg-datafusion" -version = "0.7.0" +version = "0.8.0" dependencies = [ "anyhow", "async-trait", @@ -2539,6 +2566,47 @@ version = "1.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c" +[[package]] +name = "jiff" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49cce2b81f2098e7e3efc35bc2e0a6b7abec9d34128283d7a26fa8f32a6dbb35" +dependencies = [ + "jiff-static", + "jiff-tzdb-platform", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "windows-sys 0.61.2", +] + +[[package]] +name = "jiff-static" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "980af8b43c3ad5d8d349ace167ec8170839f753a42d233ba19e08afe1850fa69" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68971ebff725b9e2ca27a601c5eb38a4c5d64422c4cbab0c535f248087eda5c2" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2824,16 +2892,6 @@ dependencies = [ "libm", ] -[[package]] -name = "num_cpus" -version = "1.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" -dependencies = [ - "hermit-abi", - "libc", -] - [[package]] name = "object" version = "0.32.2" @@ -2875,20 +2933,20 @@ checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" [[package]] name = "opendal" -version = "0.54.1" +version = "0.55.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42afda58fa2cf50914402d132cc1caacff116a85d10c72ab2082bb7c50021754" +checksum = "d075ab8a203a6ab4bc1bce0a4b9fe486a72bf8b939037f4b78d95386384bc80a" dependencies = [ "anyhow", "backon", "base64", "bytes", - "chrono", "crc32c", "futures", "getrandom 0.2.16", "http", "http-body", + "jiff", "log", "md-5", "percent-encoding", @@ -2898,6 +2956,7 @@ dependencies = [ "serde", "serde_json", "tokio", + "url", "uuid", ] @@ -3061,6 +3120,15 @@ version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f84267b20a16ea918e43c6a88433c2d54fa145c92a811b5b047ccbe153674483" +[[package]] +name = "portable-atomic-util" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8a2f0d8d040d7848a709caf78912debcc3f33ee4b3cac47d73d1e1069e83507" +dependencies = [ + "portable-atomic", +] + [[package]] name = "potential_utf" version = "0.1.4" @@ -3168,7 +3236,7 @@ dependencies = [ [[package]] name = "pyiceberg_core_rust" -version = "0.7.0" +version = "0.8.0" dependencies = [ "arrow", "datafusion-ffi", @@ -4185,15 +4253,6 @@ dependencies = [ "syn 2.0.108", ] -[[package]] -name = "threadpool" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d050e60b33d41c19108b32cea32164033a9013fe3b46cbd4457559bfbf77afaa" -dependencies = [ - "num_cpus", -] - [[package]] name = "thrift" version = "0.17.0" @@ -4202,9 +4261,7 @@ checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" dependencies = [ "byteorder", "integer-encoding", - "log", "ordered-float 2.10.1", - "threadpool", ] [[package]] diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index d8b8444195..9ec58cf807 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -20,7 +20,7 @@ edition = "2024" homepage = "https://rust.iceberg.apache.org" name = "pyiceberg_core_rust" rust-version = "1.88" -version = "0.7.0" +version = "0.8.0" # This crate is used to build python bindings, we don't want to publish it publish = false diff --git a/bindings/python/DEPENDENCIES.rust.tsv b/bindings/python/DEPENDENCIES.rust.tsv index 7565a13e1d..5fe0da5b2a 100644 --- a/bindings/python/DEPENDENCIES.rust.tsv +++ b/bindings/python/DEPENDENCIES.rust.tsv @@ -1,445 +1,439 @@ -crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 MPL-2.0 Unicode-3.0 Unlicense Zlib -abi_stable@0.11.3 X X -abi_stable_derive@0.11.3 X X -abi_stable_shared@0.11.0 X X -addr2line@0.24.2 X X -adler2@2.0.1 X X X -ahash@0.8.12 X X -aho-corasick@1.1.3 X X -alloc-no-stdlib@2.0.4 X -alloc-stdlib@0.2.2 X -allocator-api2@0.2.21 X X -android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X -array-init@2.1.0 X X -arrayref@0.3.9 X -arrayvec@0.7.6 X X -arrow@55.2.0 X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-csv@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-json@55.2.0 X -arrow-ord@55.2.0 X -arrow-pyarrow@55.2.0 X -arrow-row@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X -as-any@0.3.2 X X -as_derive_utils@0.11.0 X X -async-compression@0.4.19 X X -async-ffi@0.5.0 X -async-lock@3.4.1 X X -async-trait@0.1.89 X X -atoi@2.0.0 X -atomic-waker@1.1.2 X X -autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.22.1 X X -bigdecimal@0.4.8 X X -bimap@0.6.3 X X -bitflags@2.9.4 X X -blake2@0.10.6 X X -blake3@1.8.2 X X X -block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X -brotli@8.0.2 X X -brotli-decompressor@5.0.0 X X -bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X -byteorder@1.5.0 X X -bytes@1.10.1 X -bzip2@0.5.2 X X -bzip2-sys@0.1.13+1.0.8 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X -chrono@0.4.42 X X -chrono-tz@0.10.4 X X -comfy-table@7.2.0 X -concurrent-queue@2.5.0 X X -const-oid@0.9.6 X X -const-random@0.1.18 X X -const-random-macro@0.1.16 X X -const_panic@0.2.14 X -constant_time_eq@0.3.1 X X X -core-foundation-sys@0.8.7 X X -core_extensions@1.5.4 X X -core_extensions_proc_macros@1.5.4 X X -cpufeatures@0.2.17 X X -crc32c@0.6.8 X X -crc32fast@1.5.0 X X -crossbeam-channel@0.5.15 X X -crossbeam-epoch@0.9.18 X X -crossbeam-utils@0.8.21 X X -crunchy@0.2.4 X -crypto-common@0.1.6 X X -csv@1.3.1 X X -csv-core@0.1.12 X X -darling@0.20.11 X -darling@0.21.3 X -darling_core@0.20.11 X -darling_core@0.21.3 X -darling_macro@0.20.11 X -darling_macro@0.21.3 X -dashmap@6.1.0 X -datafusion@48.0.1 X -datafusion-catalog@48.0.1 X -datafusion-catalog-listing@48.0.1 X -datafusion-common@48.0.1 X -datafusion-common-runtime@48.0.1 X -datafusion-datasource@48.0.1 X -datafusion-datasource-csv@48.0.1 X -datafusion-datasource-json@48.0.1 X -datafusion-datasource-parquet@48.0.1 X -datafusion-doc@48.0.1 X -datafusion-execution@48.0.1 X -datafusion-expr@48.0.1 X -datafusion-expr-common@48.0.1 X -datafusion-ffi@48.0.1 X -datafusion-functions@48.0.1 X -datafusion-functions-aggregate@48.0.1 X -datafusion-functions-aggregate-common@48.0.1 X -datafusion-functions-nested@48.0.1 X -datafusion-functions-table@48.0.1 X -datafusion-functions-window@48.0.1 X -datafusion-functions-window-common@48.0.1 X -datafusion-macros@48.0.1 X -datafusion-optimizer@48.0.1 X -datafusion-physical-expr@48.0.1 X -datafusion-physical-expr-common@48.0.1 X -datafusion-physical-optimizer@48.0.1 X -datafusion-physical-plan@48.0.1 X -datafusion-proto@48.0.1 X -datafusion-proto-common@48.0.1 X -datafusion-session@48.0.1 X -datafusion-sql@48.0.1 X -derive_builder@0.20.2 X X -derive_builder_core@0.20.2 X X -derive_builder_macro@0.20.2 X X -digest@0.10.7 X X -displaydoc@0.2.5 X X -dissimilar@1.0.10 X -either@1.15.0 X X -equivalent@1.0.2 X X -errno@0.3.13 X X -event-listener@5.4.1 X X -event-listener-strategy@0.5.4 X X -expect-test@1.5.1 X X -fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -fixedbitset@0.5.7 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X -fnv@1.0.7 X X -foldhash@0.1.5 X -form_urlencoded@1.2.2 X X -futures@0.3.31 X X -futures-channel@0.3.31 X X -futures-core@0.3.31 X X -futures-executor@0.3.31 X X -futures-io@0.3.31 X X -futures-macro@0.3.31 X X -futures-sink@0.3.31 X X -futures-task@0.3.31 X X -futures-util@0.3.31 X X -generational-arena@0.2.9 X -generator@0.8.7 X X -generic-array@0.14.7 X -getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X -glob@0.3.3 X X -gloo-timers@0.3.0 X X -half@2.6.0 X X -hashbrown@0.14.5 X X -hashbrown@0.15.5 X X -heck@0.5.0 X X -hermit-abi@0.5.2 X X -hex@0.4.3 X X -hmac@0.12.1 X X -home@0.5.11 X X -http@1.3.1 X X -http-body@1.0.1 X -http-body-util@0.1.3 X -httparse@1.10.1 X X -humantime@2.2.0 X X -hyper@1.7.0 X -hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X -iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-datafusion@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X -ident_case@1.0.1 X X -idna@1.1.0 X X -idna_adapter@1.2.1 X X -indexmap@2.11.0 X X -indoc@2.0.6 X X -integer-encoding@3.0.4 X -io-uring@0.7.10 X X -ipnet@2.11.0 X X -iri-string@0.7.8 X X -itertools@0.13.0 X X -itertools@0.14.0 X X -itoa@1.0.15 X X -jobserver@0.1.34 X X -js-sys@0.3.78 X X -lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X -libloading@0.7.4 X -libm@0.2.15 X -libz-rs-sys@0.5.2 X -linux-raw-sys@0.9.4 X X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X -matchers@0.2.0 X -md-5@0.10.6 X X -memchr@2.7.5 X X -memoffset@0.9.1 X -miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X -murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X -num-bigint@0.4.6 X X -num-complex@0.4.6 X X -num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X -num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X -object_store@0.12.3 X X -once_cell@1.21.3 X X -opendal@0.54.0 X -ordered-float@2.10.1 X -ordered-float@4.6.0 X -parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X -paste@1.0.15 X X -percent-encoding@2.3.2 X X -petgraph@0.8.2 X X -phf@0.12.1 X -phf_shared@0.12.1 X -pin-project-lite@0.2.16 X X -pin-utils@0.1.0 X X -pkg-config@0.3.32 X X -portable-atomic@1.11.1 X X -potential_utf@0.1.3 X -ppv-lite86@0.2.21 X X -prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X -prost@0.13.5 X -prost-derive@0.13.5 X -psm@0.1.26 X X -pyiceberg_core_rust@0.7.0 X -pyo3@0.24.2 X X -pyo3-build-config@0.24.2 X X -pyo3-ffi@0.24.2 X X -pyo3-macros@0.24.2 X X -pyo3-macros-backend@0.24.2 X X -quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X -r-efi@5.3.0 X X X -rand@0.8.5 X X -rand@0.9.2 X X -rand_chacha@0.3.1 X X -rand_chacha@0.9.0 X X -rand_core@0.6.4 X X -rand_core@0.9.3 X X -recursive@0.1.1 X -recursive-proc-macro-impl@0.1.1 X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X -repr_offset@0.2.2 X -reqsign@0.16.5 X -reqwest@0.12.23 X X -ring@0.17.14 X X -roaring@0.11.2 X X -rust_decimal@1.37.2 X -rustc-demangle@0.1.26 X X -rustc_version@0.4.1 X X -rustix@1.0.8 X X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X -rustversion@1.0.22 X X -ryu@1.0.20 X X -same-file@1.0.6 X X -scoped-tls@1.0.1 X X -scopeguard@1.2.0 X X -semver@1.0.26 X X -seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X -serde_repr@0.1.20 X X -serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X -sha1@0.10.6 X X -sha2@0.10.9 X X -sharded-slab@0.1.7 X -shlex@1.3.0 X X -simdutf8@0.1.5 X X -siphasher@1.0.1 X X -slab@0.4.11 X -smallvec@1.15.1 X X -snap@1.1.1 X -socket2@0.6.0 X X -sqlparser@0.55.0 X -sqlparser_derive@0.3.0 X -stable_deref_trait@1.2.0 X X -stacker@0.1.21 X X -static_assertions@1.1.0 X X -strsim@0.11.1 X -strum@0.27.2 X -strum_macros@0.27.2 X -subtle@2.6.1 X -syn@1.0.109 X X -syn@2.0.106 X X -sync_wrapper@1.0.2 X -synstructure@0.13.2 X -tagptr@0.2.0 X X -target-lexicon@0.13.2 X -tempfile@3.21.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X -thread_local@1.1.9 X X -threadpool@1.8.1 X X -thrift@0.17.0 X -tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X -tower@0.5.2 X -tower-http@0.6.6 X -tower-layer@0.3.3 X -tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X -tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X -try-lock@0.2.5 X -tstr@0.2.4 X -tstr_proc_macros@0.2.2 X -twox-hash@2.1.2 X -typed-arena@2.0.2 X -typed-builder@0.20.1 X X -typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -typewit@1.14.1 X -unicode-ident@1.0.18 X X X -unicode-segmentation@1.12.0 X X -unicode-width@0.2.1 X X -unindent@0.2.4 X X -untrusted@0.9.0 X -url@2.5.7 X X -utf8_iter@1.0.4 X X -uuid@1.18.1 X X -version_check@0.9.5 X X -walkdir@2.5.0 X X -want@0.3.1 X -wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X -wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -web-time@1.1.0 X X -webpki-roots@1.0.2 X -winapi@0.3.9 X X -winapi-i686-pc-windows-gnu@0.4.0 X X -winapi-util@0.1.11 X X -winapi-x86_64-pc-windows-gnu@0.4.0 X X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X -windows-sys@0.52.0 X X -windows-sys@0.59.0 X X -windows-sys@0.60.2 X X -windows-sys@0.61.0 X X -windows-targets@0.52.6 X X -windows-targets@0.53.3 X X -windows-threading@0.1.0 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_gnullvm@0.53.0 X X -windows_aarch64_msvc@0.52.6 X X -windows_aarch64_msvc@0.53.0 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnu@0.53.0 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_gnullvm@0.53.0 X X -windows_i686_msvc@0.52.6 X X -windows_i686_msvc@0.53.0 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnu@0.53.0 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_gnullvm@0.53.0 X X -windows_x86_64_msvc@0.52.6 X X -windows_x86_64_msvc@0.53.0 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -xz2@0.1.7 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X -zerofrom@0.1.6 X -zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X -zstd@0.13.3 X -zstd-safe@7.2.4 X X -zstd-sys@2.0.16+zstd.1.5.7 X X +crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 MPL-2.0 Unicode-3.0 Unlicense Zlib bzip2-1.0.6 +abi_stable@0.11.3 X X +abi_stable_derive@0.11.3 X X +abi_stable_shared@0.11.0 X X +adler2@2.0.1 X X X +ahash@0.8.12 X X +aho-corasick@1.1.3 X X +alloc-no-stdlib@2.0.4 X +alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X +android_system_properties@0.1.5 X X +anyhow@1.0.100 X X +apache-avro@0.21.0 X +ar_archive_writer@0.2.0 X +array-init@2.1.0 X X +arrayref@0.3.9 X +arrayvec@0.7.6 X X +arrow@57.0.0 X +arrow-arith@57.0.0 X +arrow-array@57.0.0 X +arrow-buffer@57.0.0 X +arrow-cast@57.0.0 X +arrow-csv@57.0.0 X +arrow-data@57.0.0 X +arrow-ipc@57.0.0 X +arrow-json@57.0.0 X +arrow-ord@57.0.0 X +arrow-pyarrow@57.0.0 X +arrow-row@57.0.0 X +arrow-schema@57.0.0 X +arrow-select@57.0.0 X +arrow-string@57.0.0 X +as-any@0.3.2 X X +as_derive_utils@0.11.0 X X +async-compression@0.4.19 X X +async-ffi@0.5.0 X +async-lock@3.4.1 X X +async-trait@0.1.89 X X +atoi@2.0.0 X +atomic-waker@1.1.2 X X +autocfg@1.5.0 X X +backon@1.6.0 X +base64@0.22.1 X X +bigdecimal@0.4.9 X X +bimap@0.6.3 X X +bitflags@2.10.0 X X +blake2@0.10.6 X X +blake3@1.8.2 X X X +block-buffer@0.10.4 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X +brotli@8.0.2 X X +brotli-decompressor@5.0.0 X X +bumpalo@3.19.0 X X +bytemuck@1.24.0 X X X +byteorder@1.5.0 X X +bytes@1.10.1 X +bzip2@0.5.2 X X +bzip2@0.6.1 X X +bzip2-sys@0.1.13+1.0.8 X X +cc@1.2.43 X X +cfg-if@1.0.4 X X +chrono@0.4.42 X X +chrono-tz@0.10.4 X X +comfy-table@7.1.2 X +concurrent-queue@2.5.0 X X +const-oid@0.9.6 X X +const-random@0.1.18 X X +const-random-macro@0.1.16 X X +const_panic@0.2.15 X +constant_time_eq@0.3.1 X X X +core-foundation-sys@0.8.7 X X +core_extensions@1.5.4 X X +core_extensions_proc_macros@1.5.4 X X +cpufeatures@0.2.17 X X +crc32c@0.6.8 X X +crc32fast@1.5.0 X X +crossbeam-channel@0.5.15 X X +crossbeam-epoch@0.9.18 X X +crossbeam-utils@0.8.21 X X +crunchy@0.2.4 X +crypto-common@0.1.6 X X +csv@1.4.0 X X +csv-core@0.1.13 X X +darling@0.20.11 X +darling@0.21.3 X +darling_core@0.20.11 X +darling_core@0.21.3 X +darling_macro@0.20.11 X +darling_macro@0.21.3 X +dashmap@6.1.0 X +datafusion@51.0.0 X +datafusion-catalog@51.0.0 X +datafusion-catalog-listing@51.0.0 X +datafusion-common@51.0.0 X +datafusion-common-runtime@51.0.0 X +datafusion-datasource@51.0.0 X +datafusion-datasource-arrow@51.0.0 X +datafusion-datasource-csv@51.0.0 X +datafusion-datasource-json@51.0.0 X +datafusion-datasource-parquet@51.0.0 X +datafusion-doc@51.0.0 X +datafusion-execution@51.0.0 X +datafusion-expr@51.0.0 X +datafusion-expr-common@51.0.0 X +datafusion-ffi@51.0.0 X +datafusion-functions@51.0.0 X +datafusion-functions-aggregate@51.0.0 X +datafusion-functions-aggregate-common@51.0.0 X +datafusion-functions-nested@51.0.0 X +datafusion-functions-table@51.0.0 X +datafusion-functions-window@51.0.0 X +datafusion-functions-window-common@51.0.0 X +datafusion-macros@51.0.0 X +datafusion-optimizer@51.0.0 X +datafusion-physical-expr@51.0.0 X +datafusion-physical-expr-adapter@51.0.0 X +datafusion-physical-expr-common@51.0.0 X +datafusion-physical-optimizer@51.0.0 X +datafusion-physical-plan@51.0.0 X +datafusion-proto@51.0.0 X +datafusion-proto-common@51.0.0 X +datafusion-pruning@51.0.0 X +datafusion-session@51.0.0 X +datafusion-sql@51.0.0 X +derive_builder@0.20.2 X X +derive_builder_core@0.20.2 X X +derive_builder_macro@0.20.2 X X +digest@0.10.7 X X +displaydoc@0.2.5 X X +dissimilar@1.0.10 X +either@1.15.0 X X +equivalent@1.0.2 X X +errno@0.3.14 X X +event-listener@5.4.1 X X +event-listener-strategy@0.5.4 X X +expect-test@1.5.1 X X +fastrand@2.3.0 X X +find-msvc-tools@0.1.4 X X +fixedbitset@0.5.7 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +fnv@1.0.7 X X +foldhash@0.1.5 X +form_urlencoded@1.2.2 X X +futures@0.3.31 X X +futures-channel@0.3.31 X X +futures-core@0.3.31 X X +futures-executor@0.3.31 X X +futures-io@0.3.31 X X +futures-macro@0.3.31 X X +futures-sink@0.3.31 X X +futures-task@0.3.31 X X +futures-timer@3.0.3 X X +futures-util@0.3.31 X X +generational-arena@0.2.9 X +generic-array@0.14.9 X +getrandom@0.2.16 X X +getrandom@0.3.4 X X +glob@0.3.3 X X +gloo-timers@0.3.0 X X +half@2.7.1 X X +hashbrown@0.14.5 X X +hashbrown@0.15.5 X X +hashbrown@0.16.0 X X +heck@0.5.0 X X +hex@0.4.3 X X +hmac@0.12.1 X X +home@0.5.11 X X +http@1.3.1 X X +http-body@1.0.1 X +http-body-util@0.1.3 X +httparse@1.10.1 X X +humantime@2.3.0 X X +hyper@1.7.0 X +hyper-rustls@0.27.7 X X X +hyper-util@0.1.17 X +iana-time-zone@0.1.64 X X +iana-time-zone-haiku@0.1.2 X X +iceberg@0.8.0 X +iceberg-datafusion@0.8.0 X +icu_collections@2.1.0 X +icu_locale_core@2.1.0 X +icu_normalizer@2.1.0 X +icu_normalizer_data@2.1.0 X +icu_properties@2.1.0 X +icu_properties_data@2.1.0 X +icu_provider@2.1.0 X +ident_case@1.0.1 X X +idna@1.1.0 X X +idna_adapter@1.2.1 X X +indexmap@2.12.0 X X +indoc@2.0.7 X X +integer-encoding@3.0.4 X +ipnet@2.11.0 X X +iri-string@0.7.8 X X +itertools@0.13.0 X X +itertools@0.14.0 X X +itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.5 X X +jiff-tzdb-platform@0.1.3 X X +jobserver@0.1.34 X X +js-sys@0.3.82 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libbz2-rs-sys@0.2.2 X +libc@0.2.177 X X +libloading@0.7.4 X +libm@0.2.15 X +libz-rs-sys@0.5.2 X +linux-raw-sys@0.11.0 X X X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.28 X X +lz4_flex@0.11.5 X +lzma-sys@0.1.20 X X +md-5@0.10.6 X X +memchr@2.7.6 X X +memoffset@0.9.1 X +miniz_oxide@0.8.9 X X X +mio@1.1.0 X +moka@0.12.11 X X +murmur3@0.5.2 X X +num-bigint@0.4.6 X X +num-complex@0.4.6 X X +num-integer@0.1.46 X X +num-traits@0.2.19 X X +object@0.32.2 X X +object_store@0.12.4 X X +once_cell@1.21.3 X X +opendal@0.55.0 X +ordered-float@2.10.1 X +ordered-float@4.6.0 X +parking@2.2.1 X X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.0.0 X +paste@1.0.15 X X +percent-encoding@2.3.2 X X +petgraph@0.8.3 X X +phf@0.12.1 X +phf_shared@0.12.1 X +pin-project-lite@0.2.16 X X +pin-utils@0.1.0 X X +pkg-config@0.3.32 X X +portable-atomic@1.11.1 X X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X +ppv-lite86@0.2.21 X X +prettyplease@0.2.37 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X +prost@0.14.1 X +prost-derive@0.14.1 X +psm@0.1.28 X X +pyiceberg_core_rust@0.8.0 X +pyo3@0.26.0 X X +pyo3-build-config@0.26.0 X X +pyo3-ffi@0.26.0 X X +pyo3-macros@0.26.0 X X +pyo3-macros-backend@0.26.0 X X +quad-rand@0.2.3 X +quick-xml@0.38.3 X +quote@1.0.41 X X +r-efi@5.3.0 X X X +rand@0.8.5 X X +rand@0.9.2 X X +rand_chacha@0.3.1 X X +rand_chacha@0.9.0 X X +rand_core@0.6.4 X X +rand_core@0.9.3 X X +recursive@0.1.1 X +recursive-proc-macro-impl@0.1.1 X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X +relative-path@1.9.3 X X +repr_offset@0.2.2 X +reqsign@0.16.5 X +reqwest@0.12.24 X X +ring@0.17.14 X X +roaring@0.11.2 X X +rstest@0.26.1 X X +rstest_macros@0.26.1 X X +rust_decimal@1.39.0 X +rustc_version@0.4.1 X X +rustix@1.1.2 X X X +rustls@0.23.34 X X X +rustls-pki-types@1.13.0 X X +rustls-webpki@0.103.7 X +rustversion@1.0.22 X X +ryu@1.0.20 X X +same-file@1.0.6 X X +scopeguard@1.2.0 X X +semver@1.0.27 X X +seq-macro@0.3.6 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X +serde_repr@0.1.20 X X +serde_urlencoded@0.7.1 X X +serde_with@3.15.1 X X +serde_with_macros@3.15.1 X X +sha1@0.10.6 X X +sha2@0.10.9 X X +shlex@1.3.0 X X +simd-adler32@0.3.7 X +simdutf8@0.1.5 X X +siphasher@1.0.1 X X +slab@0.4.11 X +smallvec@1.15.1 X X +snap@1.1.1 X +socket2@0.6.1 X X +sqlparser@0.59.0 X +sqlparser_derive@0.3.0 X +stable_deref_trait@1.2.1 X X +stacker@0.1.22 X X +strsim@0.11.1 X +strum@0.26.3 X +strum@0.27.2 X +strum_macros@0.26.4 X +strum_macros@0.27.2 X +subtle@2.6.1 X +syn@1.0.109 X X +syn@2.0.108 X X +sync_wrapper@1.0.2 X +synstructure@0.13.2 X +tagptr@0.2.0 X X +target-lexicon@0.13.3 X +tempfile@3.23.0 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X +thrift@0.17.0 X +tiny-keccak@2.0.2 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.16 X +toml_datetime@0.7.3 X X +toml_edit@0.23.7 X X +toml_parser@1.0.4 X X +tower@0.5.2 X +tower-http@0.6.6 X +tower-layer@0.3.3 X +tower-service@0.3.3 X +tracing@0.1.41 X +tracing-attributes@0.1.30 X +tracing-core@0.1.34 X +try-lock@0.2.5 X +tstr@0.2.4 X +tstr_proc_macros@0.2.2 X +twox-hash@2.1.2 X +typed-arena@2.0.2 X +typed-builder@0.20.1 X X +typed-builder-macro@0.20.1 X X +typenum@1.19.0 X X +typewit@1.14.2 X +unicode-ident@1.0.20 X X X +unicode-segmentation@1.12.0 X X +unicode-width@0.2.2 X X +unindent@0.2.4 X X +untrusted@0.9.0 X +url@2.5.7 X X +utf8_iter@1.0.4 X X +uuid@1.18.1 X X +version_check@0.9.5 X X +walkdir@2.5.0 X X +want@0.3.1 X +wasi@0.11.1+wasi-snapshot-preview1 X X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.105 X X +wasm-bindgen-futures@0.4.55 X X +wasm-bindgen-macro@0.2.105 X X +wasm-bindgen-macro-support@0.2.105 X X +wasm-bindgen-shared@0.2.105 X X +wasm-streams@0.4.2 X X +web-sys@0.3.82 X X +web-time@1.1.0 X X +webpki-roots@1.0.3 X +winapi@0.3.9 X X +winapi-i686-pc-windows-gnu@0.4.0 X X +winapi-util@0.1.11 X X +winapi-x86_64-pc-windows-gnu@0.4.0 X X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X +windows-sys@0.52.0 X X +windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X +windows-targets@0.52.6 X X +windows-targets@0.53.5 X X +windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X +windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X +windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X +windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X +windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X +windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X +windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X +windows_x86_64_msvc@0.52.6 X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.13 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +xz2@0.1.7 X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.27 X X X +zerocopy-derive@0.8.27 X X X +zerofrom@0.1.6 X +zerofrom-derive@0.1.6 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.2 X +zstd@0.13.3 X +zstd-safe@7.2.4 X X +zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/glue/DEPENDENCIES.rust.tsv b/crates/catalog/glue/DEPENDENCIES.rust.tsv index 2d9f686262..e34ff1afc1 100644 --- a/crates/catalog/glue/DEPENDENCIES.rust.tsv +++ b/crates/catalog/glue/DEPENDENCIES.rust.tsv @@ -1,77 +1,73 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -aws-config@1.8.6 X -aws-credential-types@1.2.6 X -aws-runtime@1.5.10 X -aws-sdk-glue@1.119.0 X -aws-sdk-sso@1.83.0 X -aws-sdk-ssooidc@1.84.0 X -aws-sdk-sts@1.85.0 X -aws-sigv4@1.3.4 X -aws-smithy-async@1.2.5 X -aws-smithy-http@0.62.3 X -aws-smithy-http-client@1.1.1 X -aws-smithy-json@0.61.5 X -aws-smithy-observability@0.1.3 X -aws-smithy-query@0.60.7 X -aws-smithy-runtime@1.9.1 X -aws-smithy-runtime-api@1.9.0 X -aws-smithy-types@1.3.2 X -aws-smithy-xml@0.60.10 X -aws-types@1.3.8 X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.21.7 X X +aws-config@1.8.11 X +aws-credential-types@1.2.10 X +aws-runtime@1.5.16 X +aws-sdk-glue@1.132.0 X +aws-sdk-sso@1.90.0 X +aws-sdk-ssooidc@1.92.0 X +aws-sdk-sts@1.94.0 X +aws-sigv4@1.3.6 X +aws-smithy-async@1.2.7 X +aws-smithy-http@0.62.6 X +aws-smithy-http-client@1.1.5 X +aws-smithy-json@0.61.8 X +aws-smithy-observability@0.1.5 X +aws-smithy-query@0.60.9 X +aws-smithy-runtime@1.9.5 X +aws-smithy-runtime-api@1.9.3 X +aws-smithy-types@1.3.5 X +aws-smithy-xml@0.60.13 X +aws-types@1.3.10 X +backon@1.6.0 X base64@0.22.1 X X base64-simd@0.8.0 X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X +bytes@1.11.0 X bytes-utils@0.1.4 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X const-random@0.1.18 X X const-random-macro@0.1.16 X X core-foundation@0.10.1 X X -core-foundation@0.9.4 X X core-foundation-sys@0.8.7 X X cpufeatures@0.2.17 X X crc32c@0.6.8 X X @@ -80,14 +76,14 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X darling_core@0.21.3 X darling_macro@0.20.11 X darling_macro@0.21.3 X -deranged@0.5.3 X X +deranged@0.5.5 X X derive_builder@0.20.2 X X derive_builder_core@0.20.2 X X derive_builder_macro@0.20.2 X X @@ -100,9 +96,9 @@ event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -114,114 +110,107 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X h2@0.3.27 X h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X http@0.2.12 X X -http@1.3.1 X X +http@1.4.0 X X http-body@0.4.6 X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X httpdate@1.0.3 X X hyper@0.14.32 X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.24.2 X X X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-glue@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-glue@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-conv@0.1.0 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X openssl-probe@0.1.6 X X ordered-float@2.10.1 X ordered-float@4.6.0 X outref@0.5.2 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X powerfmt@0.2.0 X X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -229,152 +218,150 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X rustls@0.21.12 X X X -rustls@0.23.31 X X X -rustls-native-certs@0.6.3 X X X -rustls-native-certs@0.8.1 X X X -rustls-pemfile@1.0.4 X X X -rustls-pki-types@1.12.0 X X +rustls@0.23.35 X X X +rustls-native-certs@0.8.2 X X X +rustls-pki-types@1.13.1 X X rustls-webpki@0.101.7 X -rustls-webpki@0.103.4 X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -schannel@0.1.27 X -scoped-tls@1.0.1 X X +schannel@0.1.28 X scopeguard@1.2.0 X X sct@0.7.1 X X X -security-framework@2.11.1 X X -security-framework@3.4.0 X X +security-framework@3.5.1 X X security-framework-sys@2.15.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X socket2@0.5.10 X X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X -time@0.3.43 X X +time@0.3.44 X X time-core@0.1.6 X X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X tokio-rustls@0.24.1 X X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X urlencoding@2.1.3 X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X vsimd@0.8.0 X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X xmlparser@0.13.6 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/hms/DEPENDENCIES.rust.tsv b/crates/catalog/hms/DEPENDENCIES.rust.tsv index cef38cabcd..5025f7b184 100644 --- a/crates/catalog/hms/DEPENDENCIES.rust.tsv +++ b/crates/catalog/hms/DEPENDENCIES.rust.tsv @@ -1,25 +1,24 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-broadcast@0.7.2 X X async-lock@3.4.1 X X @@ -28,23 +27,22 @@ async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X cfg_aliases@0.2.1 X chrono@0.4.42 X X concurrent-queue@2.5.0 X X @@ -59,7 +57,7 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -80,9 +78,9 @@ event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X faststr@0.2.32 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -94,103 +92,95 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X -half@2.6.0 X X +half@2.7.1 X X hashbrown@0.14.5 X X -hashbrown@0.15.5 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hive_metastore@0.2.0 X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-hms@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-hms@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -integer-encoding@4.0.2 X -io-uring@0.7.10 X X +integer-encoding@4.1.0 X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X +libz-rs-sys@0.5.3 X linked-hash-map@0.5.6 X X linkedbytes@0.1.16 X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X memoffset@0.9.1 X metainfo@0.7.14 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X motore@0.4.1 X X motore-macros@0.4.3 X X mur3@0.1.0 X murmur3@0.5.2 X X nix@0.29.0 X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -num_enum@0.7.4 X X X -num_enum_derive@0.7.4 X X X -object@0.36.7 X X +num_enum@0.7.5 X X X +num_enum_derive@0.7.5 X X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pilota@0.11.10 X X @@ -200,14 +190,15 @@ pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro-crate@3.3.0 X X -proc-macro2@1.0.101 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -215,148 +206,152 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -ref-cast@1.0.24 X X -ref-cast-impl@1.0.24 X X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +ref-cast@1.0.25 X X +ref-cast-impl@1.0.25 X X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc-hash@2.1.1 X X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X socket2@0.5.10 X X -socket2@0.6.0 X X +socket2@0.6.1 X X sonic-number@0.1.0 X sonic-rs@0.3.17 X -sonic-simd@0.1.1 X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +sonic-simd@0.1.2 X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X thiserror@1.0.69 X X -thiserror@2.0.16 X X +thiserror@2.0.17 X X thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X tokio-stream@0.1.17 X -tokio-util@0.7.16 X -toml_datetime@0.6.11 X X -toml_edit@0.22.27 X X +tokio-util@0.7.17 X +toml_datetime@0.7.3 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X volo@0.10.7 X X volo-thrift@0.10.8 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -winnow@0.7.13 X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/loader/DEPENDENCIES.rust.tsv b/crates/catalog/loader/DEPENDENCIES.rust.tsv index d809a30ca4..02b06c4479 100644 --- a/crates/catalog/loader/DEPENDENCIES.rust.tsv +++ b/crates/catalog/loader/DEPENDENCIES.rust.tsv @@ -1,25 +1,25 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-broadcast@0.7.2 X X async-lock@3.4.1 X X @@ -28,46 +28,44 @@ async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -aws-config@1.8.6 X -aws-credential-types@1.2.6 X -aws-runtime@1.5.10 X -aws-sdk-glue@1.119.0 X -aws-sdk-s3tables@1.37.0 X -aws-sdk-sso@1.83.0 X -aws-sdk-ssooidc@1.84.0 X -aws-sdk-sts@1.85.0 X -aws-sigv4@1.3.4 X -aws-smithy-async@1.2.5 X -aws-smithy-http@0.62.3 X -aws-smithy-http-client@1.1.1 X -aws-smithy-json@0.61.5 X -aws-smithy-observability@0.1.3 X -aws-smithy-query@0.60.7 X -aws-smithy-runtime@1.9.1 X -aws-smithy-runtime-api@1.9.0 X -aws-smithy-types@1.3.2 X -aws-smithy-xml@0.60.10 X -aws-types@1.3.8 X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.21.7 X X +aws-config@1.8.11 X +aws-credential-types@1.2.10 X +aws-runtime@1.5.16 X +aws-sdk-glue@1.132.0 X +aws-sdk-s3tables@1.46.0 X +aws-sdk-sso@1.90.0 X +aws-sdk-ssooidc@1.92.0 X +aws-sdk-sts@1.94.0 X +aws-sigv4@1.3.6 X +aws-smithy-async@1.2.7 X +aws-smithy-http@0.62.6 X +aws-smithy-http-client@1.1.5 X +aws-smithy-json@0.61.8 X +aws-smithy-observability@0.1.5 X +aws-smithy-query@0.60.9 X +aws-smithy-runtime@1.9.5 X +aws-smithy-runtime-api@1.9.3 X +aws-smithy-types@1.3.5 X +aws-smithy-xml@0.60.13 X +aws-types@1.3.10 X +backon@1.6.0 X base64@0.22.1 X X base64-simd@0.8.0 X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X +bytes@1.11.0 X bytes-utils@0.1.4 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X cfg_aliases@0.2.1 X chrono@0.4.42 X X concurrent-queue@2.5.0 X X @@ -75,16 +73,18 @@ const-oid@0.9.6 X X const-random@0.1.18 X X const-random-macro@0.1.16 X X core-foundation@0.10.1 X X -core-foundation@0.9.4 X X core-foundation-sys@0.8.7 X X cpufeatures@0.2.17 X X +crc@3.4.0 X X +crc-catalog@2.4.0 X X crc32c@0.6.8 X X crc32fast@1.5.0 X X crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X +crossbeam-queue@0.3.12 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -92,7 +92,7 @@ darling_core@0.21.3 X darling_macro@0.20.11 X darling_macro@0.21.3 X dashmap@6.1.0 X -deranged@0.5.3 X X +deranged@0.5.5 X X derive_builder@0.20.2 X X derive_builder_core@0.20.2 X X derive_builder_macro@0.20.2 X X @@ -106,131 +106,130 @@ event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X faststr@0.2.32 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +flume@0.11.1 X X fnv@1.0.7 X X +foldhash@0.1.5 X form_urlencoded@1.2.2 X X futures@0.3.31 X X futures-channel@0.3.31 X X futures-core@0.3.31 X X futures-executor@0.3.31 X X +futures-intrusive@0.5.0 X X futures-io@0.3.31 X X futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X h2@0.3.27 X h2@0.4.12 X -half@2.6.0 X X +half@2.7.1 X X hashbrown@0.14.5 X X hashbrown@0.15.5 X X +hashbrown@0.16.1 X X +hashlink@0.10.0 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hive_metastore@0.2.0 X hmac@0.12.1 X X home@0.5.11 X X http@0.2.12 X X -http@1.3.1 X X +http@1.4.0 X X http-body@0.4.6 X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X httpdate@1.0.3 X X hyper@0.14.32 X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.24.2 X X X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-glue@0.7.0 X -iceberg-catalog-hms@0.7.0 X -iceberg-catalog-loader@0.7.0 X -iceberg-catalog-rest@0.7.0 X -iceberg-catalog-s3tables@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-glue@0.8.0 X +iceberg-catalog-hms@0.8.0 X +iceberg-catalog-loader@0.8.0 X +iceberg-catalog-rest@0.8.0 X +iceberg-catalog-s3tables@0.8.0 X +iceberg-catalog-sql@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -integer-encoding@4.0.2 X -io-uring@0.7.10 X X +integer-encoding@4.1.0 X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X +libsqlite3-sys@0.30.1 X +libz-rs-sys@0.5.3 X linked-hash-map@0.5.6 X X linkedbytes@0.1.16 X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X memoffset@0.9.1 X metainfo@0.7.14 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X motore@0.4.1 X X motore-macros@0.4.3 X X mur3@0.1.0 X murmur3@0.5.2 X X nix@0.29.0 X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-conv@0.1.0 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -num_enum@0.7.4 X X X -num_enum_derive@0.7.4 X X X -object@0.36.7 X X +num_enum@0.7.5 X X X +num_enum_derive@0.7.5 X X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X openssl-probe@0.1.6 X X ordered-float@2.10.1 X ordered-float@4.6.0 X outref@0.5.2 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pilota@0.11.10 X X @@ -240,15 +239,16 @@ pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X powerfmt@0.2.0 X X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro-crate@3.3.0 X X -proc-macro2@1.0.101 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -256,164 +256,171 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -ref-cast@1.0.24 X X -ref-cast-impl@1.0.24 X X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +ref-cast@1.0.25 X X +ref-cast-impl@1.0.25 X X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc-hash@2.1.1 X X rustc_version@0.4.1 X X rustls@0.21.12 X X X -rustls@0.23.31 X X X -rustls-native-certs@0.6.3 X X X -rustls-native-certs@0.8.1 X X X -rustls-pemfile@1.0.4 X X X -rustls-pki-types@1.12.0 X X +rustls@0.23.35 X X X +rustls-native-certs@0.8.2 X X X +rustls-pki-types@1.13.1 X X rustls-webpki@0.101.7 X -rustls-webpki@0.103.4 X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -schannel@0.1.27 X -scoped-tls@1.0.1 X X +schannel@0.1.28 X scopeguard@1.2.0 X X sct@0.7.1 X X X -security-framework@2.11.1 X X -security-framework@3.4.0 X X +security-framework@3.5.1 X X security-framework-sys@2.15.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X socket2@0.5.10 X X -socket2@0.6.0 X X +socket2@0.6.1 X X sonic-number@0.1.0 X sonic-rs@0.3.17 X -sonic-simd@0.1.1 X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +sonic-simd@0.1.2 X +spin@0.9.8 X +sqlx@0.8.6 X X +sqlx-core@0.8.6 X X +sqlx-sqlite@0.8.6 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X thiserror@1.0.69 X X -thiserror@2.0.16 X X +thiserror@2.0.17 X X thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X -time@0.3.43 X X +time@0.3.44 X X time-core@0.1.6 X X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X tokio-rustls@0.24.1 X X -tokio-rustls@0.26.2 X X +tokio-rustls@0.26.4 X X tokio-stream@0.1.17 X -tokio-util@0.7.16 X -toml_datetime@0.6.11 X X -toml_edit@0.22.27 X X +tokio-util@0.7.17 X +toml_datetime@0.7.3 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X urlencoding@2.1.3 X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X +vcpkg@0.2.15 X X version_check@0.9.5 X X volo@0.10.7 X X volo-thrift@0.10.8 X X vsimd@0.8.0 X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@0.26.11 X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -winnow@0.7.13 X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X xmlparser@0.13.6 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/rest/DEPENDENCIES.rust.tsv b/crates/catalog/rest/DEPENDENCIES.rust.tsv index cf238f4b4c..c78434fa51 100644 --- a/crates/catalog/rest/DEPENDENCIES.rust.tsv +++ b/crates/catalog/rest/DEPENDENCIES.rust.tsv @@ -1,48 +1,46 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X @@ -56,7 +54,7 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -75,9 +73,9 @@ event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -89,105 +87,98 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X httpdate@1.0.3 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-rest@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-rest@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -195,134 +186,135 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/s3tables/DEPENDENCIES.rust.tsv b/crates/catalog/s3tables/DEPENDENCIES.rust.tsv index 13d8eebe7a..7dd182e435 100644 --- a/crates/catalog/s3tables/DEPENDENCIES.rust.tsv +++ b/crates/catalog/s3tables/DEPENDENCIES.rust.tsv @@ -1,77 +1,73 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -aws-config@1.8.6 X -aws-credential-types@1.2.6 X -aws-runtime@1.5.10 X -aws-sdk-s3tables@1.37.0 X -aws-sdk-sso@1.83.0 X -aws-sdk-ssooidc@1.84.0 X -aws-sdk-sts@1.85.0 X -aws-sigv4@1.3.4 X -aws-smithy-async@1.2.5 X -aws-smithy-http@0.62.3 X -aws-smithy-http-client@1.1.1 X -aws-smithy-json@0.61.5 X -aws-smithy-observability@0.1.3 X -aws-smithy-query@0.60.7 X -aws-smithy-runtime@1.9.1 X -aws-smithy-runtime-api@1.9.0 X -aws-smithy-types@1.3.2 X -aws-smithy-xml@0.60.10 X -aws-types@1.3.8 X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.21.7 X X +aws-config@1.8.11 X +aws-credential-types@1.2.10 X +aws-runtime@1.5.16 X +aws-sdk-s3tables@1.46.0 X +aws-sdk-sso@1.90.0 X +aws-sdk-ssooidc@1.92.0 X +aws-sdk-sts@1.94.0 X +aws-sigv4@1.3.6 X +aws-smithy-async@1.2.7 X +aws-smithy-http@0.62.6 X +aws-smithy-http-client@1.1.5 X +aws-smithy-json@0.61.8 X +aws-smithy-observability@0.1.5 X +aws-smithy-query@0.60.9 X +aws-smithy-runtime@1.9.5 X +aws-smithy-runtime-api@1.9.3 X +aws-smithy-types@1.3.5 X +aws-smithy-xml@0.60.13 X +aws-types@1.3.10 X +backon@1.6.0 X base64@0.22.1 X X base64-simd@0.8.0 X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X +bytes@1.11.0 X bytes-utils@0.1.4 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X const-random@0.1.18 X X const-random-macro@0.1.16 X X core-foundation@0.10.1 X X -core-foundation@0.9.4 X X core-foundation-sys@0.8.7 X X cpufeatures@0.2.17 X X crc32c@0.6.8 X X @@ -80,14 +76,14 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X darling_core@0.21.3 X darling_macro@0.20.11 X darling_macro@0.21.3 X -deranged@0.5.3 X X +deranged@0.5.5 X X derive_builder@0.20.2 X X derive_builder_core@0.20.2 X X derive_builder_macro@0.20.2 X X @@ -100,9 +96,9 @@ event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -114,114 +110,107 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X h2@0.3.27 X h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X http@0.2.12 X X -http@1.3.1 X X +http@1.4.0 X X http-body@0.4.6 X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X httpdate@1.0.3 X X hyper@0.14.32 X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.24.2 X X X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-s3tables@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-s3tables@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-conv@0.1.0 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X openssl-probe@0.1.6 X X ordered-float@2.10.1 X ordered-float@4.6.0 X outref@0.5.2 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X powerfmt@0.2.0 X X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -229,152 +218,150 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X rustls@0.21.12 X X X -rustls@0.23.31 X X X -rustls-native-certs@0.6.3 X X X -rustls-native-certs@0.8.1 X X X -rustls-pemfile@1.0.4 X X X -rustls-pki-types@1.12.0 X X +rustls@0.23.35 X X X +rustls-native-certs@0.8.2 X X X +rustls-pki-types@1.13.1 X X rustls-webpki@0.101.7 X -rustls-webpki@0.103.4 X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -schannel@0.1.27 X -scoped-tls@1.0.1 X X +schannel@0.1.28 X scopeguard@1.2.0 X X sct@0.7.1 X X X -security-framework@2.11.1 X X -security-framework@3.4.0 X X +security-framework@3.5.1 X X security-framework-sys@2.15.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X socket2@0.5.10 X X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X -time@0.3.43 X X +time@0.3.44 X X time-core@0.1.6 X X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X tokio-rustls@0.24.1 X X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X urlencoding@2.1.3 X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X vsimd@0.8.0 X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X xmlparser@0.13.6 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/catalog/sql/DEPENDENCIES.rust.tsv b/crates/catalog/sql/DEPENDENCIES.rust.tsv index 335c980967..cc48621d0c 100644 --- a/crates/catalog/sql/DEPENDENCIES.rust.tsv +++ b/crates/catalog/sql/DEPENDENCIES.rust.tsv @@ -1,49 +1,47 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X allocator-api2@0.2.21 X X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X @@ -51,7 +49,7 @@ const-random@0.1.18 X X const-random-macro@0.1.16 X X core-foundation-sys@0.8.7 X X cpufeatures@0.2.17 X X -crc@3.3.0 X X +crc@3.4.0 X X crc-catalog@2.4.0 X X crc32c@0.6.8 X X crc32fast@1.5.0 X X @@ -60,7 +58,7 @@ crossbeam-epoch@0.9.18 X X crossbeam-queue@0.3.12 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -79,9 +77,9 @@ event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X flume@0.11.1 X X fnv@1.0.7 X X foldhash@0.1.5 X @@ -96,105 +94,99 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X -half@2.6.0 X X +half@2.7.1 X X hashbrown@0.15.5 X X +hashbrown@0.16.1 X X hashlink@0.10.0 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-sql@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-sql@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X libsqlite3-sys@0.30.1 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -202,141 +194,142 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X -socket2@0.6.0 X X +socket2@0.6.1 X X spin@0.9.8 X sqlx@0.8.6 X X sqlx-core@0.8.6 X X sqlx-sqlite@0.8.6 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X tokio-stream@0.1.17 X -tokio-util@0.7.16 X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X vcpkg@0.2.15 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X +web-sys@0.3.83 X X webpki-roots@0.26.11 X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/examples/DEPENDENCIES.rust.tsv b/crates/examples/DEPENDENCIES.rust.tsv index 20702597a7..de07f1c650 100644 --- a/crates/examples/DEPENDENCIES.rust.tsv +++ b/crates/examples/DEPENDENCIES.rust.tsv @@ -1,48 +1,46 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X @@ -56,7 +54,7 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -75,9 +73,9 @@ event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -89,106 +87,99 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X httpdate@1.0.3 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-rest@0.7.0 X -iceberg-examples@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-catalog-rest@0.8.0 X +iceberg-examples@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.11.0 X X +indexmap@2.12.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -196,135 +187,136 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/iceberg/DEPENDENCIES.rust.tsv b/crates/iceberg/DEPENDENCIES.rust.tsv index 460a531644..c446954437 100644 --- a/crates/iceberg/DEPENDENCIES.rust.tsv +++ b/crates/iceberg/DEPENDENCIES.rust.tsv @@ -1,48 +1,46 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X @@ -56,7 +54,7 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -70,13 +68,14 @@ digest@0.10.7 X X displaydoc@0.2.5 X X dissimilar@1.0.10 X either@1.15.0 X X +equivalent@1.0.2 X X event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -88,101 +87,94 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -190,134 +182,135 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/integration_tests/DEPENDENCIES.rust.tsv b/crates/integration_tests/DEPENDENCIES.rust.tsv index 891d23966e..26f6d83820 100644 --- a/crates/integration_tests/DEPENDENCIES.rust.tsv +++ b/crates/integration_tests/DEPENDENCIES.rust.tsv @@ -1,406 +1,414 @@ -crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X -adler2@2.0.1 X X X -ahash@0.8.12 X X -aho-corasick@1.1.3 X X -alloc-no-stdlib@2.0.4 X -alloc-stdlib@0.2.2 X -allocator-api2@0.2.21 X X -android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X -array-init@2.1.0 X X -arrayref@0.3.9 X -arrayvec@0.7.6 X X -arrow@55.2.0 X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-csv@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-json@55.2.0 X -arrow-ord@55.2.0 X -arrow-row@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X -as-any@0.3.2 X X -async-compression@0.4.19 X X -async-lock@3.4.1 X X -async-trait@0.1.89 X X -atoi@2.0.0 X -atomic-waker@1.1.2 X X -autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.22.1 X X -bigdecimal@0.4.8 X X -bimap@0.6.3 X X -bitflags@2.9.4 X X -blake2@0.10.6 X X -blake3@1.8.2 X X X -block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X -brotli@8.0.2 X X -brotli-decompressor@5.0.0 X X -bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X -byteorder@1.5.0 X X -bytes@1.10.1 X -bzip2@0.5.2 X X -bzip2-sys@0.1.13+1.0.8 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X -chrono@0.4.42 X X -chrono-tz@0.10.4 X X -comfy-table@7.2.0 X -concurrent-queue@2.5.0 X X -const-oid@0.9.6 X X -const-random@0.1.18 X X -const-random-macro@0.1.16 X X -constant_time_eq@0.3.1 X X X -core-foundation-sys@0.8.7 X X -cpufeatures@0.2.17 X X -crc32c@0.6.8 X X -crc32fast@1.5.0 X X -crossbeam-channel@0.5.15 X X -crossbeam-epoch@0.9.18 X X -crossbeam-utils@0.8.21 X X -crunchy@0.2.4 X -crypto-common@0.1.6 X X -csv@1.3.1 X X -csv-core@0.1.12 X X -ctor@0.2.9 X X -darling@0.20.11 X -darling@0.21.3 X -darling_core@0.20.11 X -darling_core@0.21.3 X -darling_macro@0.20.11 X -darling_macro@0.21.3 X -dashmap@6.1.0 X -datafusion@48.0.1 X -datafusion-catalog@48.0.1 X -datafusion-catalog-listing@48.0.1 X -datafusion-common@48.0.1 X -datafusion-common-runtime@48.0.1 X -datafusion-datasource@48.0.1 X -datafusion-datasource-csv@48.0.1 X -datafusion-datasource-json@48.0.1 X -datafusion-datasource-parquet@48.0.1 X -datafusion-doc@48.0.1 X -datafusion-execution@48.0.1 X -datafusion-expr@48.0.1 X -datafusion-expr-common@48.0.1 X -datafusion-functions@48.0.1 X -datafusion-functions-aggregate@48.0.1 X -datafusion-functions-aggregate-common@48.0.1 X -datafusion-functions-nested@48.0.1 X -datafusion-functions-table@48.0.1 X -datafusion-functions-window@48.0.1 X -datafusion-functions-window-common@48.0.1 X -datafusion-macros@48.0.1 X -datafusion-optimizer@48.0.1 X -datafusion-physical-expr@48.0.1 X -datafusion-physical-expr-common@48.0.1 X -datafusion-physical-optimizer@48.0.1 X -datafusion-physical-plan@48.0.1 X -datafusion-session@48.0.1 X -datafusion-sql@48.0.1 X -derive_builder@0.20.2 X X -derive_builder_core@0.20.2 X X -derive_builder_macro@0.20.2 X X -digest@0.10.7 X X -displaydoc@0.2.5 X X -dissimilar@1.0.10 X -either@1.15.0 X X -equivalent@1.0.2 X X -errno@0.3.13 X X -event-listener@5.4.1 X X -event-listener-strategy@0.5.4 X X -expect-test@1.5.1 X X -fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -fixedbitset@0.5.7 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X -fnv@1.0.7 X X -foldhash@0.1.5 X -form_urlencoded@1.2.2 X X -futures@0.3.31 X X -futures-channel@0.3.31 X X -futures-core@0.3.31 X X -futures-executor@0.3.31 X X -futures-io@0.3.31 X X -futures-macro@0.3.31 X X -futures-sink@0.3.31 X X -futures-task@0.3.31 X X -futures-util@0.3.31 X X -generator@0.8.7 X X -generic-array@0.14.7 X -getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X -glob@0.3.3 X X -gloo-timers@0.3.0 X X -h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.14.5 X X -hashbrown@0.15.5 X X -heck@0.5.0 X X -hermit-abi@0.5.2 X X -hex@0.4.3 X X -hmac@0.12.1 X X -home@0.5.11 X X -http@1.3.1 X X -http-body@1.0.1 X -http-body-util@0.1.3 X -httparse@1.10.1 X X -httpdate@1.0.3 X X -humantime@2.2.0 X X -hyper@1.7.0 X -hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X -iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-rest@0.7.0 X -iceberg-datafusion@0.7.0 X -iceberg-integration-tests@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X -ident_case@1.0.1 X X -idna@1.1.0 X X -idna_adapter@1.2.1 X X -indexmap@2.11.0 X X -integer-encoding@3.0.4 X -io-uring@0.7.10 X X -ipnet@2.11.0 X X -iri-string@0.7.8 X X -itertools@0.13.0 X X -itertools@0.14.0 X X -itoa@1.0.15 X X -jobserver@0.1.34 X X -js-sys@0.3.78 X X -lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X -libm@0.2.15 X -libz-rs-sys@0.5.2 X -linux-raw-sys@0.9.4 X X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X -matchers@0.2.0 X -md-5@0.10.6 X X -memchr@2.7.5 X X -miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X -murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X -num-bigint@0.4.6 X X -num-complex@0.4.6 X X -num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X -num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X -object_store@0.12.3 X X -once_cell@1.21.3 X X -opendal@0.54.0 X -ordered-float@2.10.1 X -ordered-float@4.6.0 X -parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X -paste@1.0.15 X X -percent-encoding@2.3.2 X X -petgraph@0.8.2 X X -phf@0.12.1 X -phf_shared@0.12.1 X -pin-project-lite@0.2.16 X X -pin-utils@0.1.0 X X -pkg-config@0.3.32 X X -portable-atomic@1.11.1 X X -potential_utf@0.1.3 X -ppv-lite86@0.2.21 X X -prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X -psm@0.1.26 X X -quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X -r-efi@5.3.0 X X X -rand@0.8.5 X X -rand@0.9.2 X X -rand_chacha@0.3.1 X X -rand_chacha@0.9.0 X X -rand_core@0.6.4 X X -rand_core@0.9.3 X X -recursive@0.1.1 X -recursive-proc-macro-impl@0.1.1 X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X -reqsign@0.16.5 X -reqwest@0.12.23 X X -ring@0.17.14 X X -roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X -rustc_version@0.4.1 X X -rustix@1.0.8 X X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X -rustversion@1.0.22 X X -ryu@1.0.20 X X -same-file@1.0.6 X X -scoped-tls@1.0.1 X X -scopeguard@1.2.0 X X -semver@1.0.26 X X -seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X -serde_repr@0.1.20 X X -serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X -sha1@0.10.6 X X -sha2@0.10.9 X X -sharded-slab@0.1.7 X -shlex@1.3.0 X X -simdutf8@0.1.5 X X -siphasher@1.0.1 X X -slab@0.4.11 X -smallvec@1.15.1 X X -snap@1.1.1 X -socket2@0.6.0 X X -sqlparser@0.55.0 X -sqlparser_derive@0.3.0 X -stable_deref_trait@1.2.0 X X -stacker@0.1.21 X X -static_assertions@1.1.0 X X -strsim@0.11.1 X -strum@0.27.2 X -strum_macros@0.27.2 X -subtle@2.6.1 X -syn@2.0.106 X X -sync_wrapper@1.0.2 X -synstructure@0.13.2 X -tagptr@0.2.0 X X -tempfile@3.22.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X -thread_local@1.1.9 X X -threadpool@1.8.1 X X -thrift@0.17.0 X -tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X -tower@0.5.2 X -tower-http@0.6.6 X -tower-layer@0.3.3 X -tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X -tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X -try-lock@0.2.5 X -twox-hash@2.1.2 X -typed-builder@0.20.1 X X -typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X -unicode-segmentation@1.12.0 X X -unicode-width@0.2.1 X X -untrusted@0.9.0 X -url@2.5.7 X X -utf8_iter@1.0.4 X X -uuid@1.18.1 X X -version_check@0.9.5 X X -walkdir@2.5.0 X X -want@0.3.1 X -wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X -wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -web-time@1.1.0 X X -webpki-roots@1.0.2 X -winapi-util@0.1.11 X X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X -windows-sys@0.52.0 X X -windows-sys@0.59.0 X X -windows-sys@0.61.0 X X -windows-targets@0.52.6 X X -windows-threading@0.1.0 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_msvc@0.52.6 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_msvc@0.52.6 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -xz2@0.1.7 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X -zerofrom@0.1.6 X -zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X -zstd@0.13.3 X -zstd-safe@7.2.4 X X -zstd-sys@2.0.16+zstd.1.5.7 X X +crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib bzip2-1.0.6 +adler2@2.0.1 X X X +ahash@0.8.12 X X +aho-corasick@1.1.4 X X +alloc-no-stdlib@2.0.4 X +alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X +android_system_properties@0.1.5 X X +anyhow@1.0.100 X X +apache-avro@0.21.0 X +ar_archive_writer@0.2.0 X +array-init@2.1.0 X X +arrayref@0.3.9 X +arrayvec@0.7.6 X X +arrow@57.1.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-csv@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-json@57.1.0 X +arrow-ord@57.1.0 X +arrow-row@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X +as-any@0.3.2 X X +async-compression@0.4.19 X X +async-lock@3.4.1 X X +async-trait@0.1.89 X X +atoi@2.0.0 X +atomic-waker@1.1.2 X X +autocfg@1.5.0 X X +backon@1.6.0 X +base64@0.22.1 X X +bigdecimal@0.4.9 X X +bimap@0.6.3 X X +bitflags@2.10.0 X X +blake2@0.10.6 X X +blake3@1.8.2 X X X +block-buffer@0.10.4 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X +brotli@8.0.2 X X +brotli-decompressor@5.0.0 X X +bumpalo@3.19.0 X X +bytemuck@1.24.0 X X X +byteorder@1.5.0 X X +bytes@1.11.0 X +bzip2@0.5.2 X X +bzip2@0.6.1 X X +bzip2-sys@0.1.13+1.0.8 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X +chrono@0.4.42 X X +chrono-tz@0.10.4 X X +comfy-table@7.2.1 X +concurrent-queue@2.5.0 X X +const-oid@0.9.6 X X +const-random@0.1.18 X X +const-random-macro@0.1.16 X X +constant_time_eq@0.3.1 X X X +core-foundation-sys@0.8.7 X X +cpufeatures@0.2.17 X X +crc32c@0.6.8 X X +crc32fast@1.5.0 X X +crossbeam-channel@0.5.15 X X +crossbeam-epoch@0.9.18 X X +crossbeam-utils@0.8.21 X X +crunchy@0.2.4 X +crypto-common@0.1.7 X X +csv@1.4.0 X X +csv-core@0.1.13 X X +ctor@0.2.9 X X +darling@0.20.11 X +darling@0.21.3 X +darling_core@0.20.11 X +darling_core@0.21.3 X +darling_macro@0.20.11 X +darling_macro@0.21.3 X +dashmap@6.1.0 X +datafusion@51.0.0 X +datafusion-catalog@51.0.0 X +datafusion-catalog-listing@51.0.0 X +datafusion-common@51.0.0 X +datafusion-common-runtime@51.0.0 X +datafusion-datasource@51.0.0 X +datafusion-datasource-arrow@51.0.0 X +datafusion-datasource-csv@51.0.0 X +datafusion-datasource-json@51.0.0 X +datafusion-datasource-parquet@51.0.0 X +datafusion-doc@51.0.0 X +datafusion-execution@51.0.0 X +datafusion-expr@51.0.0 X +datafusion-expr-common@51.0.0 X +datafusion-functions@51.0.0 X +datafusion-functions-aggregate@51.0.0 X +datafusion-functions-aggregate-common@51.0.0 X +datafusion-functions-nested@51.0.0 X +datafusion-functions-table@51.0.0 X +datafusion-functions-window@51.0.0 X +datafusion-functions-window-common@51.0.0 X +datafusion-macros@51.0.0 X +datafusion-optimizer@51.0.0 X +datafusion-physical-expr@51.0.0 X +datafusion-physical-expr-adapter@51.0.0 X +datafusion-physical-expr-common@51.0.0 X +datafusion-physical-optimizer@51.0.0 X +datafusion-physical-plan@51.0.0 X +datafusion-pruning@51.0.0 X +datafusion-session@51.0.0 X +datafusion-sql@51.0.0 X +derive_builder@0.20.2 X X +derive_builder_core@0.20.2 X X +derive_builder_macro@0.20.2 X X +digest@0.10.7 X X +displaydoc@0.2.5 X X +dissimilar@1.0.10 X +either@1.15.0 X X +equivalent@1.0.2 X X +errno@0.3.14 X X +event-listener@5.4.1 X X +event-listener-strategy@0.5.4 X X +expect-test@1.5.1 X X +fastrand@2.3.0 X X +find-msvc-tools@0.1.5 X X +fixedbitset@0.5.7 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +fnv@1.0.7 X X +foldhash@0.1.5 X +form_urlencoded@1.2.2 X X +futures@0.3.31 X X +futures-channel@0.3.31 X X +futures-core@0.3.31 X X +futures-executor@0.3.31 X X +futures-io@0.3.31 X X +futures-macro@0.3.31 X X +futures-sink@0.3.31 X X +futures-task@0.3.31 X X +futures-timer@3.0.3 X X +futures-util@0.3.31 X X +generic-array@0.14.7 X +getrandom@0.2.16 X X +getrandom@0.3.4 X X +glob@0.3.3 X X +gloo-timers@0.3.0 X X +h2@0.4.12 X +half@2.7.1 X X +hashbrown@0.14.5 X X +hashbrown@0.15.5 X X +hashbrown@0.16.1 X X +heck@0.5.0 X X +hex@0.4.3 X X +hmac@0.12.1 X X +home@0.5.11 X X +http@1.4.0 X X +http-body@1.0.1 X +http-body-util@0.1.3 X +httparse@1.10.1 X X +httpdate@1.0.3 X X +humantime@2.3.0 X X +hyper@1.8.1 X +hyper-rustls@0.27.7 X X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X +iana-time-zone-haiku@0.1.2 X X +iceberg@0.8.0 X +iceberg-catalog-rest@0.8.0 X +iceberg-datafusion@0.8.0 X +iceberg-integration-tests@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X +ident_case@1.0.1 X X +idna@1.1.0 X X +idna_adapter@1.2.1 X X +indexmap@2.12.1 X X +integer-encoding@3.0.4 X +ipnet@2.11.0 X X +iri-string@0.7.9 X X +itertools@0.13.0 X X +itertools@0.14.0 X X +itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X +jobserver@0.1.34 X X +js-sys@0.3.83 X X +lazy_static@1.5.0 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libbz2-rs-sys@0.2.2 X +libc@0.2.178 X X +libm@0.2.15 X +libz-rs-sys@0.5.3 X +linux-raw-sys@0.11.0 X X X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X +lzma-sys@0.1.20 X X +md-5@0.10.6 X X +memchr@2.7.6 X X +miniz_oxide@0.8.9 X X X +mio@1.1.1 X +moka@0.12.11 X X +murmur3@0.5.2 X X +nu-ansi-term@0.50.3 X +num-bigint@0.4.6 X X +num-complex@0.4.6 X X +num-integer@0.1.46 X X +num-traits@0.2.19 X X +object@0.32.2 X X +object_store@0.12.4 X X +once_cell@1.21.3 X X +opendal@0.55.0 X +ordered-float@2.10.1 X +ordered-float@4.6.0 X +parking@2.2.1 X X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X +paste@1.0.15 X X +percent-encoding@2.3.2 X X +petgraph@0.8.3 X X +phf@0.12.1 X +phf_shared@0.12.1 X +pin-project-lite@0.2.16 X X +pin-utils@0.1.0 X X +pkg-config@0.3.32 X X +portable-atomic@1.11.1 X X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X +ppv-lite86@0.2.21 X X +prettyplease@0.2.37 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X +psm@0.1.28 X X +quad-rand@0.2.3 X +quick-xml@0.38.4 X +quote@1.0.42 X X +r-efi@5.3.0 X X X +rand@0.8.5 X X +rand@0.9.2 X X +rand_chacha@0.3.1 X X +rand_chacha@0.9.0 X X +rand_core@0.6.4 X X +rand_core@0.9.3 X X +recursive@0.1.1 X +recursive-proc-macro-impl@0.1.1 X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X +relative-path@1.9.3 X X +reqsign@0.16.5 X +reqwest@0.12.25 X X +ring@0.17.14 X X +roaring@0.11.2 X X +rstest@0.26.1 X X +rstest_macros@0.26.1 X X +rust_decimal@1.39.0 X +rustc_version@0.4.1 X X +rustix@1.1.2 X X X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X +rustversion@1.0.22 X X +ryu@1.0.20 X X +same-file@1.0.6 X X +scopeguard@1.2.0 X X +semver@1.0.27 X X +seq-macro@0.3.6 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X +serde_repr@0.1.20 X X +serde_urlencoded@0.7.1 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X +sha1@0.10.6 X X +sha2@0.10.9 X X +sharded-slab@0.1.7 X +shlex@1.3.0 X X +simd-adler32@0.3.8 X +simdutf8@0.1.5 X X +siphasher@1.0.1 X X +slab@0.4.11 X +smallvec@1.15.1 X X +snap@1.1.1 X +socket2@0.6.1 X X +sqlparser@0.59.0 X +sqlparser_derive@0.3.0 X +stable_deref_trait@1.2.1 X X +stacker@0.1.22 X X +strsim@0.11.1 X +strum@0.27.2 X +strum_macros@0.27.2 X +subtle@2.6.1 X +syn@2.0.111 X X +sync_wrapper@1.0.2 X +synstructure@0.13.2 X +tagptr@0.2.0 X X +tempfile@3.23.0 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X +thread_local@1.1.9 X X +thrift@0.17.0 X +tiny-keccak@2.0.2 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X +toml_datetime@0.7.3 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X +tower@0.5.2 X +tower-http@0.6.8 X +tower-layer@0.3.3 X +tower-service@0.3.3 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X +tracing-log@0.2.0 X +tracing-subscriber@0.3.22 X +try-lock@0.2.5 X +twox-hash@2.1.2 X +typed-builder@0.20.1 X X +typed-builder-macro@0.20.1 X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X +unicode-segmentation@1.12.0 X X +unicode-width@0.2.2 X X +untrusted@0.9.0 X +url@2.5.7 X X +utf8_iter@1.0.4 X X +uuid@1.19.0 X X +version_check@0.9.5 X X +walkdir@2.5.0 X X +want@0.3.1 X +wasi@0.11.1+wasi-snapshot-preview1 X X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X +wasm-streams@0.4.2 X X +web-sys@0.3.83 X X +web-time@1.1.0 X X +webpki-roots@1.0.4 X +winapi-util@0.1.11 X X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X +windows-sys@0.52.0 X X +windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X +windows-targets@0.52.6 X X +windows-targets@0.53.5 X X +windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X +windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X +windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X +windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X +windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X +windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X +windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X +windows_x86_64_msvc@0.52.6 X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +xz2@0.1.7 X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X +zerofrom@0.1.6 X +zerofrom-derive@0.1.6 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X +zstd@0.13.3 X +zstd-safe@7.2.4 X X +zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/integrations/cache-moka/DEPENDENCIES.rust.tsv b/crates/integrations/cache-moka/DEPENDENCIES.rust.tsv index 59b898d3fe..4763f71ec8 100644 --- a/crates/integrations/cache-moka/DEPENDENCIES.rust.tsv +++ b/crates/integrations/cache-moka/DEPENDENCIES.rust.tsv @@ -1,48 +1,46 @@ crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X adler2@2.0.1 X X X ahash@0.8.12 X X -aho-corasick@1.1.3 X X +aho-corasick@1.1.4 X X alloc-no-stdlib@2.0.4 X alloc-stdlib@0.2.2 X android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X +anyhow@1.0.100 X X +apache-avro@0.21.0 X array-init@2.1.0 X X arrayvec@0.7.6 X X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-ord@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-ord@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X async-lock@3.4.1 X X async-trait@0.1.89 X X atoi@2.0.0 X atomic-waker@1.1.2 X X autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X +backon@1.6.0 X base64@0.22.1 X X -bigdecimal@0.4.8 X X +bigdecimal@0.4.9 X X bimap@0.6.3 X X -bitflags@2.9.4 X X +bitflags@2.10.0 X X block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X brotli@8.0.2 X X brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X +bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -cc@1.2.36 X X -cfg-if@1.0.3 X X +bytes@1.11.0 X +cc@1.2.49 X X +cfg-if@1.0.4 X X chrono@0.4.42 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X @@ -56,7 +54,7 @@ crossbeam-channel@0.5.15 X X crossbeam-epoch@0.9.18 X X crossbeam-utils@0.8.21 X X crunchy@0.2.4 X -crypto-common@0.1.6 X X +crypto-common@0.1.7 X X darling@0.20.11 X darling@0.21.3 X darling_core@0.20.11 X @@ -70,13 +68,14 @@ digest@0.10.7 X X displaydoc@0.2.5 X X dissimilar@1.0.10 X either@1.15.0 X X +equivalent@1.0.2 X X event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X +find-msvc-tools@0.1.5 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X fnv@1.0.7 X X form_urlencoded@1.2.2 X X futures@0.3.31 X X @@ -88,102 +87,95 @@ futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X -generator@0.8.7 X X generic-array@0.14.7 X getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X +getrandom@0.3.4 X X gloo-timers@0.3.0 X X -half@2.6.0 X X -hashbrown@0.15.5 X X +half@2.7.1 X X +hashbrown@0.16.1 X X heck@0.5.0 X X -hermit-abi@0.5.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X -http@1.3.1 X X +http@1.4.0 X X http-body@1.0.1 X http-body-util@0.1.3 X httparse@1.10.1 X X -hyper@1.7.0 X +hyper@1.8.1 X hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-cache-moka@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X +iceberg@0.8.0 X +iceberg-cache-moka@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X integer-encoding@3.0.4 X -io-uring@0.7.10 X X ipnet@2.11.0 X X -iri-string@0.7.8 X X +iri-string@0.7.9 X X itertools@0.13.0 X X itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X jobserver@0.1.34 X X -js-sys@0.3.78 X X +js-sys@0.3.83 X X lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libc@0.2.178 X X libm@0.2.15 X -libz-rs-sys@0.5.2 X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -matchers@0.2.0 X +libz-rs-sys@0.5.3 X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X -memchr@2.7.5 X X +memchr@2.7.6 X X miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X +mio@1.1.1 X +moka@0.12.11 X X murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X +nu-ansi-term@0.50.3 X num-bigint@0.4.6 X X num-complex@0.4.6 X X num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X once_cell@1.21.3 X X -opendal@0.54.0 X +opendal@0.55.0 X ordered-float@2.10.1 X ordered-float@4.6.0 X parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X pkg-config@0.3.32 X X portable-atomic@1.11.1 X X -potential_utf@0.1.3 X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X +proc-macro2@1.0.103 X X quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X +quick-xml@0.38.4 X +quote@1.0.42 X X r-efi@5.3.0 X X X rand@0.8.5 X X rand@0.9.2 X X @@ -191,134 +183,135 @@ rand_chacha@0.3.1 X X rand_chacha@0.9.0 X X rand_core@0.6.4 X X rand_core@0.9.3 X X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X reqsign@0.16.5 X -reqwest@0.12.23 X X +reqwest@0.12.25 X X ring@0.17.14 X X roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X +rust_decimal@1.39.0 X rustc_version@0.4.1 X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X rustversion@1.0.22 X X ryu@1.0.20 X X -scoped-tls@1.0.1 X X scopeguard@1.2.0 X X -semver@1.0.26 X X +semver@1.0.27 X X seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X serde_repr@0.1.20 X X serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X sha1@0.10.6 X X sha2@0.10.9 X X sharded-slab@0.1.7 X shlex@1.3.0 X X +simd-adler32@0.3.8 X simdutf8@0.1.5 X X slab@0.4.11 X smallvec@1.15.1 X X snap@1.1.1 X -socket2@0.6.0 X X -stable_deref_trait@1.2.0 X X -static_assertions@1.1.0 X X +socket2@0.6.1 X X +stable_deref_trait@1.2.1 X X strsim@0.11.1 X strum@0.27.2 X strum_macros@0.27.2 X subtle@2.6.1 X -syn@2.0.106 X X +syn@2.0.111 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X thread_local@1.1.9 X X -threadpool@1.8.1 X X thrift@0.17.0 X tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X tower@0.5.2 X -tower-http@0.6.6 X +tower-http@0.6.8 X tower-layer@0.3.3 X tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X +tracing-subscriber@0.3.22 X try-lock@0.2.5 X twox-hash@2.1.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -webpki-roots@1.0.2 X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X +web-sys@0.3.83 X X +webpki-roots@1.0.4 X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X windows-targets@0.52.6 X X -windows-threading@0.1.0 X X +windows-targets@0.53.5 X X windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X +windows_x86_64_msvc@0.53.1 X X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X zerofrom@0.1.6 X zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X zstd@0.13.3 X zstd-safe@7.2.4 X X zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/integrations/datafusion/DEPENDENCIES.rust.tsv b/crates/integrations/datafusion/DEPENDENCIES.rust.tsv index 8fb3e04f80..7a0f57e7a0 100644 --- a/crates/integrations/datafusion/DEPENDENCIES.rust.tsv +++ b/crates/integrations/datafusion/DEPENDENCIES.rust.tsv @@ -1,401 +1,409 @@ -crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X -adler2@2.0.1 X X X -ahash@0.8.12 X X -aho-corasick@1.1.3 X X -alloc-no-stdlib@2.0.4 X -alloc-stdlib@0.2.2 X -allocator-api2@0.2.21 X X -android_system_properties@0.1.5 X X -anyhow@1.0.99 X X -apache-avro@0.20.0 X -array-init@2.1.0 X X -arrayref@0.3.9 X -arrayvec@0.7.6 X X -arrow@55.2.0 X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-csv@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-json@55.2.0 X -arrow-ord@55.2.0 X -arrow-row@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X -as-any@0.3.2 X X -async-compression@0.4.19 X X -async-lock@3.4.1 X X -async-trait@0.1.89 X X -atoi@2.0.0 X -atomic-waker@1.1.2 X X -autocfg@1.5.0 X X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.22.1 X X -bigdecimal@0.4.8 X X -bimap@0.6.3 X X -bitflags@2.9.4 X X -blake2@0.10.6 X X -blake3@1.8.2 X X X -block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X -brotli@8.0.2 X X -brotli-decompressor@5.0.0 X X -bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X -byteorder@1.5.0 X X -bytes@1.10.1 X -bzip2@0.5.2 X X -bzip2-sys@0.1.13+1.0.8 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X -chrono@0.4.42 X X -chrono-tz@0.10.4 X X -comfy-table@7.2.0 X -concurrent-queue@2.5.0 X X -const-oid@0.9.6 X X -const-random@0.1.18 X X -const-random-macro@0.1.16 X X -constant_time_eq@0.3.1 X X X -core-foundation-sys@0.8.7 X X -cpufeatures@0.2.17 X X -crc32c@0.6.8 X X -crc32fast@1.5.0 X X -crossbeam-channel@0.5.15 X X -crossbeam-epoch@0.9.18 X X -crossbeam-utils@0.8.21 X X -crunchy@0.2.4 X -crypto-common@0.1.6 X X -csv@1.3.1 X X -csv-core@0.1.12 X X -darling@0.20.11 X -darling@0.21.3 X -darling_core@0.20.11 X -darling_core@0.21.3 X -darling_macro@0.20.11 X -darling_macro@0.21.3 X -dashmap@6.1.0 X -datafusion@48.0.1 X -datafusion-catalog@48.0.1 X -datafusion-catalog-listing@48.0.1 X -datafusion-common@48.0.1 X -datafusion-common-runtime@48.0.1 X -datafusion-datasource@48.0.1 X -datafusion-datasource-csv@48.0.1 X -datafusion-datasource-json@48.0.1 X -datafusion-datasource-parquet@48.0.1 X -datafusion-doc@48.0.1 X -datafusion-execution@48.0.1 X -datafusion-expr@48.0.1 X -datafusion-expr-common@48.0.1 X -datafusion-functions@48.0.1 X -datafusion-functions-aggregate@48.0.1 X -datafusion-functions-aggregate-common@48.0.1 X -datafusion-functions-nested@48.0.1 X -datafusion-functions-table@48.0.1 X -datafusion-functions-window@48.0.1 X -datafusion-functions-window-common@48.0.1 X -datafusion-macros@48.0.1 X -datafusion-optimizer@48.0.1 X -datafusion-physical-expr@48.0.1 X -datafusion-physical-expr-common@48.0.1 X -datafusion-physical-optimizer@48.0.1 X -datafusion-physical-plan@48.0.1 X -datafusion-session@48.0.1 X -datafusion-sql@48.0.1 X -derive_builder@0.20.2 X X -derive_builder_core@0.20.2 X X -derive_builder_macro@0.20.2 X X -digest@0.10.7 X X -displaydoc@0.2.5 X X -dissimilar@1.0.10 X -either@1.15.0 X X -equivalent@1.0.2 X X -errno@0.3.13 X X -event-listener@5.4.1 X X -event-listener-strategy@0.5.4 X X -expect-test@1.5.1 X X -fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -fixedbitset@0.5.7 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X -fnv@1.0.7 X X -foldhash@0.1.5 X -form_urlencoded@1.2.2 X X -futures@0.3.31 X X -futures-channel@0.3.31 X X -futures-core@0.3.31 X X -futures-executor@0.3.31 X X -futures-io@0.3.31 X X -futures-macro@0.3.31 X X -futures-sink@0.3.31 X X -futures-task@0.3.31 X X -futures-util@0.3.31 X X -generator@0.8.7 X X -generic-array@0.14.7 X -getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X -glob@0.3.3 X X -gloo-timers@0.3.0 X X -half@2.6.0 X X -hashbrown@0.14.5 X X -hashbrown@0.15.5 X X -heck@0.5.0 X X -hermit-abi@0.5.2 X X -hex@0.4.3 X X -hmac@0.12.1 X X -home@0.5.11 X X -http@1.3.1 X X -http-body@1.0.1 X -http-body-util@0.1.3 X -httparse@1.10.1 X X -humantime@2.2.0 X X -hyper@1.7.0 X -hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X -iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-datafusion@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X -ident_case@1.0.1 X X -idna@1.1.0 X X -idna_adapter@1.2.1 X X -indexmap@2.11.0 X X -integer-encoding@3.0.4 X -io-uring@0.7.10 X X -ipnet@2.11.0 X X -iri-string@0.7.8 X X -itertools@0.13.0 X X -itertools@0.14.0 X X -itoa@1.0.15 X X -jobserver@0.1.34 X X -js-sys@0.3.78 X X -lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X -libm@0.2.15 X -libz-rs-sys@0.5.2 X -linux-raw-sys@0.9.4 X X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X -matchers@0.2.0 X -md-5@0.10.6 X X -memchr@2.7.5 X X -miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X -murmur3@0.5.2 X X -nu-ansi-term@0.50.1 X -num@0.4.3 X X -num-bigint@0.4.6 X X -num-complex@0.4.6 X X -num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X -num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X -object_store@0.12.3 X X -once_cell@1.21.3 X X -opendal@0.54.0 X -ordered-float@2.10.1 X -ordered-float@4.6.0 X -parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X -paste@1.0.15 X X -percent-encoding@2.3.2 X X -petgraph@0.8.2 X X -phf@0.12.1 X -phf_shared@0.12.1 X -pin-project-lite@0.2.16 X X -pin-utils@0.1.0 X X -pkg-config@0.3.32 X X -portable-atomic@1.11.1 X X -potential_utf@0.1.3 X -ppv-lite86@0.2.21 X X -prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X -psm@0.1.26 X X -quad-rand@0.2.3 X -quick-xml@0.37.5 X -quote@1.0.40 X X -r-efi@5.3.0 X X X -rand@0.8.5 X X -rand@0.9.2 X X -rand_chacha@0.3.1 X X -rand_chacha@0.9.0 X X -rand_core@0.6.4 X X -rand_core@0.9.3 X X -recursive@0.1.1 X -recursive-proc-macro-impl@0.1.1 X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X -reqsign@0.16.5 X -reqwest@0.12.23 X X -ring@0.17.14 X X -roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X -rustc_version@0.4.1 X X -rustix@1.0.8 X X X -rustls@0.23.31 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X -rustversion@1.0.22 X X -ryu@1.0.20 X X -same-file@1.0.6 X X -scoped-tls@1.0.1 X X -scopeguard@1.2.0 X X -semver@1.0.26 X X -seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X -serde_repr@0.1.20 X X -serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X -sha1@0.10.6 X X -sha2@0.10.9 X X -sharded-slab@0.1.7 X -shlex@1.3.0 X X -simdutf8@0.1.5 X X -siphasher@1.0.1 X X -slab@0.4.11 X -smallvec@1.15.1 X X -snap@1.1.1 X -socket2@0.6.0 X X -sqlparser@0.55.0 X -sqlparser_derive@0.3.0 X -stable_deref_trait@1.2.0 X X -stacker@0.1.21 X X -static_assertions@1.1.0 X X -strsim@0.11.1 X -strum@0.27.2 X -strum_macros@0.27.2 X -subtle@2.6.1 X -syn@2.0.106 X X -sync_wrapper@1.0.2 X -synstructure@0.13.2 X -tagptr@0.2.0 X X -tempfile@3.22.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X -thread_local@1.1.9 X X -threadpool@1.8.1 X X -thrift@0.17.0 X -tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X -tower@0.5.2 X -tower-http@0.6.6 X -tower-layer@0.3.3 X -tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X -tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X -try-lock@0.2.5 X -twox-hash@2.1.2 X -typed-builder@0.20.1 X X -typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X -unicode-segmentation@1.12.0 X X -unicode-width@0.2.1 X X -untrusted@0.9.0 X -url@2.5.7 X X -utf8_iter@1.0.4 X X -uuid@1.18.1 X X -version_check@0.9.5 X X -walkdir@2.5.0 X X -want@0.3.1 X -wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X -wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -web-time@1.1.0 X X -webpki-roots@1.0.2 X -winapi-util@0.1.11 X X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X -windows-sys@0.52.0 X X -windows-sys@0.59.0 X X -windows-sys@0.61.0 X X -windows-targets@0.52.6 X X -windows-threading@0.1.0 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_msvc@0.52.6 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_msvc@0.52.6 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_msvc@0.52.6 X X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -xz2@0.1.7 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X -zerofrom@0.1.6 X -zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X -zstd@0.13.3 X -zstd-safe@7.2.4 X X -zstd-sys@2.0.16+zstd.1.5.7 X X +crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib bzip2-1.0.6 +adler2@2.0.1 X X X +ahash@0.8.12 X X +aho-corasick@1.1.4 X X +alloc-no-stdlib@2.0.4 X +alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X +android_system_properties@0.1.5 X X +anyhow@1.0.100 X X +apache-avro@0.21.0 X +ar_archive_writer@0.2.0 X +array-init@2.1.0 X X +arrayref@0.3.9 X +arrayvec@0.7.6 X X +arrow@57.1.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-csv@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-json@57.1.0 X +arrow-ord@57.1.0 X +arrow-row@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X +as-any@0.3.2 X X +async-compression@0.4.19 X X +async-lock@3.4.1 X X +async-trait@0.1.89 X X +atoi@2.0.0 X +atomic-waker@1.1.2 X X +autocfg@1.5.0 X X +backon@1.6.0 X +base64@0.22.1 X X +bigdecimal@0.4.9 X X +bimap@0.6.3 X X +bitflags@2.10.0 X X +blake2@0.10.6 X X +blake3@1.8.2 X X X +block-buffer@0.10.4 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X +brotli@8.0.2 X X +brotli-decompressor@5.0.0 X X +bumpalo@3.19.0 X X +bytemuck@1.24.0 X X X +byteorder@1.5.0 X X +bytes@1.11.0 X +bzip2@0.5.2 X X +bzip2@0.6.1 X X +bzip2-sys@0.1.13+1.0.8 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X +chrono@0.4.42 X X +chrono-tz@0.10.4 X X +comfy-table@7.2.1 X +concurrent-queue@2.5.0 X X +const-oid@0.9.6 X X +const-random@0.1.18 X X +const-random-macro@0.1.16 X X +constant_time_eq@0.3.1 X X X +core-foundation-sys@0.8.7 X X +cpufeatures@0.2.17 X X +crc32c@0.6.8 X X +crc32fast@1.5.0 X X +crossbeam-channel@0.5.15 X X +crossbeam-epoch@0.9.18 X X +crossbeam-utils@0.8.21 X X +crunchy@0.2.4 X +crypto-common@0.1.7 X X +csv@1.4.0 X X +csv-core@0.1.13 X X +darling@0.20.11 X +darling@0.21.3 X +darling_core@0.20.11 X +darling_core@0.21.3 X +darling_macro@0.20.11 X +darling_macro@0.21.3 X +dashmap@6.1.0 X +datafusion@51.0.0 X +datafusion-catalog@51.0.0 X +datafusion-catalog-listing@51.0.0 X +datafusion-common@51.0.0 X +datafusion-common-runtime@51.0.0 X +datafusion-datasource@51.0.0 X +datafusion-datasource-arrow@51.0.0 X +datafusion-datasource-csv@51.0.0 X +datafusion-datasource-json@51.0.0 X +datafusion-datasource-parquet@51.0.0 X +datafusion-doc@51.0.0 X +datafusion-execution@51.0.0 X +datafusion-expr@51.0.0 X +datafusion-expr-common@51.0.0 X +datafusion-functions@51.0.0 X +datafusion-functions-aggregate@51.0.0 X +datafusion-functions-aggregate-common@51.0.0 X +datafusion-functions-nested@51.0.0 X +datafusion-functions-table@51.0.0 X +datafusion-functions-window@51.0.0 X +datafusion-functions-window-common@51.0.0 X +datafusion-macros@51.0.0 X +datafusion-optimizer@51.0.0 X +datafusion-physical-expr@51.0.0 X +datafusion-physical-expr-adapter@51.0.0 X +datafusion-physical-expr-common@51.0.0 X +datafusion-physical-optimizer@51.0.0 X +datafusion-physical-plan@51.0.0 X +datafusion-pruning@51.0.0 X +datafusion-session@51.0.0 X +datafusion-sql@51.0.0 X +derive_builder@0.20.2 X X +derive_builder_core@0.20.2 X X +derive_builder_macro@0.20.2 X X +digest@0.10.7 X X +displaydoc@0.2.5 X X +dissimilar@1.0.10 X +either@1.15.0 X X +equivalent@1.0.2 X X +errno@0.3.14 X X +event-listener@5.4.1 X X +event-listener-strategy@0.5.4 X X +expect-test@1.5.1 X X +fastrand@2.3.0 X X +find-msvc-tools@0.1.5 X X +fixedbitset@0.5.7 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +fnv@1.0.7 X X +foldhash@0.1.5 X +form_urlencoded@1.2.2 X X +futures@0.3.31 X X +futures-channel@0.3.31 X X +futures-core@0.3.31 X X +futures-executor@0.3.31 X X +futures-io@0.3.31 X X +futures-macro@0.3.31 X X +futures-sink@0.3.31 X X +futures-task@0.3.31 X X +futures-timer@3.0.3 X X +futures-util@0.3.31 X X +generic-array@0.14.7 X +getrandom@0.2.16 X X +getrandom@0.3.4 X X +glob@0.3.3 X X +gloo-timers@0.3.0 X X +half@2.7.1 X X +hashbrown@0.14.5 X X +hashbrown@0.15.5 X X +hashbrown@0.16.1 X X +heck@0.5.0 X X +hex@0.4.3 X X +hmac@0.12.1 X X +home@0.5.11 X X +http@1.4.0 X X +http-body@1.0.1 X +http-body-util@0.1.3 X +httparse@1.10.1 X X +humantime@2.3.0 X X +hyper@1.8.1 X +hyper-rustls@0.27.7 X X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X +iana-time-zone-haiku@0.1.2 X X +iceberg@0.8.0 X +iceberg-datafusion@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X +ident_case@1.0.1 X X +idna@1.1.0 X X +idna_adapter@1.2.1 X X +indexmap@2.12.1 X X +integer-encoding@3.0.4 X +ipnet@2.11.0 X X +iri-string@0.7.9 X X +itertools@0.13.0 X X +itertools@0.14.0 X X +itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X +jobserver@0.1.34 X X +js-sys@0.3.83 X X +lazy_static@1.5.0 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libbz2-rs-sys@0.2.2 X +libc@0.2.178 X X +libm@0.2.15 X +libz-rs-sys@0.5.3 X +linux-raw-sys@0.11.0 X X X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X +lzma-sys@0.1.20 X X +md-5@0.10.6 X X +memchr@2.7.6 X X +miniz_oxide@0.8.9 X X X +mio@1.1.1 X +moka@0.12.11 X X +murmur3@0.5.2 X X +nu-ansi-term@0.50.3 X +num-bigint@0.4.6 X X +num-complex@0.4.6 X X +num-integer@0.1.46 X X +num-traits@0.2.19 X X +object@0.32.2 X X +object_store@0.12.4 X X +once_cell@1.21.3 X X +opendal@0.55.0 X +ordered-float@2.10.1 X +ordered-float@4.6.0 X +parking@2.2.1 X X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X +paste@1.0.15 X X +percent-encoding@2.3.2 X X +petgraph@0.8.3 X X +phf@0.12.1 X +phf_shared@0.12.1 X +pin-project-lite@0.2.16 X X +pin-utils@0.1.0 X X +pkg-config@0.3.32 X X +portable-atomic@1.11.1 X X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X +ppv-lite86@0.2.21 X X +prettyplease@0.2.37 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X +psm@0.1.28 X X +quad-rand@0.2.3 X +quick-xml@0.38.4 X +quote@1.0.42 X X +r-efi@5.3.0 X X X +rand@0.8.5 X X +rand@0.9.2 X X +rand_chacha@0.3.1 X X +rand_chacha@0.9.0 X X +rand_core@0.6.4 X X +rand_core@0.9.3 X X +recursive@0.1.1 X +recursive-proc-macro-impl@0.1.1 X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X +relative-path@1.9.3 X X +reqsign@0.16.5 X +reqwest@0.12.25 X X +ring@0.17.14 X X +roaring@0.11.2 X X +rstest@0.26.1 X X +rstest_macros@0.26.1 X X +rust_decimal@1.39.0 X +rustc_version@0.4.1 X X +rustix@1.1.2 X X X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X +rustversion@1.0.22 X X +ryu@1.0.20 X X +same-file@1.0.6 X X +scopeguard@1.2.0 X X +semver@1.0.27 X X +seq-macro@0.3.6 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X +serde_repr@0.1.20 X X +serde_urlencoded@0.7.1 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X +sha1@0.10.6 X X +sha2@0.10.9 X X +sharded-slab@0.1.7 X +shlex@1.3.0 X X +simd-adler32@0.3.8 X +simdutf8@0.1.5 X X +siphasher@1.0.1 X X +slab@0.4.11 X +smallvec@1.15.1 X X +snap@1.1.1 X +socket2@0.6.1 X X +sqlparser@0.59.0 X +sqlparser_derive@0.3.0 X +stable_deref_trait@1.2.1 X X +stacker@0.1.22 X X +strsim@0.11.1 X +strum@0.27.2 X +strum_macros@0.27.2 X +subtle@2.6.1 X +syn@2.0.111 X X +sync_wrapper@1.0.2 X +synstructure@0.13.2 X +tagptr@0.2.0 X X +tempfile@3.23.0 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X +thread_local@1.1.9 X X +thrift@0.17.0 X +tiny-keccak@2.0.2 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X +toml_datetime@0.7.3 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X +tower@0.5.2 X +tower-http@0.6.8 X +tower-layer@0.3.3 X +tower-service@0.3.3 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X +tracing-log@0.2.0 X +tracing-subscriber@0.3.22 X +try-lock@0.2.5 X +twox-hash@2.1.2 X +typed-builder@0.20.1 X X +typed-builder-macro@0.20.1 X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X +unicode-segmentation@1.12.0 X X +unicode-width@0.2.2 X X +untrusted@0.9.0 X +url@2.5.7 X X +utf8_iter@1.0.4 X X +uuid@1.19.0 X X +version_check@0.9.5 X X +walkdir@2.5.0 X X +want@0.3.1 X +wasi@0.11.1+wasi-snapshot-preview1 X X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X +wasm-streams@0.4.2 X X +web-sys@0.3.83 X X +web-time@1.1.0 X X +webpki-roots@1.0.4 X +winapi-util@0.1.11 X X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X +windows-sys@0.52.0 X X +windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X +windows-targets@0.52.6 X X +windows-targets@0.53.5 X X +windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X +windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X +windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X +windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X +windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X +windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X +windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X +windows_x86_64_msvc@0.52.6 X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +xz2@0.1.7 X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X +zerofrom@0.1.6 X +zerofrom-derive@0.1.6 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X +zstd@0.13.3 X +zstd-safe@7.2.4 X X +zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/integrations/playground/DEPENDENCIES.rust.tsv b/crates/integrations/playground/DEPENDENCIES.rust.tsv index 84dd20ed3e..8f57b1d9ae 100644 --- a/crates/integrations/playground/DEPENDENCIES.rust.tsv +++ b/crates/integrations/playground/DEPENDENCIES.rust.tsv @@ -1,509 +1,492 @@ -crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 MPL-2.0 Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X -adler2@2.0.1 X X X -adler32@1.2.0 X -ahash@0.8.12 X X -aho-corasick@1.1.3 X X -alloc-no-stdlib@2.0.4 X -alloc-stdlib@0.2.2 X -allocator-api2@0.2.21 X X -android_system_properties@0.1.5 X X -anstream@0.6.20 X X -anstyle@1.0.11 X X -anstyle-parse@0.2.7 X X -anstyle-query@1.1.4 X X -anstyle-wincon@3.0.10 X X -anyhow@1.0.99 X X -apache-avro@0.17.0 X -apache-avro@0.20.0 X -array-init@2.1.0 X X -arrayref@0.3.9 X -arrayvec@0.7.6 X X -arrow@55.2.0 X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-csv@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-json@55.2.0 X -arrow-ord@55.2.0 X -arrow-row@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X -as-any@0.3.2 X X -async-compression@0.4.19 X X -async-lock@3.4.1 X X -async-trait@0.1.89 X X -atoi@2.0.0 X -atomic-waker@1.1.2 X X -autocfg@1.5.0 X X -aws-config@1.8.6 X -aws-credential-types@1.2.6 X -aws-runtime@1.5.10 X -aws-sdk-sso@1.83.0 X -aws-sdk-ssooidc@1.84.0 X -aws-sdk-sts@1.85.0 X -aws-sigv4@1.3.4 X -aws-smithy-async@1.2.5 X -aws-smithy-http@0.62.3 X -aws-smithy-http-client@1.1.1 X -aws-smithy-json@0.61.5 X -aws-smithy-observability@0.1.3 X -aws-smithy-query@0.60.7 X -aws-smithy-runtime@1.9.1 X -aws-smithy-runtime-api@1.9.0 X -aws-smithy-types@1.3.2 X -aws-smithy-xml@0.60.10 X -aws-types@1.3.8 X -backon@1.5.2 X -backtrace@0.3.75 X X -base64@0.22.1 X X -base64-simd@0.8.0 X -bigdecimal@0.4.8 X X -bimap@0.6.3 X X -bitflags@2.9.4 X X -blake2@0.10.6 X X -blake3@1.8.2 X X X -block-buffer@0.10.4 X X -bon@3.7.2 X X -bon-macros@3.7.2 X X -brotli@8.0.2 X X -brotli-decompressor@5.0.0 X X -bumpalo@3.19.0 X X -bytemuck@1.23.2 X X X -byteorder@1.5.0 X X -bytes@1.10.1 X -bytes-utils@0.1.4 X X -bzip2@0.4.4 X X -bzip2@0.5.2 X X -bzip2-sys@0.1.13+1.0.8 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X -cfg_aliases@0.2.1 X -chrono@0.4.42 X X -chrono-tz@0.10.4 X X -clap@4.5.47 X X -clap_builder@4.5.47 X X -clap_derive@4.5.47 X X -clap_lex@0.7.5 X X -clipboard-win@5.4.1 X -colorchoice@1.0.4 X X -comfy-table@7.2.0 X -concurrent-queue@2.5.0 X X -const-oid@0.9.6 X X -const-random@0.1.18 X X -const-random-macro@0.1.16 X X -constant_time_eq@0.3.1 X X X -core-foundation@0.10.1 X X -core-foundation-sys@0.8.7 X X -core2@0.4.0 X X -cpufeatures@0.2.17 X X -crc32c@0.6.8 X X -crc32fast@1.5.0 X X -crossbeam-channel@0.5.15 X X -crossbeam-epoch@0.9.18 X X -crossbeam-utils@0.8.21 X X -crunchy@0.2.4 X -crypto-common@0.1.6 X X -csv@1.3.1 X X -csv-core@0.1.12 X X -darling@0.20.11 X -darling@0.21.3 X -darling_core@0.20.11 X -darling_core@0.21.3 X -darling_macro@0.20.11 X -darling_macro@0.21.3 X -dary_heap@0.3.7 X X -dashmap@6.1.0 X -datafusion@48.0.1 X -datafusion-catalog@48.0.1 X -datafusion-catalog-listing@48.0.1 X -datafusion-cli@48.0.1 X -datafusion-common@48.0.1 X -datafusion-common-runtime@48.0.1 X -datafusion-datasource@48.0.1 X -datafusion-datasource-avro@48.0.1 X -datafusion-datasource-csv@48.0.1 X -datafusion-datasource-json@48.0.1 X -datafusion-datasource-parquet@48.0.1 X -datafusion-doc@48.0.1 X -datafusion-execution@48.0.1 X -datafusion-expr@48.0.1 X -datafusion-expr-common@48.0.1 X -datafusion-functions@48.0.1 X -datafusion-functions-aggregate@48.0.1 X -datafusion-functions-aggregate-common@48.0.1 X -datafusion-functions-nested@48.0.1 X -datafusion-functions-table@48.0.1 X -datafusion-functions-window@48.0.1 X -datafusion-functions-window-common@48.0.1 X -datafusion-macros@48.0.1 X -datafusion-optimizer@48.0.1 X -datafusion-physical-expr@48.0.1 X -datafusion-physical-expr-common@48.0.1 X -datafusion-physical-optimizer@48.0.1 X -datafusion-physical-plan@48.0.1 X -datafusion-session@48.0.1 X -datafusion-sql@48.0.1 X -deranged@0.5.3 X X -derive_builder@0.20.2 X X -derive_builder_core@0.20.2 X X -derive_builder_macro@0.20.2 X X -digest@0.10.7 X X -dirs@6.0.0 X X -dirs-sys@0.5.0 X X -displaydoc@0.2.5 X X -dissimilar@1.0.10 X -either@1.15.0 X X -endian-type@0.1.2 X -env_filter@0.1.3 X X -env_logger@0.11.8 X X -equivalent@1.0.2 X X -errno@0.3.13 X X -error-code@3.3.2 X -event-listener@5.4.1 X X -event-listener-strategy@0.5.4 X X -expect-test@1.5.1 X X -fastrand@2.3.0 X X -fd-lock@4.0.4 X X -find-msvc-tools@0.1.1 X X -fixedbitset@0.5.7 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X -fnv@1.0.7 X X -foldhash@0.1.5 X -form_urlencoded@1.2.2 X X -fs-err@3.1.1 X X -futures@0.3.31 X X -futures-channel@0.3.31 X X -futures-core@0.3.31 X X -futures-executor@0.3.31 X X -futures-io@0.3.31 X X -futures-macro@0.3.31 X X -futures-sink@0.3.31 X X -futures-task@0.3.31 X X -futures-util@0.3.31 X X -generator@0.8.7 X X -generic-array@0.14.7 X -getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X -glob@0.3.3 X X -gloo-timers@0.3.0 X X -h2@0.4.12 X -half@2.6.0 X X -hashbrown@0.14.5 X X -hashbrown@0.15.5 X X -heck@0.5.0 X X -hermit-abi@0.5.2 X X -hex@0.4.3 X X -hmac@0.12.1 X X -home@0.5.11 X X -http@0.2.12 X X -http@1.3.1 X X -http-body@0.4.6 X -http-body@1.0.1 X -http-body-util@0.1.3 X -httparse@1.10.1 X X -httpdate@1.0.3 X X -humantime@2.2.0 X X -hyper@1.7.0 X -hyper-rustls@0.27.7 X X X -hyper-util@0.1.16 X -iana-time-zone@0.1.63 X X -iana-time-zone-haiku@0.1.2 X X -iceberg@0.7.0 X -iceberg-catalog-rest@0.7.0 X -iceberg-datafusion@0.7.0 X -iceberg-playground@0.7.0 X -iceberg_test_utils@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X -ident_case@1.0.1 X X -idna@1.1.0 X X -idna_adapter@1.2.1 X X -indexmap@2.11.0 X X -integer-encoding@3.0.4 X -io-uring@0.7.10 X X -ipnet@2.11.0 X X -iri-string@0.7.8 X X -is_terminal_polyfill@1.70.1 X X -itertools@0.13.0 X X -itertools@0.14.0 X X -itoa@1.0.15 X X -jiff@0.2.15 X X -jobserver@0.1.34 X X -js-sys@0.3.78 X X -lazy_static@1.5.0 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X -libflate@2.1.0 X -libflate_lz77@2.1.0 X -libm@0.2.15 X -libmimalloc-sys@0.1.44 X -libredox@0.1.9 X -libz-rs-sys@0.5.2 X -linux-raw-sys@0.9.4 X X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -loom@0.7.2 X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X -matchers@0.2.0 X -md-5@0.10.6 X X -memchr@2.7.5 X X -mimalloc@0.1.48 X -miniz_oxide@0.8.9 X X X -mio@1.0.4 X -moka@0.12.10 X X -murmur3@0.5.2 X X -nibble_vec@0.1.0 X -nix@0.30.1 X -nu-ansi-term@0.50.1 X -num@0.4.3 X X -num-bigint@0.4.6 X X -num-complex@0.4.6 X X -num-conv@0.1.0 X X -num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X -num-traits@0.2.19 X X -num_cpus@1.17.0 X X -object@0.36.7 X X -object_store@0.12.3 X X -once_cell@1.21.3 X X -once_cell_polyfill@1.70.1 X X -opendal@0.54.0 X -openssl-probe@0.1.6 X X -option-ext@0.2.0 X -ordered-float@2.10.1 X -ordered-float@4.6.0 X -outref@0.5.2 X -parking@2.2.1 X X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X -paste@1.0.15 X X -percent-encoding@2.3.2 X X -petgraph@0.8.2 X X -phf@0.12.1 X -phf_shared@0.12.1 X -pin-project-lite@0.2.16 X X -pin-utils@0.1.0 X X -pkg-config@0.3.32 X X -portable-atomic@1.11.1 X X -portable-atomic-util@0.2.4 X X -potential_utf@0.1.3 X -powerfmt@0.2.0 X X -ppv-lite86@0.2.21 X X -prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X -psm@0.1.26 X X -quad-rand@0.2.3 X -quick-xml@0.37.5 X -quick-xml@0.38.3 X -quote@1.0.40 X X -r-efi@5.3.0 X X X -radix_trie@0.2.1 X -rand@0.8.5 X X -rand@0.9.2 X X -rand_chacha@0.3.1 X X -rand_chacha@0.9.0 X X -rand_core@0.6.4 X X -rand_core@0.9.3 X X -recursive@0.1.1 X -recursive-proc-macro-impl@0.1.1 X -redox_syscall@0.5.17 X -redox_users@0.5.2 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X -reqsign@0.16.5 X -reqwest@0.12.23 X X -ring@0.17.14 X X -rle-decode-fast@1.0.3 X X -roaring@0.11.2 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X -rustc_version@0.4.1 X X -rustix@1.0.8 X X X -rustls@0.23.31 X X X -rustls-native-certs@0.8.1 X X X -rustls-pemfile@2.2.0 X X X -rustls-pki-types@1.12.0 X X -rustls-webpki@0.103.4 X -rustversion@1.0.22 X X -rustyline@16.0.0 X -ryu@1.0.20 X X -same-file@1.0.6 X X -schannel@0.1.27 X -scoped-tls@1.0.1 X X -scopeguard@1.2.0 X X -security-framework@3.4.0 X X -security-framework-sys@2.15.0 X X -semver@1.0.26 X X -seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_json@1.0.143 X X -serde_repr@0.1.20 X X -serde_spanned@0.6.9 X X -serde_urlencoded@0.7.1 X X -serde_with@3.14.0 X X -serde_with_macros@3.14.0 X X -sha1@0.10.6 X X -sha2@0.10.9 X X -sharded-slab@0.1.7 X -shlex@1.3.0 X X -signal-hook-registry@1.4.6 X X -simdutf8@0.1.5 X X -siphasher@1.0.1 X X -slab@0.4.11 X -smallvec@1.15.1 X X -snap@1.1.1 X -socket2@0.6.0 X X -sqlparser@0.55.0 X -sqlparser_derive@0.3.0 X -stable_deref_trait@1.2.0 X X -stacker@0.1.21 X X -static_assertions@1.1.0 X X -strsim@0.11.1 X -strum@0.26.3 X -strum@0.27.2 X -strum_macros@0.26.4 X -strum_macros@0.27.2 X -subtle@2.6.1 X -syn@2.0.106 X X -sync_wrapper@1.0.2 X -synstructure@0.13.2 X -tagptr@0.2.0 X X -tempfile@3.22.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X -thread_local@1.1.9 X X -threadpool@1.8.1 X X -thrift@0.17.0 X -time@0.3.43 X X -time-core@0.1.6 X X -tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-rustls@0.26.2 X X -tokio-util@0.7.16 X -toml@0.8.23 X X -toml_datetime@0.6.11 X X -toml_edit@0.22.27 X X -toml_write@0.1.2 X X -tower@0.5.2 X -tower-http@0.6.6 X -tower-layer@0.3.3 X -tower-service@0.3.3 X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X -tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X -try-lock@0.2.5 X -twox-hash@2.1.2 X -typed-builder@0.19.1 X X -typed-builder@0.20.1 X X -typed-builder-macro@0.19.1 X X -typed-builder-macro@0.20.1 X X -typenum@1.18.0 X X -unicode-ident@1.0.18 X X X -unicode-segmentation@1.12.0 X X -unicode-width@0.2.1 X X -untrusted@0.9.0 X -url@2.5.7 X X -urlencoding@2.1.3 X -utf8_iter@1.0.4 X X -utf8parse@0.2.2 X X -uuid@1.18.1 X X -version_check@0.9.5 X X -vsimd@0.8.0 X -walkdir@2.5.0 X X -want@0.3.1 X -wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X -wasm-streams@0.4.2 X X -web-sys@0.3.78 X X -web-time@1.1.0 X X -webpki-roots@1.0.2 X -winapi-util@0.1.11 X X -windows@0.61.3 X X -windows-collections@0.2.0 X X -windows-core@0.61.2 X X -windows-future@0.2.1 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-numerics@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X -windows-sys@0.52.0 X X -windows-sys@0.59.0 X X -windows-sys@0.60.2 X X -windows-sys@0.61.0 X X -windows-targets@0.52.6 X X -windows-targets@0.53.3 X X -windows-threading@0.1.0 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_gnullvm@0.53.0 X X -windows_aarch64_msvc@0.52.6 X X -windows_aarch64_msvc@0.53.0 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnu@0.53.0 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_gnullvm@0.53.0 X X -windows_i686_msvc@0.52.6 X X -windows_i686_msvc@0.53.0 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnu@0.53.0 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_gnullvm@0.53.0 X X -windows_x86_64_msvc@0.52.6 X X -windows_x86_64_msvc@0.53.0 X X -winnow@0.7.13 X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -xmlparser@0.13.6 X X -xz2@0.1.7 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X -zerofrom@0.1.6 X -zerofrom-derive@0.1.6 X -zeroize@1.8.1 X X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X -zstd@0.13.3 X -zstd-safe@7.2.4 X X -zstd-sys@2.0.16+zstd.1.5.7 X X +crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 MPL-2.0 Unicode-3.0 Unlicense Zlib bzip2-1.0.6 +adler2@2.0.1 X X X +ahash@0.8.12 X X +aho-corasick@1.1.4 X X +alloc-no-stdlib@2.0.4 X +alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X +android_system_properties@0.1.5 X X +anstream@0.6.21 X X +anstyle@1.0.13 X X +anstyle-parse@0.2.7 X X +anstyle-query@1.1.5 X X +anstyle-wincon@3.0.11 X X +anyhow@1.0.100 X X +apache-avro@0.20.0 X +apache-avro@0.21.0 X +ar_archive_writer@0.2.0 X +array-init@2.1.0 X X +arrayref@0.3.9 X +arrayvec@0.7.6 X X +arrow@57.1.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-csv@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-json@57.1.0 X +arrow-ord@57.1.0 X +arrow-row@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X +as-any@0.3.2 X X +async-compression@0.4.19 X X +async-lock@3.4.1 X X +async-trait@0.1.89 X X +atoi@2.0.0 X +atomic-waker@1.1.2 X X +autocfg@1.5.0 X X +aws-config@1.8.11 X +aws-credential-types@1.2.10 X +aws-runtime@1.5.16 X +aws-sdk-sso@1.90.0 X +aws-sdk-ssooidc@1.92.0 X +aws-sdk-sts@1.94.0 X +aws-sigv4@1.3.6 X +aws-smithy-async@1.2.7 X +aws-smithy-http@0.62.6 X +aws-smithy-http-client@1.1.5 X +aws-smithy-json@0.61.8 X +aws-smithy-observability@0.1.5 X +aws-smithy-query@0.60.9 X +aws-smithy-runtime@1.9.5 X +aws-smithy-runtime-api@1.9.3 X +aws-smithy-types@1.3.5 X +aws-smithy-xml@0.60.13 X +aws-types@1.3.10 X +backon@1.6.0 X +base64@0.22.1 X X +base64-simd@0.8.0 X +bigdecimal@0.4.9 X X +bimap@0.6.3 X X +bitflags@2.10.0 X X +blake2@0.10.6 X X +blake3@1.8.2 X X X +block-buffer@0.10.4 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X +brotli@8.0.2 X X +brotli-decompressor@5.0.0 X X +bumpalo@3.19.0 X X +bytemuck@1.24.0 X X X +byteorder@1.5.0 X X +bytes@1.11.0 X +bytes-utils@0.1.4 X X +bzip2@0.5.2 X X +bzip2@0.6.1 X X +bzip2-sys@0.1.13+1.0.8 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X +cfg_aliases@0.2.1 X +chrono@0.4.42 X X +chrono-tz@0.10.4 X X +clap@4.5.53 X X +clap_builder@4.5.53 X X +clap_derive@4.5.49 X X +clap_lex@0.7.6 X X +clipboard-win@5.4.1 X +colorchoice@1.0.4 X X +comfy-table@7.2.1 X +concurrent-queue@2.5.0 X X +const-oid@0.9.6 X X +const-random@0.1.18 X X +const-random-macro@0.1.16 X X +constant_time_eq@0.3.1 X X X +core-foundation@0.10.1 X X +core-foundation-sys@0.8.7 X X +cpufeatures@0.2.17 X X +crc32c@0.6.8 X X +crc32fast@1.5.0 X X +crossbeam-channel@0.5.15 X X +crossbeam-epoch@0.9.18 X X +crossbeam-utils@0.8.21 X X +crunchy@0.2.4 X +crypto-common@0.1.7 X X +csv@1.4.0 X X +csv-core@0.1.13 X X +darling@0.20.11 X +darling@0.21.3 X +darling_core@0.20.11 X +darling_core@0.21.3 X +darling_macro@0.20.11 X +darling_macro@0.21.3 X +dashmap@6.1.0 X +datafusion@51.0.0 X +datafusion-catalog@51.0.0 X +datafusion-catalog-listing@51.0.0 X +datafusion-cli@51.0.0 X +datafusion-common@51.0.0 X +datafusion-common-runtime@51.0.0 X +datafusion-datasource@51.0.0 X +datafusion-datasource-arrow@51.0.0 X +datafusion-datasource-avro@51.0.0 X +datafusion-datasource-csv@51.0.0 X +datafusion-datasource-json@51.0.0 X +datafusion-datasource-parquet@51.0.0 X +datafusion-doc@51.0.0 X +datafusion-execution@51.0.0 X +datafusion-expr@51.0.0 X +datafusion-expr-common@51.0.0 X +datafusion-functions@51.0.0 X +datafusion-functions-aggregate@51.0.0 X +datafusion-functions-aggregate-common@51.0.0 X +datafusion-functions-nested@51.0.0 X +datafusion-functions-table@51.0.0 X +datafusion-functions-window@51.0.0 X +datafusion-functions-window-common@51.0.0 X +datafusion-macros@51.0.0 X +datafusion-optimizer@51.0.0 X +datafusion-physical-expr@51.0.0 X +datafusion-physical-expr-adapter@51.0.0 X +datafusion-physical-expr-common@51.0.0 X +datafusion-physical-optimizer@51.0.0 X +datafusion-physical-plan@51.0.0 X +datafusion-pruning@51.0.0 X +datafusion-session@51.0.0 X +datafusion-sql@51.0.0 X +deranged@0.5.5 X X +derive_builder@0.20.2 X X +derive_builder_core@0.20.2 X X +derive_builder_macro@0.20.2 X X +digest@0.10.7 X X +dirs@6.0.0 X X +dirs-sys@0.5.0 X X +displaydoc@0.2.5 X X +dissimilar@1.0.10 X +either@1.15.0 X X +endian-type@0.1.2 X +env_filter@0.1.4 X X +env_logger@0.11.8 X X +equivalent@1.0.2 X X +errno@0.3.14 X X +error-code@3.3.2 X +event-listener@5.4.1 X X +event-listener-strategy@0.5.4 X X +expect-test@1.5.1 X X +fastrand@2.3.0 X X +fd-lock@4.0.4 X X +find-msvc-tools@0.1.5 X X +fixedbitset@0.5.7 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +fnv@1.0.7 X X +foldhash@0.1.5 X +form_urlencoded@1.2.2 X X +fs-err@3.2.0 X X +futures@0.3.31 X X +futures-channel@0.3.31 X X +futures-core@0.3.31 X X +futures-executor@0.3.31 X X +futures-io@0.3.31 X X +futures-macro@0.3.31 X X +futures-sink@0.3.31 X X +futures-task@0.3.31 X X +futures-timer@3.0.3 X X +futures-util@0.3.31 X X +generic-array@0.14.7 X +getrandom@0.2.16 X X +getrandom@0.3.4 X X +glob@0.3.3 X X +gloo-timers@0.3.0 X X +h2@0.4.12 X +half@2.7.1 X X +hashbrown@0.14.5 X X +hashbrown@0.15.5 X X +hashbrown@0.16.1 X X +heck@0.5.0 X X +hex@0.4.3 X X +hmac@0.12.1 X X +home@0.5.11 X X +http@0.2.12 X X +http@1.4.0 X X +http-body@0.4.6 X +http-body@1.0.1 X +http-body-util@0.1.3 X +httparse@1.10.1 X X +httpdate@1.0.3 X X +humantime@2.3.0 X X +hyper@1.8.1 X +hyper-rustls@0.27.7 X X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X +iana-time-zone-haiku@0.1.2 X X +iceberg@0.8.0 X +iceberg-catalog-rest@0.8.0 X +iceberg-datafusion@0.8.0 X +iceberg-playground@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X +ident_case@1.0.1 X X +idna@1.1.0 X X +idna_adapter@1.2.1 X X +indexmap@2.12.1 X X +integer-encoding@3.0.4 X +ipnet@2.11.0 X X +iri-string@0.7.9 X X +is_terminal_polyfill@1.70.2 X X +itertools@0.13.0 X X +itertools@0.14.0 X X +itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X +jobserver@0.1.34 X X +js-sys@0.3.83 X X +lazy_static@1.5.0 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libbz2-rs-sys@0.2.2 X +libc@0.2.178 X X +libm@0.2.15 X +libmimalloc-sys@0.1.44 X +libredox@0.1.10 X +libz-rs-sys@0.5.3 X +linux-raw-sys@0.11.0 X X X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X +lzma-sys@0.1.20 X X +md-5@0.10.6 X X +memchr@2.7.6 X X +mimalloc@0.1.48 X +miniz_oxide@0.8.9 X X X +mio@1.1.1 X +moka@0.12.11 X X +murmur3@0.5.2 X X +nibble_vec@0.1.0 X +nix@0.30.1 X +nu-ansi-term@0.50.3 X +num-bigint@0.4.6 X X +num-complex@0.4.6 X X +num-conv@0.1.0 X X +num-integer@0.1.46 X X +num-traits@0.2.19 X X +object@0.32.2 X X +object_store@0.12.4 X X +once_cell@1.21.3 X X +once_cell_polyfill@1.70.2 X X +opendal@0.55.0 X +openssl-probe@0.1.6 X X +option-ext@0.2.0 X +ordered-float@2.10.1 X +ordered-float@4.6.0 X +outref@0.5.2 X +parking@2.2.1 X X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X +paste@1.0.15 X X +percent-encoding@2.3.2 X X +petgraph@0.8.3 X X +phf@0.12.1 X +phf_shared@0.12.1 X +pin-project-lite@0.2.16 X X +pin-utils@0.1.0 X X +pkg-config@0.3.32 X X +portable-atomic@1.11.1 X X +portable-atomic-util@0.2.4 X X +potential_utf@0.1.4 X +powerfmt@0.2.0 X X +ppv-lite86@0.2.21 X X +prettyplease@0.2.37 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X +psm@0.1.28 X X +quad-rand@0.2.3 X +quick-xml@0.38.4 X +quote@1.0.42 X X +r-efi@5.3.0 X X X +radix_trie@0.2.1 X +rand@0.8.5 X X +rand@0.9.2 X X +rand_chacha@0.3.1 X X +rand_chacha@0.9.0 X X +rand_core@0.6.4 X X +rand_core@0.9.3 X X +recursive@0.1.1 X +recursive-proc-macro-impl@0.1.1 X +redox_syscall@0.5.18 X +redox_users@0.5.2 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X +relative-path@1.9.3 X X +reqsign@0.16.5 X +reqwest@0.12.25 X X +ring@0.17.14 X X +roaring@0.11.2 X X +rstest@0.26.1 X X +rstest_macros@0.26.1 X X +rust_decimal@1.39.0 X +rustc_version@0.4.1 X X +rustix@1.1.2 X X X +rustls@0.23.35 X X X +rustls-native-certs@0.8.2 X X X +rustls-pemfile@2.2.0 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X +rustversion@1.0.22 X X +rustyline@17.0.2 X +ryu@1.0.20 X X +same-file@1.0.6 X X +schannel@0.1.28 X +scopeguard@1.2.0 X X +security-framework@3.5.1 X X +security-framework-sys@2.15.0 X X +semver@1.0.27 X X +seq-macro@0.3.6 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_json@1.0.145 X X +serde_repr@0.1.20 X X +serde_spanned@0.6.9 X X +serde_urlencoded@0.7.1 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X +sha1@0.10.6 X X +sha2@0.10.9 X X +sharded-slab@0.1.7 X +shlex@1.3.0 X X +signal-hook-registry@1.4.7 X X +simd-adler32@0.3.8 X +simdutf8@0.1.5 X X +siphasher@1.0.1 X X +slab@0.4.11 X +smallvec@1.15.1 X X +snap@1.1.1 X +socket2@0.6.1 X X +sqlparser@0.59.0 X +sqlparser_derive@0.3.0 X +stable_deref_trait@1.2.1 X X +stacker@0.1.22 X X +strsim@0.11.1 X +strum@0.27.2 X +strum_macros@0.27.2 X +subtle@2.6.1 X +syn@2.0.111 X X +sync_wrapper@1.0.2 X +synstructure@0.13.2 X +tagptr@0.2.0 X X +tempfile@3.23.0 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X +thread_local@1.1.9 X X +thrift@0.17.0 X +time@0.3.44 X X +time-core@0.1.6 X X +tiny-keccak@2.0.2 X +tinystr@0.8.2 X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X +toml@0.8.23 X X +toml_datetime@0.6.11 X X +toml_datetime@0.7.3 X X +toml_edit@0.22.27 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X +toml_write@0.1.2 X X +tower@0.5.2 X +tower-http@0.6.8 X +tower-layer@0.3.3 X +tower-service@0.3.3 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X +tracing-log@0.2.0 X +tracing-subscriber@0.3.22 X +try-lock@0.2.5 X +twox-hash@2.1.2 X +typed-builder@0.20.1 X X +typed-builder-macro@0.20.1 X X +typenum@1.19.0 X X +unicode-ident@1.0.22 X X X +unicode-segmentation@1.12.0 X X +unicode-width@0.2.2 X X +untrusted@0.9.0 X +url@2.5.7 X X +urlencoding@2.1.3 X +utf8_iter@1.0.4 X X +utf8parse@0.2.2 X X +uuid@1.19.0 X X +version_check@0.9.5 X X +vsimd@0.8.0 X +walkdir@2.5.0 X X +want@0.3.1 X +wasi@0.11.1+wasi-snapshot-preview1 X X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X +wasm-streams@0.4.2 X X +web-sys@0.3.83 X X +web-time@1.1.0 X X +webpki-roots@1.0.4 X +winapi-util@0.1.11 X X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X +windows-sys@0.52.0 X X +windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X +windows-targets@0.52.6 X X +windows-targets@0.53.5 X X +windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X +windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X +windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X +windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X +windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X +windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X +windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X +windows_x86_64_msvc@0.52.6 X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +xmlparser@0.13.6 X X +xz2@0.1.7 X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X +zerofrom@0.1.6 X +zerofrom-derive@0.1.6 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X +zstd@0.13.3 X +zstd-safe@7.2.4 X X +zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/sqllogictest/DEPENDENCIES.rust.tsv b/crates/sqllogictest/DEPENDENCIES.rust.tsv index e8af062ea3..94bd88e2e6 100644 --- a/crates/sqllogictest/DEPENDENCIES.rust.tsv +++ b/crates/sqllogictest/DEPENDENCIES.rust.tsv @@ -1,398 +1,482 @@ -crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib -addr2line@0.24.2 X X -adler2@2.0.1 X X X -adler32@1.2.0 X -ahash@0.8.12 X X -aho-corasick@1.1.3 X X -alloc-no-stdlib@2.0.4 X -alloc-stdlib@0.2.2 X -allocator-api2@0.2.21 X X -android_system_properties@0.1.5 X X -anstream@0.6.20 X X -anstyle@1.0.11 X X -anstyle-parse@0.2.7 X X -anstyle-query@1.1.4 X X -anstyle-wincon@3.0.10 X X -anyhow@1.0.99 X X -apache-avro@0.17.0 X -arrayref@0.3.9 X -arrayvec@0.7.6 X X -arrow@55.2.0 X -arrow-arith@55.2.0 X -arrow-array@55.2.0 X -arrow-buffer@55.2.0 X -arrow-cast@55.2.0 X -arrow-csv@55.2.0 X -arrow-data@55.2.0 X -arrow-ipc@55.2.0 X -arrow-json@55.2.0 X -arrow-ord@55.2.0 X -arrow-row@55.2.0 X -arrow-schema@55.2.0 X -arrow-select@55.2.0 X -arrow-string@55.2.0 X -async-compression@0.4.19 X X -async-recursion@1.1.1 X X -async-trait@0.1.89 X X -atoi@2.0.0 X -autocfg@1.5.0 X X -backtrace@0.3.75 X X -base64@0.21.7 X X -base64@0.22.1 X X -bigdecimal@0.4.8 X X -bitflags@2.9.4 X X -blake2@0.10.6 X X -blake3@1.8.2 X X X -block-buffer@0.10.4 X X -brotli@8.0.2 X X -brotli-decompressor@5.0.0 X X -bumpalo@3.19.0 X X -byteorder@1.5.0 X X -bytes@1.10.1 X -bzip2@0.4.4 X X -bzip2@0.5.2 X X -bzip2-sys@0.1.13+1.0.8 X X -cc@1.2.36 X X -cfg-if@1.0.3 X X -chrono@0.4.42 X X -chrono-tz@0.10.4 X X -clap@4.5.47 X X -clap_builder@4.5.47 X X -clap_derive@4.5.47 X X -clap_lex@0.7.5 X X -colorchoice@1.0.4 X X -comfy-table@7.2.0 X -console@0.15.11 X -const-random@0.1.18 X X -const-random-macro@0.1.16 X X -constant_time_eq@0.3.1 X X X -core-foundation-sys@0.8.7 X X -core2@0.4.0 X X -cpufeatures@0.2.17 X X -crc32fast@1.5.0 X X -crossbeam-utils@0.8.21 X X -crunchy@0.2.4 X -crypto-common@0.1.6 X X -csv@1.3.1 X X -csv-core@0.1.12 X X -dary_heap@0.3.7 X X -dashmap@6.1.0 X -datafusion@48.0.1 X -datafusion-catalog@48.0.1 X -datafusion-catalog-listing@48.0.1 X -datafusion-common@48.0.1 X -datafusion-common-runtime@48.0.1 X -datafusion-datasource@48.0.1 X -datafusion-datasource-avro@48.0.1 X -datafusion-datasource-csv@48.0.1 X -datafusion-datasource-json@48.0.1 X -datafusion-datasource-parquet@48.0.1 X -datafusion-doc@48.0.1 X -datafusion-execution@48.0.1 X -datafusion-expr@48.0.1 X -datafusion-expr-common@48.0.1 X -datafusion-functions@48.0.1 X -datafusion-functions-aggregate@48.0.1 X -datafusion-functions-aggregate-common@48.0.1 X -datafusion-functions-nested@48.0.1 X -datafusion-functions-table@48.0.1 X -datafusion-functions-window@48.0.1 X -datafusion-functions-window-common@48.0.1 X -datafusion-macros@48.0.1 X -datafusion-optimizer@48.0.1 X -datafusion-physical-expr@48.0.1 X -datafusion-physical-expr-common@48.0.1 X -datafusion-physical-optimizer@48.0.1 X -datafusion-physical-plan@48.0.1 X -datafusion-session@48.0.1 X -datafusion-spark@48.0.1 X -datafusion-sql@48.0.1 X -datafusion-sqllogictest@48.0.1 X -datafusion-substrait@48.0.1 X -digest@0.10.7 X X -displaydoc@0.2.5 X X -dyn-clone@1.0.20 X X -educe@0.6.0 X -either@1.15.0 X X -encode_unicode@1.0.0 X X -enum-ordinalize@4.3.0 X -enum-ordinalize-derive@4.3.1 X -equivalent@1.0.2 X X -errno@0.3.13 X X -escape8259@0.5.3 X -fallible-iterator@0.2.0 X X -fastrand@2.3.0 X X -find-msvc-tools@0.1.1 X X -fixedbitset@0.5.7 X X -flatbuffers@25.2.10 X -flate2@1.1.2 X X -fnv@1.0.7 X X -foldhash@0.1.5 X -form_urlencoded@1.2.2 X X -fs-err@3.1.1 X X -futures@0.3.31 X X -futures-channel@0.3.31 X X -futures-core@0.3.31 X X -futures-executor@0.3.31 X X -futures-io@0.3.31 X X -futures-macro@0.3.31 X X -futures-sink@0.3.31 X X -futures-task@0.3.31 X X -futures-util@0.3.31 X X -generic-array@0.14.7 X -getrandom@0.2.16 X X -getrandom@0.3.3 X X -gimli@0.31.1 X X -glob@0.3.3 X X -half@2.6.0 X X -hashbrown@0.14.5 X X -hashbrown@0.15.5 X X -heck@0.5.0 X X -hex@0.4.3 X X -hmac@0.12.1 X X -http@1.3.1 X X -humantime@2.2.0 X X -iana-time-zone@0.1.63 X X -iana-time-zone-haiku@0.1.2 X X -iceberg-sqllogictest@0.7.0 X -icu_collections@2.0.0 X -icu_locale_core@2.0.0 X -icu_normalizer@2.0.0 X -icu_normalizer_data@2.0.0 X -icu_properties@2.0.1 X -icu_properties_data@2.0.1 X -icu_provider@2.0.0 X -idna@1.1.0 X X -idna_adapter@1.2.1 X X -indexmap@2.11.0 X X -indicatif@0.17.11 X -integer-encoding@3.0.4 X -io-uring@0.7.10 X X -is_terminal_polyfill@1.70.1 X X -itertools@0.13.0 X X -itertools@0.14.0 X X -itoa@1.0.15 X X -jobserver@0.1.34 X X -js-sys@0.3.78 X X -lexical-core@1.0.5 X X -lexical-parse-float@1.0.5 X X -lexical-parse-integer@1.0.5 X X -lexical-util@1.0.6 X X -lexical-write-float@1.0.5 X X -lexical-write-integer@1.0.5 X X -libc@0.2.175 X X -libflate@2.1.0 X -libflate_lz77@2.1.0 X -libm@0.2.15 X -libtest-mimic@0.8.1 X X -libz-rs-sys@0.5.2 X -linux-raw-sys@0.9.4 X X X -litemap@0.8.0 X -lock_api@0.4.13 X X -log@0.4.28 X X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X -md-5@0.10.6 X X -memchr@2.7.5 X X -miniz_oxide@0.8.9 X X X -mio@1.0.4 X -multimap@0.10.1 X X -num@0.4.3 X X -num-bigint@0.4.6 X X -num-complex@0.4.6 X X -num-integer@0.1.46 X X -num-iter@0.1.45 X X -num-rational@0.4.2 X X -num-traits@0.2.19 X X -number_prefix@0.4.0 X -object@0.36.7 X X -object_store@0.12.3 X X -once_cell@1.21.3 X X -once_cell_polyfill@1.70.1 X X -ordered-float@2.10.1 X -owo-colors@4.2.2 X -parking_lot@0.12.4 X X -parking_lot_core@0.9.11 X X -parquet@55.2.0 X -paste@1.0.15 X X -pbjson@0.7.0 X -pbjson-build@0.7.0 X -pbjson-types@0.7.0 X -percent-encoding@2.3.2 X X -petgraph@0.7.1 X X -petgraph@0.8.2 X X -phf@0.12.1 X -phf_shared@0.12.1 X -pin-project-lite@0.2.16 X X -pin-utils@0.1.0 X X -pkg-config@0.3.32 X X -portable-atomic@1.11.1 X X -postgres-protocol@0.6.8 X X -postgres-types@0.2.9 X X -potential_utf@0.1.3 X -ppv-lite86@0.2.21 X X -prettyplease@0.2.37 X X -proc-macro2@1.0.101 X X -prost@0.13.5 X -prost-build@0.13.5 X -prost-derive@0.13.5 X -prost-types@0.13.5 X -psm@0.1.26 X X -quad-rand@0.2.3 X -quote@1.0.40 X X -r-efi@5.3.0 X X X -rand@0.8.5 X X -rand@0.9.2 X X -rand_chacha@0.3.1 X X -rand_chacha@0.9.0 X X -rand_core@0.6.4 X X -rand_core@0.9.3 X X -recursive@0.1.1 X -recursive-proc-macro-impl@0.1.1 X -redox_syscall@0.5.17 X -regex@1.11.2 X X -regex-automata@0.4.10 X X -regex-lite@0.1.7 X X -regex-syntax@0.8.6 X X -regress@0.10.4 X X -rle-decode-fast@1.0.3 X X -rust_decimal@1.38.0 X -rustc-demangle@0.1.26 X X -rustc_version@0.4.1 X X -rustix@1.0.8 X X X -rustversion@1.0.22 X X -ryu@1.0.20 X X -same-file@1.0.6 X X -schemars@0.8.22 X -schemars_derive@0.8.22 X -scopeguard@1.2.0 X X -semver@1.0.26 X X -seq-macro@0.3.6 X X -serde@1.0.219 X X -serde_bytes@0.11.17 X X -serde_derive@1.0.219 X X -serde_derive_internals@0.29.1 X X -serde_json@1.0.143 X X -serde_spanned@0.6.9 X X -serde_tokenstream@0.2.2 X -serde_yaml@0.9.34+deprecated X X -sha2@0.10.9 X X -shlex@1.3.0 X X -simdutf8@0.1.5 X X -similar@2.7.0 X -siphasher@1.0.1 X X -slab@0.4.11 X -smallvec@1.15.1 X X -snap@1.1.1 X -sqllogictest@0.28.4 X X -sqlparser@0.55.0 X -sqlparser_derive@0.3.0 X -stable_deref_trait@1.2.0 X X -stacker@0.1.21 X X -static_assertions@1.1.0 X X -stringprep@0.1.5 X X -strsim@0.11.1 X -strum@0.26.3 X -strum_macros@0.26.4 X -subst@0.3.8 X X -substrait@0.56.0 X -subtle@2.6.1 X -syn@2.0.106 X X -synstructure@0.13.2 X -tempfile@3.22.0 X X -thiserror@1.0.69 X X -thiserror@2.0.16 X X -thiserror-impl@1.0.69 X X -thiserror-impl@2.0.16 X X -thrift@0.17.0 X -tiny-keccak@2.0.2 X -tinystr@0.8.1 X -tinyvec@1.10.0 X X X -tinyvec_macros@0.1.1 X X X -tokio@1.47.1 X -tokio-macros@2.5.0 X -tokio-util@0.7.16 X -toml@0.8.23 X X -toml_datetime@0.6.11 X X -toml_edit@0.22.27 X X -toml_write@0.1.2 X X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X -twox-hash@2.1.2 X -typed-builder@0.19.1 X X -typed-builder-macro@0.19.1 X X -typenum@1.18.0 X X -typify@0.4.3 X -typify-impl@0.4.3 X -typify-macro@0.4.3 X -unicode-bidi@0.3.18 X X -unicode-ident@1.0.18 X X X -unicode-normalization@0.1.24 X X -unicode-properties@0.1.3 X X -unicode-segmentation@1.12.0 X X -unicode-width@0.1.14 X X -unicode-width@0.2.1 X X -unsafe-libyaml@0.2.11 X -url@2.5.7 X X -utf8_iter@1.0.4 X X -utf8parse@0.2.2 X X -uuid@1.18.1 X X -version_check@0.9.5 X X -walkdir@2.5.0 X X -wasi@0.11.1+wasi-snapshot-preview1 X X X -wasi@0.14.4+wasi-0.2.4 X X X -wasm-bindgen@0.2.101 X X -wasm-bindgen-backend@0.2.101 X X -wasm-bindgen-futures@0.4.51 X X -wasm-bindgen-macro@0.2.101 X X -wasm-bindgen-macro-support@0.2.101 X X -wasm-bindgen-shared@0.2.101 X X -web-sys@0.3.78 X X -web-time@1.1.0 X X -winapi-util@0.1.11 X X -windows-core@0.61.2 X X -windows-implement@0.60.0 X X -windows-interface@0.59.1 X X -windows-link@0.1.3 X X -windows-link@0.2.0 X X -windows-result@0.3.4 X X -windows-strings@0.4.2 X X -windows-sys@0.52.0 X X -windows-sys@0.59.0 X X -windows-sys@0.60.2 X X -windows-sys@0.61.0 X X -windows-targets@0.52.6 X X -windows-targets@0.53.3 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_gnullvm@0.53.0 X X -windows_aarch64_msvc@0.52.6 X X -windows_aarch64_msvc@0.53.0 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnu@0.53.0 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_gnullvm@0.53.0 X X -windows_i686_msvc@0.52.6 X X -windows_i686_msvc@0.53.0 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnu@0.53.0 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_gnullvm@0.53.0 X X -windows_x86_64_msvc@0.52.6 X X -windows_x86_64_msvc@0.53.0 X X -winnow@0.7.13 X -wit-bindgen@0.45.1 X X X -writeable@0.6.1 X -xz2@0.1.7 X X -yoke@0.8.0 X -yoke-derive@0.8.0 X -zerocopy@0.8.27 X X X -zerofrom@0.1.6 X -zerofrom-derive@0.1.6 X -zerotrie@0.2.2 X -zerovec@0.11.4 X -zerovec-derive@0.11.1 X -zlib-rs@0.5.2 X -zstd@0.13.3 X -zstd-safe@7.2.4 X X -zstd-sys@2.0.16+zstd.1.5.7 X X +crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 CC0-1.0 CDLA-Permissive-2.0 ISC LGPL-2.1-or-later MIT MIT-0 Unicode-3.0 Unlicense Zlib bzip2-1.0.6 +adler2@2.0.1 X X X +ahash@0.8.12 X X +aho-corasick@1.1.4 X X +alloc-no-stdlib@2.0.4 X +alloc-stdlib@0.2.2 X +allocator-api2@0.2.21 X X +android_system_properties@0.1.5 X X +anstream@0.6.21 X X +anstyle@1.0.13 X X +anstyle-parse@0.2.7 X X +anstyle-query@1.1.5 X X +anstyle-wincon@3.0.11 X X +anyhow@1.0.100 X X +apache-avro@0.20.0 X +apache-avro@0.21.0 X +ar_archive_writer@0.2.0 X +array-init@2.1.0 X X +arrayref@0.3.9 X +arrayvec@0.7.6 X X +arrow@57.1.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-csv@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-json@57.1.0 X +arrow-ord@57.1.0 X +arrow-row@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X +as-any@0.3.2 X X +async-compression@0.4.19 X X +async-lock@3.4.1 X X +async-recursion@1.1.1 X X +async-trait@0.1.89 X X +atoi@2.0.0 X +atomic-waker@1.1.2 X X +autocfg@1.5.0 X X +backon@1.6.0 X +base64@0.22.1 X X +bigdecimal@0.4.9 X X +bimap@0.6.3 X X +bitflags@2.10.0 X X +blake2@0.10.6 X X +blake3@1.8.2 X X X +block-buffer@0.10.4 X X +bon@3.8.1 X X +bon-macros@3.8.1 X X +brotli@8.0.2 X X +brotli-decompressor@5.0.0 X X +bumpalo@3.19.0 X X +bytemuck@1.24.0 X X X +byteorder@1.5.0 X X +bytes@1.11.0 X +bzip2@0.5.2 X X +bzip2@0.6.1 X X +bzip2-sys@0.1.13+1.0.8 X X +cc@1.2.49 X X +cfg-if@1.0.4 X X +chrono@0.4.42 X X +chrono-tz@0.10.4 X X +clap@4.5.53 X X +clap_builder@4.5.53 X X +clap_derive@4.5.49 X X +clap_lex@0.7.6 X X +colorchoice@1.0.4 X X +comfy-table@7.2.1 X +concurrent-queue@2.5.0 X X +console@0.16.1 X +const-oid@0.9.6 X X +const-random@0.1.18 X X +const-random-macro@0.1.16 X X +constant_time_eq@0.3.1 X X X +core-foundation-sys@0.8.7 X X +cpufeatures@0.2.17 X X +crc32c@0.6.8 X X +crc32fast@1.5.0 X X +crossbeam-channel@0.5.15 X X +crossbeam-epoch@0.9.18 X X +crossbeam-utils@0.8.21 X X +crunchy@0.2.4 X +crypto-common@0.1.7 X X +csv@1.4.0 X X +csv-core@0.1.13 X X +darling@0.20.11 X +darling@0.21.3 X +darling_core@0.20.11 X +darling_core@0.21.3 X +darling_macro@0.20.11 X +darling_macro@0.21.3 X +dashmap@6.1.0 X +datafusion@51.0.0 X +datafusion-catalog@51.0.0 X +datafusion-catalog-listing@51.0.0 X +datafusion-common@51.0.0 X +datafusion-common-runtime@51.0.0 X +datafusion-datasource@51.0.0 X +datafusion-datasource-arrow@51.0.0 X +datafusion-datasource-avro@51.0.0 X +datafusion-datasource-csv@51.0.0 X +datafusion-datasource-json@51.0.0 X +datafusion-datasource-parquet@51.0.0 X +datafusion-doc@51.0.0 X +datafusion-execution@51.0.0 X +datafusion-expr@51.0.0 X +datafusion-expr-common@51.0.0 X +datafusion-functions@51.0.0 X +datafusion-functions-aggregate@51.0.0 X +datafusion-functions-aggregate-common@51.0.0 X +datafusion-functions-nested@51.0.0 X +datafusion-functions-table@51.0.0 X +datafusion-functions-window@51.0.0 X +datafusion-functions-window-common@51.0.0 X +datafusion-macros@51.0.0 X +datafusion-optimizer@51.0.0 X +datafusion-physical-expr@51.0.0 X +datafusion-physical-expr-adapter@51.0.0 X +datafusion-physical-expr-common@51.0.0 X +datafusion-physical-optimizer@51.0.0 X +datafusion-physical-plan@51.0.0 X +datafusion-pruning@51.0.0 X +datafusion-session@51.0.0 X +datafusion-spark@51.0.0 X +datafusion-sql@51.0.0 X +datafusion-sqllogictest@51.0.0 X +datafusion-substrait@51.0.0 X +derive_builder@0.20.2 X X +derive_builder_core@0.20.2 X X +derive_builder_macro@0.20.2 X X +digest@0.10.7 X X +displaydoc@0.2.5 X X +dissimilar@1.0.10 X +dyn-clone@1.0.20 X X +educe@0.6.0 X +either@1.15.0 X X +encode_unicode@1.0.0 X X +enum-ordinalize@4.3.2 X +enum-ordinalize-derive@4.3.2 X +env_filter@0.1.4 X X +env_logger@0.11.8 X X +equivalent@1.0.2 X X +errno@0.3.14 X X +escape8259@0.5.3 X +event-listener@5.4.1 X X +event-listener-strategy@0.5.4 X X +expect-test@1.5.1 X X +fallible-iterator@0.2.0 X X +fastrand@2.3.0 X X +find-msvc-tools@0.1.5 X X +fixedbitset@0.5.7 X X +flatbuffers@25.9.23 X +flate2@1.1.5 X X +fnv@1.0.7 X X +foldhash@0.1.5 X +foldhash@0.2.0 X +form_urlencoded@1.2.2 X X +fs-err@3.2.0 X X +futures@0.3.31 X X +futures-channel@0.3.31 X X +futures-core@0.3.31 X X +futures-executor@0.3.31 X X +futures-io@0.3.31 X X +futures-macro@0.3.31 X X +futures-sink@0.3.31 X X +futures-task@0.3.31 X X +futures-timer@3.0.3 X X +futures-util@0.3.31 X X +generic-array@0.14.7 X +getrandom@0.2.16 X X +getrandom@0.3.4 X X +glob@0.3.3 X X +gloo-timers@0.3.0 X X +half@2.7.1 X X +hashbrown@0.14.5 X X +hashbrown@0.15.5 X X +hashbrown@0.16.1 X X +heck@0.5.0 X X +hex@0.4.3 X X +hmac@0.12.1 X X +home@0.5.11 X X +http@1.4.0 X X +http-body@1.0.1 X +http-body-util@0.1.3 X +httparse@1.10.1 X X +humantime@2.3.0 X X +hyper@1.8.1 X +hyper-rustls@0.27.7 X X X +hyper-util@0.1.19 X +iana-time-zone@0.1.64 X X +iana-time-zone-haiku@0.1.2 X X +iceberg@0.8.0 X +iceberg-datafusion@0.8.0 X +iceberg-sqllogictest@0.8.0 X +iceberg_test_utils@0.8.0 X +icu_collections@2.1.1 X +icu_locale_core@2.1.1 X +icu_normalizer@2.1.1 X +icu_normalizer_data@2.1.1 X +icu_properties@2.1.1 X +icu_properties_data@2.1.1 X +icu_provider@2.1.1 X +ident_case@1.0.1 X X +idna@1.1.0 X X +idna_adapter@1.2.1 X X +indexmap@2.12.1 X X +indicatif@0.18.3 X +integer-encoding@3.0.4 X +ipnet@2.11.0 X X +iri-string@0.7.9 X X +is_terminal_polyfill@1.70.2 X X +itertools@0.13.0 X X +itertools@0.14.0 X X +itoa@1.0.15 X X +jiff@0.2.16 X X +jiff-tzdb@0.1.4 X X +jiff-tzdb-platform@0.1.3 X X +jobserver@0.1.34 X X +js-sys@0.3.83 X X +lazy_static@1.5.0 X X +lexical-core@1.0.6 X X +lexical-parse-float@1.0.6 X X +lexical-parse-integer@1.0.6 X X +lexical-util@1.0.7 X X +lexical-write-float@1.0.6 X X +lexical-write-integer@1.0.6 X X +libbz2-rs-sys@0.2.2 X +libc@0.2.178 X X +libm@0.2.15 X +libtest-mimic@0.8.1 X X +libz-rs-sys@0.5.3 X +linux-raw-sys@0.11.0 X X X +litemap@0.8.1 X +lock_api@0.4.14 X X +log@0.4.29 X X +lz4_flex@0.12.0 X +lzma-sys@0.1.20 X X +md-5@0.10.6 X X +memchr@2.7.6 X X +miniz_oxide@0.8.9 X X X +mio@1.1.1 X +moka@0.12.11 X X +multimap@0.10.1 X X +murmur3@0.5.2 X X +nu-ansi-term@0.50.3 X +num-bigint@0.4.6 X X +num-complex@0.4.6 X X +num-integer@0.1.46 X X +num-traits@0.2.19 X X +object@0.32.2 X X +object_store@0.12.4 X X +once_cell@1.21.3 X X +once_cell_polyfill@1.70.2 X X +opendal@0.55.0 X +ordered-float@2.10.1 X +ordered-float@4.6.0 X +owo-colors@4.2.3 X +parking@2.2.1 X X +parking_lot@0.12.5 X X +parking_lot_core@0.9.12 X X +parquet@57.1.0 X +paste@1.0.15 X X +pbjson@0.8.0 X +pbjson-build@0.8.0 X +pbjson-types@0.8.0 X +percent-encoding@2.3.2 X X +petgraph@0.7.1 X X +petgraph@0.8.3 X X +phf@0.12.1 X +phf_shared@0.12.1 X +pin-project-lite@0.2.16 X X +pin-utils@0.1.0 X X +pkg-config@0.3.32 X X +portable-atomic@1.11.1 X X +portable-atomic-util@0.2.4 X X +postgres-protocol@0.6.9 X X +postgres-types@0.2.11 X X +potential_utf@0.1.4 X +ppv-lite86@0.2.21 X X +prettyplease@0.2.37 X X +proc-macro-crate@3.4.0 X X +proc-macro2@1.0.103 X X +prost@0.14.1 X +prost-build@0.14.1 X +prost-derive@0.14.1 X +prost-types@0.14.1 X +psm@0.1.28 X X +quad-rand@0.2.3 X +quick-xml@0.38.4 X +quote@1.0.42 X X +r-efi@5.3.0 X X X +rand@0.8.5 X X +rand@0.9.2 X X +rand_chacha@0.3.1 X X +rand_chacha@0.9.0 X X +rand_core@0.6.4 X X +rand_core@0.9.3 X X +recursive@0.1.1 X +recursive-proc-macro-impl@0.1.1 X +redox_syscall@0.5.18 X +regex@1.12.2 X X +regex-automata@0.4.13 X X +regex-lite@0.1.8 X X +regex-syntax@0.8.8 X X +regress@0.10.5 X X +relative-path@1.9.3 X X +reqsign@0.16.5 X +reqwest@0.12.25 X X +ring@0.17.14 X X +roaring@0.11.2 X X +rstest@0.26.1 X X +rstest_macros@0.26.1 X X +rust_decimal@1.39.0 X +rustc_version@0.4.1 X X +rustix@1.1.2 X X X +rustls@0.23.35 X X X +rustls-pki-types@1.13.1 X X +rustls-webpki@0.103.8 X +rustversion@1.0.22 X X +ryu@1.0.20 X X +same-file@1.0.6 X X +schemars@0.8.22 X +schemars_derive@0.8.22 X +scopeguard@1.2.0 X X +semver@1.0.27 X X +seq-macro@0.3.6 X X +serde@1.0.228 X X +serde_bytes@0.11.19 X X +serde_core@1.0.228 X X +serde_derive@1.0.228 X X +serde_derive_internals@0.29.1 X X +serde_json@1.0.145 X X +serde_repr@0.1.20 X X +serde_spanned@0.6.9 X X +serde_tokenstream@0.2.2 X +serde_urlencoded@0.7.1 X X +serde_with@3.16.1 X X +serde_with_macros@3.16.1 X X +serde_yaml@0.9.34+deprecated X X +sha1@0.10.6 X X +sha2@0.10.9 X X +sharded-slab@0.1.7 X +shlex@1.3.0 X X +simd-adler32@0.3.8 X +simdutf8@0.1.5 X X +similar@2.7.0 X +siphasher@1.0.1 X X +slab@0.4.11 X +smallvec@1.15.1 X X +snap@1.1.1 X +socket2@0.6.1 X X +sqllogictest@0.28.4 X X +sqlparser@0.59.0 X +sqlparser_derive@0.3.0 X +stable_deref_trait@1.2.1 X X +stacker@0.1.22 X X +stringprep@0.1.5 X X +strsim@0.11.1 X +strum@0.27.2 X +strum_macros@0.27.2 X +subst@0.3.8 X X +substrait@0.62.0 X +subtle@2.6.1 X +syn@2.0.111 X X +sync_wrapper@1.0.2 X +synstructure@0.13.2 X +tagptr@0.2.0 X X +tempfile@3.23.0 X X +thiserror@2.0.17 X X +thiserror-impl@2.0.17 X X +thread_local@1.1.9 X X +thrift@0.17.0 X +tiny-keccak@2.0.2 X +tinystr@0.8.2 X +tinyvec@1.10.0 X X X +tinyvec_macros@0.1.1 X X X +tokio@1.48.0 X +tokio-macros@2.6.0 X +tokio-rustls@0.26.4 X X +tokio-util@0.7.17 X +toml@0.8.23 X X +toml_datetime@0.6.11 X X +toml_datetime@0.7.3 X X +toml_edit@0.22.27 X X +toml_edit@0.23.9 X X +toml_parser@1.0.4 X X +toml_write@0.1.2 X X +tower@0.5.2 X +tower-http@0.6.8 X +tower-layer@0.3.3 X +tower-service@0.3.3 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X +tracing-log@0.2.0 X +tracing-subscriber@0.3.22 X +try-lock@0.2.5 X +twox-hash@2.1.2 X +typed-builder@0.20.1 X X +typed-builder-macro@0.20.1 X X +typenum@1.19.0 X X +typify@0.5.0 X +typify-impl@0.5.0 X +typify-macro@0.5.0 X +unicode-bidi@0.3.18 X X +unicode-ident@1.0.22 X X X +unicode-normalization@0.1.25 X X +unicode-properties@0.1.4 X X +unicode-segmentation@1.12.0 X X +unicode-width@0.1.14 X X +unicode-width@0.2.2 X X +unit-prefix@0.5.2 X +unsafe-libyaml@0.2.11 X +untrusted@0.9.0 X +url@2.5.7 X X +utf8_iter@1.0.4 X X +utf8parse@0.2.2 X X +uuid@1.19.0 X X +version_check@0.9.5 X X +walkdir@2.5.0 X X +want@0.3.1 X +wasi@0.11.1+wasi-snapshot-preview1 X X X +wasip2@1.0.1+wasi-0.2.4 X X X +wasm-bindgen@0.2.106 X X +wasm-bindgen-futures@0.4.56 X X +wasm-bindgen-macro@0.2.106 X X +wasm-bindgen-macro-support@0.2.106 X X +wasm-bindgen-shared@0.2.106 X X +wasm-streams@0.4.2 X X +web-sys@0.3.83 X X +web-time@1.1.0 X X +webpki-roots@1.0.4 X +winapi-util@0.1.11 X X +windows-core@0.62.2 X X +windows-implement@0.60.2 X X +windows-interface@0.59.3 X X +windows-link@0.2.1 X X +windows-result@0.4.1 X X +windows-strings@0.5.1 X X +windows-sys@0.52.0 X X +windows-sys@0.59.0 X X +windows-sys@0.60.2 X X +windows-sys@0.61.2 X X +windows-targets@0.52.6 X X +windows-targets@0.53.5 X X +windows_aarch64_gnullvm@0.52.6 X X +windows_aarch64_gnullvm@0.53.1 X X +windows_aarch64_msvc@0.52.6 X X +windows_aarch64_msvc@0.53.1 X X +windows_i686_gnu@0.52.6 X X +windows_i686_gnu@0.53.1 X X +windows_i686_gnullvm@0.52.6 X X +windows_i686_gnullvm@0.53.1 X X +windows_i686_msvc@0.52.6 X X +windows_i686_msvc@0.53.1 X X +windows_x86_64_gnu@0.52.6 X X +windows_x86_64_gnu@0.53.1 X X +windows_x86_64_gnullvm@0.52.6 X X +windows_x86_64_gnullvm@0.53.1 X X +windows_x86_64_msvc@0.52.6 X X +windows_x86_64_msvc@0.53.1 X X +winnow@0.7.14 X +wit-bindgen@0.46.0 X X X +writeable@0.6.2 X +xz2@0.1.7 X X +yoke@0.8.1 X +yoke-derive@0.8.1 X +zerocopy@0.8.31 X X X +zerocopy-derive@0.8.31 X X X +zerofrom@0.1.6 X +zerofrom-derive@0.1.6 X +zeroize@1.8.2 X X +zerotrie@0.2.3 X +zerovec@0.11.5 X +zerovec-derive@0.11.2 X +zlib-rs@0.5.3 X +zstd@0.13.3 X +zstd-safe@7.2.4 X X +zstd-sys@2.0.16+zstd.1.5.7 X X diff --git a/crates/test_utils/DEPENDENCIES.rust.tsv b/crates/test_utils/DEPENDENCIES.rust.tsv index ef7c315df7..879ead5921 100644 --- a/crates/test_utils/DEPENDENCIES.rust.tsv +++ b/crates/test_utils/DEPENDENCIES.rust.tsv @@ -1,30 +1,22 @@ crate Apache-2.0 MIT Unicode-3.0 -cfg-if@1.0.3 X X -iceberg_test_utils@0.7.0 X +cfg-if@1.0.4 X X +iceberg_test_utils@0.8.0 X lazy_static@1.5.0 X X -log@0.4.28 X X -nu-ansi-term@0.50.1 X +log@0.4.29 X X +nu-ansi-term@0.50.3 X once_cell@1.21.3 X X pin-project-lite@0.2.16 X X -proc-macro2@1.0.101 X X -quote@1.0.40 X X +proc-macro2@1.0.103 X X +quote@1.0.42 X X sharded-slab@0.1.7 X smallvec@1.15.1 X X -syn@2.0.106 X X +syn@2.0.111 X X thread_local@1.1.9 X X -tracing@0.1.41 X -tracing-attributes@0.1.30 X -tracing-core@0.1.34 X +tracing@0.1.43 X +tracing-attributes@0.1.31 X +tracing-core@0.1.35 X tracing-log@0.2.0 X -tracing-subscriber@0.3.20 X -unicode-ident@1.0.18 X X X -windows-sys@0.52.0 X X -windows-targets@0.52.6 X X -windows_aarch64_gnullvm@0.52.6 X X -windows_aarch64_msvc@0.52.6 X X -windows_i686_gnu@0.52.6 X X -windows_i686_gnullvm@0.52.6 X X -windows_i686_msvc@0.52.6 X X -windows_x86_64_gnu@0.52.6 X X -windows_x86_64_gnullvm@0.52.6 X X -windows_x86_64_msvc@0.52.6 X X +tracing-subscriber@0.3.22 X +unicode-ident@1.0.22 X X X +windows-link@0.2.1 X X +windows-sys@0.61.2 X X diff --git a/deny.toml b/deny.toml index 6d75c5d219..0f88ba6d0e 100644 --- a/deny.toml +++ b/deny.toml @@ -26,6 +26,7 @@ allow = [ "CC0-1.0", "Zlib", "CDLA-Permissive-2.0", + "bzip2-1.0.6", # Category-A: https://issues.apache.org/jira/browse/LEGAL-660 "Unicode-3.0", # Boost Software License Version 1.0 is allowed (Category-A): @@ -39,4 +40,4 @@ exceptions = [ { allow = ["MPL-2.0"], crate = "webpki-roots" }, { allow = ["MPL-2.0"], crate = "generational-arena" }, { allow = ["MPL-2.0"], crate = "option-ext" }, -] \ No newline at end of file +] From b7ba2e8348ef79eff868715f2b7cf4ce6256d4ea Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 17 Dec 2025 16:49:13 -0800 Subject: [PATCH 121/891] fix(website): Update expected messages (#1942) ## Which issue does this PR close? - A minor doc fix ## What changes are included in this PR? - Corrected the expected messages on the instructions to verify a release ## Are these changes tested? not needed --- website/src/release.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/src/release.md b/website/src/release.md index 78c0e57525..f3a5798a2b 100644 --- a/website/src/release.md +++ b/website/src/release.md @@ -325,13 +325,13 @@ After downloading them, here are the instructions on how to verify them. ```bash gpg --verify apache-iceberg-rust-*.tar.gz.asc ``` - Expects: `"apache-iceberg-rust-0.7.0.tar.gz: OK"` + Expects: `gpg: Good signature from ...` * Verify the checksums: ```bash shasum -a 512 -c apache-iceberg-rust-*.tar.gz.sha512 ``` - Expects: `gpg: Good signature from ...` + Expects: `"apache-iceberg-rust-...tar.gz: OK"` * Verify build and test: ```bash From a329a3b756710b71c49dfd4b43060a4cc2731d3f Mon Sep 17 00:00:00 2001 From: Lo Date: Mon, 22 Dec 2025 09:14:12 +0800 Subject: [PATCH 122/891] feat: Implement shared delete file loading and caching for ArrowReader (#1941) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Currently, ArrowReader instantiates a new CachingDeleteFileLoader (and consequently a new DeleteFilter) for each FileScanTask when calling load_deletes. This results in the DeleteFilter state being isolated per task. If multiple tasks reference the same delete file (common in positional deletes), that delete file is re-read and re-parsed for every task, leading to significant performance overhead and redundant I/O. Changes * Shared State: Moved the DeleteFilter instance into the CachingDeleteFileLoader struct. Since ArrowReader holds a single CachingDeleteFileLoader instance across its lifetime, the DeleteFilter state is now effectively shared across all file scan tasks processed by that reader. * Positional Delete Caching: Implemented a state machine for loading positional delete files (PosDelState) in DeleteFilter. * Added try_start_pos_del_load: Coordinates concurrent access to the same positional delete file. * Added finish_pos_del_load: Signals completion of loading. * Synchronization: Introduced a WaitFor state. Unlike equality deletes (which are accessed asynchronously), positional deletes are accessed synchronously by ArrowReader. Therefore, if a task encounters a file that is currently being loaded by another task, it must asynchronously wait (notify.notified().await) during the loading phase to ensure the data is fully populated before ArrowReader proceeds. * Refactoring: Updated load_file_for_task and related types in CachingDeleteFileLoader to support the new caching logic and carry file paths through the loading context. ## Are these changes tested? Added test_caching_delete_file_loader_caches_results to verify that repeated loads of the same delete file return shared memory objects --- .../src/arrow/caching_delete_file_loader.rs | 111 ++++++++++++++---- crates/iceberg/src/arrow/delete_filter.rs | 66 +++++++++++ 2 files changed, 155 insertions(+), 22 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index aceeae49f7..5d0b1da712 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -23,7 +23,7 @@ use arrow_array::{Array, ArrayRef, Int64Array, StringArray, StructArray}; use futures::{StreamExt, TryStreamExt}; use tokio::sync::oneshot::{Receiver, channel}; -use super::delete_filter::DeleteFilter; +use super::delete_filter::{DeleteFilter, PosDelLoadAction}; use crate::arrow::delete_file_loader::BasicDeleteFileLoader; use crate::arrow::{arrow_primitive_to_literal, arrow_schema_to_schema}; use crate::delete_vector::DeleteVector; @@ -42,13 +42,20 @@ use crate::{Error, ErrorKind, Result}; pub(crate) struct CachingDeleteFileLoader { basic_delete_file_loader: BasicDeleteFileLoader, concurrency_limit_data_files: usize, + /// Shared filter state to allow caching loaded deletes across multiple + /// calls to `load_deletes` (e.g., across multiple file scan tasks). + delete_filter: DeleteFilter, } // Intermediate context during processing of a delete file task. enum DeleteFileContext { // TODO: Delete Vector loader from Puffin files ExistingEqDel, - PosDels(ArrowRecordBatchStream), + ExistingPosDel, + PosDels { + file_path: String, + stream: ArrowRecordBatchStream, + }, FreshEqDel { batch_stream: ArrowRecordBatchStream, equality_ids: HashSet, @@ -59,8 +66,12 @@ enum DeleteFileContext { // Final result of the processing of a delete file task before // results are fully merged into the DeleteFileManager's state enum ParsedDeleteFileContext { - DelVecs(HashMap), + DelVecs { + file_path: String, + results: HashMap, + }, EqDel, + ExistingPosDel, } #[allow(unused_variables)] @@ -69,6 +80,7 @@ impl CachingDeleteFileLoader { CachingDeleteFileLoader { basic_delete_file_loader: BasicDeleteFileLoader::new(file_io), concurrency_limit_data_files, + delete_filter: DeleteFilter::default(), } } @@ -142,7 +154,6 @@ impl CachingDeleteFileLoader { schema: SchemaRef, ) -> Receiver> { let (tx, rx) = channel(); - let del_filter = DeleteFilter::default(); let stream_items = delete_file_entries .iter() @@ -150,14 +161,14 @@ impl CachingDeleteFileLoader { ( t.clone(), self.basic_delete_file_loader.clone(), - del_filter.clone(), + self.delete_filter.clone(), schema.clone(), ) }) .collect::>(); let task_stream = futures::stream::iter(stream_items); - let del_filter = del_filter.clone(); + let del_filter = self.delete_filter.clone(); let concurrency_limit_data_files = self.concurrency_limit_data_files; let basic_delete_file_loader = self.basic_delete_file_loader.clone(); crate::runtime::spawn(async move { @@ -165,7 +176,7 @@ impl CachingDeleteFileLoader { let mut del_filter = del_filter; let basic_delete_file_loader = basic_delete_file_loader.clone(); - let results: Vec = task_stream + let mut results_stream = task_stream .map(move |(task, file_io, del_filter, schema)| { let basic_delete_file_loader = basic_delete_file_loader.clone(); async move { @@ -181,15 +192,16 @@ impl CachingDeleteFileLoader { .map(move |ctx| { Ok(async { Self::parse_file_content_for_task(ctx.await?).await }) }) - .try_buffer_unordered(concurrency_limit_data_files) - .try_collect::>() - .await?; + .try_buffer_unordered(concurrency_limit_data_files); - for item in results { - if let ParsedDeleteFileContext::DelVecs(hash_map) = item { - for (data_file_path, delete_vector) in hash_map.into_iter() { + while let Some(item) = results_stream.next().await { + let item = item?; + if let ParsedDeleteFileContext::DelVecs { file_path, results } = item { + for (data_file_path, delete_vector) in results.into_iter() { del_filter.upsert_delete_vector(data_file_path, delete_vector); } + // Mark the positional delete file as fully loaded so waiters can proceed + del_filter.finish_pos_del_load(&file_path); } } @@ -210,11 +222,24 @@ impl CachingDeleteFileLoader { schema: SchemaRef, ) -> Result { match task.file_type { - DataContentType::PositionDeletes => Ok(DeleteFileContext::PosDels( - basic_delete_file_loader - .parquet_to_batch_stream(&task.file_path) - .await?, - )), + DataContentType::PositionDeletes => { + match del_filter.try_start_pos_del_load(&task.file_path) { + PosDelLoadAction::AlreadyLoaded => Ok(DeleteFileContext::ExistingPosDel), + PosDelLoadAction::WaitFor(notify) => { + // Positional deletes are accessed synchronously by ArrowReader. + // We must wait here to ensure the data is ready before returning, + // otherwise ArrowReader might get an empty/partial result. + notify.notified().await; + Ok(DeleteFileContext::ExistingPosDel) + } + PosDelLoadAction::Load => Ok(DeleteFileContext::PosDels { + file_path: task.file_path.clone(), + stream: basic_delete_file_loader + .parquet_to_batch_stream(&task.file_path) + .await?, + }), + } + } DataContentType::EqualityDeletes => { let Some(notify) = del_filter.try_start_eq_del_load(&task.file_path) else { @@ -255,10 +280,13 @@ impl CachingDeleteFileLoader { ) -> Result { match ctx { DeleteFileContext::ExistingEqDel => Ok(ParsedDeleteFileContext::EqDel), - DeleteFileContext::PosDels(batch_stream) => { - let del_vecs = - Self::parse_positional_deletes_record_batch_stream(batch_stream).await?; - Ok(ParsedDeleteFileContext::DelVecs(del_vecs)) + DeleteFileContext::ExistingPosDel => Ok(ParsedDeleteFileContext::ExistingPosDel), + DeleteFileContext::PosDels { file_path, stream } => { + let del_vecs = Self::parse_positional_deletes_record_batch_stream(stream).await?; + Ok(ParsedDeleteFileContext::DelVecs { + file_path, + results: del_vecs, + }) } DeleteFileContext::FreshEqDel { sender, @@ -979,4 +1007,43 @@ mod tests { assert!(result.is_ok()); } + + #[tokio::test] + async fn test_caching_delete_file_loader_caches_results() { + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path(); + let file_io = FileIO::from_path(table_location.as_os_str().to_str().unwrap()) + .unwrap() + .build() + .unwrap(); + + let delete_file_loader = CachingDeleteFileLoader::new(file_io.clone(), 10); + + let file_scan_tasks = setup(table_location); + + // Load deletes for the first time + let delete_filter_1 = delete_file_loader + .load_deletes(&file_scan_tasks[0].deletes, file_scan_tasks[0].schema_ref()) + .await + .unwrap() + .unwrap(); + + // Load deletes for the second time (same task/files) + let delete_filter_2 = delete_file_loader + .load_deletes(&file_scan_tasks[0].deletes, file_scan_tasks[0].schema_ref()) + .await + .unwrap() + .unwrap(); + + let dv1 = delete_filter_1 + .get_delete_vector(&file_scan_tasks[0]) + .unwrap(); + let dv2 = delete_filter_2 + .get_delete_vector(&file_scan_tasks[0]) + .unwrap(); + + // Verify that the delete vectors point to the same memory location, + // confirming that the second load reused the result from the first. + assert!(Arc::ptr_eq(&dv1, &dv2)); + } } diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index d05e028997..e6434e72ec 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -34,10 +34,23 @@ enum EqDelState { Loaded(Predicate), } +/// State tracking for positional delete files. +/// Unlike equality deletes, positional deletes must be fully loaded before +/// the ArrowReader proceeds because retrieval is synchronous and non-blocking. +#[derive(Debug)] +enum PosDelState { + /// The file is currently being loaded by a task. + /// The notifier allows other tasks to wait for completion. + Loading(Arc), + /// The file has been fully loaded and merged into the delete vector map. + Loaded, +} + #[derive(Debug, Default)] struct DeleteFileFilterState { delete_vectors: HashMap>>, equality_deletes: HashMap, + positional_deletes: HashMap, } #[derive(Clone, Debug, Default)] @@ -45,6 +58,18 @@ pub(crate) struct DeleteFilter { state: Arc>, } +/// Action to take when trying to start loading a positional delete file +pub(crate) enum PosDelLoadAction { + /// The file is not loaded, the caller should load it. + Load, + /// The file is already loaded, nothing to do. + AlreadyLoaded, + /// The file is currently being loaded by another task. + /// The caller *must* wait for this notifier to ensure data availability + /// before returning, as subsequent access (get_delete_vector) is synchronous. + WaitFor(Arc), +} + impl DeleteFilter { /// Retrieve a delete vector for the data file associated with a given file scan task pub(crate) fn get_delete_vector( @@ -82,6 +107,47 @@ impl DeleteFilter { Some(notifier) } + /// Attempts to mark a positional delete file as "loading". + /// + /// Returns an action dictating whether the caller should load the file, + /// wait for another task to load it, or do nothing. + pub(crate) fn try_start_pos_del_load(&self, file_path: &str) -> PosDelLoadAction { + let mut state = self.state.write().unwrap(); + + if let Some(state) = state.positional_deletes.get(file_path) { + match state { + PosDelState::Loaded => return PosDelLoadAction::AlreadyLoaded, + PosDelState::Loading(notify) => return PosDelLoadAction::WaitFor(notify.clone()), + } + } + + let notifier = Arc::new(Notify::new()); + state + .positional_deletes + .insert(file_path.to_string(), PosDelState::Loading(notifier)); + + PosDelLoadAction::Load + } + + /// Marks a positional delete file as successfully loaded and notifies any waiting tasks. + pub(crate) fn finish_pos_del_load(&self, file_path: &str) { + let notify = { + let mut state = self.state.write().unwrap(); + if let Some(PosDelState::Loading(notify)) = state + .positional_deletes + .insert(file_path.to_string(), PosDelState::Loaded) + { + Some(notify) + } else { + None + } + }; + + if let Some(notify) = notify { + notify.notify_waiters(); + } + } + /// Retrieve the equality delete predicate for a given eq delete file path pub(crate) async fn get_equality_delete_predicate_for_delete_file_path( &self, From de6ffd202f77b2c53fe298f92d4d33f1482cf8a8 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 22 Dec 2025 17:25:55 -0800 Subject: [PATCH 123/891] infra: license header check ignore target/ dir (#1954) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Following the [release verification](https://rust.iceberg.apache.org/release.html#how-to-verify-a-release) step by step. The target/ dir gets picked up by license header check. This PR adds the target/ dir to the ignore list. ## Are these changes tested? --- .licenserc.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.licenserc.yaml b/.licenserc.yaml index da87374c3b..0bcb65f3b7 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -31,6 +31,7 @@ header: - "**/DEPENDENCIES.*.tsv" # Release distributions - "dist/*" + - "target" - "Cargo.lock" - "bindings/python/uv.lock" - ".github/PULL_REQUEST_TEMPLATE.md" From 152aeb645332ba9962a4e7f8a633b8f409e9a21a Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 23 Dec 2025 15:58:48 -0800 Subject: [PATCH 124/891] infra: release script, validate proper ICEBERG_VERSION variable (#1956) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Add a check that the input `ICEBERG_VERSION` variable is in the proper format, i.e. `0.8.0`. Otherwise, the source directory and zip file will have the wrong name ## Are these changes tested? Added these 3 lines: ``` echo "Release version: $release_version" echo "Git branch: $git_branch" echo "Source directory: apache-iceberg-rust-$release_version" ``` This PR: ``` ➜ iceberg-rust git:(kevinjqliu/validate-version) ✗ ICEBERG_VERSION=rc.1 ./scripts/release.sh var is set to 'rc.1' Error: ICEBERG_VERSION (rc.1) must be in the format: .. ➜ iceberg-rust git:(kevinjqliu/validate-version) ✗ ICEBERG_VERSION=0.8.0 ./scripts/release.sh var is set to '0.8.0' Release version: 0.8.0 Git branch: release-0.8.0-rc.1 Source directory: apache-iceberg-rust-0.8.0 ``` main: ``` ➜ iceberg-rust git:(kevinjqliu/validate-version) ✗ ICEBERG_VERSION=rc.1 ./scripts/release.sh var is set to 'rc.1' Release version: rc.1 Git branch: release-rc.1-rc.1 Source directory: apache-iceberg-rust-rc.1 ➜ iceberg-rust git:(kevinjqliu/validate-version) ✗ ICEBERG_VERSION=0.8.0 ./scripts/release.sh var is set to '0.8.0' Release version: 0.8.0 Git branch: release-0.8.0-rc.1 Source directory: apache-iceberg-rust-0.8.0 ``` --- scripts/release.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/release.sh b/scripts/release.sh index a790cdd8d1..e4cee342d2 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -25,6 +25,12 @@ else echo "var is set to '$ICEBERG_VERSION'" fi +# Validate version format (e.g., 1.0.0) +if [[ ! "$ICEBERG_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "Error: ICEBERG_VERSION ($ICEBERG_VERSION) must be in the format: .." + exit 1 +fi + # tar source code release_version=${ICEBERG_VERSION} # rc versions From bc86d10c8ed85865cef0effae8f6ebca0ff3e358 Mon Sep 17 00:00:00 2001 From: Robert Mu Date: Wed, 24 Dec 2025 09:25:51 +0800 Subject: [PATCH 125/891] refactor(arrow): Rename parameter in delete_filter for clarity (#1955) The parameter `delete_file_path` in `get_delete_vector_for_path` was misleading because the function expects a data file path, not a delete file path. Renaming it to `data_file_path` accurately reflects its usage. --- crates/iceberg/src/arrow/delete_filter.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index e6434e72ec..4af9f6b6ff 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -82,12 +82,12 @@ impl DeleteFilter { /// Retrieve a delete vector for a data file pub(crate) fn get_delete_vector_for_path( &self, - delete_file_path: &str, + data_file_path: &str, ) -> Option>> { self.state .read() .ok() - .and_then(|st| st.delete_vectors.get(delete_file_path).cloned()) + .and_then(|st| st.delete_vectors.get(data_file_path).cloned()) } pub(crate) fn try_start_eq_del_load(&self, file_path: &str) -> Option> { From 5dc57643aebbe1be3ac36e29581fa7462bfdd2cd Mon Sep 17 00:00:00 2001 From: Andrea Bozzo Date: Thu, 25 Dec 2025 01:45:35 +0100 Subject: [PATCH 126/891] feat(sqllogictest): use serde derived structs for schedule parsing (#1953) This PR refactors the schedule file parsing in the sqllogictest crate to use serde-derived structs instead of manual TOML parsing, as requested in #1952. ### Changes - **New structs with serde derives:** - `ScheduleConfig` - top-level configuration parsed from TOML - `EngineConfig` - per-engine configuration with `#[serde(flatten)]` for extensibility - `EngineType` - enum of supported engine types - **Refactored parsing flow:** - `Schedule::from_file()` now uses `toml::from_str()` directly - Added `instantiate_engines()` to separate parsing from engine creation - Removed manual `parse_engines()` and `parse_steps()` functions - **Forward-compatibility:** - Uses `#[serde(flatten)]` to capture extra fields in `EngineConfig.extra` - This enables PR #1943 to easily add `catalog_type` and `catalog_properties` support ### Relation to #1943 This PR was suggested by @liurenjie1024 as a prerequisite to #1943 (dynamic catalog configuration). The `#[serde(flatten)]` approach allows #1943 to simply extract the catalog configuration from `EngineConfig.extra` without modifying the parsing logic. ### Testing - All existing tests pass - Added new unit tests for deserialization behavior - Integration test with `df_test.toml` passes unchanged Closes #1952 --- crates/sqllogictest/src/engine/datafusion.rs | 18 +- crates/sqllogictest/src/engine/mod.rs | 96 ++++++--- crates/sqllogictest/src/schedule.rs | 198 ++++++++++++------- 3 files changed, 207 insertions(+), 105 deletions(-) diff --git a/crates/sqllogictest/src/engine/datafusion.rs b/crates/sqllogictest/src/engine/datafusion.rs index e3402dfa97..e9f93287d8 100644 --- a/crates/sqllogictest/src/engine/datafusion.rs +++ b/crates/sqllogictest/src/engine/datafusion.rs @@ -27,9 +27,8 @@ use iceberg::spec::{NestedField, PrimitiveType, Schema, Transform, Type, Unbound use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation}; use iceberg_datafusion::IcebergCatalogProvider; use indicatif::ProgressBar; -use toml::Table as TomlTable; -use crate::engine::{EngineRunner, run_slt_with_runner}; +use crate::engine::{DatafusionCatalogConfig, EngineRunner, run_slt_with_runner}; use crate::error::Result; pub struct DataFusionEngine { @@ -59,12 +58,15 @@ impl EngineRunner for DataFusionEngine { } impl DataFusionEngine { - pub async fn new(config: TomlTable) -> Result { + pub async fn new(catalog_config: Option) -> Result { let session_config = SessionConfig::new() .with_target_partitions(4) .with_information_schema(true); let ctx = SessionContext::new_with_config(session_config); - ctx.register_catalog("default", Self::create_catalog(&config).await?); + ctx.register_catalog( + "default", + Self::create_catalog(catalog_config.as_ref()).await?, + ); Ok(Self { test_data_path: PathBuf::from("testdata"), @@ -72,9 +74,11 @@ impl DataFusionEngine { }) } - async fn create_catalog(_: &TomlTable) -> anyhow::Result> { - // TODO: support dynamic catalog configuration - // See: https://github.com/apache/iceberg-rust/issues/1780 + async fn create_catalog( + _catalog_config: Option<&DatafusionCatalogConfig>, + ) -> anyhow::Result> { + // TODO: Use catalog_config to load different catalog types via iceberg-catalog-loader + // See: https://github.com/apache/iceberg-rust/issues/1780 let catalog = MemoryCatalogBuilder::default() .load( "memory", diff --git a/crates/sqllogictest/src/engine/mod.rs b/crates/sqllogictest/src/engine/mod.rs index 724359fbe5..a276671401 100644 --- a/crates/sqllogictest/src/engine/mod.rs +++ b/crates/sqllogictest/src/engine/mod.rs @@ -17,29 +17,45 @@ mod datafusion; +use std::collections::HashMap; use std::path::Path; use anyhow::anyhow; +use serde::Deserialize; use sqllogictest::{AsyncDB, MakeConnection, Runner, parse_file}; -use toml::Table as TomlTable; use crate::engine::datafusion::DataFusionEngine; use crate::error::{Error, Result}; -const TYPE_DATAFUSION: &str = "datafusion"; +/// Configuration for the catalog used by the DataFusion engine +#[derive(Debug, Clone, Deserialize)] +pub struct DatafusionCatalogConfig { + /// Catalog type: "memory", "rest", "glue", "hms", "s3tables", "sql" + #[serde(rename = "type")] + pub catalog_type: String, + /// Catalog properties passed to the catalog loader + #[serde(default)] + pub props: HashMap, +} + +/// Engine configuration as a tagged enum +#[derive(Debug, Clone, Deserialize)] +#[serde(tag = "type", rename_all = "lowercase")] +pub enum EngineConfig { + Datafusion { + #[serde(default)] + catalog: Option, + }, +} #[async_trait::async_trait] pub trait EngineRunner: Send { async fn run_slt_file(&mut self, path: &Path) -> Result<()>; } -pub async fn load_engine_runner( - engine_type: &str, - cfg: TomlTable, -) -> Result> { - match engine_type { - TYPE_DATAFUSION => Ok(Box::new(DataFusionEngine::new(cfg).await?)), - _ => Err(anyhow::anyhow!("Unsupported engine type: {engine_type}").into()), +pub async fn load_engine_runner(config: EngineConfig) -> Result> { + match config { + EngineConfig::Datafusion { catalog } => Ok(Box::new(DataFusionEngine::new(catalog).await?)), } } @@ -65,29 +81,63 @@ where #[cfg(test)] mod tests { - use crate::engine::{TYPE_DATAFUSION, load_engine_runner}; + use crate::engine::{DatafusionCatalogConfig, EngineConfig, load_engine_runner}; - #[tokio::test] - async fn test_engine_invalid_type() { + #[test] + fn test_deserialize_engine_config() { + let input = r#"type = "datafusion""#; + + let config: EngineConfig = toml::from_str(input).unwrap(); + assert!(matches!(config, EngineConfig::Datafusion { catalog: None })); + } + + #[test] + fn test_deserialize_engine_config_with_catalog() { + let input = r#" + type = "datafusion" + + [catalog] + type = "rest" + + [catalog.props] + uri = "http://localhost:8181" + "#; + + let config: EngineConfig = toml::from_str(input).unwrap(); + match config { + EngineConfig::Datafusion { catalog: Some(cat) } => { + assert_eq!(cat.catalog_type, "rest"); + assert_eq!( + cat.props.get("uri"), + Some(&"http://localhost:8181".to_string()) + ); + } + _ => panic!("Expected Datafusion with catalog"), + } + } + + #[test] + fn test_deserialize_catalog_config() { let input = r#" - [engines] - random = { type = "random_engine", url = "http://localhost:8181" } + type = "memory" + + [props] + warehouse = "file:///tmp/warehouse" "#; - let tbl = toml::from_str(input).unwrap(); - let result = load_engine_runner("random_engine", tbl).await; - assert!(result.is_err()); + let config: DatafusionCatalogConfig = toml::from_str(input).unwrap(); + assert_eq!(config.catalog_type, "memory"); + assert_eq!( + config.props.get("warehouse"), + Some(&"file:///tmp/warehouse".to_string()) + ); } #[tokio::test] async fn test_load_datafusion() { - let input = r#" - [engines] - df = { type = "datafusion" } - "#; - let tbl = toml::from_str(input).unwrap(); - let result = load_engine_runner(TYPE_DATAFUSION, tbl).await; + let config = EngineConfig::Datafusion { catalog: None }; + let result = load_engine_runner(config).await; assert!(result.is_ok()); } } diff --git a/crates/sqllogictest/src/schedule.rs b/crates/sqllogictest/src/schedule.rs index 7c13ad4d12..25728a2968 100644 --- a/crates/sqllogictest/src/schedule.rs +++ b/crates/sqllogictest/src/schedule.rs @@ -21,10 +21,18 @@ use std::path::{Path, PathBuf}; use anyhow::{Context, anyhow}; use serde::{Deserialize, Serialize}; -use toml::{Table as TomlTable, Value}; use tracing::info; -use crate::engine::{EngineRunner, load_engine_runner}; +use crate::engine::{EngineConfig, EngineRunner, load_engine_runner}; + +/// Raw configuration parsed from the schedule TOML file +#[derive(Debug, Clone, Deserialize)] +pub struct ScheduleConfig { + /// Engine name to engine configuration + pub engines: HashMap, + /// List of test steps to run + pub steps: Vec, +} pub struct Schedule { /// Engine names to engine instances @@ -59,15 +67,27 @@ impl Schedule { pub async fn from_file>(path: P) -> anyhow::Result { let path_str = path.as_ref().to_string_lossy().to_string(); let content = read_to_string(path)?; - let toml_value = content.parse::()?; - let toml_table = toml_value - .as_table() - .ok_or_else(|| anyhow!("Schedule file must be a TOML table"))?; - let engines = Schedule::parse_engines(toml_table).await?; - let steps = Schedule::parse_steps(toml_table)?; + let config: ScheduleConfig = toml::from_str(&content) + .with_context(|| format!("Failed to parse schedule file: {path_str}"))?; - Ok(Self::new(engines, steps, path_str)) + let engines = Self::instantiate_engines(config.engines).await?; + + Ok(Self::new(engines, config.steps, path_str)) + } + + /// Instantiate engine runners from their configurations + async fn instantiate_engines( + configs: HashMap, + ) -> anyhow::Result>> { + let mut engines = HashMap::new(); + + for (name, config) in configs { + let engine = load_engine_runner(config).await?; + engines.insert(name, engine); + } + + Ok(engines) } pub async fn run(mut self) -> anyhow::Result<()> { @@ -105,103 +125,131 @@ impl Schedule { } Ok(()) } +} - async fn parse_engines( - table: &TomlTable, - ) -> anyhow::Result>> { - let engines_tbl = table - .get("engines") - .with_context(|| "Schedule file must have an 'engines' table")? - .as_table() - .ok_or_else(|| anyhow!("'engines' must be a table"))?; - - let mut engines = HashMap::new(); - - for (name, engine_val) in engines_tbl { - let cfg_tbl = engine_val - .as_table() - .ok_or_else(|| anyhow!("Config of engine '{name}' is not a table"))? - .clone(); - - let engine_type = cfg_tbl - .get("type") - .ok_or_else(|| anyhow::anyhow!("Engine {name} doesn't have a 'type' field"))? - .as_str() - .ok_or_else(|| anyhow::anyhow!("Engine {name} type must be a string"))?; - - let engine = load_engine_runner(engine_type, cfg_tbl.clone()).await?; - - if engines.insert(name.clone(), engine).is_some() { - return Err(anyhow!("Duplicate engine '{name}'")); - } - } +#[cfg(test)] +mod tests { + use crate::engine::EngineConfig; + use crate::schedule::ScheduleConfig; - Ok(engines) - } + #[test] + fn test_deserialize_schedule_config() { + let input = r#" + [engines] + df = { type = "datafusion" } - fn parse_steps(table: &TomlTable) -> anyhow::Result> { - let steps_val = table - .get("steps") - .with_context(|| "Schedule file must have a 'steps' array")?; + [[steps]] + engine = "df" + slt = "test.slt" + "#; - let steps: Vec = steps_val - .clone() - .try_into() - .with_context(|| "Failed to deserialize steps")?; + let config: ScheduleConfig = toml::from_str(input).unwrap(); - Ok(steps) + assert_eq!(config.engines.len(), 1); + assert!(config.engines.contains_key("df")); + assert!(matches!(config.engines["df"], EngineConfig::Datafusion { + catalog: None + })); + assert_eq!(config.steps.len(), 1); + assert_eq!(config.steps[0].engine, "df"); + assert_eq!(config.steps[0].slt, "test.slt"); } -} - -#[cfg(test)] -mod tests { - use toml::Table as TomlTable; - - use crate::schedule::Schedule; #[test] - fn test_parse_steps() { + fn test_deserialize_multiple_steps() { let input = r#" + [engines] + datafusion = { type = "datafusion" } + [[steps]] engine = "datafusion" slt = "test.slt" [[steps]] - engine = "spark" + engine = "datafusion" slt = "test2.slt" "#; - let tbl: TomlTable = toml::from_str(input).unwrap(); - let steps = Schedule::parse_steps(&tbl).unwrap(); + let config: ScheduleConfig = toml::from_str(input).unwrap(); - assert_eq!(steps.len(), 2); - assert_eq!(steps[0].engine, "datafusion"); - assert_eq!(steps[0].slt, "test.slt"); - assert_eq!(steps[1].engine, "spark"); - assert_eq!(steps[1].slt, "test2.slt"); + assert_eq!(config.steps.len(), 2); + assert_eq!(config.steps[0].engine, "datafusion"); + assert_eq!(config.steps[0].slt, "test.slt"); + assert_eq!(config.steps[1].engine, "datafusion"); + assert_eq!(config.steps[1].slt, "test2.slt"); } #[test] - fn test_parse_steps_empty() { + fn test_deserialize_with_catalog_config() { let input = r#" + [engines.df] + type = "datafusion" + + [engines.df.catalog] + type = "rest" + + [engines.df.catalog.props] + uri = "http://localhost:8181" + [[steps]] + engine = "df" + slt = "test.slt" "#; - let tbl: TomlTable = toml::from_str(input).unwrap(); - let steps = Schedule::parse_steps(&tbl); + let config: ScheduleConfig = toml::from_str(input).unwrap(); - assert!(steps.is_err()); + match &config.engines["df"] { + EngineConfig::Datafusion { catalog: Some(cat) } => { + assert_eq!(cat.catalog_type, "rest"); + assert_eq!( + cat.props.get("uri"), + Some(&"http://localhost:8181".to_string()) + ); + } + _ => panic!("Expected Datafusion with catalog config"), + } } - #[tokio::test] - async fn test_parse_engines_invalid_table() { - let toml_content = r#" - engines = "not_a_table" + #[test] + fn test_deserialize_missing_engine_type() { + let input = r#" + [engines] + df = { } + + [[steps]] + engine = "df" + slt = "test.slt" "#; - let table: TomlTable = toml::from_str(toml_content).unwrap(); - let result = Schedule::parse_engines(&table).await; + let result: Result = toml::from_str(input); + assert!(result.is_err()); + } + + #[test] + fn test_deserialize_invalid_engine_type() { + let input = r#" + [engines] + df = { type = "unknown_engine" } + + [[steps]] + engine = "df" + slt = "test.slt" + "#; + + let result: Result = toml::from_str(input); + assert!(result.is_err()); + } + + #[test] + fn test_deserialize_missing_step_fields() { + let input = r#" + [engines] + df = { type = "datafusion" } + + [[steps]] + "#; + let result: Result = toml::from_str(input); assert!(result.is_err()); } } From 99ca196a7a81343ac35d993d9db2ec59c626f389 Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Thu, 25 Dec 2025 09:03:32 +0800 Subject: [PATCH 127/891] fix: follow IEEE 754 totalOrder for `float` and `double` (#1959) ## Which issue does this PR close? - Closes #1951. ## What changes are included in this PR? For `F32` and `F64`, we use the `total_cmp` method for comparison. ## Are these changes tested? Yes, I added test cases to verify whether the comparison follows the IEEE 754 rules. Signed-off-by: StandingMan Co-authored-by: Renjie Liu --- crates/iceberg/src/spec/values/datum.rs | 36 ++++++------------------- crates/iceberg/src/spec/values/tests.rs | 25 +++++++++++++++++ 2 files changed, 33 insertions(+), 28 deletions(-) diff --git a/crates/iceberg/src/spec/values/datum.rs b/crates/iceberg/src/spec/values/datum.rs index cb60fb94e9..88209ae95c 100644 --- a/crates/iceberg/src/spec/values/datum.rs +++ b/crates/iceberg/src/spec/values/datum.rs @@ -166,36 +166,16 @@ impl<'de> Deserialize<'de> for Datum { // Compare following iceberg float ordering rules: // -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN -fn iceberg_float_cmp(a: T, b: T) -> Option { - if a.is_nan() && b.is_nan() { - return match (a.is_sign_negative(), b.is_sign_negative()) { - (true, false) => Some(Ordering::Less), - (false, true) => Some(Ordering::Greater), - _ => Some(Ordering::Equal), - }; - } - - if a.is_nan() { - return Some(if a.is_sign_negative() { - Ordering::Less - } else { - Ordering::Greater - }); - } - - if b.is_nan() { - return Some(if b.is_sign_negative() { - Ordering::Greater - } else { - Ordering::Less - }); - } +fn iceberg_float_cmp_f32(a: OrderedFloat, b: OrderedFloat) -> Option { + Some(a.total_cmp(&b)) +} - a.partial_cmp(&b) +fn iceberg_float_cmp_f64(a: OrderedFloat, b: OrderedFloat) -> Option { + Some(a.total_cmp(&b)) } impl PartialOrd for Datum { - fn partial_cmp(&self, other: &Self) -> Option { + fn partial_cmp(&self, other: &Self) -> Option { match (&self.literal, &other.literal, &self.r#type, &other.r#type) { // generate the arm with same type and same literal ( @@ -221,13 +201,13 @@ impl PartialOrd for Datum { PrimitiveLiteral::Float(other_val), PrimitiveType::Float, PrimitiveType::Float, - ) => iceberg_float_cmp(*val, *other_val), + ) => iceberg_float_cmp_f32(*val, *other_val), ( PrimitiveLiteral::Double(val), PrimitiveLiteral::Double(other_val), PrimitiveType::Double, PrimitiveType::Double, - ) => iceberg_float_cmp(*val, *other_val), + ) => iceberg_float_cmp_f64(*val, *other_val), ( PrimitiveLiteral::Int(val), PrimitiveLiteral::Int(other_val), diff --git a/crates/iceberg/src/spec/values/tests.rs b/crates/iceberg/src/spec/values/tests.rs index 73343a9a1a..bb10701d87 100644 --- a/crates/iceberg/src/spec/values/tests.rs +++ b/crates/iceberg/src/spec/values/tests.rs @@ -1293,6 +1293,31 @@ fn test_iceberg_float_order() { assert_eq!(double_sorted, double_expected); } +#[test] +fn test_negative_zero_less_than_positive_zero() { + { + let neg_zero = Datum::float(-0.0); + let pos_zero = Datum::float(0.0); + + assert_eq!( + neg_zero.partial_cmp(&pos_zero), + Some(std::cmp::Ordering::Less), + "IEEE 754 totalOrder requires -0.0 < +0.0 on F32" + ); + } + + { + let neg_zero = Datum::double(-0.0); + let pos_zero = Datum::double(0.0); + + assert_eq!( + neg_zero.partial_cmp(&pos_zero), + Some(std::cmp::Ordering::Less), + "IEEE 754 totalOrder requires -0.0 < +0.0 on F64" + ); + } +} + /// Test Date deserialization from JSON as number (days since epoch). /// /// This reproduces the scenario from Iceberg Java's TestAddFilesProcedure where: From de6a2b9cd2d364f11905fc26c2d318bdc03c2d52 Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Tue, 30 Dec 2025 09:14:58 +0800 Subject: [PATCH 128/891] fix: return proper error rather than persisting error message on snapshot (#1960) ## Which issue does this PR close? - Closes #1957. ## What changes are included in this PR? Avoid storing error message on snapshot, return error instead. ## Are these changes tested? Yes --------- Signed-off-by: StandingMan --- crates/iceberg/src/spec/snapshot.rs | 97 +++++++++++++++++++++++++++-- 1 file changed, 93 insertions(+), 4 deletions(-) diff --git a/crates/iceberg/src/spec/snapshot.rs b/crates/iceberg/src/spec/snapshot.rs index 5371cf68f2..270279988b 100644 --- a/crates/iceberg/src/spec/snapshot.rs +++ b/crates/iceberg/src/spec/snapshot.rs @@ -266,9 +266,9 @@ pub(super) mod _serde { use serde::{Deserialize, Serialize}; use super::{Operation, Snapshot, Summary}; - use crate::Error; use crate::spec::SchemaId; use crate::spec::snapshot::SnapshotRowRange; + use crate::{Error, ErrorKind}; #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] @@ -408,9 +408,19 @@ pub(super) mod _serde { timestamp_ms: v1.timestamp_ms, manifest_list: match (v1.manifest_list, v1.manifests) { (Some(file), None) => file, - (Some(_), Some(_)) => "Invalid v1 snapshot, when manifest list provided, manifest files should be omitted".to_string(), - (None, _) => "Unsupported v1 snapshot, only manifest list is supported".to_string() - }, + (Some(_), Some(_)) => { + return Err(Error::new( + ErrorKind::DataInvalid, + "Invalid v1 snapshot, when manifest list provided, manifest files should be omitted", + )); + } + (None, _) => { + return Err(Error::new( + ErrorKind::DataInvalid, + "Unsupported v1 snapshot, only manifest list is supported", + )); + } + }, summary: v1.summary.unwrap_or(Summary { operation: Operation::default(), additional_properties: HashMap::new(), @@ -517,6 +527,7 @@ mod tests { use chrono::{TimeZone, Utc}; + use crate::spec::TableMetadata; use crate::spec::snapshot::_serde::SnapshotV1; use crate::spec::snapshot::{Operation, Snapshot, Summary}; @@ -604,6 +615,84 @@ mod tests { ); } + #[test] + fn test_v1_snapshot_with_manifest_list_and_manifests() { + { + let metadata = r#" + { + "format-version": 1, + "table-uuid": "d20125c8-7284-442c-9aea-15fee620737c", + "location": "s3://bucket/test/location", + "last-updated-ms": 1700000000000, + "last-column-id": 1, + "schema": { + "type": "struct", + "fields": [ + {"id": 1, "name": "x", "required": true, "type": "long"} + ] + }, + "partition-spec": [], + "properties": {}, + "current-snapshot-id": 111111111, + "snapshots": [ + { + "snapshot-id": 111111111, + "timestamp-ms": 1600000000000, + "summary": {"operation": "append"}, + "manifest-list": "s3://bucket/metadata/snap-123.avro", + "manifests": ["s3://bucket/metadata/manifest-1.avro"] + } + ] + } + "#; + + let result_both_manifest_list_and_manifest_set = + serde_json::from_str::(metadata); + assert!(result_both_manifest_list_and_manifest_set.is_err()); + assert_eq!( + result_both_manifest_list_and_manifest_set + .unwrap_err() + .to_string(), + "DataInvalid => Invalid v1 snapshot, when manifest list provided, manifest files should be omitted" + ) + } + + { + let metadata = r#" + { + "format-version": 1, + "table-uuid": "d20125c8-7284-442c-9aea-15fee620737c", + "location": "s3://bucket/test/location", + "last-updated-ms": 1700000000000, + "last-column-id": 1, + "schema": { + "type": "struct", + "fields": [ + {"id": 1, "name": "x", "required": true, "type": "long"} + ] + }, + "partition-spec": [], + "properties": {}, + "current-snapshot-id": 111111111, + "snapshots": [ + { + "snapshot-id": 111111111, + "timestamp-ms": 1600000000000, + "summary": {"operation": "append"}, + "manifests": ["s3://bucket/metadata/manifest-1.avro"] + } + ] + } + "#; + let result_missing_manifest_list = serde_json::from_str::(metadata); + assert!(result_missing_manifest_list.is_err()); + assert_eq!( + result_missing_manifest_list.unwrap_err().to_string(), + "DataInvalid => Unsupported v1 snapshot, only manifest list is supported" + ) + } + } + #[test] fn test_snapshot_v1_to_v2_with_missing_summary() { use crate::spec::snapshot::_serde::SnapshotV1; From ef851524f16a604c05683051d732fa523b6e3bdc Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Mon, 29 Dec 2025 17:24:56 -0800 Subject: [PATCH 129/891] feat(arrow): Convert Arrow schema to Iceberg schema with auto assigned field ids (#1928) ## Which issue does this PR close? - Closes #1927 ## What changes are included in this PR? - Modified ArrowSchemaConverter to enable id reassignment - Added a new pub helper: `arrow_schema_to_schema_auto_assign_ids` ## Are these changes tested? Added uts --- crates/iceberg/src/arrow/schema.rs | 233 +++++++++++++++++- crates/iceberg/src/arrow/value.rs | 4 +- crates/iceberg/src/spec/mod.rs | 1 + crates/iceberg/src/spec/schema/mod.rs | 4 +- .../src/spec/table_metadata_builder.rs | 2 +- 5 files changed, 226 insertions(+), 18 deletions(-) diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index 4f4f083c73..9ee7897cb6 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -35,8 +35,8 @@ use uuid::Uuid; use crate::error::Result; use crate::spec::{ - Datum, ListType, MapType, NestedField, NestedFieldRef, PrimitiveLiteral, PrimitiveType, Schema, - SchemaVisitor, StructType, Type, + Datum, FIRST_FIELD_ID, ListType, MapType, NestedField, NestedFieldRef, PrimitiveLiteral, + PrimitiveType, Schema, SchemaVisitor, StructType, Type, }; use crate::{Error, ErrorKind}; @@ -221,6 +221,19 @@ pub fn arrow_schema_to_schema(schema: &ArrowSchema) -> Result { visit_schema(schema, &mut visitor) } +/// Convert Arrow schema to Iceberg schema with automatically assigned field IDs. +/// +/// Unlike [`arrow_schema_to_schema`], this function does not require field IDs in the Arrow +/// schema metadata. Instead, it automatically assigns unique field IDs starting from 1, +/// following Iceberg's field ID assignment rules. +/// +/// This is useful when converting Arrow schemas that don't originate from Iceberg tables, +/// such as schemas from DataFusion or other Arrow-based systems. +pub fn arrow_schema_to_schema_auto_assign_ids(schema: &ArrowSchema) -> Result { + let mut visitor = ArrowSchemaConverter::new_with_field_ids_from(FIRST_FIELD_ID); + visit_schema(schema, &mut visitor) +} + /// Convert Arrow type to iceberg type. pub fn arrow_type_to_type(ty: &DataType) -> Result { let mut visitor = ArrowSchemaConverter::new(); @@ -229,7 +242,7 @@ pub fn arrow_type_to_type(ty: &DataType) -> Result { const ARROW_FIELD_DOC_KEY: &str = "doc"; -pub(super) fn get_field_id(field: &Field) -> Result { +pub(super) fn get_field_id_from_metadata(field: &Field) -> Result { if let Some(value) = field.metadata().get(PARQUET_FIELD_ID_META_KEY) { return value.parse::().map_err(|e| { Error::new( @@ -253,19 +266,55 @@ fn get_field_doc(field: &Field) -> Option { None } -struct ArrowSchemaConverter; +struct ArrowSchemaConverter { + /// When set, the schema builder will reassign field IDs starting from this value + /// using level-order traversal (breadth-first). + reassign_field_ids_from: Option, + /// Generates unique placeholder IDs for fields before reassignment. + /// Required because `ReassignFieldIds` builds an old-to-new ID mapping + /// that expects unique input IDs. + next_field_id: i32, +} impl ArrowSchemaConverter { fn new() -> Self { - Self {} + Self { + reassign_field_ids_from: None, + next_field_id: 0, + } + } + + fn new_with_field_ids_from(start_from: i32) -> Self { + Self { + reassign_field_ids_from: Some(start_from), + next_field_id: 0, + } + } + + fn get_field_id(&mut self, field: &Field) -> Result { + if self.reassign_field_ids_from.is_some() { + // Field IDs will be reassigned by the schema builder. + // We need unique temporary IDs because ReassignFieldIds builds an + // old->new ID mapping that requires unique input IDs. + let temp_id = self.next_field_id; + self.next_field_id += 1; + Ok(temp_id) + } else { + // Get field ID from arrow field metadata + get_field_id_from_metadata(field) + } } - fn convert_fields(fields: &Fields, field_results: &[Type]) -> Result> { + fn convert_fields( + &mut self, + fields: &Fields, + field_results: &[Type], + ) -> Result> { let mut results = Vec::with_capacity(fields.len()); for i in 0..fields.len() { let field = &fields[i]; let field_type = &field_results[i]; - let id = get_field_id(field)?; + let id = self.get_field_id(field)?; let doc = get_field_doc(field); let nested_field = NestedField { id, @@ -287,13 +336,16 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { type U = Schema; fn schema(&mut self, schema: &ArrowSchema, values: Vec) -> Result { - let fields = Self::convert_fields(schema.fields(), &values)?; - let builder = Schema::builder().with_fields(fields); + let fields = self.convert_fields(schema.fields(), &values)?; + let mut builder = Schema::builder().with_fields(fields); + if let Some(start_from) = self.reassign_field_ids_from { + builder = builder.with_reassigned_field_ids(start_from) + } builder.build() } fn r#struct(&mut self, fields: &Fields, results: Vec) -> Result { - let fields = Self::convert_fields(fields, &results)?; + let fields = self.convert_fields(fields, &results)?; Ok(Type::Struct(StructType::new(fields))) } @@ -310,7 +362,7 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } }; - let id = get_field_id(element_field)?; + let id = self.get_field_id(element_field)?; let doc = get_field_doc(element_field); let mut element_field = NestedField::list_element(id, value.clone(), !element_field.is_nullable()); @@ -335,7 +387,7 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { let key_field = &fields[0]; let value_field = &fields[1]; - let key_id = get_field_id(key_field)?; + let key_id = self.get_field_id(key_field)?; let key_doc = get_field_doc(key_field); let mut key_field = NestedField::map_key_element(key_id, key_value.clone()); if let Some(doc) = key_doc { @@ -343,7 +395,7 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } let key_field = Arc::new(key_field); - let value_id = get_field_id(value_field)?; + let value_id = self.get_field_id(value_field)?; let value_doc = get_field_doc(value_field); let mut value_field = NestedField::map_value_element( value_id, @@ -1932,4 +1984,159 @@ mod tests { assert_eq!(array.value(0), [66u8; 16]); } } + + #[test] + fn test_arrow_schema_to_schema_with_field_id() { + // Create a complex Arrow schema without field ID metadata + // Including: primitives, list, nested struct, map, and nested list of structs + let arrow_schema = ArrowSchema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new("name", DataType::Utf8, true), + Field::new("price", DataType::Decimal128(10, 2), false), + Field::new( + "created_at", + DataType::Timestamp(TimeUnit::Microsecond, Some("+00:00".into())), + true, + ), + Field::new( + "tags", + DataType::List(Arc::new(Field::new("item", DataType::Utf8, true))), + true, + ), + Field::new( + "address", + DataType::Struct(Fields::from(vec![ + Field::new("street", DataType::Utf8, true), + Field::new("city", DataType::Utf8, false), + Field::new("zip", DataType::Int32, true), + ])), + true, + ), + Field::new( + "attributes", + DataType::Map( + Arc::new(Field::new( + DEFAULT_MAP_FIELD_NAME, + DataType::Struct(Fields::from(vec![ + Field::new("key", DataType::Utf8, false), + Field::new("value", DataType::Utf8, true), + ])), + false, + )), + false, + ), + true, + ), + Field::new( + "orders", + DataType::List(Arc::new(Field::new( + "element", + DataType::Struct(Fields::from(vec![ + Field::new("order_id", DataType::Int64, false), + Field::new("amount", DataType::Float64, false), + ])), + true, + ))), + true, + ), + ]); + + let schema = arrow_schema_to_schema_auto_assign_ids(&arrow_schema).unwrap(); + + // Build expected schema with exact field IDs following level-order assignment: + // Level 0: id=1, name=2, price=3, created_at=4, tags=5, address=6, attributes=7, orders=8 + // Level 1: tags.element=9, address.{street=10,city=11,zip=12}, attributes.{key=13,value=14}, orders.element=15 + // Level 2: orders.element.{order_id=16,amount=17} + let expected = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required( + 3, + "price", + Type::Primitive(PrimitiveType::Decimal { + precision: 10, + scale: 2, + }), + ) + .into(), + NestedField::optional(4, "created_at", Type::Primitive(PrimitiveType::Timestamptz)) + .into(), + NestedField::optional( + 5, + "tags", + Type::List(ListType { + element_field: NestedField::list_element( + 9, + Type::Primitive(PrimitiveType::String), + false, + ) + .into(), + }), + ) + .into(), + NestedField::optional( + 6, + "address", + Type::Struct(StructType::new(vec![ + NestedField::optional(10, "street", Type::Primitive(PrimitiveType::String)) + .into(), + NestedField::required(11, "city", Type::Primitive(PrimitiveType::String)) + .into(), + NestedField::optional(12, "zip", Type::Primitive(PrimitiveType::Int)) + .into(), + ])), + ) + .into(), + NestedField::optional( + 7, + "attributes", + Type::Map(MapType { + key_field: NestedField::map_key_element( + 13, + Type::Primitive(PrimitiveType::String), + ) + .into(), + value_field: NestedField::map_value_element( + 14, + Type::Primitive(PrimitiveType::String), + false, + ) + .into(), + }), + ) + .into(), + NestedField::optional( + 8, + "orders", + Type::List(ListType { + element_field: NestedField::list_element( + 15, + Type::Struct(StructType::new(vec![ + NestedField::required( + 16, + "order_id", + Type::Primitive(PrimitiveType::Long), + ) + .into(), + NestedField::required( + 17, + "amount", + Type::Primitive(PrimitiveType::Double), + ) + .into(), + ])), + false, + ) + .into(), + }), + ) + .into(), + ]) + .build() + .unwrap(); + + pretty_assertions::assert_eq!(schema, expected); + assert_eq!(schema.highest_field_id(), 17); + } } diff --git a/crates/iceberg/src/arrow/value.rs b/crates/iceberg/src/arrow/value.rs index 190aba08e8..30b47d83fc 100644 --- a/crates/iceberg/src/arrow/value.rs +++ b/crates/iceberg/src/arrow/value.rs @@ -27,7 +27,7 @@ use arrow_buffer::NullBuffer; use arrow_schema::{DataType, FieldRef}; use uuid::Uuid; -use super::get_field_id; +use super::get_field_id_from_metadata; use crate::spec::{ ListType, Literal, Map, MapType, NestedField, PartnerAccessor, PrimitiveLiteral, PrimitiveType, SchemaWithPartnerVisitor, Struct, StructType, Type, visit_struct_with_partner, @@ -450,7 +450,7 @@ impl FieldMatchMode { /// Determines if an Arrow field matches an Iceberg field based on the matching mode. pub fn match_field(&self, arrow_field: &FieldRef, iceberg_field: &NestedField) -> bool { match self { - FieldMatchMode::Id => get_field_id(arrow_field) + FieldMatchMode::Id => get_field_id_from_metadata(arrow_field) .map(|id| id == iceberg_field.id) .unwrap_or(false), FieldMatchMode::Name => arrow_field.name() == &iceberg_field.name, diff --git a/crates/iceberg/src/spec/mod.rs b/crates/iceberg/src/spec/mod.rs index 44b35e5a6b..a2b540f08b 100644 --- a/crates/iceberg/src/spec/mod.rs +++ b/crates/iceberg/src/spec/mod.rs @@ -49,6 +49,7 @@ pub use snapshot_summary::*; pub use sort::*; pub use statistic_file::*; pub use table_metadata::*; +pub(crate) use table_metadata_builder::FIRST_FIELD_ID; pub use table_properties::*; pub use transform::*; pub use values::*; diff --git a/crates/iceberg/src/spec/schema/mod.rs b/crates/iceberg/src/spec/schema/mod.rs index 7080b6e700..13ad41818b 100644 --- a/crates/iceberg/src/spec/schema/mod.rs +++ b/crates/iceberg/src/spec/schema/mod.rs @@ -102,8 +102,8 @@ impl SchemaBuilder { /// Reassignment starts from the field-id specified in `start_from` (inclusive). /// /// All specified aliases and identifier fields will be updated to the new field-ids. - pub(crate) fn with_reassigned_field_ids(mut self, start_from: u32) -> Self { - self.reassign_field_ids_from = Some(start_from.try_into().unwrap_or(i32::MAX)); + pub(crate) fn with_reassigned_field_ids(mut self, start_from: i32) -> Self { + self.reassign_field_ids_from = Some(start_from); self } diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index eee4fec345..3db327d48a 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -31,7 +31,7 @@ use crate::error::{Error, ErrorKind, Result}; use crate::spec::{EncryptedKey, INITIAL_ROW_ID, MIN_FORMAT_VERSION_ROW_LINEAGE}; use crate::{TableCreation, TableUpdate}; -const FIRST_FIELD_ID: u32 = 1; +pub(crate) const FIRST_FIELD_ID: i32 = 1; /// Manipulating table metadata. /// From a4054140d9ea66022a11835afa3a68bc4eb72cec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Eickler?= <797483+eickler@users.noreply.github.com> Date: Tue, 30 Dec 2025 02:32:38 +0100 Subject: [PATCH 130/891] fix: MemoryCatalog to return absolute NamespaceIdents (#1970) ## Which issue does this PR close? - Closes #1969. ## What changes are included in this PR? The change makes `list_namespaces(parent)` return an absolute namespace identifier instead of a relative. ## Are these changes tested? The associated unit tests are updated with the same behaviour as SqlCatalog. --- crates/iceberg/src/catalog/memory/catalog.rs | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/crates/iceberg/src/catalog/memory/catalog.rs b/crates/iceberg/src/catalog/memory/catalog.rs index cfa3dc6b52..df0299acb2 100644 --- a/crates/iceberg/src/catalog/memory/catalog.rs +++ b/crates/iceberg/src/catalog/memory/catalog.rs @@ -163,8 +163,12 @@ impl Catalog for MemoryCatalog { let namespaces = root_namespace_state .list_namespaces_under(parent_namespace_ident)? .into_iter() - .map(|name| NamespaceIdent::new(name.to_string())) - .collect_vec(); + .map(|name| { + let mut names = parent_namespace_ident.iter().cloned().collect::>(); + names.push(name.to_string()); + NamespaceIdent::from_vec(names) + }) + .collect::>>()?; Ok(namespaces) } @@ -599,7 +603,7 @@ pub(crate) mod tests { .list_namespaces(Some(&namespace_ident_1)) .await .unwrap(), - vec![NamespaceIdent::new("b".into())] + vec![namespace_ident_2] ); } @@ -628,9 +632,9 @@ pub(crate) mod tests { .unwrap() ), to_set(vec![ - NamespaceIdent::new("a".into()), - NamespaceIdent::new("b".into()), - NamespaceIdent::new("c".into()), + namespace_ident_2, + namespace_ident_3, + namespace_ident_4, ]) ); } From 2301352019bc9f5cdfd488bf829f92c6c3d0da25 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Sun, 4 Jan 2026 17:57:47 -0800 Subject: [PATCH 131/891] fix(spec): Include delete file content to V3 manifest (#1979) ## Which issue does this PR close? - Closes #1973 ## What changes are included in this PR? - Write `content` to V3 manifest so the field is preserved correctly for delete entry ## Are these changes tested? Yes --- crates/iceberg/src/spec/manifest/writer.rs | 98 +++++++++++++++++++++- 1 file changed, 95 insertions(+), 3 deletions(-) diff --git a/crates/iceberg/src/spec/manifest/writer.rs b/crates/iceberg/src/spec/manifest/writer.rs index 2fb6a42062..0669651603 100644 --- a/crates/iceberg/src/spec/manifest/writer.rs +++ b/crates/iceberg/src/spec/manifest/writer.rs @@ -437,9 +437,12 @@ impl ManifestWriter { "format-version".to_string(), (self.metadata.format_version as u8).to_string(), )?; - if self.metadata.format_version == FormatVersion::V2 { - avro_writer - .add_user_metadata("content".to_string(), self.metadata.content.to_string())?; + match self.metadata.format_version { + FormatVersion::V1 => {} + FormatVersion::V2 | FormatVersion::V3 => { + avro_writer + .add_user_metadata("content".to_string(), self.metadata.content.to_string())?; + } } let partition_summary = self.construct_partition_summaries(&partition_type)?; @@ -708,4 +711,93 @@ mod tests { entries[0].file_sequence_number = None; assert_eq!(actual_manifest, Manifest::new(metadata, entries)); } + + #[tokio::test] + async fn test_v3_delete_manifest_delete_file_roundtrip() { + let schema = Arc::new( + Schema::builder() + .with_fields(vec![ + Arc::new(NestedField::optional( + 1, + "id", + Type::Primitive(PrimitiveType::Long), + )), + Arc::new(NestedField::optional( + 2, + "data", + Type::Primitive(PrimitiveType::String), + )), + ]) + .build() + .unwrap(), + ); + + let partition_spec = PartitionSpec::builder(schema.clone()) + .with_spec_id(0) + .build() + .unwrap(); + + // Create a position delete file entry + let delete_entry = ManifestEntry { + status: ManifestStatus::Added, + snapshot_id: None, + sequence_number: None, + file_sequence_number: None, + data_file: DataFile { + content: DataContentType::PositionDeletes, + file_path: "s3://bucket/table/data/delete-00000.parquet".to_string(), + file_format: DataFileFormat::Parquet, + partition: Struct::empty(), + record_count: 10, + file_size_in_bytes: 1024, + column_sizes: HashMap::new(), + value_counts: HashMap::new(), + null_value_counts: HashMap::new(), + nan_value_counts: HashMap::new(), + lower_bounds: HashMap::new(), + upper_bounds: HashMap::new(), + key_metadata: None, + split_offsets: None, + equality_ids: None, + sort_order_id: None, + partition_spec_id: 0, + first_row_id: None, + referenced_data_file: None, + content_offset: None, + content_size_in_bytes: None, + }, + }; + + // Write a V3 delete manifest + let tmp_dir = TempDir::new().unwrap(); + let path = tmp_dir.path().join("v3_delete_manifest.avro"); + let io = FileIOBuilder::new_fs_io().build().unwrap(); + let output_file = io.new_output(path.to_str().unwrap()).unwrap(); + + let mut writer = ManifestWriterBuilder::new( + output_file, + Some(1), + None, + schema.clone(), + partition_spec.clone(), + ) + .build_v3_deletes(); + + writer.add_entry(delete_entry).unwrap(); + let manifest_file = writer.write_manifest_file().await.unwrap(); + + // The returned ManifestFile correctly reports Deletes content + assert_eq!(manifest_file.content, ManifestContentType::Deletes); + + // Read back the manifest file + let actual_manifest = + Manifest::parse_avro(fs::read(&path).expect("read_file must succeed").as_slice()) + .unwrap(); + + // Verify the content type is correctly preserved as Deletes + assert_eq!( + actual_manifest.metadata().content, + ManifestContentType::Deletes, + ); + } } From a321fe59ed5fdbb14f4404f0ecc50d9c30cbf942 Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Mon, 5 Jan 2026 15:04:23 +0800 Subject: [PATCH 132/891] fix: fix typo check error (#1989) ## Which issue does this PR close? - Closes #1986. ## What changes are included in this PR? ## Are these changes tested? Signed-off-by: StandingMan --- .typos.toml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.typos.toml b/.typos.toml index 9363f17c9a..407ce8168c 100644 --- a/.typos.toml +++ b/.typos.toml @@ -15,5 +15,8 @@ # specific language governing permissions and limitations # under the License. +[type.rust] +extend-ignore-identifiers-re = ["^bimap$"] + [files] extend-exclude = ["**/testdata", "CHANGELOG.md"] From 4d09ba216657d38102b5a33e02f9e7764cbdeba1 Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Mon, 5 Jan 2026 19:45:58 +0800 Subject: [PATCH 133/891] Fix ci audit failure (#1988) --- .cargo/audit.toml | 8 ++++++++ .github/workflows/audit.yml | 1 - 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.cargo/audit.toml b/.cargo/audit.toml index a46052f3b5..09e2d35c50 100644 --- a/.cargo/audit.toml +++ b/.cargo/audit.toml @@ -25,4 +25,12 @@ ignore = [ # # Introduced by hive_metastore, tracked at https://github.com/cloudwego/pilota/issues/293 "RUSTSEC-2024-0388", + # `paste` is unmaintained; consider using an alternative + # + # Introduced by hive_metastore, tracked at https://github.com/cloudwego/pilota/issues/293 + "RUSTSEC-2024-0436", + # `rustls-pemfile` is unmaintained + # + # Introduced by object_store, see https://github.com/apache/arrow-rs-object-store/issues/564 + "RUSTSEC-2025-0134", ] diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index 28c8a3db75..fc6b1224e8 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -48,4 +48,3 @@ jobs: - uses: rustsec/audit-check@v2.0.0 with: token: ${{ secrets.GITHUB_TOKEN }} - ignore: RUSTSEC-2024-0436 From 9844638c48e2d4b123c9df4f89330cfd1e8bfcba Mon Sep 17 00:00:00 2001 From: Alan Tang Date: Tue, 6 Jan 2026 10:43:11 +0800 Subject: [PATCH 134/891] feat: make FanoutWriter writer configurable (#1962) ## Which issue does this PR close? - Closes #1834. ## What changes are included in this PR? - Fellow on #1872. ## Are these changes tested? --------- Signed-off-by: StandingMan --- crates/iceberg/src/spec/table_properties.rs | 12 ++++++++++ .../datafusion/src/physical_plan/write.rs | 24 +++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/crates/iceberg/src/spec/table_properties.rs b/crates/iceberg/src/spec/table_properties.rs index 4975456010..413604f51c 100644 --- a/crates/iceberg/src/spec/table_properties.rs +++ b/crates/iceberg/src/spec/table_properties.rs @@ -49,6 +49,8 @@ pub struct TableProperties { pub write_format_default: String, /// The target file size for files. pub write_target_file_size_bytes: usize, + /// Whether to use `FanoutWriter` for partitioned tables. + pub write_datafusion_fanout_enabled: bool, } impl TableProperties { @@ -137,6 +139,11 @@ impl TableProperties { pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES: &str = "write.target-file-size-bytes"; /// Default target file size pub const PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT: usize = 512 * 1024 * 1024; // 512 MB + /// Whether to use `FanoutWriter` for partitioned tables (handles unsorted data). + /// If false, uses `ClusteredWriter` (requires sorted data, more memory efficient). + pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled"; + /// Default value for fanout writer enabled + pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT: bool = true; } impl TryFrom<&HashMap> for TableProperties { @@ -175,6 +182,11 @@ impl TryFrom<&HashMap> for TableProperties { TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES, TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, )?, + write_datafusion_fanout_enabled: parse_property( + props, + TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED, + TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT, + )?, }) } } diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index 9eb53c235f..fdfddf877b 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -266,8 +266,28 @@ impl ExecutionPlan for IcebergWriteExec { let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); // Create TaskWriter - // TODO: Make fanout_enabled configurable via table properties - let fanout_enabled = true; + let fanout_enabled = self + .table + .metadata() + .properties() + .get(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED) + .map(|value| { + value + .parse::() + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Invalid value for {}, expected 'true' or 'false'", + TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED + ), + ) + .with_source(e) + }) + .map_err(to_datafusion_error) + }) + .transpose()? + .unwrap_or(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT); let schema = self.table.metadata().current_schema().clone(); let partition_spec = self.table.metadata().default_partition_spec().clone(); let task_writer = TaskWriter::try_new( From 434ab9603e7cd9969b45fd54f1d6a3c40951145f Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Wed, 7 Jan 2026 08:35:33 +0800 Subject: [PATCH 135/891] fix: Address RUSTSEC-2026-0001 (#1994) ## Which issue does this PR close? - Closes #1992 - Closes #1993 ## What changes are included in this PR? Update dependency to upgrade rkyv, but we still have to ignore it and wait for rust_decimal to resolve it. ## Are these changes tested? CI. --- .cargo/audit.toml | 3 + Cargo.lock | 26 ++++---- Cargo.toml | 4 +- bindings/python/Cargo.lock | 131 ++++++++++++++++++++++++++++++++++--- bindings/python/Cargo.toml | 6 ++ 5 files changed, 146 insertions(+), 24 deletions(-) diff --git a/.cargo/audit.toml b/.cargo/audit.toml index 09e2d35c50..d403f0ac5a 100644 --- a/.cargo/audit.toml +++ b/.cargo/audit.toml @@ -33,4 +33,7 @@ ignore = [ # # Introduced by object_store, see https://github.com/apache/arrow-rs-object-store/issues/564 "RUSTSEC-2025-0134", + + # Tracked here: https://github.com/paupino/rust-decimal/issues/766 + "RUSTSEC-2026-0001", ] diff --git a/Cargo.lock b/Cargo.lock index 97ee25d658..73494910b6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2715,7 +2715,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baec6a0289d7f1fe5665586ef7340af82e3037207bef60f5785e57569776f0c8" dependencies = [ "bytes", - "rkyv 0.8.12", + "rkyv 0.8.13", "serde", "simdutf8", ] @@ -5525,9 +5525,9 @@ dependencies = [ [[package]] name = "rkyv" -version = "0.7.45" +version = "0.7.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9008cd6385b9e161d8229e1f6549dd23c3d022f132a2ea37ac3a10ac4935779b" +checksum = "2297bf9c81a3f0dc96bc9521370b88f054168c29826a75e89c55ff196e7ed6a1" dependencies = [ "bitvec", "bytecheck", @@ -5535,7 +5535,7 @@ dependencies = [ "hashbrown 0.12.3", "ptr_meta 0.1.4", "rend 0.4.2", - "rkyv_derive 0.7.45", + "rkyv_derive 0.7.46", "seahash", "tinyvec", "uuid", @@ -5543,27 +5543,27 @@ dependencies = [ [[package]] name = "rkyv" -version = "0.8.12" +version = "0.8.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35a640b26f007713818e9a9b65d34da1cf58538207b052916a83d80e43f3ffa4" +checksum = "8b2e88acca7157d83d789836a3987dafc12bc3d88a050e54b8fe9ea4aaa29d20" dependencies = [ "bytes", - "hashbrown 0.15.5", + "hashbrown 0.16.1", "indexmap 2.12.1", "munge", "ptr_meta 0.3.1", "rancor", "rend 0.5.3", - "rkyv_derive 0.8.12", + "rkyv_derive 0.8.13", "tinyvec", "uuid", ] [[package]] name = "rkyv_derive" -version = "0.7.45" +version = "0.7.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "503d1d27590a2b0a3a4ca4c94755aa2875657196ecbf401a42eff41d7de532c0" +checksum = "84d7b42d4b8d06048d3ac8db0eb31bcb942cbeb709f0b5f2b2ebde398d3038f5" dependencies = [ "proc-macro2", "quote", @@ -5572,9 +5572,9 @@ dependencies = [ [[package]] name = "rkyv_derive" -version = "0.8.12" +version = "0.8.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bd83f5f173ff41e00337d97f6572e416d022ef8a19f371817259ae960324c482" +checksum = "7f6dffea3c91fa91a3c0fc8a061b0e27fef25c6304728038a6d6bcb1c58ba9bd" dependencies = [ "proc-macro2", "quote", @@ -5663,7 +5663,7 @@ dependencies = [ "num-traits", "postgres-types", "rand 0.8.5", - "rkyv 0.7.45", + "rkyv 0.7.46", "serde", "serde_json", ] diff --git a/Cargo.toml b/Cargo.toml index d099398dbd..56cd1801cc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -109,7 +109,7 @@ rand = "0.8.5" regex = "1.11.3" reqwest = { version = "0.12.12", default-features = false, features = ["json"] } roaring = { version = "0.11" } -rust_decimal = "1.37.2" +rust_decimal = { version = "1.39", default-features = false, features = ["std"] } serde = { version = "1.0.219", features = ["rc"] } serde_bytes = "0.11.17" serde_derive = "1.0.219" @@ -131,4 +131,4 @@ url = "2.5.7" uuid = { version = "1.18", features = ["v7"] } volo = "0.10.6" volo-thrift = "0.10.8" -zstd = "0.13.3" +zstd = "0.13.3" \ No newline at end of file diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 4647f9d886..d33abed581 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -668,8 +668,20 @@ version = "0.6.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2" dependencies = [ - "bytecheck_derive", - "ptr_meta", + "bytecheck_derive 0.6.12", + "ptr_meta 0.1.4", + "simdutf8", +] + +[[package]] +name = "bytecheck" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0caa33a2c0edca0419d15ac723dff03f1956f7978329b1e3b5fdaaaed9d3ca8b" +dependencies = [ + "bytecheck_derive 0.8.2", + "ptr_meta 0.3.1", + "rancor", "simdutf8", ] @@ -684,6 +696,17 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "bytecheck_derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89385e82b5d1821d2219e0b095efa2cc1f246cbf99080f3be46a1a85c0d392d9" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + [[package]] name = "bytemuck" version = "1.24.0" @@ -2350,6 +2373,7 @@ dependencies = [ "rand 0.8.5", "reqsign", "reqwest", + "rkyv 0.8.13", "roaring", "rust_decimal", "serde", @@ -2841,6 +2865,26 @@ dependencies = [ "uuid", ] +[[package]] +name = "munge" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e17401f259eba956ca16491461b6e8f72913a0a114e39736ce404410f915a0c" +dependencies = [ + "munge_macro", +] + +[[package]] +name = "munge_macro" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + [[package]] name = "murmur3" version = "0.5.2" @@ -3220,7 +3264,16 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1" dependencies = [ - "ptr_meta_derive", + "ptr_meta_derive 0.1.4", +] + +[[package]] +name = "ptr_meta" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b9a0cf95a1196af61d4f1cbdab967179516d9a4a4312af1f31948f8f6224a79" +dependencies = [ + "ptr_meta_derive 0.3.1", ] [[package]] @@ -3234,6 +3287,17 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "ptr_meta_derive" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + [[package]] name = "pyiceberg_core_rust" version = "0.8.0" @@ -3243,6 +3307,7 @@ dependencies = [ "iceberg", "iceberg-datafusion", "pyo3", + "rust_decimal", "tokio", ] @@ -3409,6 +3474,15 @@ version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" +[[package]] +name = "rancor" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a063ea72381527c2a0561da9c80000ef822bdd7c3241b1cc1b12100e3df081ee" +dependencies = [ + "ptr_meta 0.3.1", +] + [[package]] name = "rand" version = "0.8.5" @@ -3564,7 +3638,16 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c" dependencies = [ - "bytecheck", + "bytecheck 0.6.12", +] + +[[package]] +name = "rend" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cadadef317c2f20755a64d7fdc48f9e7178ee6b0e1f7fce33fa60f1d68a276e6" +dependencies = [ + "bytecheck 0.8.2", ] [[package]] @@ -3667,17 +3750,36 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9008cd6385b9e161d8229e1f6549dd23c3d022f132a2ea37ac3a10ac4935779b" dependencies = [ "bitvec", - "bytecheck", + "bytecheck 0.6.12", "bytes", "hashbrown 0.12.3", - "ptr_meta", - "rend", - "rkyv_derive", + "ptr_meta 0.1.4", + "rend 0.4.2", + "rkyv_derive 0.7.45", "seahash", "tinyvec", "uuid", ] +[[package]] +name = "rkyv" +version = "0.8.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2e88acca7157d83d789836a3987dafc12bc3d88a050e54b8fe9ea4aaa29d20" +dependencies = [ + "bytecheck 0.8.2", + "bytes", + "hashbrown 0.16.0", + "indexmap 2.12.0", + "munge", + "ptr_meta 0.3.1", + "rancor", + "rend 0.5.3", + "rkyv_derive 0.8.13", + "tinyvec", + "uuid", +] + [[package]] name = "rkyv_derive" version = "0.7.45" @@ -3689,6 +3791,17 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "rkyv_derive" +version = "0.8.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f6dffea3c91fa91a3c0fc8a061b0e27fef25c6304728038a6d6bcb1c58ba9bd" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + [[package]] name = "roaring" version = "0.11.2" @@ -3749,7 +3862,7 @@ dependencies = [ "bytes", "num-traits", "rand 0.8.5", - "rkyv", + "rkyv 0.7.45", "serde", "serde_json", ] diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index 9ec58cf807..8346d02703 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -37,6 +37,8 @@ pyo3 = { version = "0.26", features = ["extension-module", "abi3-py310"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } datafusion-ffi = { version = "51.0" } tokio = { version = "1.46.1", default-features = false } +# Security: disable rkyv feature to avoid RUSTSEC-2026-0001 (rkyv 0.7.45 vulnerability) +rust_decimal = { version = "1.39", default-features = false, features = ["std"] } [profile.release] codegen-units = 1 @@ -44,3 +46,7 @@ debug = false lto = "thin" opt-level = "z" strip = true + +[package.metadata.cargo-machete] +# rust_decimal is included to override feature flags for security (disable rkyv) +ignored = ["rust_decimal"] From 1b9fa3ae52eb1700cc2130301087e6c89c436004 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 7 Jan 2026 09:06:17 +0800 Subject: [PATCH 136/891] chore(deps): Bump reqwest from 0.12.25 to 0.12.28 (#1983) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.25 to 0.12.28.
Release notes

Sourced from reqwest's releases.

v0.12.28

What's Changed

Full Changelog: https://github.com/seanmonstar/reqwest/compare/v0.12.27...v0.12.28

v0.12.27

tl;dr

  • Add ClientBuilder::windows_named_pipe(name) option that will force all requests over that Windows Named Pipe.

What's Changed

Full Changelog: https://github.com/seanmonstar/reqwest/compare/v0.12.26...v0.12.27

v0.12.26

tl;dr

  • Fix sending Accept-Encoding header only with values configured with reqwest, regardless of underlying tower-http config.

What's Changed

Full Changelog: https://github.com/seanmonstar/reqwest/compare/v0.12.25...v0.12.26

Changelog

Sourced from reqwest's changelog.

v0.12.28

  • Fix compiling on Windows if TLS and SOCKS features are not enabled.

v0.12.27

  • Add ClientBuilder::windows_named_pipe(name) option that will force all requests over that Windows Named Piper.

v0.12.26

  • Fix sending Accept-Encoding header only with values configured with reqwest, regardless of underlying tower-http config.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=reqwest&package-manager=cargo&previous-version=0.12.25&new-version=0.12.28)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- Cargo.lock | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 73494910b6..794c82b9c8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2497,7 +2497,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2639,7 +2639,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -3312,7 +3312,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.6.1", + "socket2 0.5.10", "tokio", "tower-service", "tracing", @@ -3844,7 +3844,7 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -4365,7 +4365,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5165,7 +5165,7 @@ dependencies = [ "quinn-udp", "rustc-hash", "rustls 0.23.35", - "socket2 0.6.1", + "socket2 0.5.10", "thiserror 2.0.17", "tokio", "tracing", @@ -5202,9 +5202,9 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.6.1", + "socket2 0.5.10", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.59.0", ] [[package]] @@ -5468,9 +5468,9 @@ dependencies = [ [[package]] name = "reqwest" -version = "0.12.25" +version = "0.12.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6eff9328d40131d43bd911d42d79eb6a47312002a4daefc9e37f17e74a7701a" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ "base64", "bytes", @@ -5696,7 +5696,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -6698,7 +6698,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -7513,7 +7513,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] From 99461bf2dca63de4109a65b13a9b05e4334126f2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 7 Jan 2026 10:26:12 +0800 Subject: [PATCH 137/891] chore(deps): Bump rsa from 0.9.9 to 0.9.10 (#2004) Bumps [rsa](https://github.com/RustCrypto/RSA) from 0.9.9 to 0.9.10.
Changelog

Sourced from rsa's changelog.

0.9.10 (2026-01-06)

Fixed

  • do not panic on a prime being 1 when loading a secret key (#624)

#624: RustCrypto/RSA#624

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rsa&package-manager=cargo&previous-version=0.9.9&new-version=0.9.10)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/iceberg-rust/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 794c82b9c8..77d60a167f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5593,9 +5593,9 @@ dependencies = [ [[package]] name = "rsa" -version = "0.9.9" +version = "0.9.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40a0376c50d0358279d9d643e4bf7b7be212f1f4ff1da9070a7b54d22ef75c88" +checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d" dependencies = [ "const-oid", "digest", From 7f2dda3a3807e89b162785b4f16d655ca6b84f79 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Wed, 7 Jan 2026 10:27:15 +0800 Subject: [PATCH 138/891] Bump to version 0.8.0 Round 2 (#1999) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This PR will start another round of v0.8.0 release ## Are these changes tested? --- CHANGELOG.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 50d4576dfb..bd35e6b5d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/) and this project adheres to [Semantic Versioning](https://semver.org/). -## [v0.8.0] - 2025-12-16 +## [v0.8.0] - 2026-01-06 ### Breaking Changes @@ -167,6 +167,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/). * Remove wildcard pattern in exhaustive enums by @lgingerich in https://github.com/apache/iceberg-rust/pull/1925 * fix: prioritize delete manifests to prevent scan deadlock by @dojiong in https://github.com/apache/iceberg-rust/pull/1937 * feat: Include statistics for Reserved Fields by @Fokko in https://github.com/apache/iceberg-rust/pull/1849 +* fix(website): Update expected messages by @CTTY in https://github.com/apache/iceberg-rust/pull/1942 +* feat: Implement shared delete file loading and caching for ArrowReader by @dojiong in https://github.com/apache/iceberg-rust/pull/1941 +* infra: license header check ignore target/ dir by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1954 +* infra: release script, validate proper ICEBERG_VERSION variable by @kevinjqliu in https://github.com/apache/iceberg-rust/pull/1956 +* refactor(arrow): Rename parameter in delete_filter for clarity by @robertmu in https://github.com/apache/iceberg-rust/pull/1955 +* feat(sqllogictest): use serde derived structs for schedule parsing by @AndreaBozzo in https://github.com/apache/iceberg-rust/pull/1953 +* fix: follow IEEE 754 totalOrder for `float` and `double` by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1959 +* fix: return proper error rather than persisting error message on snapshot by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1960 +* feat(arrow): Convert Arrow schema to Iceberg schema with auto assigned field ids by @CTTY in https://github.com/apache/iceberg-rust/pull/1928 +* fix: MemoryCatalog to return absolute NamespaceIdents by @eickler in https://github.com/apache/iceberg-rust/pull/1970 +* fix(spec): Include delete file content to V3 manifest by @CTTY in https://github.com/apache/iceberg-rust/pull/1979 +* fix: fix typo check error by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1989 +* Fix ci audit failure by @liurenjie1024 in https://github.com/apache/iceberg-rust/pull/1988 +* feat: make FanoutWriter writer configurable by @Standing-Man in https://github.com/apache/iceberg-rust/pull/1962 ## [v0.7.0] - 2025-09-23 From 6fa878590a70f4078e6fbefa2751513662d421b4 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 7 Jan 2026 17:42:45 -0800 Subject: [PATCH 139/891] feat(datafusion): Apply SortExec when writing in clustered mode (#2005) ## Which issue does this PR close? - Closes #1540 ## What changes are included in this PR? - When writing in clustered mode, use `sort_by_partition` to sort the data so the clustered writer can comsume it ## Are these changes tested? - Added simple uts to verify that the `SortExec` is applied correctly --- .../datafusion/src/physical_plan/sort.rs | 3 - .../integrations/datafusion/src/table/mod.rs | 178 +++++++++++++++++- 2 files changed, 177 insertions(+), 4 deletions(-) diff --git a/crates/integrations/datafusion/src/physical_plan/sort.rs b/crates/integrations/datafusion/src/physical_plan/sort.rs index ede2547535..587ab120ca 100644 --- a/crates/integrations/datafusion/src/physical_plan/sort.rs +++ b/crates/integrations/datafusion/src/physical_plan/sort.rs @@ -42,9 +42,6 @@ use iceberg::arrow::PROJECTED_PARTITION_VALUE_COLUMN; /// # Returns /// * `Ok(Arc)` - A SortExec that sorts by partition values /// * `Err` - If the partition column is not found -/// -/// TODO remove dead_code mark when integrating with insert_into -#[allow(dead_code)] pub(crate) fn sort_by_partition(input: Arc) -> DFResult> { let schema = input.schema(); diff --git a/crates/integrations/datafusion/src/table/mod.rs b/crates/integrations/datafusion/src/table/mod.rs index 86a79611b3..ad616542a4 100644 --- a/crates/integrations/datafusion/src/table/mod.rs +++ b/crates/integrations/datafusion/src/table/mod.rs @@ -44,6 +44,7 @@ use datafusion::physical_plan::ExecutionPlan; use datafusion::physical_plan::coalesce_partitions::CoalescePartitionsExec; use iceberg::arrow::schema_to_arrow_schema; use iceberg::inspect::MetadataTableType; +use iceberg::spec::TableProperties; use iceberg::table::Table; use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableIdent}; use metadata_table::IcebergMetadataTableProvider; @@ -53,6 +54,7 @@ use crate::physical_plan::commit::IcebergCommitExec; use crate::physical_plan::project::project_with_partition; use crate::physical_plan::repartition::repartition; use crate::physical_plan::scan::IcebergTableScan; +use crate::physical_plan::sort::sort_by_partition; use crate::physical_plan::write::IcebergWriteExec; /// Catalog-backed table provider with automatic metadata refresh. @@ -185,9 +187,38 @@ impl TableProvider for IcebergTableProvider { let repartitioned_plan = repartition(plan_with_partition, table.metadata_ref(), target_partitions)?; + // Apply sort node when it's not fanout mode + let fanout_enabled = table + .metadata() + .properties() + .get(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED) + .map(|value| { + value + .parse::() + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!( + "Invalid value for {}, expected 'true' or 'false'", + TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED + ), + ) + .with_source(e) + }) + .map_err(to_datafusion_error) + }) + .transpose()? + .unwrap_or(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT); + + let write_input = if fanout_enabled { + repartitioned_plan + } else { + sort_by_partition(repartitioned_plan)? + }; + let write_plan = Arc::new(IcebergWriteExec::new( table.clone(), - repartitioned_plan, + write_input, self.schema.clone(), )); @@ -321,6 +352,7 @@ mod tests { use std::sync::Arc; use datafusion::common::Column; + use datafusion::physical_plan::ExecutionPlan; use datafusion::prelude::SessionContext; use iceberg::io::FileIO; use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; @@ -598,4 +630,148 @@ mod tests { assert_eq!(logical_field.data_type(), physical_field.data_type()); } } + + async fn get_partitioned_test_catalog_and_table( + fanout_enabled: Option, + ) -> (Arc, NamespaceIdent, String, TempDir) { + use iceberg::spec::{Transform, UnboundPartitionSpec}; + + let temp_dir = TempDir::new().unwrap(); + let warehouse_path = temp_dir.path().to_str().unwrap().to_string(); + + let catalog = MemoryCatalogBuilder::default() + .load( + "memory", + HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), warehouse_path.clone())]), + ) + .await + .unwrap(); + + let namespace = NamespaceIdent::new("test_ns".to_string()); + catalog + .create_namespace(&namespace, HashMap::new()) + .await + .unwrap(); + + let schema = Schema::builder() + .with_schema_id(0) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "category", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(); + + let partition_spec = UnboundPartitionSpec::builder() + .with_spec_id(0) + .add_partition_field(2, "category", Transform::Identity) + .unwrap() + .build(); + + let mut properties = HashMap::new(); + if let Some(enabled) = fanout_enabled { + properties.insert( + iceberg::spec::TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED + .to_string(), + enabled.to_string(), + ); + } + + let table_creation = TableCreation::builder() + .name("partitioned_table".to_string()) + .location(format!("{warehouse_path}/partitioned_table")) + .schema(schema) + .partition_spec(partition_spec) + .properties(properties) + .build(); + + catalog + .create_table(&namespace, table_creation) + .await + .unwrap(); + + ( + Arc::new(catalog), + namespace, + "partitioned_table".to_string(), + temp_dir, + ) + } + + /// Helper to check if a plan contains a SortExec node + fn plan_contains_sort(plan: &Arc) -> bool { + if plan.name() == "SortExec" { + return true; + } + for child in plan.children() { + if plan_contains_sort(child) { + return true; + } + } + false + } + + #[tokio::test] + async fn test_insert_plan_fanout_enabled_no_sort() { + use datafusion::datasource::TableProvider; + use datafusion::logical_expr::dml::InsertOp; + use datafusion::physical_plan::empty::EmptyExec; + + // When fanout is enabled (default), no sort node should be added + let (catalog, namespace, table_name, _temp_dir) = + get_partitioned_test_catalog_and_table(Some(true)).await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + let input_schema = provider.schema(); + let input = Arc::new(EmptyExec::new(input_schema)) as Arc; + + let state = ctx.state(); + let insert_plan = provider + .insert_into(&state, input, InsertOp::Append) + .await + .unwrap(); + + // With fanout enabled, there should be no SortExec in the plan + assert!( + !plan_contains_sort(&insert_plan), + "Plan should NOT contain SortExec when fanout is enabled" + ); + } + + #[tokio::test] + async fn test_insert_plan_fanout_disabled_has_sort() { + use datafusion::datasource::TableProvider; + use datafusion::logical_expr::dml::InsertOp; + use datafusion::physical_plan::empty::EmptyExec; + + // When fanout is disabled, a sort node should be added + let (catalog, namespace, table_name, _temp_dir) = + get_partitioned_test_catalog_and_table(Some(false)).await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + let input_schema = provider.schema(); + let input = Arc::new(EmptyExec::new(input_schema)) as Arc; + + let state = ctx.state(); + let insert_plan = provider + .insert_into(&state, input, InsertOp::Append) + .await + .unwrap(); + + // With fanout disabled, there should be a SortExec in the plan + assert!( + plan_contains_sort(&insert_plan), + "Plan should contain SortExec when fanout is disabled" + ); + } } From 700e62e00009a781cab5c8d321eb94173dfe7676 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 8 Jan 2026 10:20:28 +0800 Subject: [PATCH 140/891] chore(deps): Bump crate-ci/typos from 1.40.0 to 1.41.0 (#1982) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.40.0 to 1.41.0.
Release notes

Sourced from crate-ci/typos's releases.

v1.41.0

[1.41.0] - 2025-12-31

Features

v1.40.1

[1.40.1] - 2025-12-29

Fixes

  • Treat incrementer and incrementor the same for now

Fixes

  • Don't correct ITerm2
Changelog

Sourced from crate-ci/typos's changelog.

[1.41.0] - 2025-12-31

Features

[1.40.1] - 2025-12-29

Fixes

  • Treat incrementer and incrementor the same for now

Fixes

  • Don't correct ITerm2
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.40.0&new-version=1.41.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 6406b82142..b79ab0c0d1 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.40.0 + uses: crate-ci/typos@v1.41.0 From 76cdf280231bf27ea888de15507d918392f08d44 Mon Sep 17 00:00:00 2001 From: Aditya Subrahmanyan Date: Wed, 7 Jan 2026 18:20:43 -0800 Subject: [PATCH 141/891] fix: Use correct byte representation for decimal hashing (#1998) ## Which issue does this PR close? - Closes #1981. ## What changes are included in this PR? The [spec](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) states that: >"Decimal values are hashed using the minimum number of bytes required to hold the unscaled value as a two's complement big-endian". Prior to this fix, we would incorrectly consume leading `0xFF` bytes and hash them. Now, we only consume the bytes starting with the one that is used to preserve the sign, and everything that follows it. ## Are these changes tested? Added unit tests for original scenario mentioned in the issue, as well as some additional cases --- crates/iceberg/src/transform/bucket.rs | 43 +++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/crates/iceberg/src/transform/bucket.rs b/crates/iceberg/src/transform/bucket.rs index 8807fb1f79..e6786a70ca 100644 --- a/crates/iceberg/src/transform/bucket.rs +++ b/crates/iceberg/src/transform/bucket.rs @@ -78,12 +78,26 @@ impl Bucket { /// ref: https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements #[inline] fn hash_decimal(v: i128) -> i32 { + if v == 0 { + return Self::hash_bytes(&[0]); + } + let bytes = v.to_be_bytes(); - if let Some(start) = bytes.iter().position(|&x| x != 0) { - Self::hash_bytes(&bytes[start..]) + let start = if v > 0 { + // Positive: skip 0x00 unless next byte would appear negative + bytes + .windows(2) + .position(|w| w[0] != 0x00 || w[1] & 0x80 != 0) + .unwrap_or(15) } else { - Self::hash_bytes(&[0]) - } + // Negative: skip 0xFF only if next byte stays negative + bytes + .windows(2) + .position(|w| w[0] != 0xFF || w[1] & 0x80 == 0) + .unwrap_or(15) + }; + + Self::hash_bytes(&bytes[start..]) } /// def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N @@ -790,6 +804,27 @@ mod test { ); } + #[test] + fn test_hash_decimal_with_negative_value() { + // Test cases from GitHub issue #1981 + assert_eq!(Bucket::hash_decimal(1), -463810133); + assert_eq!(Bucket::hash_decimal(-1), -43192051); + + // Additional test cases for edge case values + assert_eq!(Bucket::hash_decimal(0), Bucket::hash_decimal(0)); + assert_eq!(Bucket::hash_decimal(127), Bucket::hash_decimal(127)); + assert_eq!(Bucket::hash_decimal(-128), Bucket::hash_decimal(-128)); + + // Test minimum representation is used + // -1 should hash as [0xFF] not [0xFF, 0xFF, ..., 0xFF] + // 128 should hash as [0x00, 0x80] not [0x00, 0x00, ..., 0x80] + assert_eq!(Bucket::hash_decimal(128), Bucket::hash_bytes(&[0x00, 0x80])); + assert_eq!( + Bucket::hash_decimal(-129), + Bucket::hash_bytes(&[0xFF, 0x7F]) + ); + } + #[test] fn test_int_literal() { let bucket = Bucket::new(10); From 43f1ed8362951a8be72101ac1625cb8634cd242e Mon Sep 17 00:00:00 2001 From: Aditya Subrahmanyan Date: Thu, 8 Jan 2026 18:50:04 -0800 Subject: [PATCH 142/891] fix: Reserved sort order ID cannot contain any fields (#1978) ## Which issue does this PR close? - Closes #1963. ## What changes are included in this PR? This change validates that table metadata with reserved sort order ID (0) cannot contain fields associated with it. If this is found, we error out instead of silently parsing arbitrary field values. ## Are these changes tested? Added the unit test described in the issue and verified that the check is now enforced. --- crates/iceberg/src/spec/table_metadata.rs | 66 +++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index 48b715da59..cfa25deccb 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -506,6 +506,19 @@ impl TableMetadata { /// If the default sort order is unsorted but the sort order is not present, add it fn try_normalize_sort_order(&mut self) -> Result<()> { + // Validate that sort order ID 0 (reserved for unsorted) has no fields + if let Some(sort_order) = self.sort_order_by_id(SortOrder::UNSORTED_ORDER_ID) + && !sort_order.fields.is_empty() + { + return Err(Error::new( + ErrorKind::Unexpected, + format!( + "Sort order ID {} is reserved for unsorted order", + SortOrder::UNSORTED_ORDER_ID + ), + )); + } + if self.sort_order_by_id(self.default_sort_order_id).is_some() { return Ok(()); } @@ -3795,4 +3808,57 @@ mod tests { assert!(final_metadata.name_exists_in_any_schema("new_field")); // only in current schema assert!(!final_metadata.name_exists_in_any_schema("never_existed")); } + + #[test] + fn test_invalid_sort_order_id_zero_with_fields() { + let metadata = r#" + { + "format-version": 2, + "table-uuid": "9c12d441-03fe-4693-9a96-a0705ddf69c1", + "location": "s3://bucket/test/location", + "last-sequence-number": 111, + "last-updated-ms": 1600000000000, + "last-column-id": 3, + "current-schema-id": 1, + "schemas": [ + { + "type": "struct", + "schema-id": 1, + "fields": [ + {"id": 1, "name": "x", "required": true, "type": "long"}, + {"id": 2, "name": "y", "required": true, "type": "long"} + ] + } + ], + "default-spec-id": 0, + "partition-specs": [{"spec-id": 0, "fields": []}], + "last-partition-id": 999, + "default-sort-order-id": 0, + "sort-orders": [ + { + "order-id": 0, + "fields": [ + { + "transform": "identity", + "source-id": 1, + "direction": "asc", + "null-order": "nulls-first" + } + ] + } + ], + "properties": {}, + "current-snapshot-id": -1, + "snapshots": [] + } + "#; + + let result: Result = serde_json::from_str(metadata); + + // Should fail because sort order ID 0 is reserved for unsorted order and cannot have fields + assert!( + result.is_err(), + "Parsing should fail for sort order ID 0 with fields" + ); + } } From 65e368250c1e363c3c63de06248e4bc40ad37c58 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Fri, 9 Jan 2026 11:37:44 +0800 Subject: [PATCH 143/891] feat(datafusion): Add LIMIT pushdown support (#2006) Implement LIMIT pushdown to optimize queries with LIMIT clauses by stopping data processing once the limit is reached. This reduces unnecessary I/O and computation for queries that only need a subset of rows. Changes: - Add limit field to IcebergTableScan to track row limit - Apply limit at stream level by filtering/slicing record batches - Update IcebergTableProvider and IcebergStaticTableProvider to pass limit parameter to scan - Add comprehensive tests for limit pushdown functionality ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? Co-authored-by: Claude Sonnet 4.5 --- .../datafusion/src/physical_plan/scan.rs | 30 +++++- .../integrations/datafusion/src/table/mod.rs | 98 ++++++++++++++++++- 2 files changed, 125 insertions(+), 3 deletions(-) diff --git a/crates/integrations/datafusion/src/physical_plan/scan.rs b/crates/integrations/datafusion/src/physical_plan/scan.rs index be92e93d25..d627b6a63d 100644 --- a/crates/integrations/datafusion/src/physical_plan/scan.rs +++ b/crates/integrations/datafusion/src/physical_plan/scan.rs @@ -51,6 +51,8 @@ pub struct IcebergTableScan { projection: Option>, /// Filters to apply to the table scan predicates: Option, + /// Optional limit on the number of rows to return + limit: Option, } impl IcebergTableScan { @@ -61,6 +63,7 @@ impl IcebergTableScan { schema: ArrowSchemaRef, projection: Option<&Vec>, filters: &[Expr], + limit: Option, ) -> Self { let output_schema = match projection { None => schema.clone(), @@ -76,6 +79,7 @@ impl IcebergTableScan { plan_properties, projection, predicates, + limit, } } @@ -95,6 +99,10 @@ impl IcebergTableScan { self.predicates.as_ref() } + pub fn limit(&self) -> Option { + self.limit + } + /// Computes [`PlanProperties`] used in query optimization. fn compute_properties(schema: ArrowSchemaRef) -> PlanProperties { // TODO: @@ -146,9 +154,29 @@ impl ExecutionPlan for IcebergTableScan { ); let stream = futures::stream::once(fut).try_flatten(); + // Apply limit if specified + let limited_stream: Pin> + Send>> = + if let Some(limit) = self.limit { + let mut remaining = limit; + Box::pin(stream.try_filter_map(move |batch| { + futures::future::ready(if remaining == 0 { + Ok(None) + } else if batch.num_rows() <= remaining { + remaining -= batch.num_rows(); + Ok(Some(batch)) + } else { + let limited_batch = batch.slice(0, remaining); + remaining = 0; + Ok(Some(limited_batch)) + }) + })) + } else { + Box::pin(stream) + }; + Ok(Box::pin(RecordBatchStreamAdapter::new( self.schema(), - stream, + limited_stream, ))) } } diff --git a/crates/integrations/datafusion/src/table/mod.rs b/crates/integrations/datafusion/src/table/mod.rs index ad616542a4..ae87342fa5 100644 --- a/crates/integrations/datafusion/src/table/mod.rs +++ b/crates/integrations/datafusion/src/table/mod.rs @@ -127,7 +127,7 @@ impl TableProvider for IcebergTableProvider { _state: &dyn Session, projection: Option<&Vec>, filters: &[Expr], - _limit: Option, + limit: Option, ) -> DFResult> { // Load fresh table metadata from catalog let table = self @@ -143,6 +143,7 @@ impl TableProvider for IcebergTableProvider { self.schema.clone(), projection, filters, + limit, ))) } @@ -311,7 +312,7 @@ impl TableProvider for IcebergStaticTableProvider { _state: &dyn Session, projection: Option<&Vec>, filters: &[Expr], - _limit: Option, + limit: Option, ) -> DFResult> { // Use cached table (no refresh) Ok(Arc::new(IcebergTableScan::new( @@ -320,6 +321,7 @@ impl TableProvider for IcebergStaticTableProvider { self.schema.clone(), projection, filters, + limit, ))) } @@ -774,4 +776,96 @@ mod tests { "Plan should contain SortExec when fanout is disabled" ); } + + #[tokio::test] + async fn test_limit_pushdown_static_provider() { + use datafusion::datasource::TableProvider; + + let table = get_test_table_from_metadata_file().await; + let table_provider = IcebergStaticTableProvider::try_new_from_table(table.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + let state = ctx.state(); + + // Test scan with limit + let scan_plan = table_provider + .scan(&state, None, &[], Some(10)) + .await + .unwrap(); + + // Verify that the scan plan is an IcebergTableScan + let iceberg_scan = scan_plan + .as_any() + .downcast_ref::() + .expect("Expected IcebergTableScan"); + + // Verify the limit is set + assert_eq!( + iceberg_scan.limit(), + Some(10), + "Limit should be set to 10 in the scan plan" + ); + } + + #[tokio::test] + async fn test_limit_pushdown_catalog_backed_provider() { + use datafusion::datasource::TableProvider; + + let (catalog, namespace, table_name, _temp_dir) = get_test_catalog_and_table().await; + + let provider = + IcebergTableProvider::try_new(catalog.clone(), namespace.clone(), table_name.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + let state = ctx.state(); + + // Test scan with limit + let scan_plan = provider.scan(&state, None, &[], Some(5)).await.unwrap(); + + // Verify that the scan plan is an IcebergTableScan + let iceberg_scan = scan_plan + .as_any() + .downcast_ref::() + .expect("Expected IcebergTableScan"); + + // Verify the limit is set + assert_eq!( + iceberg_scan.limit(), + Some(5), + "Limit should be set to 5 in the scan plan" + ); + } + + #[tokio::test] + async fn test_no_limit_pushdown() { + use datafusion::datasource::TableProvider; + + let table = get_test_table_from_metadata_file().await; + let table_provider = IcebergStaticTableProvider::try_new_from_table(table.clone()) + .await + .unwrap(); + + let ctx = SessionContext::new(); + let state = ctx.state(); + + // Test scan without limit + let scan_plan = table_provider.scan(&state, None, &[], None).await.unwrap(); + + // Verify that the scan plan is an IcebergTableScan + let iceberg_scan = scan_plan + .as_any() + .downcast_ref::() + .expect("Expected IcebergTableScan"); + + // Verify the limit is None + assert_eq!( + iceberg_scan.limit(), + None, + "Limit should be None when not specified" + ); + } } From a169c470f637e97dd07b82fb19d3537d62d7ca51 Mon Sep 17 00:00:00 2001 From: Nathan Metzger <39155588+xbattlax@users.noreply.github.com> Date: Sat, 10 Jan 2026 02:37:18 +0100 Subject: [PATCH 144/891] feat(spec): add table_properties() method to TableMetadata (#2002) ## Summary Add a type-safe accessor method `table_properties()` to `TableMetadata` that returns `TableProperties`, eliminating manual `TryFrom` conversions at call sites. ### Changes - Add `table_properties()` method to `TableMetadata` that parses and returns typed `TableProperties` - Update `transaction/mod.rs` to use the new method - Simplify `datafusion/write.rs` property access (~60 lines reduced to ~10) - Add unit tests for the new accessor method Closes #1878 --- crates/iceberg/src/spec/table_metadata.rs | 119 +++++++++++++++++- crates/iceberg/src/transaction/mod.rs | 7 +- .../datafusion/src/physical_plan/write.rs | 61 ++------- 3 files changed, 132 insertions(+), 55 deletions(-) diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index cfa25deccb..585cb3e2bc 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -37,6 +37,7 @@ pub use super::table_metadata_builder::{TableMetadataBuildResult, TableMetadataB use super::{ DEFAULT_PARTITION_SPEC_ID, PartitionSpecRef, PartitionStatisticsFile, SchemaId, SchemaRef, SnapshotRef, SnapshotRetention, SortOrder, SortOrderRef, StatisticsFile, StructType, + TableProperties, }; use crate::error::{Result, timestamp_ms_to_utc}; use crate::io::FileIO; @@ -360,6 +361,13 @@ impl TableMetadata { &self.properties } + /// Returns typed table properties parsed from the raw properties map with defaults. + pub fn table_properties(&self) -> Result { + TableProperties::try_from(&self.properties).map_err(|e| { + Error::new(ErrorKind::DataInvalid, "Invalid table properties").with_source(e) + }) + } + /// Return location of statistics files. #[inline] pub fn statistics_iter(&self) -> impl ExactSizeIterator { @@ -1561,7 +1569,6 @@ mod tests { use uuid::Uuid; use super::{FormatVersion, MetadataLog, SnapshotLog, TableMetadataBuilder}; - use crate::TableCreation; use crate::io::FileIOBuilder; use crate::spec::table_metadata::TableMetadata; use crate::spec::{ @@ -1570,6 +1577,7 @@ mod tests { SnapshotReference, SnapshotRetention, SortDirection, SortField, SortOrder, StatisticsFile, Summary, Transform, Type, UnboundPartitionField, }; + use crate::{ErrorKind, TableCreation}; fn check_table_metadata_serde(json: &str, expected_type: TableMetadata) { let desered_type: TableMetadata = serde_json::from_str(json).unwrap(); @@ -3861,4 +3869,113 @@ mod tests { "Parsing should fail for sort order ID 0 with fields" ); } + + #[test] + fn test_table_properties_with_defaults() { + use crate::spec::TableProperties; + + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(); + + let metadata = TableMetadataBuilder::new( + schema, + PartitionSpec::unpartition_spec().into_unbound(), + SortOrder::unsorted_order(), + "s3://test/location".to_string(), + FormatVersion::V2, + HashMap::new(), + ) + .unwrap() + .build() + .unwrap() + .metadata; + + let props = metadata.table_properties().unwrap(); + + assert_eq!( + props.commit_num_retries, + TableProperties::PROPERTY_COMMIT_NUM_RETRIES_DEFAULT + ); + assert_eq!( + props.write_target_file_size_bytes, + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT + ); + } + + #[test] + fn test_table_properties_with_custom_values() { + use crate::spec::TableProperties; + + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(); + + let properties = HashMap::from([ + ( + TableProperties::PROPERTY_COMMIT_NUM_RETRIES.to_string(), + "10".to_string(), + ), + ( + TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES.to_string(), + "1024".to_string(), + ), + ]); + + let metadata = TableMetadataBuilder::new( + schema, + PartitionSpec::unpartition_spec().into_unbound(), + SortOrder::unsorted_order(), + "s3://test/location".to_string(), + FormatVersion::V2, + properties, + ) + .unwrap() + .build() + .unwrap() + .metadata; + + let props = metadata.table_properties().unwrap(); + + assert_eq!(props.commit_num_retries, 10); + assert_eq!(props.write_target_file_size_bytes, 1024); + } + + #[test] + fn test_table_properties_with_invalid_value() { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(); + + let properties = HashMap::from([( + "commit.retry.num-retries".to_string(), + "not_a_number".to_string(), + )]); + + let metadata = TableMetadataBuilder::new( + schema, + PartitionSpec::unpartition_spec().into_unbound(), + SortOrder::unsorted_order(), + "s3://test/location".to_string(), + FormatVersion::V2, + properties, + ) + .unwrap() + .build() + .unwrap() + .metadata; + + let err = metadata.table_properties().unwrap_err(); + assert_eq!(err.kind(), ErrorKind::DataInvalid); + assert!(err.message().contains("Invalid table properties")); + } } diff --git a/crates/iceberg/src/transaction/mod.rs b/crates/iceberg/src/transaction/mod.rs index 8ddaa26698..074c7fefe4 100644 --- a/crates/iceberg/src/transaction/mod.rs +++ b/crates/iceberg/src/transaction/mod.rs @@ -76,7 +76,7 @@ use crate::transaction::update_location::UpdateLocationAction; use crate::transaction::update_properties::UpdatePropertiesAction; use crate::transaction::update_statistics::UpdateStatisticsAction; use crate::transaction::upgrade_format_version::UpgradeFormatVersionAction; -use crate::{Catalog, Error, ErrorKind, TableCommit, TableRequirement, TableUpdate}; +use crate::{Catalog, TableCommit, TableRequirement, TableUpdate}; /// Table transaction. #[derive(Clone)] @@ -163,10 +163,7 @@ impl Transaction { return Ok(self.table); } - let table_props = - TableProperties::try_from(self.table.metadata().properties()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Invalid table properties").with_source(e) - })?; + let table_props = self.table.metadata().table_properties()?; let backoff = Self::build_backoff(table_props)?; let tx = self; diff --git a/crates/integrations/datafusion/src/physical_plan/write.rs b/crates/integrations/datafusion/src/physical_plan/write.rs index fdfddf877b..0dea150d31 100644 --- a/crates/integrations/datafusion/src/physical_plan/write.rs +++ b/crates/integrations/datafusion/src/physical_plan/write.rs @@ -36,7 +36,7 @@ use datafusion::physical_plan::{ }; use futures::StreamExt; use iceberg::arrow::FieldMatchMode; -use iceberg::spec::{DataFileFormat, TableProperties, serialize_data_file_to_json}; +use iceberg::spec::{DataFileFormat, serialize_data_file_to_json}; use iceberg::table::Table; use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; use iceberg::writer::file_writer::ParquetWriterBuilder; @@ -208,15 +208,16 @@ impl ExecutionPlan for IcebergWriteExec { let partition_type = self.table.metadata().default_partition_type().clone(); let format_version = self.table.metadata().format_version(); + // Get typed table properties + let table_props = self + .table + .metadata() + .table_properties() + .map_err(to_datafusion_error)?; + // Check data file format - let file_format = DataFileFormat::from_str( - self.table - .metadata() - .properties() - .get(TableProperties::PROPERTY_DEFAULT_FILE_FORMAT) - .unwrap_or(&TableProperties::PROPERTY_DEFAULT_FILE_FORMAT_DEFAULT.to_string()), - ) - .map_err(to_datafusion_error)?; + let file_format = DataFileFormat::from_str(&table_props.write_format_default) + .map_err(to_datafusion_error)?; if file_format != DataFileFormat::Parquet { return Err(to_datafusion_error(Error::new( ErrorKind::FeatureUnsupported, @@ -230,24 +231,7 @@ impl ExecutionPlan for IcebergWriteExec { self.table.metadata().current_schema().clone(), FieldMatchMode::Name, ); - let target_file_size = match self - .table - .metadata() - .properties() - .get(TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES) - { - Some(value_str) => value_str - .parse::() - .map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Invalid value for write.target-file-size-bytes", - ) - .with_source(e) - }) - .map_err(to_datafusion_error)?, - None => TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT, - }; + let target_file_size = table_props.write_target_file_size_bytes; let file_io = self.table.file_io().clone(); // todo location_gen and file_name_gen should be configurable @@ -266,28 +250,7 @@ impl ExecutionPlan for IcebergWriteExec { let data_file_writer_builder = DataFileWriterBuilder::new(rolling_writer_builder); // Create TaskWriter - let fanout_enabled = self - .table - .metadata() - .properties() - .get(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED) - .map(|value| { - value - .parse::() - .map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - format!( - "Invalid value for {}, expected 'true' or 'false'", - TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED - ), - ) - .with_source(e) - }) - .map_err(to_datafusion_error) - }) - .transpose()? - .unwrap_or(TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT); + let fanout_enabled = table_props.write_datafusion_fanout_enabled; let schema = self.table.metadata().current_schema().clone(); let partition_spec = self.table.metadata().default_partition_spec().clone(); let task_writer = TaskWriter::try_new( From 3f71c39b7848569044bc5d3d6700a52481a6d57a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 12 Jan 2026 08:34:51 +0800 Subject: [PATCH 145/891] chore(deps): Bump crate-ci/typos from 1.41.0 to 1.42.0 (#2012) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.41.0 to 1.42.0.
Release notes

Sourced from crate-ci/typos's releases.

v1.42.0

[1.42.0] - 2026-01-07

Features

  • Dictionary updates
Changelog

Sourced from crate-ci/typos's changelog.

[1.42.0] - 2026-01-07

Features

  • Dictionary updates
Commits
  • bb4666a chore: Release
  • 6995a8a chore: Release
  • e0227ba docs: Update changelog
  • c4054db Merge pull request #1460 from epage/wiki
  • 596a0bd feat(dict): Extend from misspell's list
  • 2d45979 refactor(misspell): Reformat dict to look like ours
  • 54e1366 feat(dict): Extend from codespell's list
  • 74e0660 feat(dict): Extend from wikipedia's list
  • a2568ec Merge pull request #1459 from epage/update
  • c360cf6 fix(wikipedia): Update dict
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.41.0&new-version=1.42.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index b79ab0c0d1..514d59b66c 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.41.0 + uses: crate-ci/typos@v1.42.0 From dbc7e9f9dba7f1a5a0f640865ebfcb2201fca90a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 12 Jan 2026 08:41:54 +0800 Subject: [PATCH 146/891] chore(deps): Bump the arrow-parquet group with 6 updates (#2013) Bumps the arrow-parquet group with 6 updates: | Package | From | To | | --- | --- | --- | | [arrow-array](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | | [arrow-buffer](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | | [arrow-ord](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | | [arrow-schema](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | | [arrow-select](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | | [arrow-string](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | Updates `arrow-array` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-array's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Updates `arrow-buffer` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-buffer's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Updates `arrow-ord` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-ord's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Updates `arrow-schema` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-schema's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Updates `arrow-select` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-select's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Updates `arrow-string` from 57.1.0 to 57.2.0
Changelog

Sourced from arrow-string's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 77d60a167f..51af571d5a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -258,9 +258,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a23eaff85a44e9fa914660fb0d0bb00b79c4a3d888b5334adb3ea4330c84f002" +checksum = "65ca404ea6191e06bf30956394173337fa9c35f445bd447fe6c21ab944e1a23c" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -277,9 +277,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2819d893750cb3380ab31ebdc8c68874dd4429f90fd09180f3c93538bd21626" +checksum = "36356383099be0151dacc4245309895f16ba7917d79bdb71a7148659c9206c56" dependencies = [ "bytes", "half", @@ -326,9 +326,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05738f3d42cb922b9096f7786f606fcb8669260c2640df8490533bb2fa38c9d3" +checksum = "bf87f4ff5fc13290aa47e499a8b669a82c5977c6a1fedce22c7f542c1fd5a597" dependencies = [ "arrow-buffer", "arrow-schema", @@ -379,9 +379,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cbc94fc7adec5d1ba9e8cd1b1e8d6f72423b33fe978bf1f46d970fafab787521" +checksum = "13c4e0530272ca755d6814218dffd04425c5b7854b87fa741d5ff848bf50aa39" dependencies = [ "arrow-array", "arrow-buffer", @@ -405,9 +405,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d27609cd7dd45f006abae27995c2729ef6f4b9361cde1ddd019dc31a5aa017e0" +checksum = "6bb63203e8e0e54b288d0d8043ca8fa1013820822a27692ef1b78a977d879f2c" dependencies = [ "serde_core", "serde_json", @@ -415,9 +415,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae980d021879ea119dd6e2a13912d81e64abed372d53163e804dfe84639d8010" +checksum = "c96d8a1c180b44ecf2e66c9a2f2bbcb8b1b6f14e165ce46ac8bde211a363411b" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -429,9 +429,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf35e8ef49dcf0c5f6d175edee6b8af7b45611805333129c541a8b89a0fc0534" +checksum = "a8ad6a81add9d3ea30bf8374ee8329992c7fd246ffd8b7e2f48a3cea5aa0cc9a" dependencies = [ "arrow-array", "arrow-buffer", From 19cd1083bc408c7217980e7ff07635669de4b1da Mon Sep 17 00:00:00 2001 From: Aditya Subrahmanyan Date: Mon, 12 Jan 2026 13:16:17 -0800 Subject: [PATCH 147/891] docs: Minor update to README.md (#2018) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Minor updates to `README.md` text. ## Are these changes tested? N/A --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 27cce7664e..5fc452098e 100644 --- a/README.md +++ b/README.md @@ -74,14 +74,14 @@ The features that Iceberg Rust currently supports can be found [here](https://ic ## Supported Rust Version -Iceberg Rust is built and tested with stable rust, and will keep a rolling MSRV(minimum supported rust version). +Iceberg Rust is built and tested with stable rust, and will keep a rolling MSRV (minimum supported rust version). At least three months from latest rust release is supported. MSRV is updated when we release iceberg-rust. Check the current MSRV on [crates.io](https://crates.io/crates/iceberg). ## Contribute -Apache Iceberg is an active open-source project, governed under the Apache Software Foundation (ASF). Iceberg-rust are always open to people who want to use or contribute to it. Here are some ways to get involved. +Apache Iceberg is an active open-source project, governed under the Apache Software Foundation (ASF). Apache Iceberg Rust is always open to people who want to use or contribute to it. Here are some ways to get involved. - Start with [Contributing Guide](CONTRIBUTING.md). - Submit [Issues](https://github.com/apache/iceberg-rust/issues/new) for bug report or feature requests. From f01c4df5ec8217f1212806d15a7becd8e5157e4e Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 12 Jan 2026 20:49:36 -0500 Subject: [PATCH 148/891] docs: split up the make command in release doc (#2016) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Just a nit, I like to split up the make command so i can rerun it later Before: Screenshot 2026-01-12 at 9 36 24 AM After: Screenshot 2026-01-12 at 9 36 49 AM ## Are these changes tested? --- website/src/release.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/src/release.md b/website/src/release.md index f3a5798a2b..aec5b3da27 100644 --- a/website/src/release.md +++ b/website/src/release.md @@ -337,6 +337,9 @@ After downloading them, here are the instructions on how to verify them. ```bash tar -xzf apache-iceberg-rust-*.tar.gz cd apache-iceberg-rust-*/ + ``` + + ```bash make build && make test ``` * Verify license headers: From a5df5946a20dc58ca36ec7a8a55072d00d221b51 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 12 Jan 2026 20:52:17 -0500 Subject: [PATCH 149/891] add `make site` for local website preview (#2017) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Add a `make site` command to build and serve the iceberg-rust website. Update README to match ## Are these changes tested? --- Makefile | 6 ++++++ website/README.md | 16 ++-------------- 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/Makefile b/Makefile index ecdada3df5..6b8a039b43 100644 --- a/Makefile +++ b/Makefile @@ -60,3 +60,9 @@ test: doc-test clean: cargo clean + +install-mdbook: + cargo install mdbook@0.4.36 + +site: install-mdbook + cd website && mdbook serve diff --git a/website/README.md b/website/README.md index 9914b82226..8d0f50345d 100644 --- a/website/README.md +++ b/website/README.md @@ -19,22 +19,10 @@ # Iceberg Rust Website -## Setup - -Install mdbook first - -```shell -cargo install mdbook -``` - ## Preview -```shell -mdbook serve -``` - -## Build +This will automatically install mdbook if needed. ```shell -mdbook build +make site ``` From a0424e9e99cc71499c17c32de2dbfb0c4353793b Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Tue, 13 Jan 2026 09:58:37 +0800 Subject: [PATCH 150/891] feat(datafusion): Add schema validation for partition projection (#2008) Implement schema validation in project_with_partition to ensure the input schema matches the Iceberg table schema before calculating partition values. This prevents subtle bugs from schema mismatches and provides clear error messages when schemas don't match. Changes: - Add helper functions to recursively strip metadata from Arrow schemas - Implement schema validation that compares input schema with expected Iceberg table schema, ignoring metadata differences - Add comprehensive tests for metadata stripping and schema validation - Closes #1752 The implementation follows the approach suggested in issue #1752: - Recursively visits schema and removes metadata from all fields - Compares cleaned schemas using Arrow's built-in equality operator - Returns helpful error messages showing both schemas on mismatch ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? Co-authored-by: Claude Sonnet 4.5 --- crates/iceberg/src/arrow/schema.rs | 164 +++++++++++++ .../datafusion/src/physical_plan/project.rs | 220 +++++++++++++++++- 2 files changed, 379 insertions(+), 5 deletions(-) diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index 9ee7897cb6..b15e141c82 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -1125,6 +1125,170 @@ pub fn datum_to_arrow_type_with_ree(datum: &Datum) -> DataType { } } +/// A visitor that strips metadata from an Arrow schema. +/// +/// This visitor recursively removes all metadata from fields at every level of the schema, +/// including nested struct, list, and map fields. This is useful for schema comparison +/// where metadata differences should be ignored. +struct MetadataStripVisitor { + /// Stack to track field information during traversal + field_stack: Vec, +} + +impl MetadataStripVisitor { + fn new() -> Self { + Self { + field_stack: Vec::new(), + } + } +} + +impl ArrowSchemaVisitor for MetadataStripVisitor { + type T = Field; + type U = ArrowSchema; + + fn before_field(&mut self, field: &Field) -> Result<()> { + // Store field name and nullability for later reconstruction + self.field_stack.push(Field::new( + field.name(), + DataType::Null, // Placeholder, will be replaced + field.is_nullable(), + )); + Ok(()) + } + + fn after_field(&mut self, _field: &Field) -> Result<()> { + Ok(()) + } + + fn schema(&mut self, _schema: &ArrowSchema, values: Vec) -> Result { + Ok(ArrowSchema::new(values)) + } + + fn r#struct(&mut self, _fields: &Fields, results: Vec) -> Result { + // Pop the struct field from the stack + let field_info = self + .field_stack + .pop() + .ok_or_else(|| Error::new(ErrorKind::Unexpected, "Field stack underflow in struct"))?; + + // Reconstruct struct field without metadata + Ok(Field::new( + field_info.name(), + DataType::Struct(Fields::from(results)), + field_info.is_nullable(), + )) + } + + fn list(&mut self, list: &DataType, value: Self::T) -> Result { + // Pop the list field from the stack + let field_info = self + .field_stack + .pop() + .ok_or_else(|| Error::new(ErrorKind::Unexpected, "Field stack underflow in list"))?; + + // Reconstruct list field without metadata + let list_type = match list { + DataType::List(_) => DataType::List(Arc::new(value)), + DataType::LargeList(_) => DataType::LargeList(Arc::new(value)), + DataType::FixedSizeList(_, size) => DataType::FixedSizeList(Arc::new(value), *size), + _ => { + return Err(Error::new( + ErrorKind::Unexpected, + format!("Expected list type, got {list}"), + )); + } + }; + + Ok(Field::new( + field_info.name(), + list_type, + field_info.is_nullable(), + )) + } + + fn map(&mut self, map: &DataType, key_value: Self::T, value: Self::T) -> Result { + // Pop the map field from the stack + let field_info = self + .field_stack + .pop() + .ok_or_else(|| Error::new(ErrorKind::Unexpected, "Field stack underflow in map"))?; + + // Reconstruct the map's struct field (contains key and value) + let struct_field = Field::new( + DEFAULT_MAP_FIELD_NAME, + DataType::Struct(Fields::from(vec![key_value, value])), + false, + ); + + // Get the sorted flag from the original map type + let sorted = match map { + DataType::Map(_, sorted) => *sorted, + _ => { + return Err(Error::new( + ErrorKind::Unexpected, + format!("Expected map type, got {map}"), + )); + } + }; + + // Reconstruct map field without metadata + Ok(Field::new( + field_info.name(), + DataType::Map(Arc::new(struct_field), sorted), + field_info.is_nullable(), + )) + } + + fn primitive(&mut self, p: &DataType) -> Result { + // Pop the primitive field from the stack + let field_info = self.field_stack.pop().ok_or_else(|| { + Error::new(ErrorKind::Unexpected, "Field stack underflow in primitive") + })?; + + // Return field without metadata + Ok(Field::new( + field_info.name(), + p.clone(), + field_info.is_nullable(), + )) + } +} + +/// Strips all metadata from an Arrow schema and its nested fields. +/// +/// This function recursively removes metadata from all fields at every level of the schema, +/// including nested struct, list, and map fields. This is useful for schema comparison +/// where metadata differences should be ignored. +/// +/// # Arguments +/// * `schema` - The Arrow schema to strip metadata from +/// +/// # Returns +/// A new Arrow schema with all metadata removed, or an error if the schema structure +/// is invalid. +/// +/// # Example +/// ``` +/// use std::collections::HashMap; +/// +/// use arrow_schema::{DataType, Field, Schema as ArrowSchema}; +/// use iceberg::arrow::strip_metadata_from_schema; +/// +/// let mut metadata = HashMap::new(); +/// metadata.insert("key".to_string(), "value".to_string()); +/// +/// let field = Field::new("col1", DataType::Int32, false).with_metadata(metadata); +/// let schema = ArrowSchema::new(vec![field]); +/// +/// let stripped = strip_metadata_from_schema(&schema).unwrap(); +/// assert!(stripped.field(0).metadata().is_empty()); +/// ``` +pub fn strip_metadata_from_schema(schema: &ArrowSchema) -> Result { + let mut visitor = MetadataStripVisitor::new(); + visit_schema(schema, &mut visitor) +} + #[cfg(test)] mod tests { use std::collections::HashMap; diff --git a/crates/integrations/datafusion/src/physical_plan/project.rs b/crates/integrations/datafusion/src/physical_plan/project.rs index 17492176a4..77a02f51e2 100644 --- a/crates/integrations/datafusion/src/physical_plan/project.rs +++ b/crates/integrations/datafusion/src/physical_plan/project.rs @@ -21,12 +21,15 @@ use std::sync::Arc; use datafusion::arrow::array::RecordBatch; use datafusion::arrow::datatypes::{DataType, Schema as ArrowSchema}; -use datafusion::common::Result as DFResult; +use datafusion::common::{DataFusionError, Result as DFResult}; use datafusion::physical_expr::PhysicalExpr; use datafusion::physical_expr::expressions::Column; use datafusion::physical_plan::projection::ProjectionExec; use datafusion::physical_plan::{ColumnarValue, ExecutionPlan}; -use iceberg::arrow::{PROJECTED_PARTITION_VALUE_COLUMN, PartitionValueCalculator}; +use iceberg::arrow::{ + PROJECTED_PARTITION_VALUE_COLUMN, PartitionValueCalculator, schema_to_arrow_schema, + strip_metadata_from_schema, +}; use iceberg::spec::PartitionSpec; use iceberg::table::Table; @@ -58,8 +61,24 @@ pub fn project_with_partition( } let input_schema = input.schema(); - // TODO: Validate that input_schema matches the Iceberg table schema. - // See: https://github.com/apache/iceberg-rust/issues/1752 + + // Validate that input_schema matches the Iceberg table schema + // Strip metadata from both schemas before comparison to ignore metadata differences + let expected_arrow_schema = + schema_to_arrow_schema(table_schema.as_ref()).map_err(to_datafusion_error)?; + let input_schema_cleaned = + strip_metadata_from_schema(&input_schema).map_err(to_datafusion_error)?; + let expected_schema_cleaned = + strip_metadata_from_schema(&expected_arrow_schema).map_err(to_datafusion_error)?; + + if input_schema_cleaned != expected_schema_cleaned { + return Err(DataFusionError::Plan(format!( + "Input schema does not match Iceberg table schema.\n\ + Expected schema: {expected_schema_cleaned}\n\ + Input schema: {input_schema_cleaned}" + ))); + } + let calculator = PartitionValueCalculator::try_new(partition_spec.as_ref(), table_schema.as_ref()) .map_err(to_datafusion_error)?; @@ -172,7 +191,7 @@ impl std::hash::Hash for PartitionExpr { #[cfg(test)] mod tests { use datafusion::arrow::array::{ArrayRef, Int32Array, StructArray}; - use datafusion::arrow::datatypes::{Field, Fields}; + use datafusion::arrow::datatypes::{DataType, Field, Fields}; use datafusion::physical_plan::empty::EmptyExec; use iceberg::spec::{NestedField, PrimitiveType, Schema, StructType, Transform, Type}; @@ -377,4 +396,195 @@ mod tests { assert_eq!(city_partition.value(0), "New York"); assert_eq!(city_partition.value(1), "Los Angeles"); } + + #[test] + fn test_schema_validation_matching_schemas() { + use iceberg::TableIdent; + use iceberg::io::FileIO; + use iceberg::spec::{FormatVersion, NestedField, PrimitiveType, Schema, Type}; + + let table_schema = Arc::new( + Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + let partition_spec = iceberg::spec::PartitionSpec::builder(table_schema.clone()) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder() + .build(&table_schema) + .unwrap(); + + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + (*table_schema).clone(), + partition_spec, + sort_order, + "/test/table".to_string(), + FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + + // Create Arrow schema matching the table schema + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, false), + ])); + + let input = Arc::new(EmptyExec::new(arrow_schema)); + + let table = iceberg::table::Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/metadata.json".to_string()) + .build() + .unwrap(); + + let result = project_with_partition(input, &table); + assert!(result.is_ok(), "Schema validation should pass"); + } + + #[test] + fn test_schema_validation_mismatched_schemas() { + use iceberg::TableIdent; + use iceberg::io::FileIO; + use iceberg::spec::{FormatVersion, NestedField, PrimitiveType, Schema, Type}; + + let table_schema = Arc::new( + Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + let partition_spec = iceberg::spec::PartitionSpec::builder(table_schema.clone()) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder() + .build(&table_schema) + .unwrap(); + + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + (*table_schema).clone(), + partition_spec, + sort_order, + "/test/table".to_string(), + FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + + // Create Arrow schema with different field name (mismatched) + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("different_name", DataType::Utf8, false), // Wrong field name + ])); + + let input = Arc::new(EmptyExec::new(arrow_schema)); + + let table = iceberg::table::Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/metadata.json".to_string()) + .build() + .unwrap(); + + let result = project_with_partition(input, &table); + assert!( + result.is_err(), + "Schema validation should fail for mismatched schemas" + ); + assert!( + result + .unwrap_err() + .to_string() + .contains("Input schema does not match Iceberg table schema") + ); + } + + #[test] + fn test_schema_validation_with_metadata_differences() { + use std::collections::HashMap; + + use iceberg::TableIdent; + use iceberg::io::FileIO; + use iceberg::spec::{FormatVersion, NestedField, PrimitiveType, Schema, Type}; + + let table_schema = Arc::new( + Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "name", Type::Primitive(PrimitiveType::String)).into(), + ]) + .build() + .unwrap(), + ); + + let partition_spec = iceberg::spec::PartitionSpec::builder(table_schema.clone()) + .add_partition_field("id", "id_partition", Transform::Identity) + .unwrap() + .build() + .unwrap(); + + let sort_order = iceberg::spec::SortOrder::builder() + .build(&table_schema) + .unwrap(); + + let table_metadata_builder = iceberg::spec::TableMetadataBuilder::new( + (*table_schema).clone(), + partition_spec, + sort_order, + "/test/table".to_string(), + FormatVersion::V2, + std::collections::HashMap::new(), + ) + .unwrap(); + + let table_metadata = table_metadata_builder.build().unwrap(); + + // Create Arrow schema with metadata (should be ignored in comparison) + let mut metadata = HashMap::new(); + metadata.insert("extra".to_string(), "metadata".to_string()); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(metadata.clone()), + Field::new("name", DataType::Utf8, false).with_metadata(metadata), + ])); + + let input = Arc::new(EmptyExec::new(arrow_schema)); + + let table = iceberg::table::Table::builder() + .metadata(table_metadata.metadata) + .identifier(TableIdent::from_strs(["test", "table"]).unwrap()) + .file_io(FileIO::from_path("/tmp").unwrap().build().unwrap()) + .metadata_location("/test/metadata.json".to_string()) + .build() + .unwrap(); + + let result = project_with_partition(input, &table); + assert!( + result.is_ok(), + "Schema validation should pass even with metadata differences" + ); + } } From 5cf85dab903e171a077481597be1e49a1159715a Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 13 Jan 2026 16:00:03 -0800 Subject: [PATCH 151/891] feat(datafusion): Support `CREATE TABLE` for DataFusion (#1972) ## Which issue does this PR close? - Closes #1905 ## What changes are included in this PR? - Implement `register_table` in `IcebergSchemaProvider` - Added a new slt: `create_table.slt` - Updated existing slts Note: This does NOT cover the syntax `CREATE TABLE ... AS VALUES` ## Are these changes tested? Yes --- Cargo.lock | 1 + Cargo.toml | 1 + crates/integrations/datafusion/Cargo.toml | 1 + crates/integrations/datafusion/src/schema.rs | 272 ++++++++++++++++-- crates/sqllogictest/src/engine/datafusion.rs | 31 +- .../testdata/schedules/df_test.toml | 4 + .../testdata/slts/df_test/create_table.slt | 90 ++++++ .../testdata/slts/df_test/insert_into.slt | 4 + .../testdata/slts/df_test/show_tables.slt | 3 - 9 files changed, 358 insertions(+), 49 deletions(-) create mode 100644 crates/sqllogictest/testdata/slts/df_test/create_table.slt diff --git a/Cargo.lock b/Cargo.lock index 51af571d5a..3de43e685b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3525,6 +3525,7 @@ version = "0.8.0" dependencies = [ "anyhow", "async-trait", + "dashmap", "datafusion", "expect-test", "futures", diff --git a/Cargo.toml b/Cargo.toml index 56cd1801cc..517bfa36e8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -62,6 +62,7 @@ bytes = "1.10" chrono = "0.4.41" clap = { version = "4.5.48", features = ["derive", "cargo"] } ctor = "0.2.8" +dashmap = "6" datafusion = "51.0" datafusion-cli = "51.0" datafusion-sqllogictest = "51.0" diff --git a/crates/integrations/datafusion/Cargo.toml b/crates/integrations/datafusion/Cargo.toml index 0ee1738b4f..fd3e489e4b 100644 --- a/crates/integrations/datafusion/Cargo.toml +++ b/crates/integrations/datafusion/Cargo.toml @@ -31,6 +31,7 @@ repository = { workspace = true } [dependencies] anyhow = { workspace = true } async-trait = { workspace = true } +dashmap = { workspace = true } datafusion = { workspace = true } futures = { workspace = true } iceberg = { workspace = true } diff --git a/crates/integrations/datafusion/src/schema.rs b/crates/integrations/datafusion/src/schema.rs index 31bbdbd67f..022964ba6d 100644 --- a/crates/integrations/datafusion/src/schema.rs +++ b/crates/integrations/datafusion/src/schema.rs @@ -16,16 +16,20 @@ // under the License. use std::any::Any; -use std::collections::HashMap; use std::sync::Arc; use async_trait::async_trait; +use dashmap::DashMap; use datafusion::catalog::SchemaProvider; use datafusion::datasource::TableProvider; use datafusion::error::{DataFusionError, Result as DFResult}; +use datafusion::execution::TaskContext; +use datafusion::prelude::SessionContext; +use futures::StreamExt; use futures::future::try_join_all; +use iceberg::arrow::arrow_schema_to_schema_auto_assign_ids; use iceberg::inspect::MetadataTableType; -use iceberg::{Catalog, NamespaceIdent, Result}; +use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableCreation}; use crate::table::IcebergTableProvider; use crate::to_datafusion_error; @@ -34,10 +38,15 @@ use crate::to_datafusion_error; /// access to table providers within a specific namespace. #[derive(Debug)] pub(crate) struct IcebergSchemaProvider { - /// A `HashMap` where keys are table names + /// Reference to the Iceberg catalog + catalog: Arc, + /// The namespace this schema represents + namespace: NamespaceIdent, + /// A concurrent map where keys are table names /// and values are dynamic references to objects implementing the /// [`TableProvider`] trait. - tables: HashMap>, + /// Wrapped in Arc to allow sharing across async boundaries in register_table. + tables: Arc>>, } impl IcebergSchemaProvider { @@ -71,13 +80,16 @@ impl IcebergSchemaProvider { ) .await?; - let tables: HashMap> = table_names - .into_iter() - .zip(providers.into_iter()) - .map(|(name, provider)| (name, Arc::new(provider))) - .collect(); + let tables = Arc::new(DashMap::new()); + for (name, provider) in table_names.into_iter().zip(providers.into_iter()) { + tables.insert(name, Arc::new(provider)); + } - Ok(IcebergSchemaProvider { tables }) + Ok(IcebergSchemaProvider { + catalog: client, + namespace, + tables, + }) } } @@ -89,13 +101,16 @@ impl SchemaProvider for IcebergSchemaProvider { fn table_names(&self) -> Vec { self.tables - .keys() - .flat_map(|table_name| { + .iter() + .flat_map(|entry| { + let table_name = entry.key().clone(); [table_name.clone()] .into_iter() - .chain(MetadataTableType::all_types().map(|metadata_table_name| { - format!("{}${}", table_name.clone(), metadata_table_name.as_str()) - })) + .chain( + MetadataTableType::all_types().map(move |metadata_table_name| { + format!("{}${}", table_name, metadata_table_name.as_str()) + }), + ) }) .collect() } @@ -127,7 +142,230 @@ impl SchemaProvider for IcebergSchemaProvider { Ok(self .tables .get(name) - .cloned() - .map(|t| t as Arc)) + .map(|entry| entry.value().clone() as Arc)) + } + + fn register_table( + &self, + name: String, + table: Arc, + ) -> DFResult>> { + // Check if table already exists + if self.table_exist(name.as_str()) { + return Err(DataFusionError::Execution(format!( + "Table {name} already exists" + ))); + } + + // Convert DataFusion schema to Iceberg schema + // DataFusion schemas don't have field IDs, so we use the function that assigns them automatically + let df_schema = table.schema(); + let iceberg_schema = arrow_schema_to_schema_auto_assign_ids(df_schema.as_ref()) + .map_err(to_datafusion_error)?; + + // Create the table in the Iceberg catalog + let table_creation = TableCreation::builder() + .name(name.clone()) + .schema(iceberg_schema) + .build(); + + let catalog = self.catalog.clone(); + let namespace = self.namespace.clone(); + let tables = self.tables.clone(); + let name_clone = name.clone(); + + // Use tokio's spawn_blocking to handle the async work on a blocking thread pool + let result = tokio::task::spawn_blocking(move || { + // Create a new runtime handle to execute the async work + let rt = tokio::runtime::Handle::current(); + rt.block_on(async move { + // Verify the input table is empty - CREATE TABLE only accepts schema definition + ensure_table_is_empty(&table) + .await + .map_err(to_datafusion_error)?; + + catalog + .create_table(&namespace, table_creation) + .await + .map_err(to_datafusion_error)?; + + // Create a new table provider using the catalog reference + let table_provider = IcebergTableProvider::try_new( + catalog.clone(), + namespace.clone(), + name_clone.clone(), + ) + .await + .map_err(to_datafusion_error)?; + + // Store the new table provider + tables.insert(name_clone, Arc::new(table_provider)); + + Ok(None) + }) + }); + + // Block on the spawned task to get the result + // This is safe because spawn_blocking moves the blocking to a dedicated thread pool + futures::executor::block_on(result).map_err(|e| { + DataFusionError::Execution(format!("Failed to create Iceberg table: {e}")) + })? + } +} + +/// Verifies that a table provider contains no data by scanning with LIMIT 1. +/// Returns an error if the table has any rows. +async fn ensure_table_is_empty(table: &Arc) -> Result<()> { + let session_ctx = SessionContext::new(); + let exec_plan = table + .scan(&session_ctx.state(), None, &[], Some(1)) + .await + .map_err(|e| Error::new(ErrorKind::Unexpected, format!("Failed to scan table: {e}")))?; + + let task_ctx = Arc::new(TaskContext::default()); + let stream = exec_plan.execute(0, task_ctx).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to execute scan: {e}"), + ) + })?; + + let batches: Vec<_> = stream.collect().await; + let has_data = batches + .into_iter() + .filter_map(|r| r.ok()) + .any(|batch| batch.num_rows() > 0); + + if has_data { + return Err(Error::new( + ErrorKind::Unexpected, + "register_table does not support tables with data.", + )); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use datafusion::arrow::array::{Int32Array, StringArray}; + use datafusion::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; + use datafusion::arrow::record_batch::RecordBatch; + use datafusion::datasource::MemTable; + use iceberg::memory::{MEMORY_CATALOG_WAREHOUSE, MemoryCatalogBuilder}; + use iceberg::{Catalog, CatalogBuilder, NamespaceIdent}; + use tempfile::TempDir; + + use super::*; + + async fn create_test_schema_provider() -> (IcebergSchemaProvider, TempDir) { + let temp_dir = TempDir::new().unwrap(); + let warehouse_path = temp_dir.path().to_str().unwrap().to_string(); + + let catalog = MemoryCatalogBuilder::default() + .load( + "memory", + HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), warehouse_path.clone())]), + ) + .await + .unwrap(); + + let namespace = NamespaceIdent::new("test_ns".to_string()); + catalog + .create_namespace(&namespace, HashMap::new()) + .await + .unwrap(); + + let provider = IcebergSchemaProvider::try_new(Arc::new(catalog), namespace) + .await + .unwrap(); + + (provider, temp_dir) + } + + #[tokio::test] + async fn test_register_table_with_data_fails() { + let (schema_provider, _temp_dir) = create_test_schema_provider().await; + + // Create a MemTable with data + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, true), + ])); + + let batch = RecordBatch::try_new(arrow_schema.clone(), vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ]) + .unwrap(); + + let mem_table = MemTable::try_new(arrow_schema, vec![vec![batch]]).unwrap(); + + // Attempt to register the table with data - should fail + let result = schema_provider.register_table("test_table".to_string(), Arc::new(mem_table)); + + assert!(result.is_err()); + let err = result.unwrap_err(); + assert!( + err.to_string() + .contains("register_table does not support tables with data."), + "Expected error about tables with data, got: {err}", + ); + } + + #[tokio::test] + async fn test_register_empty_table_succeeds() { + let (schema_provider, _temp_dir) = create_test_schema_provider().await; + + // Create an empty MemTable (schema only, no data rows) + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false), + Field::new("name", DataType::Utf8, true), + ])); + + // Create an empty batch (0 rows) - MemTable requires at least one partition + let empty_batch = RecordBatch::new_empty(arrow_schema.clone()); + let mem_table = MemTable::try_new(arrow_schema, vec![vec![empty_batch]]).unwrap(); + + // Attempt to register the empty table - should succeed + let result = schema_provider.register_table("empty_table".to_string(), Arc::new(mem_table)); + + assert!(result.is_ok(), "Expected success, got: {result:?}"); + + // Verify the table was registered + assert!(schema_provider.table_exist("empty_table")); + } + + #[tokio::test] + async fn test_register_duplicate_table_fails() { + let (schema_provider, _temp_dir) = create_test_schema_provider().await; + + // Create empty MemTables + let arrow_schema = Arc::new(ArrowSchema::new(vec![Field::new( + "id", + DataType::Int32, + false, + )])); + + let empty_batch1 = RecordBatch::new_empty(arrow_schema.clone()); + let empty_batch2 = RecordBatch::new_empty(arrow_schema.clone()); + let mem_table1 = MemTable::try_new(arrow_schema.clone(), vec![vec![empty_batch1]]).unwrap(); + let mem_table2 = MemTable::try_new(arrow_schema, vec![vec![empty_batch2]]).unwrap(); + + // Register first table - should succeed + let result1 = schema_provider.register_table("dup_table".to_string(), Arc::new(mem_table1)); + assert!(result1.is_ok()); + + // Register second table with same name - should fail + let result2 = schema_provider.register_table("dup_table".to_string(), Arc::new(mem_table2)); + assert!(result2.is_err()); + let err = result2.unwrap_err(); + assert!( + err.to_string().contains("already exists"), + "Expected error about table already existing, got: {err}", + ); } } diff --git a/crates/sqllogictest/src/engine/datafusion.rs b/crates/sqllogictest/src/engine/datafusion.rs index e9f93287d8..487d8dc977 100644 --- a/crates/sqllogictest/src/engine/datafusion.rs +++ b/crates/sqllogictest/src/engine/datafusion.rs @@ -93,8 +93,7 @@ impl DataFusionEngine { let namespace = NamespaceIdent::new("default".to_string()); catalog.create_namespace(&namespace, HashMap::new()).await?; - // Create test tables - Self::create_unpartitioned_table(&catalog, &namespace).await?; + // Create partitioned test table (unpartitioned tables are now created via SQL) Self::create_partitioned_table(&catalog, &namespace).await?; Ok(Arc::new( @@ -102,35 +101,9 @@ impl DataFusionEngine { )) } - /// Create an unpartitioned test table with id and name columns - /// TODO: this can be removed when we support CREATE TABLE - async fn create_unpartitioned_table( - catalog: &impl Catalog, - namespace: &NamespaceIdent, - ) -> anyhow::Result<()> { - let schema = Schema::builder() - .with_fields(vec![ - NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), - NestedField::optional(2, "name", Type::Primitive(PrimitiveType::String)).into(), - ]) - .build()?; - - catalog - .create_table( - namespace, - TableCreation::builder() - .name("test_unpartitioned_table".to_string()) - .schema(schema) - .build(), - ) - .await?; - - Ok(()) - } - /// Create a partitioned test table with id, category, and value columns /// Partitioned by category using identity transform - /// TODO: this can be removed when we support CREATE TABLE + /// TODO: this can be removed when we support CREATE EXTERNAL TABLE async fn create_partitioned_table( catalog: &impl Catalog, namespace: &NamespaceIdent, diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index df5e638d5a..1d7f42c8d4 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -22,6 +22,10 @@ df = { type = "datafusion" } engine = "df" slt = "df_test/show_tables.slt" +[[steps]] +engine = "df" +slt = "df_test/create_table.slt" + [[steps]] engine = "df" slt = "df_test/insert_into.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/create_table.slt b/crates/sqllogictest/testdata/slts/df_test/create_table.slt new file mode 100644 index 0000000000..2eab1b6bab --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/create_table.slt @@ -0,0 +1,90 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test CREATE TABLE with explicit schema +statement ok +CREATE TABLE default.default.empty_table (id INT NOT NULL, name STRING) + +# Verify the empty table exists and has correct schema +query IT rowsort +SELECT * FROM default.default.empty_table +---- + +# Insert data into the created table +query I +INSERT INTO default.default.empty_table VALUES (1, 'Alice') +---- +1 + +# Verify the inserted data +query IT rowsort +SELECT * FROM default.default.empty_table +---- +1 Alice + +# Insert multiple rows +query I +INSERT INTO default.default.empty_table VALUES (2, 'Bob'), (3, 'Charlie') +---- +2 + +# Verify all rows +query IT rowsort +SELECT * FROM default.default.empty_table +---- +1 Alice +2 Bob +3 Charlie + +# Test CREATE TABLE with different column types +statement ok +CREATE TABLE default.default.typed_table (id BIGINT NOT NULL, value DOUBLE, flag BOOLEAN) + +# Verify the typed table exists +query IDT rowsort +SELECT * FROM default.default.typed_table +---- + +# Insert data with different types +query I +INSERT INTO default.default.typed_table VALUES (100, 3.14, true), (200, 2.71, false) +---- +2 + +# Verify typed data +query IDT rowsort +SELECT * FROM default.default.typed_table +---- +100 3.14 true +200 2.71 false + +# Test CREATE TABLE with nullable columns +statement ok +CREATE TABLE default.default.nullable_table (id INT NOT NULL, optional_name STRING) + +# Insert with NULL value +query I +INSERT INTO default.default.nullable_table VALUES (1, 'Value'), (2, NULL) +---- +2 + +# Verify NULL handling +query IT rowsort +SELECT * FROM default.default.nullable_table +---- +1 Value +2 NULL diff --git a/crates/sqllogictest/testdata/slts/df_test/insert_into.slt b/crates/sqllogictest/testdata/slts/df_test/insert_into.slt index 2ba33afcd1..1e07844326 100644 --- a/crates/sqllogictest/testdata/slts/df_test/insert_into.slt +++ b/crates/sqllogictest/testdata/slts/df_test/insert_into.slt @@ -15,6 +15,10 @@ # specific language governing permissions and limitations # under the License. +# Create unpartitioned test table +statement ok +CREATE TABLE default.default.test_unpartitioned_table (id INT NOT NULL, name STRING) + # Verify the table is initially empty query IT rowsort SELECT * FROM default.default.test_unpartitioned_table diff --git a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt index c5da5f6276..770072f9dc 100644 --- a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt +++ b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt @@ -28,9 +28,6 @@ datafusion information_schema views VIEW default default test_partitioned_table BASE TABLE default default test_partitioned_table$manifests BASE TABLE default default test_partitioned_table$snapshots BASE TABLE -default default test_unpartitioned_table BASE TABLE -default default test_unpartitioned_table$manifests BASE TABLE -default default test_unpartitioned_table$snapshots BASE TABLE default information_schema columns VIEW default information_schema df_settings VIEW default information_schema parameters VIEW From 459db8a13fd038df8ed18dd1d2685bdcaecb845e Mon Sep 17 00:00:00 2001 From: Andrea Bozzo Date: Thu, 15 Jan 2026 02:06:32 +0100 Subject: [PATCH 152/891] chore: Update copyright year in NOTICE files (#2024) ## Which issue does this PR close? - Closes #2015. ## What changes are included in this PR? Update copyright year from `2023-2024` to `2023-2026` in NOTICE files. ## Are these changes tested? No code changes, only documentation update. --- NOTICE | 2 +- crates/catalog/s3tables/NOTICE | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/NOTICE b/NOTICE index 9340680cbd..73b77516fa 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache Iceberg Rust -Copyright 2023-2024 The Apache Software Foundation +Copyright 2023-2026 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). diff --git a/crates/catalog/s3tables/NOTICE b/crates/catalog/s3tables/NOTICE index 9340680cbd..73b77516fa 100644 --- a/crates/catalog/s3tables/NOTICE +++ b/crates/catalog/s3tables/NOTICE @@ -1,5 +1,5 @@ Apache Iceberg Rust -Copyright 2023-2024 The Apache Software Foundation +Copyright 2023-2026 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). From b05a675db44645becc60422b596f16cca8816a89 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Thu, 15 Jan 2026 02:21:08 -0500 Subject: [PATCH 153/891] infra: Restrict audit workflow to main apache/iceberg-rust repository (#1974) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? I noticed this workflow will run on my forked repo periodically. Let's disable it for forked repos ## Are these changes tested? --- .github/workflows/audit.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index fc6b1224e8..a86c58e5c7 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -39,6 +39,7 @@ on: jobs: security_audit: runs-on: ubuntu-latest + if: github.repository == 'apache/iceberg-rust' steps: - uses: actions/checkout@v6 - name: Setup Rust toolchain From 4efea04db1d91b9aa0cd2f57ef97b991cac1d40c Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Thu, 15 Jan 2026 16:54:14 -0800 Subject: [PATCH 154/891] rfc: Making Storage a Trait (#1885) ## Which issue does this PR close? - Related to - #1314 - #172 ## What changes are included in this PR? ## Are these changes tested? --------- Co-authored-by: Renjie Liu --- docs/rfcs/0002_storage_trait.md | 862 ++++++++++++++++++++++++++++++++ 1 file changed, 862 insertions(+) create mode 100644 docs/rfcs/0002_storage_trait.md diff --git a/docs/rfcs/0002_storage_trait.md b/docs/rfcs/0002_storage_trait.md new file mode 100644 index 0000000000..b68141aec4 --- /dev/null +++ b/docs/rfcs/0002_storage_trait.md @@ -0,0 +1,862 @@ + + +# Making Storage a Trait + +## Background + +### Existing Implementation + +The existing code implements storage functionality through a concrete `Storage` enum that handles different storage backends (S3, local filesystem, GCS, etc.). This implementation is tightly coupled with OpenDAL as the underlying storage layer. The `FileIO` struct wraps this `Storage` enum and provides a high-level API for file operations. + +```rust +// Current: Concrete enum with variants for each backend +pub(crate) enum Storage { + #[cfg(feature = "storage-memory")] + Memory(Operator), + #[cfg(feature = "storage-fs")] + LocalFs, + #[cfg(feature = "storage-s3")] + S3 { + configured_scheme: String, + config: Arc, + customized_credential_load: Option, + }, + #[cfg(feature = "storage-gcs")] + Gcs { config: Arc }, + // ... other variants +} +``` + +Current structure: + +- **FileIO:** Main interface for file operations, wraps `Arc` +- **Storage:** Enum with variants for different storage backends +- **InputFile / OutputFile:** Concrete structs that hold an `Operator` and path + +### Problem Statement + +The original design has several limitations: + +- **Tight Coupling** – All storage logic depends on OpenDAL, limiting flexibility. Users cannot easily opt in for other storage implementations like `object_store` +- **Customization Barriers** – Users cannot easily add custom behaviors or optimizations +- **No Extensibility** – Adding new backends requires modifying the core enum in the `iceberg` crate + +As discussed in Issue #1314, making Storage a trait would allow pluggable storage and better integration with existing systems. + +--- + +## High-Level Architecture + +The new design introduces a trait-based storage abstraction with a factory pattern for creating storage instances. This enables pluggable storage backends while maintaining a clean separation between the core Iceberg library and storage implementations. + +### Component Overview + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ User Application │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Catalog │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ CatalogBuilder::with_storage_factory(storage_factory) │ │ +│ │ - Accepts optional StorageFactory injection │ │ +│ │ - Falls back to LocalFsStorageFactory if not provided │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ FileIO / FileIOBuilder │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ FileIOBuilder: │ │ +│ │ - factory: Arc │ │ +│ │ - config: StorageConfig │ │ +│ │ - Methods: new(), with_prop(), with_props(), config(), build() │ │ +│ │ │ │ +│ │ FileIO: │ │ +│ │ - config: StorageConfig (properties only, no scheme) │ │ +│ │ - factory: Arc │ │ +│ │ - storage: OnceCell> (lazy initialization) │ │ +│ │ - Methods: new_with_memory(), new_with_fs(), into_builder() │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ Methods: new_input(), new_output(), delete(), exists(), delete_prefix() │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────┴─────────────────┐ + ▼ ▼ +┌───────────────────────────────┐ ┌───────────────────────────────────────┐ +│ StorageFactory │ │ Storage │ +│ (trait in iceberg crate) │ │ (trait in iceberg crate) │ +│ │ │ │ +│ fn build(&self, config) │────▶│ async fn exists(&self, path) │ +│ -> Arc │ │ async fn read(&self, path) │ +│ │ │ async fn write(&self, path, bytes) │ +│ │ │ async fn delete(&self, path) │ +│ │ │ fn new_input(&self, path) │ +│ │ │ fn new_output(&self, path) │ +└───────────────────────────────┘ └───────────────────────────────────────┘ + │ ▲ + │ │ + ▼ │ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Storage Implementations │ +│ │ +│ ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐ │ +│ │ MemoryStorage │ │ LocalFsStorage │ │ OpenDalStorage │ │ +│ │ (iceberg crate) │ │ (iceberg crate) │ │ (iceberg-storage- │ │ +│ │ │ │ │ │ opendal) │ │ +│ │ - In-memory HashMap│ │ - std::fs ops │ │ - S3, GCS, Azure │ │ +│ │ - For testing │ │ - For local files │ │ - OSS, filesystem │ │ +│ └─────────────────────┘ └─────────────────────┘ └─────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### Data Flow + +``` +┌──────────────────────────────────────────────────────────────────────────────┐ +│ FileIO Creation Flow │ +└──────────────────────────────────────────────────────────────────────────────┘ + + User Code FileIOBuilder StorageFactory + │ │ │ + │ FileIOBuilder::new() │ │ + │─────────────────────────▶│ │ + │ │ │ + │ .with_prop() │ │ + │─────────────────────────▶│ │ + │ │ │ + │ .build() │ │ + │─────────────────────────▶│ │ + │ │ │ + │◀─────────────────────────│ │ + │ FileIO │ │ + │ │ │ + │ new_input(path) │ │ + │─────────────────────────▶│ │ + │ │ (lazy) factory.build() │ + │ │─────────────────────────────▶│ + │ │ │ + │ │◀─────────────────────────────│ + │ │ Arc │ + │ │ │ + │◀─────────────────────────│ │ + │ InputFile │ │ + + +┌──────────────────────────────────────────────────────────────────────────────┐ +│ Catalog with FileIO Injection │ +└──────────────────────────────────────────────────────────────────────────────┘ + + User Code CatalogBuilder Catalog + │ │ │ + │ ::default() │ │ + │─────────────────────────▶│ │ + │ │ │ + │ .with_storage_factory(factory) │ │ + │─────────────────────────────────▶│ │ + │ │ │ + │ .load(name, props) │ │ + │─────────────────────────▶│ │ + │ │ new(config, Some(file_io)) │ + │ │─────────────────────────────▶│ + │ │ │ + │◀─────────────────────────│◀─────────────────────────────│ + │ Catalog │ │ +``` + +### Crate Structure + +``` +crates/ +├── iceberg/ # Core Iceberg functionality +│ └── src/ +│ └── io/ +│ ├── mod.rs # Re-exports +│ ├── storage.rs # Storage + StorageFactory traits +│ ├── file_io.rs # FileIO, InputFile, OutputFile +│ ├── config/ # StorageConfig and backend configs +│ │ ├── mod.rs # StorageConfig +│ │ ├── s3.rs # S3Config constants +│ │ ├── gcs.rs # GcsConfig constants +│ │ ├── oss.rs # OssConfig constants +│ │ └── azdls.rs # AzdlsConfig constants +│ ├── memory.rs # MemoryStorage (built-in) +│ └── local_fs.rs # LocalFsStorage (built-in) +│ +├── storage/ +│ └── opendal/ # OpenDAL-based implementations +│ └── src/ +│ ├── lib.rs # Re-exports +│ ├── storage.rs # OpenDalStorage + OpenDalStorageFactory +│ ├── storage_s3.rs # S3 support +│ ├── storage_gcs.rs # GCS support +│ ├── storage_oss.rs # OSS support +│ ├── storage_azdls.rs # Azure support +│ └── storage_fs.rs # Filesystem support +│ +└── catalog/ # Catalog implementations + ├── rest/ # Uses with_storage_factory injection + ├── glue/ # Uses with_storage_factory injection + ├── hms/ # Uses with_storage_factory injection + ├── s3tables/ # Uses with_storage_factory injection + └── sql/ # Uses with_storage_factory injection +``` + +--- + +## Design Phase 1: Storage Trait and Core Types + +Phase 1 focuses on converting Storage from an enum to a trait, introducing `StorageFactory` and `StorageConfig`, and updating `FileIO`, `InputFile`, and `OutputFile` to use the trait-based abstraction. + +### Storage Trait + +The `Storage` trait is defined in the `iceberg` crate and provides the interface for all storage operations. It uses `typetag` for serialization support across process boundaries. + +```rust +#[async_trait] +#[typetag::serde(tag = "type")] +pub trait Storage: Debug + Send + Sync { + /// Check if a file exists at the given path + async fn exists(&self, path: &str) -> Result; + + /// Get metadata from an input path + async fn metadata(&self, path: &str) -> Result; + + /// Read bytes from a path + async fn read(&self, path: &str) -> Result; + + /// Get FileRead from a path + async fn reader(&self, path: &str) -> Result>; + + /// Write bytes to an output path + async fn write(&self, path: &str, bs: Bytes) -> Result<()>; + + /// Get FileWrite from a path + async fn writer(&self, path: &str) -> Result>; + + /// Delete a file at the given path + async fn delete(&self, path: &str) -> Result<()>; + + /// Delete all files with the given prefix + async fn delete_prefix(&self, path: &str) -> Result<()>; + + /// Create a new input file for reading + fn new_input(&self, path: &str) -> Result; + + /// Create a new output file for writing + fn new_output(&self, path: &str) -> Result; +} +``` + +### StorageFactory Trait + +The `StorageFactory` trait creates `Storage` instances from configuration. This enables lazy initialization and custom storage injection. + +```rust +#[typetag::serde(tag = "type")] +pub trait StorageFactory: Debug + Send + Sync { + /// Build a new Storage instance from the given configuration. + fn build(&self, config: &StorageConfig) -> Result>; +} +``` + +### StorageConfig + +`StorageConfig` is a pure property container without scheme. The storage type is determined by explicit factory selection: + +```rust +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)] +pub struct StorageConfig { + /// Configuration properties for the storage backend + props: HashMap, +} + +impl StorageConfig { + pub fn new() -> Self; + pub fn from_props(props: HashMap) -> Self; + pub fn props(&self) -> &HashMap; + pub fn get(&self, key: &str) -> Option<&String>; + pub fn with_prop(self, key: impl Into, value: impl Into) -> Self; + pub fn with_props(self, props: impl IntoIterator, impl Into)>) -> Self; +} +``` + +#### Backend-Specific Configuration Types + +In addition to `StorageConfig`, we provide typed configuration structs for each storage backend. +These can be constructed from `StorageConfig` using `TryFrom` and provide a structured way to access backend-specific settings: + +- `S3Config` - Amazon S3 configuration +- `GcsConfig` - Google Cloud Storage configuration +- `OssConfig` - Alibaba Cloud OSS configuration +- `AzdlsConfig` - Azure Data Lake Storage configuration + +Example of `S3Config`: + +```rust +#[derive(Clone, Debug, Default, Serialize, Deserialize, TypedBuilder)] +pub struct S3Config { + #[builder(default, setter(strip_option, into))] + endpoint: Option, + #[builder(default, setter(strip_option, into))] + access_key_id: Option, + #[builder(default, setter(strip_option, into))] + secret_access_key: Option, + #[builder(default, setter(strip_option, into))] + region: Option, + #[builder(default)] + allow_anonymous: bool, + // ... other S3-specific fields +} + +impl S3Config { + /// Returns the S3 endpoint URL. + pub fn endpoint(&self) -> Option<&str> { + self.endpoint.as_deref() + } + + // ... other getter methods +} + +// Fallible conversion from StorageConfig +impl TryFrom<&StorageConfig> for S3Config { + type Error = iceberg::Error; + + fn try_from(config: &StorageConfig) -> Result {/* ... */} +} +``` + +Usage with the builder pattern: + +```rust +let s3_config = S3Config::builder() + .region("us-east-1") + .access_key_id("my-access-key") + .secret_access_key("my-secret-key") + .build(); + +assert_eq!(s3_config.region(), Some("us-east-1")); +``` + +These typed configs are used internally by storage implementations (e.g., `OpenDalStorage`) to +parse properties from `StorageConfig` into strongly-typed configuration. + +### FileIO and FileIOBuilder + +`FileIO` is redesigned to use lazy storage initialization with a factory pattern. Configuration is done via `FileIOBuilder`: + +```rust +#[derive(Clone)] +pub struct FileIO { + /// Storage configuration containing properties + config: StorageConfig, + /// Factory for creating storage instances + factory: Arc, + /// Cached storage instance (lazily initialized) + storage: Arc>>, +} + +impl FileIO { + /// Create a new FileIO backed by in-memory storage. + pub fn new_with_memory() -> Self; + + /// Create a new FileIO backed by local filesystem storage. + pub fn new_with_fs() -> Self; + + /// Convert this FileIO into a FileIOBuilder for modification. + pub fn into_builder(self) -> FileIOBuilder; + + // File operations delegate to the lazily-initialized storage + pub async fn delete(&self, path: impl AsRef) -> Result<()>; + pub async fn delete_prefix(&self, path: impl AsRef) -> Result<()>; + pub async fn exists(&self, path: impl AsRef) -> Result; + pub fn new_input(&self, path: impl AsRef) -> Result; + pub fn new_output(&self, path: impl AsRef) -> Result; +} + +/// Builder for creating FileIO instances. +pub struct FileIOBuilder { + factory: Arc, + config: StorageConfig, +} + +impl FileIOBuilder { + /// Create a new FileIOBuilder with the given storage factory. + pub fn new(factory: Arc) -> Self; + + /// Add a configuration property. + pub fn with_prop(self, key: impl Into, value: impl Into) -> Self; + + /// Add multiple configuration properties. + pub fn with_props(self, props: impl IntoIterator, impl Into)>) -> Self; + + /// Get the storage configuration. + pub fn config(&self) -> &StorageConfig; + + /// Build the FileIO instance. + pub fn build(self) -> Result; +} +``` + +Key changes from the old design: +- `FileIOBuilder` is used for configuration with explicit factory injection +- `FileIO` has convenience constructors (`new_with_memory()`, `new_with_fs()`) for common cases +- Removed `Extensions` - custom behavior is now provided via `StorageFactory` +- Storage is lazily initialized on first use via `OnceCell` + +### InputFile and OutputFile Changes + +`InputFile` and `OutputFile` now hold a reference to `Arc` instead of an `Operator`: + +```rust +pub struct InputFile { + storage: Arc, + path: String, +} + +impl InputFile { + pub fn new(storage: Arc, path: String) -> Self; + pub fn location(&self) -> &str; + pub async fn exists(&self) -> Result; + pub async fn metadata(&self) -> Result; + pub async fn read(&self) -> Result; + pub async fn reader(&self) -> Result>; +} + +pub struct OutputFile { + storage: Arc, + path: String, +} + +impl OutputFile { + pub fn new(storage: Arc, path: String) -> Self; + pub fn location(&self) -> &str; + pub async fn exists(&self) -> Result; + pub async fn delete(&self) -> Result<()>; + pub fn to_input_file(self) -> InputFile; + pub async fn write(&self, bs: Bytes) -> Result<()>; + pub async fn writer(&self) -> Result>; +} +``` + +### Built-in Storage Implementations + +The `iceberg` crate includes two built-in storage implementations for testing and basic use cases: + +#### MemoryStorage + +In-memory storage using a thread-safe `HashMap`, primarily for testing: + +```rust +#[derive(Debug, Clone, Default)] +pub struct MemoryStorage { + data: Arc>>, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct MemoryStorageFactory; + +#[typetag::serde] +impl StorageFactory for MemoryStorageFactory { + fn build(&self, config: &StorageConfig) -> Result> { + if config.scheme() != "memory" { + return Err(/* error */); + } + Ok(Arc::new(MemoryStorage::new())) + } +} +``` + +#### LocalFsStorage + +Local filesystem storage using standard Rust `std::fs` operations: + +```rust +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct LocalFsStorage; + +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct LocalFsStorageFactory; + +#[typetag::serde] +impl StorageFactory for LocalFsStorageFactory { + fn build(&self, config: &StorageConfig) -> Result> { + if config.scheme() != "file" { + return Err(/* error */); + } + Ok(Arc::new(LocalFsStorage::new())) + } +} +``` + +### CatalogBuilder Changes + +The `CatalogBuilder` trait is extended with `with_storage_factory()` to allow StorageFactory injection: + +```rust +pub trait CatalogBuilder: Default + Debug + Send + Sync { + type C: Catalog; + + /// Set a custom StorageFactory to use for storage operations. + /// + /// When a StorageFactory is provided, the catalog will use it to build FileIO + /// instances for all storage operations instead of using the default factory. + fn with_storage_factory(self, storage_factory: Arc) -> Self; + + /// Create a new catalog instance. + fn load( + self, + name: impl Into, + props: HashMap, + ) -> impl Future> + Send; +} +``` + +Catalog implementations store the optional `StorageFactory` and use it when provided: + +```rust +pub struct GlueCatalogBuilder { + config: GlueCatalogConfig, + storage_factory: Option>, // New field +} + +impl CatalogBuilder for GlueCatalogBuilder { + fn with_storage_factory(mut self, storage_factory: Arc) -> Self { + self.storage_factory = Some(storage_factory); + self + } + + // In load(): + // Use provided StorageFactory or LocalFsStorageFactory as fallback + let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory)); + let file_io = FileIOBuilder::new(factory) + .with_props(file_io_props) + .build()?; +} +``` + +--- + +## Design Part 2: Separate Storage Crate + +Phase 2 moves concrete OpenDAL-based implementations to a separate crate (`iceberg-storage-opendal`). + +### iceberg-storage-opendal Crate + +This crate provides OpenDAL-based storage implementations for cloud storage backends: + +```rust +// crates/storage/opendal/src/storage.rs + +/// Explicit storage factory variants for OpenDAL-based backends. +/// +/// Each variant represents a specific storage backend. Path scheme +/// validation is handled by the underlying Storage implementation +/// when operations are performed. +#[derive(Clone, Debug, Serialize, Deserialize)] +pub enum OpenDalStorageFactory { + /// Local filesystem storage factory. + #[cfg(feature = "storage-fs")] + Fs, + + /// Amazon S3 storage factory. + #[cfg(feature = "storage-s3")] + S3, + + /// Google Cloud Storage factory. + #[cfg(feature = "storage-gcs")] + Gcs, + + /// Alibaba Cloud OSS storage factory. + #[cfg(feature = "storage-oss")] + Oss, + + /// Azure Data Lake Storage factory. + #[cfg(feature = "storage-azdls")] + Azdls, +} + +#[typetag::serde] +impl StorageFactory for OpenDalStorageFactory { + fn build(&self, config: &StorageConfig) -> Result> { + let storage = match self { + #[cfg(feature = "storage-fs")] + Self::Fs => OpenDalStorage::LocalFs, + + #[cfg(feature = "storage-s3")] + Self::S3 => { + let iceberg_s3_config = S3Config::try_from(config)?; + let opendal_s3_config = s3_config_to_opendal(&iceberg_s3_config); + OpenDalStorage::S3 { + configured_scheme: "s3".to_string(), + config: opendal_s3_config.into(), + customized_credential_load: None, + } + } + // ... other variants + }; + Ok(Arc::new(storage)) + } +} + +/// Unified OpenDAL-based storage implementation. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum OpenDalStorage { + #[cfg(feature = "storage-fs")] + LocalFs, + + #[cfg(feature = "storage-s3")] + S3 { + configured_scheme: String, + config: Arc, + #[serde(skip)] + customized_credential_load: Option, + }, + + #[cfg(feature = "storage-gcs")] + Gcs { config: Arc }, + + #[cfg(feature = "storage-oss")] + Oss { config: Arc }, + + #[cfg(feature = "storage-azdls")] + Azdls { + configured_scheme: AzureStorageScheme, + config: Arc, + }, +} + +impl OpenDalStorage { + /// Creates operator from path. + fn create_operator<'a>(&self, path: &'a str) -> Result<(Operator, &'a str)>; +} + +#[async_trait] +#[typetag::serde] +impl Storage for OpenDalStorage { + // Delegates all operations to the appropriate OpenDAL operator +} +``` + +Feature flags in `iceberg-storage-opendal`: +- `storage-s3` : Enables S3 storage backend +- `storage-gcs`: Enables Google Cloud Storage backend +- `storage-oss`: Enables Alibaba Cloud OSS backend +- `storage-azdls`: Enables Azure Data Lake Storage backend +- `storage-fs`: Enables OpenDAL filesystem backend +- `storage-all`: Enables all storage backends + +--- + +## Example Usage + +### Basic Usage with Memory Storage (Testing) + +```rust +use iceberg::io::FileIO; + +// Create in-memory FileIO for testing +let file_io = FileIO::new_with_memory(); + +// Write and read files +let output = file_io.new_output("memory://test/file.txt")?; +output.write("Hello, World!".into()).await?; + +let input = file_io.new_input("memory://test/file.txt")?; +let content = input.read().await?; +assert_eq!(content, bytes::Bytes::from("Hello, World!")); +``` + +### Using OpenDAL Storage Factory + +```rust +use std::sync::Arc; +use iceberg::io::FileIOBuilder; +use iceberg_storage_opendal::OpenDalStorageFactory; + +// Create FileIO with explicit S3 factory +let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::S3)) + .with_prop("s3.region", "us-east-1") + .with_prop("s3.access-key-id", "my-access-key") + .with_prop("s3.secret-access-key", "my-secret-key") + .build()?; + +// Use the FileIO +let input = file_io.new_input("s3://my-bucket/warehouse/table/metadata.json")?; +let metadata = input.read().await?; +``` + +### Using Catalogs with Custom Storage + +When using a catalog without injecting a custom `StorageFactory`, the catalog falls back to +`LocalFsStorageFactory`. For cloud storage, inject the appropriate factory: + +```rust +use std::collections::HashMap; +use std::sync::Arc; +use iceberg::CatalogBuilder; +use iceberg_catalog_glue::GlueCatalogBuilder; +use iceberg_storage_opendal::OpenDalStorageFactory; + +// Inject S3 storage factory for cloud storage support +let catalog = GlueCatalogBuilder::default() + .with_storage_factory(Arc::new(OpenDalStorageFactory::S3)) + .load("my_catalog", HashMap::from([ + ("warehouse".to_string(), "s3://my-bucket/warehouse".to_string()), + ("s3.region".to_string(), "us-east-1".to_string()), + ])) + .await?; + +// Load and scan a table - storage is handled automatically +let table = catalog.load_table(&TableIdent::from_strs(["db", "my_table"])?).await?; +let scan = table.scan().build()?; +``` + +### Injecting Custom StorageFactory into Catalogs + +For advanced use cases, you can inject a custom `StorageFactory` with specific storage configuration: + +```rust +use std::collections::HashMap; +use std::sync::Arc; +use iceberg::CatalogBuilder; +use iceberg::io::FileIOBuilder; +use iceberg_catalog_glue::GlueCatalogBuilder; +use iceberg_storage_opendal::OpenDalStorageFactory; + +// Create a custom StorageFactory +let storage_factory = Arc::new(OpenDalStorageFactory::S3); + +// Inject StorageFactory into catalog +let catalog = GlueCatalogBuilder::default() + .with_storage_factory(storage_factory) + .load("my_catalog", HashMap::from([ + ("warehouse".to_string(), "s3://my-bucket/warehouse".to_string()), + ("s3.region".to_string(), "us-east-1".to_string()), + ])) + .await?; +``` + +### Implementing Custom Storage + +To implement a custom storage backend, implement the `Storage` trait with `#[typetag::serde]`: + +```rust +use std::sync::Arc; +use async_trait::async_trait; +use iceberg::io::{Storage, StorageFactory, StorageConfig, InputFile, OutputFile}; + +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct MyCustomStorage { /* fields */ } + +#[async_trait] +#[typetag::serde] +impl Storage for MyCustomStorage { + // Implement all required methods: exists, metadata, read, reader, + // write, writer, delete, delete_prefix, new_input, new_output +} + +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct MyCustomStorageFactory; + +#[typetag::serde] +impl StorageFactory for MyCustomStorageFactory { + fn build(&self, config: &StorageConfig) -> iceberg::Result> { + Ok(Arc::new(MyCustomStorage { /* ... */ })) + } +} +``` + +### Routing to Multiple Storage Backends + +To use different storage implementations for different schemes (e.g., a native S3 client +for S3 and OpenDAL for other schemes), implement routing at the `Storage` level: + +```rust +use std::sync::Arc; +use async_trait::async_trait; +use iceberg::io::{Storage, StorageFactory, StorageConfig, InputFile, OutputFile}; + +/// A storage that routes to different backends based on path scheme +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct RoutingStorage { + s3_storage: NativeS3Storage, // Custom S3 implementation + opendal_storage: OpenDalStorage, // OpenDAL for other schemes +} + +#[async_trait] +#[typetag::serde] +impl Storage for RoutingStorage { + async fn read(&self, path: &str) -> iceberg::Result { + if path.starts_with("s3://") || path.starts_with("s3a://") { + self.s3_storage.read(path).await + } else { + self.opendal_storage.read(path).await + } + } + + // Route other methods similarly... +} + +/// Factory that creates RoutingStorage +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct RoutingStorageFactory; + +#[typetag::serde] +impl StorageFactory for RoutingStorageFactory { + fn build(&self, config: &StorageConfig) -> iceberg::Result> { + Ok(Arc::new(RoutingStorage { + s3_storage: NativeS3Storage::new(config)?, + opendal_storage: OpenDalStorage::build_from_config(config)?, + })) + } +} +``` + +--- + + +## Implementation Plan + +### Phase 1: Storage Trait +- Define `Storage` trait in `iceberg` crate +- Define `StorageFactory` trait in `iceberg` crate +- Introduce `StorageConfig` for configuration properties +- Update `FileIO` to use lazy storage initialization with factory pattern +- Update `InputFile`/`OutputFile` to use `Arc` +- Implement `MemoryStorage` and `LocalFsStorage` in `iceberg` crate +- Add `with_storage_factory()` to `CatalogBuilder` trait +- Update all catalog implementations to support StorageFactory injection +- Improve naming: Storage handles locations rather than paths + +### Phase 2: Separate Storage Crate +- Create `iceberg-storage-opendal` crate with `OpenDalStorage` and `OpenDalStorageFactory` +- Move S3, GCS, OSS, Azure implementations to `iceberg-storage-opendal` +- Remove storage feature flags from `iceberg` crate + +### Future Work +- Implement ResolvingStorage backed by OpenDal +- Add `object_store`-based storage implementations +- Consider introducing `IoErrorKind` for storage-specific error handling +- Introduce custom key values in StorageConfigs From d76bb5ed124e65e42bdc488e5028cb9d20366aa5 Mon Sep 17 00:00:00 2001 From: Aditya Subrahmanyan Date: Fri, 16 Jan 2026 16:08:16 -0800 Subject: [PATCH 155/891] fix: allow v2 to v3 table upgrades with existing snapshots (#2010) --- crates/iceberg/src/spec/snapshot.rs | 33 +-- crates/iceberg/src/spec/table_metadata.rs | 191 ++++++++++++++++++ .../src/spec/table_metadata_builder.rs | 2 + 3 files changed, 211 insertions(+), 15 deletions(-) diff --git a/crates/iceberg/src/spec/snapshot.rs b/crates/iceberg/src/spec/snapshot.rs index 270279988b..802cd6546e 100644 --- a/crates/iceberg/src/spec/snapshot.rs +++ b/crates/iceberg/src/spec/snapshot.rs @@ -272,7 +272,7 @@ pub(super) mod _serde { #[derive(Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] - /// Defines the structure of a v2 snapshot for serialization/deserialization + /// Defines the structure of a v3 snapshot for serialization/deserialization pub(crate) struct SnapshotV3 { pub snapshot_id: i64, #[serde(skip_serializing_if = "Option::is_none")] @@ -283,8 +283,10 @@ pub(super) mod _serde { pub summary: Summary, #[serde(skip_serializing_if = "Option::is_none")] pub schema_id: Option, - pub first_row_id: u64, - pub added_rows: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub first_row_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub added_rows: Option, #[serde(skip_serializing_if = "Option::is_none")] pub key_id: Option, } @@ -333,10 +335,13 @@ pub(super) mod _serde { summary: s.summary, schema_id: s.schema_id, encryption_key_id: s.key_id, - row_range: Some(SnapshotRowRange { - first_row_id: s.first_row_id, - added_rows: s.added_rows, - }), + row_range: match (s.first_row_id, s.added_rows) { + (Some(first_row_id), Some(added_rows)) => Some(SnapshotRowRange { + first_row_id, + added_rows, + }), + _ => None, + }, } } } @@ -345,12 +350,10 @@ pub(super) mod _serde { type Error = Error; fn try_from(s: Snapshot) -> Result { - let row_range = s.row_range.ok_or_else(|| { - Error::new( - crate::ErrorKind::DataInvalid, - "v3 Snapshots must have first-row-id and rows-added fields set.".to_string(), - ) - })?; + let (first_row_id, added_rows) = match s.row_range { + Some(row_range) => (Some(row_range.first_row_id), Some(row_range.added_rows)), + None => (None, None), + }; Ok(SnapshotV3 { snapshot_id: s.snapshot_id, @@ -360,8 +363,8 @@ pub(super) mod _serde { manifest_list: s.manifest_list, summary: s.summary, schema_id: s.schema_id, - first_row_id: row_range.first_row_id, - added_rows: row_range.added_rows, + first_row_id, + added_rows, key_id: s.encryption_key_id, }) } diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index 585cb3e2bc..d3836b2f42 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -3978,4 +3978,195 @@ mod tests { assert_eq!(err.kind(), ErrorKind::DataInvalid); assert!(err.message().contains("Invalid table properties")); } + + #[test] + fn test_v2_to_v3_upgrade_preserves_existing_snapshots_without_row_lineage() { + // Create a v2 table metadata + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(); + + let v2_metadata = TableMetadataBuilder::new( + schema, + PartitionSpec::unpartition_spec().into_unbound(), + SortOrder::unsorted_order(), + "s3://bucket/test/location".to_string(), + FormatVersion::V2, + HashMap::new(), + ) + .unwrap() + .build() + .unwrap() + .metadata; + + // Add a v2 snapshot + let snapshot = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(v2_metadata.last_updated_ms + 1) + .with_sequence_number(1) + .with_schema_id(0) + .with_manifest_list("s3://bucket/test/metadata/snap-1.avro") + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::from([( + "added-data-files".to_string(), + "1".to_string(), + )]), + }) + .build(); + + let v2_with_snapshot = v2_metadata + .into_builder(Some("s3://bucket/test/metadata/v00001.json".to_string())) + .add_snapshot(snapshot) + .unwrap() + .set_ref("main", SnapshotReference { + snapshot_id: 1, + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: None, + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + }) + .unwrap() + .build() + .unwrap() + .metadata; + + // Verify v2 serialization works fine + let v2_json = serde_json::to_string(&v2_with_snapshot); + assert!(v2_json.is_ok(), "v2 serialization should work"); + + // Upgrade to v3 + let v3_metadata = v2_with_snapshot + .into_builder(Some("s3://bucket/test/metadata/v00002.json".to_string())) + .upgrade_format_version(FormatVersion::V3) + .unwrap() + .build() + .unwrap() + .metadata; + + assert_eq!(v3_metadata.format_version, FormatVersion::V3); + assert_eq!(v3_metadata.next_row_id, INITIAL_ROW_ID); + assert_eq!(v3_metadata.snapshots.len(), 1); + + // Verify the snapshot has no row_range + let snapshot = v3_metadata.snapshots.values().next().unwrap(); + assert!( + snapshot.row_range().is_none(), + "Snapshot should have no row_range after upgrade" + ); + + // Try to serialize v3 metadata - this should now work + let v3_json = serde_json::to_string(&v3_metadata); + assert!( + v3_json.is_ok(), + "v3 serialization should work for upgraded tables" + ); + + // Verify we can deserialize it back + let deserialized: TableMetadata = serde_json::from_str(&v3_json.unwrap()).unwrap(); + assert_eq!(deserialized.format_version, FormatVersion::V3); + assert_eq!(deserialized.snapshots.len(), 1); + + // Verify the deserialized snapshot still has no row_range + let deserialized_snapshot = deserialized.snapshots.values().next().unwrap(); + assert!( + deserialized_snapshot.row_range().is_none(), + "Deserialized snapshot should have no row_range" + ); + } + + #[test] + fn test_v3_snapshot_with_row_lineage_serialization() { + // Create a v3 table metadata + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + ]) + .build() + .unwrap(); + + let v3_metadata = TableMetadataBuilder::new( + schema, + PartitionSpec::unpartition_spec().into_unbound(), + SortOrder::unsorted_order(), + "s3://bucket/test/location".to_string(), + FormatVersion::V3, + HashMap::new(), + ) + .unwrap() + .build() + .unwrap() + .metadata; + + // Add a v3 snapshot with row lineage + let snapshot = Snapshot::builder() + .with_snapshot_id(1) + .with_timestamp_ms(v3_metadata.last_updated_ms + 1) + .with_sequence_number(1) + .with_schema_id(0) + .with_manifest_list("s3://bucket/test/metadata/snap-1.avro") + .with_summary(Summary { + operation: Operation::Append, + additional_properties: HashMap::from([( + "added-data-files".to_string(), + "1".to_string(), + )]), + }) + .with_row_range(100, 50) // first_row_id=100, added_rows=50 + .build(); + + let v3_with_snapshot = v3_metadata + .into_builder(Some("s3://bucket/test/metadata/v00001.json".to_string())) + .add_snapshot(snapshot) + .unwrap() + .set_ref("main", SnapshotReference { + snapshot_id: 1, + retention: SnapshotRetention::Branch { + min_snapshots_to_keep: None, + max_snapshot_age_ms: None, + max_ref_age_ms: None, + }, + }) + .unwrap() + .build() + .unwrap() + .metadata; + + // Verify the snapshot has row_range + let snapshot = v3_with_snapshot.snapshots.values().next().unwrap(); + assert!( + snapshot.row_range().is_some(), + "Snapshot should have row_range" + ); + let (first_row_id, added_rows) = snapshot.row_range().unwrap(); + assert_eq!(first_row_id, 100); + assert_eq!(added_rows, 50); + + // Serialize v3 metadata - this should work + let v3_json = serde_json::to_string(&v3_with_snapshot); + assert!( + v3_json.is_ok(), + "v3 serialization should work for snapshots with row lineage" + ); + + // Verify we can deserialize it back + let deserialized: TableMetadata = serde_json::from_str(&v3_json.unwrap()).unwrap(); + assert_eq!(deserialized.format_version, FormatVersion::V3); + assert_eq!(deserialized.snapshots.len(), 1); + + // Verify the deserialized snapshot has the correct row_range + let deserialized_snapshot = deserialized.snapshots.values().next().unwrap(); + assert!( + deserialized_snapshot.row_range().is_some(), + "Deserialized snapshot should have row_range" + ); + let (deserialized_first_row_id, deserialized_added_rows) = + deserialized_snapshot.row_range().unwrap(); + assert_eq!(deserialized_first_row_id, 100); + assert_eq!(deserialized_added_rows, 50); + } } diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 3db327d48a..45d1ccefcd 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -233,6 +233,8 @@ impl TableMetadataBuilder { } FormatVersion::V3 => { self.metadata.format_version = format_version; + // Set next-row-id to 0 when upgrading to v3 as per Iceberg spec + self.metadata.next_row_id = INITIAL_ROW_ID; self.changes .push(TableUpdate::UpgradeFormatVersion { format_version }); } From 6ed38b28647a4fd6e14c3173903a5fa5bb710cfe Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sat, 17 Jan 2026 15:04:46 -0500 Subject: [PATCH 156/891] site: release static version from downloads page (#2039) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Remove mentions of release version from the "downloads" page (https://rust.iceberg.apache.org/download.html) So we dont have to update it after every release :) Rendered ![Screenshot 2026-01-16 at 8 12 55 AM](https://github.com/user-attachments/assets/f9668722-8465-4f7f-a545-bac3387b8ec2) hyperlinks referenced in screenshot: https://dlcdn.apache.org/iceberg/ https://github.com/apache/iceberg-rust/releases https://archive.apache.org/dist/iceberg/ ## Are these changes tested? --- website/src/download.md | 3 +-- website/src/release.md | 4 ---- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/website/src/download.md b/website/src/download.md index cc632c4c87..5158d7e94e 100644 --- a/website/src/download.md +++ b/website/src/download.md @@ -23,8 +23,7 @@ The official Apache Iceberg-Rust releases are provided as source artifacts. ## Releases -The latest source release is [0.7.0](https://www.apache.org/dyn/closer.lua/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz) ([asc](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz.asc), -[sha512](https://downloads.apache.org/iceberg/apache-iceberg-rust-0.7.0/apache-iceberg-rust-0.7.0-src.tar.gz.sha512)). +Find the latest source release at the [Apache CDN](https://dlcdn.apache.org/iceberg/). For release notes, see the [GitHub releases page](https://github.com/apache/iceberg-rust/releases). For older releases, please check the [archive](https://archive.apache.org/dist/iceberg/). diff --git a/website/src/release.md b/website/src/release.md index aec5b3da27..7549b8ef4d 100644 --- a/website/src/release.md +++ b/website/src/release.md @@ -368,10 +368,6 @@ git push origin "v${iceberg_version}" svn mv https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-rust-${release_version} https://dist.apache.org/repos/dist/release/iceberg/apache-iceberg-rust-${iceberg_version} -m "Release Apache Iceberg Rust ${iceberg_version}" ``` -### Change Iceberg Rust Website download link - -Update the download link in `website/src/download.md` to the new release version. - ### Create a GitHub Release - Click [here](https://github.com/apache/iceberg-rust/releases/new) to create a new release. From ea6004b4f5f3256e50580de25552878fe5f8975a Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sun, 18 Jan 2026 21:17:52 -0500 Subject: [PATCH 157/891] infra: upgrade spark to 3.5.8 (#2045) ## Which issue does this PR close? - Closes #2041. ## What changes are included in this PR? Upgrade spark to use 3.5.8 which is available in https://dlcdn.apache.org/spark/ 3.5.7 is gone, causing our CI to fail ## Are these changes tested? --- crates/integration_tests/testdata/spark/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/integration_tests/testdata/spark/Dockerfile b/crates/integration_tests/testdata/spark/Dockerfile index e31e2273d9..7de700bde2 100644 --- a/crates/integration_tests/testdata/spark/Dockerfile +++ b/crates/integration_tests/testdata/spark/Dockerfile @@ -27,7 +27,7 @@ ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$ RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME} && mkdir -p /home/iceberg/spark-events WORKDIR ${SPARK_HOME} -ENV SPARK_VERSION=3.5.7 +ENV SPARK_VERSION=3.5.8 ENV ICEBERG_SPARK_RUNTIME_VERSION=3.5_2.12 ENV ICEBERG_VERSION=1.10.0 From 6270ce4d03546256553b3f365cb4aa600c9b5fe1 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sun, 18 Jan 2026 21:51:11 -0500 Subject: [PATCH 158/891] infra: use spark base image for docker (#2043) ## Which issue does this PR close? - Closes #2041 ## What changes are included in this PR? We made some upgrades to the Spark Dockerfile in pyiceberg (https://github.com/apache/iceberg-python/pull/2540) (which i think rust's Dockerfile copied over previously). Porting those changes over: - Use `apache/spark` as base image (should be faster than downloading spark from apache cdn) - Upgrade to spark 4.0 - Use Spark connect for provisioning ## Are these changes tested? Yes --- .../testdata/docker-compose.yaml | 26 +++++++ .../testdata/spark/Dockerfile | 74 ++++++++++--------- .../testdata/spark/entrypoint.sh | 33 --------- .../testdata/spark/provision.py | 50 ++++++------- .../testdata/spark/spark-defaults.conf | 9 +++ 5 files changed, 97 insertions(+), 95 deletions(-) delete mode 100755 crates/integration_tests/testdata/spark/entrypoint.sh diff --git a/crates/integration_tests/testdata/docker-compose.yaml b/crates/integration_tests/testdata/docker-compose.yaml index cf0240d1a5..30fc3914f8 100644 --- a/crates/integration_tests/testdata/docker-compose.yaml +++ b/crates/integration_tests/testdata/docker-compose.yaml @@ -80,3 +80,29 @@ services: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 + ports: + - 15002:15002 # Spark Connect + - 4040:4040 # Spark UI + healthcheck: + test: ["CMD", "sh", "-c", "netstat -an | grep 15002 | grep LISTEN"] + interval: 30s + timeout: 10s + retries: 5 + start_period: 90s + + provision: + image: python:3.12-slim + networks: + rest_bridge: + depends_on: + spark-iceberg: + condition: service_healthy + entrypoint: ["/bin/sh", "-c", "pip install -q 'pyspark[connect]==4.0.1' && python3 /opt/spark/provision.py && touch /tmp/provision_complete && tail -f /dev/null"] + volumes: + - ./spark/provision.py:/opt/spark/provision.py:ro + healthcheck: + test: ["CMD-SHELL", "[ -f /tmp/provision_complete ]"] + interval: 2s + timeout: 2s + retries: 90 + start_period: 20s diff --git a/crates/integration_tests/testdata/spark/Dockerfile b/crates/integration_tests/testdata/spark/Dockerfile index 7de700bde2..4b486c9001 100644 --- a/crates/integration_tests/testdata/spark/Dockerfile +++ b/crates/integration_tests/testdata/spark/Dockerfile @@ -13,47 +13,55 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM python:3.9-bullseye +ARG BASE_IMAGE_SPARK_VERSION=4.0.1 -RUN apt-get -qq update && \ - apt-get -qq install -y --no-install-recommends sudo curl openjdk-11-jdk && \ - apt-get -qq clean && \ - rm -rf /var/lib/apt/lists/* +FROM apache/spark:${BASE_IMAGE_SPARK_VERSION} -ENV SPARK_HOME=${SPARK_HOME:-"/opt/spark"} -ENV HADOOP_HOME=${HADOOP_HOME:-"/opt/hadoop"} -ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH +# Dependency versions - keep these compatible +ARG ICEBERG_VERSION=1.10.1 +ARG ICEBERG_SPARK_RUNTIME_VERSION=4.0_2.13 +ARG HADOOP_VERSION=3.4.1 +ARG SCALA_VERSION=2.13 +ARG AWS_SDK_VERSION=2.24.6 +ARG MAVEN_MIRROR=https://repo.maven.apache.org/maven2 -RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME} && mkdir -p /home/iceberg/spark-events +USER root WORKDIR ${SPARK_HOME} -ENV SPARK_VERSION=3.5.8 -ENV ICEBERG_SPARK_RUNTIME_VERSION=3.5_2.12 -ENV ICEBERG_VERSION=1.10.0 - -RUN curl --retry 5 -s -C - https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz -o spark-${SPARK_VERSION}-bin-hadoop3.tgz \ - && tar xzf spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory /opt/spark --strip-components 1 \ - && rm -rf spark-${SPARK_VERSION}-bin-hadoop3.tgz - -# Download iceberg spark runtime -RUN curl --retry 5 -s https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}/${ICEBERG_VERSION}/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar -Lo iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar \ - && mv iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar /opt/spark/jars - -# Download AWS bundle -RUN curl --retry 5 -s https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/${ICEBERG_VERSION}/iceberg-aws-bundle-${ICEBERG_VERSION}.jar -Lo /opt/spark/jars/iceberg-aws-bundle-${ICEBERG_VERSION}.jar +# Install curl for JAR downloads +RUN apt-get update && \ + apt-get install -y --no-install-recommends curl && \ + rm -rf /var/lib/apt/lists/* -COPY spark-defaults.conf /opt/spark/conf -ENV PATH="/opt/spark/sbin:/opt/spark/bin:${PATH}" +# Copy configuration (early for better caching) +COPY --chown=spark:spark spark-defaults.conf ${SPARK_HOME}/conf/ -RUN chmod u+x /opt/spark/sbin/* && \ - chmod u+x /opt/spark/bin/* +# Create event log directory +RUN mkdir -p /home/iceberg/spark-events && \ + chown -R spark:spark /home/iceberg -WORKDIR '/home/' +# Required JAR dependencies +ENV JARS_TO_DOWNLOAD="\ + org/apache/iceberg/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}/${ICEBERG_VERSION}/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar \ + org/apache/iceberg/iceberg-aws-bundle/${ICEBERG_VERSION}/iceberg-aws-bundle-${ICEBERG_VERSION}.jar \ + org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar \ + software/amazon/awssdk/bundle/${AWS_SDK_VERSION}/bundle-${AWS_SDK_VERSION}.jar" -COPY entrypoint.sh . -COPY provision.py . +# Download JARs with retry logic +RUN set -e && \ + cd "${SPARK_HOME}/jars" && \ + for jar_path in ${JARS_TO_DOWNLOAD}; do \ + jar_name=$(basename "${jar_path}") && \ + echo "Downloading ${jar_name}..." && \ + curl -fsSL --retry 3 --retry-delay 5 \ + -o "${jar_name}" \ + "${MAVEN_MIRROR}/${jar_path}" && \ + echo "✓ Downloaded ${jar_name}"; \ + done && \ + chown -R spark:spark "${SPARK_HOME}/jars" -HEALTHCHECK --retries=120 --interval=1s \ - CMD ls /tmp/ready || exit 1 +USER spark +WORKDIR ${SPARK_HOME} -ENTRYPOINT ["./entrypoint.sh"] +# Start Spark Connect server +CMD ["sh", "-c", "SPARK_NO_DAEMONIZE=true ${SPARK_HOME}/sbin/start-connect-server.sh"] diff --git a/crates/integration_tests/testdata/spark/entrypoint.sh b/crates/integration_tests/testdata/spark/entrypoint.sh deleted file mode 100755 index 51ec329f03..0000000000 --- a/crates/integration_tests/testdata/spark/entrypoint.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -set -e - -start-master.sh -p 7077 -start-worker.sh spark://spark-iceberg:7077 - -echo "Starting provision" -python3 ./provision.py - -echo "Finished provisioning" -touch /tmp/ready - -echo "Print logs" -tail -f $SPARK_HOME/logs/* diff --git a/crates/integration_tests/testdata/spark/provision.py b/crates/integration_tests/testdata/spark/provision.py index 899a49f6ac..c53a1dd842 100755 --- a/crates/integration_tests/testdata/spark/provision.py +++ b/crates/integration_tests/testdata/spark/provision.py @@ -18,17 +18,8 @@ from pyspark.sql import SparkSession from pyspark.sql.functions import current_date, date_add, expr -# The configuration is important, otherwise we get many small -# parquet files with a single row. When a positional delete -# hits the Parquet file with one row, the parquet file gets -# dropped instead of having a merge-on-read delete file. -spark = ( - SparkSession - .builder - .config("spark.sql.shuffle.partitions", "1") - .config("spark.default.parallelism", "1") - .getOrCreate() -) +# Create SparkSession against the remote Spark Connect server +spark = SparkSession.builder.remote("sc://spark-iceberg:15002").getOrCreate() spark.sql(f"""CREATE NAMESPACE IF NOT EXISTS rest.default""") @@ -87,24 +78,25 @@ """ ) -spark.sql( - f""" -INSERT INTO rest.default.test_positional_merge_on_read_double_deletes -VALUES - (CAST('2023-03-01' AS date), 1, 'a'), - (CAST('2023-03-02' AS date), 2, 'b'), - (CAST('2023-03-03' AS date), 3, 'c'), - (CAST('2023-03-04' AS date), 4, 'd'), - (CAST('2023-03-05' AS date), 5, 'e'), - (CAST('2023-03-06' AS date), 6, 'f'), - (CAST('2023-03-07' AS date), 7, 'g'), - (CAST('2023-03-08' AS date), 8, 'h'), - (CAST('2023-03-09' AS date), 9, 'i'), - (CAST('2023-03-10' AS date), 10, 'j'), - (CAST('2023-03-11' AS date), 11, 'k'), - (CAST('2023-03-12' AS date), 12, 'l'); -""" -) +# Use coalesce(1) to write all data into a single parquet file. +# This ensures that deleting individual rows creates positional delete files +# instead of dropping entire files (which happens when a file has only one row). +spark.sql(""" + SELECT * FROM VALUES + (CAST('2023-03-01' AS date), 1, 'a'), + (CAST('2023-03-02' AS date), 2, 'b'), + (CAST('2023-03-03' AS date), 3, 'c'), + (CAST('2023-03-04' AS date), 4, 'd'), + (CAST('2023-03-05' AS date), 5, 'e'), + (CAST('2023-03-06' AS date), 6, 'f'), + (CAST('2023-03-07' AS date), 7, 'g'), + (CAST('2023-03-08' AS date), 8, 'h'), + (CAST('2023-03-09' AS date), 9, 'i'), + (CAST('2023-03-10' AS date), 10, 'j'), + (CAST('2023-03-11' AS date), 11, 'k'), + (CAST('2023-03-12' AS date), 12, 'l') + AS t(dt, number, letter) +""").coalesce(1).writeTo("rest.default.test_positional_merge_on_read_double_deletes").append() # Creates two positional deletes that should be merged spark.sql(f"DELETE FROM rest.default.test_positional_merge_on_read_double_deletes WHERE number = 9") diff --git a/crates/integration_tests/testdata/spark/spark-defaults.conf b/crates/integration_tests/testdata/spark/spark-defaults.conf index 9db0c7da9e..b576f624b9 100644 --- a/crates/integration_tests/testdata/spark/spark-defaults.conf +++ b/crates/integration_tests/testdata/spark/spark-defaults.conf @@ -16,14 +16,23 @@ # spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions + +# Configure Iceberg REST catalog spark.sql.catalog.rest org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.rest.type rest spark.sql.catalog.rest.uri http://rest:8181 spark.sql.catalog.rest.io-impl org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.rest.warehouse s3://warehouse/rest/ spark.sql.catalog.rest.s3.endpoint http://minio:9000 +spark.sql.catalog.rest.cache-enabled false + spark.sql.defaultCatalog rest + +# Configure Spark UI and event logging +spark.ui.enabled true spark.eventLog.enabled true spark.eventLog.dir /home/iceberg/spark-events + +spark.sql.ansi.enabled false spark.history.fs.logDirectory /home/iceberg/spark-events spark.sql.catalogImplementation in-memory From 0e019221e9a220aed8745870f80c7df3d74ea4b0 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sun, 18 Jan 2026 21:51:24 -0500 Subject: [PATCH 159/891] infra: auto update docker image (#2044) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Similar to https://github.com/apache/iceberg-python/pull/2885 Add the `--build` flag to `docker compose up`. This will trigger docker to rebuild if the docker-compose file or the Dockerfile is changed. Without `--build`, changes to `Dockerfile` will not trigger a new image build and we might accidentally still be using the old image thinking its updated ## Are these changes tested? --------- Co-authored-by: Renjie Liu --- crates/test_utils/src/docker.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/test_utils/src/docker.rs b/crates/test_utils/src/docker.rs index 1a6e883fde..078966ddb8 100644 --- a/crates/test_utils/src/docker.rs +++ b/crates/test_utils/src/docker.rs @@ -79,6 +79,7 @@ impl DockerCompose { self.project_name.as_str(), "up", "-d", + "--build", "--wait", "--timeout", "1200000", From fd87aab50e694861f800a020af60ceb83b5094af Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 19 Jan 2026 11:54:33 +0800 Subject: [PATCH 160/891] chore(deps): Bump rust_decimal from 1.39.0 to 1.40.0 (#2047) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.39.0 to 1.40.0.
Release notes

Sourced from rust_decimal's releases.

1.40.0

What's Changed

New Contributors

Full Changelog: https://github.com/paupino/rust-decimal/compare/1.39.0...1.40.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rust_decimal&package-manager=cargo&previous-version=1.39.0&new-version=1.40.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 3de43e685b..fdea0e2d97 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5654,9 +5654,9 @@ dependencies = [ [[package]] name = "rust_decimal" -version = "1.39.0" +version = "1.40.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35affe401787a9bd846712274d97654355d21b2a2c092a3139aabe31e9022282" +checksum = "61f703d19852dbf87cbc513643fa81428361eb6940f1ac14fd58155d295a3eb0" dependencies = [ "arrayvec", "borsh", From 8706d0837ba563a1e6cbb948e98275df0d3fbf44 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 19 Jan 2026 13:52:00 +0800 Subject: [PATCH 161/891] chore(deps): Bump the arrow-parquet group with 2 updates (#2046) Bumps the arrow-parquet group with 2 updates: [arrow-arith](https://github.com/apache/arrow-rs) and [arrow-cast](https://github.com/apache/arrow-rs). Updates `arrow-arith` from 57.1.0 to 57.2.0
Release notes

Sourced from arrow-arith's releases.

arrow 57.2.0

Changelog

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946

... (truncated)

Changelog

Sourced from arrow-arith's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits
  • 9e822e0 Update version to 57.2.0, add CHANGELOG (#9103)
  • 28f66f9 Add Union encoding documentation (#9102)
  • a8346be Minor: make it clear cache array reader is not cloning arrays (#9057)
  • 721f373 Seal Array trait (#9092)
  • 2b179b8 feat(parquet): relax type compatility check in parquet ArrowWriter (#9099)
  • 10a976f chore: increase row count and batch size for more deterministic tests (#9088)
  • 2507946 Add FlightInfo::with_endpoints method (#9075)
  • 068a7e4 Add DataType::is_decimal (#9100)
  • b1dfb69 Fix row slice bug in Union column decoding with many columns (#9000)
  • a9d6e92 [Variant] Move ArrayVariantToArrowRowBuilder to variant_to_arrow (#9094)
  • Additional commits viewable in compare view

Updates `arrow-cast` from 57.1.0 to 57.2.0
Release notes

Sourced from arrow-cast's releases.

arrow 57.2.0

Changelog

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946

... (truncated)

Changelog

Sourced from arrow-cast's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits
  • 9e822e0 Update version to 57.2.0, add CHANGELOG (#9103)
  • 28f66f9 Add Union encoding documentation (#9102)
  • a8346be Minor: make it clear cache array reader is not cloning arrays (#9057)
  • 721f373 Seal Array trait (#9092)
  • 2b179b8 feat(parquet): relax type compatility check in parquet ArrowWriter (#9099)
  • 10a976f chore: increase row count and batch size for more deterministic tests (#9088)
  • 2507946 Add FlightInfo::with_endpoints method (#9075)
  • 068a7e4 Add DataType::is_decimal (#9100)
  • b1dfb69 Fix row slice bug in Union column decoding with many columns (#9000)
  • a9d6e92 [Variant] Move ArrayVariantToArrowRowBuilder to variant_to_arrow (#9094)
  • Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index fdea0e2d97..e8805d5102 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -244,9 +244,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f377dcd19e440174596d83deb49cd724886d91060c07fec4f67014ef9d54049" +checksum = "288015089e7931843c80ed4032c5274f02b37bcb720c4a42096d50b390e70372" dependencies = [ "arrow-array", "arrow-buffer", @@ -289,9 +289,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "57.1.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3d131abb183f80c450d4591dc784f8d7750c50c6e2bc3fcaad148afc8361271" +checksum = "9c8e372ed52bd4ee88cc1e6c3859aa7ecea204158ac640b10e187936e7e87074" dependencies = [ "arrow-array", "arrow-buffer", From 3c9df9edbc90e8fe5f877c52db8ecf953b46b017 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 19 Jan 2026 00:53:05 -0500 Subject: [PATCH 162/891] infra: dynamically set msrv in github workflows (#2040) ## Which issue does this PR close? - Closes #2038 ## What changes are included in this PR? Get MSRV by parsing `rust-version` from `Cargo.toml`. This is to avoid forgetting to update MSRV version in the github workflow. ## Are these changes tested? --- .github/actions/get-msrv/action.yml | 35 ++++++++++++++++++++ .github/workflows/ci.yml | 8 ++--- .github/workflows/publish.yml | 9 ++--- .github/workflows/release_python.yml | 9 ++--- .github/workflows/release_python_nightly.yml | 9 ++--- 5 files changed, 54 insertions(+), 16 deletions(-) create mode 100644 .github/actions/get-msrv/action.yml diff --git a/.github/actions/get-msrv/action.yml b/.github/actions/get-msrv/action.yml new file mode 100644 index 0000000000..f0b03e2777 --- /dev/null +++ b/.github/actions/get-msrv/action.yml @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Get MSRV +description: Get the Minimum Supported Rust Version from Cargo.toml + +outputs: + msrv: + description: The MSRV extracted from Cargo.toml + value: ${{ steps.get-msrv.outputs.msrv }} + +runs: + using: composite + steps: + - name: Get MSRV + id: get-msrv + shell: bash + run: | + msrv=$(grep '^rust-version' Cargo.toml | sed 's/.*"\(.*\)".*/\1/') + echo "msrv=$msrv" >> $GITHUB_OUTPUT + echo "MSRV is $msrv" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f393309bcb..af9b812c77 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,9 +40,6 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }} cancel-in-progress: true -env: - rust_msrv: "1.87" - jobs: check: runs-on: ${{ matrix.os }} @@ -184,10 +181,13 @@ jobs: uses: arduino/setup-protoc@v3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Get MSRV + id: get-msrv + uses: ./.github/actions/get-msrv - name: Setup MSRV Rust toolchain uses: ./.github/actions/setup-builder with: - rust-version: ${{ env.rust_msrv }} + rust-version: ${{ steps.get-msrv.outputs.msrv }} - name: Setup Nightly Rust toolchain uses: ./.github/actions/setup-builder - name: Check MSRV diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 66c17a668d..6b4b3da841 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -26,9 +26,6 @@ on: - "v[0-9]+.[0-9]+.[0-9]+-rc.[0-9]+" workflow_dispatch: -env: - rust_msrv: "1.87" - jobs: publish: runs-on: ubuntu-latest @@ -48,10 +45,14 @@ jobs: steps: - uses: actions/checkout@v6 + - name: Get MSRV + id: get-msrv + uses: ./.github/actions/get-msrv + - name: Setup Rust toolchain uses: ./.github/actions/setup-builder with: - rust-version: ${{ env.rust_msrv }} + rust-version: ${{ steps.get-msrv.outputs.msrv }} - name: Publish ${{ matrix.package }} working-directory: ${{ matrix.package }} diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 85663fc75f..dba532a31c 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -24,9 +24,6 @@ on: - completed workflow_dispatch: -env: - rust_msrv: "1.87" - concurrency: group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch }}-${{ github.event_name }} cancel-in-progress: true @@ -147,10 +144,14 @@ jobs: - uses: actions/setup-python@v6 with: python-version: 3.12 + - name: Get MSRV + id: get-msrv + uses: ./.github/actions/get-msrv + - name: Setup Rust toolchain uses: ./.github/actions/setup-builder with: - rust-version: ${{ env.rust_msrv }} + rust-version: ${{ steps.get-msrv.outputs.msrv }} - uses: PyO3/maturin-action@v1 with: target: ${{ matrix.target }} diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 833b8ee6a6..8bdd9d1003 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -22,9 +22,6 @@ on: - cron: "0 0 * * *" # Runs at midnight UTC every day workflow_dispatch: # Allows manual triggering -env: - rust_msrv: "1.87" - permissions: contents: read @@ -88,10 +85,14 @@ jobs: with: python-version: 3.12 + - name: Get MSRV + id: get-msrv + uses: ./.github/actions/get-msrv + - name: Setup Rust toolchain uses: ./.github/actions/setup-builder with: - rust-version: ${{ env.rust_msrv }} + rust-version: ${{ steps.get-msrv.outputs.msrv }} - uses: PyO3/maturin-action@v1 with: From 2c690c226c1faa44d9fb4842299111de27c97a3b Mon Sep 17 00:00:00 2001 From: Aditya Subrahmanyan Date: Sun, 18 Jan 2026 23:46:40 -0800 Subject: [PATCH 163/891] fix: reuse partition field IDs for equivalent fields in AddSpec (#2011) ## Which issue does this PR close? - Closes #2007. ## What changes are included in this PR? Implements Iceberg spec requirement to reuse partition field IDs when adding specs with equivalent fields (same source_id + transform). - Add Hash trait to Transform enum for HashMap keys - Add field ID reuse logic in TableMetadataBuilder.add_partition_spec() ## Are these changes tested? Added a test to cover partition field id reuse when new specs are added --------- Co-authored-by: Renjie Liu --- .../src/spec/table_metadata_builder.rs | 100 ++++++++++++++++++ crates/iceberg/src/spec/transform.rs | 2 +- 2 files changed, 101 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/spec/table_metadata_builder.rs b/crates/iceberg/src/spec/table_metadata_builder.rs index 45d1ccefcd..3508c1e52f 100644 --- a/crates/iceberg/src/spec/table_metadata_builder.rs +++ b/crates/iceberg/src/spec/table_metadata_builder.rs @@ -835,6 +835,9 @@ impl TableMetadataBuilder { // Check if partition field names conflict with schema field names across all schemas self.validate_partition_field_names(&unbound_spec)?; + // Reuse field IDs for equivalent fields from existing partition specs + let unbound_spec = self.reuse_partition_field_ids(unbound_spec)?; + let spec = PartitionSpecBuilder::new_from_unbound(unbound_spec.clone(), schema)? .with_last_assigned_field_id(self.metadata.last_partition_id) .build()?; @@ -877,6 +880,44 @@ impl TableMetadataBuilder { Ok(self) } + /// Reuse partition field IDs for equivalent fields from existing partition specs. + /// + /// According to the Iceberg spec, partition field IDs must be reused if an existing + /// partition spec contains an equivalent field (same source_id and transform). + fn reuse_partition_field_ids( + &self, + unbound_spec: UnboundPartitionSpec, + ) -> Result { + // Build a map of (source_id, transform) -> field_id from existing specs + let equivalent_field_ids: HashMap<_, _> = self + .metadata + .partition_specs + .values() + .flat_map(|spec| spec.fields()) + .map(|field| ((field.source_id, &field.transform), field.field_id)) + .collect(); + + // Create new fields with reused field IDs where possible + let fields = unbound_spec + .fields + .into_iter() + .map(|mut field| { + if field.field_id.is_none() + && let Some(&existing_field_id) = + equivalent_field_ids.get(&(field.source_id, &field.transform)) + { + field.field_id = Some(existing_field_id); + } + field + }) + .collect(); + + Ok(UnboundPartitionSpec { + spec_id: unbound_spec.spec_id, + fields, + }) + } + /// Set the default partition spec. /// /// # Errors @@ -3507,4 +3548,63 @@ mod tests { let keys = build_result.metadata.encryption_keys_iter(); assert_eq!(keys.len(), 0); } + + #[test] + fn test_partition_field_id_reuse_across_specs() { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(), + NestedField::required(2, "data", Type::Primitive(PrimitiveType::String)).into(), + NestedField::required(3, "timestamp", Type::Primitive(PrimitiveType::Timestamp)) + .into(), + ]) + .build() + .unwrap(); + + // Create initial table with spec 0: identity(id) -> field_id = 1000 + let initial_spec = UnboundPartitionSpec::builder() + .add_partition_field(1, "id", Transform::Identity) + .unwrap() + .build(); + + let mut metadata = TableMetadataBuilder::new( + schema, + initial_spec, + SortOrder::unsorted_order(), + "s3://bucket/table".to_string(), + FormatVersion::V2, + HashMap::new(), + ) + .unwrap() + .build() + .unwrap() + .metadata; + + // Add spec 1: bucket(data) -> field_id = 1001 + let spec1 = UnboundPartitionSpec::builder() + .add_partition_field(2, "data_bucket", Transform::Bucket(10)) + .unwrap() + .build(); + let builder = metadata.into_builder(Some("s3://bucket/table/metadata/v1.json".to_string())); + let result = builder.add_partition_spec(spec1).unwrap().build().unwrap(); + metadata = result.metadata; + + // Add spec 2: identity(id) + bucket(data) + year(timestamp) + // Should reuse field_id 1000 for identity(id) and 1001 for bucket(data) + let spec2 = UnboundPartitionSpec::builder() + .add_partition_field(1, "id", Transform::Identity) // Should reuse 1000 + .unwrap() + .add_partition_field(2, "data_bucket", Transform::Bucket(10)) // Should reuse 1001 + .unwrap() + .add_partition_field(3, "year", Transform::Year) // Should get new 1002 + .unwrap() + .build(); + let builder = metadata.into_builder(Some("s3://bucket/table/metadata/v2.json".to_string())); + let result = builder.add_partition_spec(spec2).unwrap().build().unwrap(); + + // Verify field ID reuse: spec 2 should reuse IDs from specs 0 and 1, assign new ID for new field + let spec2 = result.metadata.partition_spec_by_id(2).unwrap(); + let field_ids: Vec = spec2.fields().iter().map(|f| f.field_id).collect(); + assert_eq!(field_ids, vec![1000, 1001, 1002]); // Reused 1000, 1001; new 1002 + } } diff --git a/crates/iceberg/src/spec/transform.rs b/crates/iceberg/src/spec/transform.rs index 354dc1889c..026b12613b 100644 --- a/crates/iceberg/src/spec/transform.rs +++ b/crates/iceberg/src/spec/transform.rs @@ -47,7 +47,7 @@ use crate::transform::{BoxedTransformFunction, create_transform_function}; /// predicates and partition predicates. /// /// All transforms must return `null` for a `null` input value. -#[derive(Debug, PartialEq, Eq, Clone, Copy)] +#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)] pub enum Transform { /// Source value, unmodified /// From fbcf58e86be9d058cbff0b4c8211b7e0d75531ff Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 19 Jan 2026 19:02:43 -0500 Subject: [PATCH 164/891] infra: fix manual trigger for the "release python" github workflow (#2049) ## Which issue does this PR close? - Closes #2038 This unblocks us to publish python binding to pypi ## What changes are included in this PR? Need to manually trigger `.github/workflows/release_python.yml` to publish python binding to pypi. This PR fixes the manual flow by allowing to specify `RELEASE_TAG` as input. Added an extra validation for the manual workflow to check the version against the library version in `bindings/python/Cargo.toml` ## Are these changes tested? --- .github/workflows/release_python.yml | 39 ++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index dba532a31c..e587b50adc 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -23,10 +23,11 @@ on: types: - completed workflow_dispatch: - -concurrency: - group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch }}-${{ github.event_name }} - cancel-in-progress: true + inputs: + release_tag: + description: 'Release tag (e.g., v0.4.0 or v0.4.0-rc.1)' + required: true + type: string permissions: contents: read @@ -46,12 +47,20 @@ jobs: cargo-version: ${{ steps.validate.outputs.cargo-version }} is-rc: ${{ steps.validate.outputs.is-rc }} steps: + - uses: actions/checkout@v6 + if: ${{ github.event_name == 'workflow_dispatch' }} + - name: Validate release tag format id: validate + # Use input for workflow_dispatch, otherwise use `workflow_run.head_branch` # Note, `workflow_run.head_branch` does not contain `refs/tags/` prefix, just the tag name, i.e. `v0.4.0` or `v0.4.0-rc.1` # Valid formats: v.. OR v..-rc. run: | - RELEASE_TAG="${{ github.event.workflow_run.head_branch }}" + if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then + RELEASE_TAG="${{ github.event.inputs.release_tag }}" + else + RELEASE_TAG="${{ github.event.workflow_run.head_branch }}" + fi echo "Validating release tag: $RELEASE_TAG" if [[ ! "$RELEASE_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$ ]]; then echo "❌ Invalid release tag format: $RELEASE_TAG" @@ -65,6 +74,26 @@ jobs: CARGO_VERSION="${RELEASE_TAG#v}" echo "Cargo version (without v prefix): $CARGO_VERSION" + # For manual triggers, validate that the tag matches the version in Cargo.toml + if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then + # Extract base version (without -rc.X suffix) for comparison with Cargo.toml + BASE_VERSION="${CARGO_VERSION%-rc.*}" + echo "Base version (for Cargo.toml comparison): $BASE_VERSION" + + # Read version from Cargo.toml and validate it matches + CARGO_TOML_VERSION=$(grep '^version = ' bindings/python/Cargo.toml | head -1 | sed 's/version = "\(.*\)"/\1/') + echo "Version in bindings/python/Cargo.toml: $CARGO_TOML_VERSION" + + if [ "$BASE_VERSION" != "$CARGO_TOML_VERSION" ]; then + echo "❌ Version mismatch!" + echo " Release tag base version: $BASE_VERSION" + echo " bindings/python/Cargo.toml version: $CARGO_TOML_VERSION" + echo "Please ensure the release tag matches the version in Cargo.toml" + exit 1 + fi + echo "✅ Version matches bindings/python/Cargo.toml" + fi + # Check if this is a release candidate if [[ "$RELEASE_TAG" =~ -rc\.[0-9]+$ ]]; then IS_RC="true" From ee21563c2032948f636eae84870f317a0b299a05 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Mon, 19 Jan 2026 20:27:32 -0500 Subject: [PATCH 165/891] perf(reader): Fast path ArrowReader::read when concurrency is 1 to avoid waker churn and add determinism to FileScanTask processing (#2020) ## Which issue does this PR close? - N/A. ## What changes are included in this PR? - Due to the way Comet maps DataFusion `SessionContext`, the tokio runtime, and Spark Tasks, we see frequent waker churn when concurrency is set to 1 in the `ArrowReader`. This adds a fast path that does not use `try_flatten_unordered` and its internal `replace_waker` calls. - This also prevents tasks from being reordered at runtime. Several Iceberg Java tests expect specific query results without an `ORDER BY`, so this enables those tests to keep working when concurrency is set to 1. See https://github.com/apache/datafusion-comet/pull/3051 and flamegraph ## Are these changes tested? New test for determinism, also running the entire Iceberg Java Spark suite via Comet in https://github.com/apache/datafusion-comet/pull/3051. --------- Co-authored-by: Renjie Liu --- crates/iceberg/src/arrow/reader.rs | 225 ++++++++++++++++++++++++++--- 1 file changed, 206 insertions(+), 19 deletions(-) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index f7f90663a5..aa45a12973 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -148,26 +148,53 @@ impl ArrowReader { let row_group_filtering_enabled = self.row_group_filtering_enabled; let row_selection_enabled = self.row_selection_enabled; - let stream = tasks - .map_ok(move |task| { - let file_io = file_io.clone(); - - Self::process_file_scan_task( - task, - batch_size, - file_io, - self.delete_file_loader.clone(), - row_group_filtering_enabled, - row_selection_enabled, - ) - }) - .map_err(|err| { - Error::new(ErrorKind::Unexpected, "file scan task generate failed").with_source(err) - }) - .try_buffer_unordered(concurrency_limit_data_files) - .try_flatten_unordered(concurrency_limit_data_files); + // Fast-path for single concurrency to avoid overhead of try_flatten_unordered + let stream: ArrowRecordBatchStream = if concurrency_limit_data_files == 1 { + Box::pin( + tasks + .and_then(move |task| { + let file_io = file_io.clone(); + + Self::process_file_scan_task( + task, + batch_size, + file_io, + self.delete_file_loader.clone(), + row_group_filtering_enabled, + row_selection_enabled, + ) + }) + .map_err(|err| { + Error::new(ErrorKind::Unexpected, "file scan task generate failed") + .with_source(err) + }) + .try_flatten(), + ) + } else { + Box::pin( + tasks + .map_ok(move |task| { + let file_io = file_io.clone(); + + Self::process_file_scan_task( + task, + batch_size, + file_io, + self.delete_file_loader.clone(), + row_group_filtering_enabled, + row_selection_enabled, + ) + }) + .map_err(|err| { + Error::new(ErrorKind::Unexpected, "file scan task generate failed") + .with_source(err) + }) + .try_buffer_unordered(concurrency_limit_data_files) + .try_flatten_unordered(concurrency_limit_data_files), + ) + }; - Ok(Box::pin(stream) as ArrowRecordBatchStream) + Ok(stream) } #[allow(clippy::too_many_arguments)] @@ -3894,6 +3921,166 @@ message schema { assert!(result.is_empty() || result.iter().all(|batch| batch.num_rows() == 0)); } + /// Test that concurrency=1 reads all files correctly and in deterministic order. + /// This verifies the fast-path optimization for single concurrency. + #[tokio::test] + async fn test_read_with_concurrency_one() { + use arrow_array::Int32Array; + + let schema = Arc::new( + Schema::builder() + .with_schema_id(1) + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::required(2, "file_num", Type::Primitive(PrimitiveType::Int)) + .into(), + ]) + .build() + .unwrap(), + ); + + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("id", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "1".to_string(), + )])), + Field::new("file_num", DataType::Int32, false).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "2".to_string(), + )])), + ])); + + let tmp_dir = TempDir::new().unwrap(); + let table_location = tmp_dir.path().to_str().unwrap().to_string(); + let file_io = FileIO::from_path(&table_location).unwrap().build().unwrap(); + + // Create 3 parquet files with different data + let props = WriterProperties::builder() + .set_compression(Compression::SNAPPY) + .build(); + + for file_num in 0..3 { + let id_data = Arc::new(Int32Array::from_iter_values( + file_num * 10..(file_num + 1) * 10, + )) as ArrayRef; + let file_num_data = Arc::new(Int32Array::from(vec![file_num; 10])) as ArrayRef; + + let to_write = + RecordBatch::try_new(arrow_schema.clone(), vec![id_data, file_num_data]).unwrap(); + + let file = File::create(format!("{table_location}/file_{file_num}.parquet")).unwrap(); + let mut writer = + ArrowWriter::try_new(file, to_write.schema(), Some(props.clone())).unwrap(); + writer.write(&to_write).expect("Writing batch"); + writer.close().unwrap(); + } + + // Read with concurrency=1 (fast-path) + let reader = ArrowReaderBuilder::new(file_io) + .with_data_file_concurrency_limit(1) + .build(); + + // Create tasks in a specific order: file_0, file_1, file_2 + let tasks = vec![ + Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{table_location}/file_0.parquet"), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, + case_sensitive: false, + }), + Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{table_location}/file_1.parquet"), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, + case_sensitive: false, + }), + Ok(FileScanTask { + start: 0, + length: 0, + record_count: None, + data_file_path: format!("{table_location}/file_2.parquet"), + data_file_format: DataFileFormat::Parquet, + schema: schema.clone(), + project_field_ids: vec![1, 2], + predicate: None, + deletes: vec![], + partition: None, + partition_spec: None, + name_mapping: None, + case_sensitive: false, + }), + ]; + + let tasks_stream = Box::pin(futures::stream::iter(tasks)) as FileScanTaskStream; + + let result = reader + .read(tasks_stream) + .unwrap() + .try_collect::>() + .await + .unwrap(); + + // Verify we got all 30 rows (10 from each file) + let total_rows: usize = result.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 30, "Should have 30 total rows"); + + // Collect all ids and file_nums to verify data + let mut all_ids = Vec::new(); + let mut all_file_nums = Vec::new(); + + for batch in &result { + let id_col = batch + .column(0) + .as_primitive::(); + let file_num_col = batch + .column(1) + .as_primitive::(); + + for i in 0..batch.num_rows() { + all_ids.push(id_col.value(i)); + all_file_nums.push(file_num_col.value(i)); + } + } + + assert_eq!(all_ids.len(), 30); + assert_eq!(all_file_nums.len(), 30); + + // With concurrency=1 and sequential processing, files should be processed in order + // file_0: ids 0-9, file_num=0 + // file_1: ids 10-19, file_num=1 + // file_2: ids 20-29, file_num=2 + for i in 0..10 { + assert_eq!(all_file_nums[i], 0, "First 10 rows should be from file_0"); + assert_eq!(all_ids[i], i as i32, "IDs should be 0-9"); + } + for i in 10..20 { + assert_eq!(all_file_nums[i], 1, "Next 10 rows should be from file_1"); + assert_eq!(all_ids[i], i as i32, "IDs should be 10-19"); + } + for i in 20..30 { + assert_eq!(all_file_nums[i], 2, "Last 10 rows should be from file_2"); + assert_eq!(all_ids[i], i as i32, "IDs should be 20-29"); + } + } + /// Test bucket partitioning reads source column from data file (not partition metadata). /// /// This is an integration test verifying the complete ArrowReader pipeline with bucket partitioning. From 20ce7a50c67344564be48d5f1f8fafca2c288854 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Wed, 21 Jan 2026 09:01:26 +0800 Subject: [PATCH 166/891] feat(datafusion): Add Binary scalar value conversion for predicate pushdown (#2048) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Add support for converting Binary and LargeBinary DataFusion ScalarValue types to Iceberg Datum, enabling binary predicates to be pushed down to the Iceberg storage layer. This conversion allows SQL queries with binary hex literals (X'...') to push predicates down to Iceberg, improving query performance by filtering data at the storage level rather than in DataFusion. The integration test verifies that binary predicates are successfully pushed down end-to-end: - Without conversion: predicate stays in FilterExec with predicate:[] - With conversion: predicate pushed to IcebergTableScan Other scalar types (Boolean, Timestamp, Decimal) were investigated but excluded because they are not reachable through practical usage: - Boolean: DataFusion aggressively optimizes comparisons (e.g., x=true becomes just x) before reaching the converter - Timestamp/Decimal: SQL literals are converted to strings/other types before reaching the converter ## Are these changes tested? --------- Co-authored-by: Claude Sonnet 4.5 --- .../src/physical_plan/expr_to_predicate.rs | 29 ++++++++ crates/sqllogictest/src/engine/datafusion.rs | 28 ++++++++ .../testdata/schedules/df_test.toml | 4 ++ .../df_test/binary_predicate_pushdown.slt | 69 +++++++++++++++++++ .../testdata/slts/df_test/show_tables.slt | 3 + 5 files changed, 133 insertions(+) create mode 100644 crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt diff --git a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs index 2974dd2642..fc95be5db5 100644 --- a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs +++ b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs @@ -212,6 +212,8 @@ fn scalar_value_to_datum(value: &ScalarValue) -> Option { ScalarValue::Float64(Some(v)) => Some(Datum::double(*v)), ScalarValue::Utf8(Some(v)) => Some(Datum::string(v.clone())), ScalarValue::LargeUtf8(Some(v)) => Some(Datum::string(v.clone())), + ScalarValue::Binary(Some(v)) => Some(Datum::binary(v.clone())), + ScalarValue::LargeBinary(Some(v)) => Some(Datum::binary(v.clone())), ScalarValue::Date32(Some(v)) => Some(Datum::date(*v)), ScalarValue::Date64(Some(v)) => Some(Datum::date((*v / MILLIS_PER_DAY) as i32)), _ => None, @@ -429,4 +431,31 @@ mod tests { let predicate = convert_to_iceberg_predicate(sql); assert_eq!(predicate, None); } + + #[test] + fn test_scalar_value_to_datum_binary() { + use datafusion::common::ScalarValue; + + let bytes = vec![1u8, 2u8, 3u8]; + let datum = super::scalar_value_to_datum(&ScalarValue::Binary(Some(bytes.clone()))); + assert_eq!(datum, Some(Datum::binary(bytes.clone()))); + + let datum = super::scalar_value_to_datum(&ScalarValue::LargeBinary(Some(bytes.clone()))); + assert_eq!(datum, Some(Datum::binary(bytes))); + + let datum = super::scalar_value_to_datum(&ScalarValue::Binary(None)); + assert_eq!(datum, None); + } + + #[test] + fn test_predicate_conversion_with_binary() { + let sql = "foo = 1 and bar = X'0102'"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + // Binary literals are converted to Datum::binary + // Note: SQL literal 1 is converted to Long by DataFusion + let expected_predicate = Reference::new("foo") + .equal_to(Datum::long(1)) + .and(Reference::new("bar").equal_to(Datum::binary(vec![1u8, 2u8]))); + assert_eq!(predicate, expected_predicate); + } } diff --git a/crates/sqllogictest/src/engine/datafusion.rs b/crates/sqllogictest/src/engine/datafusion.rs index 487d8dc977..7af647be96 100644 --- a/crates/sqllogictest/src/engine/datafusion.rs +++ b/crates/sqllogictest/src/engine/datafusion.rs @@ -95,6 +95,7 @@ impl DataFusionEngine { // Create partitioned test table (unpartitioned tables are now created via SQL) Self::create_partitioned_table(&catalog, &namespace).await?; + Self::create_binary_table(&catalog, &namespace).await?; Ok(Arc::new( IcebergCatalogProvider::try_new(Arc::new(catalog)).await?, @@ -134,4 +135,31 @@ impl DataFusionEngine { Ok(()) } + + /// Create a test table with binary type column + /// Used for testing binary predicate pushdown + /// TODO: this can be removed when we support CREATE TABLE + async fn create_binary_table( + catalog: &impl Catalog, + namespace: &NamespaceIdent, + ) -> anyhow::Result<()> { + let schema = Schema::builder() + .with_fields(vec![ + NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(), + NestedField::optional(2, "data", Type::Primitive(PrimitiveType::Binary)).into(), + ]) + .build()?; + + catalog + .create_table( + namespace, + TableCreation::builder() + .name("test_binary_table".to_string()) + .schema(schema) + .build(), + ) + .await?; + + Ok(()) + } } diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index 1d7f42c8d4..6fdd08054f 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -29,3 +29,7 @@ slt = "df_test/create_table.slt" [[steps]] engine = "df" slt = "df_test/insert_into.slt" + +[[steps]] +engine = "df" +slt = "df_test/binary_predicate_pushdown.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt new file mode 100644 index 0000000000..d3d1db1b67 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt @@ -0,0 +1,69 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test that binary predicates are pushed down to IcebergTableScan +# This validates that ScalarValue::Binary is correctly converted to Datum::binary + +# Verify EXPLAIN shows binary predicate is pushed down to IcebergTableScan +query TT +EXPLAIN SELECT * FROM default.default.test_binary_table WHERE data = X'0102' +---- +logical_plan +01)Filter: default.default.test_binary_table.data = LargeBinary("1,2") +02)--TableScan: default.default.test_binary_table projection=[id, data], partial_filters=[default.default.test_binary_table.data = LargeBinary("1,2")] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: data@1 = 0102 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,data] predicate:[data = 0102] + +# Verify empty result from empty table +query I? +SELECT * FROM default.default.test_binary_table WHERE data = X'0102' +---- + +# Insert test data +query I +INSERT INTO default.default.test_binary_table VALUES (1, X'0102') +---- +1 + +# Verify binary predicate filtering works +query I? +SELECT * FROM default.default.test_binary_table WHERE data = X'0102' +---- +1 0102 + +# Insert more rows with different binary values +query I +INSERT INTO default.default.test_binary_table VALUES (2, X'0304'), (3, X'0102'), (4, X'0506') +---- +3 + +# Verify binary predicate filters correctly +query I? rowsort +SELECT * FROM default.default.test_binary_table WHERE data = X'0102' +---- +1 0102 +3 0102 + +# Verify different binary value +query I? +SELECT * FROM default.default.test_binary_table WHERE data = X'0506' +---- +4 0506 diff --git a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt index 770072f9dc..a0f0e55b5b 100644 --- a/crates/sqllogictest/testdata/slts/df_test/show_tables.slt +++ b/crates/sqllogictest/testdata/slts/df_test/show_tables.slt @@ -25,6 +25,9 @@ datafusion information_schema routines VIEW datafusion information_schema schemata VIEW datafusion information_schema tables VIEW datafusion information_schema views VIEW +default default test_binary_table BASE TABLE +default default test_binary_table$manifests BASE TABLE +default default test_binary_table$snapshots BASE TABLE default default test_partitioned_table BASE TABLE default default test_partitioned_table$manifests BASE TABLE default default test_partitioned_table$snapshots BASE TABLE From 05651e2052723cc80f117f6f26afa20f677b1968 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 21 Jan 2026 17:16:19 -0800 Subject: [PATCH 167/891] feat(datafusion): Add support for DROP TABLE (#2033) ## Which issue does this PR close? - Closes #2060 ## What changes are included in this PR? - Support DROP TABLE syntax by implementing `SchemaProvider::deregister_table` ## Are these changes tested? Added sqllogictests --- crates/integrations/datafusion/src/schema.rs | 76 ++++++++++++++++++- .../testdata/schedules/df_test.toml | 4 + .../testdata/slts/df_test/drop_table.slt | 63 +++++++++++++++ 3 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 crates/sqllogictest/testdata/slts/df_test/drop_table.slt diff --git a/crates/integrations/datafusion/src/schema.rs b/crates/integrations/datafusion/src/schema.rs index 022964ba6d..508aeb303b 100644 --- a/crates/integrations/datafusion/src/schema.rs +++ b/crates/integrations/datafusion/src/schema.rs @@ -29,7 +29,7 @@ use futures::StreamExt; use futures::future::try_join_all; use iceberg::arrow::arrow_schema_to_schema_auto_assign_ids; use iceberg::inspect::MetadataTableType; -use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableCreation}; +use iceberg::{Catalog, Error, ErrorKind, NamespaceIdent, Result, TableCreation, TableIdent}; use crate::table::IcebergTableProvider; use crate::to_datafusion_error; @@ -211,6 +211,42 @@ impl SchemaProvider for IcebergSchemaProvider { DataFusionError::Execution(format!("Failed to create Iceberg table: {e}")) })? } + + fn deregister_table(&self, name: &str) -> DFResult>> { + // Check if table exists + if !self.table_exist(name) { + return Ok(None); + } + + let catalog = self.catalog.clone(); + let namespace = self.namespace.clone(); + let tables = self.tables.clone(); + let table_name = name.to_string(); + + // Use tokio's spawn_blocking to handle the async work on a blocking thread pool + let result = tokio::task::spawn_blocking(move || { + let rt = tokio::runtime::Handle::current(); + rt.block_on(async move { + let table_ident = TableIdent::new(namespace, table_name.clone()); + + // Drop the table from the Iceberg catalog + catalog + .drop_table(&table_ident) + .await + .map_err(to_datafusion_error)?; + + // Remove from local cache and return the removed provider + let removed = tables + .remove(&table_name) + .map(|(_, table)| table as Arc); + + Ok(removed) + }) + }); + + futures::executor::block_on(result) + .map_err(|e| DataFusionError::Execution(format!("Failed to drop Iceberg table: {e}")))? + } } /// Verifies that a table provider contains no data by scanning with LIMIT 1. @@ -368,4 +404,42 @@ mod tests { "Expected error about table already existing, got: {err}", ); } + + #[tokio::test] + async fn test_deregister_table_succeeds() { + let (schema_provider, _temp_dir) = create_test_schema_provider().await; + + // Create and register an empty table + let arrow_schema = Arc::new(ArrowSchema::new(vec![Field::new( + "id", + DataType::Int32, + false, + )])); + + let empty_batch = RecordBatch::new_empty(arrow_schema.clone()); + let mem_table = MemTable::try_new(arrow_schema, vec![vec![empty_batch]]).unwrap(); + + // Register the table + let result = schema_provider.register_table("drop_me".to_string(), Arc::new(mem_table)); + assert!(result.is_ok()); + assert!(schema_provider.table_exist("drop_me")); + + // Deregister the table + let result = schema_provider.deregister_table("drop_me"); + assert!(result.is_ok()); + assert!(result.unwrap().is_some()); + + // Verify the table no longer exists + assert!(!schema_provider.table_exist("drop_me")); + } + + #[tokio::test] + async fn test_deregister_nonexistent_table_returns_none() { + let (schema_provider, _temp_dir) = create_test_schema_provider().await; + + // Attempt to deregister a table that doesn't exist + let result = schema_provider.deregister_table("nonexistent"); + assert!(result.is_ok()); + assert!(result.unwrap().is_none()); + } } diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index 6fdd08054f..1a1a0ad334 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -33,3 +33,7 @@ slt = "df_test/insert_into.slt" [[steps]] engine = "df" slt = "df_test/binary_predicate_pushdown.slt" + +[[steps]] +engine = "df" +slt = "df_test/drop_table.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/drop_table.slt b/crates/sqllogictest/testdata/slts/df_test/drop_table.slt new file mode 100644 index 0000000000..c721c84e11 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/drop_table.slt @@ -0,0 +1,63 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test DROP TABLE functionality + +# Create a table to drop +statement ok +CREATE TABLE default.default.table_to_drop (id INT NOT NULL, name STRING) + +# Verify the table exists +query IT rowsort +SELECT * FROM default.default.table_to_drop +---- + +# Insert some data +query I +INSERT INTO default.default.table_to_drop VALUES (1, 'Test') +---- +1 + +# Verify data exists +query IT rowsort +SELECT * FROM default.default.table_to_drop +---- +1 Test + +# Drop the table +statement ok +DROP TABLE default.default.table_to_drop + +# Verify the table no longer exists (should error) +statement error +SELECT * FROM default.default.table_to_drop + +# Test DROP TABLE IF EXISTS on non-existent table (should not error) +statement ok +DROP TABLE IF EXISTS default.default.nonexistent_table + +# Create another table for IF EXISTS test +statement ok +CREATE TABLE default.default.another_table (id INT NOT NULL) + +# Drop with IF EXISTS +statement ok +DROP TABLE IF EXISTS default.default.another_table + +# Verify it's gone +statement error +SELECT * FROM default.default.another_table From 5d07594becbe9d774105b208c87a4bd9aae68065 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 21 Jan 2026 17:17:20 -0800 Subject: [PATCH 168/891] feat(io): Rename Storage to OpenDalStorage (#2059) ## Which issue does this PR close? - Closes #2052 ## What changes are included in this PR? - Rename Storage to OpenDalStorage so we can introduce Storage trait later ## Are these changes tested? Relies on existing tests --- crates/iceberg/src/io/file_io.rs | 6 +++--- crates/iceberg/src/io/storage.rs | 16 ++++++++-------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/crates/iceberg/src/io/file_io.rs b/crates/iceberg/src/io/file_io.rs index 24e91ca8a4..ef9f8aa935 100644 --- a/crates/iceberg/src/io/file_io.rs +++ b/crates/iceberg/src/io/file_io.rs @@ -24,7 +24,7 @@ use bytes::Bytes; use opendal::Operator; use url::Url; -use super::storage::Storage; +use super::storage::OpenDalStorage; use crate::{Error, ErrorKind, Result}; /// FileIO implementation, used to manipulate files in underlying storage. @@ -48,7 +48,7 @@ use crate::{Error, ErrorKind, Result}; pub struct FileIO { builder: FileIOBuilder, - inner: Arc, + inner: Arc, } impl FileIO { @@ -262,7 +262,7 @@ impl FileIOBuilder { /// Builds [`FileIO`]. pub fn build(self) -> Result { - let storage = Storage::build(self.clone())?; + let storage = OpenDalStorage::build(self.clone())?; Ok(FileIO { builder: self, inner: Arc::new(storage), diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index 03e43600dd..32a95eb1dd 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -43,7 +43,7 @@ use crate::{Error, ErrorKind}; /// The storage carries all supported storage services in iceberg #[derive(Debug)] -pub(crate) enum Storage { +pub(crate) enum OpenDalStorage { #[cfg(feature = "storage-memory")] Memory(Operator), #[cfg(feature = "storage-fs")] @@ -73,7 +73,7 @@ pub(crate) enum Storage { }, } -impl Storage { +impl OpenDalStorage { /// Convert iceberg config to opendal config. pub(crate) fn build(file_io_builder: FileIOBuilder) -> crate::Result { let (scheme_str, props, extensions) = file_io_builder.into_parts(); @@ -137,7 +137,7 @@ impl Storage { let _ = path; let (operator, relative_path): (Operator, &str) = match self { #[cfg(feature = "storage-memory")] - Storage::Memory(op) => { + OpenDalStorage::Memory(op) => { if let Some(stripped) = path.strip_prefix("memory:/") { Ok::<_, crate::Error>((op.clone(), stripped)) } else { @@ -145,7 +145,7 @@ impl Storage { } } #[cfg(feature = "storage-fs")] - Storage::LocalFs => { + OpenDalStorage::LocalFs => { let op = super::fs_config_build()?; if let Some(stripped) = path.strip_prefix("file:/") { @@ -155,7 +155,7 @@ impl Storage { } } #[cfg(feature = "storage-s3")] - Storage::S3 { + OpenDalStorage::S3 { configured_scheme, config, customized_credential_load, @@ -175,7 +175,7 @@ impl Storage { } } #[cfg(feature = "storage-gcs")] - Storage::Gcs { config } => { + OpenDalStorage::Gcs { config } => { let operator = super::gcs_config_build(config, path)?; let prefix = format!("gs://{}/", operator.info().name()); if path.starts_with(&prefix) { @@ -188,7 +188,7 @@ impl Storage { } } #[cfg(feature = "storage-oss")] - Storage::Oss { config } => { + OpenDalStorage::Oss { config } => { let op = super::oss_config_build(config, path)?; // Check prefix of oss path. @@ -203,7 +203,7 @@ impl Storage { } } #[cfg(feature = "storage-azdls")] - Storage::Azdls { + OpenDalStorage::Azdls { configured_scheme, config, } => super::azdls_create_operator(path, config, configured_scheme), From 0200cb0b5ca58f209c0df8069977fba41714936f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 09:03:09 +0800 Subject: [PATCH 169/891] chore(deps): Bump uuid from 1.19.0 to 1.20.0 (#2067) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.19.0 to 1.20.0.
Release notes

Sourced from uuid's releases.

v1.20.0

What's Changed

New Contributors

Full Changelog: https://github.com/uuid-rs/uuid/compare/v1.19.0...v1.20.0

Commits
  • c3346dd Merge pull request #857 from uuid-rs/cargo/v1.20.0
  • 66eebc3 prepare for 1.20.0 release
  • 3b66758 Merge pull request #856 from uuid-rs/docs/bytes-le-ordering
  • e2bdd44 don't run UI tests in wasm
  • b6dc7ec note that ordering applies to fields in to/from_bytes_le
  • a0281cd Merge pull request #855 from uuid-rs/feat/serde-fmt
  • db27b67 Merge pull request #854 from mivort/non-nil-uuid-ord
  • efb06f2 implement Deserialize on adapter types
  • 50d44ad Derive Ord and PartialOrd for NonNilUuid
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=uuid&package-manager=cargo&previous-version=1.19.0&new-version=1.20.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e8805d5102..3eba261bae 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7259,9 +7259,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.19.0" +version = "1.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" +checksum = "ee48d38b119b0cd71fe4141b30f5ba9c7c5d9f4e7a3a8b4a674e4b6ef789976f" dependencies = [ "getrandom 0.3.4", "js-sys", From ffe355b54f0adfd6260d40ea21be11a18f08a2e3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Jan 2026 10:14:01 +0800 Subject: [PATCH 170/891] chore(deps): Bump crate-ci/typos from 1.42.0 to 1.42.1 (#2066) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.42.0 to 1.42.1.
Release notes

Sourced from crate-ci/typos's releases.

v1.42.1

[1.42.1] - 2026-01-19

Fixes

  • Ignore hex literals with suffixes (e.g. 0xffffUL)
Changelog

Sourced from crate-ci/typos's changelog.

[1.42.1] - 2026-01-19

Fixes

  • Ignore hex literals with suffixes (e.g. 0xffffUL)
Commits
  • 6512063 chore: Release
  • 2049566 docs: Update changelog
  • cbc66c9 Merge pull request #1471 from epage/hex
  • 2071579 fix(tokens): Ignore hex literals with suffixes
  • 7300bb0 perf(token): Avoid switching to chars
  • 01955c0 perf(token): Prefer slices over characters
  • 5d4cfab test(cli): Show hex literal issue
  • 3cee018 Merge pull request #1468 from Wilfred/patch-1
  • a96a636 Fix typo in ripsecrets link
  • 837ad27 Merge pull request #1467 from Wilfred/full_examples_in_reference
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.42.0&new-version=1.42.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Renjie Liu --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 514d59b66c..60d3304828 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.42.0 + uses: crate-ci/typos@v1.42.1 From 330f21da894948fc10b57d541cb2d6f32c8bdbb8 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Mon, 26 Jan 2026 19:03:58 +0800 Subject: [PATCH 171/891] feat(datafusion): Add LIKE predicate pushdown for StartsWith patterns (#2014) Implement pushdown for LIKE predicates with simple prefix patterns (e.g., 'prefix%') to Iceberg's StartsWith operator. This optimization allows filtering to be performed at the storage layer, significantly improving query performance for prefix searches. Changes: - Add support for Expr::Like in predicate conversion - Convert LIKE 'prefix%' patterns to StartsWith operator - Convert NOT LIKE 'prefix%' patterns to NotStartsWith operator - Handle edge cases: empty prefix, unicode, special characters - Reject complex patterns that cannot be pushed down (wildcards in middle, underscore, ILIKE) - Add 8 comprehensive unit tests covering various scenarios Implementation details: - Only simple prefix patterns ending with % are converted - Patterns with % or _ wildcards in the prefix are not pushed down - Case-insensitive LIKE (ILIKE) is not supported for pushdown - Escape characters are not supported for pushdown - Works seamlessly with other predicates in AND/OR expressions ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? --------- Co-authored-by: Claude Sonnet 4.5 Co-authored-by: Renjie Liu --- .../src/physical_plan/expr_to_predicate.rs | 147 +++++++++++++++++- .../testdata/schedules/df_test.toml | 4 + .../slts/df_test/like_predicate_pushdown.slt | 113 ++++++++++++++ 3 files changed, 262 insertions(+), 2 deletions(-) create mode 100644 crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt diff --git a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs index fc95be5db5..2468606b48 100644 --- a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs +++ b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs @@ -18,10 +18,10 @@ use std::vec; use datafusion::arrow::datatypes::DataType; -use datafusion::logical_expr::{Expr, Operator}; +use datafusion::logical_expr::{Expr, Like, Operator}; use datafusion::scalar::ScalarValue; use iceberg::expr::{BinaryExpression, Predicate, PredicateOperator, Reference, UnaryExpression}; -use iceberg::spec::Datum; +use iceberg::spec::{Datum, PrimitiveLiteral}; // A datafusion expression could be an Iceberg predicate, column, or literal. enum TransformedResult { @@ -128,6 +128,56 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult { } to_iceberg_predicate(&c.expr) } + Expr::Like(Like { + negated, + expr, + pattern, + escape_char, + case_insensitive, + }) => { + // Only support simple prefix patterns (e.g., 'prefix%') + // Note: Iceberg's StartsWith operator is case-sensitive, so we cannot + // push down case-insensitive LIKE (ILIKE) patterns + // Escape characters are also not supported for pushdown + if escape_char.is_some() || *case_insensitive { + return TransformedResult::NotTransformed; + } + + // Extract the pattern string + let pattern_str = match to_iceberg_predicate(pattern) { + TransformedResult::Literal(d) => match d.literal() { + PrimitiveLiteral::String(s) => s.clone(), + _ => return TransformedResult::NotTransformed, + }, + _ => return TransformedResult::NotTransformed, + }; + + // Check if it's a simple prefix pattern (ends with % and no other wildcards) + if pattern_str.ends_with('%') + && !pattern_str[..pattern_str.len() - 1].contains(['%', '_']) + { + // Extract the prefix (remove trailing %) + let prefix = pattern_str[..pattern_str.len() - 1].to_string(); + + // Get the column reference + let column = match to_iceberg_predicate(expr) { + TransformedResult::Column(r) => r, + _ => return TransformedResult::NotTransformed, + }; + + // Create the appropriate predicate + let predicate = if *negated { + column.not_starts_with(Datum::string(prefix)) + } else { + column.starts_with(Datum::string(prefix)) + }; + + TransformedResult::Predicate(predicate) + } else { + // Complex LIKE patterns cannot be pushed down + TransformedResult::NotTransformed + } + } _ => TransformedResult::NotTransformed, } } @@ -458,4 +508,97 @@ mod tests { .and(Reference::new("bar").equal_to(Datum::binary(vec![1u8, 2u8]))); assert_eq!(predicate, expected_predicate); } + + #[test] + fn test_predicate_conversion_with_like_starts_with() { + let sql = "bar LIKE 'test%'"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + assert_eq!( + predicate, + Reference::new("bar").starts_with(Datum::string("test")) + ); + } + + #[test] + fn test_predicate_conversion_with_not_like_starts_with() { + let sql = "bar NOT LIKE 'test%'"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + assert_eq!( + predicate, + Reference::new("bar").not_starts_with(Datum::string("test")) + ); + } + + #[test] + fn test_predicate_conversion_with_like_empty_prefix() { + let sql = "bar LIKE '%'"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + assert_eq!( + predicate, + Reference::new("bar").starts_with(Datum::string("")) + ); + } + + #[test] + fn test_predicate_conversion_with_like_complex_pattern() { + // Patterns with wildcards in the middle cannot be pushed down + let sql = "bar LIKE 'te%st'"; + let predicate = convert_to_iceberg_predicate(sql); + assert_eq!(predicate, None); + } + + #[test] + fn test_predicate_conversion_with_like_underscore_wildcard() { + // Patterns with underscore wildcard cannot be pushed down + let sql = "bar LIKE 'test_'"; + let predicate = convert_to_iceberg_predicate(sql); + assert_eq!(predicate, None); + } + + #[test] + fn test_predicate_conversion_with_like_no_wildcard() { + // Patterns without trailing % cannot be pushed down as StartsWith + let sql = "bar LIKE 'test'"; + let predicate = convert_to_iceberg_predicate(sql); + assert_eq!(predicate, None); + } + + #[test] + fn test_predicate_conversion_with_ilike() { + // Case-insensitive LIKE (ILIKE) is not supported + let sql = "bar ILIKE 'test%'"; + let predicate = convert_to_iceberg_predicate(sql); + assert_eq!(predicate, None); + } + + #[test] + fn test_predicate_conversion_with_like_and_other_conditions() { + let sql = "bar LIKE 'test%' AND foo > 1"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + let expected_predicate = Predicate::and( + Reference::new("bar").starts_with(Datum::string("test")), + Reference::new("foo").greater_than(Datum::long(1)), + ); + assert_eq!(predicate, expected_predicate); + } + + #[test] + fn test_predicate_conversion_with_like_special_characters() { + // Test LIKE with special characters in prefix + let sql = "bar LIKE 'test-abc_123%'"; + let predicate = convert_to_iceberg_predicate(sql); + // This should not be pushed down because it contains underscore + assert_eq!(predicate, None); + } + + #[test] + fn test_predicate_conversion_with_like_unicode() { + // Test LIKE with unicode characters in prefix + let sql = "bar LIKE '测试%'"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + assert_eq!( + predicate, + Reference::new("bar").starts_with(Datum::string("测试")) + ); + } } diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index 1a1a0ad334..8f4860043b 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -34,6 +34,10 @@ slt = "df_test/insert_into.slt" engine = "df" slt = "df_test/binary_predicate_pushdown.slt" +[[steps]] +engine = "df" +slt = "df_test/like_predicate_pushdown.slt" + [[steps]] engine = "df" slt = "df_test/drop_table.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt new file mode 100644 index 0000000000..41d8ec0276 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt @@ -0,0 +1,113 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test LIKE predicate pushdown for StartsWith patterns +# This validates that LIKE 'prefix%' patterns are converted to StartsWith operator +# Note: Iceberg's StartsWith operator is case-sensitive + +# Note: test_unpartitioned_table already contains data from insert_into.slt: +# (1, 'Alice'), (2, 'Bob'), (3, 'Charlie'), (4, NULL) +# We'll insert additional test data for LIKE testing +query I +INSERT INTO default.default.test_unpartitioned_table VALUES (5, 'alice'), (6, 'Albert'), (7, 'Bobby'), (8, 'Carol') +---- +4 + +# Test basic LIKE prefix pattern pushdown - verify predicate is pushed to IcebergTableScan +query TT +EXPLAIN SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'Al%' +---- +logical_plan +01)Filter: default.default.test_unpartitioned_table.name LIKE Utf8("Al%") +02)--TableScan: default.default.test_unpartitioned_table projection=[id, name], partial_filters=[default.default.test_unpartitioned_table.name LIKE Utf8("Al%")] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: name@1 LIKE Al% +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,name] predicate:[name STARTS WITH "Al"] + +# Test LIKE filtering with case-sensitive match +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'Al%' +---- +1 Alice +6 Albert + +# Test NOT LIKE prefix pattern pushdown +query TT +EXPLAIN SELECT * FROM default.default.test_unpartitioned_table WHERE name NOT LIKE 'Al%' +---- +logical_plan +01)Filter: default.default.test_unpartitioned_table.name NOT LIKE Utf8("Al%") +02)--TableScan: default.default.test_unpartitioned_table projection=[id, name], partial_filters=[default.default.test_unpartitioned_table.name NOT LIKE Utf8("Al%")] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: name@1 NOT LIKE Al% +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,name] predicate:[name NOT STARTS WITH "Al"] + +# Test NOT LIKE filtering +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name NOT LIKE 'Al%' +---- +2 Bob +3 Charlie +5 alice +7 Bobby +8 Carol + +# Test case sensitivity - lowercase 'al%' should not match 'Alice' or 'Albert' +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'al%' +---- +5 alice + +# Test LIKE with empty prefix (matches everything except NULL) +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE '%' +---- +1 Alice +2 Bob +3 Charlie +5 alice +6 Albert +7 Bobby +8 Carol + +# Test LIKE with single character prefix +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'B%' +---- +2 Bob +7 Bobby + +# Test LIKE combined with other predicates +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'Al%' AND id > 1 +---- +6 Albert + +# Test LIKE in OR expression +query IT rowsort +SELECT * FROM default.default.test_unpartitioned_table WHERE name LIKE 'Al%' OR name LIKE 'Bo%' +---- +1 Alice +2 Bob +6 Albert +7 Bobby From 2a43943b3a8c252679ab86c77c2adfa1d41cd0a1 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Mon, 26 Jan 2026 17:57:50 -0800 Subject: [PATCH 172/891] feat(io): Add Storage, StorageFactory, StorageConfig (#2061) ## Which issue does this PR close? - Closes #2053 #2054 - Related to #2055 ## What changes are included in this PR? - `Storage`, `StorageFactory` trait, and `StorageConfig` - They are not used anywhere for now. ## Are these changes tested? Added some simple uts for `StorageConfig` --- Cargo.lock | 73 +++++++-- Cargo.toml | 1 + crates/iceberg/Cargo.toml | 1 + crates/iceberg/src/io/config/mod.rs | 237 ++++++++++++++++++++++++++++ crates/iceberg/src/io/mod.rs | 3 + crates/iceberg/src/io/storage.rs | 129 ++++++++++++++- 6 files changed, 425 insertions(+), 19 deletions(-) create mode 100644 crates/iceberg/src/io/config/mod.rs diff --git a/Cargo.lock b/Cargo.lock index 3eba261bae..003a52a38a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2497,7 +2497,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2632,6 +2632,17 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "erased-serde" +version = "0.4.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89e8918065695684b2b0702da20382d5ae6065cf3327bc2d6436bd49a71ce9f3" +dependencies = [ + "serde", + "serde_core", + "typeid", +] + [[package]] name = "errno" version = "0.3.14" @@ -2639,7 +2650,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3312,7 +3323,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.1", "tokio", "tower-service", "tracing", @@ -3398,6 +3409,7 @@ dependencies = [ "tempfile", "tokio", "typed-builder", + "typetag", "url", "uuid", "zstd", @@ -3787,6 +3799,15 @@ dependencies = [ "tokio", ] +[[package]] +name = "inventory" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc61209c082fbeb19919bee74b176221b27223e27b65d781eb91af24eb1fb46e" +dependencies = [ + "rustversion", +] + [[package]] name = "ipnet" version = "2.11.0" @@ -3845,7 +3866,7 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4366,7 +4387,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -5166,7 +5187,7 @@ dependencies = [ "quinn-udp", "rustc-hash", "rustls 0.23.35", - "socket2 0.5.10", + "socket2 0.6.1", "thiserror 2.0.17", "tokio", "tracing", @@ -5203,9 +5224,9 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.1", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -5697,7 +5718,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6699,7 +6720,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -7111,12 +7132,42 @@ dependencies = [ "syn 2.0.111", ] +[[package]] +name = "typeid" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c" + [[package]] name = "typenum" version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" +[[package]] +name = "typetag" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be2212c8a9b9bcfca32024de14998494cf9a5dfa59ea1b829de98bac374b86bf" +dependencies = [ + "erased-serde", + "inventory", + "once_cell", + "serde", + "typetag-impl", +] + +[[package]] +name = "typetag-impl" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27a7a9b72ba121f6f1f6c3632b85604cac41aedb5ddc70accbebb6cac83de846" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.111", +] + [[package]] name = "typify" version = "0.5.0" @@ -7514,7 +7565,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 517bfa36e8..148a45449b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -128,6 +128,7 @@ toml = "0.8" tracing = "0.1.41" tracing-subscriber = "0.3.20" typed-builder = "0.20" +typetag = "0.2" url = "2.5.7" uuid = { version = "1.18", features = ["v7"] } volo = "0.10.6" diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index 6f1332a444..9a5de7736d 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -86,6 +86,7 @@ serde_with = { workspace = true } strum = { workspace = true, features = ["derive"] } tokio = { workspace = true, optional = false, features = ["sync"] } typed-builder = { workspace = true } +typetag = { workspace = true } url = { workspace = true } uuid = { workspace = true } zstd = { workspace = true } diff --git a/crates/iceberg/src/io/config/mod.rs b/crates/iceberg/src/io/config/mod.rs new file mode 100644 index 0000000000..648e8baf47 --- /dev/null +++ b/crates/iceberg/src/io/config/mod.rs @@ -0,0 +1,237 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// TODO Add specific configs +//! Storage configuration for storage backends. +//! +//! This module provides configuration types for various storage backends. +//! The configuration types are designed to be used with the `StorageFactory` +//! trait to create storage instances. +//! +//! # Available Configurations +//! +//! - [`StorageConfig`]: Base configuration containing properties for storage backends +//! - [`S3Config`]: Amazon S3 specific configuration +//! - [`GcsConfig`]: Google Cloud Storage specific configuration +//! - [`OssConfig`]: Alibaba Cloud OSS specific configuration +//! - [`AzdlsConfig`]: Azure Data Lake Storage specific configuration + +// TODO Add specific configs +// mod azdls; +// mod gcs; +// mod oss; +// mod s3; + +use std::collections::HashMap; + +// TODO Add specific configs +// pub use azdls::*; +// pub use gcs::*; +// pub use oss::*; +// pub use s3::*; +use serde::{Deserialize, Serialize}; + +/// Configuration properties for storage backends. +/// +/// This struct contains only configuration properties without specifying +/// which storage backend to use. The storage type is determined by the +/// explicit factory selection. +/// ``` +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)] +pub struct StorageConfig { + /// Configuration properties for the storage backend + props: HashMap, +} + +impl StorageConfig { + /// Create a new empty StorageConfig. + pub fn new() -> Self { + Self { + props: HashMap::new(), + } + } + + /// Create a StorageConfig from existing properties. + /// + /// # Arguments + /// + /// * `props` - Configuration properties for the storage backend + pub fn from_props(props: HashMap) -> Self { + Self { props } + } + + /// Get all configuration properties. + pub fn props(&self) -> &HashMap { + &self.props + } + + /// Get a specific configuration property by key. + /// + /// # Arguments + /// + /// * `key` - The property key to look up + /// + /// # Returns + /// + /// An `Option` containing a reference to the property value if it exists. + pub fn get(&self, key: &str) -> Option<&String> { + self.props.get(key) + } + + /// Add a configuration property. + /// + /// This is a builder-style method that returns `self` for chaining. + /// + /// # Arguments + /// + /// * `key` - The property key + /// * `value` - The property value + pub fn with_prop(mut self, key: impl Into, value: impl Into) -> Self { + self.props.insert(key.into(), value.into()); + self + } + + /// Add multiple configuration properties. + /// + /// This is a builder-style method that returns `self` for chaining. + /// + /// # Arguments + /// + /// * `props` - An iterator of key-value pairs to add + pub fn with_props( + mut self, + props: impl IntoIterator, impl Into)>, + ) -> Self { + self.props + .extend(props.into_iter().map(|(k, v)| (k.into(), v.into()))); + self + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_storage_config_new() { + let config = StorageConfig::new(); + + assert!(config.props().is_empty()); + } + + #[test] + fn test_storage_config_from_props() { + let props = HashMap::from([ + ("region".to_string(), "us-east-1".to_string()), + ("bucket".to_string(), "my-bucket".to_string()), + ]); + let config = StorageConfig::from_props(props.clone()); + + assert_eq!(config.props(), &props); + } + + #[test] + fn test_storage_config_default() { + let config = StorageConfig::default(); + + assert!(config.props().is_empty()); + } + + #[test] + fn test_storage_config_get() { + let config = StorageConfig::new().with_prop("region", "us-east-1"); + + assert_eq!(config.get("region"), Some(&"us-east-1".to_string())); + assert_eq!(config.get("nonexistent"), None); + } + + #[test] + fn test_storage_config_with_prop() { + let config = StorageConfig::new() + .with_prop("region", "us-east-1") + .with_prop("bucket", "my-bucket"); + + assert_eq!(config.get("region"), Some(&"us-east-1".to_string())); + assert_eq!(config.get("bucket"), Some(&"my-bucket".to_string())); + } + + #[test] + fn test_storage_config_with_props() { + let additional_props = vec![("key1", "value1"), ("key2", "value2")]; + let config = StorageConfig::new().with_props(additional_props); + + assert_eq!(config.get("key1"), Some(&"value1".to_string())); + assert_eq!(config.get("key2"), Some(&"value2".to_string())); + } + + #[test] + fn test_storage_config_clone() { + let config = StorageConfig::new().with_prop("region", "us-east-1"); + let cloned = config.clone(); + + assert_eq!(config, cloned); + assert_eq!(cloned.get("region"), Some(&"us-east-1".to_string())); + } + + #[test] + fn test_storage_config_serialization_roundtrip() { + let config = StorageConfig::new() + .with_prop("region", "us-east-1") + .with_prop("bucket", "my-bucket"); + + let serialized = serde_json::to_string(&config).unwrap(); + let deserialized: StorageConfig = serde_json::from_str(&serialized).unwrap(); + + assert_eq!(config, deserialized); + } + + #[test] + fn test_storage_config_clone_independence() { + let original = StorageConfig::new().with_prop("region", "us-east-1"); + let mut cloned = original.clone(); + + // Modify the clone + cloned = cloned.with_prop("region", "eu-west-1"); + cloned = cloned.with_prop("new_key", "new_value"); + + // Original should be unchanged + assert_eq!(original.get("region"), Some(&"us-east-1".to_string())); + assert_eq!(original.get("new_key"), None); + + // Clone should have the new values + assert_eq!(cloned.get("region"), Some(&"eu-west-1".to_string())); + assert_eq!(cloned.get("new_key"), Some(&"new_value".to_string())); + } + + #[test] + fn test_storage_config_from_props_empty() { + let config = StorageConfig::from_props(HashMap::new()); + + assert!(config.props().is_empty()); + } + + #[test] + fn test_storage_config_serialization_empty() { + let config = StorageConfig::new(); + + let serialized = serde_json::to_string(&config).unwrap(); + let deserialized: StorageConfig = serde_json::from_str(&serialized).unwrap(); + + assert_eq!(config, deserialized); + assert!(deserialized.props().is_empty()); + } +} diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 5eb5964345..55f2f262cb 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -66,6 +66,7 @@ //! - `new_input`: Create input file for reading. //! - `new_output`: Create output file for writing. +mod config; mod file_io; mod storage; @@ -85,6 +86,8 @@ mod storage_oss; #[cfg(feature = "storage-s3")] mod storage_s3; +pub use config::*; +pub use storage::{Storage, StorageFactory}; #[cfg(feature = "storage-azdls")] pub use storage_azdls::*; #[cfg(feature = "storage-fs")] diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index 32a95eb1dd..77978e2239 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -15,14 +15,11 @@ // specific language governing permissions and limitations // under the License. -#[cfg(any( - feature = "storage-s3", - feature = "storage-gcs", - feature = "storage-oss", - feature = "storage-azdls", -))] +use std::fmt::Debug; use std::sync::Arc; +use async_trait::async_trait; +use bytes::Bytes; use opendal::layers::RetryLayer; #[cfg(feature = "storage-azdls")] use opendal::services::AzdlsConfig; @@ -36,10 +33,126 @@ use opendal::{Operator, Scheme}; #[cfg(feature = "storage-azdls")] use super::AzureStorageScheme; -use super::FileIOBuilder; +use super::{ + FileIOBuilder, FileMetadata, FileRead, FileWrite, InputFile, OutputFile, StorageConfig, +}; #[cfg(feature = "storage-s3")] use crate::io::CustomAwsCredentialLoader; -use crate::{Error, ErrorKind}; +use crate::{Error, ErrorKind, Result}; + +/// Trait for storage operations in Iceberg. +/// +/// The trait supports serialization via `typetag`, allowing storage instances to be +/// serialized and deserialized across process boundaries. +/// +/// Third-party implementations can implement this trait to provide custom storage backends. +/// +/// # Implementing Custom Storage +/// +/// To implement a custom storage backend: +/// +/// 1. Create a struct that implements this trait +/// 2. Add `#[typetag::serde]` attribute for serialization support +/// 3. Implement all required methods +/// +/// # Example +/// +/// ```rust,ignore +/// #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +/// struct MyStorage { +/// // custom fields +/// } +/// +/// #[async_trait] +/// #[typetag::serde] +/// impl Storage for MyStorage { +/// async fn exists(&self, path: &str) -> Result { +/// // implementation +/// todo!() +/// } +/// // ... implement other methods +/// } +/// +/// TODO remove below when the trait is integrated with FileIO and Catalog +/// # NOTE +/// This trait is under heavy development and is not used anywhere as of now +/// Please DO NOT implement it +/// ``` +#[async_trait] +#[typetag::serde(tag = "type")] +pub trait Storage: Debug + Send + Sync { + /// Check if a file exists at the given path + async fn exists(&self, path: &str) -> Result; + + /// Get metadata from an input path + async fn metadata(&self, path: &str) -> Result; + + /// Read bytes from a path + async fn read(&self, path: &str) -> Result; + + /// Get FileRead from a path + async fn reader(&self, path: &str) -> Result>; + + /// Write bytes to an output path + async fn write(&self, path: &str, bs: Bytes) -> Result<()>; + + /// Get FileWrite from a path + async fn writer(&self, path: &str) -> Result>; + + /// Delete a file at the given path + async fn delete(&self, path: &str) -> Result<()>; + + /// Delete all files with the given prefix + async fn delete_prefix(&self, path: &str) -> Result<()>; + + /// Create a new input file for reading + fn new_input(&self, path: &str) -> Result; + + /// Create a new output file for writing + fn new_output(&self, path: &str) -> Result; +} + +/// Factory for creating Storage instances from configuration. +/// +/// Implement this trait to provide custom storage backends. The factory pattern +/// allows for lazy initialization of storage instances and enables users to +/// inject custom storage implementations into catalogs. +/// +/// # Example +/// +/// ```rust,ignore +/// #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +/// struct MyCustomStorageFactory { +/// // custom configuration +/// } +/// +/// #[typetag::serde] +/// impl StorageFactory for MyCustomStorageFactory { +/// fn build(&self, config: &StorageConfig) -> Result> { +/// // Create and return custom storage implementation +/// todo!() +/// } +/// } +/// +/// TODO remove below when the trait is integrated with FileIO and Catalog +/// # NOTE +/// This trait is under heavy development and is not used anywhere as of now +/// Please DO NOT implement it +/// ``` +#[typetag::serde(tag = "type")] +pub trait StorageFactory: Debug + Send + Sync { + /// Build a new Storage instance from the given configuration. + /// + /// # Arguments + /// + /// * `config` - The storage configuration containing scheme and properties + /// + /// # Returns + /// + /// A `Result` containing an `Arc` on success, or an error + /// if the storage could not be created. + fn build(&self, config: &StorageConfig) -> Result>; +} /// The storage carries all supported storage services in iceberg #[derive(Debug)] From 946a28fa3714ec4425fb31e295020b2c4ae2d06b Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Wed, 28 Jan 2026 08:20:00 +0800 Subject: [PATCH 173/891] Simplify test setup by starting container only once. (#2071) ## Which issue does this PR close? - Closes #2070 . ## What changes are included in this PR? 1. We removed the ci action to free up disk space, which was a hack and becomes quite unstable. 2. Refactored tests to start containers only once. Also we introduced some utility methods to help making namespace unique across tests. ## Are these changes tested? Yes, ut --- .github/workflows/ci.yml | 37 +--- Cargo.lock | 25 +-- Cargo.toml | 2 - Makefile | 13 +- crates/catalog/glue/Cargo.toml | 2 - .../testdata/glue_catalog/docker-compose.yaml | 44 ---- .../catalog/glue/tests/glue_catalog_test.rs | 137 ++++++------ crates/catalog/hms/Cargo.toml | 2 - .../testdata/hms_catalog/docker-compose.yaml | 49 ----- crates/catalog/hms/tests/hms_catalog_test.rs | 124 +++++------ crates/catalog/rest/Cargo.toml | 2 - .../testdata/rest_catalog/docker-compose.yaml | 65 ------ .../catalog/rest/tests/rest_catalog_test.rs | 207 +++++++++++------- crates/iceberg/Cargo.toml | 1 - .../testdata/file_io_gcs/docker-compose.yaml | 23 -- .../testdata/file_io_s3/docker-compose.yaml | 30 --- crates/iceberg/tests/file_io_gcs_test.rs | 45 +--- crates/iceberg/tests/file_io_s3_test.rs | 75 +++---- crates/integration_tests/Cargo.toml | 1 - crates/integration_tests/src/lib.rs | 44 +++- .../testdata/docker-compose.yaml | 108 --------- crates/integration_tests/tests/shared.rs | 20 +- .../tests/shared_tests/mod.rs | 2 +- crates/test_utils/Cargo.toml | 1 + crates/test_utils/src/lib.rs | 81 +++++++ dev/docker-compose.yaml | 202 +++++++++++++++++ .../hms_catalog => dev/hms}/Dockerfile | 0 .../hms_catalog => dev/hms}/core-site.xml | 102 ++++----- .../testdata => dev}/spark/Dockerfile | 0 .../testdata => dev}/spark/provision.py | 0 .../spark/spark-defaults.conf | 0 31 files changed, 693 insertions(+), 751 deletions(-) delete mode 100644 crates/catalog/glue/testdata/glue_catalog/docker-compose.yaml delete mode 100644 crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml delete mode 100644 crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml delete mode 100644 crates/iceberg/testdata/file_io_gcs/docker-compose.yaml delete mode 100644 crates/iceberg/testdata/file_io_s3/docker-compose.yaml delete mode 100644 crates/integration_tests/testdata/docker-compose.yaml create mode 100644 dev/docker-compose.yaml rename {crates/catalog/hms/testdata/hms_catalog => dev/hms}/Dockerfile (100%) rename {crates/catalog/hms/testdata/hms_catalog => dev/hms}/core-site.xml (96%) rename {crates/integration_tests/testdata => dev}/spark/Dockerfile (100%) rename {crates/integration_tests/testdata => dev}/spark/provision.py (100%) mode change 100755 => 100644 rename {crates/integration_tests/testdata => dev}/spark/spark-defaults.conf (100%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index af9b812c77..829f64030e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -86,18 +86,6 @@ jobs: - macos-latest - windows-latest steps: - - name: Maximize build space (Ubuntu) - if: matrix.os == 'ubuntu-latest' - uses: easimon/maximize-build-space@master - with: - remove-dotnet: "true" - remove-android: "true" - remove-haskell: "true" - remove-codeql: "true" - remove-docker-images: "true" - root-reserve-mb: 10240 - temp-reserve-mb: 10240 - - uses: actions/checkout@v6 - name: Setup Rust toolchain @@ -134,26 +122,15 @@ jobs: - name: Build run: cargo build -p iceberg --no-default-features - unit: + tests: runs-on: ubuntu-latest strategy: matrix: test-suite: - { name: "default", args: "--all-targets --all-features --workspace" } - { name: "doc", args: "--doc --all-features --workspace" } - name: Unit Tests (${{ matrix.test-suite.name }}) + name: Tests (${{ matrix.test-suite.name }}) steps: - - name: Maximize build space - uses: easimon/maximize-build-space@master - with: - remove-dotnet: "true" - remove-android: "true" - remove-haskell: "true" - remove-codeql: "true" - remove-docker-images: "true" - root-reserve-mb: 10240 - temp-reserve-mb: 10240 - - uses: actions/checkout@v6 - name: Setup Rust toolchain @@ -169,9 +146,17 @@ jobs: with: key: ${{ matrix.test-suite.name }} - - name: Test + - name: Start Docker containers + if: matrix.test-suite.name == 'default' + run: make docker-up + + - name: Run tests run: cargo test --no-fail-fast ${{ matrix.test-suite.args }} + - name: Stop Docker containers + if: always() && matrix.test-suite.name == 'default' + run: make docker-down + msrv: name: Verify MSRV runs-on: ubuntu-latest diff --git a/Cargo.lock b/Cargo.lock index 003a52a38a..c0c46ce672 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1546,16 +1546,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "ctor" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a2785755761f3ddc1492979ce1e48d2c00d09311c39e4466429188f3dd6501" -dependencies = [ - "quote", - "syn 2.0.111", -] - [[package]] name = "darling" version = "0.20.11" @@ -3375,7 +3365,6 @@ dependencies = [ "bimap", "bytes", "chrono", - "ctor", "derive_builder", "expect-test", "flate2", @@ -3431,10 +3420,8 @@ dependencies = [ "async-trait", "aws-config", "aws-sdk-glue", - "ctor", "iceberg", "iceberg_test_utils", - "port_scanner", "serde_json", "tokio", "tracing", @@ -3447,7 +3434,6 @@ dependencies = [ "anyhow", "async-trait", "chrono", - "ctor", "faststr", "hive_metastore", "iceberg", @@ -3456,7 +3442,6 @@ dependencies = [ "metainfo", "motore-macros", "pilota", - "port_scanner", "serde_json", "tokio", "tracing", @@ -3486,13 +3471,11 @@ version = "0.8.0" dependencies = [ "async-trait", "chrono", - "ctor", "http 1.4.0", "iceberg", "iceberg_test_utils", "itertools 0.13.0", "mockito", - "port_scanner", "reqwest", "serde", "serde_derive", @@ -3564,7 +3547,6 @@ version = "0.8.0" dependencies = [ "arrow-array", "arrow-schema", - "ctor", "datafusion", "futures", "iceberg", @@ -3625,6 +3607,7 @@ dependencies = [ name = "iceberg_test_utils" version = "0.8.0" dependencies = [ + "iceberg", "tracing", "tracing-subscriber", ] @@ -4908,12 +4891,6 @@ version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" -[[package]] -name = "port_scanner" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "325a6d2ac5dee293c3b2612d4993b98aec1dff096b0a2dae70ed7d95784a05da" - [[package]] name = "portable-atomic" version = "1.11.1" diff --git a/Cargo.toml b/Cargo.toml index 148a45449b..46ac4736b8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -61,7 +61,6 @@ bimap = "0.6" bytes = "1.10" chrono = "0.4.41" clap = { version = "4.5.48", features = ["derive", "cargo"] } -ctor = "0.2.8" dashmap = "6" datafusion = "51.0" datafusion-cli = "51.0" @@ -104,7 +103,6 @@ opendal = "0.55.0" ordered-float = "4" parquet = "57.0" pilota = "0.11.10" -port_scanner = "0.1.5" pretty_assertions = "1.4" rand = "0.8.5" regex = "1.11.3" diff --git a/Makefile b/Makefile index 6b8a039b43..14093e1829 100644 --- a/Makefile +++ b/Makefile @@ -55,7 +55,8 @@ doc-test: unit-test: doc-test cargo test --no-fail-fast --lib --all-features --workspace -test: doc-test +test: docker-up + @trap '$(MAKE) docker-down' EXIT; \ cargo test --no-fail-fast --all-targets --all-features --workspace clean: @@ -66,3 +67,13 @@ install-mdbook: site: install-mdbook cd website && mdbook serve + +# Docker targets for integration tests +docker-up: + docker compose -f dev/docker-compose.yaml up -d --build --wait + +docker-down: + docker compose -f dev/docker-compose.yaml down -v --remove-orphans --timeout 0 + +docker-logs: + docker compose -f dev/docker-compose.yaml logs -f diff --git a/crates/catalog/glue/Cargo.toml b/crates/catalog/glue/Cargo.toml index b6126021f5..f42fedeae1 100644 --- a/crates/catalog/glue/Cargo.toml +++ b/crates/catalog/glue/Cargo.toml @@ -39,6 +39,4 @@ tokio = { workspace = true } tracing = { workspace = true } [dev-dependencies] -ctor = { workspace = true } iceberg_test_utils = { path = "../../test_utils", features = ["tests"] } -port_scanner = { workspace = true } diff --git a/crates/catalog/glue/testdata/glue_catalog/docker-compose.yaml b/crates/catalog/glue/testdata/glue_catalog/docker-compose.yaml deleted file mode 100644 index bb77d5f8b5..0000000000 --- a/crates/catalog/glue/testdata/glue_catalog/docker-compose.yaml +++ /dev/null @@ -1,44 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -services: - minio: - image: minio/minio:RELEASE.2025-05-24T17-08-30Z - expose: - - 9000 - - 9001 - environment: - - MINIO_ROOT_USER=admin - - MINIO_ROOT_PASSWORD=password - - MINIO_DOMAIN=minio - command: [ "server", "/data", "--console-address", ":9001" ] - - mc: - depends_on: - - minio - image: minio/mc:RELEASE.2025-05-21T01-59-54Z - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - entrypoint: > - /bin/sh -c " until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " - - moto: - image: motoserver/moto:5.0.3 - expose: - - 5000 diff --git a/crates/catalog/glue/tests/glue_catalog_test.rs b/crates/catalog/glue/tests/glue_catalog_test.rs index 4917032149..f6e2060c0f 100644 --- a/crates/catalog/glue/tests/glue_catalog_test.rs +++ b/crates/catalog/glue/tests/glue_catalog_test.rs @@ -16,12 +16,12 @@ // under the License. //! Integration tests for glue catalog. +//! +//! These tests assume Docker containers are started externally via `make docker-up`. +//! Each test uses unique namespaces based on module path to avoid conflicts. use std::collections::HashMap; -use std::net::SocketAddr; -use std::sync::RwLock; -use ctor::{ctor, dtor}; use iceberg::io::{S3_ACCESS_KEY_ID, S3_ENDPOINT, S3_REGION, S3_SECRET_ACCESS_KEY}; use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; use iceberg::transaction::{ApplyTransactionAction, Transaction}; @@ -32,55 +32,18 @@ use iceberg_catalog_glue::{ AWS_ACCESS_KEY_ID, AWS_REGION_NAME, AWS_SECRET_ACCESS_KEY, GLUE_CATALOG_PROP_URI, GLUE_CATALOG_PROP_WAREHOUSE, GlueCatalog, GlueCatalogBuilder, }; -use iceberg_test_utils::docker::DockerCompose; -use iceberg_test_utils::{normalize_test_name, set_up}; -use port_scanner::scan_port_addr; +use iceberg_test_utils::{ + cleanup_namespace, get_glue_endpoint, get_minio_endpoint, normalize_test_name_with_parts, + set_up, +}; use tokio::time::sleep; use tracing::info; -const GLUE_CATALOG_PORT: u16 = 5000; -const MINIO_PORT: u16 = 9000; -static DOCKER_COMPOSE_ENV: RwLock> = RwLock::new(None); - -#[ctor] -fn before_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - let docker_compose = DockerCompose::new( - normalize_test_name(module_path!()), - format!("{}/testdata/glue_catalog", env!("CARGO_MANIFEST_DIR")), - ); - docker_compose.up(); - guard.replace(docker_compose); -} - -#[dtor] -fn after_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - guard.take(); -} - async fn get_catalog() -> GlueCatalog { set_up(); - let (glue_catalog_ip, minio_ip) = { - let guard = DOCKER_COMPOSE_ENV.read().unwrap(); - let docker_compose = guard.as_ref().unwrap(); - ( - docker_compose.get_container_ip("moto"), - docker_compose.get_container_ip("minio"), - ) - }; - let glue_socket_addr = SocketAddr::new(glue_catalog_ip, GLUE_CATALOG_PORT); - let minio_socket_addr = SocketAddr::new(minio_ip, MINIO_PORT); - while !scan_port_addr(glue_socket_addr) { - info!("Waiting for 1s glue catalog to ready..."); - sleep(std::time::Duration::from_millis(1000)).await; - } - - while !scan_port_addr(minio_socket_addr) { - info!("Waiting for 1s minio to ready..."); - sleep(std::time::Duration::from_millis(1000)).await; - } + let glue_endpoint = get_glue_endpoint(); + let minio_endpoint = get_minio_endpoint(); let props = HashMap::from([ (AWS_ACCESS_KEY_ID.to_string(), "my_access_id".to_string()), @@ -89,10 +52,7 @@ async fn get_catalog() -> GlueCatalog { "my_secret_key".to_string(), ), (AWS_REGION_NAME.to_string(), "us-east-1".to_string()), - ( - S3_ENDPOINT.to_string(), - format!("http://{minio_socket_addr}"), - ), + (S3_ENDPOINT.to_string(), minio_endpoint), (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), (S3_REGION.to_string(), "us-east-1".to_string()), @@ -117,10 +77,7 @@ async fn get_catalog() -> GlueCatalog { } let mut glue_props = HashMap::from([ - ( - GLUE_CATALOG_PROP_URI.to_string(), - format!("http://{glue_socket_addr}"), - ), + (GLUE_CATALOG_PROP_URI.to_string(), glue_endpoint), ( GLUE_CATALOG_PROP_WAREHOUSE.to_string(), "s3a://warehouse/hive".to_string(), @@ -163,7 +120,11 @@ fn set_table_creation(location: Option, name: impl ToString) -> Result Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_rename_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_rename_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; catalog .create_namespace(namespace.name(), HashMap::new()) @@ -190,7 +151,11 @@ async fn test_rename_table() -> Result<()> { async fn test_table_exists() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_table_exists".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_table_exists" + ))); + cleanup_namespace(&catalog, namespace.name()).await; catalog .create_namespace(namespace.name(), HashMap::new()) @@ -214,7 +179,11 @@ async fn test_table_exists() -> Result<()> { async fn test_drop_table() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_drop_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_drop_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; catalog .create_namespace(namespace.name(), HashMap::new()) @@ -235,7 +204,11 @@ async fn test_drop_table() -> Result<()> { async fn test_load_table() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_load_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_load_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; catalog .create_namespace(namespace.name(), HashMap::new()) @@ -260,7 +233,9 @@ async fn test_load_table() -> Result<()> { #[tokio::test] async fn test_create_table() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_create_table".to_string()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_create_table")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; // inject custom location, ignore the namespace prefix let creation = set_table_creation(Some("s3a://warehouse/hive".into()), "my_table")?; @@ -285,7 +260,9 @@ async fn test_create_table() -> Result<()> { #[tokio::test] async fn test_list_tables() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_list_tables".to_string()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_list_tables")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; let expected = vec![]; @@ -299,7 +276,9 @@ async fn test_list_tables() -> Result<()> { #[tokio::test] async fn test_drop_namespace() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_drop_namespace".to_string()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_drop_namespace")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; let exists = catalog.namespace_exists(&namespace).await?; @@ -316,7 +295,9 @@ async fn test_drop_namespace() -> Result<()> { #[tokio::test] async fn test_update_namespace() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_update_namespace".into()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_update_namespace")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; let before_update = catalog.get_namespace(&namespace).await?; @@ -340,7 +321,9 @@ async fn test_update_namespace() -> Result<()> { async fn test_namespace_exists() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_namespace_exists".into()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_namespace_exists")); + cleanup_namespace(&catalog, &namespace).await; let exists = catalog.namespace_exists(&namespace).await?; assert!(!exists); @@ -357,7 +340,9 @@ async fn test_namespace_exists() -> Result<()> { async fn test_get_namespace() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_get_namespace".into()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_get_namespace")); + cleanup_namespace(&catalog, &namespace).await; let does_not_exist = catalog.get_namespace(&namespace).await; assert!(does_not_exist.is_err()); @@ -377,7 +362,9 @@ async fn test_create_namespace() -> Result<()> { let catalog = get_catalog().await; let properties = HashMap::new(); - let namespace = NamespaceIdent::new("test_create_namespace".into()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_create_namespace")); + cleanup_namespace(&catalog, &namespace).await; let expected = Namespace::new(namespace.clone()); @@ -392,7 +379,9 @@ async fn test_create_namespace() -> Result<()> { async fn test_list_namespace() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_list_namespace".to_string()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_list_namespace")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; let result = catalog.list_namespaces(None).await?; @@ -408,7 +397,11 @@ async fn test_list_namespace() -> Result<()> { async fn test_update_table() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_update_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_update_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; catalog .create_namespace(namespace.name(), HashMap::new()) @@ -473,13 +466,13 @@ async fn test_update_table() -> Result<()> { #[tokio::test] async fn test_register_table() -> Result<()> { let catalog = get_catalog().await; - let namespace = NamespaceIdent::new("test_register_table".into()); + // Use unique namespace to avoid conflicts + let namespace = NamespaceIdent::new(normalize_test_name_with_parts!("test_register_table")); + cleanup_namespace(&catalog, &namespace).await; set_test_namespace(&catalog, &namespace).await?; - let creation = set_table_creation( - Some("s3a://warehouse/hive/test_register_table".into()), - "my_table", - )?; + let location = format!("s3a://warehouse/hive/{namespace}"); + let creation = set_table_creation(Some(location), "my_table")?; let table = catalog.create_table(&namespace, creation).await?; let metadata_location = table .metadata_location() diff --git a/crates/catalog/hms/Cargo.toml b/crates/catalog/hms/Cargo.toml index 549dbb9c02..a6517fb7b0 100644 --- a/crates/catalog/hms/Cargo.toml +++ b/crates/catalog/hms/Cargo.toml @@ -54,9 +54,7 @@ motore-macros = { workspace = true } volo = { workspace = true } [dev-dependencies] -ctor = { workspace = true } iceberg_test_utils = { path = "../../test_utils", features = ["tests"] } -port_scanner = { workspace = true } [package.metadata.cargo-machete] # These dependencies are added to ensure minimal dependency version diff --git a/crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml b/crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml deleted file mode 100644 index 2729f562c4..0000000000 --- a/crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml +++ /dev/null @@ -1,49 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -services: - minio: - image: minio/minio:RELEASE.2025-05-24T17-08-30Z - expose: - - 9000 - - 9001 - environment: - - MINIO_ROOT_USER=admin - - MINIO_ROOT_PASSWORD=password - - MINIO_DOMAIN=minio - command: [ "server", "/data", "--console-address", ":9001" ] - - mc: - depends_on: - - minio - image: minio/mc:RELEASE.2025-05-21T01-59-54Z - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - entrypoint: > - /bin/sh -c " until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " - - hive-metastore: - image: iceberg-hive-metastore - build: ./ - platform: ${DOCKER_DEFAULT_PLATFORM} - expose: - - 9083 - environment: - SERVICE_NAME: "metastore" - SERVICE_OPTS: "-Dmetastore.warehouse.dir=s3a://warehouse/hive/" diff --git a/crates/catalog/hms/tests/hms_catalog_test.rs b/crates/catalog/hms/tests/hms_catalog_test.rs index 9793b7f738..bc036d0c6b 100644 --- a/crates/catalog/hms/tests/hms_catalog_test.rs +++ b/crates/catalog/hms/tests/hms_catalog_test.rs @@ -16,12 +16,12 @@ // under the License. //! Integration tests for hms catalog. +//! +//! These tests assume Docker containers are started externally via `make docker-up`. +//! Each test uses unique namespaces based on module path to avoid conflicts. use std::collections::HashMap; -use std::net::SocketAddr; -use std::sync::RwLock; -use ctor::{ctor, dtor}; use iceberg::io::{S3_ACCESS_KEY_ID, S3_ENDPOINT, S3_REGION, S3_SECRET_ACCESS_KEY}; use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; use iceberg::{Catalog, CatalogBuilder, Namespace, NamespaceIdent, TableCreation, TableIdent}; @@ -29,63 +29,22 @@ use iceberg_catalog_hms::{ HMS_CATALOG_PROP_THRIFT_TRANSPORT, HMS_CATALOG_PROP_URI, HMS_CATALOG_PROP_WAREHOUSE, HmsCatalog, HmsCatalogBuilder, THRIFT_TRANSPORT_BUFFERED, }; -use iceberg_test_utils::docker::DockerCompose; -use iceberg_test_utils::{normalize_test_name, set_up}; -use port_scanner::scan_port_addr; +use iceberg_test_utils::{ + cleanup_namespace, get_hms_endpoint, get_minio_endpoint, normalize_test_name_with_parts, set_up, +}; use tokio::time::sleep; use tracing::info; -const HMS_CATALOG_PORT: u16 = 9083; -const MINIO_PORT: u16 = 9000; -static DOCKER_COMPOSE_ENV: RwLock> = RwLock::new(None); type Result = std::result::Result; -#[ctor] -fn before_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - let docker_compose = DockerCompose::new( - normalize_test_name(module_path!()), - format!("{}/testdata/hms_catalog", env!("CARGO_MANIFEST_DIR")), - ); - docker_compose.up(); - guard.replace(docker_compose); -} - -#[dtor] -fn after_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - guard.take(); -} - async fn get_catalog() -> HmsCatalog { set_up(); - let (hms_catalog_ip, minio_ip) = { - let guard = DOCKER_COMPOSE_ENV.read().unwrap(); - let docker_compose = guard.as_ref().unwrap(); - ( - docker_compose.get_container_ip("hive-metastore"), - docker_compose.get_container_ip("minio"), - ) - }; - let hms_socket_addr = SocketAddr::new(hms_catalog_ip, HMS_CATALOG_PORT); - let minio_socket_addr = SocketAddr::new(minio_ip, MINIO_PORT); - while !scan_port_addr(hms_socket_addr) { - info!("scan hms_socket_addr {} check", hms_socket_addr); - info!("Waiting for 1s hms catalog to ready..."); - sleep(std::time::Duration::from_millis(1000)).await; - } - - while !scan_port_addr(minio_socket_addr) { - info!("Waiting for 1s minio to ready..."); - sleep(std::time::Duration::from_millis(1000)).await; - } + let hms_endpoint = get_hms_endpoint(); + let minio_endpoint = get_minio_endpoint(); let props = HashMap::from([ - ( - HMS_CATALOG_PROP_URI.to_string(), - hms_socket_addr.to_string(), - ), + (HMS_CATALOG_PROP_URI.to_string(), hms_endpoint), ( HMS_CATALOG_PROP_THRIFT_TRANSPORT.to_string(), THRIFT_TRANSPORT_BUFFERED.to_string(), @@ -94,10 +53,7 @@ async fn get_catalog() -> HmsCatalog { HMS_CATALOG_PROP_WAREHOUSE.to_string(), "s3a://warehouse/hive".to_string(), ), - ( - S3_ENDPOINT.to_string(), - format!("http://{minio_socket_addr}"), - ), + (S3_ENDPOINT.to_string(), minio_endpoint), (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), (S3_REGION.to_string(), "us-east-1".to_string()), @@ -157,7 +113,12 @@ fn set_table_creation(location: Option, name: impl ToString) -> Result Result<()> { let catalog = get_catalog().await; let creation: TableCreation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_rename_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_rename_table" + ))); + // Clean up from any previous test runs + cleanup_namespace(&catalog, namespace.name()).await; set_test_namespace(&catalog, namespace.name()).await?; let table: iceberg::table::Table = catalog.create_table(namespace.name(), creation).await?; @@ -177,7 +138,11 @@ async fn test_rename_table() -> Result<()> { async fn test_table_exists() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_table_exists".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_table_exists" + ))); + cleanup_namespace(&catalog, namespace.name()).await; set_test_namespace(&catalog, namespace.name()).await?; let table = catalog.create_table(namespace.name(), creation).await?; @@ -193,7 +158,11 @@ async fn test_table_exists() -> Result<()> { async fn test_drop_table() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_drop_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_drop_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; set_test_namespace(&catalog, namespace.name()).await?; let table = catalog.create_table(namespace.name(), creation).await?; @@ -211,7 +180,11 @@ async fn test_drop_table() -> Result<()> { async fn test_load_table() -> Result<()> { let catalog = get_catalog().await; let creation = set_table_creation(None, "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_load_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_load_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; set_test_namespace(&catalog, namespace.name()).await?; let expected = catalog.create_table(namespace.name(), creation).await?; @@ -235,7 +208,11 @@ async fn test_create_table() -> Result<()> { let catalog = get_catalog().await; // inject custom location, ignore the namespace prefix let creation = set_table_creation(Some("s3a://warehouse/hive".into()), "my_table")?; - let namespace = Namespace::new(NamespaceIdent::new("test_create_table".into())); + // Use unique namespace to avoid conflicts + let namespace = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_create_table" + ))); + cleanup_namespace(&catalog, namespace.name()).await; set_test_namespace(&catalog, namespace.name()).await?; let result = catalog.create_table(namespace.name(), creation).await?; @@ -259,9 +236,14 @@ async fn test_create_table() -> Result<()> { #[tokio::test] async fn test_list_tables() -> Result<()> { let catalog = get_catalog().await; - let ns = Namespace::new(NamespaceIdent::new("test_list_tables".into())); - let result = catalog.list_tables(ns.name()).await?; + // Use unique namespace to avoid conflicts + let ns = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_list_tables" + ))); + // Clean up and create namespace, then verify it's empty + cleanup_namespace(&catalog, ns.name()).await; set_test_namespace(&catalog, ns.name()).await?; + let result = catalog.list_tables(ns.name()).await?; assert_eq!(result, vec![]); @@ -311,10 +293,12 @@ async fn test_create_namespace() -> Result<()> { ("key1".to_string(), "value1".to_string()), ]); + // Use unique namespace to avoid conflicts let ns = Namespace::with_properties( - NamespaceIdent::new("test_create_namespace".into()), + NamespaceIdent::new(normalize_test_name_with_parts!("test_create_namespace")), properties.clone(), ); + cleanup_namespace(&catalog, ns.name()).await; let result = catalog.create_namespace(ns.name(), properties).await?; @@ -355,7 +339,11 @@ async fn test_namespace_exists() -> Result<()> { let catalog = get_catalog().await; let ns_exists = Namespace::new(NamespaceIdent::new("default".into())); - let ns_not_exists = Namespace::new(NamespaceIdent::new("test_namespace_exists".into())); + // Use unique namespace to ensure it doesn't exist + let ns_not_exists = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_namespace_exists" + ))); + cleanup_namespace(&catalog, ns_not_exists.name()).await; let result_exists = catalog.namespace_exists(ns_exists.name()).await?; let result_not_exists = catalog.namespace_exists(ns_not_exists.name()).await?; @@ -370,7 +358,9 @@ async fn test_namespace_exists() -> Result<()> { async fn test_update_namespace() -> Result<()> { let catalog = get_catalog().await; - let ns = NamespaceIdent::new("test_update_namespace".into()); + // Use unique namespace to avoid conflicts + let ns = NamespaceIdent::new(normalize_test_name_with_parts!("test_update_namespace")); + cleanup_namespace(&catalog, &ns).await; set_test_namespace(&catalog, &ns).await?; let properties = HashMap::from([("comment".to_string(), "my_update".to_string())]); @@ -390,7 +380,11 @@ async fn test_update_namespace() -> Result<()> { async fn test_drop_namespace() -> Result<()> { let catalog = get_catalog().await; - let ns = Namespace::new(NamespaceIdent::new("delete_me".into())); + // Use unique namespace to avoid conflicts + let ns = Namespace::new(NamespaceIdent::new(normalize_test_name_with_parts!( + "test_drop_namespace" + ))); + cleanup_namespace(&catalog, ns.name()).await; catalog.create_namespace(ns.name(), HashMap::new()).await?; diff --git a/crates/catalog/rest/Cargo.toml b/crates/catalog/rest/Cargo.toml index 916b5ccf75..de72b6c61b 100644 --- a/crates/catalog/rest/Cargo.toml +++ b/crates/catalog/rest/Cargo.toml @@ -44,8 +44,6 @@ typed-builder = { workspace = true } uuid = { workspace = true, features = ["v4"] } [dev-dependencies] -ctor = { workspace = true } iceberg_test_utils = { path = "../../test_utils", features = ["tests"] } mockito = { workspace = true } -port_scanner = { workspace = true } tokio = { workspace = true } diff --git a/crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml b/crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml deleted file mode 100644 index 29b44b2a23..0000000000 --- a/crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml +++ /dev/null @@ -1,65 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -networks: - rest_bridge: - -services: - rest: - image: apache/iceberg-rest-fixture:1.10.0 - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - - CATALOG_CATALOG__IMPL=org.apache.iceberg.jdbc.JdbcCatalog - - CATALOG_URI=jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory - - CATALOG_WAREHOUSE=s3://icebergdata/demo - - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - - CATALOG_S3_ENDPOINT=http://minio:9000 - depends_on: - - minio - networks: - rest_bridge: - expose: - - 8181 - - minio: - image: minio/minio:RELEASE.2025-05-24T17-08-30Z - environment: - - MINIO_ROOT_USER=admin - - MINIO_ROOT_PASSWORD=password - - MINIO_DOMAIN=minio - hostname: icebergdata.minio - networks: - rest_bridge: - expose: - - 9001 - - 9000 - command: ["server", "/data", "--console-address", ":9001"] - - mc: - depends_on: - - minio - image: minio/mc:RELEASE.2025-05-21T01-59-54Z - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - entrypoint: > - /bin/sh -c " until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/icebergdata; /usr/bin/mc mb minio/icebergdata; /usr/bin/mc policy set public minio/icebergdata; tail -f /dev/null " - networks: - rest_bridge: diff --git a/crates/catalog/rest/tests/rest_catalog_test.rs b/crates/catalog/rest/tests/rest_catalog_test.rs index 59fea0b51f..60c67caab3 100644 --- a/crates/catalog/rest/tests/rest_catalog_test.rs +++ b/crates/catalog/rest/tests/rest_catalog_test.rs @@ -16,64 +16,55 @@ // under the License. //! Integration tests for rest catalog. +//! +//! These tests assume Docker containers are started externally via `make docker-up`. +//! Each test uses unique namespaces based on module path to avoid conflicts. use std::collections::HashMap; -use std::net::SocketAddr; -use std::sync::RwLock; -use ctor::{ctor, dtor}; use iceberg::spec::{FormatVersion, NestedField, PrimitiveType, Schema, Type}; use iceberg::transaction::{ApplyTransactionAction, Transaction}; use iceberg::{Catalog, CatalogBuilder, Namespace, NamespaceIdent, TableCreation, TableIdent}; use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalog, RestCatalogBuilder}; -use iceberg_test_utils::docker::DockerCompose; -use iceberg_test_utils::{normalize_test_name, set_up}; -use port_scanner::scan_port_addr; +use iceberg_test_utils::{ + cleanup_namespace, get_rest_catalog_endpoint, normalize_test_name_with_parts, set_up, +}; use tokio::time::sleep; use tracing::info; -const REST_CATALOG_PORT: u16 = 8181; -static DOCKER_COMPOSE_ENV: RwLock> = RwLock::new(None); - -#[ctor] -fn before_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - let docker_compose = DockerCompose::new( - normalize_test_name(module_path!()), - format!("{}/testdata/rest_catalog", env!("CARGO_MANIFEST_DIR")), - ); - docker_compose.up(); - guard.replace(docker_compose); -} - -#[dtor] -fn after_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - guard.take(); -} - async fn get_catalog() -> RestCatalog { set_up(); - let rest_catalog_ip = { - let guard = DOCKER_COMPOSE_ENV.read().unwrap(); - let docker_compose = guard.as_ref().unwrap(); - docker_compose.get_container_ip("rest") - }; + let rest_endpoint = get_rest_catalog_endpoint(); - let rest_socket_addr = SocketAddr::new(rest_catalog_ip, REST_CATALOG_PORT); - while !scan_port_addr(rest_socket_addr) { - info!("Waiting for 1s rest catalog to ready..."); - sleep(std::time::Duration::from_millis(1000)).await; + // Wait for catalog to be ready + let client = reqwest::Client::new(); + let mut retries = 0; + while retries < 30 { + match client + .get(format!("{rest_endpoint}/v1/config")) + .send() + .await + { + Ok(resp) if resp.status().is_success() => { + info!("REST catalog is ready at {}", rest_endpoint); + break; + } + _ => { + info!( + "Waiting for REST catalog to be ready... (attempt {})", + retries + 1 + ); + sleep(std::time::Duration::from_millis(1000)).await; + retries += 1; + } + } } RestCatalogBuilder::default() .load( "rest", - HashMap::from([( - REST_CATALOG_PROP_URI.to_string(), - format!("http://{rest_socket_addr}"), - )]), + HashMap::from([(REST_CATALOG_PROP_URI.to_string(), rest_endpoint)]), ) .await .unwrap() @@ -83,9 +74,14 @@ async fn get_catalog() -> RestCatalog { async fn test_get_non_exist_namespace() { let catalog = get_catalog().await; - let result = catalog - .get_namespace(&NamespaceIdent::from_strs(["test_get_non_exist_namespace"]).unwrap()) - .await; + // Use unique namespace name to ensure it doesn't exist + let ns_ident = NamespaceIdent::new(normalize_test_name_with_parts!( + "test_get_non_exist_namespace" + )); + // Clean up from any previous test runs + cleanup_namespace(&catalog, &ns_ident).await; + + let result = catalog.get_namespace(&ns_ident).await; assert!(result.is_err()); assert!(result.unwrap_err().to_string().contains("does not exist")); @@ -95,14 +91,23 @@ async fn test_get_non_exist_namespace() { async fn test_get_namespace() { let catalog = get_catalog().await; + // Use unique namespace to avoid conflicts with other tests let ns = Namespace::with_properties( - NamespaceIdent::from_strs(["apple", "ios"]).unwrap(), + NamespaceIdent::from_strs([ + "apple", + "ios", + &normalize_test_name_with_parts!("test_get_namespace"), + ]) + .unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns.name()).await; + // Verify that namespace doesn't exist assert!(catalog.get_namespace(ns.name()).await.is_err()); @@ -125,8 +130,12 @@ async fn test_get_namespace() { async fn test_list_namespace() { let catalog = get_catalog().await; + // Use unique parent namespace to avoid conflicts + let parent_ns_name = normalize_test_name_with_parts!("test_list_namespace"); + let parent_ident = NamespaceIdent::from_strs([&parent_ns_name]).unwrap(); + let ns1 = Namespace::with_properties( - NamespaceIdent::from_strs(["test_list_namespace", "ios"]).unwrap(), + NamespaceIdent::from_strs([&parent_ns_name, "ios"]).unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), @@ -134,22 +143,20 @@ async fn test_list_namespace() { ); let ns2 = Namespace::with_properties( - NamespaceIdent::from_strs(["test_list_namespace", "macos"]).unwrap(), + NamespaceIdent::from_strs([&parent_ns_name, "macos"]).unwrap(), HashMap::from([ ("owner".to_string(), "xuanwo".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns1.name()).await; + cleanup_namespace(&catalog, ns2.name()).await; + cleanup_namespace(&catalog, &parent_ident).await; + // Currently this namespace doesn't exist, so it should return error. - assert!( - catalog - .list_namespaces(Some( - &NamespaceIdent::from_strs(["test_list_namespace"]).unwrap() - )) - .await - .is_err() - ); + assert!(catalog.list_namespaces(Some(&parent_ident)).await.is_err()); // Create namespaces catalog @@ -162,12 +169,7 @@ async fn test_list_namespace() { .unwrap(); // List namespace - let nss = catalog - .list_namespaces(Some( - &NamespaceIdent::from_strs(["test_list_namespace"]).unwrap(), - )) - .await - .unwrap(); + let nss = catalog.list_namespaces(Some(&parent_ident)).await.unwrap(); assert!(nss.contains(ns1.name())); assert!(nss.contains(ns2.name())); @@ -177,14 +179,23 @@ async fn test_list_namespace() { async fn test_list_empty_namespace() { let catalog = get_catalog().await; + // Use unique namespace to avoid conflicts let ns_apple = Namespace::with_properties( - NamespaceIdent::from_strs(["test_list_empty_namespace", "apple"]).unwrap(), + NamespaceIdent::from_strs([ + "list_empty", + "apple", + &normalize_test_name_with_parts!("test_list_empty_namespace"), + ]) + .unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns_apple.name()).await; + // Currently this namespace doesn't exist, so it should return error. assert!( catalog @@ -211,8 +222,12 @@ async fn test_list_empty_namespace() { async fn test_list_root_namespace() { let catalog = get_catalog().await; + // Use unique root namespace to avoid conflicts + let root_ns_name = normalize_test_name_with_parts!("test_list_root_namespace"); + let root_ident = NamespaceIdent::from_strs([&root_ns_name]).unwrap(); + let ns1 = Namespace::with_properties( - NamespaceIdent::from_strs(["test_list_root_namespace", "apple", "ios"]).unwrap(), + NamespaceIdent::from_strs([&root_ns_name, "apple", "ios"]).unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), @@ -220,22 +235,20 @@ async fn test_list_root_namespace() { ); let ns2 = Namespace::with_properties( - NamespaceIdent::from_strs(["test_list_root_namespace", "google", "android"]).unwrap(), + NamespaceIdent::from_strs([&root_ns_name, "google", "android"]).unwrap(), HashMap::from([ ("owner".to_string(), "xuanwo".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns1.name()).await; + cleanup_namespace(&catalog, ns2.name()).await; + cleanup_namespace(&catalog, &root_ident).await; + // Currently this namespace doesn't exist, so it should return error. - assert!( - catalog - .list_namespaces(Some( - &NamespaceIdent::from_strs(["test_list_root_namespace"]).unwrap() - )) - .await - .is_err() - ); + assert!(catalog.list_namespaces(Some(&root_ident)).await.is_err()); // Create namespaces catalog @@ -249,21 +262,31 @@ async fn test_list_root_namespace() { // List namespace let nss = catalog.list_namespaces(None).await.unwrap(); - assert!(nss.contains(&NamespaceIdent::from_strs(["test_list_root_namespace"]).unwrap())); + assert!(nss.contains(&root_ident)); } #[tokio::test] async fn test_create_table() { let catalog = get_catalog().await; + // Use unique namespace to avoid conflicts let ns = Namespace::with_properties( - NamespaceIdent::from_strs(["test_create_table", "apple", "ios"]).unwrap(), + NamespaceIdent::from_strs([ + "create_table", + "apple", + "ios", + &normalize_test_name_with_parts!("test_create_table"), + ]) + .unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns.name()).await; + // Create namespaces catalog .create_namespace(ns.name(), ns.properties().clone()) @@ -311,14 +334,24 @@ async fn test_create_table() { async fn test_update_table() { let catalog = get_catalog().await; + // Use unique namespace to avoid conflicts let ns = Namespace::with_properties( - NamespaceIdent::from_strs(["test_update_table", "apple", "ios"]).unwrap(), + NamespaceIdent::from_strs([ + "update_table", + "apple", + "ios", + &normalize_test_name_with_parts!("test_update_table"), + ]) + .unwrap(), HashMap::from([ ("owner".to_string(), "ray".to_string()), ("community".to_string(), "apache".to_string()), ]), ); + // Clean up from any previous test runs + cleanup_namespace(&catalog, ns.name()).await; + // Create namespaces catalog .create_namespace(ns.name(), ns.properties().clone()) @@ -380,15 +413,24 @@ fn assert_map_contains(map1: &HashMap, map2: &HashMap> = RwLock::new(None); - static FAKE_GCS_PORT: u16 = 4443; static FAKE_GCS_BUCKET: &str = "test-bucket"; - #[ctor] - fn before_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - let docker_compose = DockerCompose::new( - normalize_test_name(module_path!()), - format!("{}/testdata/file_io_gcs", env!("CARGO_MANIFEST_DIR")), - ); - docker_compose.up(); - guard.replace(docker_compose); - } - - #[dtor] - fn after_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - guard.take(); - } - async fn get_file_io_gcs() -> FileIO { set_up(); - let ip = DOCKER_COMPOSE_ENV - .read() - .unwrap() - .as_ref() - .unwrap() - .get_container_ip("gcs-server"); - let addr = SocketAddr::new(ip, FAKE_GCS_PORT); + let gcs_endpoint = get_gcs_endpoint(); // A bucket must exist for FileIO - create_bucket(FAKE_GCS_BUCKET, addr.to_string()) - .await - .unwrap(); + create_bucket(FAKE_GCS_BUCKET, &gcs_endpoint).await.unwrap(); FileIOBuilder::new("gcs") .with_props(vec![ - (GCS_SERVICE_PATH, format!("http://{addr}")), + (GCS_SERVICE_PATH, gcs_endpoint), (GCS_NO_AUTH, "true".to_string()), ]) .build() @@ -76,12 +47,12 @@ mod tests { } // Create a bucket against the emulated GCS storage server. - async fn create_bucket(name: &str, server_addr: String) -> anyhow::Result<()> { + async fn create_bucket(name: &str, server_endpoint: &str) -> anyhow::Result<()> { let mut bucket_data = HashMap::new(); bucket_data.insert("name", name); let client = reqwest::Client::new(); - let endpoint = format!("http://{server_addr}/storage/v1/b"); + let endpoint = format!("{server_endpoint}/storage/v1/b"); client.post(endpoint).json(&bucket_data).send().await?; Ok(()) } diff --git a/crates/iceberg/tests/file_io_s3_test.rs b/crates/iceberg/tests/file_io_s3_test.rs index b044128327..f28538e73e 100644 --- a/crates/iceberg/tests/file_io_s3_test.rs +++ b/crates/iceberg/tests/file_io_s3_test.rs @@ -16,51 +16,30 @@ // under the License. //! Integration tests for FileIO S3. +//! +//! These tests assume Docker containers are started externally via `make docker-up`. +//! Each test uses unique file paths based on module path to avoid conflicts. #[cfg(all(test, feature = "storage-s3"))] mod tests { - use std::net::{IpAddr, SocketAddr}; - use std::sync::{Arc, RwLock}; + use std::sync::Arc; use async_trait::async_trait; - use ctor::{ctor, dtor}; use iceberg::io::{ CustomAwsCredentialLoader, FileIO, FileIOBuilder, S3_ACCESS_KEY_ID, S3_ENDPOINT, S3_REGION, S3_SECRET_ACCESS_KEY, }; - use iceberg_test_utils::docker::DockerCompose; - use iceberg_test_utils::{normalize_test_name, set_up}; + use iceberg_test_utils::{get_minio_endpoint, normalize_test_name_with_parts, set_up}; use reqsign::{AwsCredential, AwsCredentialLoad}; use reqwest::Client; - const MINIO_PORT: u16 = 9000; - static DOCKER_COMPOSE_ENV: RwLock> = RwLock::new(None); - - #[ctor] - fn before_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - let docker_compose = DockerCompose::new( - normalize_test_name(module_path!()), - format!("{}/testdata/file_io_s3", env!("CARGO_MANIFEST_DIR")), - ); - docker_compose.up(); - guard.replace(docker_compose); - } - - #[dtor] - fn after_all() { - let mut guard = DOCKER_COMPOSE_ENV.write().unwrap(); - guard.take(); - } - async fn get_file_io() -> FileIO { set_up(); - let container_ip = get_container_ip("minio"); - let minio_socket_addr = SocketAddr::new(container_ip, MINIO_PORT); + let minio_endpoint = get_minio_endpoint(); FileIOBuilder::new("s3") .with_props(vec![ - (S3_ENDPOINT, format!("http://{minio_socket_addr}")), + (S3_ENDPOINT, minio_endpoint), (S3_ACCESS_KEY_ID, "admin".to_string()), (S3_SECRET_ACCESS_KEY, "password".to_string()), (S3_REGION, "us-east-1".to_string()), @@ -69,12 +48,6 @@ mod tests { .unwrap() } - fn get_container_ip(service_name: &str) -> IpAddr { - let guard = DOCKER_COMPOSE_ENV.read().unwrap(); - let docker_compose = guard.as_ref().unwrap(); - docker_compose.get_container_ip(service_name) - } - #[tokio::test] async fn test_file_io_s3_exists() { let file_io = get_file_io().await; @@ -85,23 +58,35 @@ mod tests { #[tokio::test] async fn test_file_io_s3_output() { let file_io = get_file_io().await; - assert!(!file_io.exists("s3://bucket1/test_output").await.unwrap()); - let output_file = file_io.new_output("s3://bucket1/test_output").unwrap(); + // Use unique file path based on module path to avoid conflicts + let output_path = format!( + "s3://bucket1/{}", + normalize_test_name_with_parts!("test_file_io_s3_output") + ); + // Clean up from any previous test runs + let _ = file_io.delete(&output_path).await; + assert!(!file_io.exists(&output_path).await.unwrap()); + let output_file = file_io.new_output(&output_path).unwrap(); { output_file.write("123".into()).await.unwrap(); } - assert!(file_io.exists("s3://bucket1/test_output").await.unwrap()); + assert!(file_io.exists(&output_path).await.unwrap()); } #[tokio::test] async fn test_file_io_s3_input() { let file_io = get_file_io().await; - let output_file = file_io.new_output("s3://bucket1/test_input").unwrap(); + // Use unique file path based on module path to avoid conflicts + let file_path = format!( + "s3://bucket1/{}", + normalize_test_name_with_parts!("test_file_io_s3_input") + ); + let output_file = file_io.new_output(&file_path).unwrap(); { output_file.write("test_input".into()).await.unwrap(); } - let input_file = file_io.new_input("s3://bucket1/test_input").unwrap(); + let input_file = file_io.new_input(&file_path).unwrap(); { let buffer = input_file.read().await.unwrap(); @@ -200,15 +185,13 @@ mod tests { let mock_loader = MockCredentialLoader::new_minio(); let custom_loader = CustomAwsCredentialLoader::new(Arc::new(mock_loader)); - // Get container info for endpoint - let container_ip = get_container_ip("minio"); - let minio_socket_addr = SocketAddr::new(container_ip, MINIO_PORT); + let minio_endpoint = get_minio_endpoint(); // Build FileIO with custom credential loader let file_io_with_custom_creds = FileIOBuilder::new("s3") .with_extension(custom_loader) .with_props(vec![ - (S3_ENDPOINT, format!("http://{minio_socket_addr}")), + (S3_ENDPOINT, minio_endpoint), (S3_REGION, "us-east-1".to_string()), ]) .build() @@ -229,15 +212,13 @@ mod tests { let mock_loader = MockCredentialLoader::new(None); let custom_loader = CustomAwsCredentialLoader::new(Arc::new(mock_loader)); - // Get container info for endpoint - let container_ip = get_container_ip("minio"); - let minio_socket_addr = SocketAddr::new(container_ip, MINIO_PORT); + let minio_endpoint = get_minio_endpoint(); // Build FileIO with custom credential loader let file_io_with_custom_creds = FileIOBuilder::new("s3") .with_extension(custom_loader) .with_props(vec![ - (S3_ENDPOINT, format!("http://{minio_socket_addr}")), + (S3_ENDPOINT, minio_endpoint), (S3_REGION, "us-east-1".to_string()), ]) .build() diff --git a/crates/integration_tests/Cargo.toml b/crates/integration_tests/Cargo.toml index 07eea5f375..07291f29ef 100644 --- a/crates/integration_tests/Cargo.toml +++ b/crates/integration_tests/Cargo.toml @@ -27,7 +27,6 @@ version = { workspace = true } [dependencies] arrow-array = { workspace = true } arrow-schema = { workspace = true } -ctor = { workspace = true } datafusion = { workspace = true } futures = { workspace = true } iceberg = { workspace = true } diff --git a/crates/integration_tests/src/lib.rs b/crates/integration_tests/src/lib.rs index 44f6c30240..e99c74df6b 100644 --- a/crates/integration_tests/src/lib.rs +++ b/crates/integration_tests/src/lib.rs @@ -16,19 +16,61 @@ // under the License. use std::collections::HashMap; +use std::sync::OnceLock; use iceberg::io::{S3_ACCESS_KEY_ID, S3_ENDPOINT, S3_REGION, S3_SECRET_ACCESS_KEY}; use iceberg_catalog_rest::REST_CATALOG_PROP_URI; use iceberg_test_utils::docker::DockerCompose; -use iceberg_test_utils::{normalize_test_name, set_up}; +use iceberg_test_utils::{ + get_minio_endpoint, get_rest_catalog_endpoint, normalize_test_name, set_up, +}; const REST_CATALOG_PORT: u16 = 8181; +/// Test fixture that manages Docker containers. +/// This is kept for backward compatibility but deprecated in favor of GlobalTestFixture. pub struct TestFixture { pub _docker_compose: DockerCompose, pub catalog_config: HashMap, } +/// Global test fixture that uses environment-based configuration. +/// This assumes Docker containers are started externally (e.g., via `make docker-up`). +pub struct GlobalTestFixture { + pub catalog_config: HashMap, +} + +static GLOBAL_FIXTURE: OnceLock = OnceLock::new(); + +impl GlobalTestFixture { + /// Creates a new GlobalTestFixture from environment variables. + /// Uses default localhost endpoints if environment variables are not set. + pub fn from_env() -> Self { + set_up(); + + let rest_endpoint = get_rest_catalog_endpoint(); + let minio_endpoint = get_minio_endpoint(); + + let catalog_config = HashMap::from([ + (REST_CATALOG_PROP_URI.to_string(), rest_endpoint), + (S3_ENDPOINT.to_string(), minio_endpoint), + (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), + (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), + (S3_REGION.to_string(), "us-east-1".to_string()), + ]); + + GlobalTestFixture { catalog_config } + } +} + +/// Returns a reference to the global test fixture. +/// This fixture assumes Docker containers are started externally. +pub fn get_test_fixture() -> &'static GlobalTestFixture { + GLOBAL_FIXTURE.get_or_init(GlobalTestFixture::from_env) +} + +/// Legacy function to create a test fixture with Docker container management. +/// Deprecated: prefer using `get_test_fixture()` with externally managed containers. pub fn set_test_fixture(func: &str) -> TestFixture { set_up(); let docker_compose = DockerCompose::new( diff --git a/crates/integration_tests/testdata/docker-compose.yaml b/crates/integration_tests/testdata/docker-compose.yaml deleted file mode 100644 index 30fc3914f8..0000000000 --- a/crates/integration_tests/testdata/docker-compose.yaml +++ /dev/null @@ -1,108 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -networks: - rest_bridge: - -services: - rest: - image: apache/iceberg-rest-fixture:1.10.0 - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - - CATALOG_CATALOG__IMPL=org.apache.iceberg.jdbc.JdbcCatalog - - CATALOG_URI=jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory - - CATALOG_WAREHOUSE=s3://icebergdata/demo - - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - - CATALOG_S3_ENDPOINT=http://minio:9000 - depends_on: - - minio - networks: - rest_bridge: - ports: - - 8181:8181 - expose: - - 8181 - - minio: - image: minio/minio:RELEASE.2025-05-24T17-08-30Z - environment: - - MINIO_ROOT_USER=admin - - MINIO_ROOT_PASSWORD=password - - MINIO_DOMAIN=minio - - MINIO_DEFAULT_BUCKETS=icebergdata - hostname: icebergdata.minio - networks: - rest_bridge: - ports: - - 9001:9001 - expose: - - 9001 - - 9000 - command: ["server", "/data", "--console-address", ":9001"] - - mc: - depends_on: - - minio - image: minio/mc:RELEASE.2025-05-21T01-59-54Z - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - entrypoint: > - /bin/sh -c " until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/icebergdata; /usr/bin/mc mb minio/icebergdata; /usr/bin/mc policy set public minio/icebergdata; tail -f /dev/null " - networks: - rest_bridge: - - spark-iceberg: - build: spark/ - networks: - rest_bridge: - depends_on: - - rest - - minio - environment: - - AWS_ACCESS_KEY_ID=admin - - AWS_SECRET_ACCESS_KEY=password - - AWS_REGION=us-east-1 - ports: - - 15002:15002 # Spark Connect - - 4040:4040 # Spark UI - healthcheck: - test: ["CMD", "sh", "-c", "netstat -an | grep 15002 | grep LISTEN"] - interval: 30s - timeout: 10s - retries: 5 - start_period: 90s - - provision: - image: python:3.12-slim - networks: - rest_bridge: - depends_on: - spark-iceberg: - condition: service_healthy - entrypoint: ["/bin/sh", "-c", "pip install -q 'pyspark[connect]==4.0.1' && python3 /opt/spark/provision.py && touch /tmp/provision_complete && tail -f /dev/null"] - volumes: - - ./spark/provision.py:/opt/spark/provision.py:ro - healthcheck: - test: ["CMD-SHELL", "[ -f /tmp/provision_complete ]"] - interval: 2s - timeout: 2s - retries: 90 - start_period: 20s diff --git a/crates/integration_tests/tests/shared.rs b/crates/integration_tests/tests/shared.rs index 6bdddaa6cf..ab85e4d1ae 100644 --- a/crates/integration_tests/tests/shared.rs +++ b/crates/integration_tests/tests/shared.rs @@ -15,22 +15,12 @@ // specific language governing permissions and limitations // under the License. -use std::sync::{Arc, OnceLock}; - -use ctor::dtor; -use iceberg_integration_tests::{TestFixture, set_test_fixture}; +use iceberg_integration_tests::{GlobalTestFixture, get_test_fixture}; pub mod shared_tests; -static DOCKER_CONTAINERS: OnceLock> = OnceLock::new(); - -pub fn get_shared_containers() -> &'static Arc { - DOCKER_CONTAINERS.get_or_init(|| Arc::new(set_test_fixture("shared_tests"))) -} - -#[dtor] -fn shutdown() { - if let Some(fixture) = DOCKER_CONTAINERS.get() { - fixture._docker_compose.down() - } +/// Returns a reference to the shared test fixture. +/// This assumes Docker containers are started externally via `make docker-up`. +pub fn get_shared_containers() -> &'static GlobalTestFixture { + get_test_fixture() } diff --git a/crates/integration_tests/tests/shared_tests/mod.rs b/crates/integration_tests/tests/shared_tests/mod.rs index 065b14d5da..1e695c5884 100644 --- a/crates/integration_tests/tests/shared_tests/mod.rs +++ b/crates/integration_tests/tests/shared_tests/mod.rs @@ -54,7 +54,7 @@ pub async fn random_ns() -> Namespace { ns } -fn test_schema() -> Schema { +pub fn test_schema() -> Schema { Schema::builder() .with_schema_id(1) .with_identifier_field_ids(vec![2]) diff --git a/crates/test_utils/Cargo.toml b/crates/test_utils/Cargo.toml index 9a486b15ed..58f9307790 100644 --- a/crates/test_utils/Cargo.toml +++ b/crates/test_utils/Cargo.toml @@ -26,6 +26,7 @@ license = { workspace = true } repository = { workspace = true } [dependencies] +iceberg = { workspace = true } tracing = { workspace = true } tracing-subscriber = { workspace = true } diff --git a/crates/test_utils/src/lib.rs b/crates/test_utils/src/lib.rs index 6d1104a1b9..54ec54193f 100644 --- a/crates/test_utils/src/lib.rs +++ b/crates/test_utils/src/lib.rs @@ -31,6 +31,8 @@ pub use common::*; mod common { use std::sync::Once; + use iceberg::{Catalog, NamespaceIdent}; + static INIT: Once = Once::new(); pub fn set_up() { INIT.call_once(tracing_subscriber::fmt::init); @@ -38,4 +40,83 @@ mod common { pub fn normalize_test_name(s: impl ToString) -> String { s.to_string().replace("::", "__").replace('.', "_") } + + // Environment variable names for service endpoints + pub const ENV_MINIO_ENDPOINT: &str = "ICEBERG_TEST_MINIO_ENDPOINT"; + pub const ENV_REST_CATALOG_ENDPOINT: &str = "ICEBERG_TEST_REST_ENDPOINT"; + pub const ENV_HMS_ENDPOINT: &str = "ICEBERG_TEST_HMS_ENDPOINT"; + pub const ENV_GLUE_ENDPOINT: &str = "ICEBERG_TEST_GLUE_ENDPOINT"; + pub const ENV_GCS_ENDPOINT: &str = "ICEBERG_TEST_GCS_ENDPOINT"; + + // Default ports matching dev/docker-compose.yaml + pub const DEFAULT_MINIO_PORT: u16 = 9000; + pub const DEFAULT_REST_CATALOG_PORT: u16 = 8181; + pub const DEFAULT_HMS_PORT: u16 = 9083; + pub const DEFAULT_GLUE_PORT: u16 = 5000; + pub const DEFAULT_GCS_PORT: u16 = 4443; + + /// Returns the MinIO S3-compatible endpoint. + /// Checks ICEBERG_TEST_MINIO_ENDPOINT env var, otherwise returns localhost default. + pub fn get_minio_endpoint() -> String { + std::env::var(ENV_MINIO_ENDPOINT) + .unwrap_or_else(|_| format!("http://localhost:{DEFAULT_MINIO_PORT}")) + } + + /// Returns the REST catalog endpoint. + /// Checks ICEBERG_TEST_REST_ENDPOINT env var, otherwise returns localhost default. + pub fn get_rest_catalog_endpoint() -> String { + std::env::var(ENV_REST_CATALOG_ENDPOINT) + .unwrap_or_else(|_| format!("http://localhost:{DEFAULT_REST_CATALOG_PORT}")) + } + + /// Returns the HMS (Hive Metastore) endpoint. + /// Checks ICEBERG_TEST_HMS_ENDPOINT env var, otherwise returns localhost default. + pub fn get_hms_endpoint() -> String { + std::env::var(ENV_HMS_ENDPOINT).unwrap_or_else(|_| format!("localhost:{DEFAULT_HMS_PORT}")) + } + + /// Returns the Glue (Moto mock) endpoint. + /// Checks ICEBERG_TEST_GLUE_ENDPOINT env var, otherwise returns localhost default. + pub fn get_glue_endpoint() -> String { + std::env::var(ENV_GLUE_ENDPOINT) + .unwrap_or_else(|_| format!("http://localhost:{DEFAULT_GLUE_PORT}")) + } + + /// Returns the GCS (fake-gcs-server) endpoint. + /// Checks ICEBERG_TEST_GCS_ENDPOINT env var, otherwise returns localhost default. + pub fn get_gcs_endpoint() -> String { + std::env::var(ENV_GCS_ENDPOINT) + .unwrap_or_else(|_| format!("http://localhost:{DEFAULT_GCS_PORT}")) + } + + /// Helper to clean up a namespace and its tables before a test runs. + /// This handles the case where previous test runs left data in the persistent database. + pub async fn cleanup_namespace(catalog: &C, ns: &NamespaceIdent) { + // Try to drop all tables in the namespace first + if let Ok(tables) = catalog.list_tables(ns).await { + for table in tables { + let _ = catalog.drop_table(&table).await; + } + } + // Then try to drop the namespace itself + let _ = catalog.drop_namespace(ns).await; + } + + /// Macro to generate a normalized test name with module path prefix. + /// Takes one or more string parts and joins them with the module path. + /// + /// Example: + /// ```ignore + /// // Returns something like "rest_catalog_test__test_create_table" + /// let name = normalize_test_name_with_parts!("test_create_table"); + /// + /// // Returns something like "rest_catalog_test__apple__ios" + /// let name = normalize_test_name_with_parts!("apple", "ios"); + /// ``` + #[macro_export] + macro_rules! normalize_test_name_with_parts { + ($($part:expr),+) => { + $crate::normalize_test_name([module_path!(), $($part),+].join("_")) + }; + } } diff --git a/dev/docker-compose.yaml b/dev/docker-compose.yaml new file mode 100644 index 0000000000..0df017a75f --- /dev/null +++ b/dev/docker-compose.yaml @@ -0,0 +1,202 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Unified Docker Compose for all integration tests. +# This file combines services needed for REST, HMS, Glue, S3, and GCS tests. +# Start once with: docker compose -f dev/docker-compose.yaml up -d --wait +# Stop with: docker compose -f dev/docker-compose.yaml down -v + +networks: + iceberg_test: + +services: + # ============================================================================= + # MinIO - S3-compatible storage (shared by all tests) + # ============================================================================= + minio: + image: minio/minio:RELEASE.2025-05-24T17-08-30Z + environment: + - MINIO_ROOT_USER=admin + - MINIO_ROOT_PASSWORD=password + - MINIO_DOMAIN=minio + hostname: minio + networks: + iceberg_test: + # Add aliases for virtual-hosted style bucket access + aliases: + - icebergdata.minio + - warehouse.minio + - bucket1.minio + ports: + - "9000:9000" + - "9001:9001" + command: ["server", "/data", "--console-address", ":9001"] + healthcheck: + test: ["CMD", "mc", "ready", "local"] + interval: 5s + timeout: 5s + retries: 5 + + # MinIO client - creates buckets for tests + mc: + depends_on: + minio: + condition: service_healthy + image: minio/mc:RELEASE.2025-05-21T01-59-54Z + environment: + - AWS_ACCESS_KEY_ID=admin + - AWS_SECRET_ACCESS_KEY=password + - AWS_REGION=us-east-1 + entrypoint: > + /bin/sh -c " + /usr/bin/mc alias set minio http://minio:9000 admin password; + /usr/bin/mc mb --ignore-existing minio/icebergdata; + /usr/bin/mc mb --ignore-existing minio/warehouse; + /usr/bin/mc mb --ignore-existing minio/bucket1; + /usr/bin/mc policy set public minio/icebergdata; + /usr/bin/mc policy set public minio/warehouse; + /usr/bin/mc policy set public minio/bucket1; + echo 'Buckets created successfully'; + tail -f /dev/null + " + networks: + iceberg_test: + + # ============================================================================= + # REST Catalog - Apache Iceberg REST Catalog + # ============================================================================= + rest: + image: apache/iceberg-rest-fixture:1.10.0 + environment: + - AWS_ACCESS_KEY_ID=admin + - AWS_SECRET_ACCESS_KEY=password + - AWS_REGION=us-east-1 + - CATALOG_CATALOG__IMPL=org.apache.iceberg.jdbc.JdbcCatalog + - CATALOG_URI=jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory + - CATALOG_WAREHOUSE=s3://icebergdata/demo + - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO + - CATALOG_S3_ENDPOINT=http://minio:9000 + depends_on: + minio: + condition: service_healthy + networks: + iceberg_test: + ports: + - "8181:8181" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8181/v1/config"] + interval: 5s + timeout: 5s + retries: 10 + start_period: 10s + + # ============================================================================= + # Hive Metastore - HMS Catalog + # ============================================================================= + hive-metastore: + build: + context: ./hms + dockerfile: Dockerfile + platform: ${DOCKER_DEFAULT_PLATFORM:-linux/amd64} + depends_on: + minio: + condition: service_healthy + networks: + iceberg_test: + ports: + - "9083:9083" + environment: + SERVICE_NAME: "metastore" + SERVICE_OPTS: "-Dmetastore.warehouse.dir=s3a://warehouse/hive/" + healthcheck: + test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/localhost/9083"] + interval: 5s + timeout: 5s + retries: 20 + start_period: 30s + + # ============================================================================= + # Moto - AWS Glue Mock for Glue Catalog tests + # ============================================================================= + moto: + image: motoserver/moto:5.0.3 + networks: + iceberg_test: + ports: + - "5000:5000" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:5000/moto-api/"] + interval: 5s + timeout: 5s + retries: 5 + + # ============================================================================= + # Fake GCS Server - GCS emulator for GCS tests + # ============================================================================= + gcs-server: + image: fsouza/fake-gcs-server@sha256:36b0116fae5236e8def76ccb07761a9ca323e476f366a5f4bf449cac19deaf2d + networks: + iceberg_test: + ports: + - "4443:4443" + command: --scheme http + + # ============================================================================= + # Spark - Spark with Iceberg for integration tests + # ============================================================================= + spark-iceberg: + build: + context: ./spark + dockerfile: Dockerfile + networks: + iceberg_test: + depends_on: + rest: + condition: service_healthy + minio: + condition: service_healthy + environment: + - AWS_ACCESS_KEY_ID=admin + - AWS_SECRET_ACCESS_KEY=password + - AWS_REGION=us-east-1 + ports: + - "15002:15002" + - "4040:4040" + healthcheck: + test: ["CMD", "sh", "-c", "netstat -an | grep 15002 | grep LISTEN"] + interval: 30s + timeout: 10s + retries: 5 + start_period: 90s + + # Provision service - creates test data via Spark + provision: + image: python:3.12-slim + networks: + iceberg_test: + depends_on: + spark-iceberg: + condition: service_healthy + entrypoint: ["/bin/sh", "-c", "pip install -q 'pyspark[connect]==4.0.1' && python3 /opt/spark/provision.py && touch /tmp/provision_complete && tail -f /dev/null"] + volumes: + - ./spark/provision.py:/opt/spark/provision.py:ro + healthcheck: + test: ["CMD-SHELL", "[ -f /tmp/provision_complete ]"] + interval: 2s + timeout: 2s + retries: 90 + start_period: 20s diff --git a/crates/catalog/hms/testdata/hms_catalog/Dockerfile b/dev/hms/Dockerfile similarity index 100% rename from crates/catalog/hms/testdata/hms_catalog/Dockerfile rename to dev/hms/Dockerfile diff --git a/crates/catalog/hms/testdata/hms_catalog/core-site.xml b/dev/hms/core-site.xml similarity index 96% rename from crates/catalog/hms/testdata/hms_catalog/core-site.xml rename to dev/hms/core-site.xml index f0583a0bc0..f23efa8885 100644 --- a/crates/catalog/hms/testdata/hms_catalog/core-site.xml +++ b/dev/hms/core-site.xml @@ -1,51 +1,51 @@ - - - - - fs.defaultFS - s3a://warehouse/hive - - - fs.s3a.impl - org.apache.hadoop.fs.s3a.S3AFileSystem - - - fs.s3a.fast.upload - true - - - fs.s3a.endpoint - http://minio:9000 - - - fs.s3a.access.key - admin - - - fs.s3a.secret.key - password - - - fs.s3a.connection.ssl.enabled - false - - - fs.s3a.path.style.access - true - - \ No newline at end of file + + + + + fs.defaultFS + s3a://warehouse/hive + + + fs.s3a.impl + org.apache.hadoop.fs.s3a.S3AFileSystem + + + fs.s3a.fast.upload + true + + + fs.s3a.endpoint + http://minio:9000 + + + fs.s3a.access.key + admin + + + fs.s3a.secret.key + password + + + fs.s3a.connection.ssl.enabled + false + + + fs.s3a.path.style.access + true + + diff --git a/crates/integration_tests/testdata/spark/Dockerfile b/dev/spark/Dockerfile similarity index 100% rename from crates/integration_tests/testdata/spark/Dockerfile rename to dev/spark/Dockerfile diff --git a/crates/integration_tests/testdata/spark/provision.py b/dev/spark/provision.py old mode 100755 new mode 100644 similarity index 100% rename from crates/integration_tests/testdata/spark/provision.py rename to dev/spark/provision.py diff --git a/crates/integration_tests/testdata/spark/spark-defaults.conf b/dev/spark/spark-defaults.conf similarity index 100% rename from crates/integration_tests/testdata/spark/spark-defaults.conf rename to dev/spark/spark-defaults.conf From 04d90bfd3af1db3890696ae8b88102d08562a585 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 27 Jan 2026 17:42:47 -0800 Subject: [PATCH 174/891] feat(io): Add specific storage configs (#2072) ## Which issue does this PR close? - Closes #2055 ## What changes are included in this PR? - Added specific storage configs: `S3Config`, `GcsConfig`, `AzdlsConfig`, `OssConfig` - Moved config keys from files like `storage_s3.rs` to `config/s3.rs` ## Are these changes tested? Added uts to test `TryFrom` --------- Co-authored-by: Renjie Liu --- crates/iceberg/src/io/config/azdls.rs | 180 +++++++++++++++ crates/iceberg/src/io/config/gcs.rs | 189 +++++++++++++++ crates/iceberg/src/io/config/mod.rs | 18 +- crates/iceberg/src/io/config/oss.rs | 124 ++++++++++ crates/iceberg/src/io/config/s3.rs | 303 +++++++++++++++++++++++++ crates/iceberg/src/io/mod.rs | 10 +- crates/iceberg/src/io/storage_azdls.rs | 33 +-- crates/iceberg/src/io/storage_gcs.rs | 28 +-- crates/iceberg/src/io/storage_oss.rs | 12 +- crates/iceberg/src/io/storage_s3.rs | 48 +--- 10 files changed, 824 insertions(+), 121 deletions(-) create mode 100644 crates/iceberg/src/io/config/azdls.rs create mode 100644 crates/iceberg/src/io/config/gcs.rs create mode 100644 crates/iceberg/src/io/config/oss.rs create mode 100644 crates/iceberg/src/io/config/s3.rs diff --git a/crates/iceberg/src/io/config/azdls.rs b/crates/iceberg/src/io/config/azdls.rs new file mode 100644 index 0000000000..059012942d --- /dev/null +++ b/crates/iceberg/src/io/config/azdls.rs @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Azure Data Lake Storage configuration. +//! +//! This module provides configuration constants and types for Azure Data Lake Storage. + +use serde::{Deserialize, Serialize}; +use typed_builder::TypedBuilder; + +use super::StorageConfig; +use crate::Result; + +/// A connection string. +/// +/// Note, this string is parsed first, and any other passed adls.* properties +/// will override values from the connection string. +pub const ADLS_CONNECTION_STRING: &str = "adls.connection-string"; +/// The account that you want to connect to. +pub const ADLS_ACCOUNT_NAME: &str = "adls.account-name"; +/// The key to authentication against the account. +pub const ADLS_ACCOUNT_KEY: &str = "adls.account-key"; +/// The shared access signature. +pub const ADLS_SAS_TOKEN: &str = "adls.sas-token"; +/// The tenant-id. +pub const ADLS_TENANT_ID: &str = "adls.tenant-id"; +/// The client-id. +pub const ADLS_CLIENT_ID: &str = "adls.client-id"; +/// The client-secret. +pub const ADLS_CLIENT_SECRET: &str = "adls.client-secret"; +/// The authority host of the service principal. +/// - required for client_credentials authentication +/// - default value: `https://login.microsoftonline.com` +pub const ADLS_AUTHORITY_HOST: &str = "adls.authority-host"; + +/// Azure Data Lake Storage configuration. +/// +/// This struct contains all the configuration options for connecting to Azure Data Lake Storage. +/// Use the builder pattern via `AzdlsConfig::builder()` to construct instances. +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)] +pub struct AzdlsConfig { + /// Connection string. + #[builder(default, setter(strip_option, into))] + pub connection_string: Option, + /// Account name. + #[builder(default, setter(strip_option, into))] + pub account_name: Option, + /// Account key. + #[builder(default, setter(strip_option, into))] + pub account_key: Option, + /// SAS token. + #[builder(default, setter(strip_option, into))] + pub sas_token: Option, + /// Tenant ID. + #[builder(default, setter(strip_option, into))] + pub tenant_id: Option, + /// Client ID. + #[builder(default, setter(strip_option, into))] + pub client_id: Option, + /// Client secret. + #[builder(default, setter(strip_option, into))] + pub client_secret: Option, + /// Authority host. + #[builder(default, setter(strip_option, into))] + pub authority_host: Option, + /// Endpoint URL. + #[builder(default, setter(strip_option, into))] + pub endpoint: Option, + /// Filesystem name. + #[builder(default, setter(into))] + pub filesystem: String, +} + +impl TryFrom<&StorageConfig> for AzdlsConfig { + type Error = crate::Error; + + fn try_from(config: &StorageConfig) -> Result { + let props = config.props(); + + let mut cfg = AzdlsConfig::default(); + + if let Some(connection_string) = props.get(ADLS_CONNECTION_STRING) { + cfg.connection_string = Some(connection_string.clone()); + } + if let Some(account_name) = props.get(ADLS_ACCOUNT_NAME) { + cfg.account_name = Some(account_name.clone()); + } + if let Some(account_key) = props.get(ADLS_ACCOUNT_KEY) { + cfg.account_key = Some(account_key.clone()); + } + if let Some(sas_token) = props.get(ADLS_SAS_TOKEN) { + cfg.sas_token = Some(sas_token.clone()); + } + if let Some(tenant_id) = props.get(ADLS_TENANT_ID) { + cfg.tenant_id = Some(tenant_id.clone()); + } + if let Some(client_id) = props.get(ADLS_CLIENT_ID) { + cfg.client_id = Some(client_id.clone()); + } + if let Some(client_secret) = props.get(ADLS_CLIENT_SECRET) { + cfg.client_secret = Some(client_secret.clone()); + } + if let Some(authority_host) = props.get(ADLS_AUTHORITY_HOST) { + cfg.authority_host = Some(authority_host.clone()); + } + + Ok(cfg) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_azdls_config_builder() { + let config = AzdlsConfig::builder() + .account_name("myaccount") + .account_key("my-account-key") + .build(); + + assert_eq!(config.account_name.as_deref(), Some("myaccount")); + assert_eq!(config.account_key.as_deref(), Some("my-account-key")); + } + + #[test] + fn test_azdls_config_from_storage_config() { + let storage_config = StorageConfig::new() + .with_prop(ADLS_ACCOUNT_NAME, "myaccount") + .with_prop(ADLS_ACCOUNT_KEY, "my-account-key"); + + let azdls_config = AzdlsConfig::try_from(&storage_config).unwrap(); + + assert_eq!(azdls_config.account_name.as_deref(), Some("myaccount")); + assert_eq!(azdls_config.account_key.as_deref(), Some("my-account-key")); + } + + #[test] + fn test_azdls_config_with_sas_token() { + let storage_config = StorageConfig::new() + .with_prop(ADLS_ACCOUNT_NAME, "myaccount") + .with_prop(ADLS_SAS_TOKEN, "my-sas-token"); + + let azdls_config = AzdlsConfig::try_from(&storage_config).unwrap(); + + assert_eq!(azdls_config.account_name.as_deref(), Some("myaccount")); + assert_eq!(azdls_config.sas_token.as_deref(), Some("my-sas-token")); + } + + #[test] + fn test_azdls_config_with_client_credentials() { + let storage_config = StorageConfig::new() + .with_prop(ADLS_ACCOUNT_NAME, "myaccount") + .with_prop(ADLS_TENANT_ID, "my-tenant") + .with_prop(ADLS_CLIENT_ID, "my-client") + .with_prop(ADLS_CLIENT_SECRET, "my-secret"); + + let azdls_config = AzdlsConfig::try_from(&storage_config).unwrap(); + + assert_eq!(azdls_config.account_name.as_deref(), Some("myaccount")); + assert_eq!(azdls_config.tenant_id.as_deref(), Some("my-tenant")); + assert_eq!(azdls_config.client_id.as_deref(), Some("my-client")); + assert_eq!(azdls_config.client_secret.as_deref(), Some("my-secret")); + } +} diff --git a/crates/iceberg/src/io/config/gcs.rs b/crates/iceberg/src/io/config/gcs.rs new file mode 100644 index 0000000000..5b11567f4d --- /dev/null +++ b/crates/iceberg/src/io/config/gcs.rs @@ -0,0 +1,189 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Google Cloud Storage configuration. +//! +//! This module provides configuration constants and types for Google Cloud Storage. +//! Reference: https://github.com/apache/iceberg/blob/main/gcp/src/main/java/org/apache/iceberg/gcp/GCPProperties.java + +use serde::{Deserialize, Serialize}; +use typed_builder::TypedBuilder; + +use super::StorageConfig; +use crate::Result; +use crate::io::is_truthy; + +/// Google Cloud Project ID. +pub const GCS_PROJECT_ID: &str = "gcs.project-id"; +/// Google Cloud Storage endpoint. +pub const GCS_SERVICE_PATH: &str = "gcs.service.path"; +/// Google Cloud user project. +pub const GCS_USER_PROJECT: &str = "gcs.user-project"; +/// Allow unauthenticated requests. +pub const GCS_NO_AUTH: &str = "gcs.no-auth"; +/// Google Cloud Storage credentials JSON string, base64 encoded. +/// +/// E.g. base64::prelude::BASE64_STANDARD.encode(serde_json::to_string(credential).as_bytes()) +pub const GCS_CREDENTIALS_JSON: &str = "gcs.credentials-json"; +/// Google Cloud Storage token. +pub const GCS_TOKEN: &str = "gcs.oauth2.token"; +/// Option to skip signing requests (e.g. for public buckets/folders). +pub const GCS_ALLOW_ANONYMOUS: &str = "gcs.allow-anonymous"; +/// Option to skip loading the credential from GCE metadata server. +pub const GCS_DISABLE_VM_METADATA: &str = "gcs.disable-vm-metadata"; +/// Option to skip loading configuration from config file and the env. +pub const GCS_DISABLE_CONFIG_LOAD: &str = "gcs.disable-config-load"; + +/// Google Cloud Storage configuration. +/// +/// This struct contains all the configuration options for connecting to Google Cloud Storage. +/// Use the builder pattern via `GcsConfig::builder()` to construct instances. +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)] +pub struct GcsConfig { + /// Google Cloud Project ID. + #[builder(default, setter(strip_option, into))] + pub project_id: Option, + /// GCS service endpoint. + #[builder(default, setter(strip_option, into))] + pub endpoint: Option, + /// User project for requester pays buckets. + #[builder(default, setter(strip_option, into))] + pub user_project: Option, + /// Credentials JSON (base64 encoded). + #[builder(default, setter(strip_option, into))] + pub credential: Option, + /// OAuth2 token. + #[builder(default, setter(strip_option, into))] + pub token: Option, + /// Allow anonymous access. + #[builder(default)] + pub allow_anonymous: bool, + /// Disable VM metadata. + #[builder(default)] + pub disable_vm_metadata: bool, + /// Disable config load. + #[builder(default)] + pub disable_config_load: bool, +} + +impl TryFrom<&StorageConfig> for GcsConfig { + type Error = crate::Error; + + fn try_from(config: &StorageConfig) -> Result { + let props = config.props(); + + let mut cfg = GcsConfig::default(); + + if let Some(project_id) = props.get(GCS_PROJECT_ID) { + cfg.project_id = Some(project_id.clone()); + } + if let Some(endpoint) = props.get(GCS_SERVICE_PATH) { + cfg.endpoint = Some(endpoint.clone()); + } + if let Some(user_project) = props.get(GCS_USER_PROJECT) { + cfg.user_project = Some(user_project.clone()); + } + if let Some(credential) = props.get(GCS_CREDENTIALS_JSON) { + cfg.credential = Some(credential.clone()); + } + if let Some(token) = props.get(GCS_TOKEN) { + cfg.token = Some(token.clone()); + } + + // GCS_NO_AUTH enables all anonymous/no-auth options + if props.get(GCS_NO_AUTH).is_some() { + cfg.allow_anonymous = true; + cfg.disable_vm_metadata = true; + cfg.disable_config_load = true; + } + + if let Some(allow_anonymous) = props.get(GCS_ALLOW_ANONYMOUS) + && is_truthy(allow_anonymous.to_lowercase().as_str()) + { + cfg.allow_anonymous = true; + } + if let Some(disable_vm_metadata) = props.get(GCS_DISABLE_VM_METADATA) + && is_truthy(disable_vm_metadata.to_lowercase().as_str()) + { + cfg.disable_vm_metadata = true; + } + if let Some(disable_config_load) = props.get(GCS_DISABLE_CONFIG_LOAD) + && is_truthy(disable_config_load.to_lowercase().as_str()) + { + cfg.disable_config_load = true; + } + + Ok(cfg) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_gcs_config_builder() { + let config = GcsConfig::builder() + .project_id("my-project") + .credential("base64-creds") + .endpoint("http://localhost:4443") + .build(); + + assert_eq!(config.project_id.as_deref(), Some("my-project")); + assert_eq!(config.credential.as_deref(), Some("base64-creds")); + assert_eq!(config.endpoint.as_deref(), Some("http://localhost:4443")); + } + + #[test] + fn test_gcs_config_from_storage_config() { + let storage_config = StorageConfig::new() + .with_prop(GCS_PROJECT_ID, "my-project") + .with_prop(GCS_CREDENTIALS_JSON, "base64-creds") + .with_prop(GCS_SERVICE_PATH, "http://localhost:4443"); + + let gcs_config = GcsConfig::try_from(&storage_config).unwrap(); + + assert_eq!(gcs_config.project_id.as_deref(), Some("my-project")); + assert_eq!(gcs_config.credential.as_deref(), Some("base64-creds")); + assert_eq!( + gcs_config.endpoint.as_deref(), + Some("http://localhost:4443") + ); + } + + #[test] + fn test_gcs_config_no_auth() { + let storage_config = StorageConfig::new().with_prop(GCS_NO_AUTH, "true"); + + let gcs_config = GcsConfig::try_from(&storage_config).unwrap(); + + assert!(gcs_config.allow_anonymous); + assert!(gcs_config.disable_vm_metadata); + assert!(gcs_config.disable_config_load); + } + + #[test] + fn test_gcs_config_allow_anonymous() { + let storage_config = StorageConfig::new().with_prop(GCS_ALLOW_ANONYMOUS, "true"); + + let gcs_config = GcsConfig::try_from(&storage_config).unwrap(); + + assert!(gcs_config.allow_anonymous); + assert!(!gcs_config.disable_vm_metadata); + } +} diff --git a/crates/iceberg/src/io/config/mod.rs b/crates/iceberg/src/io/config/mod.rs index 648e8baf47..cbdb537303 100644 --- a/crates/iceberg/src/io/config/mod.rs +++ b/crates/iceberg/src/io/config/mod.rs @@ -30,19 +30,17 @@ //! - [`OssConfig`]: Alibaba Cloud OSS specific configuration //! - [`AzdlsConfig`]: Azure Data Lake Storage specific configuration -// TODO Add specific configs -// mod azdls; -// mod gcs; -// mod oss; -// mod s3; +mod azdls; +mod gcs; +mod oss; +mod s3; use std::collections::HashMap; -// TODO Add specific configs -// pub use azdls::*; -// pub use gcs::*; -// pub use oss::*; -// pub use s3::*; +pub use azdls::*; +pub use gcs::*; +pub use oss::*; +pub use s3::*; use serde::{Deserialize, Serialize}; /// Configuration properties for storage backends. diff --git a/crates/iceberg/src/io/config/oss.rs b/crates/iceberg/src/io/config/oss.rs new file mode 100644 index 0000000000..986710ef56 --- /dev/null +++ b/crates/iceberg/src/io/config/oss.rs @@ -0,0 +1,124 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Alibaba Cloud OSS storage configuration. +//! +//! This module provides configuration constants and types for Alibaba Cloud OSS storage. + +use serde::{Deserialize, Serialize}; +use typed_builder::TypedBuilder; + +use super::StorageConfig; +use crate::Result; + +/// Aliyun OSS endpoint. +pub const OSS_ENDPOINT: &str = "oss.endpoint"; +/// Aliyun OSS access key ID. +pub const OSS_ACCESS_KEY_ID: &str = "oss.access-key-id"; +/// Aliyun OSS access key secret. +pub const OSS_ACCESS_KEY_SECRET: &str = "oss.access-key-secret"; + +/// Alibaba Cloud OSS storage configuration. +/// +/// This struct contains all the configuration options for connecting to Alibaba Cloud OSS. +/// Use the builder pattern via `OssConfig::builder()` to construct instances. +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)] +pub struct OssConfig { + /// OSS endpoint URL. + #[builder(default, setter(strip_option, into))] + pub endpoint: Option, + /// OSS access key ID. + #[builder(default, setter(strip_option, into))] + pub access_key_id: Option, + /// OSS access key secret. + #[builder(default, setter(strip_option, into))] + pub access_key_secret: Option, +} + +impl TryFrom<&StorageConfig> for OssConfig { + type Error = crate::Error; + + fn try_from(config: &StorageConfig) -> Result { + let props = config.props(); + + let mut cfg = OssConfig::default(); + if let Some(endpoint) = props.get(OSS_ENDPOINT) { + cfg.endpoint = Some(endpoint.clone()); + } + if let Some(access_key_id) = props.get(OSS_ACCESS_KEY_ID) { + cfg.access_key_id = Some(access_key_id.clone()); + } + if let Some(access_key_secret) = props.get(OSS_ACCESS_KEY_SECRET) { + cfg.access_key_secret = Some(access_key_secret.clone()); + } + + Ok(cfg) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_oss_config_builder() { + let config = OssConfig::builder() + .endpoint("https://oss-cn-hangzhou.aliyuncs.com") + .access_key_id("my-access-key") + .access_key_secret("my-secret-key") + .build(); + + assert_eq!( + config.endpoint.as_deref(), + Some("https://oss-cn-hangzhou.aliyuncs.com") + ); + assert_eq!(config.access_key_id.as_deref(), Some("my-access-key")); + assert_eq!(config.access_key_secret.as_deref(), Some("my-secret-key")); + } + + #[test] + fn test_oss_config_from_storage_config() { + let storage_config = StorageConfig::new() + .with_prop(OSS_ENDPOINT, "https://oss-cn-hangzhou.aliyuncs.com") + .with_prop(OSS_ACCESS_KEY_ID, "my-access-key") + .with_prop(OSS_ACCESS_KEY_SECRET, "my-secret-key"); + + let oss_config = OssConfig::try_from(&storage_config).unwrap(); + + assert_eq!( + oss_config.endpoint.as_deref(), + Some("https://oss-cn-hangzhou.aliyuncs.com") + ); + assert_eq!(oss_config.access_key_id.as_deref(), Some("my-access-key")); + assert_eq!( + oss_config.access_key_secret.as_deref(), + Some("my-secret-key") + ); + } + + #[test] + fn test_oss_config_empty() { + let storage_config = StorageConfig::new(); + + let oss_config = OssConfig::try_from(&storage_config).unwrap(); + + assert_eq!(oss_config.endpoint, None); + assert_eq!(oss_config.access_key_id, None); + assert_eq!(oss_config.access_key_secret, None); + } +} diff --git a/crates/iceberg/src/io/config/s3.rs b/crates/iceberg/src/io/config/s3.rs new file mode 100644 index 0000000000..fae3a14757 --- /dev/null +++ b/crates/iceberg/src/io/config/s3.rs @@ -0,0 +1,303 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Amazon S3 storage configuration. +//! +//! This module provides configuration constants and types for Amazon S3 storage. +//! These are based on the [Iceberg S3 FileIO configuration](https://py.iceberg.apache.org/configuration/#s3). + +use serde::{Deserialize, Serialize}; +use typed_builder::TypedBuilder; + +use super::StorageConfig; +use crate::io::is_truthy; +use crate::{Error, ErrorKind, Result}; + +/// S3 endpoint URL. +pub const S3_ENDPOINT: &str = "s3.endpoint"; +/// S3 access key ID. +pub const S3_ACCESS_KEY_ID: &str = "s3.access-key-id"; +/// S3 secret access key. +pub const S3_SECRET_ACCESS_KEY: &str = "s3.secret-access-key"; +/// S3 session token (required when using temporary credentials). +pub const S3_SESSION_TOKEN: &str = "s3.session-token"; +/// S3 region. +pub const S3_REGION: &str = "s3.region"; +/// Region to use for the S3 client (takes precedence over [`S3_REGION`]). +pub const CLIENT_REGION: &str = "client.region"; +/// S3 Path Style Access. +pub const S3_PATH_STYLE_ACCESS: &str = "s3.path-style-access"; +/// S3 Server Side Encryption Type. +pub const S3_SSE_TYPE: &str = "s3.sse.type"; +/// S3 Server Side Encryption Key. +/// If S3 encryption type is kms, input is a KMS Key ID. +/// In case this property is not set, default key "aws/s3" is used. +/// If encryption type is custom, input is a custom base-64 AES256 symmetric key. +pub const S3_SSE_KEY: &str = "s3.sse.key"; +/// S3 Server Side Encryption MD5. +pub const S3_SSE_MD5: &str = "s3.sse.md5"; +/// If set, all AWS clients will assume a role of the given ARN, instead of using the default +/// credential chain. +pub const S3_ASSUME_ROLE_ARN: &str = "client.assume-role.arn"; +/// Optional external ID used to assume an IAM role. +pub const S3_ASSUME_ROLE_EXTERNAL_ID: &str = "client.assume-role.external-id"; +/// Optional session name used to assume an IAM role. +pub const S3_ASSUME_ROLE_SESSION_NAME: &str = "client.assume-role.session-name"; +/// Option to skip signing requests (e.g. for public buckets/folders). +pub const S3_ALLOW_ANONYMOUS: &str = "s3.allow-anonymous"; +/// Option to skip loading the credential from EC2 metadata (typically used in conjunction with +/// `S3_ALLOW_ANONYMOUS`). +pub const S3_DISABLE_EC2_METADATA: &str = "s3.disable-ec2-metadata"; +/// Option to skip loading configuration from config file and the env. +pub const S3_DISABLE_CONFIG_LOAD: &str = "s3.disable-config-load"; + +/// Amazon S3 storage configuration. +/// +/// This struct contains all the configuration options for connecting to Amazon S3. +/// Use the builder pattern via `S3Config::builder()` to construct instances. +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)] +pub struct S3Config { + /// S3 endpoint URL. + #[builder(default, setter(strip_option, into))] + pub endpoint: Option, + /// S3 access key ID. + #[builder(default, setter(strip_option, into))] + pub access_key_id: Option, + /// S3 secret access key. + #[builder(default, setter(strip_option, into))] + pub secret_access_key: Option, + /// S3 session token. + #[builder(default, setter(strip_option, into))] + pub session_token: Option, + /// S3 region. + #[builder(default, setter(strip_option, into))] + pub region: Option, + /// Enable virtual host style (opposite of path style access). + #[builder(default)] + pub enable_virtual_host_style: bool, + /// Server side encryption type. + #[builder(default, setter(strip_option, into))] + pub server_side_encryption: Option, + /// Server side encryption AWS KMS key ID. + #[builder(default, setter(strip_option, into))] + pub server_side_encryption_aws_kms_key_id: Option, + /// Server side encryption customer algorithm. + #[builder(default, setter(strip_option, into))] + pub server_side_encryption_customer_algorithm: Option, + /// Server side encryption customer key. + #[builder(default, setter(strip_option, into))] + pub server_side_encryption_customer_key: Option, + /// Server side encryption customer key MD5. + #[builder(default, setter(strip_option, into))] + pub server_side_encryption_customer_key_md5: Option, + /// Role ARN for assuming a role. + #[builder(default, setter(strip_option, into))] + pub role_arn: Option, + /// External ID for assuming a role. + #[builder(default, setter(strip_option, into))] + pub external_id: Option, + /// Session name for assuming a role. + #[builder(default, setter(strip_option, into))] + pub role_session_name: Option, + /// Allow anonymous access. + #[builder(default)] + pub allow_anonymous: bool, + /// Disable EC2 metadata. + #[builder(default)] + pub disable_ec2_metadata: bool, + /// Disable config load. + #[builder(default)] + pub disable_config_load: bool, +} + +impl TryFrom<&StorageConfig> for S3Config { + type Error = crate::Error; + + fn try_from(config: &StorageConfig) -> Result { + let props = config.props(); + + let mut cfg = S3Config::default(); + + if let Some(endpoint) = props.get(S3_ENDPOINT) { + cfg.endpoint = Some(endpoint.clone()); + } + if let Some(access_key_id) = props.get(S3_ACCESS_KEY_ID) { + cfg.access_key_id = Some(access_key_id.clone()); + } + if let Some(secret_access_key) = props.get(S3_SECRET_ACCESS_KEY) { + cfg.secret_access_key = Some(secret_access_key.clone()); + } + if let Some(session_token) = props.get(S3_SESSION_TOKEN) { + cfg.session_token = Some(session_token.clone()); + } + if let Some(region) = props.get(S3_REGION) { + cfg.region = Some(region.clone()); + } + // CLIENT_REGION takes precedence over S3_REGION + if let Some(region) = props.get(CLIENT_REGION) { + cfg.region = Some(region.clone()); + } + if let Some(path_style_access) = props.get(S3_PATH_STYLE_ACCESS) { + cfg.enable_virtual_host_style = !is_truthy(path_style_access.to_lowercase().as_str()); + } + if let Some(arn) = props.get(S3_ASSUME_ROLE_ARN) { + cfg.role_arn = Some(arn.clone()); + } + if let Some(external_id) = props.get(S3_ASSUME_ROLE_EXTERNAL_ID) { + cfg.external_id = Some(external_id.clone()); + } + if let Some(session_name) = props.get(S3_ASSUME_ROLE_SESSION_NAME) { + cfg.role_session_name = Some(session_name.clone()); + } + + // Handle SSE configuration + let s3_sse_key = props.get(S3_SSE_KEY).cloned(); + if let Some(sse_type) = props.get(S3_SSE_TYPE) { + match sse_type.to_lowercase().as_str() { + // No Server Side Encryption + "none" => {} + // S3 SSE-S3 encryption (S3 managed keys). https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html + "s3" => { + cfg.server_side_encryption = Some("AES256".to_string()); + } + // S3 SSE KMS, either using default or custom KMS key. https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html + "kms" => { + cfg.server_side_encryption = Some("aws:kms".to_string()); + cfg.server_side_encryption_aws_kms_key_id = s3_sse_key; + } + // S3 SSE-C, using customer managed keys. https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html + "custom" => { + cfg.server_side_encryption_customer_algorithm = Some("AES256".to_string()); + cfg.server_side_encryption_customer_key = s3_sse_key; + cfg.server_side_encryption_customer_key_md5 = props.get(S3_SSE_MD5).cloned(); + } + _ => { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Invalid {S3_SSE_TYPE}: {sse_type}. Expected one of (custom, kms, s3, none)" + ), + )); + } + } + } + + if let Some(allow_anonymous) = props.get(S3_ALLOW_ANONYMOUS) + && is_truthy(allow_anonymous.to_lowercase().as_str()) + { + cfg.allow_anonymous = true; + } + if let Some(disable_ec2_metadata) = props.get(S3_DISABLE_EC2_METADATA) + && is_truthy(disable_ec2_metadata.to_lowercase().as_str()) + { + cfg.disable_ec2_metadata = true; + } + if let Some(disable_config_load) = props.get(S3_DISABLE_CONFIG_LOAD) + && is_truthy(disable_config_load.to_lowercase().as_str()) + { + cfg.disable_config_load = true; + } + + Ok(cfg) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_s3_config_builder() { + let config = S3Config::builder() + .region("us-east-1") + .access_key_id("my-access-key") + .secret_access_key("my-secret-key") + .endpoint("http://localhost:9000") + .build(); + + assert_eq!(config.region.as_deref(), Some("us-east-1")); + assert_eq!(config.access_key_id.as_deref(), Some("my-access-key")); + assert_eq!(config.secret_access_key.as_deref(), Some("my-secret-key")); + assert_eq!(config.endpoint.as_deref(), Some("http://localhost:9000")); + } + + #[test] + fn test_s3_config_from_storage_config() { + let storage_config = StorageConfig::new() + .with_prop(S3_REGION, "us-east-1") + .with_prop(S3_ACCESS_KEY_ID, "my-access-key") + .with_prop(S3_SECRET_ACCESS_KEY, "my-secret-key") + .with_prop(S3_ENDPOINT, "http://localhost:9000"); + + let s3_config = S3Config::try_from(&storage_config).unwrap(); + + assert_eq!(s3_config.region.as_deref(), Some("us-east-1")); + assert_eq!(s3_config.access_key_id.as_deref(), Some("my-access-key")); + assert_eq!( + s3_config.secret_access_key.as_deref(), + Some("my-secret-key") + ); + assert_eq!(s3_config.endpoint.as_deref(), Some("http://localhost:9000")); + } + + #[test] + fn test_s3_config_client_region_precedence() { + let storage_config = StorageConfig::new() + .with_prop(S3_REGION, "us-east-1") + .with_prop(CLIENT_REGION, "eu-west-1"); + + let s3_config = S3Config::try_from(&storage_config).unwrap(); + + // CLIENT_REGION should take precedence + assert_eq!(s3_config.region.as_deref(), Some("eu-west-1")); + } + + #[test] + fn test_s3_config_path_style_access() { + let storage_config = StorageConfig::new().with_prop(S3_PATH_STYLE_ACCESS, "true"); + + let s3_config = S3Config::try_from(&storage_config).unwrap(); + + // path style access = true means virtual host style = false + assert!(!s3_config.enable_virtual_host_style); + } + + #[test] + fn test_s3_config_sse_kms() { + let storage_config = StorageConfig::new() + .with_prop(S3_SSE_TYPE, "kms") + .with_prop(S3_SSE_KEY, "my-kms-key-id"); + + let s3_config = S3Config::try_from(&storage_config).unwrap(); + + assert_eq!(s3_config.server_side_encryption.as_deref(), Some("aws:kms")); + assert_eq!( + s3_config.server_side_encryption_aws_kms_key_id.as_deref(), + Some("my-kms-key-id") + ); + } + + #[test] + fn test_s3_config_allow_anonymous() { + let storage_config = StorageConfig::new().with_prop(S3_ALLOW_ANONYMOUS, "true"); + + let s3_config = S3Config::try_from(&storage_config).unwrap(); + + assert!(s3_config.allow_anonymous); + } +} diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 55f2f262cb..8e40bae976 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -70,7 +70,9 @@ mod config; mod file_io; mod storage; +pub use config::*; pub use file_io::*; +pub use storage::{Storage, StorageFactory}; pub(crate) mod object_cache; #[cfg(feature = "storage-azdls")] @@ -86,18 +88,16 @@ mod storage_oss; #[cfg(feature = "storage-s3")] mod storage_s3; -pub use config::*; -pub use storage::{Storage, StorageFactory}; #[cfg(feature = "storage-azdls")] -pub use storage_azdls::*; +use storage_azdls::*; #[cfg(feature = "storage-fs")] use storage_fs::*; #[cfg(feature = "storage-gcs")] -pub use storage_gcs::*; +use storage_gcs::*; #[cfg(feature = "storage-memory")] use storage_memory::*; #[cfg(feature = "storage-oss")] -pub use storage_oss::*; +use storage_oss::*; #[cfg(feature = "storage-s3")] pub use storage_s3::*; diff --git a/crates/iceberg/src/io/storage_azdls.rs b/crates/iceberg/src/io/storage_azdls.rs index 5abb0cd6e0..ce2b7427bc 100644 --- a/crates/iceberg/src/io/storage_azdls.rs +++ b/crates/iceberg/src/io/storage_azdls.rs @@ -23,37 +23,12 @@ use opendal::Configurator; use opendal::services::AzdlsConfig; use url::Url; +use crate::io::config::{ + ADLS_ACCOUNT_KEY, ADLS_ACCOUNT_NAME, ADLS_AUTHORITY_HOST, ADLS_CLIENT_ID, ADLS_CLIENT_SECRET, + ADLS_CONNECTION_STRING, ADLS_SAS_TOKEN, ADLS_TENANT_ID, +}; use crate::{Error, ErrorKind, Result, ensure_data_valid}; -/// A connection string. -/// -/// Note, this string is parsed first, and any other passed adls.* properties -/// will override values from the connection string. -const ADLS_CONNECTION_STRING: &str = "adls.connection-string"; - -/// The account that you want to connect to. -pub const ADLS_ACCOUNT_NAME: &str = "adls.account-name"; - -/// The key to authentication against the account. -pub const ADLS_ACCOUNT_KEY: &str = "adls.account-key"; - -/// The shared access signature. -pub const ADLS_SAS_TOKEN: &str = "adls.sas-token"; - -/// The tenant-id. -pub const ADLS_TENANT_ID: &str = "adls.tenant-id"; - -/// The client-id. -pub const ADLS_CLIENT_ID: &str = "adls.client-id"; - -/// The client-secret. -pub const ADLS_CLIENT_SECRET: &str = "adls.client-secret"; - -/// The authority host of the service principal. -/// - required for client_credentials authentication -/// - default value: `https://login.microsoftonline.com` -pub const ADLS_AUTHORITY_HOST: &str = "adls.authority-host"; - /// Parses adls.* prefixed configuration properties. pub(crate) fn azdls_config_parse(mut properties: HashMap) -> Result { let mut config = AzdlsConfig::default(); diff --git a/crates/iceberg/src/io/storage_gcs.rs b/crates/iceberg/src/io/storage_gcs.rs index 7718df603f..5c6145d32b 100644 --- a/crates/iceberg/src/io/storage_gcs.rs +++ b/crates/iceberg/src/io/storage_gcs.rs @@ -22,33 +22,13 @@ use opendal::Operator; use opendal::services::GcsConfig; use url::Url; +use crate::io::config::{ + GCS_ALLOW_ANONYMOUS, GCS_CREDENTIALS_JSON, GCS_DISABLE_CONFIG_LOAD, GCS_DISABLE_VM_METADATA, + GCS_NO_AUTH, GCS_SERVICE_PATH, GCS_TOKEN, +}; use crate::io::is_truthy; use crate::{Error, ErrorKind, Result}; -// Reference: https://github.com/apache/iceberg/blob/main/gcp/src/main/java/org/apache/iceberg/gcp/GCPProperties.java - -/// Google Cloud Project ID -pub const GCS_PROJECT_ID: &str = "gcs.project-id"; -/// Google Cloud Storage endpoint -pub const GCS_SERVICE_PATH: &str = "gcs.service.path"; -/// Google Cloud user project -pub const GCS_USER_PROJECT: &str = "gcs.user-project"; -/// Allow unauthenticated requests -pub const GCS_NO_AUTH: &str = "gcs.no-auth"; -/// Google Cloud Storage credentials JSON string, base64 encoded. -/// -/// E.g. base64::prelude::BASE64_STANDARD.encode(serde_json::to_string(credential).as_bytes()) -pub const GCS_CREDENTIALS_JSON: &str = "gcs.credentials-json"; -/// Google Cloud Storage token -pub const GCS_TOKEN: &str = "gcs.oauth2.token"; - -/// Option to skip signing requests (e.g. for public buckets/folders). -pub const GCS_ALLOW_ANONYMOUS: &str = "gcs.allow-anonymous"; -/// Option to skip loading the credential from GCE metadata server (typically used in conjunction with `GCS_ALLOW_ANONYMOUS`). -pub const GCS_DISABLE_VM_METADATA: &str = "gcs.disable-vm-metadata"; -/// Option to skip loading configuration from config file and the env. -pub const GCS_DISABLE_CONFIG_LOAD: &str = "gcs.disable-config-load"; - /// Parse iceberg properties to [`GcsConfig`]. pub(crate) fn gcs_config_parse(mut m: HashMap) -> Result { let mut cfg = GcsConfig::default(); diff --git a/crates/iceberg/src/io/storage_oss.rs b/crates/iceberg/src/io/storage_oss.rs index e82dda23a5..83fc1424aa 100644 --- a/crates/iceberg/src/io/storage_oss.rs +++ b/crates/iceberg/src/io/storage_oss.rs @@ -21,19 +21,9 @@ use opendal::services::OssConfig; use opendal::{Configurator, Operator}; use url::Url; +use crate::io::config::{OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_ENDPOINT}; use crate::{Error, ErrorKind, Result}; -/// Required configuration arguments for creating an Aliyun OSS Operator with OpenDAL: -/// - `oss.endpoint`: The OSS service endpoint URL -/// - `oss.access-key-id`: The access key ID for authentication -/// - `oss.access-key-secret`: The access key secret for authentication -/// Aliyun oss endpoint. -pub const OSS_ENDPOINT: &str = "oss.endpoint"; -/// Aliyun oss access key id. -pub const OSS_ACCESS_KEY_ID: &str = "oss.access-key-id"; -/// Aliyun oss access key secret. -pub const OSS_ACCESS_KEY_SECRET: &str = "oss.access-key-secret"; - /// Parse iceberg props to oss config. pub(crate) fn oss_config_parse(mut m: HashMap) -> Result { let mut cfg: OssConfig = OssConfig::default(); diff --git a/crates/iceberg/src/io/storage_s3.rs b/crates/iceberg/src/io/storage_s3.rs index f069e0e2f9..bf7399e01b 100644 --- a/crates/iceberg/src/io/storage_s3.rs +++ b/crates/iceberg/src/io/storage_s3.rs @@ -25,51 +25,15 @@ pub use reqsign::{AwsCredential, AwsCredentialLoad}; use reqwest::Client; use url::Url; +use crate::io::config::{ + CLIENT_REGION, S3_ACCESS_KEY_ID, S3_ALLOW_ANONYMOUS, S3_ASSUME_ROLE_ARN, + S3_ASSUME_ROLE_EXTERNAL_ID, S3_ASSUME_ROLE_SESSION_NAME, S3_DISABLE_CONFIG_LOAD, + S3_DISABLE_EC2_METADATA, S3_ENDPOINT, S3_PATH_STYLE_ACCESS, S3_REGION, S3_SECRET_ACCESS_KEY, + S3_SESSION_TOKEN, S3_SSE_KEY, S3_SSE_MD5, S3_SSE_TYPE, +}; use crate::io::is_truthy; use crate::{Error, ErrorKind, Result}; -/// Following are arguments for [s3 file io](https://py.iceberg.apache.org/configuration/#s3). -/// S3 endpoint. -pub const S3_ENDPOINT: &str = "s3.endpoint"; -/// S3 access key id. -pub const S3_ACCESS_KEY_ID: &str = "s3.access-key-id"; -/// S3 secret access key. -pub const S3_SECRET_ACCESS_KEY: &str = "s3.secret-access-key"; -/// S3 session token. -/// This is required when using temporary credentials. -pub const S3_SESSION_TOKEN: &str = "s3.session-token"; -/// S3 region. -pub const S3_REGION: &str = "s3.region"; -/// Region to use for the S3 client. -/// -/// This takes precedence over [`S3_REGION`]. -pub const CLIENT_REGION: &str = "client.region"; -/// S3 Path Style Access. -pub const S3_PATH_STYLE_ACCESS: &str = "s3.path-style-access"; -/// S3 Server Side Encryption Type. -pub const S3_SSE_TYPE: &str = "s3.sse.type"; -/// S3 Server Side Encryption Key. -/// If S3 encryption type is kms, input is a KMS Key ID. -/// In case this property is not set, default key "aws/s3" is used. -/// If encryption type is custom, input is a custom base-64 AES256 symmetric key. -pub const S3_SSE_KEY: &str = "s3.sse.key"; -/// S3 Server Side Encryption MD5. -pub const S3_SSE_MD5: &str = "s3.sse.md5"; -/// If set, all AWS clients will assume a role of the given ARN, instead of using the default -/// credential chain. -pub const S3_ASSUME_ROLE_ARN: &str = "client.assume-role.arn"; -/// Optional external ID used to assume an IAM role. -pub const S3_ASSUME_ROLE_EXTERNAL_ID: &str = "client.assume-role.external-id"; -/// Optional session name used to assume an IAM role. -pub const S3_ASSUME_ROLE_SESSION_NAME: &str = "client.assume-role.session-name"; -/// Option to skip signing requests (e.g. for public buckets/folders). -pub const S3_ALLOW_ANONYMOUS: &str = "s3.allow-anonymous"; -/// Option to skip loading the credential from EC2 metadata (typically used in conjunction with -/// `S3_ALLOW_ANONYMOUS`). -pub const S3_DISABLE_EC2_METADATA: &str = "s3.disable-ec2-metadata"; -/// Option to skip loading configuration from config file and the env. -pub const S3_DISABLE_CONFIG_LOAD: &str = "s3.disable-config-load"; - /// Parse iceberg props to s3 config. pub(crate) fn s3_config_parse(mut m: HashMap) -> Result { let mut cfg = S3Config::default(); From 045c0b2e298fee03e6d658bba78092a7fcf65747 Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Wed, 28 Jan 2026 12:23:50 +0800 Subject: [PATCH 175/891] Cleanup integration tests (#2074) --- Cargo.lock | 3 - crates/integration_tests/Cargo.toml | 2 - crates/integration_tests/src/lib.rs | 50 +-- .../shared_tests/append_data_file_test.rs | 138 ------- .../append_partition_data_file_test.rs | 235 ----------- .../tests/shared_tests/datafusion.rs | 142 ------- .../tests/shared_tests/mod.rs | 4 - .../tests/shared_tests/scan_all_type.rs | 366 ------------------ .../testdata/schedules/df_test.toml | 8 + .../testdata/slts/df_test/basic_queries.slt | 212 ++++++++++ .../testdata/slts/df_test/scan_all_types.slt | 128 ++++++ crates/test_utils/Cargo.toml | 1 - crates/test_utils/src/cmd.rs | 58 --- crates/test_utils/src/docker.rs | 165 -------- crates/test_utils/src/lib.rs | 5 - 15 files changed, 349 insertions(+), 1168 deletions(-) delete mode 100644 crates/integration_tests/tests/shared_tests/append_data_file_test.rs delete mode 100644 crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs delete mode 100644 crates/integration_tests/tests/shared_tests/datafusion.rs delete mode 100644 crates/integration_tests/tests/shared_tests/scan_all_type.rs create mode 100644 crates/sqllogictest/testdata/slts/df_test/basic_queries.slt create mode 100644 crates/sqllogictest/testdata/slts/df_test/scan_all_types.slt delete mode 100644 crates/test_utils/src/cmd.rs delete mode 100644 crates/test_utils/src/docker.rs diff --git a/Cargo.lock b/Cargo.lock index c0c46ce672..211b6416cb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3547,11 +3547,9 @@ version = "0.8.0" dependencies = [ "arrow-array", "arrow-schema", - "datafusion", "futures", "iceberg", "iceberg-catalog-rest", - "iceberg-datafusion", "iceberg_test_utils", "ordered-float 2.10.1", "parquet", @@ -3608,7 +3606,6 @@ name = "iceberg_test_utils" version = "0.8.0" dependencies = [ "iceberg", - "tracing", "tracing-subscriber", ] diff --git a/crates/integration_tests/Cargo.toml b/crates/integration_tests/Cargo.toml index 07291f29ef..2ed211769f 100644 --- a/crates/integration_tests/Cargo.toml +++ b/crates/integration_tests/Cargo.toml @@ -27,11 +27,9 @@ version = { workspace = true } [dependencies] arrow-array = { workspace = true } arrow-schema = { workspace = true } -datafusion = { workspace = true } futures = { workspace = true } iceberg = { workspace = true } iceberg-catalog-rest = { workspace = true } -iceberg-datafusion = { workspace = true } iceberg_test_utils = { path = "../test_utils", features = ["tests"] } parquet = { workspace = true } tokio = { workspace = true } diff --git a/crates/integration_tests/src/lib.rs b/crates/integration_tests/src/lib.rs index e99c74df6b..4bf8f4d19c 100644 --- a/crates/integration_tests/src/lib.rs +++ b/crates/integration_tests/src/lib.rs @@ -20,19 +20,7 @@ use std::sync::OnceLock; use iceberg::io::{S3_ACCESS_KEY_ID, S3_ENDPOINT, S3_REGION, S3_SECRET_ACCESS_KEY}; use iceberg_catalog_rest::REST_CATALOG_PROP_URI; -use iceberg_test_utils::docker::DockerCompose; -use iceberg_test_utils::{ - get_minio_endpoint, get_rest_catalog_endpoint, normalize_test_name, set_up, -}; - -const REST_CATALOG_PORT: u16 = 8181; - -/// Test fixture that manages Docker containers. -/// This is kept for backward compatibility but deprecated in favor of GlobalTestFixture. -pub struct TestFixture { - pub _docker_compose: DockerCompose, - pub catalog_config: HashMap, -} +use iceberg_test_utils::{get_minio_endpoint, get_rest_catalog_endpoint, set_up}; /// Global test fixture that uses environment-based configuration. /// This assumes Docker containers are started externally (e.g., via `make docker-up`). @@ -68,39 +56,3 @@ impl GlobalTestFixture { pub fn get_test_fixture() -> &'static GlobalTestFixture { GLOBAL_FIXTURE.get_or_init(GlobalTestFixture::from_env) } - -/// Legacy function to create a test fixture with Docker container management. -/// Deprecated: prefer using `get_test_fixture()` with externally managed containers. -pub fn set_test_fixture(func: &str) -> TestFixture { - set_up(); - let docker_compose = DockerCompose::new( - normalize_test_name(format!("{}_{func}", module_path!())), - format!("{}/testdata", env!("CARGO_MANIFEST_DIR")), - ); - - // Stop any containers from previous runs and start new ones - docker_compose.down(); - docker_compose.up(); - - let rest_catalog_ip = docker_compose.get_container_ip("rest"); - let minio_ip = docker_compose.get_container_ip("minio"); - - let catalog_config = HashMap::from([ - ( - REST_CATALOG_PROP_URI.to_string(), - format!("http://{rest_catalog_ip}:{REST_CATALOG_PORT}"), - ), - ( - S3_ENDPOINT.to_string(), - format!("http://{}:{}", minio_ip, 9000), - ), - (S3_ACCESS_KEY_ID.to_string(), "admin".to_string()), - (S3_SECRET_ACCESS_KEY.to_string(), "password".to_string()), - (S3_REGION.to_string(), "us-east-1".to_string()), - ]); - - TestFixture { - _docker_compose: docker_compose, - catalog_config, - } -} diff --git a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_data_file_test.rs deleted file mode 100644 index bedc975102..0000000000 --- a/crates/integration_tests/tests/shared_tests/append_data_file_test.rs +++ /dev/null @@ -1,138 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -//! Integration tests for rest catalog. - -use std::sync::Arc; - -use arrow_array::{ArrayRef, BooleanArray, Int32Array, RecordBatch, StringArray}; -use futures::TryStreamExt; -use iceberg::transaction::{ApplyTransactionAction, Transaction}; -use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; -use iceberg::writer::file_writer::ParquetWriterBuilder; -use iceberg::writer::file_writer::location_generator::{ - DefaultFileNameGenerator, DefaultLocationGenerator, -}; -use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; -use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; -use iceberg::{Catalog, CatalogBuilder, TableCreation}; -use iceberg_catalog_rest::RestCatalogBuilder; -use parquet::arrow::arrow_reader::ArrowReaderOptions; -use parquet::file::properties::WriterProperties; - -use crate::get_shared_containers; -use crate::shared_tests::{random_ns, test_schema}; - -#[tokio::test] -async fn test_append_data_file() { - let fixture = get_shared_containers(); - let rest_catalog = RestCatalogBuilder::default() - .load("rest", fixture.catalog_config.clone()) - .await - .unwrap(); - let ns = random_ns().await; - let schema = test_schema(); - - let table_creation = TableCreation::builder() - .name("t1".to_string()) - .schema(schema.clone()) - .build(); - - let table = rest_catalog - .create_table(ns.name(), table_creation) - .await - .unwrap(); - - // Create the writer and write the data - let schema: Arc = Arc::new( - table - .metadata() - .current_schema() - .as_ref() - .try_into() - .unwrap(), - ); - let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); - let file_name_generator = DefaultFileNameGenerator::new( - "test".to_string(), - None, - iceberg::spec::DataFileFormat::Parquet, - ); - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::default(), - table.metadata().current_schema().clone(), - ); - let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( - parquet_writer_builder, - table.file_io().clone(), - location_generator.clone(), - file_name_generator.clone(), - ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); - let mut data_file_writer = data_file_writer_builder.build(None).await.unwrap(); - let col1 = StringArray::from(vec![Some("foo"), Some("bar"), None, Some("baz")]); - let col2 = Int32Array::from(vec![Some(1), Some(2), Some(3), Some(4)]); - let col3 = BooleanArray::from(vec![Some(true), Some(false), None, Some(false)]); - let batch = RecordBatch::try_new(schema.clone(), vec![ - Arc::new(col1) as ArrayRef, - Arc::new(col2) as ArrayRef, - Arc::new(col3) as ArrayRef, - ]) - .unwrap(); - data_file_writer.write(batch.clone()).await.unwrap(); - let data_file = data_file_writer.close().await.unwrap(); - - // check parquet file schema - let content = table - .file_io() - .new_input(data_file[0].file_path()) - .unwrap() - .read() - .await - .unwrap(); - let parquet_reader = parquet::arrow::arrow_reader::ArrowReaderMetadata::load( - &content, - ArrowReaderOptions::default(), - ) - .unwrap(); - let field_ids: Vec = parquet_reader - .parquet_schema() - .columns() - .iter() - .map(|col| col.self_type().get_basic_info().id()) - .collect(); - assert_eq!(field_ids, vec![1, 2, 3]); - - // commit result - let tx = Transaction::new(&table); - let append_action = tx.fast_append().add_data_files(data_file.clone()); - let tx = append_action.apply(tx).unwrap(); - let table = tx.commit(&rest_catalog).await.unwrap(); - - // check result - let batch_stream = table - .scan() - .select_all() - .build() - .unwrap() - .to_arrow() - .await - .unwrap(); - let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); - assert_eq!(batches.len(), 1); - assert_eq!(batches[0], batch); -} diff --git a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs b/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs deleted file mode 100644 index a305ec0842..0000000000 --- a/crates/integration_tests/tests/shared_tests/append_partition_data_file_test.rs +++ /dev/null @@ -1,235 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -//! Integration test for partition data file - -use std::sync::Arc; - -use arrow_array::{ArrayRef, BooleanArray, Int32Array, RecordBatch, StringArray}; -use futures::TryStreamExt; -use iceberg::spec::{ - Literal, PartitionKey, PrimitiveLiteral, Struct, Transform, UnboundPartitionSpec, -}; -use iceberg::table::Table; -use iceberg::transaction::{ApplyTransactionAction, Transaction}; -use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; -use iceberg::writer::file_writer::ParquetWriterBuilder; -use iceberg::writer::file_writer::location_generator::{ - DefaultFileNameGenerator, DefaultLocationGenerator, -}; -use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; -use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; -use iceberg::{Catalog, CatalogBuilder, TableCreation}; -use iceberg_catalog_rest::RestCatalogBuilder; -use parquet::file::properties::WriterProperties; - -use crate::get_shared_containers; -use crate::shared_tests::{random_ns, test_schema}; - -#[tokio::test] -async fn test_append_partition_data_file() { - let fixture = get_shared_containers(); - let rest_catalog = RestCatalogBuilder::default() - .load("rest", fixture.catalog_config.clone()) - .await - .unwrap(); - let ns = random_ns().await; - let schema = test_schema(); - - let unbound_partition_spec = UnboundPartitionSpec::builder() - .add_partition_field(2, "id", Transform::Identity) - .expect("could not add partition field") - .build(); - - let partition_spec = unbound_partition_spec - .bind(schema.clone()) - .expect("could not bind to schema"); - - let table_creation = TableCreation::builder() - .name("t1".to_string()) - .schema(schema.clone()) - .partition_spec(partition_spec.clone()) - .build(); - - let table = rest_catalog - .create_table(ns.name(), table_creation) - .await - .unwrap(); - - // Create the writer and write the data - let schema: Arc = Arc::new( - table - .metadata() - .current_schema() - .as_ref() - .try_into() - .unwrap(), - ); - - let first_partition_id_value = 100; - let partition_key = PartitionKey::new( - partition_spec.clone(), - table.metadata().current_schema().clone(), - Struct::from_iter(vec![Some(Literal::int(first_partition_id_value))]), - ); - - let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); - let file_name_generator = DefaultFileNameGenerator::new( - "test".to_string(), - None, - iceberg::spec::DataFileFormat::Parquet, - ); - - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::default(), - table.metadata().current_schema().clone(), - ); - - let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( - parquet_writer_builder.clone(), - table.file_io().clone(), - location_generator.clone(), - file_name_generator.clone(), - ); - - let mut data_file_writer_valid = - DataFileWriterBuilder::new(rolling_file_writer_builder.clone()) - .build(Some(partition_key.clone())) - .await - .unwrap(); - - let col1 = StringArray::from(vec![Some("foo1"), Some("foo2")]); - let col2 = Int32Array::from(vec![ - Some(first_partition_id_value), - Some(first_partition_id_value), - ]); - let col3 = BooleanArray::from(vec![Some(true), Some(false)]); - let batch = RecordBatch::try_new(schema.clone(), vec![ - Arc::new(col1) as ArrayRef, - Arc::new(col2) as ArrayRef, - Arc::new(col3) as ArrayRef, - ]) - .unwrap(); - - data_file_writer_valid.write(batch.clone()).await.unwrap(); - let data_file_valid = data_file_writer_valid.close().await.unwrap(); - - // commit result - let tx = Transaction::new(&table); - let append_action = tx.fast_append().add_data_files(data_file_valid.clone()); - let tx = append_action.apply(tx).unwrap(); - let table = tx.commit(&rest_catalog).await.unwrap(); - - // check result - let batch_stream = table - .scan() - .select_all() - .build() - .unwrap() - .to_arrow() - .await - .unwrap(); - let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); - assert_eq!(batches.len(), 1); - assert_eq!(batches[0], batch); - - let partition_key = partition_key.copy_with_data(Struct::from_iter([Some( - Literal::Primitive(PrimitiveLiteral::Boolean(true)), - )])); - test_schema_incompatible_partition_type( - rolling_file_writer_builder.clone(), - batch.clone(), - partition_key.clone(), - table.clone(), - &rest_catalog, - ) - .await; - - let partition_key = partition_key.copy_with_data(Struct::from_iter([ - Some(Literal::Primitive(PrimitiveLiteral::Int( - first_partition_id_value, - ))), - Some(Literal::Primitive(PrimitiveLiteral::Int( - first_partition_id_value, - ))), - ])); - test_schema_incompatible_partition_fields( - rolling_file_writer_builder.clone(), - batch, - partition_key, - table, - &rest_catalog, - ) - .await; -} - -async fn test_schema_incompatible_partition_type( - rolling_file_writer_builder: RollingFileWriterBuilder< - ParquetWriterBuilder, - DefaultLocationGenerator, - DefaultFileNameGenerator, - >, - batch: RecordBatch, - partition_key: PartitionKey, - table: Table, - catalog: &dyn Catalog, -) { - // test writing different "type" of partition than mentioned in schema - let mut data_file_writer_invalid = DataFileWriterBuilder::new(rolling_file_writer_builder) - .build(Some(partition_key)) - .await - .unwrap(); - - data_file_writer_invalid.write(batch.clone()).await.unwrap(); - let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); - - let tx = Transaction::new(&table); - let append_action = tx.fast_append().add_data_files(data_file_invalid.clone()); - let tx = append_action.apply(tx).unwrap(); - - if tx.commit(catalog).await.is_ok() { - panic!("diverging partition info should have returned error"); - } -} - -async fn test_schema_incompatible_partition_fields( - rolling_file_writer_builder: RollingFileWriterBuilder< - ParquetWriterBuilder, - DefaultLocationGenerator, - DefaultFileNameGenerator, - >, - batch: RecordBatch, - partition_key: PartitionKey, - table: Table, - catalog: &dyn Catalog, -) { - // test writing different number of partition fields than mentioned in schema - let mut data_file_writer_invalid = DataFileWriterBuilder::new(rolling_file_writer_builder) - .build(Some(partition_key)) - .await - .unwrap(); - - data_file_writer_invalid.write(batch.clone()).await.unwrap(); - let data_file_invalid = data_file_writer_invalid.close().await.unwrap(); - - let tx = Transaction::new(&table); - let append_action = tx.fast_append().add_data_files(data_file_invalid.clone()); - let tx = append_action.apply(tx).unwrap(); - if tx.commit(catalog).await.is_ok() { - panic!("passing different number of partition fields should have returned error"); - } -} diff --git a/crates/integration_tests/tests/shared_tests/datafusion.rs b/crates/integration_tests/tests/shared_tests/datafusion.rs deleted file mode 100644 index 60dd9f36c8..0000000000 --- a/crates/integration_tests/tests/shared_tests/datafusion.rs +++ /dev/null @@ -1,142 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -use std::collections::HashMap; -use std::sync::Arc; - -use arrow_schema::TimeUnit; -use datafusion::arrow::datatypes::{DataType, Field, Schema}; -use datafusion::assert_batches_eq; -use datafusion::catalog::TableProvider; -use datafusion::error::DataFusionError; -use datafusion::prelude::SessionContext; -use iceberg::{Catalog, CatalogBuilder, TableIdent}; -use iceberg_catalog_rest::RestCatalogBuilder; -use iceberg_datafusion::IcebergStaticTableProvider; -use parquet::arrow::PARQUET_FIELD_ID_META_KEY; - -use crate::get_shared_containers; - -#[tokio::test] -async fn test_basic_queries() -> Result<(), DataFusionError> { - let fixture = get_shared_containers(); - let rest_catalog = RestCatalogBuilder::default() - .load("rest", fixture.catalog_config.clone()) - .await - .unwrap(); - - let table = rest_catalog - .load_table(&TableIdent::from_strs(["default", "types_test"]).unwrap()) - .await - .unwrap(); - - let ctx = SessionContext::new(); - - let table_provider = Arc::new( - IcebergStaticTableProvider::try_new_from_table(table) - .await - .unwrap(), - ); - - let schema = table_provider.schema(); - - assert_eq!( - schema.as_ref(), - &Schema::new(vec![ - Field::new("cboolean", DataType::Boolean, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "1".to_string(), - )])), - Field::new("ctinyint", DataType::Int32, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "2".to_string(), - )])), - Field::new("csmallint", DataType::Int32, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "3".to_string(), - )])), - Field::new("cint", DataType::Int32, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "4".to_string(), - )])), - Field::new("cbigint", DataType::Int64, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "5".to_string(), - )])), - Field::new("cfloat", DataType::Float32, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "6".to_string(), - )])), - Field::new("cdouble", DataType::Float64, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "7".to_string(), - )])), - Field::new("cdecimal", DataType::Decimal128(8, 2), true).with_metadata(HashMap::from( - [(PARQUET_FIELD_ID_META_KEY.to_string(), "8".to_string(),)] - )), - Field::new("cdate", DataType::Date32, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "9".to_string(), - )])), - Field::new( - "ctimestamp_ntz", - DataType::Timestamp(TimeUnit::Microsecond, None), - true - ) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "10".to_string(), - )])), - Field::new( - "ctimestamp", - DataType::Timestamp(TimeUnit::Microsecond, Some(Arc::from("+00:00"))), - true - ) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "11".to_string(), - )])), - Field::new("cstring", DataType::Utf8, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "12".to_string(), - )])), - Field::new("cbinary", DataType::LargeBinary, true).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "13".to_string(), - )])), - ]) - ); - - ctx.register_table("types_table", table_provider)?; - - let batches = ctx - .sql("SELECT * FROM types_table ORDER BY cbigint LIMIT 3") - .await? - .collect() - .await?; - let expected = [ - "+----------+----------+-----------+------+---------+--------+---------+----------+------------+---------------------+----------------------+---------+----------+", - "| cboolean | ctinyint | csmallint | cint | cbigint | cfloat | cdouble | cdecimal | cdate | ctimestamp_ntz | ctimestamp | cstring | cbinary |", - "+----------+----------+-----------+------+---------+--------+---------+----------+------------+---------------------+----------------------+---------+----------+", - "| false | -128 | 0 | 0 | 0 | 0.0 | 0.0 | 0.00 | 1970-01-01 | 1970-01-01T00:00:00 | 1970-01-01T00:00:00Z | 0 | 00000000 |", - "| true | -127 | 1 | 1 | 1 | 1.0 | 1.0 | 0.01 | 1970-01-02 | 1970-01-01T00:00:01 | 1970-01-01T00:00:01Z | 1 | 00000001 |", - "| false | -126 | 2 | 2 | 2 | 2.0 | 2.0 | 0.02 | 1970-01-03 | 1970-01-01T00:00:02 | 1970-01-01T00:00:02Z | 2 | 00000002 |", - "+----------+----------+-----------+------+---------+--------+---------+----------+------------+---------------------+----------------------+---------+----------+", - ]; - assert_batches_eq!(expected, &batches); - Ok(()) -} diff --git a/crates/integration_tests/tests/shared_tests/mod.rs b/crates/integration_tests/tests/shared_tests/mod.rs index 1e695c5884..8fd56ab521 100644 --- a/crates/integration_tests/tests/shared_tests/mod.rs +++ b/crates/integration_tests/tests/shared_tests/mod.rs @@ -23,13 +23,9 @@ use iceberg_catalog_rest::RestCatalogBuilder; use crate::get_shared_containers; -mod append_data_file_test; -mod append_partition_data_file_test; mod conflict_commit_test; -mod datafusion; mod read_evolved_schema; mod read_positional_deletes; -mod scan_all_type; pub async fn random_ns() -> Namespace { let fixture = get_shared_containers(); diff --git a/crates/integration_tests/tests/shared_tests/scan_all_type.rs b/crates/integration_tests/tests/shared_tests/scan_all_type.rs deleted file mode 100644 index 7a2907d4cb..0000000000 --- a/crates/integration_tests/tests/shared_tests/scan_all_type.rs +++ /dev/null @@ -1,366 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -//! Integration tests for rest catalog. - -use std::collections::HashMap; -use std::sync::Arc; - -use arrow_array::builder::{Int32Builder, ListBuilder, MapBuilder, StringBuilder}; -use arrow_array::{ - Array, ArrayRef, BooleanArray, Date32Array, Decimal128Array, FixedSizeBinaryArray, - Float32Array, Float64Array, Int32Array, Int64Array, LargeBinaryArray, MapArray, RecordBatch, - StringArray, StructArray, Time64MicrosecondArray, TimestampMicrosecondArray, -}; -use arrow_schema::{DataType, Field, Fields}; -use futures::TryStreamExt; -use iceberg::arrow::{DEFAULT_MAP_FIELD_NAME, UTC_TIME_ZONE}; -use iceberg::spec::{ - LIST_FIELD_NAME, ListType, MAP_KEY_FIELD_NAME, MAP_VALUE_FIELD_NAME, MapType, NestedField, - PrimitiveType, Schema, StructType, Type, -}; -use iceberg::transaction::{ApplyTransactionAction, Transaction}; -use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; -use iceberg::writer::file_writer::ParquetWriterBuilder; -use iceberg::writer::file_writer::location_generator::{ - DefaultFileNameGenerator, DefaultLocationGenerator, -}; -use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; -use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; -use iceberg::{Catalog, CatalogBuilder, TableCreation}; -use iceberg_catalog_rest::RestCatalogBuilder; -use parquet::arrow::PARQUET_FIELD_ID_META_KEY; -use parquet::file::properties::WriterProperties; -use uuid::Uuid; - -use crate::get_shared_containers; -use crate::shared_tests::random_ns; - -#[tokio::test] -async fn test_scan_all_type() { - let fixture = get_shared_containers(); - let rest_catalog = RestCatalogBuilder::default() - .load("rest", fixture.catalog_config.clone()) - .await - .unwrap(); - let ns = random_ns().await; - - let schema = Schema::builder() - .with_schema_id(1) - .with_identifier_field_ids(vec![2]) - .with_fields(vec![ - // test all type - NestedField::required(1, "int", Type::Primitive(PrimitiveType::Int)).into(), - NestedField::required(2, "long", Type::Primitive(PrimitiveType::Long)).into(), - NestedField::required(3, "float", Type::Primitive(PrimitiveType::Float)).into(), - NestedField::required(4, "double", Type::Primitive(PrimitiveType::Double)).into(), - NestedField::required( - 5, - "decimal", - Type::Primitive(PrimitiveType::Decimal { - precision: 20, - scale: 5, - }), - ) - .into(), - NestedField::required(6, "string", Type::Primitive(PrimitiveType::String)).into(), - NestedField::required(7, "boolean", Type::Primitive(PrimitiveType::Boolean)).into(), - NestedField::required(8, "binary", Type::Primitive(PrimitiveType::Binary)).into(), - NestedField::required(9, "date", Type::Primitive(PrimitiveType::Date)).into(), - NestedField::required(10, "time", Type::Primitive(PrimitiveType::Time)).into(), - NestedField::required(11, "timestamp", Type::Primitive(PrimitiveType::Timestamp)) - .into(), - NestedField::required(12, "fixed", Type::Primitive(PrimitiveType::Fixed(10))).into(), - NestedField::required(13, "uuid", Type::Primitive(PrimitiveType::Uuid)).into(), - NestedField::required( - 14, - "timestamptz", - Type::Primitive(PrimitiveType::Timestamptz), - ) - .into(), - NestedField::required( - 15, - "struct", - Type::Struct(StructType::new(vec![ - NestedField::required(18, "int", Type::Primitive(PrimitiveType::Int)).into(), - NestedField::required(19, "string", Type::Primitive(PrimitiveType::String)) - .into(), - ])), - ) - .into(), - NestedField::required( - 16, - "list", - Type::List(ListType::new( - NestedField::list_element(20, Type::Primitive(PrimitiveType::Int), true).into(), - )), - ) - .into(), - NestedField::required( - 17, - "map", - Type::Map(MapType::new( - NestedField::map_key_element(21, Type::Primitive(PrimitiveType::Int)).into(), - NestedField::map_value_element( - 22, - Type::Primitive(PrimitiveType::String), - true, - ) - .into(), - )), - ) - .into(), - ]) - .build() - .unwrap(); - - let table_creation = TableCreation::builder() - .name("t1".to_string()) - .schema(schema.clone()) - .build(); - - let table = rest_catalog - .create_table(ns.name(), table_creation) - .await - .unwrap(); - - // Create the writer and write the data - let schema: Arc = Arc::new( - table - .metadata() - .current_schema() - .as_ref() - .try_into() - .unwrap(), - ); - let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); - let file_name_generator = DefaultFileNameGenerator::new( - "test".to_string(), - None, - iceberg::spec::DataFileFormat::Parquet, - ); - let parquet_writer_builder = ParquetWriterBuilder::new( - WriterProperties::default(), - table.metadata().current_schema().clone(), - ); - let rolling_file_writer_builder = RollingFileWriterBuilder::new_with_default_file_size( - parquet_writer_builder, - table.file_io().clone(), - location_generator.clone(), - file_name_generator.clone(), - ); - let data_file_writer_builder = DataFileWriterBuilder::new(rolling_file_writer_builder); - let mut data_file_writer = data_file_writer_builder.build(None).await.unwrap(); - - // Prepare data - let col1 = Int32Array::from(vec![1, 2, 3, 4, 5]); - let col2 = Int64Array::from(vec![1, 2, 3, 4, 5]); - let col3 = Float32Array::from(vec![1.1, 2.2, 3.3, 4.4, 5.5]); - let col4 = Float64Array::from(vec![1.1, 2.2, 3.3, 4.4, 5.5]); - let col5 = Decimal128Array::from(vec![ - Some(1.into()), - Some(2.into()), - Some(3.into()), - Some(4.into()), - Some(5.into()), - ]) - .with_data_type(DataType::Decimal128(20, 5)); - let col6 = StringArray::from(vec!["a", "b", "c", "d", "e"]); - let col7 = BooleanArray::from(vec![true, false, true, false, true]); - let col8 = LargeBinaryArray::from_opt_vec(vec![ - Some(b"a"), - Some(b"b"), - Some(b"c"), - Some(b"d"), - Some(b"e"), - ]); - let col9 = Date32Array::from(vec![1, 2, 3, 4, 5]); - let col10 = Time64MicrosecondArray::from(vec![1, 2, 3, 4, 5]); - let col11 = TimestampMicrosecondArray::from(vec![1, 2, 3, 4, 5]); - let col12 = FixedSizeBinaryArray::try_from_iter( - vec![ - vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10], - vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10], - vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10], - vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10], - vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10], - ] - .into_iter(), - ) - .unwrap(); - assert_eq!(col12.data_type(), &DataType::FixedSizeBinary(10)); - let col13 = FixedSizeBinaryArray::try_from_iter( - vec![ - Uuid::new_v4().as_bytes().to_vec(), - Uuid::new_v4().as_bytes().to_vec(), - Uuid::new_v4().as_bytes().to_vec(), - Uuid::new_v4().as_bytes().to_vec(), - Uuid::new_v4().as_bytes().to_vec(), - ] - .into_iter(), - ) - .unwrap(); - assert_eq!(col13.data_type(), &DataType::FixedSizeBinary(16)); - let col14 = TimestampMicrosecondArray::from(vec![1, 2, 3, 4, 5]).with_timezone(UTC_TIME_ZONE); - let col15 = StructArray::from(vec![ - ( - Arc::new( - Field::new("int", DataType::Int32, false).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - 18.to_string(), - )])), - ), - Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5])) as ArrayRef, - ), - ( - Arc::new( - Field::new("string", DataType::Utf8, false).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - 19.to_string(), - )])), - ), - Arc::new(StringArray::from(vec!["a", "b", "c", "d", "e"])) as ArrayRef, - ), - ]); - let col16 = { - let mut builder = ListBuilder::new(Int32Builder::new()).with_field(Arc::new( - Field::new(LIST_FIELD_NAME, DataType::Int32, false).with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - 20.to_string(), - )])), - )); - builder.append_value([Some(1), Some(2), Some(3), Some(4), Some(5)]); - builder.append_value([Some(1), Some(2), Some(3), Some(4), Some(5)]); - builder.append_value([Some(1), Some(2), Some(3), Some(4), Some(5)]); - builder.append_value([Some(1), Some(2), Some(3), Some(4), Some(5)]); - builder.append_value([Some(1), Some(2), Some(3), Some(4), Some(5)]); - builder.finish() - }; - let col17 = { - let string_builder = StringBuilder::new(); - let int_builder = Int32Builder::with_capacity(4); - let mut builder = MapBuilder::new(None, int_builder, string_builder); - builder.keys().append_value(1); - builder.values().append_value("a"); - builder.append(true).unwrap(); - builder.keys().append_value(2); - builder.values().append_value("b"); - builder.append(true).unwrap(); - builder.keys().append_value(3); - builder.values().append_value("c"); - builder.append(true).unwrap(); - builder.keys().append_value(4); - builder.values().append_value("d"); - builder.append(true).unwrap(); - builder.keys().append_value(5); - builder.values().append_value("e"); - builder.append(true).unwrap(); - let array = builder.finish(); - let (_field, offsets, entries, nulls, ordered) = array.into_parts(); - let new_struct_fields = Fields::from(vec![ - Field::new(MAP_KEY_FIELD_NAME, DataType::Int32, false).with_metadata(HashMap::from([ - (PARQUET_FIELD_ID_META_KEY.to_string(), 21.to_string()), - ])), - Field::new(MAP_VALUE_FIELD_NAME, DataType::Utf8, false).with_metadata(HashMap::from([ - (PARQUET_FIELD_ID_META_KEY.to_string(), 22.to_string()), - ])), - ]); - let entries = { - let (_, arrays, nulls) = entries.into_parts(); - StructArray::new(new_struct_fields.clone(), arrays, nulls) - }; - let field = Arc::new(Field::new( - DEFAULT_MAP_FIELD_NAME, - DataType::Struct(new_struct_fields), - false, - )); - MapArray::new(field, offsets, entries, nulls, ordered) - }; - - let batch = RecordBatch::try_new(schema.clone(), vec![ - Arc::new(col1) as ArrayRef, - Arc::new(col2) as ArrayRef, - Arc::new(col3) as ArrayRef, - Arc::new(col4) as ArrayRef, - Arc::new(col5) as ArrayRef, - Arc::new(col6) as ArrayRef, - Arc::new(col7) as ArrayRef, - Arc::new(col8) as ArrayRef, - Arc::new(col9) as ArrayRef, - Arc::new(col10) as ArrayRef, - Arc::new(col11) as ArrayRef, - Arc::new(col12) as ArrayRef, - Arc::new(col13) as ArrayRef, - Arc::new(col14) as ArrayRef, - Arc::new(col15) as ArrayRef, - Arc::new(col16) as ArrayRef, - Arc::new(col17) as ArrayRef, - ]) - .unwrap(); - data_file_writer.write(batch.clone()).await.unwrap(); - let data_file = data_file_writer.close().await.unwrap(); - - // commit result - let tx = Transaction::new(&table); - let append_action = tx.fast_append().add_data_files(data_file.clone()); - let tx = append_action.apply(tx).unwrap(); - let table = tx.commit(&rest_catalog).await.unwrap(); - - // check result - let batch_stream = table - .scan() - .select(vec![ - "int", - "long", - "float", - "double", - "decimal", - "string", - "boolean", - "binary", - "date", - "time", - "timestamp", - "fixed", - "uuid", - "timestamptz", - "struct", - "list", - "map", - ]) - .build() - .unwrap() - .to_arrow() - .await - .unwrap(); - let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); - assert_eq!(batches.len(), 1); - assert_eq!(batches[0], batch); - - // check result - let batch_stream = table - .scan() - .select_all() - .build() - .unwrap() - .to_arrow() - .await - .unwrap(); - let batches: Vec<_> = batch_stream.try_collect().await.unwrap(); - assert_eq!(batches.len(), 1); - assert_eq!(batches[0], batch); -} diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_test.toml index 8f4860043b..ef5a2e4927 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_test.toml @@ -30,6 +30,14 @@ slt = "df_test/create_table.slt" engine = "df" slt = "df_test/insert_into.slt" +[[steps]] +engine = "df" +slt = "df_test/scan_all_types.slt" + +[[steps]] +engine = "df" +slt = "df_test/basic_queries.slt" + [[steps]] engine = "df" slt = "df_test/binary_predicate_pushdown.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/basic_queries.slt b/crates/sqllogictest/testdata/slts/df_test/basic_queries.slt new file mode 100644 index 0000000000..5d8889f158 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/basic_queries.slt @@ -0,0 +1,212 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test basic SELECT queries with ORDER BY, LIMIT, and various clauses +# This covers the functionality from the original datafusion.rs integration test + +# Create a test table with multiple columns for query testing +statement ok +CREATE TABLE default.default.query_test_table ( + id INT NOT NULL, + name STRING, + score DOUBLE, + category STRING +) + +# Insert test data +query I +INSERT INTO default.default.query_test_table VALUES + (1, 'Alice', 95.5, 'A'), + (2, 'Bob', 87.0, 'B'), + (3, 'Charlie', 92.5, 'A'), + (4, 'Diana', 78.5, 'C'), + (5, 'Eve', 88.0, 'B'), + (6, 'Frank', 91.0, 'A'), + (7, 'Grace', 85.5, 'B'), + (8, 'Henry', 79.0, 'C'), + (9, 'Ivy', 96.5, 'A'), + (10, 'Jack', 82.0, 'C') +---- +10 + +# Test SELECT * with ORDER BY and LIMIT +query ITRT +SELECT * FROM default.default.query_test_table ORDER BY id LIMIT 3 +---- +1 Alice 95.5 A +2 Bob 87 B +3 Charlie 92.5 A + +# Test ORDER BY DESC with LIMIT +query ITRT +SELECT * FROM default.default.query_test_table ORDER BY score DESC LIMIT 5 +---- +9 Ivy 96.5 A +1 Alice 95.5 A +3 Charlie 92.5 A +6 Frank 91 A +5 Eve 88 B + +# Test ORDER BY with multiple columns +query ITR +SELECT id, name, score FROM default.default.query_test_table ORDER BY category, score DESC +---- +9 Ivy 96.5 +1 Alice 95.5 +3 Charlie 92.5 +6 Frank 91 +5 Eve 88 +2 Bob 87 +7 Grace 85.5 +10 Jack 82 +8 Henry 79 +4 Diana 78.5 + +# Test LIMIT with OFFSET +query IT +SELECT id, name FROM default.default.query_test_table ORDER BY id LIMIT 3 OFFSET 5 +---- +6 Frank +7 Grace +8 Henry + +# Test WHERE clause with ORDER BY +query ITR +SELECT id, name, score FROM default.default.query_test_table WHERE score > 90 ORDER BY score DESC +---- +9 Ivy 96.5 +1 Alice 95.5 +3 Charlie 92.5 +6 Frank 91 + +# Test WHERE clause with string comparison +query ITT rowsort +SELECT id, name, category FROM default.default.query_test_table WHERE category = 'A' +---- +1 Alice A +3 Charlie A +6 Frank A +9 Ivy A + +# Test aggregation with GROUP BY +query TIR rowsort +SELECT category, COUNT(*), AVG(score) FROM default.default.query_test_table GROUP BY category +---- +A 4 93.875 +B 3 86.833333333333 +C 3 79.833333333333 + +# Test aggregation with HAVING clause +query TI rowsort +SELECT category, COUNT(*) as cnt FROM default.default.query_test_table GROUP BY category HAVING COUNT(*) >= 4 +---- +A 4 + +# Test DISTINCT +query T rowsort +SELECT DISTINCT category FROM default.default.query_test_table +---- +A +B +C + +# Test COUNT with DISTINCT +query I +SELECT COUNT(DISTINCT category) FROM default.default.query_test_table +---- +3 + +# Test MIN and MAX +query RR +SELECT MIN(score), MAX(score) FROM default.default.query_test_table +---- +78.5 96.5 + +# Test SUM +query R +SELECT SUM(score) FROM default.default.query_test_table +---- +875.5 + +# Test combined WHERE, ORDER BY, and LIMIT +query ITR +SELECT id, name, score FROM default.default.query_test_table +WHERE category IN ('A', 'B') AND score > 85 +ORDER BY score DESC +LIMIT 4 +---- +9 Ivy 96.5 +1 Alice 95.5 +3 Charlie 92.5 +6 Frank 91 + +# Test aliasing +query IT +SELECT id AS user_id, name AS user_name FROM default.default.query_test_table ORDER BY user_id LIMIT 2 +---- +1 Alice +2 Bob + +# Test expression in SELECT +query ITR +SELECT id, name, score * 1.1 AS adjusted_score FROM default.default.query_test_table ORDER BY id LIMIT 3 +---- +1 Alice 105.05 +2 Bob 95.7 +3 Charlie 101.75 + +# Test BETWEEN +query IT rowsort +SELECT id, name FROM default.default.query_test_table WHERE score BETWEEN 85 AND 92 +---- +2 Bob +5 Eve +6 Frank +7 Grace + +# Test NOT IN +query IT rowsort +SELECT id, name FROM default.default.query_test_table WHERE category NOT IN ('A') +---- +10 Jack +2 Bob +4 Diana +5 Eve +7 Grace +8 Henry + +# Test LIKE pattern matching +query IT rowsort +SELECT id, name FROM default.default.query_test_table WHERE name LIKE 'A%' +---- +1 Alice + +# Test OR condition +query ITT rowsort +SELECT id, name, category FROM default.default.query_test_table WHERE category = 'A' OR score > 90 +---- +1 Alice A +3 Charlie A +6 Frank A +9 Ivy A + +# Test AND condition +query ITR rowsort +SELECT id, name, score FROM default.default.query_test_table WHERE category = 'A' AND score > 93 +---- +1 Alice 95.5 +9 Ivy 96.5 diff --git a/crates/sqllogictest/testdata/slts/df_test/scan_all_types.slt b/crates/sqllogictest/testdata/slts/df_test/scan_all_types.slt new file mode 100644 index 0000000000..d93e461333 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/scan_all_types.slt @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test scanning all primitive types supported by Iceberg +# This tests INT, BIGINT, FLOAT, DOUBLE, DECIMAL, STRING, BOOLEAN, DATE, TIMESTAMP + +# Create a table with all primitive types +statement ok +CREATE TABLE default.default.all_primitive_types ( + col_int INT NOT NULL, + col_long BIGINT NOT NULL, + col_float FLOAT NOT NULL, + col_double DOUBLE NOT NULL, + col_decimal DECIMAL(10, 2), + col_string STRING, + col_boolean BOOLEAN +) + +# Verify the table is initially empty +query IIRRTTT rowsort +SELECT * FROM default.default.all_primitive_types +---- + +# Insert test data with various primitive types +query I +INSERT INTO default.default.all_primitive_types VALUES + (1, 100, 1.5, 2.5, 123.45, 'hello', true), + (2, 200, 2.5, 3.5, 234.56, 'world', false), + (3, 300, 3.5, 4.5, 345.67, NULL, true), + (-1, -100, -1.5, -2.5, -123.45, 'negative', false), + (0, 0, 0.0, 0.0, 0.00, 'zero', NULL) +---- +5 + +# Verify all inserted rows +query IIRRTTT rowsort +SELECT * FROM default.default.all_primitive_types +---- +-1 -100 -1.5 -2.5 -123.45 negative false +0 0 0 0 0 zero NULL +1 100 1.5 2.5 123.45 hello true +2 200 2.5 3.5 234.56 world false +3 300 3.5 4.5 345.67 NULL true + +# Test selecting specific columns with string filter +query IT rowsort +SELECT col_int, col_string FROM default.default.all_primitive_types WHERE col_string = 'hello' +---- +1 hello + +# Test numeric comparisons +query IR rowsort +SELECT col_int, col_decimal FROM default.default.all_primitive_types WHERE col_decimal > 200.00 +---- +2 234.56 +3 345.67 + +# Test filtering on int column +query IIRRTTT rowsort +SELECT * FROM default.default.all_primitive_types WHERE col_int >= 0 ORDER BY col_int +---- +0 0 0 0 0 zero NULL +1 100 1.5 2.5 123.45 hello true +2 200 2.5 3.5 234.56 world false +3 300 3.5 4.5 345.67 NULL true + +# Test filtering with NULL +query IT rowsort +SELECT col_int, col_string FROM default.default.all_primitive_types WHERE col_string IS NOT NULL +---- +-1 negative +0 zero +1 hello +2 world + +# Test aggregations on numeric types +query IIRR +SELECT COUNT(*), SUM(col_int), AVG(col_float), MAX(col_double) FROM default.default.all_primitive_types +---- +5 5 1.2 4.5 + +# Create a table with date and timestamp types +statement ok +CREATE TABLE default.default.temporal_types ( + id INT NOT NULL, + col_date DATE, + col_timestamp TIMESTAMP +) + +# Insert temporal data +query I +INSERT INTO default.default.temporal_types VALUES + (1, '2024-01-15', '2024-01-15 10:30:00'), + (2, '2024-06-20', '2024-06-20 14:45:30'), + (3, '2023-12-31', '2023-12-31 23:59:59'), + (4, NULL, NULL) +---- +4 + +# Verify temporal data +query IDP rowsort +SELECT * FROM default.default.temporal_types +---- +1 2024-01-15 2024-01-15T10:30:00 +2 2024-06-20 2024-06-20T14:45:30 +3 2023-12-31 2023-12-31T23:59:59 +4 NULL NULL + +# Test date comparisons +query ID rowsort +SELECT id, col_date FROM default.default.temporal_types WHERE col_date >= '2024-01-01' +---- +1 2024-01-15 +2 2024-06-20 diff --git a/crates/test_utils/Cargo.toml b/crates/test_utils/Cargo.toml index 58f9307790..383448881f 100644 --- a/crates/test_utils/Cargo.toml +++ b/crates/test_utils/Cargo.toml @@ -27,7 +27,6 @@ repository = { workspace = true } [dependencies] iceberg = { workspace = true } -tracing = { workspace = true } tracing-subscriber = { workspace = true } [features] diff --git a/crates/test_utils/src/cmd.rs b/crates/test_utils/src/cmd.rs deleted file mode 100644 index e5e3f09e4b..0000000000 --- a/crates/test_utils/src/cmd.rs +++ /dev/null @@ -1,58 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -use std::process::Command; - -use tracing::{error, info}; - -pub fn run_command(mut cmd: Command, desc: impl ToString) -> bool { - let desc = desc.to_string(); - info!("Starting to {}, command: {:?}", &desc, cmd); - let exit = cmd.status().unwrap(); - if exit.success() { - info!("{} succeed!", desc); - true - } else { - error!("{} failed: {:?}", desc, exit); - false - } -} - -pub fn get_cmd_output_result(mut cmd: Command, desc: impl ToString) -> Result { - let desc = desc.to_string(); - info!("Starting to {}, command: {:?}", &desc, cmd); - let result = cmd.output(); - match result { - Ok(output) => { - if output.status.success() { - info!("{} succeed!", desc); - Ok(String::from_utf8(output.stdout).unwrap()) - } else { - Err(format!("{} failed with rc: {:?}", desc, output.status)) - } - } - Err(err) => Err(format!("{} failed with error: {}", desc, { err })), - } -} - -pub fn get_cmd_output(cmd: Command, desc: impl ToString) -> String { - let result = get_cmd_output_result(cmd, desc); - match result { - Ok(output_str) => output_str, - Err(err) => panic!("{}", err), - } -} diff --git a/crates/test_utils/src/docker.rs b/crates/test_utils/src/docker.rs deleted file mode 100644 index 078966ddb8..0000000000 --- a/crates/test_utils/src/docker.rs +++ /dev/null @@ -1,165 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -use std::net::IpAddr; -use std::process::Command; - -use tracing::error; - -use crate::cmd::{get_cmd_output, get_cmd_output_result, run_command}; - -/// A utility to manage the lifecycle of `docker compose`. -/// -/// It will start `docker compose` when calling the `run` method and will be stopped via [`Drop`]. -#[derive(Debug)] -pub struct DockerCompose { - project_name: String, - docker_compose_dir: String, -} - -impl DockerCompose { - pub fn new(project_name: impl ToString, docker_compose_dir: impl ToString) -> Self { - Self { - project_name: project_name.to_string(), - docker_compose_dir: docker_compose_dir.to_string(), - } - } - - pub fn project_name(&self) -> &str { - self.project_name.as_str() - } - - fn get_os_arch() -> String { - let mut cmd = Command::new("docker"); - cmd.arg("info") - .arg("--format") - .arg("{{.OSType}}/{{.Architecture}}"); - - let result = get_cmd_output_result(cmd, "Get os arch".to_string()); - match result { - Ok(value) => value.trim().to_string(), - Err(_err) => { - // docker/podman do not consistently place OSArch info in the same json path across OS and versions - // Below tries an alternative path if the above path fails - let mut alt_cmd = Command::new("docker"); - alt_cmd - .arg("info") - .arg("--format") - .arg("{{.Version.OsArch}}"); - get_cmd_output(alt_cmd, "Get os arch".to_string()) - .trim() - .to_string() - } - } - } - - pub fn up(&self) { - let mut cmd = Command::new("docker"); - cmd.current_dir(&self.docker_compose_dir); - - cmd.env("DOCKER_DEFAULT_PLATFORM", Self::get_os_arch()); - - cmd.args(vec![ - "compose", - "-p", - self.project_name.as_str(), - "up", - "-d", - "--build", - "--wait", - "--timeout", - "1200000", - ]); - - let ret = run_command( - cmd, - format!( - "Starting docker compose in {}, project name: {}", - self.docker_compose_dir, self.project_name - ), - ); - - if !ret { - let mut cmd = Command::new("docker"); - cmd.current_dir(&self.docker_compose_dir); - - cmd.env("DOCKER_DEFAULT_PLATFORM", Self::get_os_arch()); - - cmd.args(vec![ - "compose", - "-p", - self.project_name.as_str(), - "logs", - "spark-iceberg", - ]); - run_command(cmd, "Docker compose logs"); - panic!("Docker compose up failed!") - } - } - - pub fn down(&self) { - let mut cmd = Command::new("docker"); - cmd.current_dir(&self.docker_compose_dir); - - cmd.args(vec![ - "compose", - "-p", - self.project_name.as_str(), - "down", - "-v", - "--remove-orphans", - ]); - - let ret = run_command( - cmd, - format!( - "Stopping docker compose in {}, project name: {}", - self.docker_compose_dir, self.project_name - ), - ); - - if !ret { - panic!("Failed to stop docker compose") - } - } - - pub fn get_container_ip(&self, service_name: impl AsRef) -> IpAddr { - let container_name = format!("{}-{}-1", self.project_name, service_name.as_ref()); - let mut cmd = Command::new("docker"); - cmd.arg("inspect") - .arg("-f") - .arg("{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}") - .arg(&container_name); - - let ip_result = get_cmd_output(cmd, format!("Get container ip of {container_name}")) - .trim() - .parse::(); - match ip_result { - Ok(ip) => ip, - Err(e) => { - error!("Invalid IP, {e}"); - panic!("Failed to parse IP for {container_name}") - } - } - } -} - -impl Drop for DockerCompose { - fn drop(&mut self) { - self.down() - } -} diff --git a/crates/test_utils/src/lib.rs b/crates/test_utils/src/lib.rs index 54ec54193f..4511231d76 100644 --- a/crates/test_utils/src/lib.rs +++ b/crates/test_utils/src/lib.rs @@ -19,11 +19,6 @@ //! //! It's not intended for use outside of `iceberg-rust`. -#[cfg(feature = "tests")] -mod cmd; -#[cfg(feature = "tests")] -pub mod docker; - #[cfg(feature = "tests")] pub use common::*; From ac4e4b08c73d3545a2b6ae8be67e10edd02a5f94 Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Wed, 28 Jan 2026 15:02:18 +0800 Subject: [PATCH 176/891] Let integration tests run in parallel (#2076) ## Which issue does this PR close? - Closes #2075 . ## What changes are included in this PR? Refactor integrationt-tests module so that they can run in parallel. ## Are these changes tested? Yes, ut. --- .../tests/{shared_tests => common}/mod.rs | 9 ++----- .../conflict_commit_test.rs | 9 ++++--- .../{shared_tests => }/read_evolved_schema.rs | 5 ++-- .../read_positional_deletes.rs | 5 ++-- crates/integration_tests/tests/shared.rs | 26 ------------------- 5 files changed, 11 insertions(+), 43 deletions(-) rename crates/integration_tests/tests/{shared_tests => common}/mod.rs (92%) rename crates/integration_tests/tests/{shared_tests => }/conflict_commit_test.rs (97%) rename crates/integration_tests/tests/{shared_tests => }/read_evolved_schema.rs (98%) rename crates/integration_tests/tests/{shared_tests => }/read_positional_deletes.rs (96%) delete mode 100644 crates/integration_tests/tests/shared.rs diff --git a/crates/integration_tests/tests/shared_tests/mod.rs b/crates/integration_tests/tests/common/mod.rs similarity index 92% rename from crates/integration_tests/tests/shared_tests/mod.rs rename to crates/integration_tests/tests/common/mod.rs index 8fd56ab521..00039d8896 100644 --- a/crates/integration_tests/tests/shared_tests/mod.rs +++ b/crates/integration_tests/tests/common/mod.rs @@ -20,15 +20,10 @@ use std::collections::HashMap; use iceberg::spec::{NestedField, PrimitiveType, Schema, Type}; use iceberg::{Catalog, CatalogBuilder, Namespace, NamespaceIdent}; use iceberg_catalog_rest::RestCatalogBuilder; - -use crate::get_shared_containers; - -mod conflict_commit_test; -mod read_evolved_schema; -mod read_positional_deletes; +use iceberg_integration_tests::get_test_fixture; pub async fn random_ns() -> Namespace { - let fixture = get_shared_containers(); + let fixture = get_test_fixture(); let rest_catalog = RestCatalogBuilder::default() .load("rest", fixture.catalog_config.clone()) .await diff --git a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs b/crates/integration_tests/tests/conflict_commit_test.rs similarity index 97% rename from crates/integration_tests/tests/shared_tests/conflict_commit_test.rs rename to crates/integration_tests/tests/conflict_commit_test.rs index fc529cc3d2..d171217b04 100644 --- a/crates/integration_tests/tests/shared_tests/conflict_commit_test.rs +++ b/crates/integration_tests/tests/conflict_commit_test.rs @@ -17,9 +17,12 @@ //! Integration tests for rest catalog. +mod common; + use std::sync::Arc; use arrow_array::{ArrayRef, BooleanArray, Int32Array, RecordBatch, StringArray}; +use common::{random_ns, test_schema}; use futures::TryStreamExt; use iceberg::transaction::{ApplyTransactionAction, Transaction}; use iceberg::writer::base_writer::data_file_writer::DataFileWriterBuilder; @@ -31,14 +34,12 @@ use iceberg::writer::file_writer::rolling_writer::RollingFileWriterBuilder; use iceberg::writer::{IcebergWriter, IcebergWriterBuilder}; use iceberg::{Catalog, CatalogBuilder, TableCreation}; use iceberg_catalog_rest::RestCatalogBuilder; +use iceberg_integration_tests::get_test_fixture; use parquet::file::properties::WriterProperties; -use crate::get_shared_containers; -use crate::shared_tests::{random_ns, test_schema}; - #[tokio::test] async fn test_append_data_file_conflict() { - let fixture = get_shared_containers(); + let fixture = get_test_fixture(); let rest_catalog = RestCatalogBuilder::default() .load("rest", fixture.catalog_config.clone()) .await diff --git a/crates/integration_tests/tests/shared_tests/read_evolved_schema.rs b/crates/integration_tests/tests/read_evolved_schema.rs similarity index 98% rename from crates/integration_tests/tests/shared_tests/read_evolved_schema.rs rename to crates/integration_tests/tests/read_evolved_schema.rs index fe8a488fa3..62ec610a63 100644 --- a/crates/integration_tests/tests/shared_tests/read_evolved_schema.rs +++ b/crates/integration_tests/tests/read_evolved_schema.rs @@ -23,13 +23,12 @@ use iceberg::expr::Reference; use iceberg::spec::Datum; use iceberg::{Catalog, CatalogBuilder, TableIdent}; use iceberg_catalog_rest::RestCatalogBuilder; +use iceberg_integration_tests::get_test_fixture; use ordered_float::OrderedFloat; -use crate::get_shared_containers; - #[tokio::test] async fn test_evolved_schema() { - let fixture = get_shared_containers(); + let fixture = get_test_fixture(); let rest_catalog = RestCatalogBuilder::default() .load("rest", fixture.catalog_config.clone()) .await diff --git a/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs b/crates/integration_tests/tests/read_positional_deletes.rs similarity index 96% rename from crates/integration_tests/tests/shared_tests/read_positional_deletes.rs rename to crates/integration_tests/tests/read_positional_deletes.rs index 318b06bf0e..96add41299 100644 --- a/crates/integration_tests/tests/shared_tests/read_positional_deletes.rs +++ b/crates/integration_tests/tests/read_positional_deletes.rs @@ -20,12 +20,11 @@ use futures::TryStreamExt; use iceberg::{Catalog, CatalogBuilder, TableIdent}; use iceberg_catalog_rest::RestCatalogBuilder; - -use crate::get_shared_containers; +use iceberg_integration_tests::get_test_fixture; #[tokio::test] async fn test_read_table_with_positional_deletes() { - let fixture = get_shared_containers(); + let fixture = get_test_fixture(); let rest_catalog = RestCatalogBuilder::default() .load("rest", fixture.catalog_config.clone()) .await diff --git a/crates/integration_tests/tests/shared.rs b/crates/integration_tests/tests/shared.rs deleted file mode 100644 index ab85e4d1ae..0000000000 --- a/crates/integration_tests/tests/shared.rs +++ /dev/null @@ -1,26 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -use iceberg_integration_tests::{GlobalTestFixture, get_test_fixture}; - -pub mod shared_tests; - -/// Returns a reference to the shared test fixture. -/// This assumes Docker containers are started externally via `make docker-up`. -pub fn get_shared_containers() -> &'static GlobalTestFixture { - get_test_fixture() -} From 4c10b027e962aa99198ede75e710c89abd9f17c6 Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Thu, 29 Jan 2026 08:43:19 +0800 Subject: [PATCH 177/891] Use nextest to speedup tests (#2078) ## Which issue does this PR close? - Closes #2077. ## What changes are included in this PR? Use nextest instead of cargo test to run tests. ## Are these changes tested? Yes, ut. --- .github/workflows/ci.yml | 16 +++++++++++++++- Makefile | 8 +++++++- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 829f64030e..fe7476a80e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -146,12 +146,26 @@ jobs: with: key: ${{ matrix.test-suite.name }} + - name: Install cargo-nextest + if: matrix.test-suite.name == 'default' + uses: taiki-e/install-action@v2 + with: + tool: cargo-nextest + - name: Start Docker containers if: matrix.test-suite.name == 'default' run: make docker-up - name: Run tests - run: cargo test --no-fail-fast ${{ matrix.test-suite.args }} + env: + # Disable debug info to speed up compilation and reduce artifact size + RUSTFLAGS: "-C debuginfo=0" + run: | + if [ "${{ matrix.test-suite.name }}" = "default" ]; then + cargo nextest run ${{ matrix.test-suite.args }} + else + cargo test --no-fail-fast ${{ matrix.test-suite.args }} + fi - name: Stop Docker containers if: always() && matrix.test-suite.name == 'default' diff --git a/Makefile b/Makefile index 14093e1829..6579140ff5 100644 --- a/Makefile +++ b/Makefile @@ -32,6 +32,12 @@ install-cargo-machete: cargo-machete: install-cargo-machete cargo machete +install-cargo-nextest: + cargo install --locked cargo-nextest + +nextest: install-cargo-nextest + cargo nextest run --all-targets --all-features --workspace + install-taplo-cli: cargo install taplo-cli@0.9.3 @@ -57,7 +63,7 @@ unit-test: doc-test test: docker-up @trap '$(MAKE) docker-down' EXIT; \ - cargo test --no-fail-fast --all-targets --all-features --workspace + $(MAKE) nextest clean: cargo clean From f93f01334cf34e618d157bbe0a10a108be2ddb0a Mon Sep 17 00:00:00 2001 From: emkornfield Date: Wed, 28 Jan 2026 17:32:30 -0800 Subject: [PATCH 178/891] feat: Factor out CompressionCodec to common location and add Gzip (#2081) ## Which issue does this PR close? As discussed in the PR for [allowing compressed metadata.json](https://github.com/apache/iceberg-rust/pull/1876#discussion_r2693266830) writes we want CompressionCodec in a central place so it can be used in puffin and for other compression use-cases. Happy to move it to `spec/` as suggested in the original comment but this seemed more natural. ## What changes are included in this PR? This moves compression.rs to a top level (seems like the best location for common code but please let me know if there is a better place. - It adds Gzip as a compression option and replaces current direct use the GzEncoder package. - It adds validation to puffin that Gzip is not currently supported (per spec) ## Are these changes tested? Added unit tests to cover additions and existing tests cover the rest. --- .../iceberg/src/{puffin => }/compression.rs | 79 +++++++++++-------- crates/iceberg/src/lib.rs | 1 + crates/iceberg/src/puffin/metadata.rs | 32 +++++++- crates/iceberg/src/puffin/mod.rs | 47 ++++++++++- crates/iceberg/src/puffin/reader.rs | 41 +++++++++- crates/iceberg/src/puffin/test_utils.rs | 2 +- crates/iceberg/src/puffin/writer.rs | 30 +++++-- crates/iceberg/src/spec/table_metadata.rs | 33 ++++---- 8 files changed, 204 insertions(+), 61 deletions(-) rename crates/iceberg/src/{puffin => }/compression.rs (57%) diff --git a/crates/iceberg/src/puffin/compression.rs b/crates/iceberg/src/compression.rs similarity index 57% rename from crates/iceberg/src/puffin/compression.rs rename to crates/iceberg/src/compression.rs index a9a56ef12c..1218d81df6 100644 --- a/crates/iceberg/src/puffin/compression.rs +++ b/crates/iceberg/src/compression.rs @@ -15,6 +15,13 @@ // specific language governing permissions and limitations // under the License. +//! Compression codec support for data compression and decompression. + +use std::io::{Read, Write}; + +use flate2::Compression; +use flate2::read::GzDecoder; +use flate2::write::GzEncoder; use serde::{Deserialize, Serialize}; use crate::{Error, ErrorKind, Result}; @@ -30,6 +37,8 @@ pub enum CompressionCodec { Lz4, /// Zstandard single compression frame with content size present Zstd, + /// Gzip compression + Gzip, } impl CompressionCodec { @@ -40,8 +49,11 @@ impl CompressionCodec { ErrorKind::FeatureUnsupported, "LZ4 decompression is not supported currently", )), - CompressionCodec::Zstd => { - let decompressed = zstd::stream::decode_all(&bytes[..])?; + CompressionCodec::Zstd => Ok(zstd::stream::decode_all(&bytes[..])?), + CompressionCodec::Gzip => { + let mut decoder = GzDecoder::new(&bytes[..]); + let mut decompressed = Vec::new(); + decoder.read_to_end(&mut decompressed)?; Ok(decompressed) } } @@ -60,8 +72,12 @@ impl CompressionCodec { encoder.include_checksum(true)?; encoder.set_pledged_src_size(Some(bytes.len().try_into()?))?; std::io::copy(&mut &bytes[..], &mut encoder)?; - let compressed = encoder.finish()?; - Ok(compressed) + Ok(encoder.finish()?) + } + CompressionCodec::Gzip => { + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + encoder.write_all(&bytes)?; + Ok(encoder.finish()?) } } } @@ -73,51 +89,48 @@ impl CompressionCodec { #[cfg(test)] mod tests { - use crate::puffin::compression::CompressionCodec; + use super::CompressionCodec; #[tokio::test] async fn test_compression_codec_none() { - let compression_codec = CompressionCodec::None; let bytes_vec = [0_u8; 100].to_vec(); - let compressed = compression_codec.compress(bytes_vec.clone()).unwrap(); + let codec = CompressionCodec::None; + let compressed = codec.compress(bytes_vec.clone()).unwrap(); assert_eq!(bytes_vec, compressed); - - let decompressed = compression_codec.decompress(compressed.clone()).unwrap(); - assert_eq!(compressed, decompressed) + let decompressed = codec.decompress(compressed).unwrap(); + assert_eq!(bytes_vec, decompressed); } #[tokio::test] - async fn test_compression_codec_lz4() { - let compression_codec = CompressionCodec::Lz4; + async fn test_compression_codec_compress() { let bytes_vec = [0_u8; 100].to_vec(); - assert_eq!( - compression_codec - .compress(bytes_vec.clone()) - .unwrap_err() - .to_string(), - "FeatureUnsupported => LZ4 compression is not supported currently", - ); - - assert_eq!( - compression_codec - .decompress(bytes_vec.clone()) - .unwrap_err() - .to_string(), - "FeatureUnsupported => LZ4 decompression is not supported currently", - ) + let compression_codecs = [CompressionCodec::Zstd, CompressionCodec::Gzip]; + + for codec in compression_codecs { + let compressed = codec.compress(bytes_vec.clone()).unwrap(); + assert!(compressed.len() < bytes_vec.len()); + let decompressed = codec.decompress(compressed).unwrap(); + assert_eq!(decompressed, bytes_vec); + } } #[tokio::test] - async fn test_compression_codec_zstd() { - let compression_codec = CompressionCodec::Zstd; + async fn test_compression_codec_unsupported() { + let unsupported_codecs = [(CompressionCodec::Lz4, "LZ4")]; let bytes_vec = [0_u8; 100].to_vec(); - let compressed = compression_codec.compress(bytes_vec.clone()).unwrap(); - assert!(compressed.len() < bytes_vec.len()); + for (codec, name) in unsupported_codecs { + assert_eq!( + codec.compress(bytes_vec.clone()).unwrap_err().to_string(), + format!("FeatureUnsupported => {name} compression is not supported currently"), + ); - let decompressed = compression_codec.decompress(compressed.clone()).unwrap(); - assert_eq!(decompressed, bytes_vec) + assert_eq!( + codec.decompress(bytes_vec.clone()).unwrap_err().to_string(), + format!("FeatureUnsupported => {name} decompression is not supported currently"), + ); + } } } diff --git a/crates/iceberg/src/lib.rs b/crates/iceberg/src/lib.rs index 8d8f40f72d..8b345deb6e 100644 --- a/crates/iceberg/src/lib.rs +++ b/crates/iceberg/src/lib.rs @@ -77,6 +77,7 @@ pub mod table; mod avro; pub mod cache; +pub mod compression; pub mod io; pub mod spec; diff --git a/crates/iceberg/src/puffin/metadata.rs b/crates/iceberg/src/puffin/metadata.rs index 15a8e9b33a..b09f2c7c1e 100644 --- a/crates/iceberg/src/puffin/metadata.rs +++ b/crates/iceberg/src/puffin/metadata.rs @@ -20,8 +20,8 @@ use std::collections::{HashMap, HashSet}; use bytes::Bytes; use serde::{Deserialize, Serialize}; +use crate::compression::CompressionCodec; use crate::io::{FileRead, InputFile}; -use crate::puffin::compression::CompressionCodec; use crate::{Error, ErrorKind, Result}; /// Human-readable identification of the application writing the file, along with its version. @@ -954,4 +954,34 @@ mod tests { assert_eq!(file_metadata, zstd_compressed_metric_file_metadata()); } + + #[tokio::test] + async fn test_gzip_compression_allowed_in_metadata() { + let temp_dir = TempDir::new().unwrap(); + + // Create a JSON payload with Gzip compression codec + // Metadata should be readable, but accessing the blob will fail + let payload = r#"{ + "blobs": [ + { + "type": "test-type", + "fields": [1], + "snapshot-id": 1, + "sequence-number": 1, + "offset": 4, + "length": 10, + "compression-codec": "gzip" + } + ] + }"#; + + let input_file = input_file_with_payload(&temp_dir, payload).await; + + // Reading metadata should succeed (lazy validation) + let result = FileMetadata::read(&input_file).await; + assert!(result.is_ok()); + let metadata = result.unwrap(); + assert_eq!(metadata.blobs.len(), 1); + assert_eq!(metadata.blobs[0].compression_codec, CompressionCodec::Gzip); + } } diff --git a/crates/iceberg/src/puffin/mod.rs b/crates/iceberg/src/puffin/mod.rs index 0a03781655..854d4070ff 100644 --- a/crates/iceberg/src/puffin/mod.rs +++ b/crates/iceberg/src/puffin/mod.rs @@ -19,11 +19,38 @@ #![deny(missing_docs)] +use crate::{Error, ErrorKind, Result}; + mod blob; pub use blob::{APACHE_DATASKETCHES_THETA_V1, Blob, DELETION_VECTOR_V1}; -mod compression; -pub use compression::CompressionCodec; +pub use crate::compression::CompressionCodec; + +/// Compression codecs supported by the Puffin spec. +const SUPPORTED_PUFFIN_CODECS: &[CompressionCodec] = &[ + CompressionCodec::None, + CompressionCodec::Lz4, + CompressionCodec::Zstd, +]; + +/// Validates that the compression codec is supported for Puffin files. +/// Returns an error if the codec is not supported. +fn validate_puffin_compression(codec: CompressionCodec) -> Result<()> { + if !SUPPORTED_PUFFIN_CODECS.contains(&codec) { + let supported_names: Vec = SUPPORTED_PUFFIN_CODECS + .iter() + .map(|c| format!("{c:?}")) + .collect(); + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Compression codec {codec:?} is not supported for Puffin files. Only {} are supported.", + supported_names.join(", ") + ), + )); + } + Ok(()) +} mod metadata; pub use metadata::{BlobMetadata, CREATED_BY_PROPERTY, FileMetadata}; @@ -36,3 +63,19 @@ pub use writer::PuffinWriter; #[cfg(test)] mod test_utils; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_puffin_codec_validation() { + // All codecs in SUPPORTED_PUFFIN_CODECS should be valid + for codec in SUPPORTED_PUFFIN_CODECS { + assert!(validate_puffin_compression(*codec).is_ok()); + } + + // Gzip should not be supported for Puffin files + assert!(validate_puffin_compression(CompressionCodec::Gzip).is_err()); + } +} diff --git a/crates/iceberg/src/puffin/reader.rs b/crates/iceberg/src/puffin/reader.rs index dce53d93f0..a6308dcebf 100644 --- a/crates/iceberg/src/puffin/reader.rs +++ b/crates/iceberg/src/puffin/reader.rs @@ -17,6 +17,7 @@ use tokio::sync::OnceCell; +use super::validate_puffin_compression; use crate::Result; use crate::io::{FileRead, InputFile}; use crate::puffin::blob::Blob; @@ -46,11 +47,13 @@ impl PuffinReader { /// Returns blob pub async fn blob(&self, blob_metadata: &BlobMetadata) -> Result { + validate_puffin_compression(blob_metadata.compression_codec)?; + let file_read = self.input_file.reader().await?; let start = blob_metadata.offset; let end = start + blob_metadata.length; - let bytes = file_read.read(start..end).await?.to_vec(); - let data = blob_metadata.compression_codec.decompress(bytes)?; + let bytes = file_read.read(start..end).await?; + let data = blob_metadata.compression_codec.decompress(bytes.to_vec())?; Ok(Blob { r#type: blob_metadata.r#type.clone(), @@ -65,7 +68,11 @@ impl PuffinReader { #[cfg(test)] mod tests { + use std::collections::HashMap; + use crate::ErrorKind; + use crate::compression::CompressionCodec; + use crate::puffin::metadata::BlobMetadata; use crate::puffin::reader::PuffinReader; use crate::puffin::test_utils::{ blob_0, blob_1, java_uncompressed_metric_input_file, @@ -122,4 +129,34 @@ mod tests { blob_1(), ) } + + #[tokio::test] + async fn test_gzip_compression_rejected_on_blob_access() { + // Use a real puffin file + let input_file = java_uncompressed_metric_input_file(); + let reader = PuffinReader::new(input_file); + + // Create a BlobMetadata with Gzip compression + let gzip_blob_metadata = BlobMetadata { + r#type: "test-type".to_string(), + fields: vec![1], + snapshot_id: 1, + sequence_number: 1, + offset: 4, + length: 10, + compression_codec: CompressionCodec::Gzip, + properties: HashMap::new(), + }; + + // Attempting to access the blob should fail + let result = reader.blob(&gzip_blob_metadata).await; + assert!(result.is_err()); + let err = result.unwrap_err(); + assert_eq!(err.kind(), ErrorKind::DataInvalid); + assert!(err.to_string().contains("Gzip")); + assert!( + err.to_string() + .contains("is not supported for Puffin files") + ); + } } diff --git a/crates/iceberg/src/puffin/test_utils.rs b/crates/iceberg/src/puffin/test_utils.rs index ca91fa217b..a3232f53ef 100644 --- a/crates/iceberg/src/puffin/test_utils.rs +++ b/crates/iceberg/src/puffin/test_utils.rs @@ -18,8 +18,8 @@ use std::collections::HashMap; use super::blob::Blob; +use crate::compression::CompressionCodec; use crate::io::{FileIOBuilder, InputFile}; -use crate::puffin::compression::CompressionCodec; use crate::puffin::metadata::{BlobMetadata, CREATED_BY_PROPERTY, FileMetadata}; const JAVA_TESTDATA: &str = "testdata/puffin/java-generated"; diff --git a/crates/iceberg/src/puffin/writer.rs b/crates/iceberg/src/puffin/writer.rs index f68efda889..c8e4233a94 100644 --- a/crates/iceberg/src/puffin/writer.rs +++ b/crates/iceberg/src/puffin/writer.rs @@ -19,10 +19,11 @@ use std::collections::{HashMap, HashSet}; use bytes::Bytes; +use super::validate_puffin_compression; use crate::Result; +use crate::compression::CompressionCodec; use crate::io::{FileWrite, OutputFile}; use crate::puffin::blob::Blob; -use crate::puffin::compression::CompressionCodec; use crate::puffin::metadata::{BlobMetadata, FileMetadata, Flag}; /// Puffin writer @@ -64,6 +65,8 @@ impl PuffinWriter { /// Adds blob to Puffin file pub async fn add(&mut self, blob: Blob, compression_codec: CompressionCodec) -> Result<()> { + validate_puffin_compression(compression_codec)?; + self.write_header_once().await?; let offset = self.num_bytes_written; @@ -114,8 +117,7 @@ impl PuffinWriter { properties: self.properties.clone(), }; let json = serde_json::to_string::(&file_metadata)?; - let bytes = json.as_bytes(); - self.footer_compression_codec.compress(bytes.to_vec()) + self.footer_compression_codec.compress(json.into_bytes()) } fn flags_bytes(&self) -> [u8; FileMetadata::FOOTER_STRUCT_FLAGS_LENGTH as usize] { @@ -148,10 +150,9 @@ mod tests { use tempfile::TempDir; - use crate::Result; + use crate::compression::CompressionCodec; use crate::io::{FileIOBuilder, InputFile, OutputFile}; use crate::puffin::blob::Blob; - use crate::puffin::compression::CompressionCodec; use crate::puffin::metadata::FileMetadata; use crate::puffin::reader::PuffinReader; use crate::puffin::test_utils::{ @@ -161,6 +162,7 @@ mod tests { zstd_compressed_metric_file_metadata, }; use crate::puffin::writer::PuffinWriter; + use crate::{ErrorKind, Result}; async fn write_puffin_file( temp_dir: &TempDir, @@ -331,4 +333,22 @@ mod tests { ) .await } + + #[tokio::test] + async fn test_gzip_compression_rejected() { + let temp_dir = TempDir::new().unwrap(); + let blobs = vec![blob_0()]; + let blobs_with_compression = blobs_with_compression(blobs, CompressionCodec::Gzip); + + let result = write_puffin_file(&temp_dir, blobs_with_compression, file_properties()).await; + + assert!(result.is_err()); + let err = result.unwrap_err(); + assert_eq!(err.kind(), ErrorKind::DataInvalid); + assert!(err.to_string().contains("Gzip")); + assert!( + err.to_string() + .contains("is not supported for Puffin files") + ); + } } diff --git a/crates/iceberg/src/spec/table_metadata.rs b/crates/iceberg/src/spec/table_metadata.rs index d3836b2f42..28f753e9f1 100644 --- a/crates/iceberg/src/spec/table_metadata.rs +++ b/crates/iceberg/src/spec/table_metadata.rs @@ -22,12 +22,10 @@ use std::cmp::Ordering; use std::collections::HashMap; use std::fmt::{Display, Formatter}; use std::hash::Hash; -use std::io::Read as _; use std::sync::Arc; use _serde::TableMetadataEnum; use chrono::{DateTime, Utc}; -use flate2::read::GzDecoder; use serde::{Deserialize, Serialize}; use serde_repr::{Deserialize_repr, Serialize_repr}; use uuid::Uuid; @@ -39,6 +37,7 @@ use super::{ SnapshotRef, SnapshotRetention, SortOrder, SortOrderRef, StatisticsFile, StructType, TableProperties, }; +use crate::compression::CompressionCodec; use crate::error::{Result, timestamp_ms_to_utc}; use crate::io::FileIO; use crate::spec::EncryptedKey; @@ -445,16 +444,16 @@ impl TableMetadata { && metadata_content[0] == 0x1F && metadata_content[1] == 0x8B { - let mut decoder = GzDecoder::new(metadata_content.as_ref()); - let mut decompressed_data = Vec::new(); - decoder.read_to_end(&mut decompressed_data).map_err(|e| { - Error::new( - ErrorKind::DataInvalid, - "Trying to read compressed metadata file", - ) - .with_context("file_path", metadata_location) - .with_source(e) - })?; + let decompressed_data = CompressionCodec::Gzip + .decompress(metadata_content.to_vec()) + .map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + "Trying to read compressed metadata file", + ) + .with_context("file_path", metadata_location) + .with_source(e) + })?; serde_json::from_slice(&decompressed_data)? } else { serde_json::from_slice(&metadata_content)? @@ -1559,7 +1558,6 @@ impl SnapshotLog { mod tests { use std::collections::HashMap; use std::fs; - use std::io::Write as _; use std::sync::Arc; use anyhow::Result; @@ -1569,6 +1567,7 @@ mod tests { use uuid::Uuid; use super::{FormatVersion, MetadataLog, SnapshotLog, TableMetadataBuilder}; + use crate::compression::CompressionCodec; use crate::io::FileIOBuilder; use crate::spec::table_metadata::TableMetadata; use crate::spec::{ @@ -3576,10 +3575,10 @@ mod tests { let original_metadata: TableMetadata = get_test_table_metadata("TableMetadataV2Valid.json"); let json = serde_json::to_string(&original_metadata).unwrap(); - let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); - encoder.write_all(json.as_bytes()).unwrap(); - std::fs::write(&metadata_location, encoder.finish().unwrap()) - .expect("failed to write metadata"); + let compressed = CompressionCodec::Gzip + .compress(json.into_bytes()) + .expect("failed to compress metadata"); + std::fs::write(&metadata_location, &compressed).expect("failed to write metadata"); // Read the metadata back let file_io = FileIOBuilder::new_fs_io().build().unwrap(); From d5572bd23a08f5932ac69e3dc96c8c9548c33094 Mon Sep 17 00:00:00 2001 From: Renjie Liu Date: Thu, 29 Jan 2026 13:02:34 +0800 Subject: [PATCH 179/891] Split sqllogictests into different schedule (#2085) ## Which issue does this PR close? - Closes #2084 . ## What changes are included in this PR? Split sqllogictests into different schedules to run in parallel. ## Are these changes tested? Yes, ci. --- .../testdata/schedules/df_basic_queries.toml | 23 ++++++++++++++++++ .../df_binary_predicate_pushdown.toml | 23 ++++++++++++++++++ .../testdata/schedules/df_create_table.toml | 23 ++++++++++++++++++ .../testdata/schedules/df_drop_table.toml | 23 ++++++++++++++++++ .../{df_test.toml => df_insert_and_like.toml} | 24 ------------------- .../testdata/schedules/df_scan_all_types.toml | 23 ++++++++++++++++++ .../testdata/schedules/df_show_tables.toml | 23 ++++++++++++++++++ 7 files changed, 138 insertions(+), 24 deletions(-) create mode 100644 crates/sqllogictest/testdata/schedules/df_basic_queries.toml create mode 100644 crates/sqllogictest/testdata/schedules/df_binary_predicate_pushdown.toml create mode 100644 crates/sqllogictest/testdata/schedules/df_create_table.toml create mode 100644 crates/sqllogictest/testdata/schedules/df_drop_table.toml rename crates/sqllogictest/testdata/schedules/{df_test.toml => df_insert_and_like.toml} (72%) create mode 100644 crates/sqllogictest/testdata/schedules/df_scan_all_types.toml create mode 100644 crates/sqllogictest/testdata/schedules/df_show_tables.toml diff --git a/crates/sqllogictest/testdata/schedules/df_basic_queries.toml b/crates/sqllogictest/testdata/schedules/df_basic_queries.toml new file mode 100644 index 0000000000..c0d5439d30 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_basic_queries.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/basic_queries.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_binary_predicate_pushdown.toml b/crates/sqllogictest/testdata/schedules/df_binary_predicate_pushdown.toml new file mode 100644 index 0000000000..de0093acb2 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_binary_predicate_pushdown.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/binary_predicate_pushdown.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_create_table.toml b/crates/sqllogictest/testdata/schedules/df_create_table.toml new file mode 100644 index 0000000000..1935c9fbd7 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_create_table.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/create_table.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_drop_table.toml b/crates/sqllogictest/testdata/schedules/df_drop_table.toml new file mode 100644 index 0000000000..8a6804dff7 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_drop_table.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/drop_table.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_test.toml b/crates/sqllogictest/testdata/schedules/df_insert_and_like.toml similarity index 72% rename from crates/sqllogictest/testdata/schedules/df_test.toml rename to crates/sqllogictest/testdata/schedules/df_insert_and_like.toml index ef5a2e4927..0c89a0426a 100644 --- a/crates/sqllogictest/testdata/schedules/df_test.toml +++ b/crates/sqllogictest/testdata/schedules/df_insert_and_like.toml @@ -18,34 +18,10 @@ [engines] df = { type = "datafusion" } -[[steps]] -engine = "df" -slt = "df_test/show_tables.slt" - -[[steps]] -engine = "df" -slt = "df_test/create_table.slt" - [[steps]] engine = "df" slt = "df_test/insert_into.slt" -[[steps]] -engine = "df" -slt = "df_test/scan_all_types.slt" - -[[steps]] -engine = "df" -slt = "df_test/basic_queries.slt" - -[[steps]] -engine = "df" -slt = "df_test/binary_predicate_pushdown.slt" - [[steps]] engine = "df" slt = "df_test/like_predicate_pushdown.slt" - -[[steps]] -engine = "df" -slt = "df_test/drop_table.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_scan_all_types.toml b/crates/sqllogictest/testdata/schedules/df_scan_all_types.toml new file mode 100644 index 0000000000..cb97fec641 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_scan_all_types.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/scan_all_types.slt" diff --git a/crates/sqllogictest/testdata/schedules/df_show_tables.toml b/crates/sqllogictest/testdata/schedules/df_show_tables.toml new file mode 100644 index 0000000000..5fdd8eec93 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_show_tables.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/show_tables.slt" From 6165cd94c61c4e25af8d2e742c8d1671c9a5d01c Mon Sep 17 00:00:00 2001 From: Nathan Metzger <39155588+xbattlax@users.noreply.github.com> Date: Thu, 29 Jan 2026 18:20:57 +0100 Subject: [PATCH 180/891] feat(spec): replace rust_decimal with fastnum for 38-digit precision (#2063) ## Summary Replace rust_decimal with fastnum::D128 to support 38-digit decimal precision as required by the Iceberg spec, addressing issue #669. ## Changes - Add `crates/iceberg/src/spec/values/decimal_utils.rs` with: - Compatibility layer providing rust_decimal-like API using fastnum::D128 - `decimal_from_i128_with_scale()`, `decimal_mantissa()`, `decimal_scale()` helpers - `i128_from_be_bytes()` / `i128_to_be_bytes_min()` for binary serialization - Unit tests for all helper functions including 38-digit precision validation - Update `Cargo.toml` (workspace and iceberg crate): - Add fastnum dependency with std and serde features - Remove rust_decimal, num-bigint, num-traits dependencies - Update `crates/iceberg/src/spec/values/datum.rs`: - Replace rust_decimal imports with decimal_utils - Update `from_bytes()` and `to_bytes()` for new decimal API - Update `decimal_with_precision()` validation - Update `crates/iceberg/src/spec/values/literal.rs`: - Update JSON serialization/deserialization for decimals - Update `crates/iceberg/src/transform/bucket.rs` and `truncate.rs`: - Update decimal transform implementations - Update `crates/iceberg/src/arrow/schema.rs` and `parquet_writer.rs`: - Update Arrow/Parquet decimal statistics handling - Replace `BigInt` byte conversions with helper functions - Update `.cargo/audit.toml`: - Remove RUSTSEC-2024-0399 ignore (rust_decimal vulnerability no longer applies) ## Notes fastnum::D128 provides exactly 38-digit precision with stack-based storage (no heap allocation), meeting the Iceberg spec requirement that rust_decimal (28-digit max) could not satisfy. The decimal_utils module abstracts the API differences between rust_decimal and fastnum, making the migration transparent to the rest of the codebase. Closes #669 --- .cargo/audit.toml | 3 - Cargo.lock | 34 +- Cargo.toml | 3 +- bindings/python/Cargo.toml | 5 +- crates/iceberg/Cargo.toml | 3 +- crates/iceberg/src/arrow/schema.rs | 20 +- crates/iceberg/src/error.rs | 6 - crates/iceberg/src/spec/mod.rs | 1 + crates/iceberg/src/spec/transform.rs | 13 +- crates/iceberg/src/spec/values/datum.rs | 66 ++-- .../iceberg/src/spec/values/decimal_utils.rs | 322 ++++++++++++++++++ crates/iceberg/src/spec/values/literal.rs | 21 +- crates/iceberg/src/spec/values/mod.rs | 2 + crates/iceberg/src/spec/values/tests.rs | 23 +- crates/iceberg/src/transform/bucket.rs | 3 +- crates/iceberg/src/transform/truncate.rs | 11 +- .../src/writer/file_writer/parquet_writer.rs | 27 +- 17 files changed, 447 insertions(+), 116 deletions(-) create mode 100644 crates/iceberg/src/spec/values/decimal_utils.rs diff --git a/.cargo/audit.toml b/.cargo/audit.toml index d403f0ac5a..09e2d35c50 100644 --- a/.cargo/audit.toml +++ b/.cargo/audit.toml @@ -33,7 +33,4 @@ ignore = [ # # Introduced by object_store, see https://github.com/apache/arrow-rs-object-store/issues/564 "RUSTSEC-2025-0134", - - # Tracked here: https://github.com/paupino/rust-decimal/issues/766 - "RUSTSEC-2026-0001", ] diff --git a/Cargo.lock b/Cargo.lock index 211b6416cb..577229645a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1058,6 +1058,16 @@ dependencies = [ "generic-array", ] +[[package]] +name = "bnum" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f781dba93de3a5ef6dc5b17c9958b208f6f3f021623b360fb605ea51ce443f10" +dependencies = [ + "serde", + "serde-big-array", +] + [[package]] name = "bon" version = "3.8.1" @@ -2703,6 +2713,18 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7" +[[package]] +name = "fastnum" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4089ab2dfd45d8ddc92febb5ca80644389d5ebb954f40231274a3f18341762e2" +dependencies = [ + "bnum", + "num-integer", + "num-traits", + "serde", +] + [[package]] name = "fastrand" version = "2.3.0" @@ -3367,6 +3389,7 @@ dependencies = [ "chrono", "derive_builder", "expect-test", + "fastnum", "flate2", "fnv", "futures", @@ -3376,7 +3399,6 @@ dependencies = [ "mockall", "moka", "murmur3", - "num-bigint", "once_cell", "opendal", "ordered-float 4.6.0", @@ -3387,7 +3409,6 @@ dependencies = [ "reqsign", "reqwest", "roaring", - "rust_decimal", "serde", "serde_bytes", "serde_derive", @@ -5966,6 +5987,15 @@ dependencies = [ "serde_derive", ] +[[package]] +name = "serde-big-array" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11fc7cc2c76d73e0f27ee52abbd64eec84d46f370c88371120433196934e4b7f" +dependencies = [ + "serde", +] + [[package]] name = "serde_bytes" version = "0.11.19" diff --git a/Cargo.toml b/Cargo.toml index 46ac4736b8..900377c22d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -97,7 +97,6 @@ mockall = "0.13.1" mockito = "1" motore-macros = "0.4.3" murmur3 = "0.5.2" -num-bigint = "0.4.6" once_cell = "1.20" opendal = "0.55.0" ordered-float = "4" @@ -108,7 +107,7 @@ rand = "0.8.5" regex = "1.11.3" reqwest = { version = "0.12.12", default-features = false, features = ["json"] } roaring = { version = "0.11" } -rust_decimal = { version = "1.39", default-features = false, features = ["std"] } +fastnum = { version = "0.7", default-features = false, features = ["std", "serde"] } serde = { version = "1.0.219", features = ["rc"] } serde_bytes = "0.11.17" serde_derive = "1.0.219" diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index 8346d02703..c179e89f99 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -37,8 +37,6 @@ pyo3 = { version = "0.26", features = ["extension-module", "abi3-py310"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } datafusion-ffi = { version = "51.0" } tokio = { version = "1.46.1", default-features = false } -# Security: disable rkyv feature to avoid RUSTSEC-2026-0001 (rkyv 0.7.45 vulnerability) -rust_decimal = { version = "1.39", default-features = false, features = ["std"] } [profile.release] codegen-units = 1 @@ -48,5 +46,4 @@ opt-level = "z" strip = true [package.metadata.cargo-machete] -# rust_decimal is included to override feature flags for security (disable rkyv) -ignored = ["rust_decimal"] +ignored = [] diff --git a/crates/iceberg/Cargo.toml b/crates/iceberg/Cargo.toml index 3835e77d1b..d6d931c86d 100644 --- a/crates/iceberg/Cargo.toml +++ b/crates/iceberg/Cargo.toml @@ -67,7 +67,6 @@ futures = { workspace = true } itertools = { workspace = true } moka = { version = "0.12.10", features = ["future"] } murmur3 = { workspace = true } -num-bigint = { workspace = true } once_cell = { workspace = true } opendal = { workspace = true } ordered-float = { workspace = true } @@ -76,7 +75,7 @@ rand = { workspace = true } reqsign = { version = "0.16.3", optional = true, default-features = false } reqwest = { workspace = true } roaring = { workspace = true } -rust_decimal = { workspace = true } +fastnum = { workspace = true } serde = { workspace = true } serde_bytes = { workspace = true } serde_derive = { workspace = true } diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index b15e141c82..e00f79a3bc 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -27,13 +27,12 @@ use arrow_array::{ TimestampMicrosecondArray, }; use arrow_schema::{DataType, Field, Fields, Schema as ArrowSchema, TimeUnit}; -use num_bigint::BigInt; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use parquet::file::statistics::Statistics; -use rust_decimal::prelude::ToPrimitive; use uuid::Uuid; use crate::error::Result; +use crate::spec::decimal_utils::i128_from_be_bytes; use crate::spec::{ Datum, FIRST_FIELD_ID, ListType, MapType, NestedField, NestedFieldRef, PrimitiveLiteral, PrimitiveType, Schema, SchemaVisitor, StructType, Type, @@ -680,7 +679,8 @@ impl SchemaVisitor for ToArrowSchemaConverter { DataType::FixedSizeBinary(16), )), crate::spec::PrimitiveType::Fixed(len) => Ok(ArrowSchemaOrFieldOrType::Type( - len.to_i32() + i32::try_from(*len) + .ok() .map(DataType::FixedSizeBinary) .unwrap_or(DataType::LargeBinary), )), @@ -722,10 +722,10 @@ pub(crate) fn get_arrow_datum(datum: &Datum) -> Result { - Ok(Arc::new(Float32Array::new_scalar(value.to_f32().unwrap()))) + Ok(Arc::new(Float32Array::new_scalar(value.into_inner()))) } (PrimitiveType::Double, PrimitiveLiteral::Double(value)) => { - Ok(Arc::new(Float64Array::new_scalar(value.to_f64().unwrap()))) + Ok(Arc::new(Float64Array::new_scalar(value.into_inner()))) } (PrimitiveType::String, PrimitiveLiteral::String(value)) => { Ok(Arc::new(StringArray::new_scalar(value.as_str()))) @@ -835,10 +835,9 @@ pub(crate) fn get_parquet_stat_min_as_datum( let Some(bytes) = stats.min_bytes_opt() else { return Ok(None); }; - let unscaled_value = BigInt::from_signed_bytes_be(bytes); Some(Datum::new( primitive_type.clone(), - PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { + PrimitiveLiteral::Int128(i128_from_be_bytes(bytes).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, format!("Can't convert bytes to i128: {bytes:?}"), @@ -982,10 +981,9 @@ pub(crate) fn get_parquet_stat_max_as_datum( let Some(bytes) = stats.max_bytes_opt() else { return Ok(None); }; - let unscaled_value = BigInt::from_signed_bytes_be(bytes); Some(Datum::new( primitive_type.clone(), - PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { + PrimitiveLiteral::Int128(i128_from_be_bytes(bytes).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, format!("Can't convert bytes to i128: {bytes:?}"), @@ -1295,9 +1293,9 @@ mod tests { use std::sync::Arc; use arrow_schema::{DataType, Field, Schema as ArrowSchema, TimeUnit}; - use rust_decimal::Decimal; use super::*; + use crate::spec::decimal_utils::decimal_new; use crate::spec::{Literal, Schema}; /// Create a simple field with metadata. @@ -2127,7 +2125,7 @@ mod tests { assert_eq!(array.value(0), 42); } { - let datum = Datum::decimal_with_precision(Decimal::new(123, 2), 30).unwrap(); + let datum = Datum::decimal_with_precision(decimal_new(123, 2), 30).unwrap(); let arrow_datum = get_arrow_datum(&datum).unwrap(); let (array, is_scalar) = arrow_datum.get(); let array = array.as_any().downcast_ref::().unwrap(); diff --git a/crates/iceberg/src/error.rs b/crates/iceberg/src/error.rs index 6ab3a78c8b..8810ef5009 100644 --- a/crates/iceberg/src/error.rs +++ b/crates/iceberg/src/error.rs @@ -408,12 +408,6 @@ define_from_err!( "Failed to parse json string" ); -define_from_err!( - rust_decimal::Error, - ErrorKind::DataInvalid, - "Failed to convert decimal literal to rust decimal" -); - define_from_err!( parquet::errors::ParquetError, ErrorKind::Unexpected, diff --git a/crates/iceberg/src/spec/mod.rs b/crates/iceberg/src/spec/mod.rs index a2b540f08b..707ebbb630 100644 --- a/crates/iceberg/src/spec/mod.rs +++ b/crates/iceberg/src/spec/mod.rs @@ -52,6 +52,7 @@ pub use table_metadata::*; pub(crate) use table_metadata_builder::FIRST_FIELD_ID; pub use table_properties::*; pub use transform::*; +pub(crate) use values::decimal_utils; pub use values::*; pub use view_metadata::*; pub use view_version::*; diff --git a/crates/iceberg/src/spec/transform.rs b/crates/iceberg/src/spec/transform.rs index 026b12613b..73fd290ee2 100644 --- a/crates/iceberg/src/spec/transform.rs +++ b/crates/iceberg/src/spec/transform.rs @@ -24,6 +24,7 @@ use std::str::FromStr; use fnv::FnvHashSet; use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use super::values::decimal_utils::decimal_from_i128_with_scale; use super::{Datum, PrimitiveLiteral}; use crate::ErrorKind; use crate::error::{Error, Result}; @@ -660,7 +661,7 @@ impl Transform { (PrimitiveType::Int, PrimitiveLiteral::Int(v)) => Some(Datum::int(v - 1)), (PrimitiveType::Long, PrimitiveLiteral::Long(v)) => Some(Datum::long(v - 1)), (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => { - Some(Datum::decimal(v - 1)?) + Some(Datum::decimal(decimal_from_i128_with_scale(v - 1, 0))?) } (PrimitiveType::Date, PrimitiveLiteral::Int(v)) => Some(Datum::date(v - 1)), (PrimitiveType::Timestamp, PrimitiveLiteral::Long(v)) => { @@ -672,7 +673,7 @@ impl Transform { (PrimitiveType::Int, PrimitiveLiteral::Int(v)) => Some(Datum::int(v + 1)), (PrimitiveType::Long, PrimitiveLiteral::Long(v)) => Some(Datum::long(v + 1)), (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => { - Some(Datum::decimal(v + 1)?) + Some(Datum::decimal(decimal_from_i128_with_scale(v + 1, 0))?) } (PrimitiveType::Date, PrimitiveLiteral::Int(v)) => Some(Datum::date(v + 1)), (PrimitiveType::Timestamp, PrimitiveLiteral::Long(v)) => { @@ -806,7 +807,9 @@ impl Transform { match (datum.data_type(), datum.literal()) { (PrimitiveType::Int, PrimitiveLiteral::Int(v)) => Ok(Datum::int(v + 1)), (PrimitiveType::Long, PrimitiveLiteral::Long(v)) => Ok(Datum::long(v + 1)), - (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => Datum::decimal(v + 1), + (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => { + Datum::decimal(decimal_from_i128_with_scale(v + 1, 0)) + } (PrimitiveType::Date, PrimitiveLiteral::Int(v)) => Ok(Datum::date(v + 1)), (PrimitiveType::Timestamp, PrimitiveLiteral::Long(v)) => { Ok(Datum::timestamp_micros(v + 1)) @@ -842,7 +845,9 @@ impl Transform { match (datum.data_type(), datum.literal()) { (PrimitiveType::Int, PrimitiveLiteral::Int(v)) => Ok(Datum::int(v - 1)), (PrimitiveType::Long, PrimitiveLiteral::Long(v)) => Ok(Datum::long(v - 1)), - (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => Datum::decimal(v - 1), + (PrimitiveType::Decimal { .. }, PrimitiveLiteral::Int128(v)) => { + Datum::decimal(decimal_from_i128_with_scale(v - 1, 0)) + } (PrimitiveType::Date, PrimitiveLiteral::Int(v)) => Ok(Datum::date(v - 1)), (PrimitiveType::Timestamp, PrimitiveLiteral::Long(v)) => { Ok(Datum::timestamp_micros(v - 1)) diff --git a/crates/iceberg/src/spec/values/datum.rs b/crates/iceberg/src/spec/values/datum.rs index 88209ae95c..3d4abc019d 100644 --- a/crates/iceberg/src/spec/values/datum.rs +++ b/crates/iceberg/src/spec/values/datum.rs @@ -22,15 +22,16 @@ use std::fmt::{Display, Formatter}; use std::str::FromStr; use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeZone, Utc}; -use num_bigint::BigInt; use ordered_float::{Float, OrderedFloat}; -use rust_decimal::Decimal; -use rust_decimal::prelude::ToPrimitive; use serde::de::{self, MapAccess}; use serde::ser::SerializeStruct; use serde::{Deserialize, Serialize}; use serde_bytes::ByteBuf; +use super::decimal_utils::{ + Decimal, decimal_from_i128_with_scale, decimal_from_str_exact, decimal_mantissa, decimal_scale, + i128_from_be_bytes, i128_to_be_bytes_min, +}; use super::literal::Literal; use super::primitive::PrimitiveLiteral; use super::serde::_serde::RawLiteral; @@ -268,8 +269,8 @@ impl PartialOrd for Datum { scale: other_scale, }, ) => { - let val = Decimal::from_i128_with_scale(*val, *scale); - let other_val = Decimal::from_i128_with_scale(*other_val, *other_scale); + let val = decimal_from_i128_with_scale(*val, *scale); + let other_val = decimal_from_i128_with_scale(*other_val, *other_scale); val.partial_cmp(&other_val) } _ => None, @@ -315,7 +316,7 @@ impl Display for Datum { }, PrimitiveLiteral::Int128(val), ) => { - write!(f, "{}", Decimal::from_i128_with_scale(*val, *scale)) + write!(f, "{}", decimal_from_i128_with_scale(*val, *scale)) } (_, _) => { unreachable!() @@ -407,8 +408,7 @@ impl Datum { PrimitiveType::Fixed(_) => PrimitiveLiteral::Binary(Vec::from(bytes)), PrimitiveType::Binary => PrimitiveLiteral::Binary(Vec::from(bytes)), PrimitiveType::Decimal { .. } => { - let unscaled_value = BigInt::from_signed_bytes_be(bytes); - PrimitiveLiteral::Int128(unscaled_value.to_i128().ok_or_else(|| { + PrimitiveLiteral::Int128(i128_from_be_bytes(bytes).ok_or_else(|| { Error::new( ErrorKind::DataInvalid, format!("Can't convert bytes to i128: {bytes:?}"), @@ -461,10 +461,8 @@ impl Datum { }; // The primitive literal is unscaled value. - let unscaled_value = BigInt::from(*val); - // Convert into two's-complement byte representation of the BigInt - // in big-endian byte order. - let mut bytes = unscaled_value.to_signed_bytes_be(); + // Convert into two's-complement byte representation in big-endian byte order. + let mut bytes = i128_to_be_bytes_min(*val); // Truncate with required bytes to make sure. bytes.truncate(required_bytes as usize); @@ -993,22 +991,18 @@ impl Datum { } } - /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. + /// Creates decimal literal from string. /// /// Example: /// /// ```rust /// use iceberg::spec::Datum; - /// use itertools::assert_equal; - /// use rust_decimal::Decimal; /// let t = Datum::decimal_from_str("123.45").unwrap(); /// /// assert_eq!(&format!("{t}"), "123.45"); /// ``` pub fn decimal_from_str>(s: S) -> Result { - let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) - })?; + let decimal = decimal_from_str_exact(s.as_ref())?; Self::decimal(decimal) } @@ -1019,21 +1013,19 @@ impl Datum { /// /// ```rust /// use iceberg::spec::Datum; - /// use rust_decimal::Decimal; /// - /// let t = Datum::decimal(Decimal::new(123, 2)).unwrap(); + /// let t = Datum::decimal_from_str("1.23").unwrap(); /// /// assert_eq!(&format!("{t}"), "1.23"); /// ``` - pub fn decimal(value: impl Into) -> Result { - let decimal = value.into(); - let scale = decimal.scale(); + pub fn decimal(value: Decimal) -> Result { + let scale = decimal_scale(&value); let r#type = Type::decimal(MAX_DECIMAL_PRECISION, scale)?; if let Type::Primitive(p) = r#type { Ok(Self { r#type: p, - literal: PrimitiveLiteral::Int128(decimal.mantissa()), + literal: PrimitiveLiteral::Int128(decimal_mantissa(&value)), }) } else { unreachable!("Decimal type must be primitive.") @@ -1042,27 +1034,19 @@ impl Datum { /// Try to create a decimal literal from [`Decimal`] with precision. /// - /// Example: - /// - /// ```rust - /// use iceberg::spec::Datum; - /// use rust_decimal::Decimal; - /// - /// let t = Datum::decimal_with_precision(Decimal::new(123, 2), 30).unwrap(); - /// - /// assert_eq!(&format!("{t}"), "1.23"); - /// ``` - pub fn decimal_with_precision(value: impl Into, precision: u32) -> Result { - let decimal = value.into(); - let scale = decimal.scale(); + /// This method allows specifying a custom precision for the decimal type, + /// which is useful when you need to control the storage requirements. + /// Use [`Datum::decimal`] if you want to use the maximum precision (38). + pub fn decimal_with_precision(value: Decimal, precision: u32) -> Result { + let scale = decimal_scale(&value); + let mantissa = decimal_mantissa(&value); let available_bytes = Type::decimal_required_bytes(precision)? as usize; - let unscaled_value = BigInt::from(decimal.mantissa()); - let actual_bytes = unscaled_value.to_signed_bytes_be(); + let actual_bytes = i128_to_be_bytes_min(mantissa); if actual_bytes.len() > available_bytes { return Err(Error::new( ErrorKind::DataInvalid, - format!("Decimal value {decimal} is too large for precision {precision}"), + format!("Decimal value {value} is too large for precision {precision}"), )); } @@ -1070,7 +1054,7 @@ impl Datum { if let Type::Primitive(p) = r#type { Ok(Self { r#type: p, - literal: PrimitiveLiteral::Int128(decimal.mantissa()), + literal: PrimitiveLiteral::Int128(mantissa), }) } else { unreachable!("Decimal type must be primitive.") diff --git a/crates/iceberg/src/spec/values/decimal_utils.rs b/crates/iceberg/src/spec/values/decimal_utils.rs new file mode 100644 index 0000000000..88e3f72f65 --- /dev/null +++ b/crates/iceberg/src/spec/values/decimal_utils.rs @@ -0,0 +1,322 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Compatibility layer for decimal operations. +//! +//! Provides rust_decimal-compatible API using fastnum's D128 internally. +//! D128 supports 38-digit precision, meeting the Iceberg spec requirement. + +use fastnum::D128; +use fastnum::decimal::Context; + +use crate::{Error, ErrorKind, Result}; + +/// Re-export D128 as the Decimal type for use throughout the crate. +pub type Decimal = D128; + +/// Create a D128 from mantissa (i128) and scale (u32). +/// +/// This is equivalent to rust_decimal's `Decimal::from_i128_with_scale`. +/// The value is computed as: mantissa * 10^(-scale) +/// +/// For example: +/// - mantissa=12345, scale=2 => 123.45 +/// - mantissa=-456, scale=3 => -0.456 +pub fn decimal_from_i128_with_scale(mantissa: i128, scale: u32) -> Decimal { + if scale == 0 { + return D128::from_i128(mantissa).expect("i128 always fits in D128"); + } + + // Convert mantissa to string and insert decimal point at the right position + let is_negative = mantissa < 0; + let abs_str = mantissa.unsigned_abs().to_string(); + let scale_usize = scale as usize; + + let decimal_str = if abs_str.len() <= scale_usize { + // Need leading zeros: e.g., mantissa=456, scale=3 => "0.456" + // Or mantissa=5, scale=3 => "0.005" + let zeros_needed = scale_usize - abs_str.len(); + format!( + "{}0.{}{}", + if is_negative { "-" } else { "" }, + "0".repeat(zeros_needed), + abs_str + ) + } else { + // Insert decimal point: e.g., mantissa=12345, scale=2 => "123.45" + let decimal_pos = abs_str.len() - scale_usize; + format!( + "{}{}.{}", + if is_negative { "-" } else { "" }, + &abs_str[..decimal_pos], + &abs_str[decimal_pos..] + ) + }; + + D128::from_str(&decimal_str, Context::default()) + .expect("constructed decimal string is always valid") +} + +/// Try to create a D128 from mantissa and scale, with validation. +/// +/// This is equivalent to rust_decimal's `Decimal::try_from_i128_with_scale`. +/// Currently always succeeds for 38-digit decimals. +pub fn try_decimal_from_i128_with_scale(mantissa: i128, scale: u32) -> Result { + // For now, always succeeds since D128 supports full 38-digit precision + Ok(decimal_from_i128_with_scale(mantissa, scale)) +} + +/// Create a D128 from i64 mantissa and scale. +/// +/// This is equivalent to rust_decimal's `Decimal::new`. +#[allow(dead_code)] +pub fn decimal_new(mantissa: i64, scale: u32) -> Decimal { + decimal_from_i128_with_scale(mantissa as i128, scale) +} + +/// Parse a decimal from string with exact representation. +/// +/// This is equivalent to rust_decimal's `Decimal::from_str_exact`. +pub fn decimal_from_str_exact(s: &str) -> Result { + D128::from_str(s, Context::default()) + .map_err(|e| Error::new(ErrorKind::DataInvalid, format!("Can't parse decimal: {e}"))) +} + +/// Get the mantissa (unscaled coefficient) as i128. +/// +/// This is equivalent to rust_decimal's `decimal.mantissa()`. +/// +/// The mantissa is signed: negative decimals return negative mantissa. +pub fn decimal_mantissa(d: &Decimal) -> i128 { + // digits() returns unsigned coefficient as UInt + // For Iceberg decimals (max 38 digits), this always fits in u128/i128 + let digits = d.digits(); + + // Convert UInt<2> to u128 - this always succeeds for Iceberg-compliant decimals + // since 38 digits requires ~127 bits and u128 has 128 bits + let unsigned: u128 = digits + .to_u128() + .expect("Iceberg decimals (max 38 digits) always fit in u128"); + + let signed = unsigned as i128; + if d.is_sign_negative() { + -signed + } else { + signed + } +} + +/// Get the scale (number of digits after decimal point). +/// +/// This is equivalent to rust_decimal's `decimal.scale()`. +pub fn decimal_scale(d: &Decimal) -> u32 { + let frac = d.fractional_digits_count(); + if frac < 0 { 0 } else { frac as u32 } +} + +/// Rescale a decimal to the given scale, returning the rescaled value. +/// +/// This is equivalent to rust_decimal's `decimal.rescale(scale)`. +pub fn decimal_rescale(d: Decimal, scale: u32) -> Decimal { + d.rescale(scale as i16) +} + +/// Convert big-endian signed bytes to i128. +/// +/// This handles variable-length byte arrays (up to 16 bytes) with sign extension. +/// Returns None if the byte array is longer than 16 bytes. +pub fn i128_from_be_bytes(bytes: &[u8]) -> Option { + if bytes.is_empty() { + return Some(0); + } + if bytes.len() > 16 { + return None; // Too large for i128 + } + + // Check sign bit (most significant bit of first byte) + let is_negative = bytes[0] & 0x80 != 0; + + // Pad to 16 bytes with sign extension + let mut padded = if is_negative { [0xFF; 16] } else { [0; 16] }; + let start = 16 - bytes.len(); + padded[start..].copy_from_slice(bytes); + + Some(i128::from_be_bytes(padded)) +} + +/// Convert i128 to big-endian signed bytes with minimum length. +/// +/// This produces the shortest two's complement representation of the value. +/// The result is suitable for Iceberg decimal binary serialization. +pub fn i128_to_be_bytes_min(value: i128) -> Vec { + let bytes = value.to_be_bytes(); + + // Find the first significant byte + // For positive numbers, skip leading 0x00 bytes (but keep sign bit) + // For negative numbers, skip leading 0xFF bytes (but keep sign bit) + let is_negative = value < 0; + let skip_byte = if is_negative { 0xFF } else { 0x00 }; + + let mut start = 0; + while start < 15 && bytes[start] == skip_byte { + // Check if the next byte has the correct sign bit + let next_byte = bytes[start + 1]; + let next_is_negative = (next_byte & 0x80) != 0; + if next_is_negative == is_negative { + start += 1; + } else { + break; + } + } + + bytes[start..].to_vec() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_decimal_from_i128_with_scale() { + let d = decimal_from_i128_with_scale(12345, 2); + assert_eq!(d.to_string(), "123.45"); + + let d = decimal_from_i128_with_scale(-12345, 2); + assert_eq!(d.to_string(), "-123.45"); + + let d = decimal_from_i128_with_scale(0, 5); + assert_eq!(d.to_string(), "0.00000"); + } + + #[test] + fn test_decimal_new() { + let d = decimal_new(123, 2); + assert_eq!(d.to_string(), "1.23"); + + let d = decimal_new(-456, 3); + assert_eq!(d.to_string(), "-0.456"); + } + + #[test] + fn test_decimal_from_str_exact() { + let d = decimal_from_str_exact("123.45").unwrap(); + assert_eq!(d.to_string(), "123.45"); + + let d = decimal_from_str_exact("-0.001").unwrap(); + assert_eq!(d.to_string(), "-0.001"); + + let d = decimal_from_str_exact("99999999999999999999999999999999999999").unwrap(); + assert_eq!(d.to_string(), "99999999999999999999999999999999999999"); + } + + #[test] + fn test_decimal_mantissa() { + let d = decimal_from_i128_with_scale(12345, 2); + assert_eq!(decimal_mantissa(&d), 12345); + + let d = decimal_from_i128_with_scale(-12345, 2); + assert_eq!(decimal_mantissa(&d), -12345); + } + + #[test] + fn test_decimal_scale() { + let d = decimal_from_i128_with_scale(12345, 2); + assert_eq!(decimal_scale(&d), 2); + + let d = decimal_from_i128_with_scale(12345, 0); + assert_eq!(decimal_scale(&d), 0); + } + + #[test] + fn test_decimal_rescale() { + let d = decimal_from_str_exact("123.45").unwrap(); + let rescaled = decimal_rescale(d, 4); + assert_eq!(decimal_scale(&rescaled), 4); + assert_eq!(decimal_mantissa(&rescaled), 1234500); + } + + #[test] + fn test_38_digit_precision() { + // Test that we can handle 38-digit decimals (Iceberg spec requirement) + let max_38_digits = "99999999999999999999999999999999999999"; + let d = decimal_from_str_exact(max_38_digits).unwrap(); + assert_eq!(d.to_string(), max_38_digits); + + let min_38_digits = "-99999999999999999999999999999999999999"; + let d = decimal_from_str_exact(min_38_digits).unwrap(); + assert_eq!(d.to_string(), min_38_digits); + } + + #[test] + fn test_i128_from_be_bytes() { + // Empty bytes + assert_eq!(i128_from_be_bytes(&[]), Some(0)); + + // Positive values + assert_eq!(i128_from_be_bytes(&[0x01]), Some(1)); + assert_eq!(i128_from_be_bytes(&[0x7F]), Some(127)); + assert_eq!(i128_from_be_bytes(&[0x00, 0xFF]), Some(255)); + assert_eq!(i128_from_be_bytes(&[0x04, 0xD2]), Some(1234)); + + // Negative values (sign extension) + assert_eq!(i128_from_be_bytes(&[0xFF]), Some(-1)); + assert_eq!(i128_from_be_bytes(&[0x80]), Some(-128)); + assert_eq!(i128_from_be_bytes(&[0xFB, 0x2E]), Some(-1234)); + + // Too large (> 16 bytes) + assert_eq!(i128_from_be_bytes(&[0; 17]), None); + } + + #[test] + fn test_i128_to_be_bytes_min() { + // Positive values + assert_eq!(i128_to_be_bytes_min(0), vec![0x00]); + assert_eq!(i128_to_be_bytes_min(1), vec![0x01]); + assert_eq!(i128_to_be_bytes_min(127), vec![0x7F]); + assert_eq!(i128_to_be_bytes_min(128), vec![0x00, 0x80]); + assert_eq!(i128_to_be_bytes_min(255), vec![0x00, 0xFF]); + assert_eq!(i128_to_be_bytes_min(1234), vec![0x04, 0xD2]); + + // Negative values + assert_eq!(i128_to_be_bytes_min(-1), vec![0xFF]); + assert_eq!(i128_to_be_bytes_min(-128), vec![0x80]); + assert_eq!(i128_to_be_bytes_min(-129), vec![0xFF, 0x7F]); + assert_eq!(i128_to_be_bytes_min(-1234), vec![0xFB, 0x2E]); + + // Round trip test + for val in [ + 0i128, + 1, + -1, + 127, + -128, + 255, + -256, + 12345, + -12345, + i128::MAX, + i128::MIN, + ] { + let bytes = i128_to_be_bytes_min(val); + assert_eq!( + i128_from_be_bytes(&bytes), + Some(val), + "Round trip failed for {val}" + ); + } + } +} diff --git a/crates/iceberg/src/spec/values/literal.rs b/crates/iceberg/src/spec/values/literal.rs index d6e502e8fd..e82fa197cd 100644 --- a/crates/iceberg/src/spec/values/literal.rs +++ b/crates/iceberg/src/spec/values/literal.rs @@ -22,11 +22,13 @@ use std::str::FromStr; use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, TimeZone, Utc}; use ordered_float::OrderedFloat; -use rust_decimal::Decimal; use serde_json::{Map as JsonMap, Number, Value as JsonValue}; use uuid::Uuid; use super::Map; +use super::decimal_utils::{ + decimal_from_str_exact, decimal_mantissa, decimal_rescale, try_decimal_from_i128_with_scale, +}; use super::primitive::PrimitiveLiteral; use super::struct_value::Struct; use super::temporal::{date, time, timestamp, timestamptz}; @@ -396,23 +398,20 @@ impl Literal { Self::Primitive(PrimitiveLiteral::Int128(decimal)) } - /// Creates decimal literal from string. See [`Decimal::from_str_exact`]. + /// Creates decimal literal from string. /// /// Example: /// /// ```rust /// use iceberg::spec::Literal; - /// use rust_decimal::Decimal; /// let t1 = Literal::decimal(12345); /// let t2 = Literal::decimal_from_str("123.45").unwrap(); /// /// assert_eq!(t1, t2); /// ``` pub fn decimal_from_str>(s: S) -> Result { - let decimal = Decimal::from_str_exact(s.as_ref()).map_err(|e| { - Error::new(ErrorKind::DataInvalid, "Can't parse decimal.").with_source(e) - })?; - Ok(Self::decimal(decimal.mantissa())) + let decimal = decimal_from_str_exact(s.as_ref())?; + Ok(Self::decimal(decimal_mantissa(&decimal))) } /// Attempts to convert the Literal to a PrimitiveLiteral @@ -509,10 +508,10 @@ impl Literal { }, JsonValue::String(s), ) => { - let mut decimal = Decimal::from_str_exact(&s)?; - decimal.rescale(*scale); + let decimal = decimal_from_str_exact(&s)?; + let rescaled = decimal_rescale(decimal, *scale); Ok(Some(Literal::Primitive(PrimitiveLiteral::Int128( - decimal.mantissa(), + decimal_mantissa(&rescaled), )))) } (_, JsonValue::Null) => Ok(None), @@ -672,7 +671,7 @@ impl Literal { precision: _precision, scale, }) => { - let decimal = Decimal::try_from_i128_with_scale(val, *scale)?; + let decimal = try_decimal_from_i128_with_scale(val, *scale)?; Ok(JsonValue::String(decimal.to_string())) } _ => Err(Error::new( diff --git a/crates/iceberg/src/spec/values/mod.rs b/crates/iceberg/src/spec/values/mod.rs index 2bc967191b..65374c74b7 100644 --- a/crates/iceberg/src/spec/values/mod.rs +++ b/crates/iceberg/src/spec/values/mod.rs @@ -18,6 +18,7 @@ //! This module contains Iceberg value types pub(crate) mod datum; +pub(crate) mod decimal_utils; mod literal; mod map; mod primitive; @@ -30,6 +31,7 @@ mod tests; // Re-export all public types pub use datum::Datum; +pub use decimal_utils::Decimal; pub use literal::Literal; pub use map::Map; pub use primitive::PrimitiveLiteral; diff --git a/crates/iceberg/src/spec/values/tests.rs b/crates/iceberg/src/spec/values/tests.rs index bb10701d87..41238ed899 100644 --- a/crates/iceberg/src/spec/values/tests.rs +++ b/crates/iceberg/src/spec/values/tests.rs @@ -20,11 +20,11 @@ use apache_avro::to_value; use apache_avro::types::Value; use ordered_float::OrderedFloat; -use rust_decimal::Decimal; use serde_bytes::ByteBuf; use serde_json::Value as JsonValue; use uuid::Uuid; +use super::decimal_utils::{decimal_from_i128_with_scale, decimal_new}; use crate::ErrorKind; use crate::avro::schema_to_avro_schema; use crate::spec::Schema; @@ -395,11 +395,8 @@ fn avro_bytes_decimal() { for (input_bytes, decimal_num, expect_scale, expect_precision) in cases { check_avro_bytes_serde( input_bytes, - Datum::decimal_with_precision( - Decimal::new(decimal_num, expect_scale), - expect_precision, - ) - .unwrap(), + Datum::decimal_with_precision(decimal_new(decimal_num, expect_scale), expect_precision) + .unwrap(), &PrimitiveType::Decimal { precision: expect_precision, scale: expect_scale, @@ -414,10 +411,8 @@ fn avro_bytes_decimal_expect_error() { let cases = vec![(1234, 2, 1)]; for (decimal_num, expect_scale, expect_precision) in cases { - let result = Datum::decimal_with_precision( - Decimal::new(decimal_num, expect_scale), - expect_precision, - ); + let result = + Datum::decimal_with_precision(decimal_new(decimal_num, expect_scale), expect_precision); assert!(result.is_err(), "expect error but got {result:?}"); assert_eq!( result.unwrap_err().kind(), @@ -1053,7 +1048,7 @@ fn test_datum_ser_deser() { test_fn(datum); let datum = Datum::uuid(Uuid::parse_str("f79c3e09-677c-4bbd-a479-3f349cb785e7").unwrap()); test_fn(datum); - let datum = Datum::decimal(1420).unwrap(); + let datum = Datum::decimal(decimal_new(1420, 0)).unwrap(); test_fn(datum); let datum = Datum::binary(vec![1, 2, 3, 4, 5]); test_fn(datum); @@ -1140,7 +1135,7 @@ fn test_datum_long_convert_to_timestamptz() { #[test] fn test_datum_decimal_convert_to_long() { - let datum = Datum::decimal(12345).unwrap(); + let datum = Datum::decimal(decimal_new(12345, 0)).unwrap(); let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); @@ -1151,7 +1146,7 @@ fn test_datum_decimal_convert_to_long() { #[test] fn test_datum_decimal_convert_to_long_above_max() { - let datum = Datum::decimal(LONG_MAX as i128 + 1).unwrap(); + let datum = Datum::decimal(decimal_from_i128_with_scale(LONG_MAX as i128 + 1, 0)).unwrap(); let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); @@ -1162,7 +1157,7 @@ fn test_datum_decimal_convert_to_long_above_max() { #[test] fn test_datum_decimal_convert_to_long_below_min() { - let datum = Datum::decimal(LONG_MIN as i128 - 1).unwrap(); + let datum = Datum::decimal(decimal_from_i128_with_scale(LONG_MIN as i128 - 1, 0)).unwrap(); let result = datum.to(&Primitive(PrimitiveType::Long)).unwrap(); diff --git a/crates/iceberg/src/transform/bucket.rs b/crates/iceberg/src/transform/bucket.rs index e6786a70ca..52fb72f1cf 100644 --- a/crates/iceberg/src/transform/bucket.rs +++ b/crates/iceberg/src/transform/bucket.rs @@ -294,6 +294,7 @@ mod test { TimestampNs, Timestamptz, TimestamptzNs, Uuid, }; use crate::spec::Type::{Primitive, Struct}; + use crate::spec::decimal_utils::decimal_new; use crate::spec::{Datum, NestedField, PrimitiveType, StructType, Transform, Type}; use crate::transform::TransformFunction; use crate::transform::test::{TestProjectionFixture, TestTransformFixture}; @@ -848,7 +849,7 @@ mod test { let bucket = Bucket::new(10); assert_eq!( bucket - .transform_literal(&Datum::decimal(1420).unwrap()) + .transform_literal(&Datum::decimal(decimal_new(1420, 0)).unwrap()) .unwrap() .unwrap(), Datum::int(9) diff --git a/crates/iceberg/src/transform/truncate.rs b/crates/iceberg/src/transform/truncate.rs index 84ef7c0da3..4fac48f7d5 100644 --- a/crates/iceberg/src/transform/truncate.rs +++ b/crates/iceberg/src/transform/truncate.rs @@ -22,6 +22,7 @@ use arrow_schema::DataType; use super::TransformFunction; use crate::Error; +use crate::spec::decimal_utils::decimal_from_i128_with_scale; use crate::spec::{Datum, PrimitiveLiteral}; #[derive(Debug)] @@ -163,7 +164,10 @@ impl TransformFunction for Truncate { })), PrimitiveLiteral::Int128(v) => Ok(Some({ let width = self.width as i128; - Datum::decimal(Self::truncate_decimal_i128(*v, width))? + Datum::decimal(decimal_from_i128_with_scale( + Self::truncate_decimal_i128(*v, width), + 0, + ))? })), PrimitiveLiteral::String(v) => Ok(Some({ let len = self.width as usize; @@ -195,6 +199,7 @@ mod test { TimestampNs, Timestamptz, TimestamptzNs, Uuid, }; use crate::spec::Type::{Primitive, Struct}; + use crate::spec::decimal_utils::decimal_new; use crate::spec::{Datum, NestedField, PrimitiveType, StructType, Transform, Type}; use crate::transform::TransformFunction; use crate::transform::test::{TestProjectionFixture, TestTransformFixture}; @@ -831,12 +836,12 @@ mod test { #[test] fn test_decimal_literal() { - let input = Datum::decimal(1065).unwrap(); + let input = Datum::decimal(decimal_new(1065, 0)).unwrap(); let res = super::Truncate::new(50) .transform_literal(&input) .unwrap() .unwrap(); - assert_eq!(res, Datum::decimal(1050).unwrap(),); + assert_eq!(res, Datum::decimal(decimal_new(1050, 0)).unwrap(),); } #[test] diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index 8fe40df71c..a75c74b3b2 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -622,13 +622,13 @@ mod tests { use arrow_select::concat::concat_batches; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; use parquet::file::statistics::ValueStatistics; - use rust_decimal::Decimal; use tempfile::TempDir; use uuid::Uuid; use super::*; use crate::arrow::schema_to_arrow_schema; use crate::io::FileIOBuilder; + use crate::spec::decimal_utils::{decimal_mantissa, decimal_new, decimal_scale}; use crate::spec::{PrimitiveLiteral, Struct, *}; use crate::writer::file_writer::location_generator::{ DefaultFileNameGenerator, DefaultLocationGenerator, FileNameGenerator, LocationGenerator, @@ -1378,12 +1378,12 @@ mod tests { .unwrap(); assert_eq!( data_file.upper_bounds().get(&0), - Some(Datum::decimal_with_precision(Decimal::new(22000000000_i64, 10), 28).unwrap()) + Some(Datum::decimal_with_precision(decimal_new(22000000000_i64, 10), 28).unwrap()) .as_ref() ); assert_eq!( data_file.lower_bounds().get(&0), - Some(Datum::decimal_with_precision(Decimal::new(11000000000_i64, 10), 28).unwrap()) + Some(Datum::decimal_with_precision(decimal_new(11000000000_i64, 10), 28).unwrap()) .as_ref() ); @@ -1430,19 +1430,22 @@ mod tests { .unwrap(); assert_eq!( data_file.upper_bounds().get(&0), - Some(Datum::decimal_with_precision(Decimal::new(-11000000000_i64, 10), 28).unwrap()) + Some(Datum::decimal_with_precision(decimal_new(-11000000000_i64, 10), 28).unwrap()) .as_ref() ); assert_eq!( data_file.lower_bounds().get(&0), - Some(Datum::decimal_with_precision(Decimal::new(-22000000000_i64, 10), 28).unwrap()) + Some(Datum::decimal_with_precision(decimal_new(-22000000000_i64, 10), 28).unwrap()) .as_ref() ); - // test max and min of rust_decimal - let decimal_max = Decimal::MAX; - let decimal_min = Decimal::MIN; - assert_eq!(decimal_max.scale(), decimal_min.scale()); + // test 38-digit precision decimal values (Iceberg spec max) + // Note: fastnum D128::MAX/MIN have impractical exponents, so we use meaningful values + use crate::spec::decimal_utils::decimal_from_str_exact; + let decimal_max = decimal_from_str_exact("99999999999999999999999999999999999999").unwrap(); + let decimal_min = + decimal_from_str_exact("-99999999999999999999999999999999999999").unwrap(); + assert_eq!(decimal_scale(&decimal_max), decimal_scale(&decimal_min)); let schema = Arc::new( Schema::builder() .with_fields(vec![ @@ -1451,7 +1454,7 @@ mod tests { "decimal", Type::Primitive(PrimitiveType::Decimal { precision: 38, - scale: decimal_max.scale(), + scale: decimal_scale(&decimal_max), }), ) .into(), @@ -1468,8 +1471,8 @@ mod tests { .await?; let col0 = Arc::new( Decimal128Array::from(vec![ - Some(decimal_max.mantissa()), - Some(decimal_min.mantissa()), + Some(decimal_mantissa(&decimal_max)), + Some(decimal_mantissa(&decimal_min)), ]) .with_data_type(DataType::Decimal128(38, 0)), ) as ArrayRef; From da9e6137070cfd2b932ae7423adbf7b73c16e29f Mon Sep 17 00:00:00 2001 From: Raminder Singh Date: Fri, 30 Jan 2026 01:32:26 +0530 Subject: [PATCH 181/891] Add ETL project link to README (#2087) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5fc452098e..693180c6df 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ The Apache Iceberg community is built on the principles described in the [Apache - [Moonlink](https://github.com/Mooncake-Labs/moonlink): A Rust library that enables sub-second mirroring (CDC) of Postgres tables into Iceberg. - [RisingWave](https://github.com/risingwavelabs/risingwave): A Postgres-compatible SQL database designed for real-time event streaming data processing, analysis, and management. - [Wrappers](https://github.com/supabase/wrappers): Postgres Foreign Data Wrapper development framework in Rust. +- [ETL](https://github.com/supabase/etl): Stream your Postgres data anywhere in real-time. ## License From 5d3152f11ec8fbb210a48d1ce490bc1ea73b6ca0 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Thu, 29 Jan 2026 19:06:12 -0500 Subject: [PATCH 182/891] site: merge container docs and add a section about docker desktop (#2091) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Closes #719 ## What changes are included in this PR? With all the recent changes to our test setup and containers (#2071, #2074, #2078), `make test` now works with Docker Desktop. I tested this locally! (requires #2090) This PR updates our documentation to reflect that docker desktop works. We used to have 2 pages for containers, 1 for orbstack another for podman. I merged them together and also added a section for Docker Desktop Preview: Screenshot 2026-01-29 at 11 35
55 AM ## Are these changes tested? --- CONTRIBUTING.md | 5 +- website/src/SUMMARY.md | 3 +- .../{podman.md => container-runtimes.md} | 93 +++++++++++++++---- website/src/reference/orbstack.md | 39 -------- 4 files changed, 77 insertions(+), 63 deletions(-) rename website/src/reference/{podman.md => container-runtimes.md} (53%) delete mode 100644 website/src/reference/orbstack.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0d05c3ddac..d1a736ed16 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -101,10 +101,7 @@ cargo 1.69.0 (6e9a83356 2023-04-12) #### Install Docker or Podman -Currently, iceberg-rust uses Docker to set up environment for integration tests. Native Docker has some limitations, please check (https://github.com/apache/iceberg-rust/pull/748). Please use Orbstack or Podman. - -For MacOS users, you can install [OrbStack as a docker alternative](website/src/reference/orbstack.md). -For Podman users, refer to [Using Podman instead of Docker](website/src/reference/podman.md) +Currently, iceberg-rust uses Docker to set up environment for integration tests. See [Container Runtimes](website/src/reference/container-runtimes.md) for setup instructions. ## Build diff --git a/website/src/SUMMARY.md b/website/src/SUMMARY.md index da82364f58..dfe0043ffd 100644 --- a/website/src/SUMMARY.md +++ b/website/src/SUMMARY.md @@ -32,6 +32,5 @@ # Reference -- [Using Podman instead of Docker](./reference/podman.md) -- [OrbStack as a docker alternative](./reference/orbstack.md) +- [Container Runtimes](./reference/container-runtimes.md) - [Setup GPG key](./reference/setup_gpg.md) diff --git a/website/src/reference/podman.md b/website/src/reference/container-runtimes.md similarity index 53% rename from website/src/reference/podman.md rename to website/src/reference/container-runtimes.md index eee93a0b35..7b4ac38511 100644 --- a/website/src/reference/podman.md +++ b/website/src/reference/container-runtimes.md @@ -17,18 +17,74 @@ ~ under the License. --> -# Using Podman instead of Docker +# Container Runtimes -Iceberg-rust does not require containerization, except for integration tests, where "docker" and "docker-compose" are used to start containers for minio and various catalogs. Below instructions setup "rootful podman" and docker's official docker-compose plugin to run integration tests as an alternative to docker or Orbstack. +Iceberg-rust uses containers for integration tests, where `docker` and `docker compose` start containers for MinIO and various catalogs. You can use any of the following container runtimes. + +## Docker Desktop + +[Docker Desktop](https://www.docker.com/products/docker-desktop/) is available for macOS, Windows, and Linux. + +1. Install Docker Desktop by downloading the [installer](https://www.docker.com/products/docker-desktop/) or using Homebrew on macOS. + ```shell + brew install --cask docker + ``` + +2. Launch Docker Desktop and complete the setup. + +3. Verify the installation. + ```shell + docker --version + docker compose version + ``` + +4. Try some integration tests! + ```shell + make test + ``` + +## OrbStack (macOS) + +[OrbStack](https://orbstack.dev/) is a lightweight alternative to Docker Desktop on macOS. + +1. Install OrbStack by downloading the [installer](https://orbstack.dev/download) or using Homebrew. + ```shell + brew install orbstack + ``` + +2. Migrate Docker data (if switching from Docker Desktop). + ```shell + orb migrate docker + ``` + +3. (Optional) Add registry mirrors. + + You can edit the config directly at `~/.orbstack/config/docker.json` and restart the engine with `orb restart docker`. + + ```json + { + "registry-mirrors": [""] + } + ``` + +4. Try some integration tests! + ```shell + make test + ``` + +## Podman + +[Podman](https://podman.io/) is a daemonless container engine. The instructions below set up "rootful podman" with docker's official docker-compose plugin. 1. Have podman v4 or newer. ```shell $ podman --version podman version 4.9.4-rhel ``` + 2. Create a docker wrapper script: - * Create a fresh `/usr/bin/docker` file and add the below contents: + Create a fresh `/usr/bin/docker` file and add the below contents: ```bash #!/bin/sh [ -e /etc/containers/nodocker ] || \ @@ -36,7 +92,7 @@ Iceberg-rust does not require containerization, except for integration tests, wh exec sudo /usr/bin/podman "$@" ``` - * Set new `/usr/bin/docker` file to executable. + Set new `/usr/bin/docker` file to executable. ```shell sudo chmod +x /usr/bin/docker ``` @@ -78,23 +134,24 @@ Iceberg-rust does not require containerization, except for integration tests, wh cargo test -p iceberg --test file_io_s3_test ``` -# References +### Note on rootless containers -* -* +As of podman v4, ["To be succinct and simple, when running rootless containers, the container itself does not have an IP address"](https://www.redhat.com/sysadmin/container-ip-address-podman). This causes issues with iceberg-rust's integration tests, which rely upon IP-addressable containers via docker-compose. As a result, podman "rootful" containers are required to ensure containers have IP addresses. + +### Podman troubleshooting + +**Error:** `short-name "apache/iceberg-rest-fixture" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"` -# Note on rootless containers +**Fix:** Add or modify the `/etc/containers/registries.conf` file: +```toml +[[registry]] +prefix = "docker.io" +location = "docker.io" +``` -As of podman v4, ["To be succinct and simple, when running rootless containers, the container itself does not have an IP address"](https://www.redhat.com/sysadmin/container-ip-address-podman) This causes issues with iceberg-rust's integration tests, which rely upon ip-addressable containers via docker-compose. As a result, podman "rootful" containers are required throughout to ensure containers have IP addresses. Perhaps as a future work or with updates to default podman networking, the need for "rootful" podman containers can be eliminated. +### Podman references +* +* * * - -# Debugging Note: - - Fix for error: `Error: short-name "apache/iceberg-rest-fixture" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"` - - Add or modify the `/etc/containers/registries.conf` file: - ```toml - [[registry]] - prefix = "docker.io" - location = "docker.io" - ``` diff --git a/website/src/reference/orbstack.md b/website/src/reference/orbstack.md deleted file mode 100644 index cbd6566359..0000000000 --- a/website/src/reference/orbstack.md +++ /dev/null @@ -1,39 +0,0 @@ - - -# OrbStack as a docker alternative on macOS -1. Install OrbStack by downloading [installer](https://orbstack.dev/download) or using Homebrew. - ```shell - brew install orbstack - ``` - -2. Migrate Docker data - ```shell - orb migrate docker - ``` - -3. (Optional) Add registry mirrors - - You can edit the config directly at `~/.orbstack/config/docker.json` and restart the engine with `orb restart docker`. - - ``` - { - "registry-mirrors": [""] - } - ``` From 1aee142c7f74a888c8135c06826490d9fbf44b5b Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Thu, 29 Jan 2026 19:11:05 -0500 Subject: [PATCH 183/891] Fix moto server port conflict with macOS AirPlay Receiver (#2090) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This PR changes the default moto server port from `5000` to `5001`. Port 5000 is used by AirPlay Receiver on MacOS. The same was done in pyiceberg: https://github.com/apache/iceberg-python/pull/292 ## Are these changes tested? Yes locally. I also did a global search for 5000 --- crates/catalog/glue/src/utils.rs | 2 +- crates/test_utils/src/lib.rs | 2 +- dev/docker-compose.yaml | 7 +++++-- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/crates/catalog/glue/src/utils.rs b/crates/catalog/glue/src/utils.rs index 457471b34a..f3be58381a 100644 --- a/crates/catalog/glue/src/utils.rs +++ b/crates/catalog/glue/src/utils.rs @@ -435,7 +435,7 @@ mod tests { #[tokio::test] async fn test_config_with_custom_endpoint() { let properties = HashMap::new(); - let endpoint_url = "http://custom_url:5000"; + let endpoint_url = "http://custom_url:5001"; let sdk_config = create_sdk_config(&properties, Some(&endpoint_url.to_string())).await; diff --git a/crates/test_utils/src/lib.rs b/crates/test_utils/src/lib.rs index 4511231d76..e44d96c385 100644 --- a/crates/test_utils/src/lib.rs +++ b/crates/test_utils/src/lib.rs @@ -47,7 +47,7 @@ mod common { pub const DEFAULT_MINIO_PORT: u16 = 9000; pub const DEFAULT_REST_CATALOG_PORT: u16 = 8181; pub const DEFAULT_HMS_PORT: u16 = 9083; - pub const DEFAULT_GLUE_PORT: u16 = 5000; + pub const DEFAULT_GLUE_PORT: u16 = 5001; pub const DEFAULT_GCS_PORT: u16 = 4443; /// Returns the MinIO S3-compatible endpoint. diff --git a/dev/docker-compose.yaml b/dev/docker-compose.yaml index 0df017a75f..2f0d08fbd9 100644 --- a/dev/docker-compose.yaml +++ b/dev/docker-compose.yaml @@ -137,9 +137,12 @@ services: networks: iceberg_test: ports: - - "5000:5000" + - "5001:5001" + environment: + # Use port 5001 instead of default 5000 to avoid conflict with macOS AirPlay Receiver. + - MOTO_PORT=5001 healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:5000/moto-api/"] + test: ["CMD", "curl", "-f", "http://localhost:5001/moto-api/"] interval: 5s timeout: 5s retries: 5 From 8e826ec1a7b76d597960af8fcbe881b811332f32 Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Fri, 30 Jan 2026 08:18:26 +0800 Subject: [PATCH 184/891] feat(datafusion): Add Boolean predicate pushdown support (#2082) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This commit adds comprehensive support for pushing down Boolean predicates to the Iceberg table scan layer, improving query performance by filtering data at the storage level. Changes: - Enhanced expr_to_predicate.rs to handle boolean column expressions: * Bare boolean columns in filters (e.g., WHERE is_active) are converted to column = true predicates * NOT of boolean columns (e.g., WHERE NOT is_active) are converted to column = false predicates * Added Boolean scalar value to Datum conversion - Added comprehensive sqllogictest (boolean_predicate_pushdown.slt) with: * Tests for is_active = true/false with EXPLAIN verification * Tests for is_active != true with EXPLAIN verification * Tests for combined predicates (AND/OR) * Tests for IS NULL/IS NOT NULL on boolean columns - Created test_boolean_table in engine setup for testing - Updated test schedule and show_tables baseline All tests verify that predicates are successfully pushed down to IcebergTableScan, not just executed in FilterExec. ## Are these changes tested? --------- Co-authored-by: Claude Sonnet 4.5 --- .../src/physical_plan/expr_to_predicate.rs | 40 +++++- .../df_boolean_predicate_pushdown.toml | 23 ++++ .../df_test/boolean_predicate_pushdown.slt | 130 ++++++++++++++++++ 3 files changed, 191 insertions(+), 2 deletions(-) create mode 100644 crates/sqllogictest/testdata/schedules/df_boolean_predicate_pushdown.toml create mode 100644 crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt diff --git a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs index 2468606b48..c69fdbb880 100644 --- a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs +++ b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs @@ -51,8 +51,18 @@ pub fn convert_filters_to_predicate(filters: &[Expr]) -> Option { fn convert_filter_to_predicate(expr: &Expr) -> Option { match to_iceberg_predicate(expr) { TransformedResult::Predicate(predicate) => Some(predicate), - TransformedResult::Column(_) | TransformedResult::Literal(_) => { - unreachable!("Not a valid expression: {:?}", expr) + TransformedResult::Column(column) => { + // A bare column in a filter context represents a boolean column check + // Convert it to: column = true + Some(Predicate::Binary(BinaryExpression::new( + PredicateOperator::Eq, + column, + Datum::bool(true), + ))) + } + TransformedResult::Literal(_) => { + // Literal values in filter context cannot be pushed down + None } _ => None, } @@ -75,6 +85,14 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult { let expr = to_iceberg_predicate(exp); match expr { TransformedResult::Predicate(p) => TransformedResult::Predicate(!p), + TransformedResult::Column(column) => { + // NOT of a bare boolean column: NOT col => col = false + TransformedResult::Predicate(Predicate::Binary(BinaryExpression::new( + PredicateOperator::Eq, + column, + Datum::bool(false), + ))) + } _ => TransformedResult::NotTransformed, } } @@ -254,6 +272,7 @@ const MILLIS_PER_DAY: i64 = 24 * 60 * 60 * 1000; /// Convert a scalar value to an iceberg datum. fn scalar_value_to_datum(value: &ScalarValue) -> Option { match value { + ScalarValue::Boolean(Some(v)) => Some(Datum::bool(*v)), ScalarValue::Int8(Some(v)) => Some(Datum::int(*v as i32)), ScalarValue::Int16(Some(v)) => Some(Datum::int(*v as i32)), ScalarValue::Int32(Some(v)) => Some(Datum::int(*v)), @@ -509,6 +528,23 @@ mod tests { assert_eq!(predicate, expected_predicate); } + #[test] + fn test_scalar_value_to_datum_boolean() { + use datafusion::common::ScalarValue; + + // Test boolean true + let datum = super::scalar_value_to_datum(&ScalarValue::Boolean(Some(true))); + assert_eq!(datum, Some(Datum::bool(true))); + + // Test boolean false + let datum = super::scalar_value_to_datum(&ScalarValue::Boolean(Some(false))); + assert_eq!(datum, Some(Datum::bool(false))); + + // Test None boolean + let datum = super::scalar_value_to_datum(&ScalarValue::Boolean(None)); + assert_eq!(datum, None); + } + #[test] fn test_predicate_conversion_with_like_starts_with() { let sql = "bar LIKE 'test%'"; diff --git a/crates/sqllogictest/testdata/schedules/df_boolean_predicate_pushdown.toml b/crates/sqllogictest/testdata/schedules/df_boolean_predicate_pushdown.toml new file mode 100644 index 0000000000..5b4f63dc7c --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_boolean_predicate_pushdown.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/boolean_predicate_pushdown.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt new file mode 100644 index 0000000000..c7269b68b9 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Create test table with boolean column +statement ok +CREATE TABLE default.default.test_boolean_table (id INT NOT NULL, is_active BOOLEAN, description STRING) + +# Insert test data into test_boolean_table +statement ok +INSERT INTO default.default.test_boolean_table VALUES + (1, true, 'Active user'), + (2, false, 'Inactive user'), + (3, true, 'Premium member'), + (4, false, 'Trial expired'), + (5, true, 'Verified account'), + (6, NULL, 'Pending verification') + +# Verify boolean equality predicate is pushed down to IcebergTableScan +query TT +EXPLAIN SELECT * FROM default.default.test_boolean_table WHERE is_active = true +---- +logical_plan +01)Filter: default.default.test_boolean_table.is_active +02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[default.default.test_boolean_table.is_active] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: is_active@1 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = true] + +# Query with is_active = true +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active = true +---- +1 true Active user +3 true Premium member +5 true Verified account + +# Verify boolean false predicate is pushed down to IcebergTableScan +query TT +EXPLAIN SELECT * FROM default.default.test_boolean_table WHERE is_active = false +---- +logical_plan +01)Filter: NOT default.default.test_boolean_table.is_active +02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[NOT default.default.test_boolean_table.is_active] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: NOT is_active@1 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] + +# Query with is_active = false +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active = false +---- +2 false Inactive user +4 false Trial expired + +# Verify boolean NOT EQUAL predicate is pushed down +query TT +EXPLAIN SELECT * FROM default.default.test_boolean_table WHERE is_active != true +---- +logical_plan +01)Filter: NOT default.default.test_boolean_table.is_active +02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[NOT default.default.test_boolean_table.is_active] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: NOT is_active@1 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] + +# Query with is_active != true (includes false and NULL) +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active != true +---- +2 false Inactive user +4 false Trial expired + +# Test combined boolean predicates with AND +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active = true AND id > 2 +---- +3 true Premium member +5 true Verified account + +# Test combined boolean predicates with OR +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active = true OR id = 2 +---- +1 true Active user +2 false Inactive user +3 true Premium member +5 true Verified account + +# Test IS NULL on boolean column +query ITT +SELECT * FROM default.default.test_boolean_table WHERE is_active IS NULL +---- +6 NULL Pending verification + +# Test IS NOT NULL on boolean column +query ITT rowsort +SELECT * FROM default.default.test_boolean_table WHERE is_active IS NOT NULL +---- +1 true Active user +2 false Inactive user +3 true Premium member +4 false Trial expired +5 true Verified account + +# Clean up: Drop the test table +statement ok +DROP TABLE default.default.test_boolean_table From c56411a31945591dc600b95206d956ce0a7bb85c Mon Sep 17 00:00:00 2001 From: Brodie Alexander Date: Thu, 29 Jan 2026 18:51:19 -0600 Subject: [PATCH 185/891] fix: SqlCatalogBuilder does not save supplied name in SqlCatalogConfig before initializing SqlCatalog (#2079) There hasn't been any movement on #1991 for a while so I wanted to see if I could help out. I'm not sure what the correct way of trying to continue where another PR left off is, so please let me know if there is a better way of going about this! ## Which issue does this PR close? Closes #1990 Refer to PR #1991 ## What changes are included in this PR? Changes to SqlCatalogBuilder from #1991 are included but the new_sql_catalog function has been refactored to accept an optional name parameter. Relevant tests have been adjusted accordingly. ## Are these changes tested? Yes, refer to #1991 for cornerwings' added test. I've run the local `make test` and everything seems to check out. Thanks! Brodie --------- Co-authored-by: Sainath Mallidi Co-authored-by: blackmwk --- crates/catalog/sql/src/catalog.rs | 130 ++++++++++++++++++------------ 1 file changed, 79 insertions(+), 51 deletions(-) diff --git a/crates/catalog/sql/src/catalog.rs b/crates/catalog/sql/src/catalog.rs index 8209cd04c1..e3b8a17be3 100644 --- a/crates/catalog/sql/src/catalog.rs +++ b/crates/catalog/sql/src/catalog.rs @@ -136,8 +136,6 @@ impl CatalogBuilder for SqlCatalogBuilder { name: impl Into, props: HashMap, ) -> impl Future> + Send { - let name = name.into(); - for (k, v) in props { self.0.props.insert(k, v); } @@ -149,6 +147,8 @@ impl CatalogBuilder for SqlCatalogBuilder { self.0.warehouse_location = warehouse_location; } + let name = name.into(); + let mut valid_sql_bind_style = true; if let Some(sql_bind_style) = self.0.props.remove(SQL_CATALOG_PROP_BIND_STYLE) { if let Ok(sql_bind_style) = SqlBindStyle::from_str(&sql_bind_style) { @@ -158,8 +158,10 @@ impl CatalogBuilder for SqlCatalogBuilder { } } + let valid_name = !name.trim().is_empty(); + async move { - if name.trim().is_empty() { + if !valid_name { Err(Error::new( ErrorKind::DataInvalid, "Catalog name cannot be empty", @@ -175,6 +177,7 @@ impl CatalogBuilder for SqlCatalogBuilder { ), )) } else { + self.0.name = name; SqlCatalog::new(self.0).await } } @@ -1004,7 +1007,16 @@ mod tests { HashMap::from([("exists".to_string(), "true".to_string())]) } - async fn new_sql_catalog(warehouse_location: String) -> impl Catalog { + /// Create a new SQLite catalog for testing. If name is not specified it defaults to "iceberg". + async fn new_sql_catalog( + warehouse_location: String, + name: Option, + ) -> impl Catalog { + let name = if let Some(name) = name { + name.to_string() + } else { + "iceberg".to_string() + }; let sql_lite_uri = format!("sqlite:{}", temp_path()); sqlx::Sqlite::create_database(&sql_lite_uri).await.unwrap(); @@ -1017,7 +1029,7 @@ mod tests { ), ]); SqlCatalogBuilder::default() - .load("iceberg", props) + .load(&name, props) .await .unwrap() } @@ -1112,10 +1124,10 @@ mod tests { #[tokio::test] async fn test_initialized() { let warehouse_loc = temp_path(); - new_sql_catalog(warehouse_loc.clone()).await; + new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; // catalog instantiation should not fail even if tables exist - new_sql_catalog(warehouse_loc.clone()).await; - new_sql_catalog(warehouse_loc.clone()).await; + new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; + new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; } #[tokio::test] @@ -1318,15 +1330,31 @@ mod tests { #[tokio::test] async fn test_list_namespaces_returns_empty_vector() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; assert_eq!(catalog.list_namespaces(None).await.unwrap(), vec![]); } + #[tokio::test] + async fn test_list_namespaces_returns_empty_different_name() { + let warehouse_loc = temp_path(); + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; + let namespace_ident_1 = NamespaceIdent::new("a".into()); + let namespace_ident_2 = NamespaceIdent::new("b".into()); + create_namespaces(&catalog, &vec![&namespace_ident_1, &namespace_ident_2]).await; + assert_eq!( + to_set(catalog.list_namespaces(None).await.unwrap()), + to_set(vec![namespace_ident_1, namespace_ident_2]) + ); + + let catalog2 = new_sql_catalog(warehouse_loc, Some("test")).await; + assert_eq!(catalog2.list_namespaces(None).await.unwrap(), vec![]); + } + #[tokio::test] async fn test_list_namespaces_returns_multiple_namespaces() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_1 = NamespaceIdent::new("a".into()); let namespace_ident_2 = NamespaceIdent::new("b".into()); create_namespaces(&catalog, &vec![&namespace_ident_1, &namespace_ident_2]).await; @@ -1340,7 +1368,7 @@ mod tests { #[tokio::test] async fn test_list_namespaces_returns_only_top_level_namespaces() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_1 = NamespaceIdent::new("a".into()); let namespace_ident_2 = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); let namespace_ident_3 = NamespaceIdent::new("b".into()); @@ -1360,7 +1388,7 @@ mod tests { #[tokio::test] async fn test_list_namespaces_returns_no_namespaces_under_parent() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_1 = NamespaceIdent::new("a".into()); let namespace_ident_2 = NamespaceIdent::new("b".into()); create_namespaces(&catalog, &vec![&namespace_ident_1, &namespace_ident_2]).await; @@ -1377,7 +1405,7 @@ mod tests { #[tokio::test] async fn test_list_namespaces_returns_namespace_under_parent() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_1 = NamespaceIdent::new("a".into()); let namespace_ident_2 = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); let namespace_ident_3 = NamespaceIdent::new("c".into()); @@ -1405,7 +1433,7 @@ mod tests { #[tokio::test] async fn test_list_namespaces_returns_multiple_namespaces_under_parent() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_1 = NamespaceIdent::new("a".to_string()); let namespace_ident_2 = NamespaceIdent::from_strs(vec!["a", "a"]).unwrap(); let namespace_ident_3 = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); @@ -1438,7 +1466,7 @@ mod tests { #[tokio::test] async fn test_namespace_exists_returns_false() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1453,7 +1481,7 @@ mod tests { #[tokio::test] async fn test_namespace_exists_returns_true() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1463,7 +1491,7 @@ mod tests { #[tokio::test] async fn test_create_namespace_with_properties() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("abc".into()); let mut properties = default_properties(); @@ -1486,7 +1514,7 @@ mod tests { #[tokio::test] async fn test_create_namespace_throws_error_if_namespace_already_exists() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1511,7 +1539,7 @@ mod tests { #[tokio::test] async fn test_create_nested_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let parent_namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &parent_namespace_ident).await; @@ -1534,7 +1562,7 @@ mod tests { #[tokio::test] async fn test_create_deeply_nested_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_a = NamespaceIdent::new("a".into()); let namespace_ident_a_b = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); create_namespaces(&catalog, &vec![&namespace_ident_a, &namespace_ident_a_b]).await; @@ -1558,7 +1586,7 @@ mod tests { #[tokio::test] async fn test_update_namespace_noop() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1580,7 +1608,7 @@ mod tests { #[tokio::test] async fn test_update_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1609,7 +1637,7 @@ mod tests { #[tokio::test] async fn test_update_nested_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::from_strs(["a", "b"]).unwrap(); create_namespace(&catalog, &namespace_ident).await; @@ -1638,7 +1666,7 @@ mod tests { #[tokio::test] async fn test_update_namespace_errors_if_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); let props = HashMap::from_iter([ @@ -1660,7 +1688,7 @@ mod tests { #[tokio::test] async fn test_update_namespace_errors_if_nested_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::from_strs(["a", "b"]).unwrap(); let props = HashMap::from_iter([ @@ -1682,7 +1710,7 @@ mod tests { #[tokio::test] async fn test_drop_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("abc".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1694,7 +1722,7 @@ mod tests { #[tokio::test] async fn test_drop_nested_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_a = NamespaceIdent::new("a".into()); let namespace_ident_a_b = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); create_namespaces(&catalog, &vec![&namespace_ident_a, &namespace_ident_a_b]).await; @@ -1714,7 +1742,7 @@ mod tests { #[tokio::test] async fn test_drop_deeply_nested_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_a = NamespaceIdent::new("a".into()); let namespace_ident_a_b = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); let namespace_ident_a_b_c = NamespaceIdent::from_strs(vec!["a", "b", "c"]).unwrap(); @@ -1750,7 +1778,7 @@ mod tests { #[tokio::test] async fn test_drop_namespace_throws_error_if_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let non_existent_namespace_ident = NamespaceIdent::new("abc".into()); assert_eq!( @@ -1766,7 +1794,7 @@ mod tests { #[tokio::test] async fn test_drop_namespace_throws_error_if_nested_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; create_namespace(&catalog, &NamespaceIdent::new("a".into())).await; let non_existent_namespace_ident = @@ -1784,7 +1812,7 @@ mod tests { #[tokio::test] async fn test_dropping_a_namespace_does_not_drop_namespaces_nested_under_that_one() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_a = NamespaceIdent::new("a".into()); let namespace_ident_a_b = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); create_namespaces(&catalog, &vec![&namespace_ident_a, &namespace_ident_a_b]).await; @@ -1804,7 +1832,7 @@ mod tests { #[tokio::test] async fn test_list_tables_returns_empty_vector() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1814,7 +1842,7 @@ mod tests { #[tokio::test] async fn test_list_tables_throws_error_if_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let non_existent_namespace_ident = NamespaceIdent::new("n1".into()); @@ -1831,7 +1859,7 @@ mod tests { #[tokio::test] async fn test_create_table_with_location() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -1870,7 +1898,7 @@ mod tests { #[tokio::test] async fn test_create_table_falls_back_to_namespace_location_if_table_location_is_missing() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); let mut namespace_properties = HashMap::new(); @@ -1912,7 +1940,7 @@ mod tests { async fn test_create_table_in_nested_namespace_falls_back_to_nested_namespace_location_if_table_location_is_missing() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); let mut namespace_properties = HashMap::new(); @@ -1968,7 +1996,7 @@ mod tests { async fn test_create_table_falls_back_to_warehouse_location_if_both_table_location_and_namespace_location_are_missing() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); // note: no location specified in namespace_properties @@ -2006,7 +2034,7 @@ mod tests { async fn test_create_table_in_nested_namespace_falls_back_to_warehouse_location_if_both_table_location_and_namespace_location_are_missing() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -2042,7 +2070,7 @@ mod tests { #[tokio::test] async fn test_create_table_throws_error_if_table_with_same_name_already_exists() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; let table_name = "tbl1"; @@ -2072,7 +2100,7 @@ mod tests { #[tokio::test] async fn test_rename_table_in_same_namespace() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("n1".into()); create_namespace(&catalog, &namespace_ident).await; let src_table_ident = TableIdent::new(namespace_ident.clone(), "tbl1".into()); @@ -2092,7 +2120,7 @@ mod tests { #[tokio::test] async fn test_rename_table_across_namespaces() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let src_namespace_ident = NamespaceIdent::new("a".into()); let dst_namespace_ident = NamespaceIdent::new("b".into()); create_namespaces(&catalog, &vec![&src_namespace_ident, &dst_namespace_ident]).await; @@ -2119,7 +2147,7 @@ mod tests { #[tokio::test] async fn test_rename_table_src_table_is_same_as_dst_table() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("n1".into()); create_namespace(&catalog, &namespace_ident).await; let table_ident = TableIdent::new(namespace_ident.clone(), "tbl".into()); @@ -2138,7 +2166,7 @@ mod tests { #[tokio::test] async fn test_rename_table_across_nested_namespaces() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident_a = NamespaceIdent::new("a".into()); let namespace_ident_a_b = NamespaceIdent::from_strs(vec!["a", "b"]).unwrap(); let namespace_ident_a_b_c = NamespaceIdent::from_strs(vec!["a", "b", "c"]).unwrap(); @@ -2166,7 +2194,7 @@ mod tests { #[tokio::test] async fn test_rename_table_throws_error_if_dst_namespace_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let src_namespace_ident = NamespaceIdent::new("n1".into()); let src_table_ident = TableIdent::new(src_namespace_ident.clone(), "tbl1".into()); create_namespace(&catalog, &src_namespace_ident).await; @@ -2188,7 +2216,7 @@ mod tests { #[tokio::test] async fn test_rename_table_throws_error_if_src_table_doesnt_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("n1".into()); create_namespace(&catalog, &namespace_ident).await; let src_table_ident = TableIdent::new(namespace_ident.clone(), "tbl1".into()); @@ -2207,7 +2235,7 @@ mod tests { #[tokio::test] async fn test_rename_table_throws_error_if_dst_table_already_exists() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("n1".into()); create_namespace(&catalog, &namespace_ident).await; let src_table_ident = TableIdent::new(namespace_ident.clone(), "tbl1".into()); @@ -2227,7 +2255,7 @@ mod tests { #[tokio::test] async fn test_drop_table_throws_error_if_table_not_exist() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); let table_name = "tbl1"; let table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); @@ -2247,7 +2275,7 @@ mod tests { #[tokio::test] async fn test_drop_table() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); let table_name = "tbl1"; let table_ident = TableIdent::new(namespace_ident.clone(), table_name.into()); @@ -2283,7 +2311,7 @@ mod tests { #[tokio::test] async fn test_register_table_throws_error_if_table_with_same_name_already_exists() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; let table_name = "tbl1"; @@ -2303,7 +2331,7 @@ mod tests { #[tokio::test] async fn test_register_table() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc.clone()).await; + let catalog = new_sql_catalog(warehouse_loc.clone(), Some("iceberg")).await; let namespace_ident = NamespaceIdent::new("a".into()); create_namespace(&catalog, &namespace_ident).await; @@ -2342,7 +2370,7 @@ mod tests { #[tokio::test] async fn test_update_table() { let warehouse_loc = temp_path(); - let catalog = new_sql_catalog(warehouse_loc).await; + let catalog = new_sql_catalog(warehouse_loc, Some("iceberg")).await; // Create a test namespace and table let namespace_ident = NamespaceIdent::new("ns1".into()); From 0ffd2f24dcba928803d59b14915597b29e40cedf Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Fri, 30 Jan 2026 14:12:09 +0800 Subject: [PATCH 186/891] feat(datafusion): Add Timestamp scalar value conversion for predicate pushdown (#2069) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Add support for converting DataFusion Timestamp scalar values to Iceberg Datum for predicate pushdown. This enables timestamp literal comparisons to be pushed down to the storage layer when DataFusion provides timestamp literals in ScalarValue form, improving query performance. Changes: - Add conversion for 2 timestamp time units (Microsecond, Nanosecond) to Iceberg's native representations - Add comprehensive unit tests covering all time units and edge cases - Add sqllogictest (timestamp_predicate_pushdown.slt) to validate end-to-end timestamp predicate filtering and pushdown behavior Implementation details: - Iceberg uses microseconds and nanoseconds as the native timestamp representations - Timezone information in ScalarValue is preserved but not used in conversion (Iceberg timestamp type is timezone-agnostic) ## Are these changes tested? --------- Co-authored-by: Claude Sonnet 4.5 --- crates/iceberg/src/arrow/schema.rs | 8 +- crates/iceberg/src/spec/values/datum.rs | 12 ++ .../src/physical_plan/expr_to_predicate.rs | 44 ++++ .../df_timestamp_predicate_pushdown.toml | 23 +++ .../df_test/timestamp_predicate_pushdown.slt | 192 ++++++++++++++++++ 5 files changed, 278 insertions(+), 1 deletion(-) create mode 100644 crates/sqllogictest/testdata/schedules/df_timestamp_predicate_pushdown.toml create mode 100644 crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt diff --git a/crates/iceberg/src/arrow/schema.rs b/crates/iceberg/src/arrow/schema.rs index e00f79a3bc..bd9e249f48 100644 --- a/crates/iceberg/src/arrow/schema.rs +++ b/crates/iceberg/src/arrow/schema.rs @@ -24,7 +24,7 @@ use arrow_array::types::{Decimal128Type, validate_decimal_precision_and_scale}; use arrow_array::{ BinaryArray, BooleanArray, Date32Array, Datum as ArrowDatum, Decimal128Array, FixedSizeBinaryArray, Float32Array, Float64Array, Int32Array, Int64Array, Scalar, StringArray, - TimestampMicrosecondArray, + TimestampMicrosecondArray, TimestampNanosecondArray, }; use arrow_schema::{DataType, Field, Fields, Schema as ArrowSchema, TimeUnit}; use parquet::arrow::PARQUET_FIELD_ID_META_KEY; @@ -742,6 +742,12 @@ pub(crate) fn get_arrow_datum(datum: &Datum) -> Result Ok(Arc::new(Scalar::new( TimestampMicrosecondArray::new(vec![*value; 1].into(), None).with_timezone_utc(), ))), + (PrimitiveType::TimestampNs, PrimitiveLiteral::Long(value)) => { + Ok(Arc::new(TimestampNanosecondArray::new_scalar(*value))) + } + (PrimitiveType::TimestamptzNs, PrimitiveLiteral::Long(value)) => Ok(Arc::new(Scalar::new( + TimestampNanosecondArray::new(vec![*value; 1].into(), None).with_timezone_utc(), + ))), (PrimitiveType::Decimal { precision, scale }, PrimitiveLiteral::Int128(value)) => { let array = Decimal128Array::from_value(*value, 1) .with_precision_and_scale(*precision as _, *scale as _) diff --git a/crates/iceberg/src/spec/values/datum.rs b/crates/iceberg/src/spec/values/datum.rs index 3d4abc019d..68ea6b3d46 100644 --- a/crates/iceberg/src/spec/values/datum.rs +++ b/crates/iceberg/src/spec/values/datum.rs @@ -233,6 +233,18 @@ impl PartialOrd for Datum { PrimitiveType::Timestamptz, PrimitiveType::Timestamptz, ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::TimestampNs, + PrimitiveType::TimestampNs, + ) => val.partial_cmp(other_val), + ( + PrimitiveLiteral::Long(val), + PrimitiveLiteral::Long(other_val), + PrimitiveType::TimestamptzNs, + PrimitiveType::TimestamptzNs, + ) => val.partial_cmp(other_val), ( PrimitiveLiteral::String(val), PrimitiveLiteral::String(other_val), diff --git a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs index c69fdbb880..9f37345f86 100644 --- a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs +++ b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs @@ -269,6 +269,7 @@ fn reverse_predicate_operator(op: PredicateOperator) -> PredicateOperator { } const MILLIS_PER_DAY: i64 = 24 * 60 * 60 * 1000; + /// Convert a scalar value to an iceberg datum. fn scalar_value_to_datum(value: &ScalarValue) -> Option { match value { @@ -285,6 +286,13 @@ fn scalar_value_to_datum(value: &ScalarValue) -> Option { ScalarValue::LargeBinary(Some(v)) => Some(Datum::binary(v.clone())), ScalarValue::Date32(Some(v)) => Some(Datum::date(*v)), ScalarValue::Date64(Some(v)) => Some(Datum::date((*v / MILLIS_PER_DAY) as i32)), + // Timestamp conversions + // Note: TimestampSecond and TimestampMillisecond are not handled here because + // DataFusion's type coercion always converts them to match the column type + // (either TimestampMicrosecond or TimestampNanosecond) before predicate pushdown. + // See unit tests for how those conversions would work if needed. + ScalarValue::TimestampMicrosecond(Some(v), _) => Some(Datum::timestamp_micros(*v)), + ScalarValue::TimestampNanosecond(Some(v), _) => Some(Datum::timestamp_nanos(*v)), _ => None, } } @@ -501,6 +509,42 @@ mod tests { assert_eq!(predicate, None); } + #[test] + fn test_scalar_value_to_datum_timestamp() { + use datafusion::common::ScalarValue; + + // Test TimestampMicrosecond - maps directly to Datum::timestamp_micros + let ts_micros = 1672876800000000i64; // 2023-01-05 00:00:00 UTC in microseconds + let datum = + super::scalar_value_to_datum(&ScalarValue::TimestampMicrosecond(Some(ts_micros), None)); + assert_eq!(datum, Some(Datum::timestamp_micros(ts_micros))); + + // Test TimestampNanosecond - maps to Datum::timestamp_nanos to preserve precision + let ts_nanos = 1672876800000000500i64; // 2023-01-05 00:00:00.000000500 UTC in nanoseconds + let datum = + super::scalar_value_to_datum(&ScalarValue::TimestampNanosecond(Some(ts_nanos), None)); + assert_eq!(datum, Some(Datum::timestamp_nanos(ts_nanos))); + + // Test None timestamp + let datum = super::scalar_value_to_datum(&ScalarValue::TimestampMicrosecond(None, None)); + assert_eq!(datum, None); + + // Note: TimestampSecond and TimestampMillisecond are not supported because + // DataFusion's type coercion converts them to TimestampMicrosecond or TimestampNanosecond + // before they reach scalar_value_to_datum in SQL queries. + // + // These return None (not pushed down): + let ts_seconds = 1672876800i64; // 2023-01-05 00:00:00 UTC in seconds + let datum = + super::scalar_value_to_datum(&ScalarValue::TimestampSecond(Some(ts_seconds), None)); + assert_eq!(datum, None); + + let ts_millis = 1672876800000i64; // 2023-01-05 00:00:00 UTC in milliseconds + let datum = + super::scalar_value_to_datum(&ScalarValue::TimestampMillisecond(Some(ts_millis), None)); + assert_eq!(datum, None); + } + #[test] fn test_scalar_value_to_datum_binary() { use datafusion::common::ScalarValue; diff --git a/crates/sqllogictest/testdata/schedules/df_timestamp_predicate_pushdown.toml b/crates/sqllogictest/testdata/schedules/df_timestamp_predicate_pushdown.toml new file mode 100644 index 0000000000..849bba50a2 --- /dev/null +++ b/crates/sqllogictest/testdata/schedules/df_timestamp_predicate_pushdown.toml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[engines] +df = { type = "datafusion" } + +[[steps]] +engine = "df" +slt = "df_test/timestamp_predicate_pushdown.slt" diff --git a/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt new file mode 100644 index 0000000000..255731c046 --- /dev/null +++ b/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt @@ -0,0 +1,192 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Test timestamp predicate pushdown behavior +# +# When you CREATE TABLE with TIMESTAMP in DataFusion, it creates a TimestampNs column +# (nanosecond precision) in Iceberg, since DataFusion's default TIMESTAMP type is nanoseconds. +# +# We use Datum::timestamp_nanos() for nanosecond timestamp predicates to preserve +# full precision. This allows predicates to be correctly pushed down to Iceberg. + +# Create test table with timestamp column +statement ok +CREATE TABLE default.default.test_timestamp_table (id INT NOT NULL, ts TIMESTAMP) + +# Insert test data with timestamps +# We use CAST to convert string timestamps to proper timestamp values +query I +INSERT INTO default.default.test_timestamp_table +VALUES + (1, CAST('2023-01-01 00:00:00' AS TIMESTAMP)), + (2, CAST('2023-01-05 12:30:00' AS TIMESTAMP)), + (3, CAST('2023-01-10 15:45:30' AS TIMESTAMP)), + (4, CAST('2023-01-15 09:00:00' AS TIMESTAMP)), + (5, CAST('2023-01-20 18:20:10' AS TIMESTAMP)) +---- +5 + +# Verify timestamp equality predicate IS pushed down +query TT +EXPLAIN SELECT * FROM default.default.test_timestamp_table WHERE ts = CAST('2023-01-05 12:30:00' AS TIMESTAMP) +---- +logical_plan +01)Filter: default.default.test_timestamp_table.ts = TimestampNanosecond(1672921800000000000, None) +02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts = TimestampNanosecond(1672921800000000000, None)] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: ts@1 = 1672921800000000000 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,ts] predicate:[ts = 2023-01-05 12:30:00] + +# Verify timestamp equality filtering works +query I? +SELECT * FROM default.default.test_timestamp_table WHERE ts = CAST('2023-01-05 12:30:00' AS TIMESTAMP) +---- +2 2023-01-05T12:30:00 + +# Verify timestamp greater than predicate IS pushed down +query TT +EXPLAIN SELECT * FROM default.default.test_timestamp_table WHERE ts > CAST('2023-01-10 00:00:00' AS TIMESTAMP) +---- +logical_plan +01)Filter: default.default.test_timestamp_table.ts > TimestampNanosecond(1673308800000000000, None) +02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts > TimestampNanosecond(1673308800000000000, None)] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: ts@1 > 1673308800000000000 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-10 00:00:00] + +# Verify timestamp greater than filtering +query I? rowsort +SELECT * FROM default.default.test_timestamp_table WHERE ts > CAST('2023-01-10 00:00:00' AS TIMESTAMP) +---- +3 2023-01-10T15:45:30 +4 2023-01-15T09:00:00 +5 2023-01-20T18:20:10 + +# Test timestamp less than or equal filtering +query I? rowsort +SELECT * FROM default.default.test_timestamp_table WHERE ts <= CAST('2023-01-05 12:30:00' AS TIMESTAMP) +---- +1 2023-01-01T00:00:00 +2 2023-01-05T12:30:00 + +# Verify timestamp range predicate (AND of two comparisons) IS pushed down +query TT +EXPLAIN SELECT * FROM default.default.test_timestamp_table +WHERE ts >= CAST('2023-01-05 00:00:00' AS TIMESTAMP) + AND ts <= CAST('2023-01-15 23:59:59' AS TIMESTAMP) +---- +logical_plan +01)Filter: default.default.test_timestamp_table.ts >= TimestampNanosecond(1672876800000000000, None) AND default.default.test_timestamp_table.ts <= TimestampNanosecond(1673827199000000000, None) +02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts >= TimestampNanosecond(1672876800000000000, None), default.default.test_timestamp_table.ts <= TimestampNanosecond(1673827199000000000, None)] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: ts@1 >= 1672876800000000000 AND ts@1 <= 1673827199000000000 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,ts] predicate:[(ts >= 2023-01-05 00:00:00) AND (ts <= 2023-01-15 23:59:59)] + +# Test timestamp range predicate filtering +query I? rowsort +SELECT * FROM default.default.test_timestamp_table +WHERE ts >= CAST('2023-01-05 00:00:00' AS TIMESTAMP) + AND ts <= CAST('2023-01-15 23:59:59' AS TIMESTAMP) +---- +2 2023-01-05T12:30:00 +3 2023-01-10T15:45:30 +4 2023-01-15T09:00:00 + +# Test timestamp predicate combined with other predicates +query I? rowsort +SELECT * FROM default.default.test_timestamp_table +WHERE ts >= CAST('2023-01-10 00:00:00' AS TIMESTAMP) AND id < 5 +---- +3 2023-01-10T15:45:30 +4 2023-01-15T09:00:00 + +# Test timestamp NOT EQUAL predicate +query I? rowsort +SELECT * FROM default.default.test_timestamp_table WHERE ts != CAST('2023-01-05 12:30:00' AS TIMESTAMP) +---- +1 2023-01-01T00:00:00 +3 2023-01-10T15:45:30 +4 2023-01-15T09:00:00 +5 2023-01-20T18:20:10 + +# Test timestamp less than filtering +query I? rowsort +SELECT * FROM default.default.test_timestamp_table WHERE ts < CAST('2023-01-05 00:00:00' AS TIMESTAMP) +---- +1 2023-01-01T00:00:00 + +# Clean up: Drop the test table +statement ok +DROP TABLE default.default.test_timestamp_table + +# ============================================================================ +# Test timestamp predicate pushdown with different precisions +# ============================================================================ + +# Test with TIMESTAMP(6) - microsecond precision +statement ok +CREATE TABLE default.default.test_timestamp_micros (id INT NOT NULL, ts TIMESTAMP(6)) + +query I +INSERT INTO default.default.test_timestamp_micros +VALUES + (1, CAST('2023-01-01 00:00:00' AS TIMESTAMP)), + (2, CAST('2023-01-05 12:30:00' AS TIMESTAMP)) +---- +2 + +# Verify microsecond timestamp predicate is pushed down +query TT +EXPLAIN SELECT * FROM default.default.test_timestamp_micros WHERE ts > CAST('2023-01-01 00:00:00' AS TIMESTAMP) +---- +logical_plan +01)Filter: default.default.test_timestamp_micros.ts > TimestampMicrosecond(1672531200000000, None) +02)--TableScan: default.default.test_timestamp_micros projection=[id, ts], partial_filters=[default.default.test_timestamp_micros.ts > TimestampMicrosecond(1672531200000000, None)] +physical_plan +01)CoalesceBatchesExec: target_batch_size=8192 +02)--FilterExec: ts@1 > 1672531200000000 +03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +04)------CooperativeExec +05)--------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-01 00:00:00] + +query I? +SELECT * FROM default.default.test_timestamp_micros WHERE ts > CAST('2023-01-01 00:00:00' AS TIMESTAMP) +---- +2 2023-01-05T12:30:00 + +statement ok +DROP TABLE default.default.test_timestamp_micros + +# Test with TIMESTAMP(3) - millisecond precision +# This should fail because Iceberg doesn't support millisecond precision +statement error DataFusion error: External error: DataInvalid => Unsupported Arrow data type: Timestamp\(ms\) +CREATE TABLE default.default.test_timestamp_millis (id INT NOT NULL, ts TIMESTAMP(3)) + +# Test with TIMESTAMP(0) - second precision +# This should fail because Iceberg doesn't support second precision +statement error DataFusion error: External error: DataInvalid => Unsupported Arrow data type: Timestamp\(s\) +CREATE TABLE default.default.test_timestamp_seconds (id INT NOT NULL, ts TIMESTAMP(0)) + From 544aa71ebaa9635fceb6a3cae542396ed814a646 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Fri, 30 Jan 2026 01:24:16 -0500 Subject: [PATCH 187/891] chore(python): Use PyPI release of pyiceberg instead of git dependency (#2095) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Following up to https://github.com/apache/iceberg-rust/pull/1796/files#diff-bf903f17157a753e10c075d464383a2e36fc657641d17673f054239f8d1a3999R55 This PR changes pyiceberg dependency to use the latest version instead of commit hash ## Are these changes tested? --- bindings/python/Cargo.lock | 397 ++++++------------------- bindings/python/pyproject.toml | 2 +- bindings/python/tests/test_manifest.py | 34 ++- bindings/python/uv.lock | 149 ++++------ 4 files changed, 176 insertions(+), 406 deletions(-) diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index d33abed581..2bd52ef4cc 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -56,17 +56,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" -[[package]] -name = "ahash" -version = "0.7.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9" -dependencies = [ - "getrandom 0.2.16", - "once_cell", - "version_check", -] - [[package]] name = "ahash" version = "0.8.12" @@ -220,7 +209,7 @@ version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "219fe420e6800979744c8393b687afb0252b3f8a89b91027d27887b72aa36d31" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-buffer", "arrow-data", "arrow-schema", @@ -389,7 +378,7 @@ version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bafa595babaad59f2455f4957d0f26448fb472722c186739f4fac0823a1bdb47" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-array", "arrow-buffer", "arrow-data", @@ -544,18 +533,6 @@ version = "2.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "812e12b5285cc515a9c72a5c1d3b6d46a19dac5acfef5265968c166106e31dd3" -[[package]] -name = "bitvec" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" -dependencies = [ - "funty", - "radium", - "tap", - "wyz", -] - [[package]] name = "blake2" version = "0.10.6" @@ -587,6 +564,16 @@ dependencies = [ "generic-array", ] +[[package]] +name = "bnum" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f781dba93de3a5ef6dc5b17c9958b208f6f3f021623b360fb605ea51ce443f10" +dependencies = [ + "serde", + "serde-big-array", +] + [[package]] name = "bon" version = "3.8.1" @@ -612,29 +599,6 @@ dependencies = [ "syn 2.0.108", ] -[[package]] -name = "borsh" -version = "1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8646f98db542e39fc66e68a20b2144f6a732636df7c2354e74645faaa433ce" -dependencies = [ - "borsh-derive", - "cfg_aliases", -] - -[[package]] -name = "borsh-derive" -version = "1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdd1d3c0c2f5833f22386f252fe8ed005c7f59fdcddeef025c01b4c3b9fd9ac3" -dependencies = [ - "once_cell", - "proc-macro-crate", - "proc-macro2", - "quote", - "syn 2.0.108", -] - [[package]] name = "brotli" version = "8.0.2" @@ -662,51 +626,6 @@ version = "3.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43" -[[package]] -name = "bytecheck" -version = "0.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2" -dependencies = [ - "bytecheck_derive 0.6.12", - "ptr_meta 0.1.4", - "simdutf8", -] - -[[package]] -name = "bytecheck" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0caa33a2c0edca0419d15ac723dff03f1956f7978329b1e3b5fdaaaed9d3ca8b" -dependencies = [ - "bytecheck_derive 0.8.2", - "ptr_meta 0.3.1", - "rancor", - "simdutf8", -] - -[[package]] -name = "bytecheck_derive" -version = "0.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3db406d29fbcd95542e92559bed4d8ad92636d1ca8b3b72ede10b4bcc010e659" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "bytecheck_derive" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89385e82b5d1821d2219e0b095efa2cc1f246cbf99080f3be46a1a85c0d392d9" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.108", -] - [[package]] name = "bytemuck" version = "1.24.0" @@ -1166,7 +1085,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "arrow-ipc", "chrono", @@ -1450,7 +1369,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-doc", @@ -1471,7 +1390,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-expr-common", @@ -1582,7 +1501,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-expr", @@ -1619,7 +1538,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-expr-common", @@ -1652,7 +1571,7 @@ version = "51.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "arrow-ord", "arrow-schema", @@ -1861,6 +1780,17 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "erased-serde" +version = "0.4.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89e8918065695684b2b0702da20382d5ae6065cf3327bc2d6436bd49a71ce9f3" +dependencies = [ + "serde", + "serde_core", + "typeid", +] + [[package]] name = "errno" version = "0.3.14" @@ -1902,6 +1832,18 @@ dependencies = [ "once_cell", ] +[[package]] +name = "fastnum" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4089ab2dfd45d8ddc92febb5ca80644389d5ebb954f40231274a3f18341762e2" +dependencies = [ + "bnum", + "num-integer", + "num-traits", + "serde", +] + [[package]] name = "fastrand" version = "2.3.0" @@ -1962,12 +1904,6 @@ dependencies = [ "percent-encoding", ] -[[package]] -name = "funty" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" - [[package]] name = "futures" version = "0.3.31" @@ -2144,9 +2080,6 @@ name = "hashbrown" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" -dependencies = [ - "ahash 0.7.8", -] [[package]] name = "hashbrown" @@ -2154,7 +2087,7 @@ version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" dependencies = [ - "ahash 0.8.12", + "ahash", "allocator-api2", ] @@ -2359,13 +2292,13 @@ dependencies = [ "chrono", "derive_builder", "expect-test", + "fastnum", "flate2", "fnv", "futures", "itertools 0.13.0", "moka", "murmur3", - "num-bigint", "once_cell", "opendal", "ordered-float 4.6.0", @@ -2373,9 +2306,7 @@ dependencies = [ "rand 0.8.5", "reqsign", "reqwest", - "rkyv 0.8.13", "roaring", - "rust_decimal", "serde", "serde_bytes", "serde_derive", @@ -2385,6 +2316,7 @@ dependencies = [ "strum 0.27.2", "tokio", "typed-builder", + "typetag", "url", "uuid", "zstd", @@ -2396,6 +2328,7 @@ version = "0.8.0" dependencies = [ "anyhow", "async-trait", + "dashmap", "datafusion", "futures", "iceberg", @@ -2550,6 +2483,15 @@ version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" +[[package]] +name = "inventory" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc61209c082fbeb19919bee74b176221b27223e27b65d781eb91af24eb1fb46e" +dependencies = [ + "rustversion", +] + [[package]] name = "ipnet" version = "2.11.0" @@ -2865,26 +2807,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "munge" -version = "0.4.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e17401f259eba956ca16491461b6e8f72913a0a114e39736ce404410f915a0c" -dependencies = [ - "munge_macro", -] - -[[package]] -name = "munge_macro" -version = "0.4.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.108", -] - [[package]] name = "murmur3" version = "0.5.2" @@ -3067,7 +2989,7 @@ version = "57.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a0f31027ef1af7549f7cec603a9a21dce706d3f8d7c2060a68f43c1773be95a" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-array", "arrow-buffer", "arrow-cast", @@ -3258,46 +3180,6 @@ dependencies = [ "cc", ] -[[package]] -name = "ptr_meta" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1" -dependencies = [ - "ptr_meta_derive 0.1.4", -] - -[[package]] -name = "ptr_meta" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b9a0cf95a1196af61d4f1cbdab967179516d9a4a4312af1f31948f8f6224a79" -dependencies = [ - "ptr_meta_derive 0.3.1", -] - -[[package]] -name = "ptr_meta_derive" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "ptr_meta_derive" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.108", -] - [[package]] name = "pyiceberg_core_rust" version = "0.8.0" @@ -3307,7 +3189,6 @@ dependencies = [ "iceberg", "iceberg-datafusion", "pyo3", - "rust_decimal", "tokio", ] @@ -3468,21 +3349,6 @@ version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" -[[package]] -name = "radium" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" - -[[package]] -name = "rancor" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a063ea72381527c2a0561da9c80000ef822bdd7c3241b1cc1b12100e3df081ee" -dependencies = [ - "ptr_meta 0.3.1", -] - [[package]] name = "rand" version = "0.8.5" @@ -3632,24 +3498,6 @@ version = "1.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" -[[package]] -name = "rend" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c" -dependencies = [ - "bytecheck 0.6.12", -] - -[[package]] -name = "rend" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cadadef317c2f20755a64d7fdc48f9e7178ee6b0e1f7fce33fa60f1d68a276e6" -dependencies = [ - "bytecheck 0.8.2", -] - [[package]] name = "repr_offset" version = "0.2.2" @@ -3743,65 +3591,6 @@ dependencies = [ "windows-sys 0.52.0", ] -[[package]] -name = "rkyv" -version = "0.7.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9008cd6385b9e161d8229e1f6549dd23c3d022f132a2ea37ac3a10ac4935779b" -dependencies = [ - "bitvec", - "bytecheck 0.6.12", - "bytes", - "hashbrown 0.12.3", - "ptr_meta 0.1.4", - "rend 0.4.2", - "rkyv_derive 0.7.45", - "seahash", - "tinyvec", - "uuid", -] - -[[package]] -name = "rkyv" -version = "0.8.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b2e88acca7157d83d789836a3987dafc12bc3d88a050e54b8fe9ea4aaa29d20" -dependencies = [ - "bytecheck 0.8.2", - "bytes", - "hashbrown 0.16.0", - "indexmap 2.12.0", - "munge", - "ptr_meta 0.3.1", - "rancor", - "rend 0.5.3", - "rkyv_derive 0.8.13", - "tinyvec", - "uuid", -] - -[[package]] -name = "rkyv_derive" -version = "0.7.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "503d1d27590a2b0a3a4ca4c94755aa2875657196ecbf401a42eff41d7de532c0" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "rkyv_derive" -version = "0.8.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f6dffea3c91fa91a3c0fc8a061b0e27fef25c6304728038a6d6bcb1c58ba9bd" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.108", -] - [[package]] name = "roaring" version = "0.11.2" @@ -3851,22 +3640,6 @@ dependencies = [ "ordered-multimap", ] -[[package]] -name = "rust_decimal" -version = "1.39.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35affe401787a9bd846712274d97654355d21b2a2c092a3139aabe31e9022282" -dependencies = [ - "arrayvec", - "borsh", - "bytes", - "num-traits", - "rand 0.8.5", - "rkyv 0.7.45", - "serde", - "serde_json", -] - [[package]] name = "rustc-hash" version = "2.1.1" @@ -3981,12 +3754,6 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" -[[package]] -name = "seahash" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b" - [[package]] name = "semver" version = "1.0.27" @@ -4009,6 +3776,15 @@ dependencies = [ "serde_derive", ] +[[package]] +name = "serde-big-array" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11fc7cc2c76d73e0f27ee52abbd64eec84d46f370c88371120433196934e4b7f" +dependencies = [ + "serde", +] + [[package]] name = "serde_bytes" version = "0.11.19" @@ -4321,12 +4097,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" -[[package]] -name = "tap" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" - [[package]] name = "target-lexicon" version = "0.13.3" @@ -4650,12 +4420,42 @@ dependencies = [ "syn 2.0.108", ] +[[package]] +name = "typeid" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c" + [[package]] name = "typenum" version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" +[[package]] +name = "typetag" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be2212c8a9b9bcfca32024de14998494cf9a5dfa59ea1b829de98bac374b86bf" +dependencies = [ + "erased-serde", + "inventory", + "once_cell", + "serde", + "typetag-impl", +] + +[[package]] +name = "typetag-impl" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27a7a9b72ba121f6f1f6c3632b85604cac41aedb5ddc70accbebb6cac83de846" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.108", +] + [[package]] name = "typewit" version = "1.14.2" @@ -5138,15 +4938,6 @@ version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" -[[package]] -name = "wyz" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" -dependencies = [ - "tap", -] - [[package]] name = "xz2" version = "0.1.7" diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index c79c9b520c..14454b4408 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -52,6 +52,6 @@ dev = [ "maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==50.*", - "pyiceberg[sql-sqlite,pyarrow] @ git+https://github.com/apache/iceberg-python.git@d3eb149fe99b14e714a06e727b69cd617c6c052d", + "pyiceberg[sql-sqlite,pyarrow]", "fastavro>=1.11.1", ] diff --git a/bindings/python/tests/test_manifest.py b/bindings/python/tests/test_manifest.py index 701eac2509..096e549937 100644 --- a/bindings/python/tests/test_manifest.py +++ b/bindings/python/tests/test_manifest.py @@ -50,22 +50,24 @@ def fetch_manifest_entry( # have the write part in there as well def _convert_entry(entry: Any) -> ManifestEntry: data_file = DataFile( - DataFileContent(entry.data_file.content), - entry.data_file.file_path, - FileFormat(entry.data_file.file_format), - [p.value() if p is not None else None for p in entry.data_file.partition], - entry.data_file.record_count, - entry.data_file.file_size_in_bytes, - entry.data_file.column_sizes, - entry.data_file.value_counts, - entry.data_file.null_value_counts, - entry.data_file.nan_value_counts, - entry.data_file.lower_bounds, - entry.data_file.upper_bounds, - entry.data_file.key_metadata, - entry.data_file.split_offsets, - entry.data_file.equality_ids, - entry.data_file.sort_order_id, + content=DataFileContent(entry.data_file.content), + file_path=entry.data_file.file_path, + file_format=FileFormat(entry.data_file.file_format), + partition=[ + p.value() if p is not None else None for p in entry.data_file.partition + ], + record_count=entry.data_file.record_count, + file_size_in_bytes=entry.data_file.file_size_in_bytes, + column_sizes=entry.data_file.column_sizes, + value_counts=entry.data_file.value_counts, + null_value_counts=entry.data_file.null_value_counts, + nan_value_counts=entry.data_file.nan_value_counts, + lower_bounds=entry.data_file.lower_bounds, + upper_bounds=entry.data_file.upper_bounds, + key_metadata=entry.data_file.key_metadata, + split_offsets=entry.data_file.split_offsets, + equality_ids=entry.data_file.equality_ids, + sort_order_id=entry.data_file.sort_order_id, ) return ManifestEntry( diff --git a/bindings/python/uv.lock b/bindings/python/uv.lock index 614478d338..3c8ceb3bff 100644 --- a/bindings/python/uv.lock +++ b/bindings/python/uv.lock @@ -13,11 +13,11 @@ wheels = [ [[package]] name = "cachetools" -version = "6.2.1" +version = "5.5.2" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/cc/7e/b975b5814bd36faf009faebe22c1072a1fa1168db34d285ef0ba071ad78c/cachetools-6.2.1.tar.gz", hash = "sha256:3f391e4bd8f8bf0931169baf7456cc822705f4e2a31f840d218f445b9a854201", size = 31325, upload-time = "2025-10-12T14:55:30.139Z" } +sdist = { url = "https://files.pythonhosted.org/packages/6c/81/3747dad6b14fa2cf53fcf10548cf5aea6913e96fab41a3c198676f8948a5/cachetools-5.5.2.tar.gz", hash = "sha256:1a661caa9175d26759571b2e19580f9d6393969e5dfca11fdb1f947a23e640d4", size = 28380, upload-time = "2025-02-20T21:01:19.524Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/96/c5/1e741d26306c42e2bf6ab740b2202872727e0f606033c9dd713f8b93f5a8/cachetools-6.2.1-py3-none-any.whl", hash = "sha256:09868944b6dde876dfd44e1d47e18484541eaf12f26f29b7af91b26cc892d701", size = 11280, upload-time = "2025-10-12T14:55:28.382Z" }, + { url = "https://files.pythonhosted.org/packages/72/76/20fa66124dbe6be5cafeb312ece67de6b61dd91a0247d1ea13db4ebb33c2/cachetools-5.5.2-py3-none-any.whl", hash = "sha256:d26a22bcc62eb95c3beabd9f1ee5e820d3d2704fe2967cbe350e20c8ffcd3f0a", size = 10080, upload-time = "2025-02-20T21:01:16.647Z" }, ] [[package]] @@ -185,6 +185,8 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7f/91/ae2eb6b7979e2f9b035a9f612cf70f1bf54aad4e1d125129bef1eae96f19/greenlet-3.2.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c2ca18a03a8cfb5b25bc1cbe20f3d9a4c80d8c3b13ba3df49ac3961af0b1018d", size = 584358, upload-time = "2025-08-07T13:18:23.708Z" }, { url = "https://files.pythonhosted.org/packages/f7/85/433de0c9c0252b22b16d413c9407e6cb3b41df7389afc366ca204dbc1393/greenlet-3.2.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9fe0a28a7b952a21e2c062cd5756d34354117796c6d9215a87f55e38d15402c5", size = 1113550, upload-time = "2025-08-07T13:42:37.467Z" }, { url = "https://files.pythonhosted.org/packages/a1/8d/88f3ebd2bc96bf7747093696f4335a0a8a4c5acfcf1b757717c0d2474ba3/greenlet-3.2.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8854167e06950ca75b898b104b63cc646573aa5fef1353d4508ecdd1ee76254f", size = 1137126, upload-time = "2025-08-07T13:18:20.239Z" }, + { url = "https://files.pythonhosted.org/packages/f1/29/74242b7d72385e29bcc5563fba67dad94943d7cd03552bac320d597f29b2/greenlet-3.2.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f47617f698838ba98f4ff4189aef02e7343952df3a615f847bb575c3feb177a7", size = 1544904, upload-time = "2025-11-04T12:42:04.763Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e2/1572b8eeab0f77df5f6729d6ab6b141e4a84ee8eb9bc8c1e7918f94eda6d/greenlet-3.2.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:af41be48a4f60429d5cad9d22175217805098a9ef7c40bfef44f7669fb9d74d8", size = 1611228, upload-time = "2025-11-04T12:42:08.423Z" }, { url = "https://files.pythonhosted.org/packages/d6/6f/b60b0291d9623c496638c582297ead61f43c4b72eef5e9c926ef4565ec13/greenlet-3.2.4-cp310-cp310-win_amd64.whl", hash = "sha256:73f49b5368b5359d04e18d15828eecc1806033db5233397748f4ca813ff1056c", size = 298654, upload-time = "2025-08-07T13:50:00.469Z" }, { url = "https://files.pythonhosted.org/packages/a4/de/f28ced0a67749cac23fecb02b694f6473f47686dff6afaa211d186e2ef9c/greenlet-3.2.4-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:96378df1de302bc38e99c3a9aa311967b7dc80ced1dcc6f171e99842987882a2", size = 272305, upload-time = "2025-08-07T13:15:41.288Z" }, { url = "https://files.pythonhosted.org/packages/09/16/2c3792cba130000bf2a31c5272999113f4764fd9d874fb257ff588ac779a/greenlet-3.2.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1ee8fae0519a337f2329cb78bd7a8e128ec0f881073d43f023c7b8d4831d5246", size = 632472, upload-time = "2025-08-07T13:42:55.044Z" }, @@ -194,6 +196,8 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1f/8e/abdd3f14d735b2929290a018ecf133c901be4874b858dd1c604b9319f064/greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2523e5246274f54fdadbce8494458a2ebdcdbc7b802318466ac5606d3cded1f8", size = 587684, upload-time = "2025-08-07T13:18:25.164Z" }, { url = "https://files.pythonhosted.org/packages/5d/65/deb2a69c3e5996439b0176f6651e0052542bb6c8f8ec2e3fba97c9768805/greenlet-3.2.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:1987de92fec508535687fb807a5cea1560f6196285a4cde35c100b8cd632cc52", size = 1116647, upload-time = "2025-08-07T13:42:38.655Z" }, { url = "https://files.pythonhosted.org/packages/3f/cc/b07000438a29ac5cfb2194bfc128151d52f333cee74dd7dfe3fb733fc16c/greenlet-3.2.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:55e9c5affaa6775e2c6b67659f3a71684de4c549b3dd9afca3bc773533d284fa", size = 1142073, upload-time = "2025-08-07T13:18:21.737Z" }, + { url = "https://files.pythonhosted.org/packages/67/24/28a5b2fa42d12b3d7e5614145f0bd89714c34c08be6aabe39c14dd52db34/greenlet-3.2.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c9c6de1940a7d828635fbd254d69db79e54619f165ee7ce32fda763a9cb6a58c", size = 1548385, upload-time = "2025-11-04T12:42:11.067Z" }, + { url = "https://files.pythonhosted.org/packages/6a/05/03f2f0bdd0b0ff9a4f7b99333d57b53a7709c27723ec8123056b084e69cd/greenlet-3.2.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:03c5136e7be905045160b1b9fdca93dd6727b180feeafda6818e6496434ed8c5", size = 1613329, upload-time = "2025-11-04T12:42:12.928Z" }, { url = "https://files.pythonhosted.org/packages/d8/0f/30aef242fcab550b0b3520b8e3561156857c94288f0332a79928c31a52cf/greenlet-3.2.4-cp311-cp311-win_amd64.whl", hash = "sha256:9c40adce87eaa9ddb593ccb0fa6a07caf34015a29bf8d344811665b573138db9", size = 299100, upload-time = "2025-08-07T13:44:12.287Z" }, { url = "https://files.pythonhosted.org/packages/44/69/9b804adb5fd0671f367781560eb5eb586c4d495277c93bde4307b9e28068/greenlet-3.2.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3b67ca49f54cede0186854a008109d6ee71f66bd57bb36abd6d0a0267b540cdd", size = 274079, upload-time = "2025-08-07T13:15:45.033Z" }, { url = "https://files.pythonhosted.org/packages/46/e9/d2a80c99f19a153eff70bc451ab78615583b8dac0754cfb942223d2c1a0d/greenlet-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ddf9164e7a5b08e9d22511526865780a576f19ddd00d62f8a665949327fde8bb", size = 640997, upload-time = "2025-08-07T13:42:56.234Z" }, @@ -203,6 +207,8 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/19/0d/6660d55f7373b2ff8152401a83e02084956da23ae58cddbfb0b330978fe9/greenlet-3.2.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b3812d8d0c9579967815af437d96623f45c0f2ae5f04e366de62a12d83a8fb0", size = 607586, upload-time = "2025-08-07T13:18:28.544Z" }, { url = "https://files.pythonhosted.org/packages/8e/1a/c953fdedd22d81ee4629afbb38d2f9d71e37d23caace44775a3a969147d4/greenlet-3.2.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:abbf57b5a870d30c4675928c37278493044d7c14378350b3aa5d484fa65575f0", size = 1123281, upload-time = "2025-08-07T13:42:39.858Z" }, { url = "https://files.pythonhosted.org/packages/3f/c7/12381b18e21aef2c6bd3a636da1088b888b97b7a0362fac2e4de92405f97/greenlet-3.2.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:20fb936b4652b6e307b8f347665e2c615540d4b42b3b4c8a321d8286da7e520f", size = 1151142, upload-time = "2025-08-07T13:18:22.981Z" }, + { url = "https://files.pythonhosted.org/packages/27/45/80935968b53cfd3f33cf99ea5f08227f2646e044568c9b1555b58ffd61c2/greenlet-3.2.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ee7a6ec486883397d70eec05059353b8e83eca9168b9f3f9a361971e77e0bcd0", size = 1564846, upload-time = "2025-11-04T12:42:15.191Z" }, + { url = "https://files.pythonhosted.org/packages/69/02/b7c30e5e04752cb4db6202a3858b149c0710e5453b71a3b2aec5d78a1aab/greenlet-3.2.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:326d234cbf337c9c3def0676412eb7040a35a768efc92504b947b3e9cfc7543d", size = 1633814, upload-time = "2025-11-04T12:42:17.175Z" }, { url = "https://files.pythonhosted.org/packages/e9/08/b0814846b79399e585f974bbeebf5580fbe59e258ea7be64d9dfb253c84f/greenlet-3.2.4-cp312-cp312-win_amd64.whl", hash = "sha256:a7d4e128405eea3814a12cc2605e0e6aedb4035bf32697f72deca74de4105e02", size = 299899, upload-time = "2025-08-07T13:38:53.448Z" }, ] @@ -345,31 +351,31 @@ wheels = [ [[package]] name = "pyarrow" -version = "22.0.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/30/53/04a7fdc63e6056116c9ddc8b43bc28c12cdd181b85cbeadb79278475f3ae/pyarrow-22.0.0.tar.gz", hash = "sha256:3d600dc583260d845c7d8a6db540339dd883081925da2bd1c5cb808f720b3cd9", size = 1151151, upload-time = "2025-10-24T12:30:00.762Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d9/9b/cb3f7e0a345353def531ca879053e9ef6b9f38ed91aebcf68b09ba54dec0/pyarrow-22.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:77718810bd3066158db1e95a63c160ad7ce08c6b0710bc656055033e39cdad88", size = 34223968, upload-time = "2025-10-24T10:03:31.21Z" }, - { url = "https://files.pythonhosted.org/packages/6c/41/3184b8192a120306270c5307f105b70320fdaa592c99843c5ef78aaefdcf/pyarrow-22.0.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:44d2d26cda26d18f7af7db71453b7b783788322d756e81730acb98f24eb90ace", size = 35942085, upload-time = "2025-10-24T10:03:38.146Z" }, - { url = "https://files.pythonhosted.org/packages/d9/3d/a1eab2f6f08001f9fb714b8ed5cfb045e2fe3e3e3c0c221f2c9ed1e6d67d/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b9d71701ce97c95480fecb0039ec5bb889e75f110da72005743451339262f4ce", size = 44964613, upload-time = "2025-10-24T10:03:46.516Z" }, - { url = "https://files.pythonhosted.org/packages/46/46/a1d9c24baf21cfd9ce994ac820a24608decf2710521b29223d4334985127/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:710624ab925dc2b05a6229d47f6f0dac1c1155e6ed559be7109f684eba048a48", size = 47627059, upload-time = "2025-10-24T10:03:55.353Z" }, - { url = "https://files.pythonhosted.org/packages/3a/4c/f711acb13075c1391fd54bc17e078587672c575f8de2a6e62509af026dcf/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f963ba8c3b0199f9d6b794c90ec77545e05eadc83973897a4523c9e8d84e9340", size = 47947043, upload-time = "2025-10-24T10:04:05.408Z" }, - { url = "https://files.pythonhosted.org/packages/4e/70/1f3180dd7c2eab35c2aca2b29ace6c519f827dcd4cfeb8e0dca41612cf7a/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:bd0d42297ace400d8febe55f13fdf46e86754842b860c978dfec16f081e5c653", size = 50206505, upload-time = "2025-10-24T10:04:15.786Z" }, - { url = "https://files.pythonhosted.org/packages/80/07/fea6578112c8c60ffde55883a571e4c4c6bc7049f119d6b09333b5cc6f73/pyarrow-22.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:00626d9dc0f5ef3a75fe63fd68b9c7c8302d2b5bbc7f74ecaedba83447a24f84", size = 28101641, upload-time = "2025-10-24T10:04:22.57Z" }, - { url = "https://files.pythonhosted.org/packages/2e/b7/18f611a8cdc43417f9394a3ccd3eace2f32183c08b9eddc3d17681819f37/pyarrow-22.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:3e294c5eadfb93d78b0763e859a0c16d4051fc1c5231ae8956d61cb0b5666f5a", size = 34272022, upload-time = "2025-10-24T10:04:28.973Z" }, - { url = "https://files.pythonhosted.org/packages/26/5c/f259e2526c67eb4b9e511741b19870a02363a47a35edbebc55c3178db22d/pyarrow-22.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:69763ab2445f632d90b504a815a2a033f74332997052b721002298ed6de40f2e", size = 35995834, upload-time = "2025-10-24T10:04:35.467Z" }, - { url = "https://files.pythonhosted.org/packages/50/8d/281f0f9b9376d4b7f146913b26fac0aa2829cd1ee7e997f53a27411bbb92/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:b41f37cabfe2463232684de44bad753d6be08a7a072f6a83447eeaf0e4d2a215", size = 45030348, upload-time = "2025-10-24T10:04:43.366Z" }, - { url = "https://files.pythonhosted.org/packages/f5/e5/53c0a1c428f0976bf22f513d79c73000926cb00b9c138d8e02daf2102e18/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:35ad0f0378c9359b3f297299c3309778bb03b8612f987399a0333a560b43862d", size = 47699480, upload-time = "2025-10-24T10:04:51.486Z" }, - { url = "https://files.pythonhosted.org/packages/95/e1/9dbe4c465c3365959d183e6345d0a8d1dc5b02ca3f8db4760b3bc834cf25/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8382ad21458075c2e66a82a29d650f963ce51c7708c7c0ff313a8c206c4fd5e8", size = 48011148, upload-time = "2025-10-24T10:04:59.585Z" }, - { url = "https://files.pythonhosted.org/packages/c5/b4/7caf5d21930061444c3cf4fa7535c82faf5263e22ce43af7c2759ceb5b8b/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1a812a5b727bc09c3d7ea072c4eebf657c2f7066155506ba31ebf4792f88f016", size = 50276964, upload-time = "2025-10-24T10:05:08.175Z" }, - { url = "https://files.pythonhosted.org/packages/ae/f3/cec89bd99fa3abf826f14d4e53d3d11340ce6f6af4d14bdcd54cd83b6576/pyarrow-22.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:ec5d40dd494882704fb876c16fa7261a69791e784ae34e6b5992e977bd2e238c", size = 28106517, upload-time = "2025-10-24T10:05:14.314Z" }, - { url = "https://files.pythonhosted.org/packages/af/63/ba23862d69652f85b615ca14ad14f3bcfc5bf1b99ef3f0cd04ff93fdad5a/pyarrow-22.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:bea79263d55c24a32b0d79c00a1c58bb2ee5f0757ed95656b01c0fb310c5af3d", size = 34211578, upload-time = "2025-10-24T10:05:21.583Z" }, - { url = "https://files.pythonhosted.org/packages/b1/d0/f9ad86fe809efd2bcc8be32032fa72e8b0d112b01ae56a053006376c5930/pyarrow-22.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:12fe549c9b10ac98c91cf791d2945e878875d95508e1a5d14091a7aaa66d9cf8", size = 35989906, upload-time = "2025-10-24T10:05:29.485Z" }, - { url = "https://files.pythonhosted.org/packages/b4/a8/f910afcb14630e64d673f15904ec27dd31f1e009b77033c365c84e8c1e1d/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:334f900ff08ce0423407af97e6c26ad5d4e3b0763645559ece6fbf3747d6a8f5", size = 45021677, upload-time = "2025-10-24T10:05:38.274Z" }, - { url = "https://files.pythonhosted.org/packages/13/95/aec81f781c75cd10554dc17a25849c720d54feafb6f7847690478dcf5ef8/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:c6c791b09c57ed76a18b03f2631753a4960eefbbca80f846da8baefc6491fcfe", size = 47726315, upload-time = "2025-10-24T10:05:47.314Z" }, - { url = "https://files.pythonhosted.org/packages/bb/d4/74ac9f7a54cfde12ee42734ea25d5a3c9a45db78f9def949307a92720d37/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c3200cb41cdbc65156e5f8c908d739b0dfed57e890329413da2748d1a2cd1a4e", size = 47990906, upload-time = "2025-10-24T10:05:58.254Z" }, - { url = "https://files.pythonhosted.org/packages/2e/71/fedf2499bf7a95062eafc989ace56572f3343432570e1c54e6599d5b88da/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ac93252226cf288753d8b46280f4edf3433bf9508b6977f8dd8526b521a1bbb9", size = 50306783, upload-time = "2025-10-24T10:06:08.08Z" }, - { url = "https://files.pythonhosted.org/packages/68/ed/b202abd5a5b78f519722f3d29063dda03c114711093c1995a33b8e2e0f4b/pyarrow-22.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:44729980b6c50a5f2bfcc2668d36c569ce17f8b17bccaf470c4313dcbbf13c9d", size = 27972883, upload-time = "2025-10-24T10:06:14.204Z" }, +version = "19.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7f/09/a9046344212690f0632b9c709f9bf18506522feb333c894d0de81d62341a/pyarrow-19.0.1.tar.gz", hash = "sha256:3bf266b485df66a400f282ac0b6d1b500b9d2ae73314a153dbe97d6d5cc8a99e", size = 1129437, upload-time = "2025-02-18T18:55:57.027Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/36/01/b23b514d86b839956238d3f8ef206fd2728eee87ff1b8ce150a5678d9721/pyarrow-19.0.1-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:fc28912a2dc924dddc2087679cc8b7263accc71b9ff025a1362b004711661a69", size = 30688914, upload-time = "2025-02-18T18:51:37.575Z" }, + { url = "https://files.pythonhosted.org/packages/c6/68/218ff7cf4a0652a933e5f2ed11274f724dd43b9813cb18dd72c0a35226a2/pyarrow-19.0.1-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:fca15aabbe9b8355800d923cc2e82c8ef514af321e18b437c3d782aa884eaeec", size = 32102866, upload-time = "2025-02-18T18:51:44.358Z" }, + { url = "https://files.pythonhosted.org/packages/98/01/c295050d183014f4a2eb796d7d2bbfa04b6cccde7258bb68aacf6f18779b/pyarrow-19.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad76aef7f5f7e4a757fddcdcf010a8290958f09e3470ea458c80d26f4316ae89", size = 41147682, upload-time = "2025-02-18T18:51:49.481Z" }, + { url = "https://files.pythonhosted.org/packages/40/17/a6c3db0b5f3678f33bbb552d2acbc16def67f89a72955b67b0109af23eb0/pyarrow-19.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d03c9d6f2a3dffbd62671ca070f13fc527bb1867b4ec2b98c7eeed381d4f389a", size = 42179192, upload-time = "2025-02-18T18:51:56.265Z" }, + { url = "https://files.pythonhosted.org/packages/cf/75/c7c8e599300d8cebb6cb339014800e1c720c9db2a3fcb66aa64ec84bac72/pyarrow-19.0.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:65cf9feebab489b19cdfcfe4aa82f62147218558d8d3f0fc1e9dea0ab8e7905a", size = 40517272, upload-time = "2025-02-18T18:52:02.969Z" }, + { url = "https://files.pythonhosted.org/packages/ef/c9/68ab123ee1528699c4d5055f645ecd1dd68ff93e4699527249d02f55afeb/pyarrow-19.0.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:41f9706fbe505e0abc10e84bf3a906a1338905cbbcf1177b71486b03e6ea6608", size = 42069036, upload-time = "2025-02-18T18:52:10.173Z" }, + { url = "https://files.pythonhosted.org/packages/54/e3/d5cfd7654084e6c0d9c3ce949e5d9e0ccad569ae1e2d5a68a3ec03b2be89/pyarrow-19.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:c6cb2335a411b713fdf1e82a752162f72d4a7b5dbc588e32aa18383318b05866", size = 25277951, upload-time = "2025-02-18T18:52:15.459Z" }, + { url = "https://files.pythonhosted.org/packages/a0/55/f1a8d838ec07fe3ca53edbe76f782df7b9aafd4417080eebf0b42aab0c52/pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:cc55d71898ea30dc95900297d191377caba257612f384207fe9f8293b5850f90", size = 30713987, upload-time = "2025-02-18T18:52:20.463Z" }, + { url = "https://files.pythonhosted.org/packages/13/12/428861540bb54c98a140ae858a11f71d041ef9e501e6b7eb965ca7909505/pyarrow-19.0.1-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:7a544ec12de66769612b2d6988c36adc96fb9767ecc8ee0a4d270b10b1c51e00", size = 32135613, upload-time = "2025-02-18T18:52:25.29Z" }, + { url = "https://files.pythonhosted.org/packages/2f/8a/23d7cc5ae2066c6c736bce1db8ea7bc9ac3ef97ac7e1c1667706c764d2d9/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0148bb4fc158bfbc3d6dfe5001d93ebeed253793fff4435167f6ce1dc4bddeae", size = 41149147, upload-time = "2025-02-18T18:52:30.975Z" }, + { url = "https://files.pythonhosted.org/packages/a2/7a/845d151bb81a892dfb368bf11db584cf8b216963ccce40a5cf50a2492a18/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f24faab6ed18f216a37870d8c5623f9c044566d75ec586ef884e13a02a9d62c5", size = 42178045, upload-time = "2025-02-18T18:52:36.859Z" }, + { url = "https://files.pythonhosted.org/packages/a7/31/e7282d79a70816132cf6cae7e378adfccce9ae10352d21c2fecf9d9756dd/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:4982f8e2b7afd6dae8608d70ba5bd91699077323f812a0448d8b7abdff6cb5d3", size = 40532998, upload-time = "2025-02-18T18:52:42.578Z" }, + { url = "https://files.pythonhosted.org/packages/b8/82/20f3c290d6e705e2ee9c1fa1d5a0869365ee477e1788073d8b548da8b64c/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:49a3aecb62c1be1d822f8bf629226d4a96418228a42f5b40835c1f10d42e4db6", size = 42084055, upload-time = "2025-02-18T18:52:48.749Z" }, + { url = "https://files.pythonhosted.org/packages/ff/77/e62aebd343238863f2c9f080ad2ef6ace25c919c6ab383436b5b81cbeef7/pyarrow-19.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:008a4009efdb4ea3d2e18f05cd31f9d43c388aad29c636112c2966605ba33466", size = 25283133, upload-time = "2025-02-18T18:52:54.549Z" }, + { url = "https://files.pythonhosted.org/packages/78/b4/94e828704b050e723f67d67c3535cf7076c7432cd4cf046e4bb3b96a9c9d/pyarrow-19.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:80b2ad2b193e7d19e81008a96e313fbd53157945c7be9ac65f44f8937a55427b", size = 30670749, upload-time = "2025-02-18T18:53:00.062Z" }, + { url = "https://files.pythonhosted.org/packages/7e/3b/4692965e04bb1df55e2c314c4296f1eb12b4f3052d4cf43d29e076aedf66/pyarrow-19.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee8dec072569f43835932a3b10c55973593abc00936c202707a4ad06af7cb294", size = 32128007, upload-time = "2025-02-18T18:53:06.581Z" }, + { url = "https://files.pythonhosted.org/packages/22/f7/2239af706252c6582a5635c35caa17cb4d401cd74a87821ef702e3888957/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d5d1ec7ec5324b98887bdc006f4d2ce534e10e60f7ad995e7875ffa0ff9cb14", size = 41144566, upload-time = "2025-02-18T18:53:11.958Z" }, + { url = "https://files.pythonhosted.org/packages/fb/e3/c9661b2b2849cfefddd9fd65b64e093594b231b472de08ff658f76c732b2/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3ad4c0eb4e2a9aeb990af6c09e6fa0b195c8c0e7b272ecc8d4d2b6574809d34", size = 42202991, upload-time = "2025-02-18T18:53:17.678Z" }, + { url = "https://files.pythonhosted.org/packages/fe/4f/a2c0ed309167ef436674782dfee4a124570ba64299c551e38d3fdaf0a17b/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d383591f3dcbe545f6cc62daaef9c7cdfe0dff0fb9e1c8121101cabe9098cfa6", size = 40507986, upload-time = "2025-02-18T18:53:26.263Z" }, + { url = "https://files.pythonhosted.org/packages/27/2e/29bb28a7102a6f71026a9d70d1d61df926887e36ec797f2e6acfd2dd3867/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b4c4156a625f1e35d6c0b2132635a237708944eb41df5fbe7d50f20d20c17832", size = 42087026, upload-time = "2025-02-18T18:53:33.063Z" }, + { url = "https://files.pythonhosted.org/packages/16/33/2a67c0f783251106aeeee516f4806161e7b481f7d744d0d643d2f30230a5/pyarrow-19.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:5bd1618ae5e5476b7654c7b55a6364ae87686d4724538c24185bbb2952679960", size = 25250108, upload-time = "2025-02-18T18:53:38.462Z" }, ] [[package]] @@ -468,8 +474,8 @@ wheels = [ [[package]] name = "pyiceberg" -version = "0.10.0" -source = { git = "https://github.com/apache/iceberg-python.git?rev=d3eb149fe99b14e714a06e727b69cd617c6c052d#d3eb149fe99b14e714a06e727b69cd617c6c052d" } +version = "0.9.1" +source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cachetools" }, { name = "click" }, @@ -477,18 +483,38 @@ dependencies = [ { name = "mmh3" }, { name = "pydantic" }, { name = "pyparsing" }, - { name = "pyroaring" }, { name = "requests" }, { name = "rich" }, { name = "sortedcontainers" }, { name = "strictyaml" }, { name = "tenacity" }, ] +sdist = { url = "https://files.pythonhosted.org/packages/bd/6a/6c1ac381ff0b8e03a9abc2f05722f6002d7452a2c05118697b3f3910e171/pyiceberg-0.9.1.tar.gz", hash = "sha256:3634134ce33859a441768b39df179b2c6f3de2bbbf506622884f553b013ee799", size = 617629, upload-time = "2025-04-30T14:59:34.306Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ea/5d/bb10c86b85895d4ba471b8a0e187031d4aaa82592a639242b83dd9354861/pyiceberg-0.9.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0a183d9217eb82159c01b23c683057f96c8b2375f592b921721d1c157895e2df", size = 527097, upload-time = "2025-04-30T14:58:52.39Z" }, + { url = "https://files.pythonhosted.org/packages/ec/b9/1d6f0d334bc51cd64a58b7320d521e54af3810a6bd748fe2e89db1ad8d5f/pyiceberg-0.9.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:57030bb15c397b0379242907c5611f5b4338fb799e972353fd0edafde6cfd2ef", size = 523267, upload-time = "2025-04-30T14:58:53.978Z" }, + { url = "https://files.pythonhosted.org/packages/02/f5/bd43a9c1d2cd3aeb987cbf2b7f25e2b10306fa81522ea00df250fb23cc84/pyiceberg-0.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3ba4cd9a8f6a04cfbc68e0c83f2db3ffd14244da8601a142cc05965d4b343645", size = 838616, upload-time = "2025-04-30T14:58:55.252Z" }, + { url = "https://files.pythonhosted.org/packages/d0/01/c68f9e03413dc983ddadc2c471038af2ff792449fc451731f58a958a7696/pyiceberg-0.9.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:d5a48c6a2016d0dcde8c9079cc5e6b6d2e2ac663eddfe4697e7ea03a0edc40b7", size = 838290, upload-time = "2025-04-30T14:58:56.412Z" }, + { url = "https://files.pythonhosted.org/packages/ab/80/b7cba54a33b8b7be3655ff656d6bb8594fec0316eec5cafa231ec7f6ff74/pyiceberg-0.9.1-cp310-cp310-win_amd64.whl", hash = "sha256:8bebfa5a804a95a9f3d98d88cbeb37430b09add04592238bba2a2b2e0466d60d", size = 523612, upload-time = "2025-04-30T14:58:59.507Z" }, + { url = "https://files.pythonhosted.org/packages/f6/75/c8b4ebba7d345b5e736ebf4976121b97dd7091dcad401a17ca57152704c5/pyiceberg-0.9.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0e75c502dd56ac3d77036ce8a3b2566348da5ff4367c7c671981616ef6dcc883", size = 566274, upload-time = "2025-04-30T14:59:00.626Z" }, + { url = "https://files.pythonhosted.org/packages/e0/a0/9494c7930e5e4dc951d95abba584d8ffdb7403368398796ede21ff25c26f/pyiceberg-0.9.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0a8189c9b3ba81dd12493d6bb874a656a4d4909904552b97a629d1d43b3a0e90", size = 560157, upload-time = "2025-04-30T14:59:02.082Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d4/351776b1ae83de187d7cf37b100f4f124c7a71e35337182d3aef308156d1/pyiceberg-0.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7c03065d5c5b704444ab8fb18cdd232ec43994db95b9e53444008ebc2cf9dc2c", size = 1052290, upload-time = "2025-04-30T14:59:03.232Z" }, + { url = "https://files.pythonhosted.org/packages/40/17/d8fea681afb52f20bf6a640f9044dcf621a47165f67cc5320bf3c6e82e4e/pyiceberg-0.9.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:93f2586a5da737de6e4643bf096a01772f068d1eedb7ffde6b36c60b6b9e6bd3", size = 1047503, upload-time = "2025-04-30T14:59:04.38Z" }, + { url = "https://files.pythonhosted.org/packages/d0/e0/d173fc2aa8dc252d7aac71703ba2c0491e4988b3a160cf5abb531cfb9086/pyiceberg-0.9.1-cp311-cp311-win_amd64.whl", hash = "sha256:94e45c10051110ba7a43b85a1f0a680b4a31d1d6cee593c8e62e14d22d18c47d", size = 559491, upload-time = "2025-04-30T14:59:05.615Z" }, + { url = "https://files.pythonhosted.org/packages/52/26/77983c2884b4a5f13f8a35e5c5e762ae699f6c511efd16730ab883000c1b/pyiceberg-0.9.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b8a958e3bbe919026533cee1f0fb6b7040928fce8d42c2ecea228de7c17578fa", size = 605755, upload-time = "2025-04-30T14:59:07.087Z" }, + { url = "https://files.pythonhosted.org/packages/6d/67/e6ea7fcc43aebc85aea5a67a69d01c9015283478061c3121b6b8aa158ce4/pyiceberg-0.9.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b7e956b35c6822600c45fd8f3ea8cfea328cc406fefa534afeb6fdb325d05406", size = 597325, upload-time = "2025-04-30T14:59:08.644Z" }, + { url = "https://files.pythonhosted.org/packages/7f/cf/178a9f63fac1bfdd13bc85169e7ab903955d082e2cd80507b1921a6f64dc/pyiceberg-0.9.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1e4e585164d7d86f5c9a609a1bc2abeae2f0ea0680a11a2064d3a945866b5311", size = 1277399, upload-time = "2025-04-30T14:59:10.193Z" }, + { url = "https://files.pythonhosted.org/packages/d1/6b/78d1739eb1d5b18529ee438aed75dac3e0b246f5e4d800931f9d1e37cda2/pyiceberg-0.9.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5fee08dac30e8524526f7d18468f9670f8606905b850b261314c597c6633f3b4", size = 1269083, upload-time = "2025-04-30T14:59:11.964Z" }, + { url = "https://files.pythonhosted.org/packages/67/69/c0087d19c8d8e8530acee3ba485d54aedeebf2963784a16692ca4b439566/pyiceberg-0.9.1-cp312-cp312-win_amd64.whl", hash = "sha256:124793c54a0c2fb5ac4ab19c38da116c068e277c85cbaa7e4064e635a70b595e", size = 595512, upload-time = "2025-04-30T14:59:14.464Z" }, + { url = "https://files.pythonhosted.org/packages/aa/62/0153ed3a39d6f4b3235d430123703d4684eec7ba780404bbc118ace7406a/pyiceberg-0.9.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:77aec1c77d675603e0c5358e74adcae8d13b323753d702011be3f309d26af355", size = 668261, upload-time = "2025-04-30T14:59:21.751Z" }, + { url = "https://files.pythonhosted.org/packages/24/bd/c4cec142686dd8124032c69b6b02ba3703abc114ce787d0f02088b1f43d8/pyiceberg-0.9.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:cf567438bf6267bbb67fdfdfc72ac500d523725fca9a6a38f93e8acd4146190e", size = 657439, upload-time = "2025-04-30T14:59:23.304Z" }, + { url = "https://files.pythonhosted.org/packages/ae/74/bbfc70bb1857f9d55d06fee1330a0236876b8ae4aa6fc5d815e2c4fef4f7/pyiceberg-0.9.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5992db7c00d789a33ff117700d453126803e769507a5edeb79bb6510ff72fc00", size = 1352983, upload-time = "2025-04-30T14:59:25.023Z" }, + { url = "https://files.pythonhosted.org/packages/90/20/e33e1716d1368b2471b80d9f1e338110f1e781b34ebffc5e320523102ffc/pyiceberg-0.9.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:c9e460fca26162a3822c0e8d50b49c80928a0e35cb41698748d7a26f8c016215", size = 657563, upload-time = "2025-04-30T14:59:27.004Z" }, +] [package.optional-dependencies] pyarrow = [ { name = "pyarrow" }, - { name = "pyiceberg-core" }, ] sql-sqlite = [ { name = "sqlalchemy" }, @@ -514,7 +540,7 @@ dev = [ { name = "datafusion", specifier = "==50.*" }, { name = "fastavro", specifier = ">=1.11.1" }, { name = "maturin", specifier = ">=1.0,<2.0" }, - { name = "pyiceberg", extras = ["sql-sqlite", "pyarrow"], git = "https://github.com/apache/iceberg-python.git?rev=d3eb149fe99b14e714a06e727b69cd617c6c052d" }, + { name = "pyiceberg", extras = ["sql-sqlite", "pyarrow"] }, { name = "pytest", specifier = ">=8.3.2" }, ] @@ -527,56 +553,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/5e/1aa9a93198c6b64513c9d7752de7422c06402de6600a8767da1524f9570b/pyparsing-3.2.5-py3-none-any.whl", hash = "sha256:e38a4f02064cf41fe6593d328d0512495ad1f3d8a91c4f73fc401b3079a59a5e", size = 113890, upload-time = "2025-09-21T04:11:04.117Z" }, ] -[[package]] -name = "pyroaring" -version = "1.0.3" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/0f/e4/975f0fa77fc3590820b4a3ac49704644b389795409bc12eb91729f845812/pyroaring-1.0.3.tar.gz", hash = "sha256:cd7392d1c010c9e41c11c62cd0610c8852e7e9698b1f7f6c2fcdefe50e7ef6da", size = 188688, upload-time = "2025-10-09T09:08:22.448Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/3f/87/f19328d35f29b1d634f8f2127941927b37c6b68890912a4668e1a272d54d/pyroaring-1.0.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:c10e4cfbe203a578c78808406af491e3615d5e46cf69a7709050243346cd68bc", size = 670434, upload-time = "2025-10-09T09:06:34.89Z" }, - { url = "https://files.pythonhosted.org/packages/2f/fa/d933448844925728990e2256bbfa4f18cc3956d43d548270317128038015/pyroaring-1.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:cc329c62e504f2531c4008240f31736bcd2dee4339071f1eac0648068e6d17fa", size = 367332, upload-time = "2025-10-09T09:06:36.549Z" }, - { url = "https://files.pythonhosted.org/packages/39/e4/ab74296c3aac8f3ceb800354c2f811de2c4c518b3ca3a7a28e0599740cff/pyroaring-1.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8c7fb6ddf6ef31148f0939bc5c26b681d63df301ee1e372525012dd7bfe4a30a", size = 311443, upload-time = "2025-10-09T09:06:37.617Z" }, - { url = "https://files.pythonhosted.org/packages/6b/71/13ff623f3bba340ea7cc841883d7a0eaba1bec7e2d4e0d6759b89a4ce754/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cd18446832ea04a7d33bd6b78270b0be14eabcda5937af3428d6cb3d2bf98e54", size = 1853845, upload-time = "2025-10-09T09:06:38.706Z" }, - { url = "https://files.pythonhosted.org/packages/2b/05/40c0b37d78b16842c924b87b4fa491f4b20ed0e40c6255c872df45314247/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f0cbc766df2a24e28f23d69b66bbec64e691799219fd82c2f2236f03fc88e2e", size = 2045170, upload-time = "2025-10-09T09:06:40.229Z" }, - { url = "https://files.pythonhosted.org/packages/78/c6/6560c61d2f5c30fbb8f7b9a1c7d02068e78a7e8cd336eb294ec70896a80a/pyroaring-1.0.3-cp310-cp310-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:96a51e96f8f473381615f0f852f7238ad0a47f28e4a35e9f082468c5cfe4e9c3", size = 1791073, upload-time = "2025-10-09T09:06:42.092Z" }, - { url = "https://files.pythonhosted.org/packages/33/9e/0c91d4dbc4ec7bea9dcd3c203cfb8d96ed9df3c46981c0b22e9f17e98296/pyroaring-1.0.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:654af38b1f1c9bdc27b4f6d331fc5d91599df96e72a6df1886f4d95eea60ab29", size = 1789048, upload-time = "2025-10-09T09:06:43.392Z" }, - { url = "https://files.pythonhosted.org/packages/e9/48/51af418321cda2a1cfa64a48397ea0b73da74afe5c53a862525476d8a42c/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6721036afa31c07bdcbb4fcafa166660cf9c2eac695dcd495f8778549fa55899", size = 2838811, upload-time = "2025-10-09T09:06:44.588Z" }, - { url = "https://files.pythonhosted.org/packages/b2/69/7bc7070b35f72706e3870b5856e73656b9065bedae90268da5d77be00b15/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:0caa10f20329d09233fac6550b2adce4d9f173f748a9a9a5ea3b7033827dfe2d", size = 2640299, upload-time = "2025-10-09T09:06:46.159Z" }, - { url = "https://files.pythonhosted.org/packages/d0/87/5f2f590973d454e79ee8729aca888f9bb2d6018f7c9816bf66000cbc5e88/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:f109be8af937e85c52cb920d3fd120db52b172f59460852d2e3d2e3d13a4f52a", size = 2965427, upload-time = "2025-10-09T09:06:47.558Z" }, - { url = "https://files.pythonhosted.org/packages/e4/c8/1b425503141681db94941d9d3f41333bbd61975c3fc62d95122c372da85a/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:ddc80bfcd313c7c524a2742d263e73cae088b6a611b77dcc46fa90c306f6dace", size = 3084447, upload-time = "2025-10-09T09:06:48.727Z" }, - { url = "https://files.pythonhosted.org/packages/51/b9/2ac712ea90bd1e0d7e49e5e26c0c5aad1d77f21cf520a7edf46a477f5217/pyroaring-1.0.3-cp310-cp310-win32.whl", hash = "sha256:5a183f5ec069757fe5b60e37f7c6fa8a53178eacf0d76601b739e2890edee036", size = 204956, upload-time = "2025-10-09T09:06:49.801Z" }, - { url = "https://files.pythonhosted.org/packages/6d/b9/1d4859c74d05f72b86dc0b308e6221e814a178459301cea9bcd084b4a92b/pyroaring-1.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:051bd9a66ce855a1143faa2b879ea6c6ca2905209e172ce9eedf79834897c730", size = 253778, upload-time = "2025-10-09T09:06:50.68Z" }, - { url = "https://files.pythonhosted.org/packages/64/20/b421100bd14b6a1074945af1418671630e1b8c4996ef000ac4e363785ead/pyroaring-1.0.3-cp310-cp310-win_arm64.whl", hash = "sha256:3043ff5c85375310ca3cd3e01944e03026e0ec07885e52dfabcfcd9dc303867f", size = 219330, upload-time = "2025-10-09T09:06:52.011Z" }, - { url = "https://files.pythonhosted.org/packages/39/ed/5e555dd99b12318ea1c7666b773fc4f097aeb609eeb1c1b3da519d445f71/pyroaring-1.0.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:755cdac1f9a1b7b5c621e570d4f6dbcf3b8e4a1e35a66f976104ecb35dce4ed2", size = 675916, upload-time = "2025-10-09T09:06:53.174Z" }, - { url = "https://files.pythonhosted.org/packages/da/06/dd8a9a87b90c4560f8384ab1dbafcd40c2a16f6777a07334a8e341bd7383/pyroaring-1.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ebab073db620f26f0ba11e13fa2f35e3b1298209fba47b6bc8cb6f0e2c9627f9", size = 369743, upload-time = "2025-10-09T09:06:54.421Z" }, - { url = "https://files.pythonhosted.org/packages/35/aa/da882011045ddacffe818a4fcbdd7e609a15f9c83d536222ec5b17af4aa9/pyroaring-1.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:684fb8dffe19bdb7f91897c65eac6eee23b1e46043c47eb24288f28a1170fe04", size = 313981, upload-time = "2025-10-09T09:06:55.514Z" }, - { url = "https://files.pythonhosted.org/packages/ed/3c/f6534844b02e2505ccdc9aae461c9838ab96f72b5688c045448761735512/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:678d31fc24e82945a1bfb14816c77823983382ffea76985d494782aa2f058427", size = 1923181, upload-time = "2025-10-09T09:06:56.897Z" }, - { url = "https://files.pythonhosted.org/packages/ea/82/9f1a85ba33e3d89b9cdb8183fb2fd2f25720d10742dd8827508ccccc13ae/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7d815f624e0285db3669f673d1725cb754b120ec70d0032d7c7166103a96c96d", size = 2113222, upload-time = "2025-10-09T09:06:58.388Z" }, - { url = "https://files.pythonhosted.org/packages/a7/f8/4d4340971cbc1379f987c847080bcb7f9765a57e122f392c3a3485c9587e/pyroaring-1.0.3-cp311-cp311-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:57fd5b80dacb8e888402b6b7508a734c6a527063e4e24e882ff2e0fd90721ada", size = 1837385, upload-time = "2025-10-09T09:06:59.449Z" }, - { url = "https://files.pythonhosted.org/packages/c6/58/d14cc561685e4c224af26b4fdb4f6c7e643294ac5a4b29f178b5cbb71af1/pyroaring-1.0.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ab26a7a45a0bb46c00394d1a60a9f2d57c220f84586e30d59b39784b0f94aee6", size = 1856170, upload-time = "2025-10-09T09:07:00.608Z" }, - { url = "https://files.pythonhosted.org/packages/d1/d2/d2d9790c373f6438d4d0958bc4c79f3dc77826d8553743ff3f64acdc9ab3/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9232f3f606315d59049c128154100fd05008d5c5c211e48b21848cd41ee64d26", size = 2909282, upload-time = "2025-10-09T09:07:02.124Z" }, - { url = "https://files.pythonhosted.org/packages/bc/28/4b2277982302b5b406998064ca1eaef1a79e4ea87185f511e33e7a7e3511/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f34b44b3ec3df97b978799f2901fefb2a48d367496fd1cde3cc5fe8b3bc13510", size = 2701034, upload-time = "2025-10-09T09:07:03.403Z" }, - { url = "https://files.pythonhosted.org/packages/d2/91/b2340193825fa2431cf735f0ecb23206fb31f386fecca38336935a294513/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:25a83ec6bac3106568bd3fdd316f0fee52aa0be8c72da565ad02b10ae7905924", size = 3028962, upload-time = "2025-10-09T09:07:05.558Z" }, - { url = "https://files.pythonhosted.org/packages/07/ea/ad79073cc5d8dcca35d1a955bb886d96905e9dacc58d1971fda012a5ad18/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c17d4ec53b5b6b333d9a9515051213a691293ada785dc8c025d3641482597ed3", size = 3152109, upload-time = "2025-10-09T09:07:06.887Z" }, - { url = "https://files.pythonhosted.org/packages/9a/de/f55a1093acb16d25ff9811546823e59078e4a3e56d2eb0ff5d10f696933d/pyroaring-1.0.3-cp311-cp311-win32.whl", hash = "sha256:d54024459ace600f1d1ffbc6dc3c60eb47cca3b678701f06148f59e10f6f8d7b", size = 204246, upload-time = "2025-10-09T09:07:08.036Z" }, - { url = "https://files.pythonhosted.org/packages/c6/e5/36bf3039733b8e00732892c9334b2f5309f38e72af0b3b40b8729b5857a3/pyroaring-1.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:c28750148ef579a7447a8cb60b39e5943e03f8c29bce8f2788728f6f23d1887a", size = 254637, upload-time = "2025-10-09T09:07:09.103Z" }, - { url = "https://files.pythonhosted.org/packages/d6/e8/e2b78e595b5a82a6014af327614756a55f17ec4120a2ab197f1762641316/pyroaring-1.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:535d8deccbd8db2c6bf38629243e9646756905574a742b2a72ff51d6461d616c", size = 219597, upload-time = "2025-10-09T09:07:10.38Z" }, - { url = "https://files.pythonhosted.org/packages/dd/09/a5376d55672e0535019ba1469888909d0046cea0cfb969a4aa1f99caaf22/pyroaring-1.0.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:add3e4c78eb590a76526ecce8d1566eecdd5822e351c36b3697997f4a80ed808", size = 681056, upload-time = "2025-10-09T09:07:11.497Z" }, - { url = "https://files.pythonhosted.org/packages/23/dd/78f59d361bd9ebf8de3660408b0c48664ade0a057ebcf4b207d99ac1a698/pyroaring-1.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ebaffe846cf4ba4f00ce6b8a9f39613f24e2d09447e77be4fa6e898bc36451b6", size = 375111, upload-time = "2025-10-09T09:07:12.597Z" }, - { url = "https://files.pythonhosted.org/packages/bf/03/10dc93f83a5453eb40a69c79106a8385b40aa12cf4531ca72bd9d7f45cb2/pyroaring-1.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a9459f27498f97d08031a34a5ead230b77eb0ab3cc3d85b7f54faa2fd548acd6", size = 314319, upload-time = "2025-10-09T09:07:13.579Z" }, - { url = "https://files.pythonhosted.org/packages/86/9e/b00c38a7e62a73e152055f593595c37152e61fc2896fd11538a7c71fbe4e/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2b2eb8bd1c35c772994889be9f7dda09477475d7aa1e2af9ab4ef18619326f6", size = 1869251, upload-time = "2025-10-09T09:07:14.584Z" }, - { url = "https://files.pythonhosted.org/packages/4f/33/f32d00ca105b66303deab43d027c3574c8ade8525dac0e5b50a9fb4d1b76/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d31f4c1c906f1af14ce61a3959d04a14a64c594f8a768399146a45bbd341f21f", size = 2071551, upload-time = "2025-10-09T09:07:15.713Z" }, - { url = "https://files.pythonhosted.org/packages/5d/89/e953cae181ba4c7523334855a1ca0ae8eeea3cee8d7cd39c56bd99709d3f/pyroaring-1.0.3-cp312-cp312-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53be988fc86698d56c11049bfe5113a2f6990adb1fa2782b29636509808b6aa7", size = 1781071, upload-time = "2025-10-09T09:07:17.19Z" }, - { url = "https://files.pythonhosted.org/packages/fa/db/65d4be532e68b62a84a9c89b24d0a1394f452f484fa29392142d9a3b9c48/pyroaring-1.0.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7df84d223424523b19a23781f4246cc247fd6d821e1bc0853c2f25669136f7d0", size = 1795670, upload-time = "2025-10-09T09:07:18.524Z" }, - { url = "https://files.pythonhosted.org/packages/f5/9e/684ea0568ce7d30fc4e01ad1c666e9ce1a5b1702fa630231f4f6bdb96539/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:34a781f1f9766897f63ef18be129827340ae37764015b83fdcff1efb9e29136d", size = 2849305, upload-time = "2025-10-09T09:07:20.388Z" }, - { url = "https://files.pythonhosted.org/packages/7c/fd/d7773a2adf91f45d8924197954c66b1694325afd2f27e02edaac07338402/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:1f414343b4ed0756734328cdf2a91022fc54503769e3f8d79bd0b672ea815a16", size = 2692843, upload-time = "2025-10-09T09:07:22.042Z" }, - { url = "https://files.pythonhosted.org/packages/13/72/b8a99ba138eebd8ff9bf8d15f3942e9e43e8e45723e2e6b7b09e542b7448/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:d16ae185c72dc64f76335dbe53e53a892e78115adc92194957d1b7ef74d230b9", size = 2983440, upload-time = "2025-10-09T09:07:23.419Z" }, - { url = "https://files.pythonhosted.org/packages/ca/94/e6ed1f682d850e039c71b2032bacdefc5082dc809796cf34b9e6f24c604d/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f888447bf22dde7759108bfe6dfbeb6bbb61b14948de9c4cb6843c4dd57e2215", size = 3117542, upload-time = "2025-10-09T09:07:25.104Z" }, - { url = "https://files.pythonhosted.org/packages/8f/89/d55b0ed3e098ef89c421b43b748afe3d90eb250cab50b9e53e3a3449ac58/pyroaring-1.0.3-cp312-cp312-win32.whl", hash = "sha256:fbbdc44c51a0a3efd7be3dbe04466278ce098fcd101aa1905849319042159770", size = 205118, upload-time = "2025-10-09T09:07:26.532Z" }, - { url = "https://files.pythonhosted.org/packages/c8/e1/b71fef6a73efb50110d33d714235ff7059f4ebae98dc474b6549b322f48f/pyroaring-1.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:3b217c4b3ad953b4c759a0d2f9bd95316f0c345b9f7adb49e6ded7a1f5106bd4", size = 260629, upload-time = "2025-10-09T09:07:27.528Z" }, - { url = "https://files.pythonhosted.org/packages/57/33/66ee872079c9c47512d6e17d374bcad8d91350c24dc20fbe678c34b33745/pyroaring-1.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:e6bcf838564c21bab8fe6c2748b4990d4cd90612d8c470c04889def7bb5114ea", size = 219032, upload-time = "2025-10-09T09:07:28.754Z" }, -] - [[package]] name = "pytest" version = "8.4.2" @@ -624,15 +600,16 @@ wheels = [ [[package]] name = "rich" -version = "14.2.0" +version = "13.9.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "markdown-it-py" }, { name = "pygments" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ab/3a/0316b28d0761c6734d6bc14e770d85506c986c85ffb239e688eeaab2c2bc/rich-13.9.4.tar.gz", hash = "sha256:439594978a49a09530cff7ebc4b5c7103ef57baf48d5ea3184f21d9a2befa098", size = 223149, upload-time = "2024-11-01T16:43:57.873Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" }, + { url = "https://files.pythonhosted.org/packages/19/71/39c7c0d87f8d4e6c020a393182060eaefeeae6c01dab6a84ec346f2567df/rich-13.9.4-py3-none-any.whl", hash = "sha256:6049d5e6ec054bf2779ab3358186963bac2ea89175919d699e378b99738c2a90", size = 242424, upload-time = "2024-11-01T16:43:55.817Z" }, ] [[package]] From 0459206cfc9a71e2d1bb768cb72ae688db492346 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Feb 2026 08:58:20 +0800 Subject: [PATCH 188/891] chore(deps): Bump minijinja from 2.13.0 to 2.15.1 (#2099) Bumps [minijinja](https://github.com/mitsuhiko/minijinja) from 2.13.0 to 2.15.1.
Release notes

Sourced from minijinja's releases.

2.15.1

Release Notes

  • Re-release of 2.15.0 because of a bad release.

Install minijinja-cli 2.15.1

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf
https://github.com/mitsuhiko/minijinja/releases/download/2.15.1/minijinja-cli-installer.sh
| sh

Install prebuilt binaries via powershell script

powershell -ExecutionPolicy Bypass -c "irm
https://github.com/mitsuhiko/minijinja/releases/download/2.15.1/minijinja-cli-installer.ps1
| iex"

Download minijinja-cli 2.15.1

File Platform Checksum
minijinja-cli-aarch64-apple-darwin.tar.xz Apple Silicon macOS checksum
minijinja-cli-x86_64-apple-darwin.tar.xz Intel macOS checksum
minijinja-cli-i686-pc-windows-msvc.zip x86 Windows checksum
minijinja-cli-x86_64-pc-windows-msvc.zip x64 Windows checksum
minijinja-cli-aarch64-unknown-linux-gnu.tar.xz ARM64 Linux checksum
minijinja-cli-i686-unknown-linux-gnu.tar.xz x86 Linux checksum
minijinja-cli-x86_64-unknown-linux-gnu.tar.xz x64 Linux checksum
minijinja-cli-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
minijinja-cli-i686-unknown-linux-musl.tar.xz x86 MUSL Linux checksum
minijinja-cli-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

2.15.0

Release Notes

  • Added py.typed marker for PEP 561 typing support in Python bindings. #853
  • Added optional default argument to map.get() method in pycompat. #852
  • Added a go language port. #854
  • Fixed stability guarantees for the |sort filter when using reverse=true. #856
  • Fixed missing SemiStrict undefined mapping in Python bindings. #859

Install minijinja-cli 2.15.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf
https://github.com/mitsuhiko/minijinja/releases/download/2.15.0/minijinja-cli-installer.sh
| sh

... (truncated)

Changelog

Sourced from minijinja's changelog.

2.15.1

  • Re-release of 2.15.0 because of a bad release.

2.15.0

  • Added py.typed marker for PEP 561 typing support in Python bindings. #853
  • Added optional default argument to map.get() method in pycompat. #852
  • Added a go language port. #854
  • Fixed stability guarantees for the |sort filter when using reverse=true. #856
  • Fixed missing SemiStrict undefined mapping in Python bindings. #859

2.14.0

  • Added support for tuple unpacking in {% set %} statements. #847
Commits
  • 96caba2 Release 2.15.1
  • 00c0d7c Copy LICENSE file instead of symlink for pkg.go.dev
  • 3c35add Update cargo-dist to 0.30.3 for macOS-15 runners
  • 5e6dbd1 Release 2.15.0
  • 0bc0429 Improve some go examples
  • 1841624 Update some deps
  • bbf292e Added stacktrace support go minijinja-go
  • 7c7c241 Correct go link in readme
  • 70784da Added more go examples
  • 98826c4 Ensure to tag go releases
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=minijinja&package-manager=cargo&previous-version=2.13.0&new-version=2.15.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 577229645a..902f7c76ae 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4187,9 +4187,9 @@ dependencies = [ [[package]] name = "minijinja" -version = "2.13.0" +version = "2.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0adbe6e92a6ce0fd6c4aac593fdfd3e3950b0f61b1a63aa9731eb6fd85776fa3" +checksum = "b479616bb6f0779fb0f3964246beda02d4b01144e1b0d5519616e012ccc2a245" dependencies = [ "serde", ] From d8f104a8d94db2493881c13d30ecc7d81d846565 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Feb 2026 09:13:55 +0800 Subject: [PATCH 189/891] chore(deps): Bump crate-ci/typos from 1.42.1 to 1.42.3 (#2098) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.42.1 to 1.42.3.
Release notes

Sourced from crate-ci/typos's releases.

v1.42.3

[1.42.3] - 2026-01-27

Fixes

  • Don't correct loosing

v1.42.2

[1.42.2] - 2026-01-26

Fixes

  • Don't correct substituters
Changelog

Sourced from crate-ci/typos's changelog.

[1.42.3] - 2026-01-27

Fixes

  • Don't correct loosing

[1.42.2] - 2026-01-26

Fixes

  • Don't correct substituters
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.42.1&new-version=1.42.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 60d3304828..dd258fb844 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.42.1 + uses: crate-ci/typos@v1.42.3 From 0c3e5f0ed7e00857d751581f46c787e13f2298cf Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Mon, 2 Feb 2026 16:52:25 -0800 Subject: [PATCH 190/891] feat!(io): Implement Storage for OpenDal (#2080) ## Which issue does this PR close? - Closes #2083 ## What changes are included in this PR? Wired `OpenDal` with `Storage` trait. `FileIO` still uses hardcoded `OpenDal` as of now so no customer facing behavior changes - Implement Storage for OpenDal - Added OpenDalFactory and implement StorageFactory (unused for now) - Updated Input/OutputFile to delegate operations to storage - Updated FileIO to delegate operations to inner storage ## Are these changes tested? No new tests added, rely on existing tests --- crates/iceberg/src/arrow/reader.rs | 12 +- crates/iceberg/src/io/file_io.rs | 131 ++--- crates/iceberg/src/io/mod.rs | 31 +- .../io/{storage_azdls.rs => opendal/azdls.rs} | 8 +- .../src/io/{storage_fs.rs => opendal/fs.rs} | 0 .../src/io/{storage_gcs.rs => opendal/gcs.rs} | 0 .../{storage_memory.rs => opendal/memory.rs} | 0 crates/iceberg/src/io/opendal/mod.rs | 474 ++++++++++++++++++ .../src/io/{storage_oss.rs => opendal/oss.rs} | 0 .../src/io/{storage_s3.rs => opendal/s3.rs} | 0 crates/iceberg/src/io/storage.rs | 223 +------- crates/iceberg/src/puffin/metadata.rs | 11 +- crates/iceberg/src/puffin/reader.rs | 2 +- .../src/writer/file_writer/parquet_writer.rs | 10 +- 14 files changed, 543 insertions(+), 359 deletions(-) rename crates/iceberg/src/io/{storage_azdls.rs => opendal/azdls.rs} (98%) rename crates/iceberg/src/io/{storage_fs.rs => opendal/fs.rs} (100%) rename crates/iceberg/src/io/{storage_gcs.rs => opendal/gcs.rs} (100%) rename crates/iceberg/src/io/{storage_memory.rs => opendal/memory.rs} (100%) create mode 100644 crates/iceberg/src/io/opendal/mod.rs rename crates/iceberg/src/io/{storage_oss.rs => opendal/oss.rs} (100%) rename crates/iceberg/src/io/{storage_s3.rs => opendal/s3.rs} (100%) diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index aa45a12973..5b3a7bb862 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -474,7 +474,7 @@ impl ArrowReader { file_io: FileIO, should_load_page_index: bool, arrow_reader_options: Option, - ) -> Result>> { + ) -> Result> { // Get the metadata for the Parquet file we need to read and build // a reader for the data within let parquet_file = file_io.new_input(data_file_path)?; @@ -1673,18 +1673,18 @@ impl BoundPredicateVisitor for PredicateConverter<'_> { } /// ArrowFileReader is a wrapper around a FileRead that impls parquets AsyncFileReader. -pub struct ArrowFileReader { +pub struct ArrowFileReader { meta: FileMetadata, preload_column_index: bool, preload_offset_index: bool, preload_page_index: bool, metadata_size_hint: Option, - r: R, + r: Box, } -impl ArrowFileReader { +impl ArrowFileReader { /// Create a new ArrowFileReader - pub fn new(meta: FileMetadata, r: R) -> Self { + pub fn new(meta: FileMetadata, r: Box) -> Self { Self { meta, preload_column_index: false, @@ -1723,7 +1723,7 @@ impl ArrowFileReader { } } -impl AsyncFileReader for ArrowFileReader { +impl AsyncFileReader for ArrowFileReader { fn get_bytes(&mut self, range: Range) -> BoxFuture<'_, parquet::errors::Result> { Box::pin( self.r diff --git a/crates/iceberg/src/io/file_io.rs b/crates/iceberg/src/io/file_io.rs index ef9f8aa935..1ad71d8531 100644 --- a/crates/iceberg/src/io/file_io.rs +++ b/crates/iceberg/src/io/file_io.rs @@ -21,10 +21,10 @@ use std::ops::Range; use std::sync::Arc; use bytes::Bytes; -use opendal::Operator; use url::Url; -use super::storage::OpenDalStorage; +use super::opendal::OpenDalStorage; +use super::storage::Storage; use crate::{Error, ErrorKind, Result}; /// FileIO implementation, used to manipulate files in underlying storage. @@ -89,8 +89,7 @@ impl FileIO { /// /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. pub async fn delete(&self, path: impl AsRef) -> Result<()> { - let (op, relative_path) = self.inner.create_operator(&path)?; - Ok(op.delete(relative_path).await?) + self.inner.delete(path.as_ref()).await } /// Remove the path and all nested dirs and files recursively. @@ -104,14 +103,8 @@ impl FileIO { /// - If the path is a file or not exist, this function will be no-op. /// - If the path is a empty directory, this function will remove the directory itself. /// - If the path is a non-empty directory, this function will remove the directory and all nested files and directories. - pub async fn remove_dir_all(&self, path: impl AsRef) -> Result<()> { - let (op, relative_path) = self.inner.create_operator(&path)?; - let path = if relative_path.ends_with('/') { - relative_path.to_string() - } else { - format!("{relative_path}/") - }; - Ok(op.remove_all(&path).await?) + pub async fn delete_prefix(&self, path: impl AsRef) -> Result<()> { + self.inner.delete_prefix(path.as_ref()).await } /// Check file exists. @@ -120,8 +113,7 @@ impl FileIO { /// /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. pub async fn exists(&self, path: impl AsRef) -> Result { - let (op, relative_path) = self.inner.create_operator(&path)?; - Ok(op.exists(relative_path).await?) + self.inner.exists(path.as_ref()).await } /// Creates input file. @@ -130,14 +122,7 @@ impl FileIO { /// /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. pub fn new_input(&self, path: impl AsRef) -> Result { - let (op, relative_path) = self.inner.create_operator(&path)?; - let path = path.as_ref().to_string(); - let relative_path_pos = path.len() - relative_path.len(); - Ok(InputFile { - op, - path, - relative_path_pos, - }) + self.inner.new_input(path.as_ref()) } /// Creates output file. @@ -146,14 +131,7 @@ impl FileIO { /// /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. pub fn new_output(&self, path: impl AsRef) -> Result { - let (op, relative_path) = self.inner.create_operator(&path)?; - let path = path.as_ref().to_string(); - let relative_path_pos = path.len() - relative_path.len(); - Ok(OutputFile { - op, - path, - relative_path_pos, - }) + self.inner.new_output(path.as_ref()) } } @@ -291,24 +269,20 @@ pub trait FileRead: Send + Sync + Unpin + 'static { async fn read(&self, range: Range) -> crate::Result; } -#[async_trait::async_trait] -impl FileRead for opendal::Reader { - async fn read(&self, range: Range) -> crate::Result { - Ok(opendal::Reader::read(self, range).await?.to_bytes()) - } -} - /// Input file is used for reading from files. #[derive(Debug)] pub struct InputFile { - op: Operator, - // Absolution path of file. + storage: Arc, + // Absolute path of file. path: String, - // Relative path of file to uri, starts at [`relative_path_pos`] - relative_path_pos: usize, } impl InputFile { + /// Creates a new input file. + pub fn new(storage: Arc, path: String) -> Self { + Self { storage, path } + } + /// Absolute path to root uri. pub fn location(&self) -> &str { &self.path @@ -316,34 +290,26 @@ impl InputFile { /// Check if file exists. pub async fn exists(&self) -> crate::Result { - Ok(self.op.exists(&self.path[self.relative_path_pos..]).await?) + self.storage.exists(&self.path).await } /// Fetch and returns metadata of file. pub async fn metadata(&self) -> crate::Result { - let meta = self.op.stat(&self.path[self.relative_path_pos..]).await?; - - Ok(FileMetadata { - size: meta.content_length(), - }) + self.storage.metadata(&self.path).await } /// Read and returns whole content of file. /// /// For continuous reading, use [`Self::reader`] instead. pub async fn read(&self) -> crate::Result { - Ok(self - .op - .read(&self.path[self.relative_path_pos..]) - .await? - .to_bytes()) + self.storage.read(&self.path).await } /// Creates [`FileRead`] for continuous reading. /// /// For one-time reading, use [`Self::read`] instead. - pub async fn reader(&self) -> crate::Result> { - Ok(self.op.reader(&self.path[self.relative_path_pos..]).await?) + pub async fn reader(&self) -> crate::Result> { + self.storage.reader(&self.path).await } } @@ -366,40 +332,20 @@ pub trait FileWrite: Send + Unpin + 'static { async fn close(&mut self) -> crate::Result<()>; } -#[async_trait::async_trait] -impl FileWrite for opendal::Writer { - async fn write(&mut self, bs: Bytes) -> crate::Result<()> { - Ok(opendal::Writer::write(self, bs).await?) - } - - async fn close(&mut self) -> crate::Result<()> { - let _ = opendal::Writer::close(self).await?; - Ok(()) - } -} - -#[async_trait::async_trait] -impl FileWrite for Box { - async fn write(&mut self, bs: Bytes) -> crate::Result<()> { - self.as_mut().write(bs).await - } - - async fn close(&mut self) -> crate::Result<()> { - self.as_mut().close().await - } -} - /// Output file is used for writing to files.. #[derive(Debug)] pub struct OutputFile { - op: Operator, - // Absolution path of file. + storage: Arc, + // Absolute path of file. path: String, - // Relative path of file to uri, starts at [`relative_path_pos`] - relative_path_pos: usize, } impl OutputFile { + /// Creates a new output file. + pub fn new(storage: Arc, path: String) -> Self { + Self { storage, path } + } + /// Relative path to root uri. pub fn location(&self) -> &str { &self.path @@ -407,22 +353,21 @@ impl OutputFile { /// Checks if file exists. pub async fn exists(&self) -> Result { - Ok(self.op.exists(&self.path[self.relative_path_pos..]).await?) + self.storage.exists(&self.path).await } /// Deletes file. /// /// If the file does not exist, it will not return error. pub async fn delete(&self) -> Result<()> { - Ok(self.op.delete(&self.path[self.relative_path_pos..]).await?) + self.storage.delete(&self.path).await } /// Converts into [`InputFile`]. pub fn to_input_file(self) -> InputFile { InputFile { - op: self.op, + storage: self.storage, path: self.path, - relative_path_pos: self.relative_path_pos, } } @@ -433,9 +378,7 @@ impl OutputFile { /// Calling `write` will overwrite the file if it exists. /// For continuous writing, use [`Self::writer`]. pub async fn write(&self, bs: Bytes) -> crate::Result<()> { - let mut writer = self.writer().await?; - writer.write(bs).await?; - writer.close().await + self.storage.write(&self.path, bs).await } /// Creates output file for continuous writing. @@ -444,9 +387,7 @@ impl OutputFile { /// /// For one-time writing, use [`Self::write`] instead. pub async fn writer(&self) -> crate::Result> { - Ok(Box::new( - self.op.writer(&self.path[self.relative_path_pos..]).await?, - )) + self.storage.writer(&self.path).await } } @@ -517,14 +458,14 @@ mod tests { assert!(file_io.exists(&a_path).await.unwrap()); // Remove a file should be no-op. - file_io.remove_dir_all(&a_path).await.unwrap(); + file_io.delete_prefix(&a_path).await.unwrap(); assert!(file_io.exists(&a_path).await.unwrap()); // Remove a not exist dir should be no-op. - file_io.remove_dir_all("not_exists/").await.unwrap(); + file_io.delete_prefix("not_exists/").await.unwrap(); // Remove a dir should remove all files in it. - file_io.remove_dir_all(&sub_dir_path).await.unwrap(); + file_io.delete_prefix(&sub_dir_path).await.unwrap(); assert!(!file_io.exists(&b_path).await.unwrap()); assert!(!file_io.exists(&c_path).await.unwrap()); assert!(file_io.exists(&a_path).await.unwrap()); @@ -543,7 +484,7 @@ mod tests { let file_io = create_local_file_io(); assert!(!file_io.exists(&full_path).await.unwrap()); assert!(file_io.delete(&full_path).await.is_ok()); - assert!(file_io.remove_dir_all(&full_path).await.is_ok()); + assert!(file_io.delete_prefix(&full_path).await.is_ok()); } #[tokio::test] diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 8e40bae976..888b868e33 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -68,38 +68,17 @@ mod config; mod file_io; +mod opendal; mod storage; pub use config::*; pub use file_io::*; -pub use storage::{Storage, StorageFactory}; -pub(crate) mod object_cache; - -#[cfg(feature = "storage-azdls")] -mod storage_azdls; -#[cfg(feature = "storage-fs")] -mod storage_fs; -#[cfg(feature = "storage-gcs")] -mod storage_gcs; -#[cfg(feature = "storage-memory")] -mod storage_memory; -#[cfg(feature = "storage-oss")] -mod storage_oss; #[cfg(feature = "storage-s3")] -mod storage_s3; +pub use opendal::CustomAwsCredentialLoader; +pub use opendal::{OpenDalStorage, OpenDalStorageFactory}; +pub use storage::{Storage, StorageConfig, StorageFactory}; -#[cfg(feature = "storage-azdls")] -use storage_azdls::*; -#[cfg(feature = "storage-fs")] -use storage_fs::*; -#[cfg(feature = "storage-gcs")] -use storage_gcs::*; -#[cfg(feature = "storage-memory")] -use storage_memory::*; -#[cfg(feature = "storage-oss")] -use storage_oss::*; -#[cfg(feature = "storage-s3")] -pub use storage_s3::*; +pub(crate) mod object_cache; pub(crate) fn is_truthy(value: &str) -> bool { ["true", "t", "1", "on"].contains(&value.to_lowercase().as_str()) diff --git a/crates/iceberg/src/io/storage_azdls.rs b/crates/iceberg/src/io/opendal/azdls.rs similarity index 98% rename from crates/iceberg/src/io/storage_azdls.rs rename to crates/iceberg/src/io/opendal/azdls.rs index ce2b7427bc..c957fd62a3 100644 --- a/crates/iceberg/src/io/storage_azdls.rs +++ b/crates/iceberg/src/io/opendal/azdls.rs @@ -21,6 +21,7 @@ use std::str::FromStr; use opendal::Configurator; use opendal::services::AzdlsConfig; +use serde::{Deserialize, Serialize}; use url::Url; use crate::io::config::{ @@ -100,8 +101,8 @@ pub(crate) fn azdls_create_operator<'a>( /// paths are expected to contain the `dfs` storage service. /// - `wasb[s]` is used to refer to files in Blob Storage directly; paths are /// expected to contain the `blob` storage service. -#[derive(Debug, PartialEq)] -pub(crate) enum AzureStorageScheme { +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub enum AzureStorageScheme { Abfs, Abfss, Wasb, @@ -320,8 +321,7 @@ mod tests { use opendal::services::AzdlsConfig; - use super::{AzureStoragePath, AzureStorageScheme, azdls_create_operator}; - use crate::io::azdls_config_parse; + use super::{AzureStoragePath, AzureStorageScheme, azdls_config_parse, azdls_create_operator}; #[test] fn test_azdls_config_parse() { diff --git a/crates/iceberg/src/io/storage_fs.rs b/crates/iceberg/src/io/opendal/fs.rs similarity index 100% rename from crates/iceberg/src/io/storage_fs.rs rename to crates/iceberg/src/io/opendal/fs.rs diff --git a/crates/iceberg/src/io/storage_gcs.rs b/crates/iceberg/src/io/opendal/gcs.rs similarity index 100% rename from crates/iceberg/src/io/storage_gcs.rs rename to crates/iceberg/src/io/opendal/gcs.rs diff --git a/crates/iceberg/src/io/storage_memory.rs b/crates/iceberg/src/io/opendal/memory.rs similarity index 100% rename from crates/iceberg/src/io/storage_memory.rs rename to crates/iceberg/src/io/opendal/memory.rs diff --git a/crates/iceberg/src/io/opendal/mod.rs b/crates/iceberg/src/io/opendal/mod.rs new file mode 100644 index 0000000000..fb49dc9e3f --- /dev/null +++ b/crates/iceberg/src/io/opendal/mod.rs @@ -0,0 +1,474 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! OpenDAL-based storage implementation. + +use std::ops::Range; +use std::sync::Arc; + +use async_trait::async_trait; +#[cfg(feature = "storage-azdls")] +use azdls::AzureStorageScheme; +use bytes::Bytes; +use opendal::layers::RetryLayer; +#[cfg(feature = "storage-azdls")] +use opendal::services::AzdlsConfig; +#[cfg(feature = "storage-gcs")] +use opendal::services::GcsConfig; +#[cfg(feature = "storage-oss")] +use opendal::services::OssConfig; +#[cfg(feature = "storage-s3")] +use opendal::services::S3Config; +use opendal::{Operator, Scheme}; +#[cfg(feature = "storage-s3")] +pub use s3::CustomAwsCredentialLoader; +use serde::{Deserialize, Serialize}; + +use super::{ + FileIOBuilder, FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, + StorageConfig, StorageFactory, +}; +use crate::{Error, ErrorKind, Result}; + +#[cfg(feature = "storage-azdls")] +mod azdls; +#[cfg(feature = "storage-fs")] +mod fs; +#[cfg(feature = "storage-gcs")] +mod gcs; +#[cfg(feature = "storage-memory")] +mod memory; +#[cfg(feature = "storage-oss")] +mod oss; +#[cfg(feature = "storage-s3")] +mod s3; + +#[cfg(feature = "storage-azdls")] +use azdls::*; +#[cfg(feature = "storage-fs")] +use fs::*; +#[cfg(feature = "storage-gcs")] +use gcs::*; +#[cfg(feature = "storage-memory")] +use memory::*; +#[cfg(feature = "storage-oss")] +use oss::*; +#[cfg(feature = "storage-s3")] +pub use s3::*; + +/// OpenDAL-based storage factory. +/// +/// Maps scheme to the corresponding OpenDalStorage storage variant. +/// +/// TODO this is currently not used, we still use OpenDalStorage::build() for now +#[derive(Clone, Debug, Serialize, Deserialize)] +pub enum OpenDalStorageFactory { + /// Memory storage factory. + #[cfg(feature = "storage-memory")] + Memory, + /// Local filesystem storage factory. + #[cfg(feature = "storage-fs")] + Fs, + /// S3 storage factory. + #[cfg(feature = "storage-s3")] + S3 { + /// Custom AWS credential loader. + #[serde(skip)] + customized_credential_load: Option, + }, + /// GCS storage factory. + #[cfg(feature = "storage-gcs")] + Gcs, + /// OSS storage factory. + #[cfg(feature = "storage-oss")] + Oss, + /// Azure Data Lake Storage factory. + #[cfg(feature = "storage-azdls")] + Azdls { + /// The configured Azure storage scheme. + configured_scheme: AzureStorageScheme, + }, +} + +#[typetag::serde(name = "OpenDalStorageFactory")] +impl StorageFactory for OpenDalStorageFactory { + #[allow(unused_variables)] + fn build(&self, config: &StorageConfig) -> Result> { + match self { + #[cfg(feature = "storage-memory")] + OpenDalStorageFactory::Memory => { + Ok(Arc::new(OpenDalStorage::Memory(memory_config_build()?))) + } + #[cfg(feature = "storage-fs")] + OpenDalStorageFactory::Fs => Ok(Arc::new(OpenDalStorage::LocalFs)), + #[cfg(feature = "storage-s3")] + OpenDalStorageFactory::S3 { + customized_credential_load, + } => Ok(Arc::new(OpenDalStorage::S3 { + configured_scheme: "s3".to_string(), + config: s3_config_parse(config.props().clone())?.into(), + customized_credential_load: customized_credential_load.clone(), + })), + #[cfg(feature = "storage-gcs")] + OpenDalStorageFactory::Gcs => Ok(Arc::new(OpenDalStorage::Gcs { + config: gcs_config_parse(config.props().clone())?.into(), + })), + #[cfg(feature = "storage-oss")] + OpenDalStorageFactory::Oss => Ok(Arc::new(OpenDalStorage::Oss { + config: oss_config_parse(config.props().clone())?.into(), + })), + #[cfg(feature = "storage-azdls")] + OpenDalStorageFactory::Azdls { configured_scheme } => { + Ok(Arc::new(OpenDalStorage::Azdls { + configured_scheme: configured_scheme.clone(), + config: azdls_config_parse(config.props().clone())?.into(), + })) + } + #[cfg(all( + not(feature = "storage-memory"), + not(feature = "storage-fs"), + not(feature = "storage-s3"), + not(feature = "storage-gcs"), + not(feature = "storage-oss"), + not(feature = "storage-azdls"), + ))] + _ => Err(Error::new( + ErrorKind::FeatureUnsupported, + "No storage service has been enabled", + )), + } + } +} + +/// Default memory operator for serde deserialization. +#[cfg(feature = "storage-memory")] +fn default_memory_operator() -> Operator { + memory_config_build().expect("Failed to create default memory operator") +} + +/// OpenDAL-based storage implementation. +#[derive(Clone, Debug, Serialize, Deserialize)] +pub enum OpenDalStorage { + /// Memory storage variant. + #[cfg(feature = "storage-memory")] + Memory(#[serde(skip, default = "self::default_memory_operator")] Operator), + /// Local filesystem storage variant. + #[cfg(feature = "storage-fs")] + LocalFs, + /// S3 storage variant. + #[cfg(feature = "storage-s3")] + S3 { + /// s3 storage could have `s3://` and `s3a://`. + /// Storing the scheme string here to return the correct path. + configured_scheme: String, + /// S3 configuration. + config: Arc, + /// Custom AWS credential loader. + #[serde(skip)] + customized_credential_load: Option, + }, + /// GCS storage variant. + #[cfg(feature = "storage-gcs")] + Gcs { + /// GCS configuration. + config: Arc, + }, + /// OSS storage variant. + #[cfg(feature = "storage-oss")] + Oss { + /// OSS configuration. + config: Arc, + }, + /// Azure Data Lake Storage variant. + /// Expects paths of the form + /// `abfs[s]://@.dfs./` or + /// `wasb[s]://@.blob./`. + #[cfg(feature = "storage-azdls")] + #[allow(private_interfaces)] + Azdls { + /// The configured Azure storage scheme. + /// Because Azdls accepts multiple possible schemes, we store the full + /// passed scheme here to later validate schemes passed via paths. + configured_scheme: AzureStorageScheme, + /// Azure DLS configuration. + config: Arc, + }, +} + +impl OpenDalStorage { + /// Convert iceberg config to opendal config. + /// + /// TODO Switch to use OpenDalStorageFactory::build() + pub(crate) fn build(file_io_builder: FileIOBuilder) -> Result { + let (scheme_str, props, extensions) = file_io_builder.into_parts(); + let _ = (&props, &extensions); + let scheme = Self::parse_scheme(&scheme_str)?; + + match scheme { + #[cfg(feature = "storage-memory")] + Scheme::Memory => Ok(Self::Memory(memory_config_build()?)), + #[cfg(feature = "storage-fs")] + Scheme::Fs => Ok(Self::LocalFs), + #[cfg(feature = "storage-s3")] + Scheme::S3 => Ok(Self::S3 { + configured_scheme: scheme_str, + config: s3_config_parse(props)?.into(), + customized_credential_load: extensions + .get::() + .map(Arc::unwrap_or_clone), + }), + #[cfg(feature = "storage-gcs")] + Scheme::Gcs => Ok(Self::Gcs { + config: gcs_config_parse(props)?.into(), + }), + #[cfg(feature = "storage-oss")] + Scheme::Oss => Ok(Self::Oss { + config: oss_config_parse(props)?.into(), + }), + #[cfg(feature = "storage-azdls")] + Scheme::Azdls => { + let scheme = scheme_str.parse::()?; + Ok(Self::Azdls { + config: azdls_config_parse(props)?.into(), + configured_scheme: scheme, + }) + } + // Update doc on [`FileIO`] when adding new schemes. + _ => Err(Error::new( + ErrorKind::FeatureUnsupported, + format!("Constructing file io from scheme: {scheme} not supported now",), + )), + } + } + + /// Creates operator from path. + /// + /// # Arguments + /// + /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. + /// + /// # Returns + /// + /// The return value consists of two parts: + /// + /// * An [`opendal::Operator`] instance used to operate on file. + /// * Relative path to the root uri of [`opendal::Operator`]. + #[allow(unreachable_code, unused_variables)] + pub(crate) fn create_operator<'a>( + &self, + path: &'a impl AsRef, + ) -> Result<(Operator, &'a str)> { + let path = path.as_ref(); + let (operator, relative_path): (Operator, &str) = match self { + #[cfg(feature = "storage-memory")] + OpenDalStorage::Memory(op) => { + if let Some(stripped) = path.strip_prefix("memory:/") { + (op.clone(), stripped) + } else { + (op.clone(), &path[1..]) + } + } + #[cfg(feature = "storage-fs")] + OpenDalStorage::LocalFs => { + let op = fs_config_build()?; + if let Some(stripped) = path.strip_prefix("file:/") { + (op, stripped) + } else { + (op, &path[1..]) + } + } + #[cfg(feature = "storage-s3")] + OpenDalStorage::S3 { + configured_scheme, + config, + customized_credential_load, + } => { + let op = s3_config_build(config, customized_credential_load, path)?; + let op_info = op.info(); + + // Check prefix of s3 path. + let prefix = format!("{}://{}/", configured_scheme, op_info.name()); + if path.starts_with(&prefix) { + (op, &path[prefix.len()..]) + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Invalid s3 url: {path}, should start with {prefix}"), + )); + } + } + #[cfg(feature = "storage-gcs")] + OpenDalStorage::Gcs { config } => { + let operator = gcs_config_build(config, path)?; + let prefix = format!("gs://{}/", operator.info().name()); + if path.starts_with(&prefix) { + (operator, &path[prefix.len()..]) + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Invalid gcs url: {path}, should start with {prefix}"), + )); + } + } + #[cfg(feature = "storage-oss")] + OpenDalStorage::Oss { config } => { + let op = oss_config_build(config, path)?; + let prefix = format!("oss://{}/", op.info().name()); + if path.starts_with(&prefix) { + (op, &path[prefix.len()..]) + } else { + return Err(Error::new( + ErrorKind::DataInvalid, + format!("Invalid oss url: {path}, should start with {prefix}"), + )); + } + } + #[cfg(feature = "storage-azdls")] + OpenDalStorage::Azdls { + configured_scheme, + config, + } => azdls_create_operator(path, config, configured_scheme)?, + #[cfg(all( + not(feature = "storage-s3"), + not(feature = "storage-fs"), + not(feature = "storage-gcs"), + not(feature = "storage-oss"), + not(feature = "storage-azdls"), + ))] + _ => { + return Err(Error::new( + ErrorKind::FeatureUnsupported, + "No storage service has been enabled", + )); + } + }; + + // Transient errors are common for object stores; however there's no + // harm in retrying temporary failures for other storage backends as well. + let operator = operator.layer(RetryLayer::new()); + Ok((operator, relative_path)) + } + + /// Parse scheme. + fn parse_scheme(scheme: &str) -> Result { + match scheme { + "memory" => Ok(Scheme::Memory), + "file" | "" => Ok(Scheme::Fs), + "s3" | "s3a" => Ok(Scheme::S3), + "gs" | "gcs" => Ok(Scheme::Gcs), + "oss" => Ok(Scheme::Oss), + "abfss" | "abfs" | "wasbs" | "wasb" => Ok(Scheme::Azdls), + s => Ok(s.parse::()?), + } + } +} + +#[typetag::serde(name = "OpenDalStorage")] +#[async_trait] +impl Storage for OpenDalStorage { + async fn exists(&self, path: &str) -> Result { + let (op, relative_path) = self.create_operator(&path)?; + Ok(op.exists(relative_path).await?) + } + + async fn metadata(&self, path: &str) -> Result { + let (op, relative_path) = self.create_operator(&path)?; + let meta = op.stat(relative_path).await?; + Ok(FileMetadata { + size: meta.content_length(), + }) + } + + async fn read(&self, path: &str) -> Result { + let (op, relative_path) = self.create_operator(&path)?; + Ok(op.read(relative_path).await?.to_bytes()) + } + + async fn reader(&self, path: &str) -> Result> { + let (op, relative_path) = self.create_operator(&path)?; + Ok(Box::new(op.reader(relative_path).await?)) + } + + async fn write(&self, path: &str, bs: Bytes) -> Result<()> { + let (op, relative_path) = self.create_operator(&path)?; + op.write(relative_path, bs).await?; + Ok(()) + } + + async fn writer(&self, path: &str) -> Result> { + let (op, relative_path) = self.create_operator(&path)?; + Ok(Box::new(op.writer(relative_path).await?)) + } + + async fn delete(&self, path: &str) -> Result<()> { + let (op, relative_path) = self.create_operator(&path)?; + Ok(op.delete(relative_path).await?) + } + + async fn delete_prefix(&self, path: &str) -> Result<()> { + let (op, relative_path) = self.create_operator(&path)?; + let path = if relative_path.ends_with('/') { + relative_path.to_string() + } else { + format!("{relative_path}/") + }; + Ok(op.remove_all(&path).await?) + } + + #[allow(unreachable_code, unused_variables)] + fn new_input(&self, path: &str) -> Result { + Ok(InputFile::new(Arc::new(self.clone()), path.to_string())) + } + + #[allow(unreachable_code, unused_variables)] + fn new_output(&self, path: &str) -> Result { + Ok(OutputFile::new(Arc::new(self.clone()), path.to_string())) + } +} + +// OpenDAL implementations for FileRead and FileWrite traits + +#[async_trait] +impl FileRead for opendal::Reader { + async fn read(&self, range: Range) -> Result { + Ok(opendal::Reader::read(self, range).await?.to_bytes()) + } +} + +#[async_trait] +impl FileWrite for opendal::Writer { + async fn write(&mut self, bs: Bytes) -> Result<()> { + Ok(opendal::Writer::write(self, bs).await?) + } + + async fn close(&mut self) -> Result<()> { + let _ = opendal::Writer::close(self).await?; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[cfg(feature = "storage-memory")] + #[test] + fn test_default_memory_operator() { + let op = default_memory_operator(); + assert_eq!(op.info().scheme().to_string(), "memory"); + } +} diff --git a/crates/iceberg/src/io/storage_oss.rs b/crates/iceberg/src/io/opendal/oss.rs similarity index 100% rename from crates/iceberg/src/io/storage_oss.rs rename to crates/iceberg/src/io/opendal/oss.rs diff --git a/crates/iceberg/src/io/storage_s3.rs b/crates/iceberg/src/io/opendal/s3.rs similarity index 100% rename from crates/iceberg/src/io/storage_s3.rs rename to crates/iceberg/src/io/opendal/s3.rs diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage.rs index 77978e2239..15cc85ab10 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage.rs @@ -15,30 +15,17 @@ // specific language governing permissions and limitations // under the License. +//! Storage interfaces for Iceberg. + use std::fmt::Debug; use std::sync::Arc; use async_trait::async_trait; use bytes::Bytes; -use opendal::layers::RetryLayer; -#[cfg(feature = "storage-azdls")] -use opendal::services::AzdlsConfig; -#[cfg(feature = "storage-gcs")] -use opendal::services::GcsConfig; -#[cfg(feature = "storage-oss")] -use opendal::services::OssConfig; -#[cfg(feature = "storage-s3")] -use opendal::services::S3Config; -use opendal::{Operator, Scheme}; -#[cfg(feature = "storage-azdls")] -use super::AzureStorageScheme; -use super::{ - FileIOBuilder, FileMetadata, FileRead, FileWrite, InputFile, OutputFile, StorageConfig, -}; -#[cfg(feature = "storage-s3")] -use crate::io::CustomAwsCredentialLoader; -use crate::{Error, ErrorKind, Result}; +use super::{FileMetadata, FileRead, FileWrite, InputFile, OutputFile}; +use crate::Result; +pub use crate::io::config::StorageConfig; /// Trait for storage operations in Iceberg. /// @@ -153,203 +140,3 @@ pub trait StorageFactory: Debug + Send + Sync { /// if the storage could not be created. fn build(&self, config: &StorageConfig) -> Result>; } - -/// The storage carries all supported storage services in iceberg -#[derive(Debug)] -pub(crate) enum OpenDalStorage { - #[cfg(feature = "storage-memory")] - Memory(Operator), - #[cfg(feature = "storage-fs")] - LocalFs, - /// Expects paths of the form `s3[a]:///`. - #[cfg(feature = "storage-s3")] - S3 { - /// s3 storage could have `s3://` and `s3a://`. - /// Storing the scheme string here to return the correct path. - configured_scheme: String, - config: Arc, - customized_credential_load: Option, - }, - #[cfg(feature = "storage-gcs")] - Gcs { config: Arc }, - #[cfg(feature = "storage-oss")] - Oss { config: Arc }, - /// Expects paths of the form - /// `abfs[s]://@.dfs./` or - /// `wasb[s]://@.blob./`. - #[cfg(feature = "storage-azdls")] - Azdls { - /// Because Azdls accepts multiple possible schemes, we store the full - /// passed scheme here to later validate schemes passed via paths. - configured_scheme: AzureStorageScheme, - config: Arc, - }, -} - -impl OpenDalStorage { - /// Convert iceberg config to opendal config. - pub(crate) fn build(file_io_builder: FileIOBuilder) -> crate::Result { - let (scheme_str, props, extensions) = file_io_builder.into_parts(); - let _ = (&props, &extensions); - let scheme = Self::parse_scheme(&scheme_str)?; - - match scheme { - #[cfg(feature = "storage-memory")] - Scheme::Memory => Ok(Self::Memory(super::memory_config_build()?)), - #[cfg(feature = "storage-fs")] - Scheme::Fs => Ok(Self::LocalFs), - #[cfg(feature = "storage-s3")] - Scheme::S3 => Ok(Self::S3 { - configured_scheme: scheme_str, - config: super::s3_config_parse(props)?.into(), - customized_credential_load: extensions - .get::() - .map(Arc::unwrap_or_clone), - }), - #[cfg(feature = "storage-gcs")] - Scheme::Gcs => Ok(Self::Gcs { - config: super::gcs_config_parse(props)?.into(), - }), - #[cfg(feature = "storage-oss")] - Scheme::Oss => Ok(Self::Oss { - config: super::oss_config_parse(props)?.into(), - }), - #[cfg(feature = "storage-azdls")] - Scheme::Azdls => { - let scheme = scheme_str.parse::()?; - Ok(Self::Azdls { - config: super::azdls_config_parse(props)?.into(), - configured_scheme: scheme, - }) - } - // Update doc on [`FileIO`] when adding new schemes. - _ => Err(Error::new( - ErrorKind::FeatureUnsupported, - format!("Constructing file io from scheme: {scheme} not supported now",), - )), - } - } - - /// Creates operator from path. - /// - /// # Arguments - /// - /// * path: It should be *absolute* path starting with scheme string used to construct [`FileIO`]. - /// - /// # Returns - /// - /// The return value consists of two parts: - /// - /// * An [`opendal::Operator`] instance used to operate on file. - /// * Relative path to the root uri of [`opendal::Operator`]. - pub(crate) fn create_operator<'a>( - &self, - path: &'a impl AsRef, - ) -> crate::Result<(Operator, &'a str)> { - let path = path.as_ref(); - let _ = path; - let (operator, relative_path): (Operator, &str) = match self { - #[cfg(feature = "storage-memory")] - OpenDalStorage::Memory(op) => { - if let Some(stripped) = path.strip_prefix("memory:/") { - Ok::<_, crate::Error>((op.clone(), stripped)) - } else { - Ok::<_, crate::Error>((op.clone(), &path[1..])) - } - } - #[cfg(feature = "storage-fs")] - OpenDalStorage::LocalFs => { - let op = super::fs_config_build()?; - - if let Some(stripped) = path.strip_prefix("file:/") { - Ok::<_, crate::Error>((op, stripped)) - } else { - Ok::<_, crate::Error>((op, &path[1..])) - } - } - #[cfg(feature = "storage-s3")] - OpenDalStorage::S3 { - configured_scheme, - config, - customized_credential_load, - } => { - let op = super::s3_config_build(config, customized_credential_load, path)?; - let op_info = op.info(); - - // Check prefix of s3 path. - let prefix = format!("{}://{}/", configured_scheme, op_info.name()); - if path.starts_with(&prefix) { - Ok((op, &path[prefix.len()..])) - } else { - Err(Error::new( - ErrorKind::DataInvalid, - format!("Invalid s3 url: {path}, should start with {prefix}"), - )) - } - } - #[cfg(feature = "storage-gcs")] - OpenDalStorage::Gcs { config } => { - let operator = super::gcs_config_build(config, path)?; - let prefix = format!("gs://{}/", operator.info().name()); - if path.starts_with(&prefix) { - Ok((operator, &path[prefix.len()..])) - } else { - Err(Error::new( - ErrorKind::DataInvalid, - format!("Invalid gcs url: {path}, should start with {prefix}"), - )) - } - } - #[cfg(feature = "storage-oss")] - OpenDalStorage::Oss { config } => { - let op = super::oss_config_build(config, path)?; - - // Check prefix of oss path. - let prefix = format!("oss://{}/", op.info().name()); - if path.starts_with(&prefix) { - Ok((op, &path[prefix.len()..])) - } else { - Err(Error::new( - ErrorKind::DataInvalid, - format!("Invalid oss url: {path}, should start with {prefix}"), - )) - } - } - #[cfg(feature = "storage-azdls")] - OpenDalStorage::Azdls { - configured_scheme, - config, - } => super::azdls_create_operator(path, config, configured_scheme), - #[cfg(all( - not(feature = "storage-s3"), - not(feature = "storage-fs"), - not(feature = "storage-gcs"), - not(feature = "storage-oss"), - not(feature = "storage-azdls"), - ))] - _ => Err(Error::new( - ErrorKind::FeatureUnsupported, - "No storage service has been enabled", - )), - }?; - - // Transient errors are common for object stores; however there's no - // harm in retrying temporary failures for other storage backends as well. - let operator = operator.layer(RetryLayer::new()); - - Ok((operator, relative_path)) - } - - /// Parse scheme. - fn parse_scheme(scheme: &str) -> crate::Result { - match scheme { - "memory" => Ok(Scheme::Memory), - "file" | "" => Ok(Scheme::Fs), - "s3" | "s3a" => Ok(Scheme::S3), - "gs" | "gcs" => Ok(Scheme::Gcs), - "oss" => Ok(Scheme::Oss), - "abfss" | "abfs" | "wasbs" | "wasb" => Ok(Scheme::Azdls), - s => Ok(s.parse::()?), - } - } -} diff --git a/crates/iceberg/src/puffin/metadata.rs b/crates/iceberg/src/puffin/metadata.rs index b09f2c7c1e..3794eec458 100644 --- a/crates/iceberg/src/puffin/metadata.rs +++ b/crates/iceberg/src/puffin/metadata.rs @@ -285,10 +285,13 @@ impl FileMetadata { let input_file_length = input_file.metadata().await?.size; let footer_payload_length = - FileMetadata::read_footer_payload_length(&file_read, input_file_length).await?; - let footer_bytes = - FileMetadata::read_footer_bytes(&file_read, input_file_length, footer_payload_length) - .await?; + FileMetadata::read_footer_payload_length(file_read.as_ref(), input_file_length).await?; + let footer_bytes = FileMetadata::read_footer_bytes( + file_read.as_ref(), + input_file_length, + footer_payload_length, + ) + .await?; let magic_length = FileMetadata::MAGIC_LENGTH as usize; // check first four bytes of footer diff --git a/crates/iceberg/src/puffin/reader.rs b/crates/iceberg/src/puffin/reader.rs index a6308dcebf..d272f02d41 100644 --- a/crates/iceberg/src/puffin/reader.rs +++ b/crates/iceberg/src/puffin/reader.rs @@ -19,7 +19,7 @@ use tokio::sync::OnceCell; use super::validate_puffin_compression; use crate::Result; -use crate::io::{FileRead, InputFile}; +use crate::io::InputFile; use crate::puffin::blob::Blob; use crate::puffin::metadata::{BlobMetadata, FileMetadata}; diff --git a/crates/iceberg/src/writer/file_writer/parquet_writer.rs b/crates/iceberg/src/writer/file_writer/parquet_writer.rs index a75c74b3b2..da04d5435b 100644 --- a/crates/iceberg/src/writer/file_writer/parquet_writer.rs +++ b/crates/iceberg/src/writer/file_writer/parquet_writer.rs @@ -212,7 +212,7 @@ impl SchemaVisitor for IndexByParquetPathName { pub struct ParquetWriter { schema: SchemaRef, output_file: OutputFile, - inner_writer: Option>>>, + inner_writer: Option>, writer_properties: WriterProperties, current_row_num: usize, nan_value_count_visitor: NanValueCountVisitor, @@ -577,16 +577,16 @@ impl CurrentFileStatus for ParquetWriter { /// # NOTES /// /// We keep this wrapper been used inside only. -struct AsyncFileWriter(W); +struct AsyncFileWriter(Box); -impl AsyncFileWriter { +impl AsyncFileWriter { /// Create a new `AsyncFileWriter` with the given writer. - pub fn new(writer: W) -> Self { + pub fn new(writer: Box) -> Self { Self(writer) } } -impl ArrowAsyncFileWriter for AsyncFileWriter { +impl ArrowAsyncFileWriter for AsyncFileWriter { fn write(&mut self, bs: Bytes) -> BoxFuture<'_, parquet::errors::Result<()>> { Box::pin(async { self.0 From a198aab9f068c2ade22e3c29f8c99c239d2cdc9c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 09:09:24 +0800 Subject: [PATCH 191/891] chore(deps): Bump bytes from 1.11.0 to 1.11.1 (#2102) Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.11.0 to 1.11.1.
Release notes

Sourced from bytes's releases.

Bytes v1.11.1

1.11.1 (February 3rd, 2026)

  • Fix integer overflow in BytesMut::reserve
Changelog

Sourced from bytes's changelog.

1.11.1 (February 3rd, 2026)

  • Fix integer overflow in BytesMut::reserve
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=bytes&package-manager=cargo&previous-version=1.11.0&new-version=1.11.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/iceberg-rust/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 902f7c76ae..ed3fd34730 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1179,9 +1179,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.11.0" +version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b35204fbdc0b3f4446b89fc1ac2cf84a8a68971995d0bf2e925ec7cd960f9cb3" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" dependencies = [ "serde", ] From 099f5b4bad0ad0bb0df32e3d56d7f1cc41329875 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 4 Feb 2026 09:35:48 +0800 Subject: [PATCH 192/891] chore(deps): Bump bytes from 1.10.1 to 1.11.1 in /bindings/python (#2101) --- Cargo.toml | 2 +- bindings/python/Cargo.lock | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index 900377c22d..e114217b29 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -58,7 +58,7 @@ aws-sdk-s3tables = "1.28.0" backon = "1.5.1" base64 = "0.22.1" bimap = "0.6" -bytes = "1.10" +bytes = "1.11" chrono = "0.4.41" clap = { version = "4.5.48", features = ["derive", "cargo"] } dashmap = "6" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index 2bd52ef4cc..dd52ff2be6 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -640,9 +640,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.10.1" +version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d71b6127be86fdcfddb610f7182ac57211d4b18a3e9c82eb2d17662f2227ad6a" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" [[package]] name = "bzip2" From 5124737f9946ccc61f47542138c7376fde8e3d7d Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 3 Feb 2026 17:50:03 -0800 Subject: [PATCH 193/891] feat(io): Implement native LocalFsStorage (#2094) ## Which issue does this PR close? This PR depends on #2080 - Closes #2056 ## What changes are included in this PR? - Implemented `LocalFsStorage`, this can be used for testing simple cases ## Are these changes tested? Added some uts --- crates/iceberg/src/io/local_fs.rs | 537 ++++++++++++++++++++++++++++++ crates/iceberg/src/io/mod.rs | 1 + 2 files changed, 538 insertions(+) create mode 100644 crates/iceberg/src/io/local_fs.rs diff --git a/crates/iceberg/src/io/local_fs.rs b/crates/iceberg/src/io/local_fs.rs new file mode 100644 index 0000000000..0a55199f70 --- /dev/null +++ b/crates/iceberg/src/io/local_fs.rs @@ -0,0 +1,537 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Local filesystem storage implementation for testing. +//! +//! This module provides a `LocalFsStorage` implementation that uses standard +//! Rust filesystem operations. It is primarily intended for unit testing +//! scenarios where tests need to read/write files on the local filesystem. + +use std::fs; +use std::io::{Read, Seek, SeekFrom, Write}; +use std::ops::Range; +use std::path::PathBuf; +use std::sync::Arc; + +use async_trait::async_trait; +use bytes::Bytes; +use serde::{Deserialize, Serialize}; + +use super::{ + FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, StorageConfig, + StorageFactory, +}; +use crate::{Error, ErrorKind, Result}; + +/// Local filesystem storage implementation. +/// +/// This storage implementation uses standard Rust filesystem operations, +/// making it suitable for unit tests that need to read/write files on disk. +/// +/// # Path Normalization +/// +/// The storage normalizes paths to handle various formats: +/// - `file:///path/to/file` -> `/path/to/file` +/// - `file:/path/to/file` -> `/path/to/file` +/// - `/path/to/file` -> `/path/to/file` +/// ``` +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct LocalFsStorage; + +impl LocalFsStorage { + /// Create a new `LocalFsStorage` instance. + pub fn new() -> Self { + Self + } + + /// Normalize a path by removing scheme prefixes. + /// + /// This handles the following formats: + /// - `file:///path` -> `/path` + /// - `file://path` -> `/path` (treats as absolute) + /// - `file:/path` -> `/path` + /// - `/path` -> `/path` + pub(crate) fn normalize_path(path: &str) -> PathBuf { + let path = if let Some(stripped) = path.strip_prefix("file://") { + // file:///path -> /path or file://path -> /path + if stripped.starts_with('/') { + stripped.to_string() + } else { + format!("/{stripped}") + } + } else if let Some(stripped) = path.strip_prefix("file:") { + // file:/path -> /path + if stripped.starts_with('/') { + stripped.to_string() + } else { + format!("/{stripped}") + } + } else { + path.to_string() + }; + PathBuf::from(path) + } +} + +#[async_trait] +#[typetag::serde] +impl Storage for LocalFsStorage { + async fn exists(&self, path: &str) -> Result { + let path = Self::normalize_path(path); + Ok(path.exists()) + } + + async fn metadata(&self, path: &str) -> Result { + let path = Self::normalize_path(path); + let metadata = fs::metadata(&path).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to get metadata for {}: {}", path.display(), e), + ) + })?; + Ok(FileMetadata { + size: metadata.len(), + }) + } + + async fn read(&self, path: &str) -> Result { + let path = Self::normalize_path(path); + let content = fs::read(&path).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to read file {}: {}", path.display(), e), + ) + })?; + Ok(Bytes::from(content)) + } + + async fn reader(&self, path: &str) -> Result> { + let path = Self::normalize_path(path); + let file = fs::File::open(&path).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to open file {}: {}", path.display(), e), + ) + })?; + Ok(Box::new(LocalFsFileRead::new(file))) + } + + async fn write(&self, path: &str, bs: Bytes) -> Result<()> { + let path = Self::normalize_path(path); + + // Create parent directories if they don't exist + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to create directory {}: {}", parent.display(), e), + ) + })?; + } + + fs::write(&path, &bs).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to write file {}: {}", path.display(), e), + ) + })?; + Ok(()) + } + + async fn writer(&self, path: &str) -> Result> { + let path = Self::normalize_path(path); + + // Create parent directories if they don't exist + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to create directory {}: {}", parent.display(), e), + ) + })?; + } + + let file = fs::File::create(&path).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to create file {}: {}", path.display(), e), + ) + })?; + Ok(Box::new(LocalFsFileWrite::new(file))) + } + + async fn delete(&self, path: &str) -> Result<()> { + let path = Self::normalize_path(path); + if path.exists() { + fs::remove_file(&path).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to delete file {}: {}", path.display(), e), + ) + })?; + } + Ok(()) + } + + async fn delete_prefix(&self, path: &str) -> Result<()> { + let path = Self::normalize_path(path); + if path.is_dir() { + fs::remove_dir_all(&path).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to delete directory {}: {}", path.display(), e), + ) + })?; + } + Ok(()) + } + + fn new_input(&self, path: &str) -> Result { + Ok(InputFile::new(Arc::new(self.clone()), path.to_string())) + } + + fn new_output(&self, path: &str) -> Result { + Ok(OutputFile::new(Arc::new(self.clone()), path.to_string())) + } +} + +/// File reader for local filesystem storage. +#[derive(Debug)] +pub struct LocalFsFileRead { + file: std::sync::Mutex, +} + +impl LocalFsFileRead { + /// Create a new `LocalFsFileRead` with the given file. + pub fn new(file: fs::File) -> Self { + Self { + file: std::sync::Mutex::new(file), + } + } +} + +#[async_trait] +impl FileRead for LocalFsFileRead { + async fn read(&self, range: Range) -> Result { + let mut file = self.file.lock().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire file lock: {e}"), + ) + })?; + + file.seek(SeekFrom::Start(range.start)).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to seek to position {}: {}", range.start, e), + ) + })?; + + let len = (range.end - range.start) as usize; + let mut buffer = vec![0u8; len]; + file.read_exact(&mut buffer).map_err(|e| { + Error::new( + ErrorKind::DataInvalid, + format!("Failed to read {len} bytes: {e}"), + ) + })?; + + Ok(Bytes::from(buffer)) + } +} + +/// File writer for local filesystem storage. +/// +/// This struct implements `FileWrite` for writing to local files. +#[derive(Debug)] +pub struct LocalFsFileWrite { + file: Option, +} + +impl LocalFsFileWrite { + /// Create a new `LocalFsFileWrite` for the given file. + pub fn new(file: fs::File) -> Self { + Self { file: Some(file) } + } +} + +#[async_trait] +impl FileWrite for LocalFsFileWrite { + async fn write(&mut self, bs: Bytes) -> Result<()> { + let file = self + .file + .as_mut() + .ok_or_else(|| Error::new(ErrorKind::DataInvalid, "Cannot write to closed file"))?; + + file.write_all(&bs).map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to write to file: {e}"), + ) + })?; + + Ok(()) + } + + async fn close(&mut self) -> Result<()> { + let file = self + .file + .take() + .ok_or_else(|| Error::new(ErrorKind::DataInvalid, "File already closed"))?; + + file.sync_all() + .map_err(|e| Error::new(ErrorKind::Unexpected, format!("Failed to sync file: {e}")))?; + + Ok(()) + } +} + +/// Factory for creating `LocalFsStorage` instances. +/// +/// This factory implements `StorageFactory` and creates `LocalFsStorage` +/// instances for the "file" scheme. +/// +/// # Example +/// +/// ```rust,ignore +/// use iceberg::io::{StorageConfig, StorageFactory, LocalFsStorageFactory}; +/// +/// let factory = LocalFsStorageFactory; +/// let config = StorageConfig::new(); +/// let storage = factory.build(&config)?; +/// ``` +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct LocalFsStorageFactory; + +#[typetag::serde] +impl StorageFactory for LocalFsStorageFactory { + fn build(&self, _config: &StorageConfig) -> Result> { + Ok(Arc::new(LocalFsStorage::new())) + } +} + +#[cfg(test)] +mod tests { + use tempfile::TempDir; + + use super::*; + + #[test] + fn test_normalize_path() { + // Test file:/// prefix + assert_eq!( + LocalFsStorage::normalize_path("file:///path/to/file"), + PathBuf::from("/path/to/file") + ); + + // Test file:// prefix (without leading slash in path) + assert_eq!( + LocalFsStorage::normalize_path("file://path/to/file"), + PathBuf::from("/path/to/file") + ); + + // Test file:/ prefix + assert_eq!( + LocalFsStorage::normalize_path("file:/path/to/file"), + PathBuf::from("/path/to/file") + ); + + // Test bare path + assert_eq!( + LocalFsStorage::normalize_path("/path/to/file"), + PathBuf::from("/path/to/file") + ); + } + + #[tokio::test] + async fn test_local_fs_storage_write_read() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + let content = Bytes::from("Hello, World!"); + + // Write + storage.write(path_str, content.clone()).await.unwrap(); + + // Read + let read_content = storage.read(path_str).await.unwrap(); + assert_eq!(read_content, content); + } + + #[tokio::test] + async fn test_local_fs_storage_exists() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + + // File doesn't exist initially + assert!(!storage.exists(path_str).await.unwrap()); + + // Write file + storage.write(path_str, Bytes::from("test")).await.unwrap(); + + // File exists now + assert!(storage.exists(path_str).await.unwrap()); + } + + #[tokio::test] + async fn test_local_fs_storage_metadata() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + let content = Bytes::from("Hello, World!"); + + storage.write(path_str, content.clone()).await.unwrap(); + + let metadata = storage.metadata(path_str).await.unwrap(); + assert_eq!(metadata.size, content.len() as u64); + } + + #[tokio::test] + async fn test_local_fs_storage_delete() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + + storage.write(path_str, Bytes::from("test")).await.unwrap(); + assert!(storage.exists(path_str).await.unwrap()); + + storage.delete(path_str).await.unwrap(); + assert!(!storage.exists(path_str).await.unwrap()); + } + + #[tokio::test] + async fn test_local_fs_storage_delete_prefix() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let dir_path = tmp_dir.path().join("subdir"); + let file1 = dir_path.join("file1.txt"); + let file2 = dir_path.join("file2.txt"); + + // Create files in subdirectory + storage + .write(file1.to_str().unwrap(), Bytes::from("1")) + .await + .unwrap(); + storage + .write(file2.to_str().unwrap(), Bytes::from("2")) + .await + .unwrap(); + + // Delete prefix (directory) + storage + .delete_prefix(dir_path.to_str().unwrap()) + .await + .unwrap(); + + // Directory should be deleted + assert!(!dir_path.exists()); + } + + #[tokio::test] + async fn test_local_fs_storage_reader() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + let content = Bytes::from("Hello, World!"); + + storage.write(path_str, content.clone()).await.unwrap(); + + let reader = storage.reader(path_str).await.unwrap(); + let read_content = reader.read(0..content.len() as u64).await.unwrap(); + assert_eq!(read_content, content); + + // Test partial read + let partial = reader.read(0..5).await.unwrap(); + assert_eq!(partial, Bytes::from("Hello")); + } + + #[tokio::test] + async fn test_local_fs_storage_writer() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + + let mut writer = storage.writer(path_str).await.unwrap(); + writer.write(Bytes::from("Hello, ")).await.unwrap(); + writer.write(Bytes::from("World!")).await.unwrap(); + writer.close().await.unwrap(); + + let content = storage.read(path_str).await.unwrap(); + assert_eq!(content, Bytes::from("Hello, World!")); + } + + #[tokio::test] + async fn test_local_fs_file_write_double_close() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + + let mut writer = storage.writer(path_str).await.unwrap(); + writer.write(Bytes::from("test")).await.unwrap(); + writer.close().await.unwrap(); + + // Second close should fail + let result = writer.close().await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_local_fs_file_write_after_close() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("test.txt"); + let path_str = path.to_str().unwrap(); + + let mut writer = storage.writer(path_str).await.unwrap(); + writer.close().await.unwrap(); + + // Write after close should fail + let result = writer.write(Bytes::from("test")).await; + assert!(result.is_err()); + } + + #[test] + fn test_local_fs_storage_factory() { + let factory = LocalFsStorageFactory; + let config = StorageConfig::new(); + let storage = factory.build(&config).unwrap(); + + // Verify we got a valid storage instance + assert!(format!("{storage:?}").contains("LocalFsStorage")); + } + + #[tokio::test] + async fn test_local_fs_creates_parent_directories() { + let tmp_dir = TempDir::new().unwrap(); + let storage = LocalFsStorage::new(); + let path = tmp_dir.path().join("a/b/c/test.txt"); + let path_str = path.to_str().unwrap(); + + // Write should create parent directories + storage.write(path_str, Bytes::from("test")).await.unwrap(); + + assert!(path.exists()); + } +} diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 888b868e33..8c76bcbcc4 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -68,6 +68,7 @@ mod config; mod file_io; +mod local_fs; mod opendal; mod storage; From 8587044bc3f14fe9628df9c5b761cb3d63588825 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Tue, 3 Feb 2026 22:22:26 -0800 Subject: [PATCH 194/891] feat(io): Implement native MemoryStorage (#2097) ## Which issue does this PR close? - Closes #2057 - This depends on #2083 ## What changes are included in this PR? - Add a new MemoryStorage (not used anywhere for now) ## Are these changes tested? Added uts --- crates/iceberg/src/io/memory.rs | 597 ++++++++++++++++++++++++++++++++ crates/iceberg/src/io/mod.rs | 1 + 2 files changed, 598 insertions(+) create mode 100644 crates/iceberg/src/io/memory.rs diff --git a/crates/iceberg/src/io/memory.rs b/crates/iceberg/src/io/memory.rs new file mode 100644 index 0000000000..39f1f5db9d --- /dev/null +++ b/crates/iceberg/src/io/memory.rs @@ -0,0 +1,597 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Pure Rust in-memory storage implementation for testing. +//! +//! This module provides a `MemoryStorage` implementation that stores data +//! in a thread-safe `HashMap`, without any external dependencies. +//! It is primarily intended for unit testing and scenarios where persistent +//! storage is not needed. + +use std::collections::HashMap; +use std::ops::Range; +use std::sync::{Arc, RwLock}; + +use async_trait::async_trait; +use bytes::Bytes; +use serde::{Deserialize, Serialize}; + +use super::{ + FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, StorageConfig, + StorageFactory, +}; +use crate::{Error, ErrorKind, Result}; + +/// In-memory storage implementation. +/// +/// This storage implementation stores all data in a thread-safe `HashMap`, +/// making it suitable for unit tests and scenarios where persistent storage +/// is not needed. +/// +/// # Path Normalization +/// +/// The storage normalizes paths to handle various formats: +/// - `memory://path/to/file` -> `path/to/file` +/// - `memory:/path/to/file` -> `path/to/file` +/// - `/path/to/file` -> `path/to/file` +/// - `path/to/file` -> `path/to/file` +/// +/// # Serialization +/// +/// When serialized, `MemoryStorage` serializes to an empty state. When +/// deserialized, it creates a new empty instance. This is intentional +/// because in-memory data cannot be meaningfully serialized across +/// process boundaries. +/// ``` +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct MemoryStorage { + #[serde(skip, default = "default_memory_data")] + data: Arc>>, +} + +fn default_memory_data() -> Arc>> { + Arc::new(RwLock::new(HashMap::new())) +} + +impl MemoryStorage { + /// Create a new empty `MemoryStorage` instance. + pub fn new() -> Self { + Self { + data: Arc::new(RwLock::new(HashMap::new())), + } + } + + /// Normalize a path by removing scheme prefixes and leading slashes. + /// + /// This handles the following formats: + /// - `memory://path` -> `path` + /// - `memory:/path` -> `path` + /// - `/path` -> `path` + /// - `path` -> `path` + pub(crate) fn normalize_path(path: &str) -> String { + // Handle memory:// prefix (with double slash) + let path = path.strip_prefix("memory://").unwrap_or(path); + // Handle memory:/ prefix (with single slash) + let path = path.strip_prefix("memory:/").unwrap_or(path); + // Remove any leading slashes + path.trim_start_matches('/').to_string() + } +} + +#[async_trait] +#[typetag::serde] +impl Storage for MemoryStorage { + async fn exists(&self, path: &str) -> Result { + let normalized = Self::normalize_path(path); + let data = self.data.read().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire read lock: {e}"), + ) + })?; + Ok(data.contains_key(&normalized)) + } + + async fn metadata(&self, path: &str) -> Result { + let normalized = Self::normalize_path(path); + let data = self.data.read().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire read lock: {e}"), + ) + })?; + match data.get(&normalized) { + Some(bytes) => Ok(FileMetadata { + size: bytes.len() as u64, + }), + None => Err(Error::new( + ErrorKind::DataInvalid, + format!("File not found: {path}"), + )), + } + } + + async fn read(&self, path: &str) -> Result { + let normalized = Self::normalize_path(path); + let data = self.data.read().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire read lock: {e}"), + ) + })?; + match data.get(&normalized) { + Some(bytes) => Ok(bytes.clone()), + None => Err(Error::new( + ErrorKind::DataInvalid, + format!("File not found: {path}"), + )), + } + } + + async fn reader(&self, path: &str) -> Result> { + let normalized = Self::normalize_path(path); + let data = self.data.read().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire read lock: {e}"), + ) + })?; + match data.get(&normalized) { + Some(bytes) => Ok(Box::new(MemoryFileRead::new(bytes.clone()))), + None => Err(Error::new( + ErrorKind::DataInvalid, + format!("File not found: {path}"), + )), + } + } + + async fn write(&self, path: &str, bs: Bytes) -> Result<()> { + let normalized = Self::normalize_path(path); + let mut data = self.data.write().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire write lock: {e}"), + ) + })?; + data.insert(normalized, bs); + Ok(()) + } + + async fn writer(&self, path: &str) -> Result> { + let normalized = Self::normalize_path(path); + Ok(Box::new(MemoryFileWrite::new( + self.data.clone(), + normalized, + ))) + } + + async fn delete(&self, path: &str) -> Result<()> { + let normalized = Self::normalize_path(path); + let mut data = self.data.write().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire write lock: {e}"), + ) + })?; + data.remove(&normalized); + Ok(()) + } + + async fn delete_prefix(&self, path: &str) -> Result<()> { + let normalized = Self::normalize_path(path); + let prefix = if normalized.ends_with('/') { + normalized + } else { + format!("{normalized}/") + }; + + let mut data = self.data.write().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire write lock: {e}"), + ) + })?; + + // Collect keys to remove (can't modify while iterating) + let keys_to_remove: Vec = data + .keys() + .filter(|k| k.starts_with(&prefix)) + .cloned() + .collect(); + + for key in keys_to_remove { + data.remove(&key); + } + + Ok(()) + } + + fn new_input(&self, path: &str) -> Result { + Ok(InputFile::new(Arc::new(self.clone()), path.to_string())) + } + + fn new_output(&self, path: &str) -> Result { + Ok(OutputFile::new(Arc::new(self.clone()), path.to_string())) + } +} + +/// Factory for creating `MemoryStorage` instances. +/// +/// This factory implements `StorageFactory` and creates `MemoryStorage` +/// instances. Since the factory is explicitly chosen, no scheme validation +/// is performed - the storage will validate paths during operations. +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct MemoryStorageFactory; + +#[typetag::serde] +impl StorageFactory for MemoryStorageFactory { + fn build(&self, _config: &StorageConfig) -> Result> { + Ok(Arc::new(MemoryStorage::new())) + } +} + +/// File reader for in-memory storage. +#[derive(Debug)] +pub struct MemoryFileRead { + data: Bytes, +} + +impl MemoryFileRead { + /// Create a new `MemoryFileRead` with the given data. + pub fn new(data: Bytes) -> Self { + Self { data } + } +} + +#[async_trait] +impl FileRead for MemoryFileRead { + async fn read(&self, range: Range) -> Result { + let start = range.start as usize; + let end = range.end as usize; + + if start > self.data.len() || end > self.data.len() { + return Err(Error::new( + ErrorKind::DataInvalid, + format!( + "Range {}..{} is out of bounds for data of length {}", + start, + end, + self.data.len() + ), + )); + } + + Ok(self.data.slice(start..end)) + } +} + +/// File writer for in-memory storage. +/// +/// This struct implements `FileWrite` for writing to in-memory storage. +/// Data is buffered until `close()` is called, at which point it is +/// flushed to the storage. +#[derive(Debug)] +pub struct MemoryFileWrite { + data: Arc>>, + path: String, + buffer: Vec, + closed: bool, +} + +impl MemoryFileWrite { + /// Create a new `MemoryFileWrite` for the given path. + pub fn new(data: Arc>>, path: String) -> Self { + Self { + data, + path, + buffer: Vec::new(), + closed: false, + } + } +} + +#[async_trait] +impl FileWrite for MemoryFileWrite { + async fn write(&mut self, bs: Bytes) -> Result<()> { + if self.closed { + return Err(Error::new( + ErrorKind::DataInvalid, + "Cannot write to closed file", + )); + } + self.buffer.extend_from_slice(&bs); + Ok(()) + } + + async fn close(&mut self) -> Result<()> { + if self.closed { + return Err(Error::new(ErrorKind::DataInvalid, "File already closed")); + } + + let mut data = self.data.write().map_err(|e| { + Error::new( + ErrorKind::Unexpected, + format!("Failed to acquire write lock: {e}"), + ) + })?; + + data.insert( + self.path.clone(), + Bytes::from(std::mem::take(&mut self.buffer)), + ); + self.closed = true; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_normalize_path() { + // Test memory:// prefix + assert_eq!( + MemoryStorage::normalize_path("memory://path/to/file"), + "path/to/file" + ); + + // Test memory:/ prefix + assert_eq!( + MemoryStorage::normalize_path("memory:/path/to/file"), + "path/to/file" + ); + + // Test leading slash + assert_eq!( + MemoryStorage::normalize_path("/path/to/file"), + "path/to/file" + ); + + // Test bare path + assert_eq!( + MemoryStorage::normalize_path("path/to/file"), + "path/to/file" + ); + + // Test multiple leading slashes + assert_eq!( + MemoryStorage::normalize_path("///path/to/file"), + "path/to/file" + ); + + // Test memory:// with leading slash in path + assert_eq!( + MemoryStorage::normalize_path("memory:///path/to/file"), + "path/to/file" + ); + } + + #[tokio::test] + async fn test_memory_storage_write_read() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + let content = Bytes::from("Hello, World!"); + + // Write + storage.write(path, content.clone()).await.unwrap(); + + // Read + let read_content = storage.read(path).await.unwrap(); + assert_eq!(read_content, content); + } + + #[tokio::test] + async fn test_memory_storage_exists() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + + // File doesn't exist initially + assert!(!storage.exists(path).await.unwrap()); + + // Write file + storage.write(path, Bytes::from("test")).await.unwrap(); + + // File exists now + assert!(storage.exists(path).await.unwrap()); + } + + #[tokio::test] + async fn test_memory_storage_metadata() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + let content = Bytes::from("Hello, World!"); + + storage.write(path, content.clone()).await.unwrap(); + + let metadata = storage.metadata(path).await.unwrap(); + assert_eq!(metadata.size, content.len() as u64); + } + + #[tokio::test] + async fn test_memory_storage_delete() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + + storage.write(path, Bytes::from("test")).await.unwrap(); + assert!(storage.exists(path).await.unwrap()); + + storage.delete(path).await.unwrap(); + assert!(!storage.exists(path).await.unwrap()); + } + + #[tokio::test] + async fn test_memory_storage_delete_prefix() { + let storage = MemoryStorage::new(); + + // Create multiple files + storage + .write("memory://dir/file1.txt", Bytes::from("1")) + .await + .unwrap(); + storage + .write("memory://dir/file2.txt", Bytes::from("2")) + .await + .unwrap(); + storage + .write("memory://other/file.txt", Bytes::from("3")) + .await + .unwrap(); + + // Delete prefix + storage.delete_prefix("memory://dir").await.unwrap(); + + // Files in dir should be deleted + assert!(!storage.exists("memory://dir/file1.txt").await.unwrap()); + assert!(!storage.exists("memory://dir/file2.txt").await.unwrap()); + + // File in other dir should still exist + assert!(storage.exists("memory://other/file.txt").await.unwrap()); + } + + #[tokio::test] + async fn test_memory_storage_reader() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + let content = Bytes::from("Hello, World!"); + + storage.write(path, content.clone()).await.unwrap(); + + let reader = storage.reader(path).await.unwrap(); + let read_content = reader.read(0..content.len() as u64).await.unwrap(); + assert_eq!(read_content, content); + + // Test partial read + let partial = reader.read(0..5).await.unwrap(); + assert_eq!(partial, Bytes::from("Hello")); + } + + #[tokio::test] + async fn test_memory_storage_writer() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + + let mut writer = storage.writer(path).await.unwrap(); + writer.write(Bytes::from("Hello, ")).await.unwrap(); + writer.write(Bytes::from("World!")).await.unwrap(); + writer.close().await.unwrap(); + + let content = storage.read(path).await.unwrap(); + assert_eq!(content, Bytes::from("Hello, World!")); + } + + #[tokio::test] + async fn test_memory_file_write_double_close() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + + let mut writer = storage.writer(path).await.unwrap(); + writer.write(Bytes::from("test")).await.unwrap(); + writer.close().await.unwrap(); + + // Second close should fail + let result = writer.close().await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_memory_file_write_after_close() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + + let mut writer = storage.writer(path).await.unwrap(); + writer.close().await.unwrap(); + + // Write after close should fail + let result = writer.write(Bytes::from("test")).await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_memory_file_read_out_of_bounds() { + let storage = MemoryStorage::new(); + let path = "memory://test/file.txt"; + let content = Bytes::from("Hello"); + + storage.write(path, content).await.unwrap(); + + let reader = storage.reader(path).await.unwrap(); + let result = reader.read(0..100).await; + assert!(result.is_err()); + } + + #[test] + fn test_memory_storage_serialization() { + let storage = MemoryStorage::new(); + + // Serialize + let serialized = serde_json::to_string(&storage).unwrap(); + + // Deserialize + let deserialized: MemoryStorage = serde_json::from_str(&serialized).unwrap(); + + // Deserialized storage should be empty (new instance) + assert!(deserialized.data.read().unwrap().is_empty()); + } + + #[test] + fn test_memory_storage_factory() { + let factory = MemoryStorageFactory; + let config = StorageConfig::new(); + let storage = factory.build(&config).unwrap(); + + // Verify we got a valid storage instance + assert!(format!("{storage:?}").contains("MemoryStorage")); + } + + #[test] + fn test_memory_storage_factory_serialization() { + let factory = MemoryStorageFactory; + + // Serialize + let serialized = serde_json::to_string(&factory).unwrap(); + + // Deserialize + let deserialized: MemoryStorageFactory = serde_json::from_str(&serialized).unwrap(); + + // Verify the deserialized factory works + let config = StorageConfig::new(); + let storage = deserialized.build(&config).unwrap(); + assert!(format!("{storage:?}").contains("MemoryStorage")); + } + + #[tokio::test] + async fn test_path_normalization_consistency() { + let storage = MemoryStorage::new(); + let content = Bytes::from("test content"); + + // Write with one format + storage + .write("memory://path/to/file", content.clone()) + .await + .unwrap(); + + // Read with different formats - all should work + assert_eq!( + storage.read("memory://path/to/file").await.unwrap(), + content + ); + assert_eq!(storage.read("memory:/path/to/file").await.unwrap(), content); + assert_eq!(storage.read("/path/to/file").await.unwrap(), content); + assert_eq!(storage.read("path/to/file").await.unwrap(), content); + } +} diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 8c76bcbcc4..5f65a15fc3 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -69,6 +69,7 @@ mod config; mod file_io; mod local_fs; +mod memory; mod opendal; mod storage; From b41ecb455a4cee716864a4949b20dc229d91b844 Mon Sep 17 00:00:00 2001 From: geruh Date: Thu, 5 Feb 2026 14:02:15 -0800 Subject: [PATCH 195/891] fix(python): Bump Python version upper bound (#2110) ## What changes are included in this PR? Currently, pyiceberg-core has `requires-python = ">=3.10,<3.13"`, so PyIceberg 3.13 users can't install the Rust-accelerated core package. The `<3.13` upper bound was added in #1796, and prior to this pr it was `~=3.9` which is the equivalent of `>=3.9,<4` so this change basically restores that behavior. Also, `pyiceberg-core` uses PyO3's abi3-py310 feature, which already handles forward ABI compatibility across Python 3.x. This allows PyIceberg to test against new Python releases without waiting on us core. This pr also updates the trove classifiers to reflect this change and one more for Rust. ## Are these changes tested? `make test` --- bindings/python/pyproject.toml | 6 +- bindings/python/uv.lock | 197 ++++++++++++++++++++++++++++++++- 2 files changed, 199 insertions(+), 4 deletions(-) diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 14454b4408..38efa00b34 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -25,13 +25,17 @@ classifiers = [ "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Rust", ] name = "pyiceberg-core" readme = "project-description.md" -requires-python = ">=3.10,<3.13" +requires-python = ">=3.10,<4" dynamic = ["version"] license = { file = "LICENSE" } diff --git a/bindings/python/uv.lock b/bindings/python/uv.lock index 3c8ceb3bff..b474f6340e 100644 --- a/bindings/python/uv.lock +++ b/bindings/python/uv.lock @@ -1,6 +1,6 @@ version = 1 revision = 3 -requires-python = ">=3.10, <3.13" +requires-python = ">=3.10, <4" [[package]] name = "annotated-types" @@ -83,6 +83,38 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a8/ef/89297262b8092b312d29cdb2517cb1237e51db8ecef2e9af5edbe7b683b1/charset_normalizer-3.4.4-cp312-cp312-win32.whl", hash = "sha256:5833d2c39d8896e4e19b689ffc198f08ea58116bee26dea51e362ecc7cd3ed26", size = 99694, upload-time = "2025-10-14T04:41:09.23Z" }, { url = "https://files.pythonhosted.org/packages/3d/2d/1e5ed9dd3b3803994c155cd9aacb60c82c331bad84daf75bcb9c91b3295e/charset_normalizer-3.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:a79cfe37875f822425b89a82333404539ae63dbdddf97f84dcbc3d339aae9525", size = 107131, upload-time = "2025-10-14T04:41:10.467Z" }, { url = "https://files.pythonhosted.org/packages/d0/d9/0ed4c7098a861482a7b6a95603edce4c0d9db2311af23da1fb2b75ec26fc/charset_normalizer-3.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:376bec83a63b8021bb5c8ea75e21c4ccb86e7e45ca4eb81146091b56599b80c3", size = 100390, upload-time = "2025-10-14T04:41:11.915Z" }, + { url = "https://files.pythonhosted.org/packages/97/45/4b3a1239bbacd321068ea6e7ac28875b03ab8bc0aa0966452db17cd36714/charset_normalizer-3.4.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e1f185f86a6f3403aa2420e815904c67b2f9ebc443f045edd0de921108345794", size = 208091, upload-time = "2025-10-14T04:41:13.346Z" }, + { url = "https://files.pythonhosted.org/packages/7d/62/73a6d7450829655a35bb88a88fca7d736f9882a27eacdca2c6d505b57e2e/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b39f987ae8ccdf0d2642338faf2abb1862340facc796048b604ef14919e55ed", size = 147936, upload-time = "2025-10-14T04:41:14.461Z" }, + { url = "https://files.pythonhosted.org/packages/89/c5/adb8c8b3d6625bef6d88b251bbb0d95f8205831b987631ab0c8bb5d937c2/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3162d5d8ce1bb98dd51af660f2121c55d0fa541b46dff7bb9b9f86ea1d87de72", size = 144180, upload-time = "2025-10-14T04:41:15.588Z" }, + { url = "https://files.pythonhosted.org/packages/91/ed/9706e4070682d1cc219050b6048bfd293ccf67b3d4f5a4f39207453d4b99/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:81d5eb2a312700f4ecaa977a8235b634ce853200e828fbadf3a9c50bab278328", size = 161346, upload-time = "2025-10-14T04:41:16.738Z" }, + { url = "https://files.pythonhosted.org/packages/d5/0d/031f0d95e4972901a2f6f09ef055751805ff541511dc1252ba3ca1f80cf5/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5bd2293095d766545ec1a8f612559f6b40abc0eb18bb2f5d1171872d34036ede", size = 158874, upload-time = "2025-10-14T04:41:17.923Z" }, + { url = "https://files.pythonhosted.org/packages/f5/83/6ab5883f57c9c801ce5e5677242328aa45592be8a00644310a008d04f922/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8a8b89589086a25749f471e6a900d3f662d1d3b6e2e59dcecf787b1cc3a1894", size = 153076, upload-time = "2025-10-14T04:41:19.106Z" }, + { url = "https://files.pythonhosted.org/packages/75/1e/5ff781ddf5260e387d6419959ee89ef13878229732732ee73cdae01800f2/charset_normalizer-3.4.4-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc7637e2f80d8530ee4a78e878bce464f70087ce73cf7c1caf142416923b98f1", size = 150601, upload-time = "2025-10-14T04:41:20.245Z" }, + { url = "https://files.pythonhosted.org/packages/d7/57/71be810965493d3510a6ca79b90c19e48696fb1ff964da319334b12677f0/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f8bf04158c6b607d747e93949aa60618b61312fe647a6369f88ce2ff16043490", size = 150376, upload-time = "2025-10-14T04:41:21.398Z" }, + { url = "https://files.pythonhosted.org/packages/e5/d5/c3d057a78c181d007014feb7e9f2e65905a6c4ef182c0ddf0de2924edd65/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:554af85e960429cf30784dd47447d5125aaa3b99a6f0683589dbd27e2f45da44", size = 144825, upload-time = "2025-10-14T04:41:22.583Z" }, + { url = "https://files.pythonhosted.org/packages/e6/8c/d0406294828d4976f275ffbe66f00266c4b3136b7506941d87c00cab5272/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:74018750915ee7ad843a774364e13a3db91682f26142baddf775342c3f5b1133", size = 162583, upload-time = "2025-10-14T04:41:23.754Z" }, + { url = "https://files.pythonhosted.org/packages/d7/24/e2aa1f18c8f15c4c0e932d9287b8609dd30ad56dbe41d926bd846e22fb8d/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c0463276121fdee9c49b98908b3a89c39be45d86d1dbaa22957e38f6321d4ce3", size = 150366, upload-time = "2025-10-14T04:41:25.27Z" }, + { url = "https://files.pythonhosted.org/packages/e4/5b/1e6160c7739aad1e2df054300cc618b06bf784a7a164b0f238360721ab86/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:362d61fd13843997c1c446760ef36f240cf81d3ebf74ac62652aebaf7838561e", size = 160300, upload-time = "2025-10-14T04:41:26.725Z" }, + { url = "https://files.pythonhosted.org/packages/7a/10/f882167cd207fbdd743e55534d5d9620e095089d176d55cb22d5322f2afd/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9a26f18905b8dd5d685d6d07b0cdf98a79f3c7a918906af7cc143ea2e164c8bc", size = 154465, upload-time = "2025-10-14T04:41:28.322Z" }, + { url = "https://files.pythonhosted.org/packages/89/66/c7a9e1b7429be72123441bfdbaf2bc13faab3f90b933f664db506dea5915/charset_normalizer-3.4.4-cp313-cp313-win32.whl", hash = "sha256:9b35f4c90079ff2e2edc5b26c0c77925e5d2d255c42c74fdb70fb49b172726ac", size = 99404, upload-time = "2025-10-14T04:41:29.95Z" }, + { url = "https://files.pythonhosted.org/packages/c4/26/b9924fa27db384bdcd97ab83b4f0a8058d96ad9626ead570674d5e737d90/charset_normalizer-3.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:b435cba5f4f750aa6c0a0d92c541fb79f69a387c91e61f1795227e4ed9cece14", size = 107092, upload-time = "2025-10-14T04:41:31.188Z" }, + { url = "https://files.pythonhosted.org/packages/af/8f/3ed4bfa0c0c72a7ca17f0380cd9e4dd842b09f664e780c13cff1dcf2ef1b/charset_normalizer-3.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:542d2cee80be6f80247095cc36c418f7bddd14f4a6de45af91dfad36d817bba2", size = 100408, upload-time = "2025-10-14T04:41:32.624Z" }, + { url = "https://files.pythonhosted.org/packages/2a/35/7051599bd493e62411d6ede36fd5af83a38f37c4767b92884df7301db25d/charset_normalizer-3.4.4-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:da3326d9e65ef63a817ecbcc0df6e94463713b754fe293eaa03da99befb9a5bd", size = 207746, upload-time = "2025-10-14T04:41:33.773Z" }, + { url = "https://files.pythonhosted.org/packages/10/9a/97c8d48ef10d6cd4fcead2415523221624bf58bcf68a802721a6bc807c8f/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8af65f14dc14a79b924524b1e7fffe304517b2bff5a58bf64f30b98bbc5079eb", size = 147889, upload-time = "2025-10-14T04:41:34.897Z" }, + { url = "https://files.pythonhosted.org/packages/10/bf/979224a919a1b606c82bd2c5fa49b5c6d5727aa47b4312bb27b1734f53cd/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74664978bb272435107de04e36db5a9735e78232b85b77d45cfb38f758efd33e", size = 143641, upload-time = "2025-10-14T04:41:36.116Z" }, + { url = "https://files.pythonhosted.org/packages/ba/33/0ad65587441fc730dc7bd90e9716b30b4702dc7b617e6ba4997dc8651495/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:752944c7ffbfdd10c074dc58ec2d5a8a4cd9493b314d367c14d24c17684ddd14", size = 160779, upload-time = "2025-10-14T04:41:37.229Z" }, + { url = "https://files.pythonhosted.org/packages/67/ed/331d6b249259ee71ddea93f6f2f0a56cfebd46938bde6fcc6f7b9a3d0e09/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d1f13550535ad8cff21b8d757a3257963e951d96e20ec82ab44bc64aeb62a191", size = 159035, upload-time = "2025-10-14T04:41:38.368Z" }, + { url = "https://files.pythonhosted.org/packages/67/ff/f6b948ca32e4f2a4576aa129d8bed61f2e0543bf9f5f2b7fc3758ed005c9/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ecaae4149d99b1c9e7b88bb03e3221956f68fd6d50be2ef061b2381b61d20838", size = 152542, upload-time = "2025-10-14T04:41:39.862Z" }, + { url = "https://files.pythonhosted.org/packages/16/85/276033dcbcc369eb176594de22728541a925b2632f9716428c851b149e83/charset_normalizer-3.4.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cb6254dc36b47a990e59e1068afacdcd02958bdcce30bb50cc1700a8b9d624a6", size = 149524, upload-time = "2025-10-14T04:41:41.319Z" }, + { url = "https://files.pythonhosted.org/packages/9e/f2/6a2a1f722b6aba37050e626530a46a68f74e63683947a8acff92569f979a/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c8ae8a0f02f57a6e61203a31428fa1d677cbe50c93622b4149d5c0f319c1d19e", size = 150395, upload-time = "2025-10-14T04:41:42.539Z" }, + { url = "https://files.pythonhosted.org/packages/60/bb/2186cb2f2bbaea6338cad15ce23a67f9b0672929744381e28b0592676824/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:47cc91b2f4dd2833fddaedd2893006b0106129d4b94fdb6af1f4ce5a9965577c", size = 143680, upload-time = "2025-10-14T04:41:43.661Z" }, + { url = "https://files.pythonhosted.org/packages/7d/a5/bf6f13b772fbb2a90360eb620d52ed8f796f3c5caee8398c3b2eb7b1c60d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:82004af6c302b5d3ab2cfc4cc5f29db16123b1a8417f2e25f9066f91d4411090", size = 162045, upload-time = "2025-10-14T04:41:44.821Z" }, + { url = "https://files.pythonhosted.org/packages/df/c5/d1be898bf0dc3ef9030c3825e5d3b83f2c528d207d246cbabe245966808d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7d8f6c26245217bd2ad053761201e9f9680f8ce52f0fcd8d0755aeae5b2152", size = 149687, upload-time = "2025-10-14T04:41:46.442Z" }, + { url = "https://files.pythonhosted.org/packages/a5/42/90c1f7b9341eef50c8a1cb3f098ac43b0508413f33affd762855f67a410e/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:799a7a5e4fb2d5898c60b640fd4981d6a25f1c11790935a44ce38c54e985f828", size = 160014, upload-time = "2025-10-14T04:41:47.631Z" }, + { url = "https://files.pythonhosted.org/packages/76/be/4d3ee471e8145d12795ab655ece37baed0929462a86e72372fd25859047c/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:99ae2cffebb06e6c22bdc25801d7b30f503cc87dbd283479e7b606f70aff57ec", size = 154044, upload-time = "2025-10-14T04:41:48.81Z" }, + { url = "https://files.pythonhosted.org/packages/b0/6f/8f7af07237c34a1defe7defc565a9bc1807762f672c0fde711a4b22bf9c0/charset_normalizer-3.4.4-cp314-cp314-win32.whl", hash = "sha256:f9d332f8c2a2fcbffe1378594431458ddbef721c1769d78e2cbc06280d8155f9", size = 99940, upload-time = "2025-10-14T04:41:49.946Z" }, + { url = "https://files.pythonhosted.org/packages/4b/51/8ade005e5ca5b0d80fb4aff72a3775b325bdc3d27408c8113811a7cbe640/charset_normalizer-3.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:8a6562c3700cce886c5be75ade4a5db4214fda19fede41d9792d100288d8f94c", size = 107104, upload-time = "2025-10-14T04:41:51.051Z" }, + { url = "https://files.pythonhosted.org/packages/da/5f/6b8f83a55bb8278772c5ae54a577f3099025f9ade59d0136ac24a0df4bde/charset_normalizer-3.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:de00632ca48df9daf77a2c65a484531649261ec9f25489917f09e455cb09ddb2", size = 100743, upload-time = "2025-10-14T04:41:52.122Z" }, { url = "https://files.pythonhosted.org/packages/0a/4c/925909008ed5a988ccbb72dcc897407e5d6d3bd72410d69e051fc0c14647/charset_normalizer-3.4.4-py3-none-any.whl", hash = "sha256:7a32c560861a02ff789ad905a2fe94e3f840803362c84fecf1851cb4cf3dc37f", size = 53402, upload-time = "2025-10-14T04:42:31.76Z" }, ] @@ -113,7 +145,7 @@ version = "50.1.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pyarrow" }, - { name = "typing-extensions" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/fa/cc/e8e8f7c472e93e7a560203ac40ac319b926029007c0dad873dbba97f9f2d/datafusion-50.1.0.tar.gz", hash = "sha256:d8b8f027c7ce2498cda1589d3ce6d8720798963e031660fbe4d2e26e172442ec", size = 188103, upload-time = "2025-10-20T12:39:23.802Z" } wheels = [ @@ -129,7 +161,7 @@ name = "exceptiongroup" version = "1.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0b/9f/a65090624ecf468cdca03533906e7c69ed7588582240cfe7cc9e770b50eb/exceptiongroup-1.3.0.tar.gz", hash = "sha256:b241f5885f560bc56a59ee63ca4c6a8bfa46ae4ad651af316d4e81817bb9fd88", size = 29749, upload-time = "2025-05-10T17:42:51.123Z" } wheels = [ @@ -160,6 +192,27 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/af/f8/5a8df450a9f55ca8441f22ea0351d8c77809fc121498b6970daaaf667a21/fastavro-1.12.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a275e48df0b1701bb764b18a8a21900b24cf882263cb03d35ecdba636bbc830b", size = 3295258, upload-time = "2025-10-10T15:41:31.564Z" }, { url = "https://files.pythonhosted.org/packages/99/b2/40f25299111d737e58b85696e91138a66c25b7334f5357e7ac2b0e8966f8/fastavro-1.12.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2de72d786eb38be6b16d556b27232b1bf1b2797ea09599507938cdb7a9fe3e7c", size = 3430328, upload-time = "2025-10-10T15:41:33.689Z" }, { url = "https://files.pythonhosted.org/packages/e0/07/85157a7c57c5f8b95507d7829b5946561e5ee656ff80e9dd9a757f53ddaf/fastavro-1.12.1-cp312-cp312-win_amd64.whl", hash = "sha256:9090f0dee63fe022ee9cc5147483366cc4171c821644c22da020d6b48f576b4f", size = 444140, upload-time = "2025-10-10T15:41:34.902Z" }, + { url = "https://files.pythonhosted.org/packages/bb/57/26d5efef9182392d5ac9f253953c856ccb66e4c549fd3176a1e94efb05c9/fastavro-1.12.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:78df838351e4dff9edd10a1c41d1324131ffecbadefb9c297d612ef5363c049a", size = 1000599, upload-time = "2025-10-10T15:41:36.554Z" }, + { url = "https://files.pythonhosted.org/packages/33/cb/8ab55b21d018178eb126007a56bde14fd01c0afc11d20b5f2624fe01e698/fastavro-1.12.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:780476c23175d2ae457c52f45b9ffa9d504593499a36cd3c1929662bf5b7b14b", size = 3335933, upload-time = "2025-10-10T15:41:39.07Z" }, + { url = "https://files.pythonhosted.org/packages/fe/03/9c94ec9bf873eb1ffb0aa694f4e71940154e6e9728ddfdc46046d7e8ced4/fastavro-1.12.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0714b285160fcd515eb0455540f40dd6dac93bdeacdb03f24e8eac3d8aa51f8d", size = 3402066, upload-time = "2025-10-10T15:41:41.608Z" }, + { url = "https://files.pythonhosted.org/packages/75/c8/cb472347c5a584ccb8777a649ebb28278fccea39d005fc7df19996f41df8/fastavro-1.12.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a8bc2dcec5843d499f2489bfe0747999108f78c5b29295d877379f1972a3d41a", size = 3240038, upload-time = "2025-10-10T15:41:43.743Z" }, + { url = "https://files.pythonhosted.org/packages/e1/77/569ce9474c40304b3a09e109494e020462b83e405545b78069ddba5f614e/fastavro-1.12.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3b1921ac35f3d89090a5816b626cf46e67dbecf3f054131f84d56b4e70496f45", size = 3369398, upload-time = "2025-10-10T15:41:45.719Z" }, + { url = "https://files.pythonhosted.org/packages/4a/1f/9589e35e9ea68035385db7bdbf500d36b8891db474063fb1ccc8215ee37c/fastavro-1.12.1-cp313-cp313-win_amd64.whl", hash = "sha256:5aa777b8ee595b50aa084104cd70670bf25a7bbb9fd8bb5d07524b0785ee1699", size = 444220, upload-time = "2025-10-10T15:41:47.39Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d2/78435fe737df94bd8db2234b2100f5453737cffd29adee2504a2b013de84/fastavro-1.12.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c3d67c47f177e486640404a56f2f50b165fe892cc343ac3a34673b80cc7f1dd6", size = 1086611, upload-time = "2025-10-10T15:41:48.818Z" }, + { url = "https://files.pythonhosted.org/packages/b6/be/428f99b10157230ddac77ec8cc167005b29e2bd5cbe228345192bb645f30/fastavro-1.12.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5217f773492bac43dae15ff2931432bce2d7a80be7039685a78d3fab7df910bd", size = 3541001, upload-time = "2025-10-10T15:41:50.871Z" }, + { url = "https://files.pythonhosted.org/packages/16/08/a2eea4f20b85897740efe44887e1ac08f30dfa4bfc3de8962bdcbb21a5a1/fastavro-1.12.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:469fecb25cba07f2e1bfa4c8d008477cd6b5b34a59d48715e1b1a73f6160097d", size = 3432217, upload-time = "2025-10-10T15:41:53.149Z" }, + { url = "https://files.pythonhosted.org/packages/87/bb/b4c620b9eb6e9838c7f7e4b7be0762834443adf9daeb252a214e9ad3178c/fastavro-1.12.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d71c8aa841ef65cfab709a22bb887955f42934bced3ddb571e98fdbdade4c609", size = 3366742, upload-time = "2025-10-10T15:41:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/3d/d1/e69534ccdd5368350646fea7d93be39e5f77c614cca825c990bd9ca58f67/fastavro-1.12.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:b81fc04e85dfccf7c028e0580c606e33aa8472370b767ef058aae2c674a90746", size = 3383743, upload-time = "2025-10-10T15:41:57.68Z" }, + { url = "https://files.pythonhosted.org/packages/58/54/b7b4a0c3fb5fcba38128542da1b26c4e6d69933c923f493548bdfd63ab6a/fastavro-1.12.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:9445da127751ba65975d8e4bdabf36bfcfdad70fc35b2d988e3950cce0ec0e7c", size = 1001377, upload-time = "2025-10-10T15:41:59.241Z" }, + { url = "https://files.pythonhosted.org/packages/1e/4f/0e589089c7df0d8f57d7e5293fdc34efec9a3b758a0d4d0c99a7937e2492/fastavro-1.12.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ed924233272719b5d5a6a0b4d80ef3345fc7e84fc7a382b6232192a9112d38a6", size = 3320401, upload-time = "2025-10-10T15:42:01.682Z" }, + { url = "https://files.pythonhosted.org/packages/f9/19/260110d56194ae29d7e423a336fccea8bcd103196d00f0b364b732bdb84e/fastavro-1.12.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3616e2f0e1c9265e92954fa099db79c6e7817356d3ff34f4bcc92699ae99697c", size = 3350894, upload-time = "2025-10-10T15:42:04.073Z" }, + { url = "https://files.pythonhosted.org/packages/d0/96/58b0411e8be9694d5972bee3167d6c1fd1fdfdf7ce253c1a19a327208f4f/fastavro-1.12.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:cb0337b42fd3c047fcf0e9b7597bd6ad25868de719f29da81eabb6343f08d399", size = 3229644, upload-time = "2025-10-10T15:42:06.221Z" }, + { url = "https://files.pythonhosted.org/packages/5b/db/38660660eac82c30471d9101f45b3acfdcbadfe42d8f7cdb129459a45050/fastavro-1.12.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:64961ab15b74b7c168717bbece5660e0f3d457837c3cc9d9145181d011199fa7", size = 3329704, upload-time = "2025-10-10T15:42:08.384Z" }, + { url = "https://files.pythonhosted.org/packages/9d/a9/1672910f458ecb30b596c9e59e41b7c00309b602a0494341451e92e62747/fastavro-1.12.1-cp314-cp314-win_amd64.whl", hash = "sha256:792356d320f6e757e89f7ac9c22f481e546c886454a6709247f43c0dd7058004", size = 452911, upload-time = "2025-10-10T15:42:09.795Z" }, + { url = "https://files.pythonhosted.org/packages/dc/8d/2e15d0938ded1891b33eff252e8500605508b799c2e57188a933f0bd744c/fastavro-1.12.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:120aaf82ac19d60a1016afe410935fe94728752d9c2d684e267e5b7f0e70f6d9", size = 3541999, upload-time = "2025-10-10T15:42:11.794Z" }, + { url = "https://files.pythonhosted.org/packages/a7/1c/6dfd082a205be4510543221b734b1191299e6a1810c452b6bc76dfa6968e/fastavro-1.12.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b6a3462934b20a74f9ece1daa49c2e4e749bd9a35fa2657b53bf62898fba80f5", size = 3433972, upload-time = "2025-10-10T15:42:14.485Z" }, + { url = "https://files.pythonhosted.org/packages/24/90/9de694625a1a4b727b1ad0958d220cab25a9b6cf7f16a5c7faa9ea7b2261/fastavro-1.12.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:1f81011d54dd47b12437b51dd93a70a9aa17b61307abf26542fc3c13efbc6c51", size = 3368752, upload-time = "2025-10-10T15:42:16.618Z" }, + { url = "https://files.pythonhosted.org/packages/fa/93/b44f67589e4d439913dab6720f7e3507b0fa8b8e56d06f6fc875ced26afb/fastavro-1.12.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:43ded16b3f4a9f1a42f5970c2aa618acb23ea59c4fcaa06680bdf470b255e5a8", size = 3386636, upload-time = "2025-10-10T15:42:18.974Z" }, ] [[package]] @@ -210,6 +263,26 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/27/45/80935968b53cfd3f33cf99ea5f08227f2646e044568c9b1555b58ffd61c2/greenlet-3.2.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ee7a6ec486883397d70eec05059353b8e83eca9168b9f3f9a361971e77e0bcd0", size = 1564846, upload-time = "2025-11-04T12:42:15.191Z" }, { url = "https://files.pythonhosted.org/packages/69/02/b7c30e5e04752cb4db6202a3858b149c0710e5453b71a3b2aec5d78a1aab/greenlet-3.2.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:326d234cbf337c9c3def0676412eb7040a35a768efc92504b947b3e9cfc7543d", size = 1633814, upload-time = "2025-11-04T12:42:17.175Z" }, { url = "https://files.pythonhosted.org/packages/e9/08/b0814846b79399e585f974bbeebf5580fbe59e258ea7be64d9dfb253c84f/greenlet-3.2.4-cp312-cp312-win_amd64.whl", hash = "sha256:a7d4e128405eea3814a12cc2605e0e6aedb4035bf32697f72deca74de4105e02", size = 299899, upload-time = "2025-08-07T13:38:53.448Z" }, + { url = "https://files.pythonhosted.org/packages/49/e8/58c7f85958bda41dafea50497cbd59738c5c43dbbea5ee83d651234398f4/greenlet-3.2.4-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:1a921e542453fe531144e91e1feedf12e07351b1cf6c9e8a3325ea600a715a31", size = 272814, upload-time = "2025-08-07T13:15:50.011Z" }, + { url = "https://files.pythonhosted.org/packages/62/dd/b9f59862e9e257a16e4e610480cfffd29e3fae018a68c2332090b53aac3d/greenlet-3.2.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd3c8e693bff0fff6ba55f140bf390fa92c994083f838fece0f63be121334945", size = 641073, upload-time = "2025-08-07T13:42:57.23Z" }, + { url = "https://files.pythonhosted.org/packages/f7/0b/bc13f787394920b23073ca3b6c4a7a21396301ed75a655bcb47196b50e6e/greenlet-3.2.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:710638eb93b1fa52823aa91bf75326f9ecdfd5e0466f00789246a5280f4ba0fc", size = 655191, upload-time = "2025-08-07T13:45:29.752Z" }, + { url = "https://files.pythonhosted.org/packages/f2/d6/6adde57d1345a8d0f14d31e4ab9c23cfe8e2cd39c3baf7674b4b0338d266/greenlet-3.2.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:c5111ccdc9c88f423426df3fd1811bfc40ed66264d35aa373420a34377efc98a", size = 649516, upload-time = "2025-08-07T13:53:16.314Z" }, + { url = "https://files.pythonhosted.org/packages/7f/3b/3a3328a788d4a473889a2d403199932be55b1b0060f4ddd96ee7cdfcad10/greenlet-3.2.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d76383238584e9711e20ebe14db6c88ddcedc1829a9ad31a584389463b5aa504", size = 652169, upload-time = "2025-08-07T13:18:32.861Z" }, + { url = "https://files.pythonhosted.org/packages/ee/43/3cecdc0349359e1a527cbf2e3e28e5f8f06d3343aaf82ca13437a9aa290f/greenlet-3.2.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23768528f2911bcd7e475210822ffb5254ed10d71f4028387e5a99b4c6699671", size = 610497, upload-time = "2025-08-07T13:18:31.636Z" }, + { url = "https://files.pythonhosted.org/packages/b8/19/06b6cf5d604e2c382a6f31cafafd6f33d5dea706f4db7bdab184bad2b21d/greenlet-3.2.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:00fadb3fedccc447f517ee0d3fd8fe49eae949e1cd0f6a611818f4f6fb7dc83b", size = 1121662, upload-time = "2025-08-07T13:42:41.117Z" }, + { url = "https://files.pythonhosted.org/packages/a2/15/0d5e4e1a66fab130d98168fe984c509249c833c1a3c16806b90f253ce7b9/greenlet-3.2.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:d25c5091190f2dc0eaa3f950252122edbbadbb682aa7b1ef2f8af0f8c0afefae", size = 1149210, upload-time = "2025-08-07T13:18:24.072Z" }, + { url = "https://files.pythonhosted.org/packages/1c/53/f9c440463b3057485b8594d7a638bed53ba531165ef0ca0e6c364b5cc807/greenlet-3.2.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6e343822feb58ac4d0a1211bd9399de2b3a04963ddeec21530fc426cc121f19b", size = 1564759, upload-time = "2025-11-04T12:42:19.395Z" }, + { url = "https://files.pythonhosted.org/packages/47/e4/3bb4240abdd0a8d23f4f88adec746a3099f0d86bfedb623f063b2e3b4df0/greenlet-3.2.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca7f6f1f2649b89ce02f6f229d7c19f680a6238af656f61e0115b24857917929", size = 1634288, upload-time = "2025-11-04T12:42:21.174Z" }, + { url = "https://files.pythonhosted.org/packages/0b/55/2321e43595e6801e105fcfdee02b34c0f996eb71e6ddffca6b10b7e1d771/greenlet-3.2.4-cp313-cp313-win_amd64.whl", hash = "sha256:554b03b6e73aaabec3745364d6239e9e012d64c68ccd0b8430c64ccc14939a8b", size = 299685, upload-time = "2025-08-07T13:24:38.824Z" }, + { url = "https://files.pythonhosted.org/packages/22/5c/85273fd7cc388285632b0498dbbab97596e04b154933dfe0f3e68156c68c/greenlet-3.2.4-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:49a30d5fda2507ae77be16479bdb62a660fa51b1eb4928b524975b3bde77b3c0", size = 273586, upload-time = "2025-08-07T13:16:08.004Z" }, + { url = "https://files.pythonhosted.org/packages/d1/75/10aeeaa3da9332c2e761e4c50d4c3556c21113ee3f0afa2cf5769946f7a3/greenlet-3.2.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:299fd615cd8fc86267b47597123e3f43ad79c9d8a22bebdce535e53550763e2f", size = 686346, upload-time = "2025-08-07T13:42:59.944Z" }, + { url = "https://files.pythonhosted.org/packages/c0/aa/687d6b12ffb505a4447567d1f3abea23bd20e73a5bed63871178e0831b7a/greenlet-3.2.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:c17b6b34111ea72fc5a4e4beec9711d2226285f0386ea83477cbb97c30a3f3a5", size = 699218, upload-time = "2025-08-07T13:45:30.969Z" }, + { url = "https://files.pythonhosted.org/packages/dc/8b/29aae55436521f1d6f8ff4e12fb676f3400de7fcf27fccd1d4d17fd8fecd/greenlet-3.2.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b4a1870c51720687af7fa3e7cda6d08d801dae660f75a76f3845b642b4da6ee1", size = 694659, upload-time = "2025-08-07T13:53:17.759Z" }, + { url = "https://files.pythonhosted.org/packages/92/2e/ea25914b1ebfde93b6fc4ff46d6864564fba59024e928bdc7de475affc25/greenlet-3.2.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:061dc4cf2c34852b052a8620d40f36324554bc192be474b9e9770e8c042fd735", size = 695355, upload-time = "2025-08-07T13:18:34.517Z" }, + { url = "https://files.pythonhosted.org/packages/72/60/fc56c62046ec17f6b0d3060564562c64c862948c9d4bc8aa807cf5bd74f4/greenlet-3.2.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44358b9bf66c8576a9f57a590d5f5d6e72fa4228b763d0e43fee6d3b06d3a337", size = 657512, upload-time = "2025-08-07T13:18:33.969Z" }, + { url = "https://files.pythonhosted.org/packages/23/6e/74407aed965a4ab6ddd93a7ded3180b730d281c77b765788419484cdfeef/greenlet-3.2.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2917bdf657f5859fbf3386b12d68ede4cf1f04c90c3a6bc1f013dd68a22e2269", size = 1612508, upload-time = "2025-11-04T12:42:23.427Z" }, + { url = "https://files.pythonhosted.org/packages/0d/da/343cd760ab2f92bac1845ca07ee3faea9fe52bee65f7bcb19f16ad7de08b/greenlet-3.2.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:015d48959d4add5d6c9f6c5210ee3803a830dce46356e3bc326d6776bde54681", size = 1680760, upload-time = "2025-11-04T12:42:25.341Z" }, + { url = "https://files.pythonhosted.org/packages/e3/a5/6ddab2b4c112be95601c13428db1d8b6608a8b6039816f2ba09c346c08fc/greenlet-3.2.4-cp314-cp314-win_amd64.whl", hash = "sha256:e37ab26028f12dbb0ff65f29a8d3d44a765c61e729647bf2ddfbbed621726f01", size = 303425, upload-time = "2025-08-07T13:32:27.59Z" }, ] [[package]] @@ -329,6 +402,62 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/68/8f/534ae319c6e05d714f437e7206f78c17e66daca88164dff70286b0e8ea0c/mmh3-5.2.0-cp312-cp312-win32.whl", hash = "sha256:2421b9d665a0b1ad724ec7332fb5a98d075f50bc51a6ff854f3a1882bd650d49", size = 40805, upload-time = "2025-07-29T07:42:28.032Z" }, { url = "https://files.pythonhosted.org/packages/b8/f6/f6abdcfefcedab3c964868048cfe472764ed358c2bf6819a70dd4ed4ed3a/mmh3-5.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:72d80005b7634a3a2220f81fbeb94775ebd12794623bb2e1451701ea732b4aa3", size = 41597, upload-time = "2025-07-29T07:42:28.894Z" }, { url = "https://files.pythonhosted.org/packages/15/fd/f7420e8cbce45c259c770cac5718badf907b302d3a99ec587ba5ce030237/mmh3-5.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:3d6bfd9662a20c054bc216f861fa330c2dac7c81e7fb8307b5e32ab5b9b4d2e0", size = 39350, upload-time = "2025-07-29T07:42:29.794Z" }, + { url = "https://files.pythonhosted.org/packages/d8/fa/27f6ab93995ef6ad9f940e96593c5dd24744d61a7389532b0fec03745607/mmh3-5.2.0-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:e79c00eba78f7258e5b354eccd4d7907d60317ced924ea4a5f2e9d83f5453065", size = 40874, upload-time = "2025-07-29T07:42:30.662Z" }, + { url = "https://files.pythonhosted.org/packages/11/9c/03d13bcb6a03438bc8cac3d2e50f80908d159b31a4367c2e1a7a077ded32/mmh3-5.2.0-cp313-cp313-android_21_x86_64.whl", hash = "sha256:956127e663d05edbeec54df38885d943dfa27406594c411139690485128525de", size = 42012, upload-time = "2025-07-29T07:42:31.539Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/0865d9765408a7d504f1789944e678f74e0888b96a766d578cb80b040999/mmh3-5.2.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:c3dca4cb5b946ee91b3d6bb700d137b1cd85c20827f89fdf9c16258253489044", size = 39197, upload-time = "2025-07-29T07:42:32.374Z" }, + { url = "https://files.pythonhosted.org/packages/3e/12/76c3207bd186f98b908b6706c2317abb73756d23a4e68ea2bc94825b9015/mmh3-5.2.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:e651e17bfde5840e9e4174b01e9e080ce49277b70d424308b36a7969d0d1af73", size = 39840, upload-time = "2025-07-29T07:42:33.227Z" }, + { url = "https://files.pythonhosted.org/packages/5d/0d/574b6cce5555c9f2b31ea189ad44986755eb14e8862db28c8b834b8b64dc/mmh3-5.2.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:9f64bf06f4bf623325fda3a6d02d36cd69199b9ace99b04bb2d7fd9f89688504", size = 40644, upload-time = "2025-07-29T07:42:34.099Z" }, + { url = "https://files.pythonhosted.org/packages/52/82/3731f8640b79c46707f53ed72034a58baad400be908c87b0088f1f89f986/mmh3-5.2.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ddc63328889bcaee77b743309e5c7d2d52cee0d7d577837c91b6e7cc9e755e0b", size = 56153, upload-time = "2025-07-29T07:42:35.031Z" }, + { url = "https://files.pythonhosted.org/packages/4f/34/e02dca1d4727fd9fdeaff9e2ad6983e1552804ce1d92cc796e5b052159bb/mmh3-5.2.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:bb0fdc451fb6d86d81ab8f23d881b8d6e37fc373a2deae1c02d27002d2ad7a05", size = 40684, upload-time = "2025-07-29T07:42:35.914Z" }, + { url = "https://files.pythonhosted.org/packages/8f/36/3dee40767356e104967e6ed6d102ba47b0b1ce2a89432239b95a94de1b89/mmh3-5.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:b29044e1ffdb84fe164d0a7ea05c7316afea93c00f8ed9449cf357c36fc4f814", size = 40057, upload-time = "2025-07-29T07:42:36.755Z" }, + { url = "https://files.pythonhosted.org/packages/31/58/228c402fccf76eb39a0a01b8fc470fecf21965584e66453b477050ee0e99/mmh3-5.2.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:58981d6ea9646dbbf9e59a30890cbf9f610df0e4a57dbfe09215116fd90b0093", size = 97344, upload-time = "2025-07-29T07:42:37.675Z" }, + { url = "https://files.pythonhosted.org/packages/34/82/fc5ce89006389a6426ef28e326fc065b0fbaaed230373b62d14c889f47ea/mmh3-5.2.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7e5634565367b6d98dc4aa2983703526ef556b3688ba3065edb4b9b90ede1c54", size = 103325, upload-time = "2025-07-29T07:42:38.591Z" }, + { url = "https://files.pythonhosted.org/packages/09/8c/261e85777c6aee1ebd53f2f17e210e7481d5b0846cd0b4a5c45f1e3761b8/mmh3-5.2.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0271ac12415afd3171ab9a3c7cbfc71dee2c68760a7dc9d05bf8ed6ddfa3a7a", size = 106240, upload-time = "2025-07-29T07:42:39.563Z" }, + { url = "https://files.pythonhosted.org/packages/70/73/2f76b3ad8a3d431824e9934403df36c0ddacc7831acf82114bce3c4309c8/mmh3-5.2.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:45b590e31bc552c6f8e2150ff1ad0c28dd151e9f87589e7eaf508fbdd8e8e908", size = 113060, upload-time = "2025-07-29T07:42:40.585Z" }, + { url = "https://files.pythonhosted.org/packages/9f/b9/7ea61a34e90e50a79a9d87aa1c0b8139a7eaf4125782b34b7d7383472633/mmh3-5.2.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bdde97310d59604f2a9119322f61b31546748499a21b44f6715e8ced9308a6c5", size = 120781, upload-time = "2025-07-29T07:42:41.618Z" }, + { url = "https://files.pythonhosted.org/packages/0f/5b/ae1a717db98c7894a37aeedbd94b3f99e6472a836488f36b6849d003485b/mmh3-5.2.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:fc9c5f280438cf1c1a8f9abb87dc8ce9630a964120cfb5dd50d1e7ce79690c7a", size = 99174, upload-time = "2025-07-29T07:42:42.587Z" }, + { url = "https://files.pythonhosted.org/packages/e3/de/000cce1d799fceebb6d4487ae29175dd8e81b48e314cba7b4da90bcf55d7/mmh3-5.2.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:c903e71fd8debb35ad2a4184c1316b3cb22f64ce517b4e6747f25b0a34e41266", size = 98734, upload-time = "2025-07-29T07:42:43.996Z" }, + { url = "https://files.pythonhosted.org/packages/79/19/0dc364391a792b72fbb22becfdeacc5add85cc043cd16986e82152141883/mmh3-5.2.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:eed4bba7ff8a0d37106ba931ab03bdd3915fbb025bcf4e1f0aa02bc8114960c5", size = 106493, upload-time = "2025-07-29T07:42:45.07Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b1/bc8c28e4d6e807bbb051fefe78e1156d7f104b89948742ad310612ce240d/mmh3-5.2.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:1fdb36b940e9261aff0b5177c5b74a36936b902f473180f6c15bde26143681a9", size = 110089, upload-time = "2025-07-29T07:42:46.122Z" }, + { url = "https://files.pythonhosted.org/packages/3b/a2/d20f3f5c95e9c511806686c70d0a15479cc3941c5f322061697af1c1ff70/mmh3-5.2.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7303aab41e97adcf010a09efd8f1403e719e59b7705d5e3cfed3dd7571589290", size = 97571, upload-time = "2025-07-29T07:42:47.18Z" }, + { url = "https://files.pythonhosted.org/packages/7b/23/665296fce4f33488deec39a750ffd245cfc07aafb0e3ef37835f91775d14/mmh3-5.2.0-cp313-cp313-win32.whl", hash = "sha256:03e08c6ebaf666ec1e3d6ea657a2d363bb01effd1a9acfe41f9197decaef0051", size = 40806, upload-time = "2025-07-29T07:42:48.166Z" }, + { url = "https://files.pythonhosted.org/packages/59/b0/92e7103f3b20646e255b699e2d0327ce53a3f250e44367a99dc8be0b7c7a/mmh3-5.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:7fddccd4113e7b736706e17a239a696332360cbaddf25ae75b57ba1acce65081", size = 41600, upload-time = "2025-07-29T07:42:49.371Z" }, + { url = "https://files.pythonhosted.org/packages/99/22/0b2bd679a84574647de538c5b07ccaa435dbccc37815067fe15b90fe8dad/mmh3-5.2.0-cp313-cp313-win_arm64.whl", hash = "sha256:fa0c966ee727aad5406d516375593c5f058c766b21236ab8985693934bb5085b", size = 39349, upload-time = "2025-07-29T07:42:50.268Z" }, + { url = "https://files.pythonhosted.org/packages/f7/ca/a20db059a8a47048aaf550da14a145b56e9c7386fb8280d3ce2962dcebf7/mmh3-5.2.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:e5015f0bb6eb50008bed2d4b1ce0f2a294698a926111e4bb202c0987b4f89078", size = 39209, upload-time = "2025-07-29T07:42:51.559Z" }, + { url = "https://files.pythonhosted.org/packages/98/dd/e5094799d55c7482d814b979a0fd608027d0af1b274bfb4c3ea3e950bfd5/mmh3-5.2.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:e0f3ed828d709f5b82d8bfe14f8856120718ec4bd44a5b26102c3030a1e12501", size = 39843, upload-time = "2025-07-29T07:42:52.536Z" }, + { url = "https://files.pythonhosted.org/packages/f4/6b/7844d7f832c85400e7cc89a1348e4e1fdd38c5a38415bb5726bbb8fcdb6c/mmh3-5.2.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:f35727c5118aba95f0397e18a1a5b8405425581bfe53e821f0fb444cbdc2bc9b", size = 40648, upload-time = "2025-07-29T07:42:53.392Z" }, + { url = "https://files.pythonhosted.org/packages/1f/bf/71f791f48a21ff3190ba5225807cbe4f7223360e96862c376e6e3fb7efa7/mmh3-5.2.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:3bc244802ccab5220008cb712ca1508cb6a12f0eb64ad62997156410579a1770", size = 56164, upload-time = "2025-07-29T07:42:54.267Z" }, + { url = "https://files.pythonhosted.org/packages/70/1f/f87e3d34d83032b4f3f0f528c6d95a98290fcacf019da61343a49dccfd51/mmh3-5.2.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:ff3d50dc3fe8a98059f99b445dfb62792b5d006c5e0b8f03c6de2813b8376110", size = 40692, upload-time = "2025-07-29T07:42:55.234Z" }, + { url = "https://files.pythonhosted.org/packages/a6/e2/db849eaed07117086f3452feca8c839d30d38b830ac59fe1ce65af8be5ad/mmh3-5.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:37a358cc881fe796e099c1db6ce07ff757f088827b4e8467ac52b7a7ffdca647", size = 40068, upload-time = "2025-07-29T07:42:56.158Z" }, + { url = "https://files.pythonhosted.org/packages/df/6b/209af927207af77425b044e32f77f49105a0b05d82ff88af6971d8da4e19/mmh3-5.2.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:b9a87025121d1c448f24f27ff53a5fe7b6ef980574b4a4f11acaabe702420d63", size = 97367, upload-time = "2025-07-29T07:42:57.037Z" }, + { url = "https://files.pythonhosted.org/packages/ca/e0/78adf4104c425606a9ce33fb351f790c76a6c2314969c4a517d1ffc92196/mmh3-5.2.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1ba55d6ca32eeef8b2625e1e4bfc3b3db52bc63014bd7e5df8cc11bf2b036b12", size = 103306, upload-time = "2025-07-29T07:42:58.522Z" }, + { url = "https://files.pythonhosted.org/packages/a3/79/c2b89f91b962658b890104745b1b6c9ce38d50a889f000b469b91eeb1b9e/mmh3-5.2.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c9ff37ba9f15637e424c2ab57a1a590c52897c845b768e4e0a4958084ec87f22", size = 106312, upload-time = "2025-07-29T07:42:59.552Z" }, + { url = "https://files.pythonhosted.org/packages/4b/14/659d4095528b1a209be90934778c5ffe312177d51e365ddcbca2cac2ec7c/mmh3-5.2.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a094319ec0db52a04af9fdc391b4d39a1bc72bc8424b47c4411afb05413a44b5", size = 113135, upload-time = "2025-07-29T07:43:00.745Z" }, + { url = "https://files.pythonhosted.org/packages/8d/6f/cd7734a779389a8a467b5c89a48ff476d6f2576e78216a37551a97e9e42a/mmh3-5.2.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c5584061fd3da584659b13587f26c6cad25a096246a481636d64375d0c1f6c07", size = 120775, upload-time = "2025-07-29T07:43:02.124Z" }, + { url = "https://files.pythonhosted.org/packages/1d/ca/8256e3b96944408940de3f9291d7e38a283b5761fe9614d4808fcf27bd62/mmh3-5.2.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ecbfc0437ddfdced5e7822d1ce4855c9c64f46819d0fdc4482c53f56c707b935", size = 99178, upload-time = "2025-07-29T07:43:03.182Z" }, + { url = "https://files.pythonhosted.org/packages/8a/32/39e2b3cf06b6e2eb042c984dab8680841ac2a0d3ca6e0bea30db1f27b565/mmh3-5.2.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:7b986d506a8e8ea345791897ba5d8ba0d9d8820cd4fc3e52dbe6de19388de2e7", size = 98738, upload-time = "2025-07-29T07:43:04.207Z" }, + { url = "https://files.pythonhosted.org/packages/61/d3/7bbc8e0e8cf65ebbe1b893ffa0467b7ecd1bd07c3bbf6c9db4308ada22ec/mmh3-5.2.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:38d899a156549da8ef6a9f1d6f7ef231228d29f8f69bce2ee12f5fba6d6fd7c5", size = 106510, upload-time = "2025-07-29T07:43:05.656Z" }, + { url = "https://files.pythonhosted.org/packages/10/99/b97e53724b52374e2f3859046f0eb2425192da356cb19784d64bc17bb1cf/mmh3-5.2.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d86651fa45799530885ba4dab3d21144486ed15285e8784181a0ab37a4552384", size = 110053, upload-time = "2025-07-29T07:43:07.204Z" }, + { url = "https://files.pythonhosted.org/packages/ac/62/3688c7d975ed195155671df68788c83fed6f7909b6ec4951724c6860cb97/mmh3-5.2.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c463d7c1c4cfc9d751efeaadd936bbba07b5b0ed81a012b3a9f5a12f0872bd6e", size = 97546, upload-time = "2025-07-29T07:43:08.226Z" }, + { url = "https://files.pythonhosted.org/packages/ca/3b/c6153250f03f71a8b7634cded82939546cdfba02e32f124ff51d52c6f991/mmh3-5.2.0-cp314-cp314-win32.whl", hash = "sha256:bb4fe46bdc6104fbc28db7a6bacb115ee6368ff993366bbd8a2a7f0076e6f0c0", size = 41422, upload-time = "2025-07-29T07:43:09.216Z" }, + { url = "https://files.pythonhosted.org/packages/74/01/a27d98bab083a435c4c07e9d1d720d4c8a578bf4c270bae373760b1022be/mmh3-5.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:7c7f0b342fd06044bedd0b6e72177ddc0076f54fd89ee239447f8b271d919d9b", size = 42135, upload-time = "2025-07-29T07:43:10.183Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c9/dbba5507e95429b8b380e2ba091eff5c20a70a59560934dff0ad8392b8c8/mmh3-5.2.0-cp314-cp314-win_arm64.whl", hash = "sha256:3193752fc05ea72366c2b63ff24b9a190f422e32d75fdeae71087c08fff26115", size = 39879, upload-time = "2025-07-29T07:43:11.106Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d1/c8c0ef839c17258b9de41b84f663574fabcf8ac2007b7416575e0f65ff6e/mmh3-5.2.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:69fc339d7202bea69ef9bd7c39bfdf9fdabc8e6822a01eba62fb43233c1b3932", size = 57696, upload-time = "2025-07-29T07:43:11.989Z" }, + { url = "https://files.pythonhosted.org/packages/2f/55/95e2b9ff201e89f9fe37036037ab61a6c941942b25cdb7b6a9df9b931993/mmh3-5.2.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:12da42c0a55c9d86ab566395324213c319c73ecb0c239fad4726324212b9441c", size = 41421, upload-time = "2025-07-29T07:43:13.269Z" }, + { url = "https://files.pythonhosted.org/packages/77/79/9be23ad0b7001a4b22752e7693be232428ecc0a35068a4ff5c2f14ef8b20/mmh3-5.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f7f9034c7cf05ddfaac8d7a2e63a3c97a840d4615d0a0e65ba8bdf6f8576e3be", size = 40853, upload-time = "2025-07-29T07:43:14.888Z" }, + { url = "https://files.pythonhosted.org/packages/ac/1b/96b32058eda1c1dee8264900c37c359a7325c1f11f5ff14fd2be8e24eff9/mmh3-5.2.0-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:11730eeb16dfcf9674fdea9bb6b8e6dd9b40813b7eb839bc35113649eef38aeb", size = 109694, upload-time = "2025-07-29T07:43:15.816Z" }, + { url = "https://files.pythonhosted.org/packages/8d/6f/a2ae44cd7dad697b6dea48390cbc977b1e5ca58fda09628cbcb2275af064/mmh3-5.2.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:932a6eec1d2e2c3c9e630d10f7128d80e70e2d47fe6b8c7ea5e1afbd98733e65", size = 117438, upload-time = "2025-07-29T07:43:16.865Z" }, + { url = "https://files.pythonhosted.org/packages/a0/08/bfb75451c83f05224a28afeaf3950c7b793c0b71440d571f8e819cfb149a/mmh3-5.2.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ca975c51c5028947bbcfc24966517aac06a01d6c921e30f7c5383c195f87991", size = 120409, upload-time = "2025-07-29T07:43:18.207Z" }, + { url = "https://files.pythonhosted.org/packages/9f/ea/8b118b69b2ff8df568f742387d1a159bc654a0f78741b31437dd047ea28e/mmh3-5.2.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5b0b58215befe0f0e120b828f7645e97719bbba9f23b69e268ed0ac7adde8645", size = 125909, upload-time = "2025-07-29T07:43:19.39Z" }, + { url = "https://files.pythonhosted.org/packages/3e/11/168cc0b6a30650032e351a3b89b8a47382da541993a03af91e1ba2501234/mmh3-5.2.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29c2b9ce61886809d0492a274a5a53047742dea0f703f9c4d5d223c3ea6377d3", size = 135331, upload-time = "2025-07-29T07:43:20.435Z" }, + { url = "https://files.pythonhosted.org/packages/31/05/e3a9849b1c18a7934c64e831492c99e67daebe84a8c2f2c39a7096a830e3/mmh3-5.2.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:a367d4741ac0103f8198c82f429bccb9359f543ca542b06a51f4f0332e8de279", size = 110085, upload-time = "2025-07-29T07:43:21.92Z" }, + { url = "https://files.pythonhosted.org/packages/d9/d5/a96bcc306e3404601418b2a9a370baec92af84204528ba659fdfe34c242f/mmh3-5.2.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:5a5dba98e514fb26241868f6eb90a7f7ca0e039aed779342965ce24ea32ba513", size = 111195, upload-time = "2025-07-29T07:43:23.066Z" }, + { url = "https://files.pythonhosted.org/packages/af/29/0fd49801fec5bff37198684e0849b58e0dab3a2a68382a357cfffb0fafc3/mmh3-5.2.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:941603bfd75a46023807511c1ac2f1b0f39cccc393c15039969806063b27e6db", size = 116919, upload-time = "2025-07-29T07:43:24.178Z" }, + { url = "https://files.pythonhosted.org/packages/2d/04/4f3c32b0a2ed762edca45d8b46568fc3668e34f00fb1e0a3b5451ec1281c/mmh3-5.2.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:132dd943451a7c7546978863d2f5a64977928410782e1a87d583cb60eb89e667", size = 123160, upload-time = "2025-07-29T07:43:25.26Z" }, + { url = "https://files.pythonhosted.org/packages/91/76/3d29eaa38821730633d6a240d36fa8ad2807e9dfd432c12e1a472ed211eb/mmh3-5.2.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f698733a8a494466432d611a8f0d1e026f5286dee051beea4b3c3146817e35d5", size = 110206, upload-time = "2025-07-29T07:43:26.699Z" }, + { url = "https://files.pythonhosted.org/packages/44/1c/ccf35892684d3a408202e296e56843743e0b4fb1629e59432ea88cdb3909/mmh3-5.2.0-cp314-cp314t-win32.whl", hash = "sha256:6d541038b3fc360ec538fc116de87462627944765a6750308118f8b509a8eec7", size = 41970, upload-time = "2025-07-29T07:43:27.666Z" }, + { url = "https://files.pythonhosted.org/packages/75/b2/b9e4f1e5adb5e21eb104588fcee2cd1eaa8308255173481427d5ecc4284e/mmh3-5.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e912b19cf2378f2967d0c08e86ff4c6c360129887f678e27e4dde970d21b3f4d", size = 43063, upload-time = "2025-07-29T07:43:28.582Z" }, + { url = "https://files.pythonhosted.org/packages/6a/fc/0e61d9a4e29c8679356795a40e48f647b4aad58d71bfc969f0f8f56fb912/mmh3-5.2.0-cp314-cp314t-win_arm64.whl", hash = "sha256:e7884931fe5e788163e7b3c511614130c2c59feffdc21112290a194487efb2e9", size = 40455, upload-time = "2025-07-29T07:43:29.563Z" }, ] [[package]] @@ -376,6 +505,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fe/4f/a2c0ed309167ef436674782dfee4a124570ba64299c551e38d3fdaf0a17b/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d383591f3dcbe545f6cc62daaef9c7cdfe0dff0fb9e1c8121101cabe9098cfa6", size = 40507986, upload-time = "2025-02-18T18:53:26.263Z" }, { url = "https://files.pythonhosted.org/packages/27/2e/29bb28a7102a6f71026a9d70d1d61df926887e36ec797f2e6acfd2dd3867/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b4c4156a625f1e35d6c0b2132635a237708944eb41df5fbe7d50f20d20c17832", size = 42087026, upload-time = "2025-02-18T18:53:33.063Z" }, { url = "https://files.pythonhosted.org/packages/16/33/2a67c0f783251106aeeee516f4806161e7b481f7d744d0d643d2f30230a5/pyarrow-19.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:5bd1618ae5e5476b7654c7b55a6364ae87686d4724538c24185bbb2952679960", size = 25250108, upload-time = "2025-02-18T18:53:38.462Z" }, + { url = "https://files.pythonhosted.org/packages/2b/8d/275c58d4b00781bd36579501a259eacc5c6dfb369be4ddeb672ceb551d2d/pyarrow-19.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e45274b20e524ae5c39d7fc1ca2aa923aab494776d2d4b316b49ec7572ca324c", size = 30653552, upload-time = "2025-02-18T18:53:44.357Z" }, + { url = "https://files.pythonhosted.org/packages/a0/9e/e6aca5cc4ef0c7aec5f8db93feb0bde08dbad8c56b9014216205d271101b/pyarrow-19.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:d9dedeaf19097a143ed6da37f04f4051aba353c95ef507764d344229b2b740ae", size = 32103413, upload-time = "2025-02-18T18:53:52.971Z" }, + { url = "https://files.pythonhosted.org/packages/6a/fa/a7033f66e5d4f1308c7eb0dfcd2ccd70f881724eb6fd1776657fdf65458f/pyarrow-19.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6ebfb5171bb5f4a52319344ebbbecc731af3f021e49318c74f33d520d31ae0c4", size = 41134869, upload-time = "2025-02-18T18:53:59.471Z" }, + { url = "https://files.pythonhosted.org/packages/2d/92/34d2569be8e7abdc9d145c98dc410db0071ac579b92ebc30da35f500d630/pyarrow-19.0.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2a21d39fbdb948857f67eacb5bbaaf36802de044ec36fbef7a1c8f0dd3a4ab2", size = 42192626, upload-time = "2025-02-18T18:54:06.062Z" }, + { url = "https://files.pythonhosted.org/packages/0a/1f/80c617b1084fc833804dc3309aa9d8daacd46f9ec8d736df733f15aebe2c/pyarrow-19.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:99bc1bec6d234359743b01e70d4310d0ab240c3d6b0da7e2a93663b0158616f6", size = 40496708, upload-time = "2025-02-18T18:54:12.347Z" }, + { url = "https://files.pythonhosted.org/packages/e6/90/83698fcecf939a611c8d9a78e38e7fed7792dcc4317e29e72cf8135526fb/pyarrow-19.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1b93ef2c93e77c442c979b0d596af45e4665d8b96da598db145b0fec014b9136", size = 42075728, upload-time = "2025-02-18T18:54:19.364Z" }, + { url = "https://files.pythonhosted.org/packages/40/49/2325f5c9e7a1c125c01ba0c509d400b152c972a47958768e4e35e04d13d8/pyarrow-19.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:d9d46e06846a41ba906ab25302cf0fd522f81aa2a85a71021826f34639ad31ef", size = 25242568, upload-time = "2025-02-18T18:54:25.846Z" }, + { url = "https://files.pythonhosted.org/packages/3f/72/135088d995a759d4d916ec4824cb19e066585b4909ebad4ab196177aa825/pyarrow-19.0.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:c0fe3dbbf054a00d1f162fda94ce236a899ca01123a798c561ba307ca38af5f0", size = 30702371, upload-time = "2025-02-18T18:54:30.665Z" }, + { url = "https://files.pythonhosted.org/packages/2e/01/00beeebd33d6bac701f20816a29d2018eba463616bbc07397fdf99ac4ce3/pyarrow-19.0.1-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:96606c3ba57944d128e8a8399da4812f56c7f61de8c647e3470b417f795d0ef9", size = 32116046, upload-time = "2025-02-18T18:54:35.995Z" }, + { url = "https://files.pythonhosted.org/packages/1f/c9/23b1ea718dfe967cbd986d16cf2a31fe59d015874258baae16d7ea0ccabc/pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f04d49a6b64cf24719c080b3c2029a3a5b16417fd5fd7c4041f94233af732f3", size = 41091183, upload-time = "2025-02-18T18:54:42.662Z" }, + { url = "https://files.pythonhosted.org/packages/3a/d4/b4a3aa781a2c715520aa8ab4fe2e7fa49d33a1d4e71c8fc6ab7b5de7a3f8/pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a9137cf7e1640dce4c190551ee69d478f7121b5c6f323553b319cac936395f6", size = 42171896, upload-time = "2025-02-18T18:54:49.808Z" }, + { url = "https://files.pythonhosted.org/packages/23/1b/716d4cd5a3cbc387c6e6745d2704c4b46654ba2668260d25c402626c5ddb/pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:7c1bca1897c28013db5e4c83944a2ab53231f541b9e0c3f4791206d0c0de389a", size = 40464851, upload-time = "2025-02-18T18:54:57.073Z" }, + { url = "https://files.pythonhosted.org/packages/ed/bd/54907846383dcc7ee28772d7e646f6c34276a17da740002a5cefe90f04f7/pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:58d9397b2e273ef76264b45531e9d552d8ec8a6688b7390b5be44c02a37aade8", size = 42085744, upload-time = "2025-02-18T18:55:08.562Z" }, ] [[package]] @@ -443,6 +585,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b0/6a/adf5734ffd52bf86d865093ad70b2ce543415e0e356f6cacabbc0d9ad910/pydantic_core-2.33.2-cp312-cp312-win32.whl", hash = "sha256:9cb1da0f5a471435a7bc7e439b8a728e8b61e59784b2af70d7c169f8dd8ae290", size = 1892628, upload-time = "2025-04-23T18:31:47.819Z" }, { url = "https://files.pythonhosted.org/packages/43/e4/5479fecb3606c1368d496a825d8411e126133c41224c1e7238be58b87d7e/pydantic_core-2.33.2-cp312-cp312-win_amd64.whl", hash = "sha256:f941635f2a3d96b2973e867144fde513665c87f13fe0e193c158ac51bfaaa7b2", size = 1955866, upload-time = "2025-04-23T18:31:49.635Z" }, { url = "https://files.pythonhosted.org/packages/0d/24/8b11e8b3e2be9dd82df4b11408a67c61bb4dc4f8e11b5b0fc888b38118b5/pydantic_core-2.33.2-cp312-cp312-win_arm64.whl", hash = "sha256:cca3868ddfaccfbc4bfb1d608e2ccaaebe0ae628e1416aeb9c4d88c001bb45ab", size = 1888894, upload-time = "2025-04-23T18:31:51.609Z" }, + { url = "https://files.pythonhosted.org/packages/46/8c/99040727b41f56616573a28771b1bfa08a3d3fe74d3d513f01251f79f172/pydantic_core-2.33.2-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1082dd3e2d7109ad8b7da48e1d4710c8d06c253cbc4a27c1cff4fbcaa97a9e3f", size = 2015688, upload-time = "2025-04-23T18:31:53.175Z" }, + { url = "https://files.pythonhosted.org/packages/3a/cc/5999d1eb705a6cefc31f0b4a90e9f7fc400539b1a1030529700cc1b51838/pydantic_core-2.33.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f517ca031dfc037a9c07e748cefd8d96235088b83b4f4ba8939105d20fa1dcd6", size = 1844808, upload-time = "2025-04-23T18:31:54.79Z" }, + { url = "https://files.pythonhosted.org/packages/6f/5e/a0a7b8885c98889a18b6e376f344da1ef323d270b44edf8174d6bce4d622/pydantic_core-2.33.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a9f2c9dd19656823cb8250b0724ee9c60a82f3cdf68a080979d13092a3b0fef", size = 1885580, upload-time = "2025-04-23T18:31:57.393Z" }, + { url = "https://files.pythonhosted.org/packages/3b/2a/953581f343c7d11a304581156618c3f592435523dd9d79865903272c256a/pydantic_core-2.33.2-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2b0a451c263b01acebe51895bfb0e1cc842a5c666efe06cdf13846c7418caa9a", size = 1973859, upload-time = "2025-04-23T18:31:59.065Z" }, + { url = "https://files.pythonhosted.org/packages/e6/55/f1a813904771c03a3f97f676c62cca0c0a4138654107c1b61f19c644868b/pydantic_core-2.33.2-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1ea40a64d23faa25e62a70ad163571c0b342b8bf66d5fa612ac0dec4f069d916", size = 2120810, upload-time = "2025-04-23T18:32:00.78Z" }, + { url = "https://files.pythonhosted.org/packages/aa/c3/053389835a996e18853ba107a63caae0b9deb4a276c6b472931ea9ae6e48/pydantic_core-2.33.2-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0fb2d542b4d66f9470e8065c5469ec676978d625a8b7a363f07d9a501a9cb36a", size = 2676498, upload-time = "2025-04-23T18:32:02.418Z" }, + { url = "https://files.pythonhosted.org/packages/eb/3c/f4abd740877a35abade05e437245b192f9d0ffb48bbbbd708df33d3cda37/pydantic_core-2.33.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9fdac5d6ffa1b5a83bca06ffe7583f5576555e6c8b3a91fbd25ea7780f825f7d", size = 2000611, upload-time = "2025-04-23T18:32:04.152Z" }, + { url = "https://files.pythonhosted.org/packages/59/a7/63ef2fed1837d1121a894d0ce88439fe3e3b3e48c7543b2a4479eb99c2bd/pydantic_core-2.33.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:04a1a413977ab517154eebb2d326da71638271477d6ad87a769102f7c2488c56", size = 2107924, upload-time = "2025-04-23T18:32:06.129Z" }, + { url = "https://files.pythonhosted.org/packages/04/8f/2551964ef045669801675f1cfc3b0d74147f4901c3ffa42be2ddb1f0efc4/pydantic_core-2.33.2-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:c8e7af2f4e0194c22b5b37205bfb293d166a7344a5b0d0eaccebc376546d77d5", size = 2063196, upload-time = "2025-04-23T18:32:08.178Z" }, + { url = "https://files.pythonhosted.org/packages/26/bd/d9602777e77fc6dbb0c7db9ad356e9a985825547dce5ad1d30ee04903918/pydantic_core-2.33.2-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:5c92edd15cd58b3c2d34873597a1e20f13094f59cf88068adb18947df5455b4e", size = 2236389, upload-time = "2025-04-23T18:32:10.242Z" }, + { url = "https://files.pythonhosted.org/packages/42/db/0e950daa7e2230423ab342ae918a794964b053bec24ba8af013fc7c94846/pydantic_core-2.33.2-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:65132b7b4a1c0beded5e057324b7e16e10910c106d43675d9bd87d4f38dde162", size = 2239223, upload-time = "2025-04-23T18:32:12.382Z" }, + { url = "https://files.pythonhosted.org/packages/58/4d/4f937099c545a8a17eb52cb67fe0447fd9a373b348ccfa9a87f141eeb00f/pydantic_core-2.33.2-cp313-cp313-win32.whl", hash = "sha256:52fb90784e0a242bb96ec53f42196a17278855b0f31ac7c3cc6f5c1ec4811849", size = 1900473, upload-time = "2025-04-23T18:32:14.034Z" }, + { url = "https://files.pythonhosted.org/packages/a0/75/4a0a9bac998d78d889def5e4ef2b065acba8cae8c93696906c3a91f310ca/pydantic_core-2.33.2-cp313-cp313-win_amd64.whl", hash = "sha256:c083a3bdd5a93dfe480f1125926afcdbf2917ae714bdb80b36d34318b2bec5d9", size = 1955269, upload-time = "2025-04-23T18:32:15.783Z" }, + { url = "https://files.pythonhosted.org/packages/f9/86/1beda0576969592f1497b4ce8e7bc8cbdf614c352426271b1b10d5f0aa64/pydantic_core-2.33.2-cp313-cp313-win_arm64.whl", hash = "sha256:e80b087132752f6b3d714f041ccf74403799d3b23a72722ea2e6ba2e892555b9", size = 1893921, upload-time = "2025-04-23T18:32:18.473Z" }, + { url = "https://files.pythonhosted.org/packages/a4/7d/e09391c2eebeab681df2b74bfe6c43422fffede8dc74187b2b0bf6fd7571/pydantic_core-2.33.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:61c18fba8e5e9db3ab908620af374db0ac1baa69f0f32df4f61ae23f15e586ac", size = 1806162, upload-time = "2025-04-23T18:32:20.188Z" }, + { url = "https://files.pythonhosted.org/packages/f1/3d/847b6b1fed9f8ed3bb95a9ad04fbd0b212e832d4f0f50ff4d9ee5a9f15cf/pydantic_core-2.33.2-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:95237e53bb015f67b63c91af7518a62a8660376a6a0db19b89acc77a4d6199f5", size = 1981560, upload-time = "2025-04-23T18:32:22.354Z" }, + { url = "https://files.pythonhosted.org/packages/6f/9a/e73262f6c6656262b5fdd723ad90f518f579b7bc8622e43a942eec53c938/pydantic_core-2.33.2-cp313-cp313t-win_amd64.whl", hash = "sha256:c2fc0a768ef76c15ab9238afa6da7f69895bb5d1ee83aeea2e3509af4472d0b9", size = 1935777, upload-time = "2025-04-23T18:32:25.088Z" }, { url = "https://files.pythonhosted.org/packages/30/68/373d55e58b7e83ce371691f6eaa7175e3a24b956c44628eb25d7da007917/pydantic_core-2.33.2-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:5c4aa4e82353f65e548c476b37e64189783aa5384903bfea4f41580f255fddfa", size = 2023982, upload-time = "2025-04-23T18:32:53.14Z" }, { url = "https://files.pythonhosted.org/packages/a4/16/145f54ac08c96a63d8ed6442f9dec17b2773d19920b627b18d4f10a061ea/pydantic_core-2.33.2-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:d946c8bf0d5c24bf4fe333af284c59a19358aa3ec18cb3dc4370080da1e8ad29", size = 1858412, upload-time = "2025-04-23T18:32:55.52Z" }, { url = "https://files.pythonhosted.org/packages/41/b1/c6dc6c3e2de4516c0bb2c46f6a373b91b5660312342a0cf5826e38ad82fa/pydantic_core-2.33.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:87b31b6846e361ef83fedb187bb5b4372d0da3f7e28d85415efa92d6125d6e6d", size = 1892749, upload-time = "2025-04-23T18:32:57.546Z" }, @@ -664,6 +823,14 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/40/cf/e27d7ee61a10f74b17740918e23cbc5bc62011b48282170dc4c66da8ec0f/sqlalchemy-2.0.44-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2e7b5b079055e02d06a4308d0481658e4f06bc7ef211567edc8f7d5dce52018d", size = 3301570, upload-time = "2025-10-10T15:43:48.407Z" }, { url = "https://files.pythonhosted.org/packages/3b/3d/3116a9a7b63e780fb402799b6da227435be878b6846b192f076d2f838654/sqlalchemy-2.0.44-cp312-cp312-win32.whl", hash = "sha256:846541e58b9a81cce7dee8329f352c318de25aa2f2bbe1e31587eb1f057448b4", size = 2103447, upload-time = "2025-10-10T15:03:21.678Z" }, { url = "https://files.pythonhosted.org/packages/25/83/24690e9dfc241e6ab062df82cc0df7f4231c79ba98b273fa496fb3dd78ed/sqlalchemy-2.0.44-cp312-cp312-win_amd64.whl", hash = "sha256:7cbcb47fd66ab294703e1644f78971f6f2f1126424d2b300678f419aa73c7b6e", size = 2130912, upload-time = "2025-10-10T15:03:24.656Z" }, + { url = "https://files.pythonhosted.org/packages/45/d3/c67077a2249fdb455246e6853166360054c331db4613cda3e31ab1cadbef/sqlalchemy-2.0.44-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ff486e183d151e51b1d694c7aa1695747599bb00b9f5f604092b54b74c64a8e1", size = 2135479, upload-time = "2025-10-10T16:03:37.671Z" }, + { url = "https://files.pythonhosted.org/packages/2b/91/eabd0688330d6fd114f5f12c4f89b0d02929f525e6bf7ff80aa17ca802af/sqlalchemy-2.0.44-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0b1af8392eb27b372ddb783b317dea0f650241cea5bd29199b22235299ca2e45", size = 2123212, upload-time = "2025-10-10T16:03:41.755Z" }, + { url = "https://files.pythonhosted.org/packages/b0/bb/43e246cfe0e81c018076a16036d9b548c4cc649de241fa27d8d9ca6f85ab/sqlalchemy-2.0.44-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2b61188657e3a2b9ac4e8f04d6cf8e51046e28175f79464c67f2fd35bceb0976", size = 3255353, upload-time = "2025-10-10T15:35:31.221Z" }, + { url = "https://files.pythonhosted.org/packages/b9/96/c6105ed9a880abe346b64d3b6ddef269ddfcab04f7f3d90a0bf3c5a88e82/sqlalchemy-2.0.44-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b87e7b91a5d5973dda5f00cd61ef72ad75a1db73a386b62877d4875a8840959c", size = 3260222, upload-time = "2025-10-10T15:43:50.124Z" }, + { url = "https://files.pythonhosted.org/packages/44/16/1857e35a47155b5ad927272fee81ae49d398959cb749edca6eaa399b582f/sqlalchemy-2.0.44-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:15f3326f7f0b2bfe406ee562e17f43f36e16167af99c4c0df61db668de20002d", size = 3189614, upload-time = "2025-10-10T15:35:32.578Z" }, + { url = "https://files.pythonhosted.org/packages/88/ee/4afb39a8ee4fc786e2d716c20ab87b5b1fb33d4ac4129a1aaa574ae8a585/sqlalchemy-2.0.44-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1e77faf6ff919aa8cd63f1c4e561cac1d9a454a191bb864d5dd5e545935e5a40", size = 3226248, upload-time = "2025-10-10T15:43:51.862Z" }, + { url = "https://files.pythonhosted.org/packages/32/d5/0e66097fc64fa266f29a7963296b40a80d6a997b7ac13806183700676f86/sqlalchemy-2.0.44-cp313-cp313-win32.whl", hash = "sha256:ee51625c2d51f8baadf2829fae817ad0b66b140573939dd69284d2ba3553ae73", size = 2101275, upload-time = "2025-10-10T15:03:26.096Z" }, + { url = "https://files.pythonhosted.org/packages/03/51/665617fe4f8c6450f42a6d8d69243f9420f5677395572c2fe9d21b493b7b/sqlalchemy-2.0.44-cp313-cp313-win_amd64.whl", hash = "sha256:c1c80faaee1a6c3428cecf40d16a2365bcf56c424c92c2b6f0f9ad204b899e9e", size = 2127901, upload-time = "2025-10-10T15:03:27.548Z" }, { url = "https://files.pythonhosted.org/packages/9c/5e/6a29fa884d9fb7ddadf6b69490a9d45fded3b38541713010dad16b77d015/sqlalchemy-2.0.44-py3-none-any.whl", hash = "sha256:19de7ca1246fbef9f9d1bff8f1ab25641569df226364a0e40457dc5457c54b05", size = 1928718, upload-time = "2025-10-10T15:29:45.32Z" }, ] @@ -710,6 +877,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/a1/4d6865da6a71c603cfe6ad0e6556c73c76548557a8d658f9e3b142df245f/tomli-2.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7b0882799624980785240ab732537fcfc372601015c00f7fc367c55308c186f6", size = 250296, upload-time = "2025-10-08T22:01:14.614Z" }, { url = "https://files.pythonhosted.org/packages/a0/b7/a7a7042715d55c9ba6e8b196d65d2cb662578b4d8cd17d882d45322b0d78/tomli-2.3.0-cp312-cp312-win32.whl", hash = "sha256:ff72b71b5d10d22ecb084d345fc26f42b5143c5533db5e2eaba7d2d335358876", size = 97124, upload-time = "2025-10-08T22:01:15.629Z" }, { url = "https://files.pythonhosted.org/packages/06/1e/f22f100db15a68b520664eb3328fb0ae4e90530887928558112c8d1f4515/tomli-2.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:1cb4ed918939151a03f33d4242ccd0aa5f11b3547d0cf30f7c74a408a5b99878", size = 107698, upload-time = "2025-10-08T22:01:16.51Z" }, + { url = "https://files.pythonhosted.org/packages/89/48/06ee6eabe4fdd9ecd48bf488f4ac783844fd777f547b8d1b61c11939974e/tomli-2.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5192f562738228945d7b13d4930baffda67b69425a7f0da96d360b0a3888136b", size = 154819, upload-time = "2025-10-08T22:01:17.964Z" }, + { url = "https://files.pythonhosted.org/packages/f1/01/88793757d54d8937015c75dcdfb673c65471945f6be98e6a0410fba167ed/tomli-2.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:be71c93a63d738597996be9528f4abe628d1adf5e6eb11607bc8fe1a510b5dae", size = 148766, upload-time = "2025-10-08T22:01:18.959Z" }, + { url = "https://files.pythonhosted.org/packages/42/17/5e2c956f0144b812e7e107f94f1cc54af734eb17b5191c0bbfb72de5e93e/tomli-2.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4665508bcbac83a31ff8ab08f424b665200c0e1e645d2bd9ab3d3e557b6185b", size = 240771, upload-time = "2025-10-08T22:01:20.106Z" }, + { url = "https://files.pythonhosted.org/packages/d5/f4/0fbd014909748706c01d16824eadb0307115f9562a15cbb012cd9b3512c5/tomli-2.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4021923f97266babc6ccab9f5068642a0095faa0a51a246a6a02fccbb3514eaf", size = 248586, upload-time = "2025-10-08T22:01:21.164Z" }, + { url = "https://files.pythonhosted.org/packages/30/77/fed85e114bde5e81ecf9bc5da0cc69f2914b38f4708c80ae67d0c10180c5/tomli-2.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4ea38c40145a357d513bffad0ed869f13c1773716cf71ccaa83b0fa0cc4e42f", size = 244792, upload-time = "2025-10-08T22:01:22.417Z" }, + { url = "https://files.pythonhosted.org/packages/55/92/afed3d497f7c186dc71e6ee6d4fcb0acfa5f7d0a1a2878f8beae379ae0cc/tomli-2.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ad805ea85eda330dbad64c7ea7a4556259665bdf9d2672f5dccc740eb9d3ca05", size = 248909, upload-time = "2025-10-08T22:01:23.859Z" }, + { url = "https://files.pythonhosted.org/packages/f8/84/ef50c51b5a9472e7265ce1ffc7f24cd4023d289e109f669bdb1553f6a7c2/tomli-2.3.0-cp313-cp313-win32.whl", hash = "sha256:97d5eec30149fd3294270e889b4234023f2c69747e555a27bd708828353ab606", size = 96946, upload-time = "2025-10-08T22:01:24.893Z" }, + { url = "https://files.pythonhosted.org/packages/b2/b7/718cd1da0884f281f95ccfa3a6cc572d30053cba64603f79d431d3c9b61b/tomli-2.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:0c95ca56fbe89e065c6ead5b593ee64b84a26fca063b5d71a1122bf26e533999", size = 107705, upload-time = "2025-10-08T22:01:26.153Z" }, + { url = "https://files.pythonhosted.org/packages/19/94/aeafa14a52e16163008060506fcb6aa1949d13548d13752171a755c65611/tomli-2.3.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:cebc6fe843e0733ee827a282aca4999b596241195f43b4cc371d64fc6639da9e", size = 154244, upload-time = "2025-10-08T22:01:27.06Z" }, + { url = "https://files.pythonhosted.org/packages/db/e4/1e58409aa78eefa47ccd19779fc6f36787edbe7d4cd330eeeedb33a4515b/tomli-2.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4c2ef0244c75aba9355561272009d934953817c49f47d768070c3c94355c2aa3", size = 148637, upload-time = "2025-10-08T22:01:28.059Z" }, + { url = "https://files.pythonhosted.org/packages/26/b6/d1eccb62f665e44359226811064596dd6a366ea1f985839c566cd61525ae/tomli-2.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c22a8bf253bacc0cf11f35ad9808b6cb75ada2631c2d97c971122583b129afbc", size = 241925, upload-time = "2025-10-08T22:01:29.066Z" }, + { url = "https://files.pythonhosted.org/packages/70/91/7cdab9a03e6d3d2bb11beae108da5bdc1c34bdeb06e21163482544ddcc90/tomli-2.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0eea8cc5c5e9f89c9b90c4896a8deefc74f518db5927d0e0e8d4a80953d774d0", size = 249045, upload-time = "2025-10-08T22:01:31.98Z" }, + { url = "https://files.pythonhosted.org/packages/15/1b/8c26874ed1f6e4f1fcfeb868db8a794cbe9f227299402db58cfcc858766c/tomli-2.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b74a0e59ec5d15127acdabd75ea17726ac4c5178ae51b85bfe39c4f8a278e879", size = 245835, upload-time = "2025-10-08T22:01:32.989Z" }, + { url = "https://files.pythonhosted.org/packages/fd/42/8e3c6a9a4b1a1360c1a2a39f0b972cef2cc9ebd56025168c4137192a9321/tomli-2.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b5870b50c9db823c595983571d1296a6ff3e1b88f734a4c8f6fc6188397de005", size = 253109, upload-time = "2025-10-08T22:01:34.052Z" }, + { url = "https://files.pythonhosted.org/packages/22/0c/b4da635000a71b5f80130937eeac12e686eefb376b8dee113b4a582bba42/tomli-2.3.0-cp314-cp314-win32.whl", hash = "sha256:feb0dacc61170ed7ab602d3d972a58f14ee3ee60494292d384649a3dc38ef463", size = 97930, upload-time = "2025-10-08T22:01:35.082Z" }, + { url = "https://files.pythonhosted.org/packages/b9/74/cb1abc870a418ae99cd5c9547d6bce30701a954e0e721821df483ef7223c/tomli-2.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:b273fcbd7fc64dc3600c098e39136522650c49bca95df2d11cf3b626422392c8", size = 107964, upload-time = "2025-10-08T22:01:36.057Z" }, + { url = "https://files.pythonhosted.org/packages/54/78/5c46fff6432a712af9f792944f4fcd7067d8823157949f4e40c56b8b3c83/tomli-2.3.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:940d56ee0410fa17ee1f12b817b37a4d4e4dc4d27340863cc67236c74f582e77", size = 163065, upload-time = "2025-10-08T22:01:37.27Z" }, + { url = "https://files.pythonhosted.org/packages/39/67/f85d9bd23182f45eca8939cd2bc7050e1f90c41f4a2ecbbd5963a1d1c486/tomli-2.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f85209946d1fe94416debbb88d00eb92ce9cd5266775424ff81bc959e001acaf", size = 159088, upload-time = "2025-10-08T22:01:38.235Z" }, + { url = "https://files.pythonhosted.org/packages/26/5a/4b546a0405b9cc0659b399f12b6adb750757baf04250b148d3c5059fc4eb/tomli-2.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a56212bdcce682e56b0aaf79e869ba5d15a6163f88d5451cbde388d48b13f530", size = 268193, upload-time = "2025-10-08T22:01:39.712Z" }, + { url = "https://files.pythonhosted.org/packages/42/4f/2c12a72ae22cf7b59a7fe75b3465b7aba40ea9145d026ba41cb382075b0e/tomli-2.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5f3ffd1e098dfc032d4d3af5c0ac64f6d286d98bc148698356847b80fa4de1b", size = 275488, upload-time = "2025-10-08T22:01:40.773Z" }, + { url = "https://files.pythonhosted.org/packages/92/04/a038d65dbe160c3aa5a624e93ad98111090f6804027d474ba9c37c8ae186/tomli-2.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e01decd096b1530d97d5d85cb4dff4af2d8347bd35686654a004f8dea20fc67", size = 272669, upload-time = "2025-10-08T22:01:41.824Z" }, + { url = "https://files.pythonhosted.org/packages/be/2f/8b7c60a9d1612a7cbc39ffcca4f21a73bf368a80fc25bccf8253e2563267/tomli-2.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8a35dd0e643bb2610f156cca8db95d213a90015c11fee76c946aa62b7ae7e02f", size = 279709, upload-time = "2025-10-08T22:01:43.177Z" }, + { url = "https://files.pythonhosted.org/packages/7e/46/cc36c679f09f27ded940281c38607716c86cf8ba4a518d524e349c8b4874/tomli-2.3.0-cp314-cp314t-win32.whl", hash = "sha256:a1f7f282fe248311650081faafa5f4732bdbfef5d45fe3f2e702fbc6f2d496e0", size = 107563, upload-time = "2025-10-08T22:01:44.233Z" }, + { url = "https://files.pythonhosted.org/packages/84/ff/426ca8683cf7b753614480484f6437f568fd2fda2edbdf57a2d3d8b27a0b/tomli-2.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:70a251f8d4ba2d9ac2542eecf008b3c8a9fc5c3f9f02c56a9d7952612be2fdba", size = 119756, upload-time = "2025-10-08T22:01:45.234Z" }, { url = "https://files.pythonhosted.org/packages/77/b8/0135fadc89e73be292b473cb820b4f5a08197779206b33191e801feeae40/tomli-2.3.0-py3-none-any.whl", hash = "sha256:e95b1af3c5b07d9e643909b5abbec77cd9f1217e6d0bca72b0234736b9fb1f1b", size = 14408, upload-time = "2025-10-08T22:01:46.04Z" }, ] From 3c4254641ff9c5c2952447d221e985e9293b22b3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 6 Feb 2026 08:53:39 +0800 Subject: [PATCH 196/891] chore(deps): Bump time from 0.3.44 to 0.3.47 (#2113) Bumps [time](https://github.com/time-rs/time) from 0.3.44 to 0.3.47.
Release notes

Sourced from time's releases.

v0.3.47

See the changelog for details.

v0.3.46

See the changelog for details.

v0.3.45

See the changelog for details.

Changelog

Sourced from time's changelog.

0.3.47 [2026-02-05]

Security

  • The possibility of a stack exhaustion denial of service attack when parsing RFC 2822 has been eliminated. Previously, it was possible to craft input that would cause unbounded recursion. Now, the depth of the recursion is tracked, causing an error to be returned if it exceeds a reasonable limit.

    This attack vector requires parsing user-provided input, with any type, using the RFC 2822 format.

Compatibility

  • Attempting to format a value with a well-known format (i.e. RFC 3339, RFC 2822, or ISO 8601) will error at compile time if the type being formatted does not provide sufficient information. This would previously fail at runtime. Similarly, attempting to format a value with ISO 8601 that is only configured for parsing (i.e. Iso8601::PARSING) will error at compile time.

Added

  • Builder methods for format description modifiers, eliminating the need for verbose initialization when done manually.
  • date!(2026-W01-2) is now supported. Previously, a space was required between W and 01.
  • [end] now has a trailing_input modifier which can either be prohibit (the default) or discard. When it is discard, all remaining input is ignored. Note that if there are components after [end], they will still attempt to be parsed, likely resulting in an error.

Changed

  • More performance gains when parsing.

Fixed

  • If manually formatting a value, the number of bytes written was one short for some components. This has been fixed such that the number of bytes written is always correct.
  • The possibility of integer overflow when parsing an owned format description has been effectively eliminated. This would previously wrap when overflow checks were disabled. Instead of storing the depth as u8, it is stored as u32. This would require multiple gigabytes of nested input to overflow, at which point we've got other problems and trivial mitigations are available by downstream users.

0.3.46 [2026-01-23]

Added

  • All possible panics are now documented for the relevant methods.
  • The need to use #[serde(default)] when using custom serde formats is documented. This applies only when deserializing an Option<T>.
  • Duration::nanoseconds_i128 has been made public, mirroring std::time::Duration::from_nanos_u128.

... (truncated)

Commits
  • d5144cd v0.3.47 release
  • f6206b0 Guard against integer overflow in release mode
  • 1c63dc7 Avoid denial of service when parsing Rfc2822
  • 5940df6 Add builder methods to avoid verbose construction
  • 00881a4 Manually format macros everywhere
  • bb723b6 Add trailing_input modifier to end
  • 31c4f8e Permit W12 in date! macro
  • 490a17b Mark error paths in well-known formats as cold
  • 6cb1896 Optimize Rfc2822 parsing
  • 6d264d5 Remove erroneous #[inline(never)] attributes
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=time&package-manager=cargo&previous-version=0.3.44&new-version=0.3.47)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/iceberg-rust/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index ed3fd34730..85ce133391 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4429,9 +4429,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.1.0" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" +checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" [[package]] name = "num-integer" @@ -6795,30 +6795,30 @@ dependencies = [ [[package]] name = "time" -version = "0.3.44" +version = "0.3.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", - "serde", + "serde_core", "time-core", "time-macros", ] [[package]] name = "time-core" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" [[package]] name = "time-macros" -version = "0.2.24" +version = "0.2.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" dependencies = [ "num-conv", "time-core", From 50b3740ed8a63356b146c01cf8e8f4b2b46e0d74 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 6 Feb 2026 09:53:00 +0800 Subject: [PATCH 197/891] chore(deps): Bump time from 0.3.44 to 0.3.47 in /bindings/python (#2112) Bumps [time](https://github.com/time-rs/time) from 0.3.44 to 0.3.47.
Release notes

Sourced from time's releases.

v0.3.47

See the changelog for details.

v0.3.46

See the changelog for details.

v0.3.45

See the changelog for details.

Changelog

Sourced from time's changelog.

0.3.47 [2026-02-05]

Security

  • The possibility of a stack exhaustion denial of service attack when parsing RFC 2822 has been eliminated. Previously, it was possible to craft input that would cause unbounded recursion. Now, the depth of the recursion is tracked, causing an error to be returned if it exceeds a reasonable limit.

    This attack vector requires parsing user-provided input, with any type, using the RFC 2822 format.

Compatibility

  • Attempting to format a value with a well-known format (i.e. RFC 3339, RFC 2822, or ISO 8601) will error at compile time if the type being formatted does not provide sufficient information. This would previously fail at runtime. Similarly, attempting to format a value with ISO 8601 that is only configured for parsing (i.e. Iso8601::PARSING) will error at compile time.

Added

  • Builder methods for format description modifiers, eliminating the need for verbose initialization when done manually.
  • date!(2026-W01-2) is now supported. Previously, a space was required between W and 01.
  • [end] now has a trailing_input modifier which can either be prohibit (the default) or discard. When it is discard, all remaining input is ignored. Note that if there are components after [end], they will still attempt to be parsed, likely resulting in an error.

Changed

  • More performance gains when parsing.

Fixed

  • If manually formatting a value, the number of bytes written was one short for some components. This has been fixed such that the number of bytes written is always correct.
  • The possibility of integer overflow when parsing an owned format description has been effectively eliminated. This would previously wrap when overflow checks were disabled. Instead of storing the depth as u8, it is stored as u32. This would require multiple gigabytes of nested input to overflow, at which point we've got other problems and trivial mitigations are available by downstream users.

0.3.46 [2026-01-23]

Added

  • All possible panics are now documented for the relevant methods.
  • The need to use #[serde(default)] when using custom serde formats is documented. This applies only when deserializing an Option<T>.
  • Duration::nanoseconds_i128 has been made public, mirroring std::time::Duration::from_nanos_u128.

... (truncated)

Commits
  • d5144cd v0.3.47 release
  • f6206b0 Guard against integer overflow in release mode
  • 1c63dc7 Avoid denial of service when parsing Rfc2822
  • 5940df6 Add builder methods to avoid verbose construction
  • 00881a4 Manually format macros everywhere
  • bb723b6 Add trailing_input modifier to end
  • 31c4f8e Permit W12 in date! macro
  • 490a17b Mark error paths in well-known formats as cold
  • 6cb1896 Optimize Rfc2822 parsing
  • 6d264d5 Remove erroneous #[inline(never)] attributes
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=time&package-manager=cargo&previous-version=0.3.44&new-version=0.3.47)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/iceberg-rust/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- bindings/python/Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index dd52ff2be6..e2a8650c1d 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -2835,9 +2835,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.1.0" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" +checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" [[package]] name = "num-integer" @@ -4149,30 +4149,30 @@ dependencies = [ [[package]] name = "time" -version = "0.3.44" +version = "0.3.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", - "serde", + "serde_core", "time-core", "time-macros", ] [[package]] name = "time-core" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" [[package]] name = "time-macros" -version = "0.2.24" +version = "0.2.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" dependencies = [ "num-conv", "time-core", From 77fad54dcdd74abeadc36e778bf9b2664e096a09 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 9 Feb 2026 10:05:11 +0800 Subject: [PATCH 198/891] chore(deps): Bump fs-err from 3.2.0 to 3.3.0 (#2126) Bumps [fs-err](https://github.com/andrewhickman/fs-err) from 3.2.0 to 3.3.0.
Changelog

Sourced from fs-err's changelog.

fs-err Changelog

3.3.2

  • Cleanup rustdoc links to make them clickable in rust-analyzer (#88)
  • Add wrappers for several os::unix::fs functions (#87):
    • std::os::unix::fs::FileExt::read_exact_at
    • std::os::unix::fs::FileExt::write_all_at
    • std::os::unix::fs::chroot
    • std::os::unix::fs::chown
    • std::os::unix::fs::lchown
  • Replace '/' operator in SPDX license expression with 'OR' for readability (#86)

3.2.2

  • Add wrappers for File::set_modified and File::set_times (#84)

3.2.1

  • Rename parameters of symlink, soft_link and hard_link functions to match std (#83)
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fs-err&package-manager=cargo&previous-version=3.2.0&new-version=3.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 85ce133391..ee980e0613 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2833,9 +2833,9 @@ checksum = "28dd6caf6059519a65843af8fe2a3ae298b14b80179855aeb4adc2c1934ee619" [[package]] name = "fs-err" -version = "3.2.0" +version = "3.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62d91fd049c123429b018c47887d3f75a265540dd3c30ba9cb7bae9197edb03a" +checksum = "73fde052dbfc920003cfd2c8e2c6e6d4cc7c1091538c3a24226cec0665ab08c0" dependencies = [ "autocfg", ] From e007e0716d084d3919fc976176a8262a3089f26a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 9 Feb 2026 10:21:32 +0800 Subject: [PATCH 199/891] chore(deps): Bump the arrow-parquet group with 6 updates (#2125) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps the arrow-parquet group with 6 updates: | Package | From | To | | --- | --- | --- | | [arrow-array](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-buffer](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-ord](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-schema](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-select](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-string](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | Updates `arrow-array` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-array's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-array's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-buffer` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-buffer's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-buffer's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-ord` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-ord's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-ord's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-schema` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-schema's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-schema's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-select` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-select's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-select's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-string` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-string's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-string's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- Cargo.lock | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index ee980e0613..054de94e68 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -258,9 +258,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65ca404ea6191e06bf30956394173337fa9c35f445bd447fe6c21ab944e1a23c" +checksum = "4c8955af33b25f3b175ee10af580577280b4bd01f7e823d94c7cdef7cf8c9aef" dependencies = [ "ahash 0.8.12", "arrow-buffer", @@ -277,9 +277,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36356383099be0151dacc4245309895f16ba7917d79bdb71a7148659c9206c56" +checksum = "c697ddca96183182f35b3a18e50b9110b11e916d7b7799cbfd4d34662f2c56c2" dependencies = [ "bytes", "half", @@ -326,9 +326,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf87f4ff5fc13290aa47e499a8b669a82c5977c6a1fedce22c7f542c1fd5a597" +checksum = "1fdd994a9d28e6365aa78e15da3f3950c0fdcea6b963a12fa1c391afb637b304" dependencies = [ "arrow-buffer", "arrow-schema", @@ -379,9 +379,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c4e0530272ca755d6814218dffd04425c5b7854b87fa741d5ff848bf50aa39" +checksum = "f7d8f1870e03d4cbed632959498bcc84083b5a24bded52905ae1695bd29da45b" dependencies = [ "arrow-array", "arrow-buffer", @@ -405,9 +405,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6bb63203e8e0e54b288d0d8043ca8fa1013820822a27692ef1b78a977d879f2c" +checksum = "8c872d36b7bf2a6a6a2b40de9156265f0242910791db366a2c17476ba8330d68" dependencies = [ "serde_core", "serde_json", @@ -415,9 +415,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c96d8a1c180b44ecf2e66c9a2f2bbcb8b1b6f14e165ce46ac8bde211a363411b" +checksum = "68bf3e3efbd1278f770d67e5dc410257300b161b93baedb3aae836144edcaf4b" dependencies = [ "ahash 0.8.12", "arrow-array", @@ -429,9 +429,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8ad6a81add9d3ea30bf8374ee8329992c7fd246ffd8b7e2f48a3cea5aa0cc9a" +checksum = "85e968097061b3c0e9fe3079cf2e703e487890700546b5b0647f60fca1b5a8d8" dependencies = [ "arrow-array", "arrow-buffer", From 921b8fdc6a2bb7e157d218fd1db2e185a0b562f6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 9 Feb 2026 10:31:03 +0800 Subject: [PATCH 200/891] chore(deps): Bump crate-ci/typos from 1.42.3 to 1.43.3 (#2124) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.42.3 to 1.43.3.
Release notes

Sourced from crate-ci/typos's releases.

v1.43.3

[1.43.3] - 2026-02-06

Fixes

  • (action) Adjust how typos are reported to github

v1.43.2

[1.43.2] - 2026-02-05

Fixes

  • Don't correct certifi in Python

v1.43.1

[1.43.1] - 2026-02-03

Fixes

  • Don't correct consts

v1.43.0

[1.43.0] - 2026-02-02

Features

Changelog

Sourced from crate-ci/typos's changelog.

[1.43.3] - 2026-02-06

Fixes

  • (action) Adjust how typos are reported to github

[1.43.2] - 2026-02-05

Fixes

  • Don't correct certifi in Python

[1.43.1] - 2026-02-03

Fixes

  • Don't correct consts

[1.43.0] - 2026-02-02

Compatibility

  • Bumped MSRV to 1.91

Features

Commits
  • 9066e99 chore: Release
  • 3bebc2a docs: Update changelog
  • adf71f7 Merge pull request #1496 from adangel/fix-action-annotations
  • ad3053d chore: Release
  • a23d8be docs: Update changelog
  • 63b278c Merge pull request #1497 from epage/certifi
  • 5775fa1 feat(config): Don't correct certifi in Python
  • 67e0ce7 fix(gh): Fix col offset (1-based)
  • 7f369e0 fix(gh): Relativize file paths against GITHUB_WORKSPACE
  • 3141b83 docs: Add msrv entry
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.42.3&new-version=1.43.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index dd258fb844..13ceeaf236 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.42.3 + uses: crate-ci/typos@v1.43.3 From 5cec0f36084a7cab2da2bc52abe27e7594f5d574 Mon Sep 17 00:00:00 2001 From: blackmwk Date: Mon, 9 Feb 2026 18:50:27 +0800 Subject: [PATCH 201/891] Use uv instead of pip for python packages (#2129) --- dev/docker-compose.yaml | 8 ++++++-- dev/spark/provision.py | 7 +++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/dev/docker-compose.yaml b/dev/docker-compose.yaml index 2f0d08fbd9..21920c9ce6 100644 --- a/dev/docker-compose.yaml +++ b/dev/docker-compose.yaml @@ -188,18 +188,22 @@ services: # Provision service - creates test data via Spark provision: - image: python:3.12-slim + image: ghcr.io/astral-sh/uv:python3.12-bookworm-slim networks: iceberg_test: depends_on: spark-iceberg: condition: service_healthy - entrypoint: ["/bin/sh", "-c", "pip install -q 'pyspark[connect]==4.0.1' && python3 /opt/spark/provision.py && touch /tmp/provision_complete && tail -f /dev/null"] + entrypoint: ["/bin/sh", "-c", "uv run /opt/spark/provision.py && touch /tmp/provision_complete && tail -f /dev/null"] volumes: - ./spark/provision.py:/opt/spark/provision.py:ro + - uv-cache:/root/.cache/uv healthcheck: test: ["CMD-SHELL", "[ -f /tmp/provision_complete ]"] interval: 2s timeout: 2s retries: 90 start_period: 20s + +volumes: + uv-cache: diff --git a/dev/spark/provision.py b/dev/spark/provision.py index c53a1dd842..40f9ba0f38 100644 --- a/dev/spark/provision.py +++ b/dev/spark/provision.py @@ -15,6 +15,13 @@ # specific language governing permissions and limitations # under the License. +# /// script +# requires-python = ">=3.12" +# dependencies = [ +# "pyspark[connect]==4.0.1", +# ] +# /// + from pyspark.sql import SparkSession from pyspark.sql.functions import current_date, date_add, expr From 7c2d5d199e50eb9a142bdefe632ac6109fcb2006 Mon Sep 17 00:00:00 2001 From: Shawn Chang Date: Wed, 11 Feb 2026 19:03:00 -0800 Subject: [PATCH 202/891] refactor(storage): Reorganize storage code into a new module (#2109) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? - Add a new module `storage` within `io` and put traits there ## Are these changes tested? Relying on the existing tests --- crates/iceberg/src/io/file_io.rs | 3 +-- crates/iceberg/src/io/mod.rs | 10 +--------- crates/iceberg/src/io/{ => storage}/config/azdls.rs | 0 crates/iceberg/src/io/{ => storage}/config/gcs.rs | 0 crates/iceberg/src/io/{ => storage}/config/mod.rs | 0 crates/iceberg/src/io/{ => storage}/config/oss.rs | 0 crates/iceberg/src/io/{ => storage}/config/s3.rs | 0 crates/iceberg/src/io/{ => storage}/local_fs.rs | 2 +- crates/iceberg/src/io/{ => storage}/memory.rs | 2 +- crates/iceberg/src/io/{storage.rs => storage/mod.rs} | 12 +++++++++++- crates/iceberg/src/io/{ => storage}/opendal/azdls.rs | 2 +- crates/iceberg/src/io/{ => storage}/opendal/fs.rs | 0 crates/iceberg/src/io/{ => storage}/opendal/gcs.rs | 5 ++--- .../iceberg/src/io/{ => storage}/opendal/memory.rs | 0 crates/iceberg/src/io/{ => storage}/opendal/mod.rs | 2 +- crates/iceberg/src/io/{ => storage}/opendal/oss.rs | 2 +- crates/iceberg/src/io/{ => storage}/opendal/s3.rs | 5 ++--- 17 files changed, 22 insertions(+), 23 deletions(-) rename crates/iceberg/src/io/{ => storage}/config/azdls.rs (100%) rename crates/iceberg/src/io/{ => storage}/config/gcs.rs (100%) rename crates/iceberg/src/io/{ => storage}/config/mod.rs (100%) rename crates/iceberg/src/io/{ => storage}/config/oss.rs (100%) rename crates/iceberg/src/io/{ => storage}/config/s3.rs (100%) rename crates/iceberg/src/io/{ => storage}/local_fs.rs (99%) rename crates/iceberg/src/io/{ => storage}/memory.rs (99%) rename crates/iceberg/src/io/{storage.rs => storage/mod.rs} (93%) rename crates/iceberg/src/io/{ => storage}/opendal/azdls.rs (99%) rename crates/iceberg/src/io/{ => storage}/opendal/fs.rs (100%) rename crates/iceberg/src/io/{ => storage}/opendal/gcs.rs (96%) rename crates/iceberg/src/io/{ => storage}/opendal/memory.rs (100%) rename crates/iceberg/src/io/{ => storage}/opendal/mod.rs (99%) rename crates/iceberg/src/io/{ => storage}/opendal/oss.rs (96%) rename crates/iceberg/src/io/{ => storage}/opendal/s3.rs (98%) diff --git a/crates/iceberg/src/io/file_io.rs b/crates/iceberg/src/io/file_io.rs index 1ad71d8531..267f295369 100644 --- a/crates/iceberg/src/io/file_io.rs +++ b/crates/iceberg/src/io/file_io.rs @@ -23,8 +23,7 @@ use std::sync::Arc; use bytes::Bytes; use url::Url; -use super::opendal::OpenDalStorage; -use super::storage::Storage; +use super::storage::{OpenDalStorage, Storage}; use crate::{Error, ErrorKind, Result}; /// FileIO implementation, used to manipulate files in underlying storage. diff --git a/crates/iceberg/src/io/mod.rs b/crates/iceberg/src/io/mod.rs index 5f65a15fc3..07f40d104e 100644 --- a/crates/iceberg/src/io/mod.rs +++ b/crates/iceberg/src/io/mod.rs @@ -66,19 +66,11 @@ //! - `new_input`: Create input file for reading. //! - `new_output`: Create output file for writing. -mod config; mod file_io; -mod local_fs; -mod memory; -mod opendal; mod storage; -pub use config::*; pub use file_io::*; -#[cfg(feature = "storage-s3")] -pub use opendal::CustomAwsCredentialLoader; -pub use opendal::{OpenDalStorage, OpenDalStorageFactory}; -pub use storage::{Storage, StorageConfig, StorageFactory}; +pub use storage::*; pub(crate) mod object_cache; diff --git a/crates/iceberg/src/io/config/azdls.rs b/crates/iceberg/src/io/storage/config/azdls.rs similarity index 100% rename from crates/iceberg/src/io/config/azdls.rs rename to crates/iceberg/src/io/storage/config/azdls.rs diff --git a/crates/iceberg/src/io/config/gcs.rs b/crates/iceberg/src/io/storage/config/gcs.rs similarity index 100% rename from crates/iceberg/src/io/config/gcs.rs rename to crates/iceberg/src/io/storage/config/gcs.rs diff --git a/crates/iceberg/src/io/config/mod.rs b/crates/iceberg/src/io/storage/config/mod.rs similarity index 100% rename from crates/iceberg/src/io/config/mod.rs rename to crates/iceberg/src/io/storage/config/mod.rs diff --git a/crates/iceberg/src/io/config/oss.rs b/crates/iceberg/src/io/storage/config/oss.rs similarity index 100% rename from crates/iceberg/src/io/config/oss.rs rename to crates/iceberg/src/io/storage/config/oss.rs diff --git a/crates/iceberg/src/io/config/s3.rs b/crates/iceberg/src/io/storage/config/s3.rs similarity index 100% rename from crates/iceberg/src/io/config/s3.rs rename to crates/iceberg/src/io/storage/config/s3.rs diff --git a/crates/iceberg/src/io/local_fs.rs b/crates/iceberg/src/io/storage/local_fs.rs similarity index 99% rename from crates/iceberg/src/io/local_fs.rs rename to crates/iceberg/src/io/storage/local_fs.rs index 0a55199f70..d6dd5b433b 100644 --- a/crates/iceberg/src/io/local_fs.rs +++ b/crates/iceberg/src/io/storage/local_fs.rs @@ -31,7 +31,7 @@ use async_trait::async_trait; use bytes::Bytes; use serde::{Deserialize, Serialize}; -use super::{ +use crate::io::{ FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, StorageConfig, StorageFactory, }; diff --git a/crates/iceberg/src/io/memory.rs b/crates/iceberg/src/io/storage/memory.rs similarity index 99% rename from crates/iceberg/src/io/memory.rs rename to crates/iceberg/src/io/storage/memory.rs index 39f1f5db9d..cb01ee4709 100644 --- a/crates/iceberg/src/io/memory.rs +++ b/crates/iceberg/src/io/storage/memory.rs @@ -30,7 +30,7 @@ use async_trait::async_trait; use bytes::Bytes; use serde::{Deserialize, Serialize}; -use super::{ +use crate::io::{ FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, StorageConfig, StorageFactory, }; diff --git a/crates/iceberg/src/io/storage.rs b/crates/iceberg/src/io/storage/mod.rs similarity index 93% rename from crates/iceberg/src/io/storage.rs rename to crates/iceberg/src/io/storage/mod.rs index 15cc85ab10..31ceaf2436 100644 --- a/crates/iceberg/src/io/storage.rs +++ b/crates/iceberg/src/io/storage/mod.rs @@ -17,15 +17,25 @@ //! Storage interfaces for Iceberg. +mod config; +mod local_fs; +mod memory; +mod opendal; + use std::fmt::Debug; use std::sync::Arc; use async_trait::async_trait; use bytes::Bytes; +pub use config::*; +pub use local_fs::{LocalFsStorage, LocalFsStorageFactory}; +pub use memory::{MemoryStorage, MemoryStorageFactory}; +#[cfg(feature = "storage-s3")] +pub use opendal::CustomAwsCredentialLoader; +pub use opendal::{OpenDalStorage, OpenDalStorageFactory}; use super::{FileMetadata, FileRead, FileWrite, InputFile, OutputFile}; use crate::Result; -pub use crate::io::config::StorageConfig; /// Trait for storage operations in Iceberg. /// diff --git a/crates/iceberg/src/io/opendal/azdls.rs b/crates/iceberg/src/io/storage/opendal/azdls.rs similarity index 99% rename from crates/iceberg/src/io/opendal/azdls.rs rename to crates/iceberg/src/io/storage/opendal/azdls.rs index c957fd62a3..759ff301ea 100644 --- a/crates/iceberg/src/io/opendal/azdls.rs +++ b/crates/iceberg/src/io/storage/opendal/azdls.rs @@ -24,7 +24,7 @@ use opendal::services::AzdlsConfig; use serde::{Deserialize, Serialize}; use url::Url; -use crate::io::config::{ +use crate::io::{ ADLS_ACCOUNT_KEY, ADLS_ACCOUNT_NAME, ADLS_AUTHORITY_HOST, ADLS_CLIENT_ID, ADLS_CLIENT_SECRET, ADLS_CONNECTION_STRING, ADLS_SAS_TOKEN, ADLS_TENANT_ID, }; diff --git a/crates/iceberg/src/io/opendal/fs.rs b/crates/iceberg/src/io/storage/opendal/fs.rs similarity index 100% rename from crates/iceberg/src/io/opendal/fs.rs rename to crates/iceberg/src/io/storage/opendal/fs.rs diff --git a/crates/iceberg/src/io/opendal/gcs.rs b/crates/iceberg/src/io/storage/opendal/gcs.rs similarity index 96% rename from crates/iceberg/src/io/opendal/gcs.rs rename to crates/iceberg/src/io/storage/opendal/gcs.rs index 5c6145d32b..4cb8aa8591 100644 --- a/crates/iceberg/src/io/opendal/gcs.rs +++ b/crates/iceberg/src/io/storage/opendal/gcs.rs @@ -22,11 +22,10 @@ use opendal::Operator; use opendal::services::GcsConfig; use url::Url; -use crate::io::config::{ +use crate::io::{ GCS_ALLOW_ANONYMOUS, GCS_CREDENTIALS_JSON, GCS_DISABLE_CONFIG_LOAD, GCS_DISABLE_VM_METADATA, - GCS_NO_AUTH, GCS_SERVICE_PATH, GCS_TOKEN, + GCS_NO_AUTH, GCS_SERVICE_PATH, GCS_TOKEN, is_truthy, }; -use crate::io::is_truthy; use crate::{Error, ErrorKind, Result}; /// Parse iceberg properties to [`GcsConfig`]. diff --git a/crates/iceberg/src/io/opendal/memory.rs b/crates/iceberg/src/io/storage/opendal/memory.rs similarity index 100% rename from crates/iceberg/src/io/opendal/memory.rs rename to crates/iceberg/src/io/storage/opendal/memory.rs diff --git a/crates/iceberg/src/io/opendal/mod.rs b/crates/iceberg/src/io/storage/opendal/mod.rs similarity index 99% rename from crates/iceberg/src/io/opendal/mod.rs rename to crates/iceberg/src/io/storage/opendal/mod.rs index fb49dc9e3f..ee0df20fce 100644 --- a/crates/iceberg/src/io/opendal/mod.rs +++ b/crates/iceberg/src/io/storage/opendal/mod.rs @@ -38,7 +38,7 @@ use opendal::{Operator, Scheme}; pub use s3::CustomAwsCredentialLoader; use serde::{Deserialize, Serialize}; -use super::{ +use crate::io::{ FileIOBuilder, FileMetadata, FileRead, FileWrite, InputFile, OutputFile, Storage, StorageConfig, StorageFactory, }; diff --git a/crates/iceberg/src/io/opendal/oss.rs b/crates/iceberg/src/io/storage/opendal/oss.rs similarity index 96% rename from crates/iceberg/src/io/opendal/oss.rs rename to crates/iceberg/src/io/storage/opendal/oss.rs index 83fc1424aa..98b68f7cde 100644 --- a/crates/iceberg/src/io/opendal/oss.rs +++ b/crates/iceberg/src/io/storage/opendal/oss.rs @@ -21,7 +21,7 @@ use opendal::services::OssConfig; use opendal::{Configurator, Operator}; use url::Url; -use crate::io::config::{OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_ENDPOINT}; +use crate::io::{OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_ENDPOINT}; use crate::{Error, ErrorKind, Result}; /// Parse iceberg props to oss config. diff --git a/crates/iceberg/src/io/opendal/s3.rs b/crates/iceberg/src/io/storage/opendal/s3.rs similarity index 98% rename from crates/iceberg/src/io/opendal/s3.rs rename to crates/iceberg/src/io/storage/opendal/s3.rs index bf7399e01b..6b4d64e3a6 100644 --- a/crates/iceberg/src/io/opendal/s3.rs +++ b/crates/iceberg/src/io/storage/opendal/s3.rs @@ -25,13 +25,12 @@ pub use reqsign::{AwsCredential, AwsCredentialLoad}; use reqwest::Client; use url::Url; -use crate::io::config::{ +use crate::io::{ CLIENT_REGION, S3_ACCESS_KEY_ID, S3_ALLOW_ANONYMOUS, S3_ASSUME_ROLE_ARN, S3_ASSUME_ROLE_EXTERNAL_ID, S3_ASSUME_ROLE_SESSION_NAME, S3_DISABLE_CONFIG_LOAD, S3_DISABLE_EC2_METADATA, S3_ENDPOINT, S3_PATH_STYLE_ACCESS, S3_REGION, S3_SECRET_ACCESS_KEY, - S3_SESSION_TOKEN, S3_SSE_KEY, S3_SSE_MD5, S3_SSE_TYPE, + S3_SESSION_TOKEN, S3_SSE_KEY, S3_SSE_MD5, S3_SSE_TYPE, is_truthy, }; -use crate::io::is_truthy; use crate::{Error, ErrorKind, Result}; /// Parse iceberg props to s3 config. From 28df4ec4e308c08faa29e728a62406f690d0128c Mon Sep 17 00:00:00 2001 From: emkornfield Date: Wed, 11 Feb 2026 23:16:17 -0800 Subject: [PATCH 203/891] =?UTF-8?q?fix:=20Interpret=20s3tables=20warehouse?= =?UTF-8?q?=20as=20table=5Flocation=20not=20metadata=20loca=E2=80=A6=20(#2?= =?UTF-8?q?115)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? AWS [Docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-tables.html) state: > When you create a table, Amazon S3 automatically generates a warehouse location for the table. This is a unique S3 location that stores objects associated with the table. The following example shows the format of a warehouse location: ``` s3://63a8e430-6e0b-46f5-k833abtwr6s8tmtsycedn8s4yc3xhuse1b--table-s3 ``` We were previously interpreting this as as a metadata location (i.e. the path to the metadata.json file), this changes the code use it as a table location. - Closes #2114 ## What changes are included in this PR? Change how we construct the MetadataLocation object. ## Are these changes tested? There never appears to have been a test, and I don't have an AWS account to verify this. Note the test was initially developed by Claude Code and refined by me. --- crates/catalog/s3tables/src/catalog.rs | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/crates/catalog/s3tables/src/catalog.rs b/crates/catalog/s3tables/src/catalog.rs index 3606fac99a..a6ac60bc79 100644 --- a/crates/catalog/s3tables/src/catalog.rs +++ b/crates/catalog/s3tables/src/catalog.rs @@ -448,9 +448,9 @@ impl Catalog for S3TablesCatalog { .await .map_err(from_aws_sdk_error)?; - // prepare metadata location. the warehouse location is generated by s3tables catalog, + // prepare table location. the warehouse location is generated by s3tables catalog, // which looks like: s3://e6c9bf20-991a-46fb-kni5xs1q2yxi3xxdyxzjzigdeop1quse2b--table-s3 - let metadata_location = match &creation.location { + let table_location = match &creation.location { Some(_) => { return Err(Error::new( ErrorKind::DataInvalid, @@ -467,16 +467,17 @@ impl Catalog for S3TablesCatalog { .send() .await .map_err(from_aws_sdk_error)?; - let warehouse_location = get_resp.warehouse_location().to_string(); - MetadataLocation::new_with_table_location(warehouse_location).to_string() + get_resp.warehouse_location().to_string() } }; // write metadata to file - creation.location = Some(metadata_location.clone()); + creation.location = Some(table_location.clone()); let metadata = TableMetadataBuilder::from_table_creation(creation)? .build()? .metadata; + let metadata_location = + MetadataLocation::new_with_table_location(table_location).to_string(); metadata.write_to(&self.file_io, &metadata_location).await?; // update metadata location From 6c197fe8b02e24dc020ca327926d74ed77c13743 Mon Sep 17 00:00:00 2001 From: Cole Mackenzie Date: Thu, 12 Feb 2026 18:42:11 -0800 Subject: [PATCH 204/891] fix(rest): Filter sensitive headers from error logs (#2117) (#2130) --- crates/catalog/rest/src/catalog.rs | 109 ++++++++++++++--- crates/catalog/rest/src/client.rs | 184 ++++++++++++++++++++++++++++- 2 files changed, 277 insertions(+), 16 deletions(-) diff --git a/crates/catalog/rest/src/catalog.rs b/crates/catalog/rest/src/catalog.rs index ddbf6a4e01..eeea1f13e9 100644 --- a/crates/catalog/rest/src/catalog.rs +++ b/crates/catalog/rest/src/catalog.rs @@ -50,6 +50,8 @@ use crate::types::{ pub const REST_CATALOG_PROP_URI: &str = "uri"; /// REST catalog warehouse location pub const REST_CATALOG_PROP_WAREHOUSE: &str = "warehouse"; +/// Disable header redaction in error logs (defaults to false for security) +pub const REST_CATALOG_PROP_DISABLE_HEADER_REDACTION: &str = "disable-header-redaction"; const ICEBERG_REST_SPEC_VERSION: &str = "0.14.1"; const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION"); @@ -293,6 +295,17 @@ impl RestCatalogConfig { params } + /// Check if header redaction is disabled in error logs. + /// + /// Returns true if the `disable-header-redaction` property is set to "true". + /// Defaults to false for security (headers are redacted by default). + pub(crate) fn disable_header_redaction(&self) -> bool { + self.props + .get(REST_CATALOG_PROP_DISABLE_HEADER_REDACTION) + .map(|v| v.eq_ignore_ascii_case("true")) + .unwrap_or(false) + } + /// Merge the `RestCatalogConfig` with the a [`CatalogConfig`] (fetched from the REST server). pub(crate) fn merge_with_config(mut self, mut config: CatalogConfig) -> Self { if let Some(uri) = config.overrides.remove("uri") { @@ -378,7 +391,11 @@ impl RestCatalog { match http_response.status() { StatusCode::OK => deserialize_catalog_response(http_response).await, - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + client.disable_header_redaction(), + ) + .await), } } @@ -479,7 +496,13 @@ impl Catalog for RestCatalog { "The parent parameter of the namespace provided does not exist", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } } } @@ -514,7 +537,11 @@ impl Catalog for RestCatalog { ErrorKind::Unexpected, "Tried to create a namespace that already exists", )), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -538,7 +565,11 @@ impl Catalog for RestCatalog { ErrorKind::Unexpected, "Tried to get a namespace that does not exist", )), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -555,7 +586,11 @@ impl Catalog for RestCatalog { match http_response.status() { StatusCode::NO_CONTENT | StatusCode::OK => Ok(true), StatusCode::NOT_FOUND => Ok(false), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -586,7 +621,11 @@ impl Catalog for RestCatalog { ErrorKind::Unexpected, "Tried to drop a namespace that does not exist", )), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -623,7 +662,13 @@ impl Catalog for RestCatalog { "Tried to list tables of a namespace that does not exist", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } } } @@ -677,7 +722,13 @@ impl Catalog for RestCatalog { "The table already exists", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } }; let metadata_location = response.metadata_location.as_ref().ok_or(Error::new( @@ -732,7 +783,13 @@ impl Catalog for RestCatalog { "Tried to load a table that does not exist", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } }; let config = response @@ -774,7 +831,11 @@ impl Catalog for RestCatalog { ErrorKind::Unexpected, "Tried to drop a table that does not exist", )), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -792,7 +853,11 @@ impl Catalog for RestCatalog { match http_response.status() { StatusCode::NO_CONTENT | StatusCode::OK => Ok(true), StatusCode::NOT_FOUND => Ok(false), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -821,7 +886,11 @@ impl Catalog for RestCatalog { ErrorKind::Unexpected, "Tried to rename a table to a name that already exists", )), - _ => Err(deserialize_unexpected_catalog_error(http_response).await), + _ => Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await), } } @@ -865,7 +934,13 @@ impl Catalog for RestCatalog { "The given table already exists.", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } }; let metadata_location = response.metadata_location.as_ref().ok_or(Error::new( @@ -934,7 +1009,13 @@ impl Catalog for RestCatalog { "A server-side gateway timeout occurred; the commit state is unknown.", )); } - _ => return Err(deserialize_unexpected_catalog_error(http_response).await), + _ => { + return Err(deserialize_unexpected_catalog_error( + http_response, + context.client.disable_header_redaction(), + ) + .await); + } }; let file_io = self diff --git a/crates/catalog/rest/src/client.rs b/crates/catalog/rest/src/client.rs index 361c036bb6..07dc0620da 100644 --- a/crates/catalog/rest/src/client.rs +++ b/crates/catalog/rest/src/client.rs @@ -43,6 +43,8 @@ pub(crate) struct HttpClient { extra_headers: HeaderMap, /// Extra oauth parameters to be added to each authentication request. extra_oauth_params: HashMap, + /// Whether to disable header redaction in error logs (defaults to false for security). + disable_header_redaction: bool, } impl Debug for HttpClient { @@ -65,6 +67,7 @@ impl HttpClient { credential: cfg.credential(), extra_headers, extra_oauth_params: cfg.extra_oauth_params(), + disable_header_redaction: cfg.disable_header_redaction(), }) } @@ -92,6 +95,7 @@ impl HttpClient { } else { self.extra_oauth_params }, + disable_header_redaction: cfg.disable_header_redaction(), }) } @@ -258,6 +262,11 @@ impl HttpClient { self.authenticate(&mut request).await?; self.execute(request).await } + + /// Returns whether header redaction is disabled for this client. + pub(crate) fn disable_header_redaction(&self) -> bool { + self.disable_header_redaction + } } /// Deserializes a catalog response into the given [`DeserializedOwned`] type. @@ -278,14 +287,64 @@ pub(crate) async fn deserialize_catalog_response( }) } +/// Headers that contain sensitive information and should be excluded from logs. +const SENSITIVE_HEADERS: &[&str] = &[ + "authorization", + "proxy-authorization", + "set-cookie", + "cookie", + "x-api-key", + "x-auth-token", +]; + +/// Returns true if the header name is considered sensitive. +fn is_sensitive_header(name: &str) -> bool { + let name_lower = name.to_lowercase(); + SENSITIVE_HEADERS.iter().any(|h| name_lower == *h) +} + +/// Redacts sensitive headers and returns a debug-formatted string. +/// +/// If `disable_redaction` is true, returns all headers without redaction. +/// Otherwise, replaces sensitive header values with "[REDACTED]". +fn format_headers_redacted(headers: &HeaderMap, disable_redaction: bool) -> String { + if disable_redaction { + // Return all headers as-is without redaction + let all: HashMap<&str, &str> = headers + .iter() + .filter_map(|(name, value)| value.to_str().ok().map(|v| (name.as_str(), v))) + .collect(); + return format!("{all:?}"); + } + + // Redact sensitive headers by replacing their values with "[REDACTED]" + let redacted: HashMap<&str, &str> = headers + .iter() + .filter_map(|(name, value)| { + if is_sensitive_header(name.as_str()) { + Some((name.as_str(), "[REDACTED]")) + } else { + value.to_str().ok().map(|v| (name.as_str(), v)) + } + }) + .collect(); + format!("{redacted:?}") +} + /// Deserializes a unexpected catalog response into an error. -pub(crate) async fn deserialize_unexpected_catalog_error(response: Response) -> Error { +pub(crate) async fn deserialize_unexpected_catalog_error( + response: Response, + disable_header_redaction: bool, +) -> Error { let err = Error::new( ErrorKind::Unexpected, "Received response with unexpected status code", ) .with_context("status", response.status().to_string()) - .with_context("headers", format!("{:?}", response.headers())); + .with_context( + "headers", + format_headers_redacted(response.headers(), disable_header_redaction), + ); let bytes = match response.bytes().await { Ok(bytes) => bytes, @@ -297,3 +356,124 @@ pub(crate) async fn deserialize_unexpected_catalog_error(response: Response) -> } err.with_context("json", String::from_utf8_lossy(&bytes)) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_format_headers_redacted_empty() { + let headers = HeaderMap::new(); + let result = format_headers_redacted(&headers, false); + assert_eq!(result, "{}"); + } + + #[test] + fn test_format_headers_redacted_non_sensitive() { + let mut headers = HeaderMap::new(); + headers.insert("content-type", "application/json".parse().unwrap()); + headers.insert("x-request-id", "abc123".parse().unwrap()); + + let result = format_headers_redacted(&headers, false); + + assert!(result.contains("content-type")); + assert!(result.contains("application/json")); + assert!(result.contains("x-request-id")); + assert!(result.contains("abc123")); + } + + #[test] + fn test_format_headers_redacted_filters_sensitive() { + let mut headers = HeaderMap::new(); + headers.insert("authorization", "Bearer secret-token".parse().unwrap()); + headers.insert("content-type", "application/json".parse().unwrap()); + + let result = format_headers_redacted(&headers, false); + + // Sensitive header should be present but with redacted value + assert!(result.contains("authorization")); + assert!(result.contains("[REDACTED]")); + // Sensitive value should NOT be present + assert!(!result.contains("secret-token")); + // Non-sensitive header should be present with actual value + assert!(result.contains("content-type")); + assert!(result.contains("application/json")); + } + + #[test] + fn test_format_headers_redacted_filters_set_cookie() { + let mut headers = HeaderMap::new(); + headers.insert( + "set-cookie", + "CF_Authorization=sensitive-session-token; Path=/; Secure;" + .parse() + .unwrap(), + ); + headers.insert("server", "cloudflare".parse().unwrap()); + + let result = format_headers_redacted(&headers, false); + + // Sensitive header should be present but with redacted value + assert!(result.contains("set-cookie")); + assert!(result.contains("[REDACTED]")); + // Sensitive value should NOT be present + assert!(!result.contains("sensitive-session-token")); + // Non-sensitive header should be present with actual value + assert!(result.contains("server")); + assert!(result.contains("cloudflare")); + } + + #[test] + fn test_format_headers_redacted_filters_all_sensitive() { + let mut headers = HeaderMap::new(); + headers.insert("authorization", "Bearer token".parse().unwrap()); + headers.insert("proxy-authorization", "Basic creds".parse().unwrap()); + headers.insert("set-cookie", "session=abc".parse().unwrap()); + headers.insert("cookie", "session=abc".parse().unwrap()); + headers.insert("x-api-key", "api-key-123".parse().unwrap()); + headers.insert("x-auth-token", "auth-token-456".parse().unwrap()); + headers.insert("x-request-id", "req-123".parse().unwrap()); + + let result = format_headers_redacted(&headers, false); + + // All sensitive headers should be present but with redacted values + assert!(result.contains("authorization")); + assert!(result.contains("proxy-authorization")); + assert!(result.contains("set-cookie")); + assert!(result.contains("cookie")); + assert!(result.contains("x-api-key")); + assert!(result.contains("x-auth-token")); + assert!(result.contains("[REDACTED]")); + + // Ensure no sensitive values leaked + assert!(!result.contains("Bearer token")); + assert!(!result.contains("Basic creds")); + assert!(!result.contains("session=abc")); + assert!(!result.contains("api-key-123")); + assert!(!result.contains("auth-token-456")); + + // Non-sensitive header should be present with actual value + assert!(result.contains("x-request-id")); + assert!(result.contains("req-123")); + } + + #[test] + fn test_format_headers_with_redaction_disabled() { + let mut headers = HeaderMap::new(); + headers.insert("authorization", "Bearer secret-token".parse().unwrap()); + headers.insert("x-api-key", "api-key-123".parse().unwrap()); + headers.insert("content-type", "application/json".parse().unwrap()); + + let result = format_headers_redacted(&headers, true); + + // When redaction is disabled, all headers and values should be present + assert!(result.contains("authorization")); + assert!(result.contains("Bearer secret-token")); + assert!(result.contains("x-api-key")); + assert!(result.contains("api-key-123")); + assert!(result.contains("content-type")); + assert!(result.contains("application/json")); + // [REDACTED] should NOT be present when redaction is disabled + assert!(!result.contains("[REDACTED]")); + } +} From 30a356079189db38ae7cdb0fa18394deeeb1f5c9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Feb 2026 15:46:34 +0800 Subject: [PATCH 205/891] chore(deps): Bump crate-ci/typos from 1.43.3 to 1.43.4 (#2138) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.43.3 to 1.43.4.
Release notes

Sourced from crate-ci/typos's releases.

v1.43.4

[1.43.4] - 2026-02-09

Fixes

  • Don't correct pincher
Changelog

Sourced from crate-ci/typos's changelog.

[1.43.4] - 2026-02-09

Fixes

  • Don't correct pincher
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.43.3&new-version=1.43.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 13ceeaf236..782c81dcea 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -42,4 +42,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.43.3 + uses: crate-ci/typos@v1.43.4 From e9b312a844f5bd871de33a231c308a4159299209 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Feb 2026 15:46:59 +0800 Subject: [PATCH 206/891] chore(deps): Bump tempfile from 3.23.0 to 3.25.0 (#2140) Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.23.0 to 3.25.0.
Changelog

Sourced from tempfile's changelog.

3.25.0

  • Allow getrandom 0.4.x while retaining support for getrandom 0.3.x.

3.24.0

  • Actually support WASIp2 without the nightly feature. This library is now feature complete on WASIp2 without any additional feature flags.
  • Exclude CI scripts from the published crate.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tempfile&package-manager=cargo&previous-version=3.23.0&new-version=3.25.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 054de94e68..7f7b0f6d62 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -120,7 +120,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -131,7 +131,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -2497,7 +2497,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2650,7 +2650,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -3867,7 +3867,7 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -4388,7 +4388,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5705,15 +5705,15 @@ dependencies = [ [[package]] name = "rustix" -version = "1.1.2" +version = "1.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e" +checksum = "146c9e247ccc180c1f61615433868c99f3de3ae256a30a43b49f67c2d9171f34" dependencies = [ "bitflags", "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -6716,15 +6716,15 @@ checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" [[package]] name = "tempfile" -version = "3.23.0" +version = "3.25.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16" +checksum = "0136791f7c95b1f6dd99f9cc786b91bb81c3800b639b3478e561ddb7be95e5f1" dependencies = [ "fastrand", "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -7569,7 +7569,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] From eb9d374e0f2ad9794fabef65c436288e2f0614c2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Feb 2026 15:47:46 +0800 Subject: [PATCH 207/891] chore(deps): Bump uuid from 1.20.0 to 1.21.0 (#2141) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.20.0 to 1.21.0.
Release notes

Sourced from uuid's releases.

v1.21.0

What's Changed

New Contributors

Full Changelog: https://github.com/uuid-rs/uuid/compare/v1.20.0...v1.21.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=uuid&package-manager=cargo&previous-version=1.20.0&new-version=1.21.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 166 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 7f7b0f6d62..655de3b976 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2995,6 +2995,19 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "getrandom" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec" +dependencies = [ + "cfg-if", + "libc", + "r-efi", + "wasip2", + "wasip3", +] + [[package]] name = "glob" version = "0.3.3" @@ -3711,6 +3724,12 @@ dependencies = [ "zerovec", ] +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + [[package]] name = "ident_case" version = "1.0.1" @@ -3940,6 +3959,12 @@ dependencies = [ "spin", ] +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + [[package]] name = "lexical-core" version = "1.0.6" @@ -7264,6 +7289,12 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + [[package]] name = "unit-prefix" version = "0.5.2" @@ -7314,11 +7345,11 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.20.0" +version = "1.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee48d38b119b0cd71fe4141b30f5ba9c7c5d9f4e7a3a8b4a674e4b6ef789976f" +checksum = "b672338555252d43fd2240c714dc444b8c6fb0a5c5335e65a07bba7742735ddb" dependencies = [ - "getrandom 0.3.4", + "getrandom 0.4.1", "js-sys", "serde_core", "wasm-bindgen", @@ -7435,7 +7466,16 @@ version = "1.0.1+wasi-0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.46.0", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", ] [[package]] @@ -7502,6 +7542,28 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap 2.12.1", + "wasm-encoder", + "wasmparser", +] + [[package]] name = "wasm-streams" version = "0.4.2" @@ -7515,6 +7577,18 @@ dependencies = [ "web-sys", ] +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap 2.12.1", + "semver", +] + [[package]] name = "web-sys" version = "0.3.83" @@ -7877,6 +7951,94 @@ version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap 2.12.1", + "prettyplease", + "syn 2.0.111", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn 2.0.111", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap 2.12.1", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap 2.12.1", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + [[package]] name = "writeable" version = "0.6.2" From 19fa0d755dfa1096e136e5f6e121373d586fb691 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 16 Feb 2026 21:40:04 -0500 Subject: [PATCH 208/891] infra: set github actions max-parallel to 15 (#2150) --- .github/workflows/bindings_python_ci.yml | 1 + .github/workflows/ci.yml | 4 ++++ .github/workflows/publish.yml | 2 +- .github/workflows/release_python.yml | 1 + .github/workflows/release_python_nightly.yml | 1 + 5 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index ed2c03b763..1c3350221c 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -75,6 +75,7 @@ jobs: test: runs-on: ${{ matrix.os }} strategy: + max-parallel: 15 matrix: os: - ubuntu-latest diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fe7476a80e..3b22201154 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,6 +44,7 @@ jobs: check: runs-on: ${{ matrix.os }} strategy: + max-parallel: 15 matrix: os: - ubuntu-latest @@ -80,6 +81,7 @@ jobs: build: runs-on: ${{ matrix.os }} strategy: + max-parallel: 15 matrix: os: - ubuntu-latest @@ -105,6 +107,7 @@ jobs: build_with_no_default_features: runs-on: ${{ matrix.os }} strategy: + max-parallel: 15 matrix: os: - ubuntu-latest @@ -125,6 +128,7 @@ jobs: tests: runs-on: ubuntu-latest strategy: + max-parallel: 15 matrix: test-suite: - { name: "default", args: "--all-targets --all-features --workspace" } diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6b4b3da841..47b3d83f58 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -31,7 +31,7 @@ jobs: runs-on: ubuntu-latest strategy: # Publish package one by one instead of flooding the registry - max-parallel: 1 + max-parallel: 15 matrix: # Order here is sensitive, as it will be used to determine the order of publishing package: diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index e587b50adc..2203271b33 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -142,6 +142,7 @@ jobs: runs-on: "${{ matrix.os }}" needs: [validate-release-tag] strategy: + max-parallel: 15 matrix: include: - { os: windows-latest } diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 8bdd9d1003..32614b2d0e 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -63,6 +63,7 @@ jobs: if: github.repository == 'apache/iceberg-rust' # Only run for apache repo runs-on: "${{ matrix.os }}" strategy: + max-parallel: 15 matrix: include: - { os: windows-latest } From 8cc171deac1daa82df81e2c3016e9d0243b2d584 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 17 Feb 2026 18:53:54 +0800 Subject: [PATCH 209/891] chore(deps): Bump the arrow-parquet group with 2 updates (#2139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps the arrow-parquet group with 2 updates: [arrow-arith](https://github.com/apache/arrow-rs) and [arrow-cast](https://github.com/apache/arrow-rs). Updates `arrow-arith` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-arith's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-arith's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-cast` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-cast's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-cast's changelog.

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 655de3b976..dfce67d7b5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -244,9 +244,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "288015089e7931843c80ed4032c5274f02b37bcb720c4a42096d50b390e70372" +checksum = "f7b3141e0ec5145a22d8694ea8b6d6f69305971c4fa1c1a13ef0195aef2d678b" dependencies = [ "arrow-array", "arrow-buffer", @@ -289,9 +289,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "57.2.0" +version = "57.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c8e372ed52bd4ee88cc1e6c3859aa7ecea204158ac640b10e187936e7e87074" +checksum = "646bbb821e86fd57189c10b4fcdaa941deaf4181924917b0daa92735baa6ada5" dependencies = [ "arrow-array", "arrow-buffer", From 1231877f543acda65e22ae51d67ee4a5e2461418 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Tue, 17 Feb 2026 20:17:37 -0500 Subject: [PATCH 210/891] CI: Add CodeQL workflow for GitHub Actions security scanning (#2151) --- .github/workflows/codeql.yml | 50 ++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 .github/workflows/codeql.yml diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000000..a3c9887a81 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: "CodeQL" + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + schedule: + - cron: '16 4 * * 1' + +jobs: + analyze: + name: Analyze Actions + runs-on: ubuntu-latest + permissions: + security-events: write + packages: read + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: + languages: actions + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v4 + with: + category: "/language:actions" From 3ae372293f6802274d2b5c7b53f34c75a44bb15b Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 23 Feb 2026 19:34:47 -0500 Subject: [PATCH 211/891] CI(Python): dont use --sdist in maturin build for CI (#2166) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This PR updates Python bindings CI to avoid `--sdist` in the test build step due to `maturin` regression for 1.12.3+ tracked in https://github.com/PyO3/maturin/issues/3030. We dont pin maturin in CI, so CI installs the latest version. `--sdist` is not needed here, removing it also aligns with other `maturin` usage in CI See https://github.com/search?q=repo%3Aapache%2Ficeberg-rust+%22command%3A+build%22&type=code ## Are these changes tested? --- .github/workflows/bindings_python_ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index 1c3350221c..8be856c246 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -90,7 +90,7 @@ jobs: with: working-directory: "bindings/python" command: build - args: --out dist --sdist + args: --out dist - uses: astral-sh/setup-uv@v7 with: version: "0.9.3" From 873f7adf3c351c8cba37a970efe97879d79f266a Mon Sep 17 00:00:00 2001 From: Shekhar Prasad Rajak <5774448+Shekharrajak@users.noreply.github.com> Date: Tue, 24 Feb 2026 06:07:38 +0530 Subject: [PATCH 212/891] feat(playground): add memory catalog support (#2121) ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1160 ## What changes are included in this PR? - Added MemoryCatalogBuilder - Changed the return to use the already-Arc-wrapped catalog - Replaced the hardcoded rest-only check with a match statement: Easy extensibility (just add a new match arm for new catalog types) ## Are these changes tested? After build : ``` $ cargo build -p iceberg-playground --release $ cat ~/.icebergrc [[catalogs]] name = "local" type = "memory" [catalogs.config] warehouse = "/tmp/iceberg-warehouse" $ ./target/release/iceberg-playground --rc /path/to/config.toml ``` --- crates/integrations/playground/src/catalog.rs | 90 +++++++++++++++++-- 1 file changed, 84 insertions(+), 6 deletions(-) diff --git a/crates/integrations/playground/src/catalog.rs b/crates/integrations/playground/src/catalog.rs index f7fd14ca84..66742ab893 100644 --- a/crates/integrations/playground/src/catalog.rs +++ b/crates/integrations/playground/src/catalog.rs @@ -24,6 +24,7 @@ use anyhow::anyhow; use datafusion::catalog::{CatalogProvider, CatalogProviderList}; use fs_err::read_to_string; use iceberg::CatalogBuilder; +use iceberg::memory::MemoryCatalogBuilder; use iceberg_catalog_rest::RestCatalogBuilder; use iceberg_datafusion::IcebergCatalogProvider; use toml::{Table as TomlTable, Value}; @@ -78,10 +79,6 @@ impl IcebergCatalogList { .as_str() .ok_or_else(|| anyhow::anyhow!("type is not string"))?; - if r#type != "rest" { - return Err(anyhow::anyhow!("Only rest catalog is supported for now!")); - } - let catalog_config = config .get("config") .ok_or_else(|| anyhow::anyhow!("config not found for catalog {name}"))? @@ -96,11 +93,21 @@ impl IcebergCatalogList { .ok_or_else(|| anyhow::anyhow!("props {key} is not string"))?; props.insert(key.to_string(), value_str.to_string()); } - let catalog = RestCatalogBuilder::default().load(name, props).await?; + + // Create catalog based on type using the appropriate builder + let catalog: Arc = match r#type { + "rest" => Arc::new(RestCatalogBuilder::default().load(name, props).await?), + "memory" => Arc::new(MemoryCatalogBuilder::default().load(name, props).await?), + _ => { + return Err(anyhow::anyhow!( + "Unsupported catalog type: '{type}'. Supported types: rest, memory" + )); + } + }; Ok(( name.to_string(), - Arc::new(IcebergCatalogProvider::try_new(Arc::new(catalog)).await?), + Arc::new(IcebergCatalogProvider::try_new(catalog).await?), )) } } @@ -129,3 +136,74 @@ impl CatalogProviderList for IcebergCatalogList { .map(|c| c.clone() as Arc) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn test_parse_memory_catalog() { + let config = r#" + [[catalogs]] + name = "test_memory" + type = "memory" + [catalogs.config] + warehouse = "/tmp/test-warehouse" + "#; + + let toml_table: TomlTable = toml::from_str(config).unwrap(); + let catalog_list = IcebergCatalogList::parse_table(&toml_table).await.unwrap(); + + assert!( + catalog_list + .catalog_names() + .contains(&"test_memory".to_string()) + ); + assert!(catalog_list.catalog("test_memory").is_some()); + } + + #[tokio::test] + async fn test_parse_unsupported_catalog_type() { + let config = r#" + [[catalogs]] + name = "test_hive" + type = "hive" + [catalogs.config] + uri = "thrift://localhost:9083" + "#; + + let toml_table: TomlTable = toml::from_str(config).unwrap(); + let result = IcebergCatalogList::parse_table(&toml_table).await; + + assert!(result.is_err()); + let err_msg = result.unwrap_err().to_string(); + assert!(err_msg.contains("Unsupported catalog type")); + assert!(err_msg.contains("hive")); + assert!(err_msg.contains("rest, memory")); + } + + #[tokio::test] + async fn test_catalog_names() { + let config = r#" + [[catalogs]] + name = "catalog_one" + type = "memory" + [catalogs.config] + warehouse = "/tmp/warehouse1" + + [[catalogs]] + name = "catalog_two" + type = "memory" + [catalogs.config] + warehouse = "/tmp/warehouse2" + "#; + + let toml_table: TomlTable = toml::from_str(config).unwrap(); + let catalog_list = IcebergCatalogList::parse_table(&toml_table).await.unwrap(); + + let names = catalog_list.catalog_names(); + assert_eq!(names.len(), 2); + assert!(names.contains(&"catalog_one".to_string())); + assert!(names.contains(&"catalog_two".to_string())); + } +} From 430b26e963f0a92b4b269a63a7930e3312fdecea Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 24 Feb 2026 08:55:09 +0800 Subject: [PATCH 213/891] chore(deps): Bump minijinja from 2.15.1 to 2.16.0 (#2162) Bumps [minijinja](https://github.com/mitsuhiko/minijinja) from 2.15.1 to 2.16.0.
Release notes

Sourced from minijinja's releases.

2.16.0

Release Notes

  • Added musllinux wheel builds for Python release artifacts.
  • Fixed |escape to honor custom formatters. #861
  • Aligned undefined behavior handling in the Go port with Rust.
  • Removed non-Rust keys and values filters from the Go port for parity. #863

Install minijinja-cli 2.16.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf
https://github.com/mitsuhiko/minijinja/releases/download/minijinja-go/v2.16.0/minijinja-cli-installer.sh
| sh

Install prebuilt binaries via powershell script

powershell -ExecutionPolicy Bypass -c "irm
https://github.com/mitsuhiko/minijinja/releases/download/minijinja-go/v2.16.0/minijinja-cli-installer.ps1
| iex"

Download minijinja-cli 2.16.0

File Platform Checksum
minijinja-cli-aarch64-apple-darwin.tar.xz Apple Silicon macOS checksum
minijinja-cli-x86_64-apple-darwin.tar.xz Intel macOS checksum
minijinja-cli-i686-pc-windows-msvc.zip x86 Windows checksum
minijinja-cli-x86_64-pc-windows-msvc.zip x64 Windows checksum
minijinja-cli-aarch64-unknown-linux-gnu.tar.xz ARM64 Linux checksum
minijinja-cli-i686-unknown-linux-gnu.tar.xz x86 Linux checksum
minijinja-cli-x86_64-unknown-linux-gnu.tar.xz x64 Linux checksum
minijinja-cli-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
minijinja-cli-i686-unknown-linux-musl.tar.xz x86 MUSL Linux checksum
minijinja-cli-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum
Changelog

Sourced from minijinja's changelog.

2.16.0

  • Added musllinux wheel builds for Python release artifacts.
  • Fixed |escape to honor custom formatters. #861
  • Aligned undefined behavior handling in the Go port with Rust.
  • Removed non-Rust keys and values filters from the Go port for parity. #863
Commits
  • b9afca4 Release 2.16.0
  • c610e03 fix(minijinja-go): remove keys and values filters
  • b2a70a0 chore(ci): add wheel workflow dry-run mode
  • 1bb42c3 feat(py): build musllinux wheels on release
  • 84741ce fix(filters): honor custom formatter in escape (#861)
  • 42845cb chore(ci): remove typos workflow
  • b98dd05 Claude -> Pi
  • 3b5b753 fix(minijinja-go): align undefined behavior with rust
  • 8d587da fix(minijinja-go): use MakeObjectMap in self_referential_context example
  • 30a4c77 ci(release): trigger pkg.go.dev indexing after Go module proxy warm
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=minijinja&package-manager=cargo&previous-version=2.15.1&new-version=2.16.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index dfce67d7b5..8c1ff336a6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4212,9 +4212,9 @@ dependencies = [ [[package]] name = "minijinja" -version = "2.15.1" +version = "2.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b479616bb6f0779fb0f3964246beda02d4b01144e1b0d5519616e012ccc2a245" +checksum = "5c54f3bcc034dd74496b5ca929fd0b710186672d5ff0b0f255a9ceb259042ece" dependencies = [ "serde", ] From 8427966d495326460cb10febfc6f74e8ba029473 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 24 Feb 2026 09:14:49 +0800 Subject: [PATCH 214/891] chore(deps): Bump actions/stale from 10.1.1 to 10.2.0 (#2159) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/stale](https://github.com/actions/stale) from 10.1.1 to 10.2.0.
Release notes

Sourced from actions/stale's releases.

v10.2.0

What's Changed

Bug Fix

Dependency Updates

New Contributors

Full Changelog: https://github.com/actions/stale/compare/v10...v10.2.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/stale&package-manager=github_actions&previous-version=10.1.1&new-version=10.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- .github/workflows/stale.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 95a4fdc256..051e9395fc 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -31,7 +31,7 @@ jobs: if: github.repository_owner == 'apache' runs-on: ubuntu-22.04 steps: - - uses: actions/stale@v10.1.1 + - uses: actions/stale@v10.2.0 with: stale-issue-label: 'stale' exempt-issue-labels: 'not-stale' From d84ae1709d59eb2ebb772db81e54eb641a10390e Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 23 Feb 2026 20:52:17 -0500 Subject: [PATCH 215/891] chore(ci): add explicit least-privilege workflow permissions (#2163) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? Added explicit permissions blocks to GitHub Actions workflows to satisfy CodeQL actions/missing-workflow-permissions. (See the [Security tab on Github](https://github.com/apache/iceberg-rust/security/code-scanning)) Defaulted workflows to `contents: read`. ## Are these changes tested? Co-authored-by: blackmwk --- .github/workflows/audit.yml | 3 +++ .github/workflows/bindings_python_ci.yml | 3 +++ .github/workflows/ci.yml | 3 +++ .github/workflows/ci_typos.yml | 3 +++ .github/workflows/codeql.yml | 4 ++++ .github/workflows/publish.yml | 3 +++ .github/workflows/website.yml | 3 +++ 7 files changed, 22 insertions(+) diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index a86c58e5c7..abe0c377c5 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -36,6 +36,9 @@ on: schedule: - cron: '0 0 * * *' +permissions: + contents: read + jobs: security_audit: runs-on: ubuntu-latest diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index 8be856c246..a78f27b0b1 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -40,6 +40,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }} cancel-in-progress: true +permissions: + contents: read + jobs: check-rust: runs-on: ubuntu-latest diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3b22201154..ea2257b676 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,6 +40,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }} cancel-in-progress: true +permissions: + contents: read + jobs: check: runs-on: ${{ matrix.os }} diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 782c81dcea..c9ca4f4973 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -32,6 +32,9 @@ concurrency: env: RUST_BACKTRACE: 1 +permissions: + contents: read + jobs: typos-check: name: typos check diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index a3c9887a81..e81c715896 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -27,11 +27,15 @@ on: schedule: - cron: '16 4 * * 1' +permissions: + contents: read + jobs: analyze: name: Analyze Actions runs-on: ubuntu-latest permissions: + contents: read security-events: write packages: read diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 47b3d83f58..4504f6e2f0 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -26,6 +26,9 @@ on: - "v[0-9]+.[0-9]+.[0-9]+-rc.[0-9]+" workflow_dispatch: +permissions: + contents: read + jobs: publish: runs-on: ubuntu-latest diff --git a/.github/workflows/website.yml b/.github/workflows/website.yml index 1a52482b08..59bd2c6f2c 100644 --- a/.github/workflows/website.yml +++ b/.github/workflows/website.yml @@ -30,6 +30,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }} cancel-in-progress: true +permissions: + contents: read + jobs: build: runs-on: ubuntu-latest From 6a01be7d46334aedabb4e320893a62514ef31be3 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Mon, 23 Feb 2026 20:52:37 -0500 Subject: [PATCH 216/891] Potential fix for code scanning alert no. 1: Cache Poisoning via low-privileged code injection (#2164) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Potential fix for [https://github.com/apache/iceberg-rust/security/code-scanning/1](https://github.com/apache/iceberg-rust/security/code-scanning/1) To fix the issue, pass `github.event.workflow_run.head_branch` into the shell as an environment variable instead of interpolating it directly in the script, and then reference only the environment variable inside the `run` block. This follows the safer pattern from the “Secure Workflow” example, where GitHub expressions are resolved into environment variables and then treated as inert data. Concretely, update the `Validate release tag format` step: - Add an `env:` section with two variables: - `DISPATCH_RELEASE_TAG: ${{ github.event.inputs.release_tag }}` - `RUN_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}` - Replace the direct usages of `${{ github.event.inputs.release_tag }}` and `${{ github.event.workflow_run.head_branch }}` in the shell script with `$DISPATCH_RELEASE_TAG` and `$RUN_HEAD_BRANCH` respectively. This change is all within `.github/workflows/release_python.yml`, in the `validate-release-tag` job, `Validate release tag format` step. No new methods, external definitions, or imports are required. _Suggested fixes powered by Copilot Autofix. Review carefully before merging._ ### Tested On fork repo github action run: https://github.com/kevinjqliu/iceberg-rust/actions/runs/22290533306 Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- .github/workflows/release_python.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index 2203271b33..f4cf4a761a 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -55,11 +55,14 @@ jobs: # Use input for workflow_dispatch, otherwise use `workflow_run.head_branch` # Note, `workflow_run.head_branch` does not contain `refs/tags/` prefix, just the tag name, i.e. `v0.4.0` or `v0.4.0-rc.1` # Valid formats: v.. OR v..-rc. + env: + DISPATCH_RELEASE_TAG: ${{ github.event.inputs.release_tag }} + RUN_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }} run: | if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - RELEASE_TAG="${{ github.event.inputs.release_tag }}" + RELEASE_TAG="$DISPATCH_RELEASE_TAG" else - RELEASE_TAG="${{ github.event.workflow_run.head_branch }}" + RELEASE_TAG="$RUN_HEAD_BRANCH" fi echo "Validating release tag: $RELEASE_TAG" if [[ ! "$RELEASE_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$ ]]; then From 85c07d7acc49d3ce35c6e86166db9c2e47381eca Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 23 Feb 2026 18:22:24 -0800 Subject: [PATCH 217/891] chore(deps): Bump actions/checkout from 4 to 6 (#2161) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6.
Release notes

Sourced from actions/checkout's releases.

v6.0.0

What's Changed

Full Changelog: https://github.com/actions/checkout/compare/v5.0.0...v6.0.0

v6-beta

What's Changed

Updated persist-credentials to store the credentials under $RUNNER_TEMP instead of directly in the local git config.

This requires a minimum Actions Runner version of v2.329.0 to access the persisted credentials for Docker container action scenarios.

v5.0.1

What's Changed

Full Changelog: https://github.com/actions/checkout/compare/v5...v5.0.1

v5.0.0

What's Changed

⚠️ Minimum Compatible Runner Version

v2.327.1
Release Notes

Make sure your runner is updated to this version or newer to use this release.

Full Changelog: https://github.com/actions/checkout/compare/v4...v5.0.0

v4.3.1

What's Changed

Full Changelog: https://github.com/actions/checkout/compare/v4...v4.3.1

v4.3.0

What's Changed

... (truncated)

Changelog

Sourced from actions/checkout's changelog.

Changelog

v6.0.2

v6.0.1

v6.0.0

v5.0.1

v5.0.0

v4.3.1

v4.3.0

v4.2.2

v4.2.1

v4.2.0

v4.1.7

v4.1.6

... (truncated)

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/checkout&package-manager=github_actions&previous-version=4&new-version=6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: blackmwk --- .github/workflows/codeql.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index e81c715896..7fef7973e5 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -41,7 +41,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Initialize CodeQL uses: github/codeql-action/init@v4 From 18178140fc4414d62048ab2ee8da1e3ffaea164e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 24 Feb 2026 11:14:24 +0800 Subject: [PATCH 218/891] chore(deps): Bump crate-ci/typos from 1.43.4 to 1.43.5 (#2160) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.43.4 to 1.43.5.
Release notes

Sourced from crate-ci/typos's releases.

v1.43.5

[1.43.5] - 2026-02-16

Fixes

  • (pypi) Hopefully fix the sdist build
Changelog

Sourced from crate-ci/typos's changelog.

[1.43.5] - 2026-02-16

Fixes

  • (pypi) Hopefully fix the sdist build
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.43.4&new-version=1.43.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index c9ca4f4973..38541bffbc 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -45,4 +45,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.43.4 + uses: crate-ci/typos@v1.43.5 From 88fdfedf1f589ea118aa8bab0c20a3191cf4a0a1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 24 Feb 2026 17:29:35 +0800 Subject: [PATCH 219/891] chore(deps): Bump urllib3 from 2.5.0 to 2.6.3 in /bindings/python (#2170) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.3.
Release notes

Sourced from urllib3's releases.

2.6.3

🚀 urllib3 is fundraising for HTTP/2 support

urllib3 is raising ~$40,000 USD to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects please consider contributing financially to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.

Thank you for your support.

Changes

  • Fixed a security issue where decompression-bomb safeguards of the streaming API were bypassed when HTTP redirects were followed. (CVE-2026-21441 reported by @​D47A, 8.9 High, GHSA-38jv-5279-wg99)
  • Started treating Retry-After times greater than 6 hours as 6 hours by default. (urllib3/urllib3#3743)
  • Fixed urllib3.connection.VerifiedHTTPSConnection on Emscripten. (urllib3/urllib3#3752)

2.6.2

🚀 urllib3 is fundraising for HTTP/2 support

urllib3 is raising ~$40,000 USD to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects please consider contributing financially to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.

Thank you for your support.

Changes

  • Fixed HTTPResponse.read_chunked() to properly handle leftover data in the decoder's buffer when reading compressed chunked responses. (urllib3/urllib3#3734)

2.6.1

🚀 urllib3 is fundraising for HTTP/2 support

urllib3 is raising ~$40,000 USD to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects please consider contributing financially to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.

Thank you for your support.

Changes

  • Restore previously removed HTTPResponse.getheaders() and HTTPResponse.getheader() methods. (#3731)

2.6.0

🚀 urllib3 is fundraising for HTTP/2 support

urllib3 is raising ~$40,000 USD to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects please consider contributing financially to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.

Thank you for your support.

Security

  • Fixed a security issue where streaming API could improperly handle highly compressed HTTP content ("decompression bombs") leading to excessive resource consumption even when a small amount of data was requested. Reading small chunks of compressed data is safer and much more efficient now. (CVE-2025-66471 reported by @​Cycloctane, 8.9 High, GHSA-2xpw-w6gg-jr37)
  • Fixed a security issue where an attacker could compose an HTTP response with virtually unlimited links in the Content-Encoding header, potentially leading to a denial of service (DoS) attack by exhausting system resources during decoding. The number of allowed chained encodings is now limited to 5. (CVE-2025-66418 reported by @​illia-v, 8.9 High, GHSA-gm62-xv2j-4w53)

[!IMPORTANT]

  • If urllib3 is not installed with the optional urllib3[brotli] extra, but your environment contains a Brotli/brotlicffi/brotlipy package anyway, make sure to upgrade it to at least Brotli 1.2.0 or brotlicffi 1.2.0.0 to benefit from the security fixes and avoid warnings. Prefer using urllib3[brotli] to install a compatible Brotli package automatically.

... (truncated)

Changelog

Sourced from urllib3's changelog.

2.6.3 (2026-01-07)

  • Fixed a high-severity security issue where decompression-bomb safeguards of the streaming API were bypassed when HTTP redirects were followed. (GHSA-38jv-5279-wg99 <https://github.com/urllib3/urllib3/security/advisories/GHSA-38jv-5279-wg99>__)
  • Started treating Retry-After times greater than 6 hours as 6 hours by default. ([#3743](https://github.com/urllib3/urllib3/issues/3743) <https://github.com/urllib3/urllib3/issues/3743>__)
  • Fixed urllib3.connection.VerifiedHTTPSConnection on Emscripten. ([#3752](https://github.com/urllib3/urllib3/issues/3752) <https://github.com/urllib3/urllib3/issues/3752>__)

2.6.2 (2025-12-11)

  • Fixed HTTPResponse.read_chunked() to properly handle leftover data in the decoder's buffer when reading compressed chunked responses. ([#3734](https://github.com/urllib3/urllib3/issues/3734) <https://github.com/urllib3/urllib3/issues/3734>__)

2.6.1 (2025-12-08)

  • Restore previously removed HTTPResponse.getheaders() and HTTPResponse.getheader() methods. ([#3731](https://github.com/urllib3/urllib3/issues/3731) <https://github.com/urllib3/urllib3/issues/3731>__)

2.6.0 (2025-12-05)

Security

  • Fixed a security issue where streaming API could improperly handle highly compressed HTTP content ("decompression bombs") leading to excessive resource consumption even when a small amount of data was requested. Reading small chunks of compressed data is safer and much more efficient now. (GHSA-2xpw-w6gg-jr37 <https://github.com/urllib3/urllib3/security/advisories/GHSA-2xpw-w6gg-jr37>__)
  • Fixed a security issue where an attacker could compose an HTTP response with virtually unlimited links in the Content-Encoding header, potentially leading to a denial of service (DoS) attack by exhausting system resources during decoding. The number of allowed chained encodings is now limited to 5. (GHSA-gm62-xv2j-4w53 <https://github.com/urllib3/urllib3/security/advisories/GHSA-gm62-xv2j-4w53>__)

.. caution::

  • If urllib3 is not installed with the optional urllib3[brotli] extra, but your environment contains a Brotli/brotlicffi/brotlipy package anyway, make sure to upgrade it to at least Brotli 1.2.0 or brotlicffi 1.2.0.0 to benefit from the security fixes and avoid warnings. Prefer using

... (truncated)

Commits
  • 0248277 Release 2.6.3
  • 8864ac4 Merge commit from fork
  • 70cecb2 Fix Scorecard issues related to vulnerable dev dependencies (#3755)
  • 41f249a Move "v2.0 Migration Guide" to the end of the table of contents (#3747)
  • fd4dffd Patch VerifiedHTTPSConnection for Emscripten (#3752)
  • 13f0bfd Handle massive values in Retry-After when calculating time to sleep for (#3743)
  • 8c480bf Bump actions/upload-artifact from 5.0.0 to 6.0.0 (#3748)
  • 4b40616 Bump actions/cache from 4.3.0 to 5.0.1 (#3750)
  • 82b8479 Bump actions/download-artifact from 6.0.0 to 7.0.0 (#3749)
  • 34284cb Mention experimental features in the security policy (#3746)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=uv&previous-version=2.5.0&new-version=2.6.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/iceberg-rust/network/alerts).
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- bindings/python/uv.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bindings/python/uv.lock b/bindings/python/uv.lock index b474f6340e..a298afb63c 100644 --- a/bindings/python/uv.lock +++ b/bindings/python/uv.lock @@ -927,9 +927,9 @@ wheels = [ [[package]] name = "urllib3" -version = "2.5.0" +version = "2.6.3" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/15/22/9ee70a2574a4f4599c47dd506532914ce044817c7752a79b6a51286319bc/urllib3-2.5.0.tar.gz", hash = "sha256:3fc47733c7e419d4bc3f6b3dc2b4f890bb743906a30d56ba4a5bfa4bbff92760", size = 393185, upload-time = "2025-06-18T14:07:41.644Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556, upload-time = "2026-01-07T16:24:43.925Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a7/c2/fe1e52489ae3122415c51f387e221dd0773709bad6c6cdaa599e8a2c5185/urllib3-2.5.0-py3-none-any.whl", hash = "sha256:e6b01673c0fa6a13e374b50871808eb3bf7046c4b125b216f6bf1cc604cff0dc", size = 129795, upload-time = "2025-06-18T14:07:40.39Z" }, + { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" }, ] From 118f5a5786831119393c0c2226d05fe432df9e4e Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 24 Feb 2026 19:28:54 -0500 Subject: [PATCH 220/891] ci: Update stale.yml GitHub workflow to handle stale PRs (#2171) Bring Iceberg Java's stale.yml workflow over to handle stale PRs in addition to the existing stale issues behavior. ## Which issue does this PR close? - Closes #2168. ## What changes are included in this PR? - Brought [Iceberg Java's stale.yml](https://github.com/apache/iceberg/blob/main/.github/workflows/stale.yml) over to match that repo's behavior. ## Are these changes tested? Existing CI. --------- Co-authored-by: Kevin Liu --- .github/workflows/stale.yml | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 051e9395fc..e2afce4c71 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -17,7 +17,7 @@ # under the License. # -name: "Close Stale Issues" +name: "Close Stale Issues and PRs" on: schedule: - cron: '0 0 * * *' @@ -25,20 +25,20 @@ on: permissions: # All other permissions are set to none issues: write + pull-requests: write jobs: stale: if: github.repository_owner == 'apache' - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 steps: - uses: actions/stale@v10.2.0 with: - stale-issue-label: 'stale' + # stale issues + stale-issue-label: 'stale,security' exempt-issue-labels: 'not-stale' days-before-issue-stale: 180 days-before-issue-close: 14 - # Only close stale issues, leave PRs alone - days-before-pr-stale: -1 stale-issue-message: > This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To @@ -47,3 +47,12 @@ jobs: close-issue-message: > This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' + # stale PRs + stale-pr-label: 'stale' + exempt-pr-labels: 'not-stale,security' + stale-pr-message: 'This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.' + close-pr-message: 'This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.' + days-before-pr-stale: 30 + days-before-pr-close: 7 + ascending: true + operations-per-run: 200 From 709a244b8a31cc0575b9b197663e5853c48b956b Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Tue, 24 Feb 2026 20:32:42 -0500 Subject: [PATCH 221/891] perf(reader): Add Parquet metadata size hint option to ArrowReaderBuilder (#2173) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Partially address #2172. ## What changes are included in this PR? Add `with_metadata_size_hint` to `ArrowReaderBuilder`, allowing callers to configure the number of bytes to prefetch when reading Parquet footer metadata. No default is set—callers opt in via the builder. When unset, behavior is unchanged. ## Are these changes tested? Existing tests. --- .../iceberg/src/arrow/delete_file_loader.rs | 1 + crates/iceberg/src/arrow/reader.rs | 26 ++++++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index e12daf5324..fa47076fef 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -64,6 +64,7 @@ impl BasicDeleteFileLoader { self.file_io.clone(), false, None, + None, ) .await? .build()? diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index 5b3a7bb862..c4c2fa0036 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -67,6 +67,7 @@ pub struct ArrowReaderBuilder { concurrency_limit_data_files: usize, row_group_filtering_enabled: bool, row_selection_enabled: bool, + metadata_size_hint: Option, } impl ArrowReaderBuilder { @@ -80,6 +81,7 @@ impl ArrowReaderBuilder { concurrency_limit_data_files: num_cpus, row_group_filtering_enabled: true, row_selection_enabled: false, + metadata_size_hint: None, } } @@ -108,6 +110,15 @@ impl ArrowReaderBuilder { self } + /// Provide a hint as to the number of bytes to prefetch for parsing the Parquet metadata + /// + /// This hint can help reduce the number of fetch requests. For more details see the + /// [ParquetMetaDataReader documentation](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataReader.html#method.with_prefetch_hint). + pub fn with_metadata_size_hint(mut self, metadata_size_hint: usize) -> Self { + self.metadata_size_hint = Some(metadata_size_hint); + self + } + /// Build the ArrowReader. pub fn build(self) -> ArrowReader { ArrowReader { @@ -120,6 +131,7 @@ impl ArrowReaderBuilder { concurrency_limit_data_files: self.concurrency_limit_data_files, row_group_filtering_enabled: self.row_group_filtering_enabled, row_selection_enabled: self.row_selection_enabled, + metadata_size_hint: self.metadata_size_hint, } } } @@ -136,6 +148,7 @@ pub struct ArrowReader { row_group_filtering_enabled: bool, row_selection_enabled: bool, + metadata_size_hint: Option, } impl ArrowReader { @@ -147,6 +160,7 @@ impl ArrowReader { let concurrency_limit_data_files = self.concurrency_limit_data_files; let row_group_filtering_enabled = self.row_group_filtering_enabled; let row_selection_enabled = self.row_selection_enabled; + let metadata_size_hint = self.metadata_size_hint; // Fast-path for single concurrency to avoid overhead of try_flatten_unordered let stream: ArrowRecordBatchStream = if concurrency_limit_data_files == 1 { @@ -162,6 +176,7 @@ impl ArrowReader { self.delete_file_loader.clone(), row_group_filtering_enabled, row_selection_enabled, + metadata_size_hint, ) }) .map_err(|err| { @@ -183,6 +198,7 @@ impl ArrowReader { self.delete_file_loader.clone(), row_group_filtering_enabled, row_selection_enabled, + metadata_size_hint, ) }) .map_err(|err| { @@ -205,6 +221,7 @@ impl ArrowReader { delete_file_loader: CachingDeleteFileLoader, row_group_filtering_enabled: bool, row_selection_enabled: bool, + metadata_size_hint: Option, ) -> Result { let should_load_page_index = (row_selection_enabled && task.predicate.is_some()) || !task.deletes.is_empty(); @@ -219,6 +236,7 @@ impl ArrowReader { file_io.clone(), should_load_page_index, None, + metadata_size_hint, ) .await?; @@ -271,6 +289,7 @@ impl ArrowReader { file_io.clone(), should_load_page_index, Some(options), + metadata_size_hint, ) .await? } else { @@ -474,17 +493,22 @@ impl ArrowReader { file_io: FileIO, should_load_page_index: bool, arrow_reader_options: Option, + metadata_size_hint: Option, ) -> Result> { // Get the metadata for the Parquet file we need to read and build // a reader for the data within let parquet_file = file_io.new_input(data_file_path)?; let (parquet_metadata, parquet_reader) = try_join!(parquet_file.metadata(), parquet_file.reader())?; - let parquet_file_reader = ArrowFileReader::new(parquet_metadata, parquet_reader) + let mut parquet_file_reader = ArrowFileReader::new(parquet_metadata, parquet_reader) .with_preload_column_index(true) .with_preload_offset_index(true) .with_preload_page_index(should_load_page_index); + if let Some(hint) = metadata_size_hint { + parquet_file_reader = parquet_file_reader.with_metadata_size_hint(hint); + } + // Create the record batch stream builder, which wraps the parquet file reader let options = arrow_reader_options.unwrap_or_default(); let record_batch_stream_builder = From 8de1826897a0e461e2b459e37fb6d55978a72e59 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Wed, 25 Feb 2026 19:48:06 -0500 Subject: [PATCH 222/891] infra: improve python binding setup (#2178) ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? While reviewing #1997, i noticed a couple of improvements we can make 1. `make install` should install the local editable `pyiceberg-core` so that i can do `make install && make test` 2. CI should fail on warnings 3. test should not read local env files (we made a similar fix in pyiceberg https://github.com/apache/iceberg-python/pull/3006/). Otherwise, tests were reading `~/.pyiceberg.yaml` and polluting the runs ## Are these changes tested? Yes. For (2), see that [it fails in CI](https://github.com/apache/iceberg-rust/actions/runs/22410058261/job/64880767855?pr=2178) before removing the use of deprecated `register_table_provider` --- bindings/python/Makefile | 9 ++++++--- bindings/python/pyproject.toml | 5 +++++ bindings/python/tests/conftest.py | 15 +++++++++++++++ .../tests/test_datafusion_table_provider.py | 4 ++-- 4 files changed, 28 insertions(+), 5 deletions(-) diff --git a/bindings/python/Makefile b/bindings/python/Makefile index c36a1091dd..0e8b9632dc 100644 --- a/bindings/python/Makefile +++ b/bindings/python/Makefile @@ -15,12 +15,15 @@ # specific language governing permissions and limitations # under the License. - -install: - uv sync --group dev --no-install-project +.PHONY: build install test build: uv run maturin develop +# uv sync may remove local editable pyiceberg-core, so install rebuilds it afterwards. +install: + uv sync --group dev --no-install-project + $(MAKE) build + test: uv run --no-sync pytest diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 38efa00b34..9f62b5716d 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -51,6 +51,11 @@ include = [ [tool.ruff.lint] ignore = ["F403", "F405"] +[tool.pytest.ini_options] +filterwarnings = [ + "error", +] + [dependency-groups] dev = [ "maturin>=1.0,<2.0", diff --git a/bindings/python/tests/conftest.py b/bindings/python/tests/conftest.py index 75ee3f36a3..db672c7335 100644 --- a/bindings/python/tests/conftest.py +++ b/bindings/python/tests/conftest.py @@ -25,6 +25,7 @@ import pytest from pydantic_core import to_json +from pyiceberg.utils.config import Config from pyiceberg.partitioning import PartitionField, PartitionSpec from pyiceberg.schema import Schema @@ -36,6 +37,20 @@ ) +@pytest.fixture(scope="session", autouse=True) +def isolate_pyiceberg_config() -> Generator[None, None, None]: + monkeypatch = pytest.MonkeyPatch() + with TemporaryDirectory() as empty_home_dir: + monkeypatch.setenv("HOME", empty_home_dir) + monkeypatch.setenv("PYICEBERG_HOME", empty_home_dir) + + import pyiceberg.catalog as catalog + + monkeypatch.setattr(catalog, "_ENV_CONFIG", Config()) + yield + monkeypatch.undo() + + @pytest.fixture(scope="session") def avro_schema_manifest_entry() -> Dict[str, Any]: return { diff --git a/bindings/python/tests/test_datafusion_table_provider.py b/bindings/python/tests/test_datafusion_table_provider.py index 915b787697..3b2d881e18 100644 --- a/bindings/python/tests/test_datafusion_table_provider.py +++ b/bindings/python/tests/test_datafusion_table_provider.py @@ -106,7 +106,7 @@ def test_register_iceberg_table_provider( ) ctx = SessionContext() - ctx.register_table_provider("test", iceberg_table_provider) + ctx.register_table("test", iceberg_table_provider) datafusion_table = ctx.table("test") assert datafusion_table is not None @@ -154,7 +154,7 @@ def __datafusion_table_provider__(self): ) ctx = SessionContext() - ctx.register_table_provider("test", iceberg_table) + ctx.register_table("test", iceberg_table) datafusion_table = ctx.table("test") assert datafusion_table is not None From 2e28f895326fa5218ae256c6bb1a9ce9cddf1eb2 Mon Sep 17 00:00:00 2001 From: Matt Butrovich Date: Wed, 25 Feb 2026 19:50:36 -0500 Subject: [PATCH 223/891] perf(reader): Pass data file size and delete file size to reader to avoid `stat()` calls (#2175) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? - Partially address #2172. - In a workload I tested, calling `stat()` is taking over 17% of the CPU time in `create_record_batch_stream_builder`: Screenshot 2026-02-24 at 2 03 07 PM ## What changes are included in this PR? - Pass through data file size to `FileScanTask`. Iceberg Java does this by wrapping a reference to a `DataFile` in its `FileScanTask`. In this case we're just cherry-picking some fields until we decide we need more. - Remove redundant data file creation code in tests. ## Are these changes tested? Existing tests. --- .../src/arrow/caching_delete_file_loader.rs | 21 ++- .../iceberg/src/arrow/delete_file_loader.rs | 6 +- crates/iceberg/src/arrow/delete_filter.rs | 22 +++ crates/iceberg/src/arrow/reader.rs | 68 ++++++++- crates/iceberg/src/scan/context.rs | 1 + crates/iceberg/src/scan/mod.rs | 137 +++--------------- crates/iceberg/src/scan/task.rs | 7 + 7 files changed, 135 insertions(+), 127 deletions(-) diff --git a/crates/iceberg/src/arrow/caching_delete_file_loader.rs b/crates/iceberg/src/arrow/caching_delete_file_loader.rs index 5d0b1da712..dfee7ed87b 100644 --- a/crates/iceberg/src/arrow/caching_delete_file_loader.rs +++ b/crates/iceberg/src/arrow/caching_delete_file_loader.rs @@ -235,7 +235,7 @@ impl CachingDeleteFileLoader { PosDelLoadAction::Load => Ok(DeleteFileContext::PosDels { file_path: task.file_path.clone(), stream: basic_delete_file_loader - .parquet_to_batch_stream(&task.file_path) + .parquet_to_batch_stream(&task.file_path, task.file_size_in_bytes) .await?, }), } @@ -254,7 +254,7 @@ impl CachingDeleteFileLoader { let equality_ids_vec = task.equality_ids.clone().unwrap(); let evolved_stream = BasicDeleteFileLoader::evolve_schema( basic_delete_file_loader - .parquet_to_batch_stream(&task.file_path) + .parquet_to_batch_stream(&task.file_path, task.file_size_in_bytes) .await?, schema, &equality_ids_vec, @@ -614,7 +614,10 @@ mod tests { let basic_delete_file_loader = BasicDeleteFileLoader::new(file_io.clone()); let record_batch_stream = basic_delete_file_loader - .parquet_to_batch_stream(&eq_delete_file_path) + .parquet_to_batch_stream( + &eq_delete_file_path, + std::fs::metadata(&eq_delete_file_path).unwrap().len(), + ) .await .expect("could not get batch stream"); @@ -811,7 +814,10 @@ mod tests { let basic_delete_file_loader = BasicDeleteFileLoader::new(file_io.clone()); let batch_stream = basic_delete_file_loader - .parquet_to_batch_stream(&delete_file_path) + .parquet_to_batch_stream( + &delete_file_path, + std::fs::metadata(&delete_file_path).unwrap().len(), + ) .await .unwrap(); @@ -913,7 +919,8 @@ mod tests { // Create FileScanTask with BOTH positional and equality deletes let pos_del = FileScanTaskDeleteFile { - file_path: pos_del_path, + file_path: pos_del_path.clone(), + file_size_in_bytes: std::fs::metadata(&pos_del_path).unwrap().len(), file_type: DataContentType::PositionDeletes, partition_spec_id: 0, equality_ids: None, @@ -921,12 +928,14 @@ mod tests { let eq_del = FileScanTaskDeleteFile { file_path: eq_delete_path.clone(), + file_size_in_bytes: std::fs::metadata(&eq_delete_path).unwrap().len(), file_type: DataContentType::EqualityDeletes, partition_spec_id: 0, equality_ids: Some(vec![2, 3]), // Only use field IDs that exist in both schemas }; let file_scan_task = FileScanTask { + file_size_in_bytes: 0, start: 0, length: 0, record_count: None, @@ -993,7 +1002,7 @@ mod tests { let basic_delete_file_loader = BasicDeleteFileLoader::new(file_io.clone()); let record_batch_stream = basic_delete_file_loader - .parquet_to_batch_stream(&path) + .parquet_to_batch_stream(&path, std::fs::metadata(&path).unwrap().len()) .await .expect("could not get batch stream"); diff --git a/crates/iceberg/src/arrow/delete_file_loader.rs b/crates/iceberg/src/arrow/delete_file_loader.rs index fa47076fef..33744d876f 100644 --- a/crates/iceberg/src/arrow/delete_file_loader.rs +++ b/crates/iceberg/src/arrow/delete_file_loader.rs @@ -54,6 +54,7 @@ impl BasicDeleteFileLoader { pub(crate) async fn parquet_to_batch_stream( &self, data_file_path: &str, + file_size_in_bytes: u64, ) -> Result { /* Essentially a super-cut-down ArrowReader. We can't use ArrowReader directly @@ -65,6 +66,7 @@ impl BasicDeleteFileLoader { false, None, None, + file_size_in_bytes, ) .await? .build()? @@ -102,7 +104,9 @@ impl DeleteFileLoader for BasicDeleteFileLoader { task: &FileScanTaskDeleteFile, schema: SchemaRef, ) -> Result { - let raw_batch_stream = self.parquet_to_batch_stream(&task.file_path).await?; + let raw_batch_stream = self + .parquet_to_batch_stream(&task.file_path, task.file_size_in_bytes) + .await?; // For equality deletes, only evolve the equality_ids columns. // For positional deletes (equality_ids is None), use all field IDs. diff --git a/crates/iceberg/src/arrow/delete_filter.rs b/crates/iceberg/src/arrow/delete_filter.rs index 4af9f6b6ff..3d074ca32e 100644 --- a/crates/iceberg/src/arrow/delete_filter.rs +++ b/crates/iceberg/src/arrow/delete_filter.rs @@ -378,6 +378,12 @@ pub(crate) mod tests { let pos_del_1 = FileScanTaskDeleteFile { file_path: format!("{}/pos-del-1.parquet", table_location.to_str().unwrap()), + file_size_in_bytes: std::fs::metadata(format!( + "{}/pos-del-1.parquet", + table_location.to_str().unwrap() + )) + .unwrap() + .len(), file_type: DataContentType::PositionDeletes, partition_spec_id: 0, equality_ids: None, @@ -385,6 +391,12 @@ pub(crate) mod tests { let pos_del_2 = FileScanTaskDeleteFile { file_path: format!("{}/pos-del-2.parquet", table_location.to_str().unwrap()), + file_size_in_bytes: std::fs::metadata(format!( + "{}/pos-del-2.parquet", + table_location.to_str().unwrap() + )) + .unwrap() + .len(), file_type: DataContentType::PositionDeletes, partition_spec_id: 0, equality_ids: None, @@ -392,6 +404,12 @@ pub(crate) mod tests { let pos_del_3 = FileScanTaskDeleteFile { file_path: format!("{}/pos-del-3.parquet", table_location.to_str().unwrap()), + file_size_in_bytes: std::fs::metadata(format!( + "{}/pos-del-3.parquet", + table_location.to_str().unwrap() + )) + .unwrap() + .len(), file_type: DataContentType::PositionDeletes, partition_spec_id: 0, equality_ids: None, @@ -399,6 +417,7 @@ pub(crate) mod tests { let file_scan_tasks = vec![ FileScanTask { + file_size_in_bytes: 0, start: 0, length: 0, record_count: None, @@ -414,6 +433,7 @@ pub(crate) mod tests { case_sensitive: false, }, FileScanTask { + file_size_in_bytes: 0, start: 0, length: 0, record_count: None, @@ -464,6 +484,7 @@ pub(crate) mod tests { // ---------- fake FileScanTask ---------- let task = FileScanTask { + file_size_in_bytes: 0, start: 0, length: 0, record_count: None, @@ -474,6 +495,7 @@ pub(crate) mod tests { predicate: None, deletes: vec![FileScanTaskDeleteFile { file_path: "eq-del.parquet".to_string(), + file_size_in_bytes: 1, // never read; this test fails before opening the file file_type: DataContentType::EqualityDeletes, partition_spec_id: 0, equality_ids: None, diff --git a/crates/iceberg/src/arrow/reader.rs b/crates/iceberg/src/arrow/reader.rs index c4c2fa0036..93dbdaa35d 100644 --- a/crates/iceberg/src/arrow/reader.rs +++ b/crates/iceberg/src/arrow/reader.rs @@ -33,7 +33,7 @@ use arrow_string::like::starts_with; use bytes::Bytes; use fnv::FnvHashSet; use futures::future::BoxFuture; -use futures::{FutureExt, StreamExt, TryFutureExt, TryStreamExt, try_join}; +use futures::{FutureExt, StreamExt, TryFutureExt, TryStreamExt}; use parquet::arrow::arrow_reader::{ ArrowPredicateFn, ArrowReaderOptions, RowFilter, RowSelection, RowSelector, }; @@ -237,6 +237,7 @@ impl ArrowReader { should_load_page_index, None, metadata_size_hint, + task.file_size_in_bytes, ) .await?; @@ -290,6 +291,7 @@ impl ArrowReader { should_load_page_index, Some(options), metadata_size_hint, + task.file_size_in_bytes, ) .await? } else { @@ -494,16 +496,21 @@ impl ArrowReader { should_load_page_index: bool, arrow_reader_options: Option, metadata_size_hint: Option, + file_size_in_bytes: u64, ) -> Result> { // Get the metadata for the Parquet file we need to read and build // a reader for the data within let parquet_file = file_io.new_input(data_file_path)?; - let (parquet_metadata, parquet_reader) = - try_join!(parquet_file.metadata(), parquet_file.reader())?; - let mut parquet_file_reader = ArrowFileReader::new(parquet_metadata, parquet_reader) - .with_preload_column_index(true) - .with_preload_offset_index(true) - .with_preload_page_index(should_load_page_index); + let parquet_reader = parquet_file.reader().await?; + let mut parquet_file_reader = ArrowFileReader::new( + FileMetadata { + size: file_size_in_bytes, + }, + parquet_reader, + ) + .with_preload_column_index(true) + .with_preload_offset_index(true) + .with_preload_page_index(should_load_page_index); if let Some(hint) = metadata_size_hint { parquet_file_reader = parquet_file_reader.with_metadata_size_hint(hint); @@ -2121,6 +2128,9 @@ message schema { ) -> Vec> { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -2443,6 +2453,7 @@ message schema { // Task 1: read only the first row group let task1 = FileScanTask { + file_size_in_bytes: std::fs::metadata(&file_path).unwrap().len(), start: rg0_start, length: row_group_0.compressed_size() as u64, record_count: Some(100), @@ -2460,6 +2471,7 @@ message schema { // Task 2: read the second and third row groups let task2 = FileScanTask { + file_size_in_bytes: std::fs::metadata(&file_path).unwrap().len(), start: rg1_start, length: file_end - rg1_start, record_count: Some(200), @@ -2588,6 +2600,9 @@ message schema { let reader = ArrowReaderBuilder::new(file_io).build(); let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/old_file.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -2755,6 +2770,7 @@ message schema { let reader = ArrowReaderBuilder::new(file_io).build(); let task = FileScanTask { + file_size_in_bytes: std::fs::metadata(&data_file_path).unwrap().len(), start: 0, length: 0, record_count: Some(200), @@ -2764,6 +2780,7 @@ message schema { project_field_ids: vec![1], predicate: None, deletes: vec![FileScanTaskDeleteFile { + file_size_in_bytes: std::fs::metadata(&delete_file_path).unwrap().len(), file_path: delete_file_path, file_type: DataContentType::PositionDeletes, partition_spec_id: 0, @@ -2973,6 +2990,7 @@ message schema { // Create FileScanTask that reads ONLY row group 1 via byte range filtering let task = FileScanTask { + file_size_in_bytes: std::fs::metadata(&data_file_path).unwrap().len(), start: rg1_start, length: rg1_length, record_count: Some(100), // Row group 1 has 100 rows @@ -2982,6 +3000,7 @@ message schema { project_field_ids: vec![1], predicate: None, deletes: vec![FileScanTaskDeleteFile { + file_size_in_bytes: std::fs::metadata(&delete_file_path).unwrap().len(), file_path: delete_file_path, file_type: DataContentType::PositionDeletes, partition_spec_id: 0, @@ -3184,6 +3203,7 @@ message schema { // Create FileScanTask that reads ONLY row group 1 via byte range filtering let task = FileScanTask { + file_size_in_bytes: std::fs::metadata(&data_file_path).unwrap().len(), start: rg1_start, length: rg1_length, record_count: Some(100), // Row group 1 has 100 rows @@ -3193,6 +3213,7 @@ message schema { project_field_ids: vec![1], predicate: None, deletes: vec![FileScanTaskDeleteFile { + file_size_in_bytes: std::fs::metadata(&delete_file_path).unwrap().len(), file_path: delete_file_path, file_type: DataContentType::PositionDeletes, partition_spec_id: 0, @@ -3293,6 +3314,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3391,6 +3415,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3478,6 +3505,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3579,6 +3609,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3709,6 +3742,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3806,6 +3842,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -3916,6 +3955,9 @@ message schema { let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -4007,6 +4049,9 @@ message schema { // Create tasks in a specific order: file_0, file_1, file_2 let tasks = vec![ Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/file_0.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -4022,6 +4067,9 @@ message schema { case_sensitive: false, }), Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/file_1.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -4037,6 +4085,9 @@ message schema { case_sensitive: false, }), Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/file_2.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, @@ -4216,6 +4267,9 @@ message schema { let reader = ArrowReaderBuilder::new(file_io).build(); let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask { + file_size_in_bytes: std::fs::metadata(format!("{table_location}/data.parquet")) + .unwrap() + .len(), start: 0, length: 0, record_count: None, diff --git a/crates/iceberg/src/scan/context.rs b/crates/iceberg/src/scan/context.rs index 169d8e6405..aa28ffd5a2 100644 --- a/crates/iceberg/src/scan/context.rs +++ b/crates/iceberg/src/scan/context.rs @@ -117,6 +117,7 @@ impl ManifestEntryContext { .await; Ok(FileScanTask { + file_size_in_bytes: self.manifest_entry.file_size_in_bytes(), start: 0, length: self.manifest_entry.file_size_in_bytes(), record_count: Some(self.manifest_entry.record_count()), diff --git a/crates/iceberg/src/scan/mod.rs b/crates/iceberg/src/scan/mod.rs index c055c12c9a..2a685cd171 100644 --- a/crates/iceberg/src/scan/mod.rs +++ b/crates/iceberg/src/scan/mod.rs @@ -756,7 +756,9 @@ pub mod tests { let current_schema = current_snapshot.schema(self.table.metadata()).unwrap(); let current_partition_spec = self.table.metadata().default_partition_spec(); - // Write data files + // Write the data files first, then use the file size in the manifest entries + let parquet_file_size = self.write_parquet_data_files(); + let mut writer = ManifestWriterBuilder::new( self.next_manifest_file(), Some(current_snapshot.snapshot_id()), @@ -775,7 +777,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/1.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(Struct::from_iter([Some(Literal::long(100))])) .key_metadata(None) @@ -798,7 +800,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/2.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(Struct::from_iter([Some(Literal::long(200))])) .build() @@ -820,7 +822,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/3.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(Struct::from_iter([Some(Literal::long(300))])) .build() @@ -845,8 +847,13 @@ pub mod tests { .add_manifests(vec![data_file_manifest].into_iter()) .unwrap(); manifest_list_write.close().await.unwrap(); + } + + /// Writes identical Parquet data files (1.parquet, 2.parquet, 3.parquet) + /// and returns the file size in bytes. + fn write_parquet_data_files(&self) -> u64 { + std::fs::create_dir_all(&self.table_location).unwrap(); - // prepare data let schema = { let fields = vec![ arrow_schema::Field::new("x", arrow_schema::DataType::Int64, false) @@ -958,6 +965,10 @@ pub mod tests { // writer must be closed to write footer writer.close().unwrap(); } + + std::fs::metadata(format!("{}/1.parquet", &self.table_location)) + .unwrap() + .len() } pub async fn setup_unpartitioned_manifest_files(&mut self) { @@ -968,6 +979,9 @@ pub mod tests { let current_schema = current_snapshot.schema(self.table.metadata()).unwrap(); let current_partition_spec = Arc::new(PartitionSpec::unpartition_spec()); + // Write the data files first, then use the file size in the manifest entries + let parquet_file_size = self.write_parquet_data_files(); + // Write data files using an empty partition for unpartitioned tables. let mut writer = ManifestWriterBuilder::new( self.next_manifest_file(), @@ -991,7 +1005,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/1.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(empty_partition.clone()) .key_metadata(None) @@ -1015,7 +1029,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/2.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(empty_partition.clone()) .build() @@ -1038,7 +1052,7 @@ pub mod tests { .content(DataContentType::Data) .file_path(format!("{}/3.parquet", &self.table_location)) .file_format(DataFileFormat::Parquet) - .file_size_in_bytes(100) + .file_size_in_bytes(parquet_file_size) .record_count(1) .partition(empty_partition.clone()) .build() @@ -1064,111 +1078,6 @@ pub mod tests { .add_manifests(vec![data_file_manifest].into_iter()) .unwrap(); manifest_list_write.close().await.unwrap(); - - // prepare data for parquet files - let schema = { - let fields = vec![ - arrow_schema::Field::new("x", arrow_schema::DataType::Int64, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "1".to_string(), - )])), - arrow_schema::Field::new("y", arrow_schema::DataType::Int64, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "2".to_string(), - )])), - arrow_schema::Field::new("z", arrow_schema::DataType::Int64, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "3".to_string(), - )])), - arrow_schema::Field::new("a", arrow_schema::DataType::Utf8, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "4".to_string(), - )])), - arrow_schema::Field::new("dbl", arrow_schema::DataType::Float64, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "5".to_string(), - )])), - arrow_schema::Field::new("i32", arrow_schema::DataType::Int32, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "6".to_string(), - )])), - arrow_schema::Field::new("i64", arrow_schema::DataType::Int64, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "7".to_string(), - )])), - arrow_schema::Field::new("bool", arrow_schema::DataType::Boolean, false) - .with_metadata(HashMap::from([( - PARQUET_FIELD_ID_META_KEY.to_string(), - "8".to_string(), - )])), - ]; - Arc::new(arrow_schema::Schema::new(fields)) - }; - - // Build the arrays for the RecordBatch - let col1 = Arc::new(Int64Array::from_iter_values(vec![1; 1024])) as ArrayRef; - - let mut values = vec![2; 512]; - values.append(vec![3; 200].as_mut()); - values.append(vec![4; 300].as_mut()); - values.append(vec![5; 12].as_mut()); - let col2 = Arc::new(Int64Array::from_iter_values(values)) as ArrayRef; - - let mut values = vec![3; 512]; - values.append(vec![4; 512].as_mut()); - let col3 = Arc::new(Int64Array::from_iter_values(values)) as ArrayRef; - - let mut values = vec!["Apache"; 512]; - values.append(vec!["Iceberg"; 512].as_mut()); - let col4 = Arc::new(StringArray::from_iter_values(values)) as ArrayRef; - - let mut values = vec![100.0f64; 512]; - values.append(vec![150.0f64; 12].as_mut()); - values.append(vec![200.0f64; 500].as_mut()); - let col5 = Arc::new(Float64Array::from_iter_values(values)) as ArrayRef; - - let mut values = vec![100i32; 512]; - values.append(vec![150i32; 12].as_mut()); - values.append(vec![200i32; 500].as_mut()); - let col6 = Arc::new(Int32Array::from_iter_values(values)) as ArrayRef; - - let mut values = vec![100i64; 512]; - values.append(vec![150i64; 12].as_mut()); - values.append(vec![200i64; 500].as_mut()); - let col7 = Arc::new(Int64Array::from_iter_values(values)) as ArrayRef; - - let mut values = vec![false; 512]; - values.append(vec![true; 512].as_mut()); - let values: BooleanArray = values.into(); - let col8 = Arc::new(values) as ArrayRef; - - let to_write = RecordBatch::try_new(schema.clone(), vec![ - col1, col2, col3, col4, col5, col6, col7, col8, - ]) - .unwrap(); - - // Write the Parquet files - let props = WriterProperties::builder() - .set_compression(Compression::SNAPPY) - .build(); - - for n in 1..=3 { - let file = File::create(format!("{}/{}.parquet", &self.table_location, n)).unwrap(); - let mut writer = - ArrowWriter::try_new(file, to_write.schema(), Some(props.clone())).unwrap(); - - writer.write(&to_write).expect("Writing batch"); - - // writer must be closed to write footer - writer.close().unwrap(); - } } pub async fn setup_deadlock_manifests(&mut self) { @@ -1874,6 +1783,7 @@ pub mod tests { ); let task = FileScanTask { data_file_path: "data_file_path".to_string(), + file_size_in_bytes: 0, start: 0, length: 100, project_field_ids: vec![1, 2, 3], @@ -1892,6 +1802,7 @@ pub mod tests { // with predicate let task = FileScanTask { data_file_path: "data_file_path".to_string(), + file_size_in_bytes: 0, start: 0, length: 100, project_field_ids: vec![1, 2, 3], diff --git a/crates/iceberg/src/scan/task.rs b/crates/iceberg/src/scan/task.rs index 5349a9bdd2..67615c351e 100644 --- a/crates/iceberg/src/scan/task.rs +++ b/crates/iceberg/src/scan/task.rs @@ -51,6 +51,9 @@ where D: serde::Deserializer<'de> { /// A task to scan part of file. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct FileScanTask { + /// The total size of the data file in bytes, from the manifest entry. + /// Used to skip a stat/HEAD request when reading Parquet footers. + pub file_size_in_bytes: u64, /// The start offset of the file to scan. pub start: u64, /// The length of the file to scan. @@ -146,6 +149,7 @@ impl From<&DeleteFileContext> for FileScanTaskDeleteFile { fn from(ctx: &DeleteFileContext) -> Self { FileScanTaskDeleteFile { file_path: ctx.manifest_entry.file_path().to_string(), + file_size_in_bytes: ctx.manifest_entry.file_size_in_bytes(), file_type: ctx.manifest_entry.content_type(), partition_spec_id: ctx.partition_spec_id, equality_ids: ctx.manifest_entry.data_file.equality_ids.clone(), @@ -159,6 +163,9 @@ pub struct FileScanTaskDeleteFile { /// The delete file path pub file_path: String, + /// The total size of the delete file in bytes, from the manifest entry. + pub file_size_in_bytes: u64, + /// delete file type pub file_type: DataContentType, From 4f5274a8afd835e746167e48683207b71b20115a Mon Sep 17 00:00:00 2001 From: Kaiqi Dong Date: Thu, 26 Feb 2026 02:38:34 +0100 Subject: [PATCH 224/891] feat(datafusion): support isnan predicate pushdown to Iceberg (#2142) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Which issue does this PR close? Part of the ongoing effort to improve predicate pushdown coverage in the DataFusion integration. - Closes #2143 ## What changes are included in this PR? This PR adds support for pushing down `isnan()` predicates from DataFusion to Iceberg's native `IsNan` / `NotNan` predicate operators. In DataFusion, `isnan()` is represented as a scalar function (`Expr::ScalarFunction`) rather than a dedicated `Expr` variant (unlike `IsNull` / `IsNotNull`). This PR introduces a new `scalar_function_to_iceberg_predicate` helper in `expr_to_predicate.rs` that matches scalar functions by name at runtime and converts `isnan(col)` into `Predicate::Unary(IsNan, col)`. Negation (`NOT isnan(col)`) is handled automatically: the existing `Expr::Not` arm wraps the result in `Predicate::Not(...)`, and Iceberg's downstream `rewrite_not` visitor normalizes it into `Predicate::Unary(NotNan, col)`. This enables file pruning using `nan_value_counts` in manifest metadata for float/double columns, as defined in the [Iceberg spec — Manifest Files: field summaries and column statistics](https://iceberg.apache.org/spec/#manifests). ## Are these changes tested? Yes --- .../src/physical_plan/expr_to_predicate.rs | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs index 9f37345f86..17c9416d54 100644 --- a/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs +++ b/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs @@ -18,6 +18,7 @@ use std::vec; use datafusion::arrow::datatypes::DataType; +use datafusion::logical_expr::expr::ScalarFunction; use datafusion::logical_expr::{Expr, Like, Operator}; use datafusion::scalar::ScalarValue; use iceberg::expr::{BinaryExpression, Predicate, PredicateOperator, Reference, UnaryExpression}; @@ -196,6 +197,9 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult { TransformedResult::NotTransformed } } + Expr::ScalarFunction(ScalarFunction { func, args }) => { + scalar_function_to_iceberg_predicate(func.name(), args) + } _ => TransformedResult::NotTransformed, } } @@ -216,6 +220,25 @@ fn to_iceberg_operation(op: Operator) -> OpTransformedResult { } } +/// Translates a DataFusion scalar function into an Iceberg predicate. +/// Unlike dedicated Expr variants (e.g. `Expr::IsNull`), scalar functions are +/// identified by name at runtime, so we need to handle them here. +fn scalar_function_to_iceberg_predicate(func_name: &str, args: &[Expr]) -> TransformedResult { + match func_name { + // TODO: support complex expression arguments to scalar functions + "isnan" if args.len() == 1 => { + let operand = to_iceberg_predicate(&args[0]); + match operand { + TransformedResult::Column(r) => TransformedResult::Predicate(Predicate::Unary( + UnaryExpression::new(PredicateOperator::IsNan, r), + )), + _ => TransformedResult::NotTransformed, + } + } + _ => TransformedResult::NotTransformed, + } +} + fn to_iceberg_and_predicate( left: TransformedResult, right: TransformedResult, @@ -324,6 +347,10 @@ mod tests { Field::new("ts", DataType::Timestamp(TimeUnit::Second, None), true).with_metadata( HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), "3".to_string())]), ), + Field::new("qux", DataType::Float64, true).with_metadata(HashMap::from([( + PARQUET_FIELD_ID_META_KEY.to_string(), + "4".to_string(), + )])), ]); DFSchema::try_from_qualified_schema("my_table", &arrow_schema).unwrap() } @@ -681,4 +708,35 @@ mod tests { Reference::new("bar").starts_with(Datum::string("测试")) ); } + + #[test] + fn test_predicate_conversion_with_isnan() { + let predicate = convert_to_iceberg_predicate("isnan(qux)").unwrap(); + assert_eq!(predicate, Reference::new("qux").is_nan()); + } + + #[test] + fn test_predicate_conversion_with_not_isnan() { + let predicate = convert_to_iceberg_predicate("NOT isnan(qux)").unwrap(); + assert_eq!(predicate, !Reference::new("qux").is_nan()); + } + + #[test] + fn test_predicate_conversion_with_isnan_and_other_condition() { + let sql = "isnan(qux) AND foo > 1"; + let predicate = convert_to_iceberg_predicate(sql).unwrap(); + let expected_predicate = Predicate::and( + Reference::new("qux").is_nan(), + Reference::new("foo").greater_than(Datum::long(1)), + ); + assert_eq!(predicate, expected_predicate); + } + + #[test] + fn test_predicate_conversion_with_isnan_unsupported_arg() { + // isnan on a complex expression (not a bare column) cannot be pushed down + let sql = "isnan(qux + 1)"; + let predicate = convert_to_iceberg_predicate(sql); + assert_eq!(predicate, None); + } } From b24ab6310235f71907f4b6b6dc14a8e5d9291acc Mon Sep 17 00:00:00 2001 From: Ethan Urbanski Date: Thu, 26 Feb 2026 15:11:42 -0500 Subject: [PATCH 225/891] chore(deps): upgrade to DataFusion 52 (#1997) ## Which issue does this PR close? Validates and adopts DataFusion 52 ## What changes are included in this PR? - Upgrade DataFusion integration from 51.x to 52.x - Keep the DataFusion Python dependency dynamic within major version 52 - Update the Python FFI table provider bridge for DataFusion 52 API/ABI expectations: - session-aware `__datafusion_table_provider__(session)` integration - DF52 compatible FFI table provider construction with task context and logical codec handling - Update sqllogictest physical plan expectations for DataFusion 52 planner output changes - Refresh lockfiles impacted by the upgrade ### DataFusion FFI API change DataFusion 52 expanded table provider FFI construction to include task context and optional logical codec parameters. Updated the Rust/Python bridge accordingly allowing filter/logical expression serialization remains compatible across the FFI boundary. ## Are these changes tested? Yes --------- Signed-off-by: Ethan Urbanski --- Cargo.lock | 1268 ++++++----------- Cargo.toml | 26 +- bindings/python/Cargo.lock | 587 ++++---- bindings/python/Cargo.toml | 4 +- bindings/python/DEPENDENCIES.rust.tsv | 143 +- bindings/python/pyproject.toml | 2 +- .../python/src/datafusion_table_provider.rs | 52 +- .../tests/test_datafusion_table_provider.py | 13 +- bindings/python/tests/test_manifest.py | 78 +- bindings/python/uv.lock | 344 +++-- .../df_test/binary_predicate_pushdown.slt | 9 +- .../df_test/boolean_predicate_pushdown.slt | 27 +- .../slts/df_test/like_predicate_pushdown.slt | 18 +- .../df_test/timestamp_predicate_pushdown.slt | 37 +- 14 files changed, 1169 insertions(+), 1439 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 8c1ff336a6..f3f32a2457 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -19,17 +19,6 @@ dependencies = [ "cpufeatures", ] -[[package]] -name = "ahash" -version = "0.7.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9" -dependencies = [ - "getrandom 0.2.16", - "once_cell", - "version_check", -] - [[package]] name = "ahash" version = "0.8.12" @@ -120,7 +109,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -131,7 +120,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -140,35 +129,6 @@ version = "1.0.100" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61" -[[package]] -name = "apache-avro" -version = "0.20.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a033b4ced7c585199fb78ef50fca7fe2f444369ec48080c5fd072efa1a03cc7" -dependencies = [ - "bigdecimal", - "bon", - "bzip2 0.6.1", - "crc32fast", - "digest", - "log", - "miniz_oxide", - "num-bigint", - "quad-rand", - "rand 0.9.2", - "regex-lite", - "serde", - "serde_bytes", - "serde_json", - "snap", - "strum", - "strum_macros", - "thiserror 2.0.17", - "uuid", - "xz2", - "zstd", -] - [[package]] name = "apache-avro" version = "0.21.0" @@ -177,7 +137,10 @@ checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf" dependencies = [ "bigdecimal", "bon", + "bzip2", + "crc32fast", "digest", + "liblzma", "log", "miniz_oxide", "num-bigint", @@ -187,6 +150,7 @@ dependencies = [ "serde", "serde_bytes", "serde_json", + "snap", "strum", "strum_macros", "thiserror 2.0.17", @@ -244,9 +208,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7b3141e0ec5145a22d8694ea8b6d6f69305971c4fa1c1a13ef0195aef2d678b" +checksum = "288015089e7931843c80ed4032c5274f02b37bcb720c4a42096d50b390e70372" dependencies = [ "arrow-array", "arrow-buffer", @@ -258,11 +222,11 @@ dependencies = [ [[package]] name = "arrow-array" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c8955af33b25f3b175ee10af580577280b4bd01f7e823d94c7cdef7cf8c9aef" +checksum = "65ca404ea6191e06bf30956394173337fa9c35f445bd447fe6c21ab944e1a23c" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-buffer", "arrow-data", "arrow-schema", @@ -277,9 +241,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c697ddca96183182f35b3a18e50b9110b11e916d7b7799cbfd4d34662f2c56c2" +checksum = "36356383099be0151dacc4245309895f16ba7917d79bdb71a7148659c9206c56" dependencies = [ "bytes", "half", @@ -289,9 +253,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "646bbb821e86fd57189c10b4fcdaa941deaf4181924917b0daa92735baa6ada5" +checksum = "9c8e372ed52bd4ee88cc1e6c3859aa7ecea204158ac640b10e187936e7e87074" dependencies = [ "arrow-array", "arrow-buffer", @@ -326,9 +290,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fdd994a9d28e6365aa78e15da3f3950c0fdcea6b963a12fa1c391afb637b304" +checksum = "bf87f4ff5fc13290aa47e499a8b669a82c5977c6a1fedce22c7f542c1fd5a597" dependencies = [ "arrow-buffer", "arrow-schema", @@ -379,9 +343,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7d8f1870e03d4cbed632959498bcc84083b5a24bded52905ae1695bd29da45b" +checksum = "13c4e0530272ca755d6814218dffd04425c5b7854b87fa741d5ff848bf50aa39" dependencies = [ "arrow-array", "arrow-buffer", @@ -405,9 +369,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c872d36b7bf2a6a6a2b40de9156265f0242910791db366a2c17476ba8330d68" +checksum = "6bb63203e8e0e54b288d0d8043ca8fa1013820822a27692ef1b78a977d879f2c" dependencies = [ "serde_core", "serde_json", @@ -415,11 +379,11 @@ dependencies = [ [[package]] name = "arrow-select" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68bf3e3efbd1278f770d67e5dc410257300b161b93baedb3aae836144edcaf4b" +checksum = "c96d8a1c180b44ecf2e66c9a2f2bbcb8b1b6f14e165ce46ac8bde211a363411b" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-array", "arrow-buffer", "arrow-data", @@ -429,9 +393,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "57.3.0" +version = "57.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85e968097061b3c0e9fe3079cf2e703e487890700546b5b0647f60fca1b5a8d8" +checksum = "a8ad6a81add9d3ea30bf8374ee8329992c7fd246ffd8b7e2f48a3cea5aa0cc9a" dependencies = [ "arrow-array", "arrow-buffer", @@ -474,26 +438,22 @@ dependencies = [ [[package]] name = "async-compression" -version = "0.4.19" +version = "0.4.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06575e6a9673580f52661c92107baabffbf41e2141373441cbcdc47cb733003c" +checksum = "98ec5f6c2f8bc326c994cb9e241cc257ddaba9afa8555a43cffbb5dd86efaa37" dependencies = [ - "bzip2 0.5.2", - "flate2", + "compression-codecs", + "compression-core", "futures-core", - "memchr", "pin-project-lite", "tokio", - "xz2", - "zstd", - "zstd-safe", ] [[package]] name = "async-lock" -version = "3.4.1" +version = "3.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fd03604047cee9b6ce9de9f70c6cd540a0520c813cbd49bae61f33ab80ed1dc" +checksum = "290f7f2596bd5b78a9fec8088ccd89180d7f9f55b94b0576823bbbdc72ee8311" dependencies = [ "event-listener", "event-listener-strategy", @@ -508,7 +468,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -519,7 +479,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -545,9 +505,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-config" -version = "1.8.11" +version = "1.8.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0149602eeaf915158e14029ba0c78dedb8c08d554b024d54c8f239aab46511d" +checksum = "96571e6996817bf3d58f6b569e4b9fd2e9d2fcf9f7424eed07b2ce9bb87535e5" dependencies = [ "aws-credential-types", "aws-runtime", @@ -575,9 +535,9 @@ dependencies = [ [[package]] name = "aws-credential-types" -version = "1.2.10" +version = "1.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b01c9521fa01558f750d183c8c68c81b0155b9d193a4ba7f84c36bd1b6d04a06" +checksum = "3cd362783681b15d136480ad555a099e82ecd8e2d10a841e14dfd0078d67fee3" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -587,9 +547,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.15.1" +version = "1.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b5ce75405893cd713f9ab8e297d8e438f624dde7d706108285f7e17a25a180f" +checksum = "6a88aab2464f1f25453baa7a07c84c5b7684e274054ba06817f382357f77a288" dependencies = [ "aws-lc-sys", "zeroize", @@ -597,9 +557,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.34.0" +version = "0.35.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "179c3777a8b5e70e90ea426114ffc565b2c1a9f82f6c4a0c5a34aa6ef5e781b6" +checksum = "b45afffdee1e7c9126814751f88dddc747f41d91da16c9551a0f1e8a11e788a1" dependencies = [ "cc", "cmake", @@ -609,9 +569,9 @@ dependencies = [ [[package]] name = "aws-runtime" -version = "1.5.16" +version = "1.5.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ce527fb7e53ba9626fc47824f25e256250556c40d8f81d27dd92aa38239d632" +checksum = "d81b5b2898f6798ad58f484856768bca817e3cd9de0974c24ae0f1113fe88f1b" dependencies = [ "aws-credential-types", "aws-sigv4", @@ -633,9 +593,9 @@ dependencies = [ [[package]] name = "aws-sdk-glue" -version = "1.132.0" +version = "1.133.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35638d8e6ef97adb7f3154ffc618bbe1d631a503b6d8328b94af77b7615fbeb2" +checksum = "46e5b45757688d5c99ef0ab38d064fefcc46ee7923fb6d7b23b035323e300c7d" dependencies = [ "aws-credential-types", "aws-runtime", @@ -655,9 +615,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3tables" -version = "1.46.0" +version = "1.47.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce68b5d4652e6248827e472c67df8773ae6ab3946ff176de8d3ee7c295299efd" +checksum = "a4252a364a800f00ce90e0fba31906ea812f263ab9f72c08ac5517eb8e33fb5a" dependencies = [ "aws-credential-types", "aws-runtime", @@ -677,9 +637,9 @@ dependencies = [ [[package]] name = "aws-sdk-sso" -version = "1.90.0" +version = "1.91.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f18e53542c522459e757f81e274783a78f8c81acdfc8d1522ee8a18b5fb1c66" +checksum = "8ee6402a36f27b52fe67661c6732d684b2635152b676aa2babbfb5204f99115d" dependencies = [ "aws-credential-types", "aws-runtime", @@ -699,9 +659,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.92.0" +version = "1.93.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "532f4d866012ffa724a4385c82e8dd0e59f0ca0e600f3f22d4c03b6824b34e4a" +checksum = "a45a7f750bbd170ee3677671ad782d90b894548f4e4ae168302c57ec9de5cb3e" dependencies = [ "aws-credential-types", "aws-runtime", @@ -721,9 +681,9 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.94.0" +version = "1.95.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1be6fbbfa1a57724788853a623378223fe828fc4c09b146c992f0c95b6256174" +checksum = "55542378e419558e6b1f398ca70adb0b2088077e79ad9f14eb09441f2f7b2164" dependencies = [ "aws-credential-types", "aws-runtime", @@ -744,9 +704,9 @@ dependencies = [ [[package]] name = "aws-sigv4" -version = "1.3.6" +version = "1.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c35452ec3f001e1f2f6db107b6373f1f48f05ec63ba2c5c9fa91f07dad32af11" +checksum = "69e523e1c4e8e7e8ff219d732988e22bfeae8a1cafdbe6d9eca1546fa080be7c" dependencies = [ "aws-credential-types", "aws-smithy-http", @@ -806,7 +766,7 @@ dependencies = [ "aws-smithy-runtime-api", "aws-smithy-types", "h2 0.3.27", - "h2 0.4.12", + "h2 0.4.13", "http 0.2.12", "http 1.4.0", "http-body 0.4.6", @@ -817,7 +777,7 @@ dependencies = [ "hyper-util", "pin-project-lite", "rustls 0.21.12", - "rustls 0.23.35", + "rustls 0.23.36", "rustls-native-certs", "rustls-pki-types", "tokio", @@ -828,9 +788,9 @@ dependencies = [ [[package]] name = "aws-smithy-json" -version = "0.61.8" +version = "0.61.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6864c190cbb8e30cf4b77b2c8f3b6dfffa697a09b7218d2f7cd3d4c4065a9f7" +checksum = "49fa1213db31ac95288d981476f78d05d9cbb0353d22cdf3472cc05bb02f6551" dependencies = [ "aws-smithy-types", ] @@ -932,9 +892,9 @@ dependencies = [ [[package]] name = "aws-types" -version = "1.3.10" +version = "1.3.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d79fb68e3d7fe5d4833ea34dc87d2e97d26d3086cb3da660bb6b1f76d98680b6" +checksum = "1d980627d2dd7bfc32a3c025685a033eeab8d365cc840c631ef59d1b8f428164" dependencies = [ "aws-credential-types", "aws-smithy-async", @@ -973,15 +933,15 @@ dependencies = [ [[package]] name = "base64ct" -version = "1.8.1" +version = "1.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e050f626429857a27ddccb31e0aca21356bfa709c04041aefddac081a8f068a" +checksum = "7d809780667f4410e7c41b07f52439b94d2bdf8528eeedc287fa38d3b7f95d82" [[package]] name = "bigdecimal" -version = "0.4.9" +version = "0.4.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "560f42649de9fa436b73517378a147ec21f6c997a546581df4b4b31677828934" +checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" dependencies = [ "autocfg", "libm", @@ -1006,18 +966,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "bitvec" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" -dependencies = [ - "funty", - "radium", - "tap", - "wyz", -] - [[package]] name = "blake2" version = "0.10.6" @@ -1090,30 +1038,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.111", -] - -[[package]] -name = "borsh" -version = "1.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1da5ab77c1437701eeff7c88d968729e7766172279eab0676857b3d63af7a6f" -dependencies = [ - "borsh-derive", - "cfg_aliases", -] - -[[package]] -name = "borsh-derive" -version = "1.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0686c856aa6aac0c4498f936d7d6a02df690f614c03e4d906d1018062b5c5e2c" -dependencies = [ - "once_cell", - "proc-macro-crate", - "proc-macro2", - "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -1139,31 +1064,9 @@ dependencies = [ [[package]] name = "bumpalo" -version = "3.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43" - -[[package]] -name = "bytecheck" -version = "0.6.12" +version = "3.19.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2" -dependencies = [ - "bytecheck_derive", - "ptr_meta 0.1.4", - "simdutf8", -] - -[[package]] -name = "bytecheck_derive" -version = "0.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3db406d29fbcd95542e92559bed4d8ad92636d1ca8b3b72ede10b4bcc010e659" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] +checksum = "5dd9dc738b7a8311c7ade152424974d8115f2cdad61e8dab8dac9f2362298510" [[package]] name = "bytemuck" @@ -1196,15 +1099,6 @@ dependencies = [ "either", ] -[[package]] -name = "bzip2" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" -dependencies = [ - "bzip2-sys", -] - [[package]] name = "bzip2" version = "0.6.1" @@ -1214,16 +1108,6 @@ dependencies = [ "libbz2-rs-sys", ] -[[package]] -name = "bzip2-sys" -version = "0.1.13+1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "cbc" version = "0.1.2" @@ -1235,9 +1119,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.49" +version = "1.2.51" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90583009037521a116abf44494efecd645ba48b6622457080f080b85544e2215" +checksum = "7a0aeaff4ff1a90589618835a598e545176939b97874f7abc7851caa0618f203" dependencies = [ "find-msvc-tools", "jobserver", @@ -1293,9 +1177,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.53" +version = "4.5.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9e340e012a1bf4935f5282ed1436d1489548e8f72308207ea5df0e23d2d03f8" +checksum = "c6e6ff9dcd79cff5cd969a17a545d79e84ab086e444102a591e288a8aa3ce394" dependencies = [ "clap_builder", "clap_derive", @@ -1303,9 +1187,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.53" +version = "4.5.54" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d76b5d13eaa18c901fd2f7fca939fefe3a0727a953561fefdf3b2922b8569d00" +checksum = "fa42cf4d2b7a41bc8f663a7cab4031ebafa1bf3875705bfaf8466dc60ab52c00" dependencies = [ "anstream", "anstyle", @@ -1322,7 +1206,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -1342,9 +1226,9 @@ dependencies = [ [[package]] name = "cmake" -version = "0.1.54" +version = "0.1.57" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7caa3f9de89ddbe2c607f4101924c5abec803763ae9534e4f4d7d8f84aa81f0" +checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d" dependencies = [ "cc", ] @@ -1361,7 +1245,7 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fde0e0ec90c9dfb3b4b1a0891a7dcd0e2bffde2f7efed5fe7c9bb00e5bfb915e" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.48.0", ] [[package]] @@ -1374,6 +1258,27 @@ dependencies = [ "unicode-width 0.2.2", ] +[[package]] +name = "compression-codecs" +version = "0.4.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0f7ac3e5b97fdce45e8922fb05cae2c37f7bbd63d30dd94821dacfd8f3f2bf2" +dependencies = [ + "bzip2", + "compression-core", + "flate2", + "liblzma", + "memchr", + "zstd", + "zstd-safe", +] + +[[package]] +name = "compression-core" +version = "0.4.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75984efb6ed102a0d42db99afb6c1948f0380d1d91808d5529916e6c08b49d8d" + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -1385,9 +1290,9 @@ dependencies = [ [[package]] name = "console" -version = "0.16.1" +version = "0.16.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b430743a6eb14e9764d4260d4c0d8123087d504eeb9c48f2b2a5e810dd369df4" +checksum = "03e45a4a8926227e4197636ba97a9fc9b00477e9f4bd711395687c5f0734bec4" dependencies = [ "encode_unicode", "libc", @@ -1587,7 +1492,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.111", + "syn", ] [[package]] @@ -1601,7 +1506,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.111", + "syn", ] [[package]] @@ -1612,7 +1517,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -1623,7 +1528,7 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -1642,15 +1547,15 @@ dependencies = [ [[package]] name = "datafusion" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ba7cb113e9c0bedf9e9765926031e132fa05a1b09ba6e93a6d1a4d7044457b8" +checksum = "d12ee9fdc6cdb5898c7691bb994f0ba606c4acc93a2258d78bb9f26ff8158bb3" dependencies = [ "arrow", "arrow-schema", "async-trait", "bytes", - "bzip2 0.6.1", + "bzip2", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1681,27 +1586,26 @@ dependencies = [ "flate2", "futures", "itertools 0.14.0", + "liblzma", "log", "object_store", "parking_lot", "parquet", "rand 0.9.2", "regex", - "rstest", "sqlparser", "tempfile", "tokio", "url", "uuid", - "xz2", "zstd", ] [[package]] name = "datafusion-catalog" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "66a3a799f914a59b1ea343906a0486f17061f39509af74e874a866428951130d" +checksum = "462dc9ef45e5d688aeaae49a7e310587e81b6016b9d03bace5626ad0043e5a9e" dependencies = [ "arrow", "async-trait", @@ -1724,9 +1628,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db1b113c80d7a0febcd901476a57aef378e717c54517a163ed51417d87621b0" +checksum = "1b96dbf1d728fc321817b744eb5080cdd75312faa6980b338817f68f3caa4208" dependencies = [ "arrow", "async-trait", @@ -1743,14 +1647,13 @@ dependencies = [ "itertools 0.14.0", "log", "object_store", - "tokio", ] [[package]] name = "datafusion-cli" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fab982df44f818a749cb5200504ccb919f4608cb9808daf8b3fb98aa7955fd1e" +checksum = "e765fd9f2ce5b451861a7858987f47ce11bb963dff7785f667e68dcea212c3de" dependencies = [ "arrow", "async-trait", @@ -1776,17 +1679,17 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" +checksum = "3237a6ff0d2149af4631290074289cae548c9863c885d821315d54c6673a074a" dependencies = [ - "ahash 0.8.12", - "apache-avro 0.20.0", + "ahash", + "apache-avro", "arrow", "arrow-ipc", "chrono", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "hex", "indexmap 2.12.1", "libc", @@ -1802,9 +1705,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b92065bbc6532c6651e2f7dd30b55cba0c7a14f860c7e1d15f165c41a1868d95" +checksum = "70b5e34026af55a1bfccb1ef0a763cf1f64e77c696ffcf5a128a278c31236528" dependencies = [ "futures", "log", @@ -1813,15 +1716,15 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fde13794244bc7581cd82f6fff217068ed79cdc344cafe4ab2c3a1c3510b38d6" +checksum = "1b2a6be734cc3785e18bbf2a7f2b22537f6b9fb960d79617775a51568c281842" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2 0.6.1", + "bzip2", "chrono", "datafusion-common", "datafusion-common-runtime", @@ -1836,21 +1739,21 @@ dependencies = [ "futures", "glob", "itertools 0.14.0", + "liblzma", "log", "object_store", "rand 0.9.2", "tokio", "tokio-util", "url", - "xz2", "zstd", ] [[package]] name = "datafusion-datasource-arrow" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "804fa9b4ecf3157982021770617200ef7c1b2979d57bec9044748314775a9aea" +checksum = "1739b9b07c9236389e09c74f770e88aff7055250774e9def7d3f4f56b3dcc7be" dependencies = [ "arrow", "arrow-ipc", @@ -1872,11 +1775,11 @@ dependencies = [ [[package]] name = "datafusion-datasource-avro" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "388ed8be535f562cc655b9c3d22edbfb0f1a50a25c242647a98b6d92a75b55a1" +checksum = "828088c2fb681cc0e06fb42f541f76c82a0c10278f9fd6334e22c8d1e3574ee7" dependencies = [ - "apache-avro 0.20.0", + "apache-avro", "arrow", "async-trait", "bytes", @@ -1892,9 +1795,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61a1641a40b259bab38131c5e6f48fac0717bedb7dc93690e604142a849e0568" +checksum = "61c73bc54b518bbba7c7650299d07d58730293cfba4356f6f428cc94c20b7600" dependencies = [ "arrow", "async-trait", @@ -1915,9 +1818,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adeacdb00c1d37271176f8fb6a1d8ce096baba16ea7a4b2671840c5c9c64fe85" +checksum = "37812c8494c698c4d889374ecfabbff780f1f26d9ec095dd1bddfc2a8ca12559" dependencies = [ "arrow", "async-trait", @@ -1937,9 +1840,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43d0b60ffd66f28bfb026565d62b0a6cbc416da09814766a3797bba7d85a3cd9" +checksum = "2210937ecd9f0e824c397e73f4b5385c97cd1aff43ab2b5836fcfd2d321523fb" dependencies = [ "arrow", "async-trait", @@ -1967,18 +1870,19 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b99e13947667b36ad713549237362afb054b2d8f8cc447751e23ec61202db07" +checksum = "2c825f969126bc2ef6a6a02d94b3c07abff871acf4d6dd759ce1255edb7923ce" [[package]] name = "datafusion-execution" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63695643190679037bc946ad46a263b62016931547bf119859c511f7ff2f5178" +checksum = "fa03ef05a2c2f90dd6c743e3e111078e322f4b395d20d4b4d431a245d79521ae" dependencies = [ "arrow", "async-trait", + "chrono", "dashmap", "datafusion-common", "datafusion-expr", @@ -1994,9 +1898,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9a4787cbf5feb1ab351f789063398f67654a6df75c4d37d7f637dc96f951a91" +checksum = "ef33934c1f98ee695cc51192cc5f9ed3a8febee84fdbcd9131bf9d3a9a78276f" dependencies = [ "arrow", "async-trait", @@ -2017,9 +1921,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ce2fb1b8c15c9ac45b0863c30b268c69dc9ee7a1ee13ecf5d067738338173dc" +checksum = "000c98206e3dd47d2939a94b6c67af4bfa6732dd668ac4fafdbde408fd9134ea" dependencies = [ "arrow", "datafusion-common", @@ -2030,9 +1934,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "794a9db7f7b96b3346fc007ff25e994f09b8f0511b4cf7dff651fadfe3ebb28f" +checksum = "379b01418ab95ca947014066248c22139fe9af9289354de10b445bd000d5d276" dependencies = [ "arrow", "arrow-buffer", @@ -2040,6 +1944,7 @@ dependencies = [ "blake2", "blake3", "chrono", + "chrono-tz", "datafusion-common", "datafusion-doc", "datafusion-execution", @@ -2060,11 +1965,11 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" +checksum = "fd00d5454ba4c3f8ebbd04bd6a6a9dc7ced7c56d883f70f2076c188be8459e4c" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-doc", @@ -2081,11 +1986,11 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" +checksum = "aec06b380729a87210a4e11f555ec2d729a328142253f8d557b87593622ecc9f" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-expr-common", @@ -2094,9 +1999,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae5c06eed03918dc7fe7a9f082a284050f0e9ecf95d72f57712d1496da03b8c4" +checksum = "904f48d45e0f1eb7d0eb5c0f80f2b5c6046a85454364a6b16a2e0b46f62e7dff" dependencies = [ "arrow", "arrow-ord", @@ -2117,9 +2022,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db4fed1d71738fbe22e2712d71396db04c25de4111f1ec252b8f4c6d3b25d7f5" +checksum = "e9a0d20e2b887e11bee24f7734d780a2588b925796ac741c3118dd06d5aa77f0" dependencies = [ "arrow", "async-trait", @@ -2133,9 +2038,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d92206aa5ae21892f1552b4d61758a862a70956e6fd7a95cb85db1de74bc6d1" +checksum = "d3414b0a07e39b6979fe3a69c7aa79a9f1369f1d5c8e52146e66058be1b285ee" dependencies = [ "arrow", "datafusion-common", @@ -2151,9 +2056,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53ae9bcc39800820d53a22d758b3b8726ff84a5a3e24cecef04ef4e5fdf1c7cc" +checksum = "5bf2feae63cd4754e31add64ce75cae07d015bce4bb41cd09872f93add32523a" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -2161,20 +2066,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1063ad4c9e094b3f798acee16d9a47bd7372d9699be2de21b05c3bd3f34ab848" +checksum = "c4fe888aeb6a095c4bcbe8ac1874c4b9a4c7ffa2ba849db7922683ba20875aaf" dependencies = [ "datafusion-doc", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "datafusion-optimizer" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f35f9ec5d08b87fd1893a30c2929f2559c2f9806ca072d8fefca5009dc0f06a" +checksum = "8a6527c063ae305c11be397a86d8193936f4b84d137fe40bd706dfc178cf733c" dependencies = [ "arrow", "chrono", @@ -2192,11 +2097,11 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" +checksum = "0bb028323dd4efd049dd8a78d78fe81b2b969447b39c51424167f973ac5811d9" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "datafusion-common", "datafusion-expr", @@ -2204,19 +2109,21 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "indexmap 2.12.1", "itertools 0.14.0", "parking_lot", "paste", "petgraph 0.8.3", + "recursive", + "tokio", ] [[package]] name = "datafusion-physical-expr-adapter" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f9ff2dbd476221b1f67337699eff432781c4e6e1713d2aefdaa517dfbf79768" +checksum = "78fe0826aef7eab6b4b61533d811234a7a9e5e458331ebbf94152a51fc8ab433" dependencies = [ "arrow", "datafusion-common", @@ -2229,23 +2136,26 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" +checksum = "cfccd388620734c661bd8b7ca93c44cdd59fecc9b550eea416a78ffcbb29475f" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", + "chrono", "datafusion-common", "datafusion-expr-common", - "hashbrown 0.14.5", + "hashbrown 0.16.1", + "indexmap 2.12.1", "itertools 0.14.0", + "parking_lot", ] [[package]] name = "datafusion-physical-optimizer" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce9804f799acd7daef3be7aaffe77c0033768ed8fdbf5fb82fc4c5f2e6bc14e6" +checksum = "bde5fa10e73259a03b705d5fddc136516814ab5f441b939525618a4070f5a059" dependencies = [ "arrow", "datafusion-common", @@ -2262,27 +2172,27 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" +checksum = "0e1098760fb29127c24cc9ade3277051dc73c9ed0ac0131bd7bcd742e0ad7470" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow", "arrow-ord", "arrow-schema", "async-trait", - "chrono", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", + "datafusion-functions", "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "futures", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "indexmap 2.12.1", "itertools 0.14.0", "log", @@ -2293,9 +2203,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac2c2498a1f134a9e11a9f5ed202a2a7d7e9774bd9249295593053ea3be999db" +checksum = "64d0fef4201777b52951edec086c21a5b246f3c82621569ddb4a26f488bc38a9" dependencies = [ "arrow", "datafusion-common", @@ -2310,9 +2220,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f96eebd17555386f459037c65ab73aae8df09f464524c709d6a3134ad4f4776" +checksum = "f71f1e39e8f2acbf1c63b0e93756c2e970a64729dab70ac789587d6237c4fde0" dependencies = [ "async-trait", "datafusion-common", @@ -2324,9 +2234,9 @@ dependencies = [ [[package]] name = "datafusion-spark" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97a8d6fed24c80dd403dcc6afec33766a599d1b72575f222237f01429b2e58ba" +checksum = "556c431f5f2259620c8223254c0ef57aa9a85c576d4da0166157260f71eb0e25" dependencies = [ "arrow", "bigdecimal", @@ -2337,7 +2247,9 @@ dependencies = [ "datafusion-execution", "datafusion-expr", "datafusion-functions", + "datafusion-functions-nested", "log", + "percent-encoding", "rand 0.9.2", "sha1", "url", @@ -2345,9 +2257,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fc195fe60634b2c6ccfd131b487de46dc30eccae8a3c35a13f136e7f440414f" +checksum = "f44693cfcaeb7a9f12d71d1c576c3a6dc025a12cef209375fa2d16fb3b5670ee" dependencies = [ "arrow", "bigdecimal", @@ -2363,9 +2275,9 @@ dependencies = [ [[package]] name = "datafusion-sqllogictest" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6830e357705e0d54fda6e3ce70a87c2b255197563c6463d668520cbfc1e0b7c" +checksum = "9d388fec80647198ae041d314dd7d9e2305207836ecec3ad48908eac6844cdef" dependencies = [ "arrow", "async-trait", @@ -2380,7 +2292,6 @@ dependencies = [ "itertools 0.14.0", "log", "object_store", - "rust_decimal", "sqllogictest", "sqlparser", "tempfile", @@ -2390,9 +2301,9 @@ dependencies = [ [[package]] name = "datafusion-substrait" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2505af06d103a55b4e8ded0c6aeb6c72a771948da939c0bd3f8eee67af475a9c" +checksum = "6042adacd0bd64e56c22f6a7f9ce0ce1793dd367c899d868179d029f110d9215" dependencies = [ "async-recursion", "async-trait", @@ -2448,7 +2359,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -2458,7 +2369,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.111", + "syn", ] [[package]] @@ -2497,7 +2408,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2508,7 +2419,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -2559,7 +2470,7 @@ dependencies = [ "enum-ordinalize", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -2600,7 +2511,7 @@ checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -2650,7 +2561,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -2707,12 +2618,6 @@ dependencies = [ "once_cell", ] -[[package]] -name = "fallible-iterator" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7" - [[package]] name = "fastnum" version = "0.7.4" @@ -2738,7 +2643,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baec6a0289d7f1fe5665586ef7340af82e3037207bef60f5785e57569776f0c8" dependencies = [ "bytes", - "rkyv 0.8.13", + "rkyv", "serde", "simdutf8", ] @@ -2756,9 +2661,9 @@ dependencies = [ [[package]] name = "find-msvc-tools" -version = "0.1.5" +version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a3076410a55c90011c298b04d0cfa770b00fa04e1e3c97d3f6c9de105a03844" +checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff" [[package]] name = "fixedbitset" @@ -2768,9 +2673,9 @@ checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" [[package]] name = "flatbuffers" -version = "25.9.23" +version = "25.12.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09b6620799e7340ebd9968d2e0708eb82cf1971e9a16821e2091b6d6e475eed5" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" dependencies = [ "bitflags", "rustc_version", @@ -2833,9 +2738,9 @@ checksum = "28dd6caf6059519a65843af8fe2a3ae298b14b80179855aeb4adc2c1934ee619" [[package]] name = "fs-err" -version = "3.3.0" +version = "3.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73fde052dbfc920003cfd2c8e2c6e6d4cc7c1091538c3a24226cec0665ab08c0" +checksum = "baf68cef89750956493a66a10f512b9e58d9db21f2a573c079c0bdf1207a54a7" dependencies = [ "autocfg", ] @@ -2846,12 +2751,6 @@ version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" -[[package]] -name = "funty" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" - [[package]] name = "futures" version = "0.3.31" @@ -2919,7 +2818,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -2934,12 +2833,6 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" -[[package]] -name = "futures-timer" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" - [[package]] name = "futures-util" version = "0.3.31" @@ -2995,19 +2888,6 @@ dependencies = [ "wasm-bindgen", ] -[[package]] -name = "getrandom" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec" -dependencies = [ - "cfg-if", - "libc", - "r-efi", - "wasip2", - "wasip3", -] - [[package]] name = "glob" version = "0.3.3" @@ -3047,9 +2927,9 @@ dependencies = [ [[package]] name = "h2" -version = "0.4.12" +version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3c0b69cfcb4e1b9f1bf2f53f95f766e4661169728ec61cd3fe5a0166f2d1386" +checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54" dependencies = [ "atomic-waker", "bytes", @@ -3081,19 +2961,12 @@ name = "hashbrown" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" -dependencies = [ - "ahash 0.7.8", -] [[package]] name = "hashbrown" version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" -dependencies = [ - "ahash 0.8.12", - "allocator-api2", -] [[package]] name = "hashbrown" @@ -3284,7 +3157,7 @@ dependencies = [ "bytes", "futures-channel", "futures-core", - "h2 0.4.12", + "h2 0.4.13", "http 1.4.0", "http-body 1.0.1", "httparse", @@ -3321,13 +3194,13 @@ dependencies = [ "http 1.4.0", "hyper 1.8.1", "hyper-util", - "rustls 0.23.35", + "rustls 0.23.36", "rustls-native-certs", "rustls-pki-types", "tokio", "tokio-rustls 0.26.4", "tower-service", - "webpki-roots 1.0.4", + "webpki-roots 1.0.5", ] [[package]] @@ -3383,7 +3256,7 @@ name = "iceberg" version = "0.8.0" dependencies = [ "anyhow", - "apache-avro 0.21.0", + "apache-avro", "array-init", "arrow-arith", "arrow-array", @@ -3691,9 +3564,9 @@ checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" [[package]] name = "icu_properties" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e93fcd3157766c0c8da2f8cff6ce651a31f0810eaa1c51ec363ef790bbb5fb99" +checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec" dependencies = [ "icu_collections", "icu_locale_core", @@ -3705,9 +3578,9 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "02845b3647bb045f1100ecd6480ff52f34c35f82d9880e029d329c21d1054899" +checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af" [[package]] name = "icu_provider" @@ -3724,12 +3597,6 @@ dependencies = [ "zerovec", ] -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - [[package]] name = "ident_case" version = "1.0.1" @@ -3821,9 +3688,9 @@ dependencies = [ [[package]] name = "inventory" -version = "0.3.21" +version = "0.3.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc61209c082fbeb19919bee74b176221b27223e27b65d781eb91af24eb1fb46e" +checksum = "009ae045c87e7082cb72dab0ccd01ae075dd00141ddc108f43a0ea150a9e7227" dependencies = [ "rustversion", ] @@ -3836,9 +3703,9 @@ checksum = "469fb0b9cefa57e3ef31275ee7cacb78f2fdca44e4765491884a2b119d4eb130" [[package]] name = "iri-string" -version = "0.7.9" +version = "0.7.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f867b9d1d896b67beb18518eda36fdb77a32ea590de864f1325b294a6d14397" +checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a" dependencies = [ "memchr", "serde", @@ -3870,15 +3737,15 @@ dependencies = [ [[package]] name = "itoa" -version = "1.0.15" +version = "1.0.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c" +checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" [[package]] name = "jiff" -version = "0.2.16" +version = "0.2.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49cce2b81f2098e7e3efc35bc2e0a6b7abec9d34128283d7a26fa8f32a6dbb35" +checksum = "e67e8da4c49d6d9909fe03361f9b620f58898859f5c7aded68351e85e71ecf50" dependencies = [ "jiff-static", "jiff-tzdb-platform", @@ -3886,25 +3753,25 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] name = "jiff-static" -version = "0.2.16" +version = "0.2.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "980af8b43c3ad5d8d349ace167ec8170839f753a42d233ba19e08afe1850fa69" +checksum = "e0c84ee7f197eca9a86c6fd6cb771e55eb991632f15f2bc3ca6ec838929e6e78" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "jiff-tzdb" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1283705eb0a21404d2bfd6eef2a7593d240bc42a0bdb39db0ad6fa2ec026524" +checksum = "68971ebff725b9e2ca27a601c5eb38a4c5d64422c4cbab0c535f248087eda5c2" [[package]] name = "jiff-tzdb-platform" @@ -3959,12 +3826,6 @@ dependencies = [ "spin", ] -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - [[package]] name = "lexical-core" version = "1.0.6" @@ -4030,9 +3891,29 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" [[package]] name = "libc" -version = "0.2.178" +version = "0.2.179" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37c93d8daa9d8a012fd8ab92f088405fb202ea0b6ab73ee2482ae66af4f42091" +checksum = "c5a2d376baa530d1238d133232d15e239abad80d05838b4b59354e5268af431f" + +[[package]] +name = "liblzma" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73c36d08cad03a3fbe2c4e7bb3a9e84c57e4ee4135ed0b065cade3d98480c648" +dependencies = [ + "liblzma-sys", +] + +[[package]] +name = "liblzma-sys" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01b9596486f6d60c3bbe644c0e1be1aa6ccc472ad630fe8927b456973d7cb736" +dependencies = [ + "cc", + "libc", + "pkg-config", +] [[package]] name = "libm" @@ -4052,13 +3933,13 @@ dependencies = [ [[package]] name = "libredox" -version = "0.1.10" +version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "416f7e718bdb06000964960ffa43b4335ad4012ae8b99060261aa4a8088d5ccb" +checksum = "3d0b95e02c851351f877147b7deea7b1afb1df71b63aa5f8270716e0c5720616" dependencies = [ "bitflags", "libc", - "redox_syscall", + "redox_syscall 0.7.0", ] [[package]] @@ -4086,9 +3967,9 @@ dependencies = [ [[package]] name = "libz-rs-sys" -version = "0.5.3" +version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b484ba8d4f775eeca644c452a56650e544bf7e617f1d170fe7298122ead5222" +checksum = "c10501e7805cee23da17c7790e59df2870c0d4043ec6d03f67d31e2b53e77415" dependencies = [ "zlib-rs", ] @@ -4152,17 +4033,6 @@ dependencies = [ "twox-hash", ] -[[package]] -name = "lzma-sys" -version = "0.1.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" -dependencies = [ - "cc", - "libc", - "pkg-config", -] - [[package]] name = "md-5" version = "0.10.6" @@ -4194,7 +4064,7 @@ version = "0.7.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cdc67a1d6ef0340a7f5152b9fd34ad7477b4d518920f7557267f1fc6e5a62641" dependencies = [ - "ahash 0.8.12", + "ahash", "faststr", "paste", "rustc-hash", @@ -4212,9 +4082,9 @@ dependencies = [ [[package]] name = "minijinja" -version = "2.16.0" +version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c54f3bcc034dd74496b5ca929fd0b710186672d5ff0b0f255a9ceb259042ece" +checksum = "12ea9ac0a51fb5112607099560fdf0f90366ab088a2a9e6e8ae176794e9806aa" dependencies = [ "serde", ] @@ -4263,7 +4133,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -4293,9 +4163,9 @@ dependencies = [ [[package]] name = "moka" -version = "0.12.11" +version = "0.12.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8261cd88c312e0004c1d51baad2980c66528dfdb2bee62003e643a4d8f86b077" +checksum = "a3dec6bd31b08944e08b58fd99373893a6c17054d6f3ea5006cc894f4f4eee2a" dependencies = [ "async-lock", "crossbeam-channel", @@ -4306,7 +4176,6 @@ dependencies = [ "futures-util", "parking_lot", "portable-atomic", - "rustc_version", "smallvec", "tagptr", "uuid", @@ -4332,7 +4201,7 @@ checksum = "b40e46c845ac234bcba19db7ab252bc2778cbadd516a466d2f12b1580852d136" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -4358,7 +4227,7 @@ checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -4413,7 +4282,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4507,7 +4376,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -4599,9 +4468,9 @@ dependencies = [ [[package]] name = "openssl-probe" -version = "0.1.6" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" +checksum = "9f50d9b3dabb09ecd771ad0aa242ca6894994c130308ca3d7684634df8037391" [[package]] name = "option-ext" @@ -4675,7 +4544,7 @@ checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" dependencies = [ "cfg-if", "libc", - "redox_syscall", + "redox_syscall 0.5.18", "smallvec", "windows-link", ] @@ -4686,7 +4555,7 @@ version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be3e4f6d320dd92bfa7d612e265d7d08bba0a240bab86af3425e1d255a511d89" dependencies = [ - "ahash 0.8.12", + "ahash", "arrow-array", "arrow-buffer", "arrow-cast", @@ -4842,7 +4711,7 @@ version = "0.11.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "520e32b295867834617b8a352b6fd4837e578f328f4e48813db4904ebc0d70b2" dependencies = [ - "ahash 0.8.12", + "ahash", "anyhow", "async-recursion", "bytes", @@ -4875,7 +4744,7 @@ checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -4936,9 +4805,9 @@ checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" [[package]] name = "portable-atomic" -version = "1.11.1" +version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f84267b20a16ea918e43c6a88433c2d54fa145c92a811b5b047ccbe153674483" +checksum = "f89776e4d69bb58bc6993e99ffa1d11f228b839984854c7daeb5d37f87cbe950" [[package]] name = "portable-atomic-util" @@ -4949,35 +4818,6 @@ dependencies = [ "portable-atomic", ] -[[package]] -name = "postgres-protocol" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbef655056b916eb868048276cfd5d6a7dea4f81560dfd047f97c8c6fe3fcfd4" -dependencies = [ - "base64", - "byteorder", - "bytes", - "fallible-iterator", - "hmac", - "md-5", - "memchr", - "rand 0.9.2", - "sha2", - "stringprep", -] - -[[package]] -name = "postgres-types" -version = "0.2.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef4605b7c057056dd35baeb6ac0c0338e4975b1f2bef0f65da953285eb007095" -dependencies = [ - "bytes", - "fallible-iterator", - "postgres-protocol", -] - [[package]] name = "potential_utf" version = "0.1.4" @@ -5045,7 +4885,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.111", + "syn", ] [[package]] @@ -5054,14 +4894,14 @@ version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" dependencies = [ - "toml_edit 0.23.9", + "toml_edit 0.23.10+spec-1.0.0", ] [[package]] name = "proc-macro2" -version = "1.0.103" +version = "1.0.105" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ee95bc4ef87b8d5ba32e8b7714ccc834865276eab0aed5c9958d00ec45f49e8" +checksum = "535d180e0ecab6268a3e718bb9fd44db66bbbc256257165fc699dadf70d16fe7" dependencies = [ "unicode-ident", ] @@ -5092,7 +4932,7 @@ dependencies = [ "prost", "prost-types", "regex", - "syn 2.0.111", + "syn", "tempfile", ] @@ -5106,7 +4946,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -5128,33 +4968,13 @@ dependencies = [ "cc", ] -[[package]] -name = "ptr_meta" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1" -dependencies = [ - "ptr_meta_derive 0.1.4", -] - [[package]] name = "ptr_meta" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b9a0cf95a1196af61d4f1cbdab967179516d9a4a4312af1f31948f8f6224a79" dependencies = [ - "ptr_meta_derive 0.3.1", -] - -[[package]] -name = "ptr_meta_derive" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", + "ptr_meta_derive", ] [[package]] @@ -5165,7 +4985,7 @@ checksum = "7347867d0a7e1208d93b46767be83e2b8f978c3dad35f775ac8d8847551d6fe1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -5206,7 +5026,7 @@ dependencies = [ "quinn-proto", "quinn-udp", "rustc-hash", - "rustls 0.23.35", + "rustls 0.23.36", "socket2 0.6.1", "thiserror 2.0.17", "tokio", @@ -5226,7 +5046,7 @@ dependencies = [ "rand 0.9.2", "ring", "rustc-hash", - "rustls 0.23.35", + "rustls 0.23.36", "rustls-pki-types", "slab", "thiserror 2.0.17", @@ -5251,9 +5071,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.42" +version = "1.0.43" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a338cc41d27e6cc6dce6cefc13a0729dfbb81c262b1f519331575dd80ef3067f" +checksum = "dc74d9a594b72ae6656596548f56f667211f8a97b3d4c3d467150794690dc40a" dependencies = [ "proc-macro2", ] @@ -5264,12 +5084,6 @@ version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" -[[package]] -name = "radium" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" - [[package]] name = "radix_trie" version = "0.2.1" @@ -5286,7 +5100,7 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a063ea72381527c2a0561da9c80000ef822bdd7c3241b1cc1b12100e3df081ee" dependencies = [ - "ptr_meta 0.3.1", + "ptr_meta", ] [[package]] @@ -5367,7 +5181,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -5379,6 +5193,15 @@ dependencies = [ "bitflags", ] +[[package]] +name = "redox_syscall" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49f3fe0889e69e2ae9e41f4d6c4c0181701d00e4697b356fb1f74173a5e0ee27" +dependencies = [ + "bitflags", +] + [[package]] name = "redox_users" version = "0.5.2" @@ -5407,7 +5230,7 @@ checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -5455,21 +5278,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "relative-path" -version = "1.9.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" - -[[package]] -name = "rend" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c" -dependencies = [ - "bytecheck", -] - [[package]] name = "rend" version = "0.5.3" @@ -5518,7 +5326,7 @@ dependencies = [ "bytes", "futures-core", "futures-util", - "h2 0.4.12", + "h2 0.4.13", "http 1.4.0", "http-body 1.0.1", "http-body-util", @@ -5530,7 +5338,7 @@ dependencies = [ "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.35", + "rustls 0.23.36", "rustls-native-certs", "rustls-pki-types", "serde", @@ -5548,7 +5356,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams", "web-sys", - "webpki-roots 1.0.4", + "webpki-roots 1.0.5", ] [[package]] @@ -5565,24 +5373,6 @@ dependencies = [ "windows-sys 0.52.0", ] -[[package]] -name = "rkyv" -version = "0.7.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2297bf9c81a3f0dc96bc9521370b88f054168c29826a75e89c55ff196e7ed6a1" -dependencies = [ - "bitvec", - "bytecheck", - "bytes", - "hashbrown 0.12.3", - "ptr_meta 0.1.4", - "rend 0.4.2", - "rkyv_derive 0.7.46", - "seahash", - "tinyvec", - "uuid", -] - [[package]] name = "rkyv" version = "0.8.13" @@ -5593,25 +5383,14 @@ dependencies = [ "hashbrown 0.16.1", "indexmap 2.12.1", "munge", - "ptr_meta 0.3.1", + "ptr_meta", "rancor", - "rend 0.5.3", - "rkyv_derive 0.8.13", + "rend", + "rkyv_derive", "tinyvec", "uuid", ] -[[package]] -name = "rkyv_derive" -version = "0.7.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "84d7b42d4b8d06048d3ac8db0eb31bcb942cbeb709f0b5f2b2ebde398d3038f5" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - [[package]] name = "rkyv_derive" version = "0.8.13" @@ -5620,14 +5399,14 @@ checksum = "7f6dffea3c91fa91a3c0fc8a061b0e27fef25c6304728038a6d6bcb1c58ba9bd" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "roaring" -version = "0.11.2" +version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f08d6a905edb32d74a5d5737a0c9d7e950c312f3c46cb0ca0a2ca09ea11878a0" +checksum = "8ba9ce64a8f45d7fc86358410bb1a82e8c987504c0d4900e9141d69a9f26c885" dependencies = [ "bytemuck", "byteorder", @@ -5654,35 +5433,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rstest" -version = "0.26.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5a3193c063baaa2a95a33f03035c8a72b83d97a54916055ba22d35ed3839d49" -dependencies = [ - "futures-timer", - "futures-util", - "rstest_macros", -] - -[[package]] -name = "rstest_macros" -version = "0.26.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0" -dependencies = [ - "cfg-if", - "glob", - "proc-macro-crate", - "proc-macro2", - "quote", - "regex", - "relative-path", - "rustc_version", - "syn 2.0.111", - "unicode-ident", -] - [[package]] name = "rust-ini" version = "0.21.3" @@ -5693,23 +5443,6 @@ dependencies = [ "ordered-multimap", ] -[[package]] -name = "rust_decimal" -version = "1.40.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61f703d19852dbf87cbc513643fa81428361eb6940f1ac14fd58155d295a3eb0" -dependencies = [ - "arrayvec", - "borsh", - "bytes", - "num-traits", - "postgres-types", - "rand 0.8.5", - "rkyv 0.7.46", - "serde", - "serde_json", -] - [[package]] name = "rustc-hash" version = "2.1.1" @@ -5738,7 +5471,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -5755,9 +5488,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.35" +version = "0.23.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f" +checksum = "c665f33d38cea657d9614f766881e4d510e0eda4239891eea56b4cadcf01801b" dependencies = [ "aws-lc-rs", "once_cell", @@ -5770,9 +5503,9 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9980d917ebb0c0536119ba501e90834767bffc3d60641457fd84a1f3fd337923" +checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" dependencies = [ "openssl-probe", "rustls-pki-types", @@ -5791,9 +5524,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.13.1" +version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "708c0f9d5f54ba0272468c1d306a52c495b31fa155e91bc25371e6df7996908c" +checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282" dependencies = [ "web-time", "zeroize", @@ -5851,9 +5584,9 @@ dependencies = [ [[package]] name = "ryu" -version = "1.0.20" +version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28d3b2b1366ec20994f1fd18c3c594f05c5dd4bc44d8bb0c1c632c8d6829481f" +checksum = "a50f4cf475b65d88e057964e0e9bb1f0aa9bbb2036dc65c64596b42932536984" [[package]] name = "salsa20" @@ -5908,9 +5641,9 @@ dependencies = [ [[package]] name = "schemars" -version = "1.1.0" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9558e172d4e8533736ba97870c4b2cd63f84b382a3d6eb063da41b91cce17289" +checksum = "54e910108742c57a770f492731f99be216a52fadd361b06c8fb59d74ccc267d2" dependencies = [ "dyn-clone", "ref-cast", @@ -5927,7 +5660,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.111", + "syn", ] [[package]] @@ -5957,12 +5690,6 @@ dependencies = [ "untrusted", ] -[[package]] -name = "seahash" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b" - [[package]] name = "security-framework" version = "3.5.1" @@ -6048,7 +5775,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6059,20 +5786,20 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "serde_json" -version = "1.0.145" +version = "1.0.149" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "402a6f66d8c709116cf22f558eab210f5a50187f702eb4d7e5ef38d9a7f1c79c" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" dependencies = [ "itoa", "memchr", - "ryu", "serde", "serde_core", + "zmij", ] [[package]] @@ -6083,7 +5810,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6104,7 +5831,7 @@ dependencies = [ "proc-macro2", "quote", "serde", - "syn 2.0.111", + "syn", ] [[package]] @@ -6131,7 +5858,7 @@ dependencies = [ "indexmap 1.9.3", "indexmap 2.12.1", "schemars 0.9.0", - "schemars 1.1.0", + "schemars 1.2.0", "serde_core", "serde_json", "serde_with_macros", @@ -6147,7 +5874,7 @@ dependencies = [ "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6202,10 +5929,11 @@ checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" [[package]] name = "signal-hook-registry" -version = "1.4.7" +version = "1.4.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7664a098b8e616bdfcc2dc0e9ac44eb231eedf41db4e9fe95d8d32ec728dedad" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" dependencies = [ + "errno", "libc", ] @@ -6397,7 +6125,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6436,7 +6164,7 @@ dependencies = [ "memchr", "once_cell", "percent-encoding", - "rustls 0.23.35", + "rustls 0.23.36", "serde", "serde_json", "sha2", @@ -6459,7 +6187,7 @@ dependencies = [ "quote", "sqlx-core", "sqlx-macros-core", - "syn 2.0.111", + "syn", ] [[package]] @@ -6480,7 +6208,7 @@ dependencies = [ "sha2", "sqlx-core", "sqlx-sqlite", - "syn 2.0.111", + "syn", "tokio", "url", ] @@ -6641,7 +6369,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6656,9 +6384,9 @@ dependencies = [ [[package]] name = "substrait" -version = "0.62.0" +version = "0.62.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21f1cb6d0bcd097a39fc25f7236236be29881fe122e282e4173d6d007a929927" +checksum = "62fc4b483a129b9772ccb9c3f7945a472112fdd9140da87f8a4e7f1d44e045d0" dependencies = [ "heck", "pbjson", @@ -6674,7 +6402,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.111", + "syn", "typify", "walkdir", ] @@ -6687,20 +6415,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "syn" -version = "1.0.109" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "2.0.111" +version = "2.0.114" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "390cc9a294ab71bdb1aa2e99d13be9c753cd2d7bd6560c77118597410c4d2e87" +checksum = "d4d107df263a3013ef9b1879b0df87d706ff80f65a86ea879bd9c31f9b307c2a" dependencies = [ "proc-macro2", "quote", @@ -6724,7 +6441,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6733,23 +6450,17 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" -[[package]] -name = "tap" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" - [[package]] name = "tempfile" -version = "3.25.0" +version = "3.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0136791f7c95b1f6dd99f9cc786b91bb81c3800b639b3478e561ddb7be95e5f1" +checksum = "655da9c7eb6305c55742045d5a8d2037996d61d8de95806335c7c86ce0f82e9c" dependencies = [ "fastrand", "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -6784,7 +6495,7 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6795,7 +6506,7 @@ checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6885,9 +6596,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.48.0" +version = "1.49.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff360e02eab121e0bc37a2d3b4d4dc622e6eda3a8e5253d5435ecf5bd4c68408" +checksum = "72a2903cd7736441aac9df9d7688bd0ce48edccaadf181c3b90be801e81d3d86" dependencies = [ "bytes", "libc", @@ -6908,7 +6619,7 @@ checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -6927,15 +6638,15 @@ version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.35", + "rustls 0.23.36", "tokio", ] [[package]] name = "tokio-stream" -version = "0.1.17" +version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eca58d7bba4a75707817a2c44174253f9236b2d5fbd055602e9d5c07c139a047" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" dependencies = [ "futures-core", "pin-project-lite", @@ -6944,9 +6655,9 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.17" +version = "0.7.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2efa149fe76073d6e8fd97ef4f4eca7b67f599660115591483572e406e165594" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" dependencies = [ "bytes", "futures-core", @@ -6978,9 +6689,9 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "0.7.3" +version = "0.7.5+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2cdb639ebbc97961c51720f858597f7f24c4fc295327923af55b74c3c724533" +checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347" dependencies = [ "serde_core", ] @@ -7001,21 +6712,21 @@ dependencies = [ [[package]] name = "toml_edit" -version = "0.23.9" +version = "0.23.10+spec-1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d7cbc3b4b49633d57a0509303158ca50de80ae32c265093b24c414705807832" +checksum = "84c8b9f757e028cee9fa244aea147aab2a9ec09d5325a9b01e0a49730c2b5269" dependencies = [ "indexmap 2.12.1", - "toml_datetime 0.7.3", + "toml_datetime 0.7.5+spec-1.1.0", "toml_parser", "winnow", ] [[package]] name = "toml_parser" -version = "1.0.4" +version = "1.0.6+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0cbe268d35bdb4bb5a56a2de88d0ad0eb70af5384a99d648cd4b3d04039800e" +checksum = "a3198b4b0a8e11f09dd03e133c0280504d0801269e9afa46362ffde1cbeebf44" dependencies = [ "winnow", ] @@ -7073,9 +6784,9 @@ checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" [[package]] name = "tracing" -version = "0.1.43" +version = "0.1.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d15d90a0b5c19378952d479dc858407149d7bb45a14de0142f6c534b16fc647" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" dependencies = [ "log", "pin-project-lite", @@ -7091,14 +6802,14 @@ checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "tracing-core" -version = "0.1.35" +version = "0.1.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a04e24fab5c89c6a36eb8558c9656f30d81de51dfa4d3b45f26b21d61fa0a6c" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" dependencies = [ "once_cell", "valuable", @@ -7158,7 +6869,7 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -7194,7 +6905,7 @@ checksum = "27a7a9b72ba121f6f1f6c3632b85604cac41aedb5ddc70accbebb6cac83de846" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -7222,7 +6933,7 @@ dependencies = [ "semver", "serde", "serde_json", - "syn 2.0.111", + "syn", "thiserror 2.0.17", "unicode-ident", ] @@ -7240,7 +6951,7 @@ dependencies = [ "serde", "serde_json", "serde_tokenstream", - "syn 2.0.111", + "syn", "typify-impl", ] @@ -7289,12 +7000,6 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" -[[package]] -name = "unicode-xid" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" - [[package]] name = "unit-prefix" version = "0.5.2" @@ -7315,9 +7020,9 @@ checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" [[package]] name = "url" -version = "2.5.7" +version = "2.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08bc136a29a3d1758e07a9cca267be308aeebf5cfd5a10f3f67ab2097683ef5b" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" dependencies = [ "form_urlencoded", "idna", @@ -7345,11 +7050,11 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.21.0" +version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b672338555252d43fd2240c714dc444b8c6fb0a5c5335e65a07bba7742735ddb" +checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" dependencies = [ - "getrandom 0.4.1", + "getrandom 0.3.4", "js-sys", "serde_core", "wasm-bindgen", @@ -7405,7 +7110,7 @@ version = "0.10.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4a41a936651c3b8206339898466fd4f39596cfedf789e5ee6a8d02a45feb45f" dependencies = [ - "ahash 0.8.12", + "ahash", "anyhow", "bytes", "chrono", @@ -7466,16 +7171,7 @@ version = "1.0.1+wasi-0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" dependencies = [ - "wit-bindgen 0.46.0", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" -dependencies = [ - "wit-bindgen 0.51.0", + "wit-bindgen", ] [[package]] @@ -7529,7 +7225,7 @@ dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 2.0.111", + "syn", "wasm-bindgen-shared", ] @@ -7542,28 +7238,6 @@ dependencies = [ "unicode-ident", ] -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap 2.12.1", - "wasm-encoder", - "wasmparser", -] - [[package]] name = "wasm-streams" version = "0.4.2" @@ -7577,18 +7251,6 @@ dependencies = [ "web-sys", ] -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags", - "hashbrown 0.15.5", - "indexmap 2.12.1", - "semver", -] - [[package]] name = "web-sys" version = "0.3.83" @@ -7615,14 +7277,14 @@ version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.4", + "webpki-roots 1.0.5", ] [[package]] name = "webpki-roots" -version = "1.0.4" +version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2878ef029c47c6e8cf779119f20fcf52bde7ad42a731b2a304bc221df17571e" +checksum = "12bed680863276c63889429bfd6cab3b99943659923822de1c8a39c49e4d722c" dependencies = [ "rustls-pki-types", ] @@ -7667,7 +7329,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -7678,7 +7340,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -7951,124 +7613,18 @@ version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" -[[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap 2.12.1", - "prettyplease", - "syn 2.0.111", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn 2.0.111", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" -dependencies = [ - "anyhow", - "bitflags", - "indexmap 2.12.1", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", -] - -[[package]] -name = "wit-parser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap 2.12.1", - "log", - "semver", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] - [[package]] name = "writeable" version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" -[[package]] -name = "wyz" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" -dependencies = [ - "tap", -] - [[package]] name = "xmlparser" version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" -[[package]] -name = "xz2" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" -dependencies = [ - "lzma-sys", -] - [[package]] name = "yansi" version = "1.0.1" @@ -8094,28 +7650,28 @@ checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", "synstructure", ] [[package]] name = "zerocopy" -version = "0.8.31" +version = "0.8.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fd74ec98b9250adb3ca554bdde269adf631549f51d8a8f8f0a10b50f1cb298c3" +checksum = "668f5168d10b9ee831de31933dc111a459c97ec93225beb307aed970d1372dfd" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.31" +version = "0.8.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8a8d209fdf45cf5138cbb5a506f6b52522a25afccc534d1475dad8e31105c6a" +checksum = "2c7962b26b0a8685668b671ee4b54d007a67d4eaf05fda79ac0ecf41e32270f1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] @@ -8135,7 +7691,7 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", "synstructure", ] @@ -8175,14 +7731,20 @@ checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.111", + "syn", ] [[package]] name = "zlib-rs" -version = "0.5.3" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40990edd51aae2c2b6907af74ffb635029d5788228222c4bb811e9351c0caad3" + +[[package]] +name = "zmij" +version = "1.0.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36134c44663532e6519d7a6dfdbbe06f6f8192bde8ae9ed076e9b213f0e31df7" +checksum = "2fc5a66a20078bf1251bde995aa2fdcc4b800c70b5d92dd2c62abc5c60f679f8" [[package]] name = "zstd" diff --git a/Cargo.toml b/Cargo.toml index e114217b29..23b498214a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -42,14 +42,14 @@ rust-version = "1.88" anyhow = "1.0.72" apache-avro = { version = "0.21", features = ["zstandard"] } array-init = "2" -arrow-arith = "57.0" -arrow-array = "57.0" -arrow-buffer = "57.0" -arrow-cast = "57.0" -arrow-ord = "57.0" -arrow-schema = "57.0" -arrow-select = "57.0" -arrow-string = "57.0" +arrow-arith = "57.1" +arrow-array = "57.1" +arrow-buffer = "57.1" +arrow-cast = "57.1" +arrow-ord = "57.1" +arrow-schema = "57.1" +arrow-select = "57.1" +arrow-string = "57.1" as-any = "0.3.2" async-trait = "0.1.89" aws-config = "1.8.7" @@ -62,9 +62,9 @@ bytes = "1.11" chrono = "0.4.41" clap = { version = "4.5.48", features = ["derive", "cargo"] } dashmap = "6" -datafusion = "51.0" -datafusion-cli = "51.0" -datafusion-sqllogictest = "51.0" +datafusion = "52.1" +datafusion-cli = "52.1" +datafusion-sqllogictest = "52.1" derive_builder = "0.20" dirs = "6" enum-ordinalize = "4.3.0" @@ -100,7 +100,7 @@ murmur3 = "0.5.2" once_cell = "1.20" opendal = "0.55.0" ordered-float = "4" -parquet = "57.0" +parquet = "57.1" pilota = "0.11.10" pretty_assertions = "1.4" rand = "0.8.5" @@ -130,4 +130,4 @@ url = "2.5.7" uuid = { version = "1.18", features = ["v7"] } volo = "0.10.6" volo-thrift = "0.10.8" -zstd = "0.13.3" \ No newline at end of file +zstd = "0.13.3" diff --git a/bindings/python/Cargo.lock b/bindings/python/Cargo.lock index e2a8650c1d..760ea8c1a3 100644 --- a/bindings/python/Cargo.lock +++ b/bindings/python/Cargo.lock @@ -169,9 +169,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4df8bb5b0bd64c0b9bc61317fcc480bad0f00e56d3bc32c69a4c8dada4786bae" +checksum = "cb372a7cbcac02a35d3fb7b3fc1f969ec078e871f9bb899bf00a2e1809bec8a3" dependencies = [ "arrow-arith", "arrow-array", @@ -191,9 +191,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1a640186d3bd30a24cb42264c2dafb30e236a6f50d510e56d40b708c9582491" +checksum = "0f377dcd19e440174596d83deb49cd724886d91060c07fec4f67014ef9d54049" dependencies = [ "arrow-array", "arrow-buffer", @@ -205,9 +205,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "219fe420e6800979744c8393b687afb0252b3f8a89b91027d27887b72aa36d31" +checksum = "a23eaff85a44e9fa914660fb0d0bb00b79c4a3d888b5334adb3ea4330c84f002" dependencies = [ "ahash", "arrow-buffer", @@ -216,7 +216,7 @@ dependencies = [ "chrono", "chrono-tz", "half", - "hashbrown 0.16.0", + "hashbrown 0.16.1", "num-complex", "num-integer", "num-traits", @@ -224,9 +224,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76885a2697a7edf6b59577f568b456afc94ce0e2edc15b784ce3685b6c3c5c27" +checksum = "a2819d893750cb3380ab31ebdc8c68874dd4429f90fd09180f3c93538bd21626" dependencies = [ "bytes", "half", @@ -236,13 +236,14 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c9ebb4c987e6b3b236fb4a14b20b34835abfdd80acead3ccf1f9bf399e1f168" +checksum = "e3d131abb183f80c450d4591dc784f8d7750c50c6e2bc3fcaad148afc8361271" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", + "arrow-ord", "arrow-schema", "arrow-select", "atoi", @@ -257,9 +258,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92386159c8d4bce96f8bd396b0642a0d544d471bdc2ef34d631aec80db40a09c" +checksum = "2275877a0e5e7e7c76954669366c2aa1a829e340ab1f612e647507860906fb6b" dependencies = [ "arrow-array", "arrow-cast", @@ -272,9 +273,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "727681b95de313b600eddc2a37e736dcb21980a40f640314dcf360e2f36bc89b" +checksum = "05738f3d42cb922b9096f7786f606fcb8669260c2640df8490533bb2fa38c9d3" dependencies = [ "arrow-buffer", "arrow-schema", @@ -285,9 +286,9 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da9ba92e3de170295c98a84e5af22e2b037f0c7b32449445e6c493b5fca27f27" +checksum = "3d09446e8076c4b3f235603d9ea7c5494e73d441b01cd61fb33d7254c11964b3" dependencies = [ "arrow-array", "arrow-buffer", @@ -301,9 +302,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b969b4a421ae83828591c6bf5450bd52e6d489584142845ad6a861f42fe35df8" +checksum = "371ffd66fa77f71d7628c63f209c9ca5341081051aa32f9c8020feb0def787c0" dependencies = [ "arrow-array", "arrow-buffer", @@ -312,7 +313,7 @@ dependencies = [ "arrow-schema", "chrono", "half", - "indexmap 2.12.0", + "indexmap 2.12.1", "itoa", "lexical-core", "memchr", @@ -325,9 +326,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "141c05298b21d03e88062317a1f1a73f5ba7b6eb041b350015b1cd6aabc0519b" +checksum = "cbc94fc7adec5d1ba9e8cd1b1e8d6f72423b33fe978bf1f46d970fafab787521" dependencies = [ "arrow-array", "arrow-buffer", @@ -338,9 +339,9 @@ dependencies = [ [[package]] name = "arrow-pyarrow" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcfb2be2e9096236f449c11f425cddde18c4cc540f516d90f066f10a29ed515" +checksum = "fbd810e3997bae72f58cda57231ccb0a2fda07911ca1b0a5718cbf9379abb297" dependencies = [ "arrow-array", "arrow-data", @@ -350,9 +351,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5f3c06a6abad6164508ed283c7a02151515cef3de4b4ff2cebbcaeb85533db2" +checksum = "169676f317157dc079cc5def6354d16db63d8861d61046d2f3883268ced6f99f" dependencies = [ "arrow-array", "arrow-buffer", @@ -363,9 +364,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cfa7a03d1eee2a4d061476e1840ad5c9867a544ca6c4c59256496af5d0a8be5" +checksum = "d27609cd7dd45f006abae27995c2729ef6f4b9361cde1ddd019dc31a5aa017e0" dependencies = [ "bitflags", "serde_core", @@ -374,9 +375,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bafa595babaad59f2455f4957d0f26448fb472722c186739f4fac0823a1bdb47" +checksum = "ae980d021879ea119dd6e2a13912d81e64abed372d53163e804dfe84639d8010" dependencies = [ "ahash", "arrow-array", @@ -388,9 +389,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32f46457dbbb99f2650ff3ac23e46a929e0ab81db809b02aa5511c258348bef2" +checksum = "cf35e8ef49dcf0c5f6d175edee6b8af7b45611805333129c541a8b89a0fc0534" dependencies = [ "arrow-array", "arrow-buffer", @@ -423,19 +424,15 @@ dependencies = [ [[package]] name = "async-compression" -version = "0.4.19" +version = "0.4.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06575e6a9673580f52661c92107baabffbf41e2141373441cbcdc47cb733003c" +checksum = "98ec5f6c2f8bc326c994cb9e241cc257ddaba9afa8555a43cffbb5dd86efaa37" dependencies = [ - "bzip2 0.5.2", - "flate2", + "compression-codecs", + "compression-core", "futures-core", - "memchr", "pin-project-lite", "tokio", - "xz2", - "zstd", - "zstd-safe", ] [[package]] @@ -466,7 +463,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -596,7 +593,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -640,18 +637,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" - -[[package]] -name = "bzip2" -version = "0.5.2" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" -dependencies = [ - "bzip2-sys", -] +checksum = "b35204fbdc0b3f4446b89fc1ac2cf84a8a68971995d0bf2e925ec7cd960f9cb3" [[package]] name = "bzip2" @@ -662,16 +650,6 @@ dependencies = [ "libbz2-rs-sys", ] -[[package]] -name = "bzip2-sys" -version = "0.1.13+1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "cc" version = "1.2.43" @@ -731,6 +709,27 @@ dependencies = [ "unicode-width", ] +[[package]] +name = "compression-codecs" +version = "0.4.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0f7ac3e5b97fdce45e8922fb05cae2c37f7bbd63d30dd94821dacfd8f3f2bf2" +dependencies = [ + "bzip2", + "compression-core", + "flate2", + "liblzma", + "memchr", + "zstd", + "zstd-safe", +] + +[[package]] +name = "compression-core" +version = "0.4.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75984efb6ed102a0d42db99afb6c1948f0380d1d91808d5529916e6c08b49d8d" + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -921,7 +920,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -935,7 +934,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -946,7 +945,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -957,7 +956,7 @@ checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" dependencies = [ "darling_core 0.21.3", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -976,15 +975,15 @@ dependencies = [ [[package]] name = "datafusion" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ba7cb113e9c0bedf9e9765926031e132fa05a1b09ba6e93a6d1a4d7044457b8" +checksum = "d12ee9fdc6cdb5898c7691bb994f0ba606c4acc93a2258d78bb9f26ff8158bb3" dependencies = [ "arrow", "arrow-schema", "async-trait", "bytes", - "bzip2 0.6.1", + "bzip2", "chrono", "datafusion-catalog", "datafusion-catalog-listing", @@ -1014,27 +1013,26 @@ dependencies = [ "flate2", "futures", "itertools 0.14.0", + "liblzma", "log", "object_store", "parking_lot", "parquet", "rand 0.9.2", "regex", - "rstest", "sqlparser", "tempfile", "tokio", "url", "uuid", - "xz2", "zstd", ] [[package]] name = "datafusion-catalog" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "66a3a799f914a59b1ea343906a0486f17061f39509af74e874a866428951130d" +checksum = "462dc9ef45e5d688aeaae49a7e310587e81b6016b9d03bace5626ad0043e5a9e" dependencies = [ "arrow", "async-trait", @@ -1057,9 +1055,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db1b113c80d7a0febcd901476a57aef378e717c54517a163ed51417d87621b0" +checksum = "1b96dbf1d728fc321817b744eb5080cdd75312faa6980b338817f68f3caa4208" dependencies = [ "arrow", "async-trait", @@ -1076,22 +1074,21 @@ dependencies = [ "itertools 0.14.0", "log", "object_store", - "tokio", ] [[package]] name = "datafusion-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" +checksum = "3237a6ff0d2149af4631290074289cae548c9863c885d821315d54c6673a074a" dependencies = [ "ahash", "arrow", "arrow-ipc", "chrono", "half", - "hashbrown 0.14.5", - "indexmap 2.12.0", + "hashbrown 0.16.1", + "indexmap 2.12.1", "libc", "log", "object_store", @@ -1105,9 +1102,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b92065bbc6532c6651e2f7dd30b55cba0c7a14f860c7e1d15f165c41a1868d95" +checksum = "70b5e34026af55a1bfccb1ef0a763cf1f64e77c696ffcf5a128a278c31236528" dependencies = [ "futures", "log", @@ -1116,15 +1113,15 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fde13794244bc7581cd82f6fff217068ed79cdc344cafe4ab2c3a1c3510b38d6" +checksum = "1b2a6be734cc3785e18bbf2a7f2b22537f6b9fb960d79617775a51568c281842" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", - "bzip2 0.6.1", + "bzip2", "chrono", "datafusion-common", "datafusion-common-runtime", @@ -1139,21 +1136,21 @@ dependencies = [ "futures", "glob", "itertools 0.14.0", + "liblzma", "log", "object_store", "rand 0.9.2", "tokio", "tokio-util", "url", - "xz2", "zstd", ] [[package]] name = "datafusion-datasource-arrow" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "804fa9b4ecf3157982021770617200ef7c1b2979d57bec9044748314775a9aea" +checksum = "1739b9b07c9236389e09c74f770e88aff7055250774e9def7d3f4f56b3dcc7be" dependencies = [ "arrow", "arrow-ipc", @@ -1175,9 +1172,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61a1641a40b259bab38131c5e6f48fac0717bedb7dc93690e604142a849e0568" +checksum = "61c73bc54b518bbba7c7650299d07d58730293cfba4356f6f428cc94c20b7600" dependencies = [ "arrow", "async-trait", @@ -1198,9 +1195,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adeacdb00c1d37271176f8fb6a1d8ce096baba16ea7a4b2671840c5c9c64fe85" +checksum = "37812c8494c698c4d889374ecfabbff780f1f26d9ec095dd1bddfc2a8ca12559" dependencies = [ "arrow", "async-trait", @@ -1220,9 +1217,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43d0b60ffd66f28bfb026565d62b0a6cbc416da09814766a3797bba7d85a3cd9" +checksum = "2210937ecd9f0e824c397e73f4b5385c97cd1aff43ab2b5836fcfd2d321523fb" dependencies = [ "arrow", "async-trait", @@ -1250,18 +1247,19 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b99e13947667b36ad713549237362afb054b2d8f8cc447751e23ec61202db07" +checksum = "2c825f969126bc2ef6a6a02d94b3c07abff871acf4d6dd759ce1255edb7923ce" [[package]] name = "datafusion-execution" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63695643190679037bc946ad46a263b62016931547bf119859c511f7ff2f5178" +checksum = "fa03ef05a2c2f90dd6c743e3e111078e322f4b395d20d4b4d431a245d79521ae" dependencies = [ "arrow", "async-trait", + "chrono", "dashmap", "datafusion-common", "datafusion-expr", @@ -1276,9 +1274,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9a4787cbf5feb1ab351f789063398f67654a6df75c4d37d7f637dc96f951a91" +checksum = "ef33934c1f98ee695cc51192cc5f9ed3a8febee84fdbcd9131bf9d3a9a78276f" dependencies = [ "arrow", "async-trait", @@ -1289,7 +1287,7 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "paste", "recursive", @@ -1299,33 +1297,40 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ce2fb1b8c15c9ac45b0863c30b268c69dc9ee7a1ee13ecf5d067738338173dc" +checksum = "000c98206e3dd47d2939a94b6c67af4bfa6732dd668ac4fafdbde408fd9134ea" dependencies = [ "arrow", "datafusion-common", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "paste", ] [[package]] name = "datafusion-ffi" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec510e7787641279b0336e8b79e4b7bd1385d5976875ff9b97f4269ce5231a67" +checksum = "30f57f7f63a25a0b78b3f2a5e18c0ecbd54851b64064ac0d5a9eb05efd5586d2" dependencies = [ "abi_stable", "arrow", "arrow-schema", "async-ffi", "async-trait", - "datafusion", + "datafusion-catalog", "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "datafusion-proto", "datafusion-proto-common", + "datafusion-session", "futures", "log", "prost", @@ -1335,9 +1340,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "794a9db7f7b96b3346fc007ff25e994f09b8f0511b4cf7dff651fadfe3ebb28f" +checksum = "379b01418ab95ca947014066248c22139fe9af9289354de10b445bd000d5d276" dependencies = [ "arrow", "arrow-buffer", @@ -1345,6 +1350,7 @@ dependencies = [ "blake2", "blake3", "chrono", + "chrono-tz", "datafusion-common", "datafusion-doc", "datafusion-execution", @@ -1365,9 +1371,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" +checksum = "fd00d5454ba4c3f8ebbd04bd6a6a9dc7ced7c56d883f70f2076c188be8459e4c" dependencies = [ "ahash", "arrow", @@ -1386,9 +1392,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" +checksum = "aec06b380729a87210a4e11f555ec2d729a328142253f8d557b87593622ecc9f" dependencies = [ "ahash", "arrow", @@ -1399,9 +1405,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae5c06eed03918dc7fe7a9f082a284050f0e9ecf95d72f57712d1496da03b8c4" +checksum = "904f48d45e0f1eb7d0eb5c0f80f2b5c6046a85454364a6b16a2e0b46f62e7dff" dependencies = [ "arrow", "arrow-ord", @@ -1422,9 +1428,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db4fed1d71738fbe22e2712d71396db04c25de4111f1ec252b8f4c6d3b25d7f5" +checksum = "e9a0d20e2b887e11bee24f7734d780a2588b925796ac741c3118dd06d5aa77f0" dependencies = [ "arrow", "async-trait", @@ -1438,9 +1444,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d92206aa5ae21892f1552b4d61758a862a70956e6fd7a95cb85db1de74bc6d1" +checksum = "d3414b0a07e39b6979fe3a69c7aa79a9f1369f1d5c8e52146e66058be1b285ee" dependencies = [ "arrow", "datafusion-common", @@ -1456,9 +1462,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53ae9bcc39800820d53a22d758b3b8726ff84a5a3e24cecef04ef4e5fdf1c7cc" +checksum = "5bf2feae63cd4754e31add64ce75cae07d015bce4bb41cd09872f93add32523a" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1466,20 +1472,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1063ad4c9e094b3f798acee16d9a47bd7372d9699be2de21b05c3bd3f34ab848" +checksum = "c4fe888aeb6a095c4bcbe8ac1874c4b9a4c7ffa2ba849db7922683ba20875aaf" dependencies = [ "datafusion-doc", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] name = "datafusion-optimizer" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f35f9ec5d08b87fd1893a30c2929f2559c2f9806ca072d8fefca5009dc0f06a" +checksum = "8a6527c063ae305c11be397a86d8193936f4b84d137fe40bd706dfc178cf733c" dependencies = [ "arrow", "chrono", @@ -1487,7 +1493,7 @@ dependencies = [ "datafusion-expr", "datafusion-expr-common", "datafusion-physical-expr", - "indexmap 2.12.0", + "indexmap 2.12.1", "itertools 0.14.0", "log", "recursive", @@ -1497,9 +1503,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" +checksum = "0bb028323dd4efd049dd8a78d78fe81b2b969447b39c51424167f973ac5811d9" dependencies = [ "ahash", "arrow", @@ -1509,19 +1515,21 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", "half", - "hashbrown 0.14.5", - "indexmap 2.12.0", + "hashbrown 0.16.1", + "indexmap 2.12.1", "itertools 0.14.0", "parking_lot", "paste", "petgraph", + "recursive", + "tokio", ] [[package]] name = "datafusion-physical-expr-adapter" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f9ff2dbd476221b1f67337699eff432781c4e6e1713d2aefdaa517dfbf79768" +checksum = "78fe0826aef7eab6b4b61533d811234a7a9e5e458331ebbf94152a51fc8ab433" dependencies = [ "arrow", "datafusion-common", @@ -1534,23 +1542,26 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" +checksum = "cfccd388620734c661bd8b7ca93c44cdd59fecc9b550eea416a78ffcbb29475f" dependencies = [ "ahash", "arrow", + "chrono", "datafusion-common", "datafusion-expr-common", - "hashbrown 0.14.5", + "hashbrown 0.16.1", + "indexmap 2.12.1", "itertools 0.14.0", + "parking_lot", ] [[package]] name = "datafusion-physical-optimizer" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce9804f799acd7daef3be7aaffe77c0033768ed8fdbf5fb82fc4c5f2e6bc14e6" +checksum = "bde5fa10e73259a03b705d5fddc136516814ab5f441b939525618a4070f5a059" dependencies = [ "arrow", "datafusion-common", @@ -1567,28 +1578,28 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" +checksum = "0e1098760fb29127c24cc9ade3277051dc73c9ed0ac0131bd7bcd742e0ad7470" dependencies = [ "ahash", "arrow", "arrow-ord", "arrow-schema", "async-trait", - "chrono", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", + "datafusion-functions", "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "futures", "half", - "hashbrown 0.14.5", - "indexmap 2.12.0", + "hashbrown 0.16.1", + "indexmap 2.12.1", "itertools 0.14.0", "log", "parking_lot", @@ -1598,9 +1609,9 @@ dependencies = [ [[package]] name = "datafusion-proto" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d368093a98a17d1449b1083ac22ed16b7128e4c67789991869480d8c4a40ecb9" +checksum = "0cf75daf56aa6b1c6867cc33ff0fb035d517d6d06737fd355a3e1ef67cba6e7a" dependencies = [ "arrow", "chrono", @@ -1625,9 +1636,9 @@ dependencies = [ [[package]] name = "datafusion-proto-common" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b6aef3d5e5c1d2bc3114c4876730cb76a9bdc5a8df31ef1b6db48f0c1671895" +checksum = "12a0cb3cce232a3de0d14ef44b58a6537aeb1362cfb6cf4d808691ddbb918956" dependencies = [ "arrow", "datafusion-common", @@ -1636,9 +1647,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac2c2498a1f134a9e11a9f5ed202a2a7d7e9774bd9249295593053ea3be999db" +checksum = "64d0fef4201777b52951edec086c21a5b246f3c82621569ddb4a26f488bc38a9" dependencies = [ "arrow", "datafusion-common", @@ -1653,9 +1664,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f96eebd17555386f459037c65ab73aae8df09f464524c709d6a3134ad4f4776" +checksum = "f71f1e39e8f2acbf1c63b0e93756c2e970a64729dab70ac789587d6237c4fde0" dependencies = [ "async-trait", "datafusion-common", @@ -1667,16 +1678,16 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "51.0.0" +version = "52.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fc195fe60634b2c6ccfd131b487de46dc30eccae8a3c35a13f136e7f440414f" +checksum = "f44693cfcaeb7a9f12d71d1c576c3a6dc025a12cef209375fa2d16fb3b5670ee" dependencies = [ "arrow", "bigdecimal", "chrono", "datafusion-common", "datafusion-expr", - "indexmap 2.12.0", + "indexmap 2.12.1", "log", "recursive", "regex", @@ -1711,7 +1722,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -1721,7 +1732,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -1744,7 +1755,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -1895,6 +1906,12 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + [[package]] name = "form_urlencoded" version = "1.2.2" @@ -1960,7 +1977,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -1975,12 +1992,6 @@ version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" -[[package]] -name = "futures-timer" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" - [[package]] name = "futures-util" version = "0.3.31" @@ -2086,10 +2097,6 @@ name = "hashbrown" version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" -dependencies = [ - "ahash", - "allocator-api2", -] [[package]] name = "hashbrown" @@ -2097,14 +2104,19 @@ version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ - "foldhash", + "foldhash 0.1.5", ] [[package]] name = "hashbrown" -version = "0.16.0" +version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] [[package]] name = "heck" @@ -2458,12 +2470,12 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.12.0" +version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6717a8d2a5a929a1a2eb43a12812498ed141a0bcfb7e8f7844fbdbe4303bba9f" +checksum = "0ad4bb2b565bca0645f4d68c5c9af97fba094e9791da685bf83cb5f3ce74acf2" dependencies = [ "equivalent", - "hashbrown 0.16.0", + "hashbrown 0.16.1", "serde", "serde_core", ] @@ -2485,9 +2497,9 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" [[package]] name = "inventory" -version = "0.3.21" +version = "0.3.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc61209c082fbeb19919bee74b176221b27223e27b65d781eb91af24eb1fb46e" +checksum = "009ae045c87e7082cb72dab0ccd01ae075dd00141ddc108f43a0ea150a9e7227" dependencies = [ "rustversion", ] @@ -2555,7 +2567,7 @@ checksum = "980af8b43c3ad5d8d349ace167ec8170839f753a42d233ba19e08afe1850fa69" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -2672,6 +2684,26 @@ dependencies = [ "winapi", ] +[[package]] +name = "liblzma" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73c36d08cad03a3fbe2c4e7bb3a9e84c57e4ee4135ed0b065cade3d98480c648" +dependencies = [ + "liblzma-sys", +] + +[[package]] +name = "liblzma-sys" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01b9596486f6d60c3bbe644c0e1be1aa6ccc472ad630fe8927b456973d7cb736" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + [[package]] name = "libm" version = "0.2.15" @@ -2722,24 +2754,13 @@ checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" [[package]] name = "lz4_flex" -version = "0.11.5" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08ab2867e3eeeca90e844d1940eab391c9dc5228783db2ed999acbc0a9ed375a" +checksum = "ab6473172471198271ff72e9379150e9dfd70d8e533e0752a27e515b48dd375e" dependencies = [ "twox-hash", ] -[[package]] -name = "lzma-sys" -version = "0.1.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" -dependencies = [ - "cc", - "libc", - "pkg-config", -] - [[package]] name = "md-5" version = "0.10.6" @@ -2835,9 +2856,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.2.0" +version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" +checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" [[package]] name = "num-integer" @@ -2985,9 +3006,9 @@ dependencies = [ [[package]] name = "parquet" -version = "57.0.0" +version = "57.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0f31027ef1af7549f7cec603a9a21dce706d3f8d7c2060a68f43c1773be95a" +checksum = "be3e4f6d320dd92bfa7d612e265d7d08bba0a240bab86af3425e1d255a511d89" dependencies = [ "ahash", "arrow-array", @@ -3004,7 +3025,7 @@ dependencies = [ "flate2", "futures", "half", - "hashbrown 0.16.0", + "hashbrown 0.16.1", "lz4_flex", "num-bigint", "num-integer", @@ -3040,7 +3061,7 @@ checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset", "hashbrown 0.15.5", - "indexmap 2.12.0", + "indexmap 2.12.1", "serde", ] @@ -3126,16 +3147,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.108", -] - -[[package]] -name = "proc-macro-crate" -version = "3.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" -dependencies = [ - "toml_edit", + "syn 2.0.113", ] [[package]] @@ -3167,7 +3179,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3237,7 +3249,7 @@ dependencies = [ "proc-macro2", "pyo3-macros-backend", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3250,7 +3262,7 @@ dependencies = [ "proc-macro2", "pyo3-build-config", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3425,7 +3437,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3454,7 +3466,7 @@ checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3492,12 +3504,6 @@ version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58" -[[package]] -name = "relative-path" -version = "1.9.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" - [[package]] name = "repr_offset" version = "0.2.2" @@ -3601,35 +3607,6 @@ dependencies = [ "byteorder", ] -[[package]] -name = "rstest" -version = "0.26.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5a3193c063baaa2a95a33f03035c8a72b83d97a54916055ba22d35ed3839d49" -dependencies = [ - "futures-timer", - "futures-util", - "rstest_macros", -] - -[[package]] -name = "rstest_macros" -version = "0.26.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0" -dependencies = [ - "cfg-if", - "glob", - "proc-macro-crate", - "proc-macro2", - "quote", - "regex", - "relative-path", - "rustc_version", - "syn 2.0.108", - "unicode-ident", -] - [[package]] name = "rust-ini" version = "0.21.3" @@ -3812,7 +3789,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3836,7 +3813,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3861,7 +3838,7 @@ dependencies = [ "chrono", "hex", "indexmap 1.9.3", - "indexmap 2.12.0", + "indexmap 2.12.1", "schemars 0.9.0", "schemars 1.0.4", "serde_core", @@ -3879,7 +3856,7 @@ dependencies = [ "darling 0.21.3", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -3975,7 +3952,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4028,7 +4005,7 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4040,7 +4017,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4062,9 +4039,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.108" +version = "2.0.113" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da58917d35242480a05c2897064da0a80589a2a0476c9a3f2fdc83b53502e917" +checksum = "678faa00651c9eb72dd2020cbdf275d92eccb2400d568e419efdd64838145cb4" dependencies = [ "proc-macro2", "quote", @@ -4088,7 +4065,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4133,7 +4110,7 @@ checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4149,30 +4126,30 @@ dependencies = [ [[package]] name = "time" -version = "0.3.47" +version = "0.3.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", - "serde_core", + "serde", "time-core", "time-macros", ] [[package]] name = "time-core" -version = "0.1.8" +version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" +checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b" [[package]] name = "time-macros" -version = "0.2.27" +version = "0.2.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3" dependencies = [ "num-conv", "time-core", @@ -4235,7 +4212,7 @@ checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4250,9 +4227,9 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.16" +version = "0.7.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14307c986784f72ef81c89db7d9e28d6ac26d16213b109ea501696195e6e3ce5" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" dependencies = [ "bytes", "futures-core", @@ -4261,36 +4238,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "toml_datetime" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2cdb639ebbc97961c51720f858597f7f24c4fc295327923af55b74c3c724533" -dependencies = [ - "serde_core", -] - -[[package]] -name = "toml_edit" -version = "0.23.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6485ef6d0d9b5d0ec17244ff7eb05310113c3f316f2d14200d4de56b3cb98f8d" -dependencies = [ - "indexmap 2.12.0", - "toml_datetime", - "toml_parser", - "winnow", -] - -[[package]] -name = "toml_parser" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0cbe268d35bdb4bb5a56a2de88d0ad0eb70af5384a99d648cd4b3d04039800e" -dependencies = [ - "winnow", -] - [[package]] name = "tower" version = "0.5.2" @@ -4355,7 +4302,7 @@ checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4417,7 +4364,7 @@ checksum = "3c36781cc0e46a83726d9879608e4cf6c2505237e263a8eb8c24502989cfdb28" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4453,7 +4400,7 @@ checksum = "27a7a9b72ba121f6f1f6c3632b85604cac41aedb5ddc70accbebb6cac83de846" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4512,13 +4459,13 @@ checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" [[package]] name = "uuid" -version = "1.18.1" +version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f87b8aa10b915a06587d0dec516c282ff295b475d94abf425d62b57710070a2" +checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" dependencies = [ "getrandom 0.3.4", "js-sys", - "serde", + "serde_core", "wasm-bindgen", ] @@ -4607,7 +4554,7 @@ dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", "wasm-bindgen-shared", ] @@ -4714,7 +4661,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4725,7 +4672,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -4917,15 +4864,6 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" -[[package]] -name = "winnow" -version = "0.7.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21a0236b59786fed61e2a80582dd500fe61f18b5dca67a4a067d0bc9039339cf" -dependencies = [ - "memchr", -] - [[package]] name = "wit-bindgen" version = "0.46.0" @@ -4938,15 +4876,6 @@ version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" -[[package]] -name = "xz2" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" -dependencies = [ - "lzma-sys", -] - [[package]] name = "yoke" version = "0.8.1" @@ -4966,7 +4895,7 @@ checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", "synstructure", ] @@ -4987,7 +4916,7 @@ checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] @@ -5007,7 +4936,7 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", "synstructure", ] @@ -5047,7 +4976,7 @@ checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.108", + "syn 2.0.113", ] [[package]] diff --git a/bindings/python/Cargo.toml b/bindings/python/Cargo.toml index c179e89f99..9279983f96 100644 --- a/bindings/python/Cargo.toml +++ b/bindings/python/Cargo.toml @@ -31,11 +31,11 @@ license = "Apache-2.0" crate-type = ["cdylib"] [dependencies] -arrow = { version = "57.0", features = ["pyarrow", "chrono-tz"] } +arrow = { version = "57.1", features = ["pyarrow", "chrono-tz"] } iceberg = { path = "../../crates/iceberg" } pyo3 = { version = "0.26", features = ["extension-module", "abi3-py310"] } iceberg-datafusion = { path = "../../crates/integrations/datafusion" } -datafusion-ffi = { version = "51.0" } +datafusion-ffi = { version = "52.1" } tokio = { version = "1.46.1", default-features = false } [profile.release] diff --git a/bindings/python/DEPENDENCIES.rust.tsv b/bindings/python/DEPENDENCIES.rust.tsv index 5fe0da5b2a..f514c29bca 100644 --- a/bindings/python/DEPENDENCIES.rust.tsv +++ b/bindings/python/DEPENDENCIES.rust.tsv @@ -15,24 +15,24 @@ ar_archive_writer@0.2.0 X array-init@2.1.0 X X arrayref@0.3.9 X arrayvec@0.7.6 X X -arrow@57.0.0 X -arrow-arith@57.0.0 X -arrow-array@57.0.0 X -arrow-buffer@57.0.0 X -arrow-cast@57.0.0 X -arrow-csv@57.0.0 X -arrow-data@57.0.0 X -arrow-ipc@57.0.0 X -arrow-json@57.0.0 X -arrow-ord@57.0.0 X -arrow-pyarrow@57.0.0 X -arrow-row@57.0.0 X -arrow-schema@57.0.0 X -arrow-select@57.0.0 X -arrow-string@57.0.0 X +arrow@57.1.0 X +arrow-arith@57.1.0 X +arrow-array@57.1.0 X +arrow-buffer@57.1.0 X +arrow-cast@57.1.0 X +arrow-csv@57.1.0 X +arrow-data@57.1.0 X +arrow-ipc@57.1.0 X +arrow-json@57.1.0 X +arrow-ord@57.1.0 X +arrow-pyarrow@57.1.0 X +arrow-row@57.1.0 X +arrow-schema@57.1.0 X +arrow-select@57.1.0 X +arrow-string@57.1.0 X as-any@0.3.2 X X as_derive_utils@0.11.0 X X -async-compression@0.4.19 X X +async-compression@0.4.36 X X async-ffi@0.5.0 X async-lock@3.4.1 X X async-trait@0.1.89 X X @@ -47,6 +47,7 @@ bitflags@2.10.0 X X blake2@0.10.6 X X blake3@1.8.2 X X X block-buffer@0.10.4 X X +bnum@0.12.1 X X bon@3.8.1 X X bon-macros@3.8.1 X X brotli@8.0.2 X X @@ -54,15 +55,15 @@ brotli-decompressor@5.0.0 X X bumpalo@3.19.0 X X bytemuck@1.24.0 X X X byteorder@1.5.0 X X -bytes@1.10.1 X -bzip2@0.5.2 X X +bytes@1.11.0 X bzip2@0.6.1 X X -bzip2-sys@0.1.13+1.0.8 X X cc@1.2.43 X X cfg-if@1.0.4 X X chrono@0.4.42 X X chrono-tz@0.10.4 X X comfy-table@7.1.2 X +compression-codecs@0.4.35 X X +compression-core@0.4.31 X X concurrent-queue@2.5.0 X X const-oid@0.9.6 X X const-random@0.1.18 X X @@ -89,40 +90,40 @@ darling_core@0.21.3 X darling_macro@0.20.11 X darling_macro@0.21.3 X dashmap@6.1.0 X -datafusion@51.0.0 X -datafusion-catalog@51.0.0 X -datafusion-catalog-listing@51.0.0 X -datafusion-common@51.0.0 X -datafusion-common-runtime@51.0.0 X -datafusion-datasource@51.0.0 X -datafusion-datasource-arrow@51.0.0 X -datafusion-datasource-csv@51.0.0 X -datafusion-datasource-json@51.0.0 X -datafusion-datasource-parquet@51.0.0 X -datafusion-doc@51.0.0 X -datafusion-execution@51.0.0 X -datafusion-expr@51.0.0 X -datafusion-expr-common@51.0.0 X -datafusion-ffi@51.0.0 X -datafusion-functions@51.0.0 X -datafusion-functions-aggregate@51.0.0 X -datafusion-functions-aggregate-common@51.0.0 X -datafusion-functions-nested@51.0.0 X -datafusion-functions-table@51.0.0 X -datafusion-functions-window@51.0.0 X -datafusion-functions-window-common@51.0.0 X -datafusion-macros@51.0.0 X -datafusion-optimizer@51.0.0 X -datafusion-physical-expr@51.0.0 X -datafusion-physical-expr-adapter@51.0.0 X -datafusion-physical-expr-common@51.0.0 X -datafusion-physical-optimizer@51.0.0 X -datafusion-physical-plan@51.0.0 X -datafusion-proto@51.0.0 X -datafusion-proto-common@51.0.0 X -datafusion-pruning@51.0.0 X -datafusion-session@51.0.0 X -datafusion-sql@51.0.0 X +datafusion@52.1.0 X +datafusion-catalog@52.1.0 X +datafusion-catalog-listing@52.1.0 X +datafusion-common@52.1.0 X +datafusion-common-runtime@52.1.0 X +datafusion-datasource@52.1.0 X +datafusion-datasource-arrow@52.1.0 X +datafusion-datasource-csv@52.1.0 X +datafusion-datasource-json@52.1.0 X +datafusion-datasource-parquet@52.1.0 X +datafusion-doc@52.1.0 X +datafusion-execution@52.1.0 X +datafusion-expr@52.1.0 X +datafusion-expr-common@52.1.0 X +datafusion-ffi@52.1.0 X +datafusion-functions@52.1.0 X +datafusion-functions-aggregate@52.1.0 X +datafusion-functions-aggregate-common@52.1.0 X +datafusion-functions-nested@52.1.0 X +datafusion-functions-table@52.1.0 X +datafusion-functions-window@52.1.0 X +datafusion-functions-window-common@52.1.0 X +datafusion-macros@52.1.0 X +datafusion-optimizer@52.1.0 X +datafusion-physical-expr@52.1.0 X +datafusion-physical-expr-adapter@52.1.0 X +datafusion-physical-expr-common@52.1.0 X +datafusion-physical-optimizer@52.1.0 X +datafusion-physical-plan@52.1.0 X +datafusion-proto@52.1.0 X +datafusion-proto-common@52.1.0 X +datafusion-pruning@52.1.0 X +datafusion-session@52.1.0 X +datafusion-sql@52.1.0 X derive_builder@0.20.2 X X derive_builder_core@0.20.2 X X derive_builder_macro@0.20.2 X X @@ -131,10 +132,12 @@ displaydoc@0.2.5 X X dissimilar@1.0.10 X either@1.15.0 X X equivalent@1.0.2 X X +erased-serde@0.4.9 X X errno@0.3.14 X X event-listener@5.4.1 X X event-listener-strategy@0.5.4 X X expect-test@1.5.1 X X +fastnum@0.7.4 X X fastrand@2.3.0 X X find-msvc-tools@0.1.4 X X fixedbitset@0.5.7 X X @@ -142,6 +145,7 @@ flatbuffers@25.9.23 X flate2@1.1.5 X X fnv@1.0.7 X X foldhash@0.1.5 X +foldhash@0.2.0 X form_urlencoded@1.2.2 X X futures@0.3.31 X X futures-channel@0.3.31 X X @@ -151,7 +155,6 @@ futures-io@0.3.31 X X futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X -futures-timer@3.0.3 X X futures-util@0.3.31 X X generational-arena@0.2.9 X generic-array@0.14.9 X @@ -162,7 +165,7 @@ gloo-timers@0.3.0 X X half@2.7.1 X X hashbrown@0.14.5 X X hashbrown@0.15.5 X X -hashbrown@0.16.0 X X +hashbrown@0.16.1 X X heck@0.5.0 X X hex@0.4.3 X X hmac@0.12.1 X X @@ -189,9 +192,10 @@ icu_provider@2.1.0 X ident_case@1.0.1 X X idna@1.1.0 X X idna_adapter@1.2.1 X X -indexmap@2.12.0 X X +indexmap@2.12.1 X X indoc@2.0.7 X X integer-encoding@3.0.4 X +inventory@0.3.22 X X ipnet@2.11.0 X X iri-string@0.7.8 X X itertools@0.13.0 X X @@ -211,14 +215,15 @@ lexical-write-integer@1.0.6 X X libbz2-rs-sys@0.2.2 X libc@0.2.177 X X libloading@0.7.4 X +liblzma@0.4.5 X X +liblzma-sys@0.4.4 X X libm@0.2.15 X libz-rs-sys@0.5.2 X linux-raw-sys@0.11.0 X X X litemap@0.8.1 X lock_api@0.4.14 X X log@0.4.28 X X -lz4_flex@0.11.5 X -lzma-sys@0.1.20 X X +lz4_flex@0.12.0 X md-5@0.10.6 X X memchr@2.7.6 X X memoffset@0.9.1 X @@ -239,7 +244,7 @@ ordered-float@4.6.0 X parking@2.2.1 X X parking_lot@0.12.5 X X parking_lot_core@0.9.12 X X -parquet@57.0.0 X +parquet@57.1.0 X paste@1.0.15 X X percent-encoding@2.3.2 X X petgraph@0.8.3 X X @@ -253,7 +258,6 @@ portable-atomic-util@0.2.4 X X potential_utf@0.1.4 X ppv-lite86@0.2.21 X X prettyplease@0.2.37 X X -proc-macro-crate@3.4.0 X X proc-macro2@1.0.103 X X prost@0.14.1 X prost-derive@0.14.1 X @@ -281,15 +285,11 @@ regex@1.12.2 X X regex-automata@0.4.13 X X regex-lite@0.1.8 X X regex-syntax@0.8.8 X X -relative-path@1.9.3 X X repr_offset@0.2.2 X reqsign@0.16.5 X reqwest@0.12.24 X X ring@0.17.14 X X roaring@0.11.2 X X -rstest@0.26.1 X X -rstest_macros@0.26.1 X X -rust_decimal@1.39.0 X rustc_version@0.4.1 X X rustix@1.1.2 X X X rustls@0.23.34 X X X @@ -302,6 +302,7 @@ scopeguard@1.2.0 X X semver@1.0.27 X X seq-macro@0.3.6 X X serde@1.0.228 X X +serde-big-array@0.5.1 X X serde_bytes@0.11.19 X X serde_core@1.0.228 X X serde_derive@1.0.228 X X @@ -331,7 +332,7 @@ strum_macros@0.26.4 X strum_macros@0.27.2 X subtle@2.6.1 X syn@1.0.109 X X -syn@2.0.108 X X +syn@2.0.113 X X sync_wrapper@1.0.2 X synstructure@0.13.2 X tagptr@0.2.0 X X @@ -345,10 +346,7 @@ tinystr@0.8.2 X tokio@1.48.0 X tokio-macros@2.6.0 X tokio-rustls@0.26.4 X X -tokio-util@0.7.16 X -toml_datetime@0.7.3 X X -toml_edit@0.23.7 X X -toml_parser@1.0.4 X X +tokio-util@0.7.18 X tower@0.5.2 X tower-http@0.6.6 X tower-layer@0.3.3 X @@ -363,7 +361,10 @@ twox-hash@2.1.2 X typed-arena@2.0.2 X typed-builder@0.20.1 X X typed-builder-macro@0.20.1 X X +typeid@1.0.3 X X typenum@1.19.0 X X +typetag@0.2.21 X X +typetag-impl@0.2.21 X X typewit@1.14.2 X unicode-ident@1.0.20 X X X unicode-segmentation@1.12.0 X X @@ -372,7 +373,7 @@ unindent@0.2.4 X X untrusted@0.9.0 X url@2.5.7 X X utf8_iter@1.0.4 X X -uuid@1.18.1 X X +uuid@1.19.0 X X version_check@0.9.5 X X walkdir@2.5.0 X X want@0.3.1 X @@ -419,10 +420,8 @@ windows_x86_64_gnullvm@0.52.6 X X windows_x86_64_gnullvm@0.53.1 X X windows_x86_64_msvc@0.52.6 X X windows_x86_64_msvc@0.53.1 X X -winnow@0.7.13 X wit-bindgen@0.46.0 X X X writeable@0.6.2 X -xz2@0.1.7 X X yoke@0.8.1 X yoke-derive@0.8.1 X zerocopy@0.8.27 X X X diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml index 9f62b5716d..4c72f1203e 100644 --- a/bindings/python/pyproject.toml +++ b/bindings/python/pyproject.toml @@ -60,7 +60,7 @@ filterwarnings = [ dev = [ "maturin>=1.0,<2.0", "pytest>=8.3.2", - "datafusion==50.*", + "datafusion==52.*", "pyiceberg[sql-sqlite,pyarrow]", "fastavro>=1.11.1", ] diff --git a/bindings/python/src/datafusion_table_provider.rs b/bindings/python/src/datafusion_table_provider.rs index 8db7223b34..6a85ad2297 100644 --- a/bindings/python/src/datafusion_table_provider.rs +++ b/bindings/python/src/datafusion_table_provider.rs @@ -19,17 +19,53 @@ use std::collections::HashMap; use std::ffi::CString; use std::sync::Arc; +use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_ffi::table_provider::FFI_TableProvider; use iceberg::TableIdent; use iceberg::io::FileIO; use iceberg::table::StaticTable; use iceberg_datafusion::table::IcebergStaticTableProvider; -use pyo3::exceptions::PyRuntimeError; -use pyo3::prelude::*; -use pyo3::types::PyCapsule; +use pyo3::exceptions::{PyRuntimeError, PyValueError}; +use pyo3::prelude::{PyAnyMethods, PyCapsuleMethods, *}; +use pyo3::types::{PyAny, PyCapsule}; use crate::runtime::runtime; +pub(crate) fn validate_pycapsule(capsule: &Bound, name: &str) -> PyResult<()> { + let capsule_name = capsule.name()?; + if capsule_name.is_none() { + return Err(PyValueError::new_err(format!( + "Expected {name} PyCapsule to have name set." + ))); + } + + let capsule_name = capsule_name.unwrap().to_str()?; + if capsule_name != name { + return Err(PyValueError::new_err(format!( + "Expected name '{name}' in PyCapsule, instead got '{capsule_name}'" + ))); + } + + Ok(()) +} + +pub(crate) fn ffi_logical_codec_from_pycapsule( + obj: Bound, +) -> PyResult { + let attr_name = "__datafusion_logical_extension_codec__"; + let capsule = if obj.hasattr(attr_name)? { + obj.getattr(attr_name)?.call0()? + } else { + obj + }; + + let capsule = capsule.downcast::()?; + validate_pycapsule(capsule, "datafusion_logical_extension_codec")?; + + let codec = unsafe { capsule.reference::() }; + Ok(codec.clone()) +} + #[pyclass(name = "IcebergDataFusionTable")] pub struct PyIcebergDataFusionTable { inner: Arc, @@ -84,10 +120,18 @@ impl PyIcebergDataFusionTable { fn __datafusion_table_provider__<'py>( &self, py: Python<'py>, + session: Bound<'py, PyAny>, ) -> PyResult> { let capsule_name = CString::new("datafusion_table_provider").unwrap(); - let ffi_provider = FFI_TableProvider::new(self.inner.clone(), false, Some(runtime())); + let logical_codec = ffi_logical_codec_from_pycapsule(session)?; + + let ffi_provider = FFI_TableProvider::new_with_ffi_codec( + self.inner.clone(), + false, + Some(runtime()), + logical_codec, + ); PyCapsule::new(py, ffi_provider, Some(capsule_name)) } diff --git a/bindings/python/tests/test_datafusion_table_provider.py b/bindings/python/tests/test_datafusion_table_provider.py index 3b2d881e18..fa8ff4494a 100644 --- a/bindings/python/tests/test_datafusion_table_provider.py +++ b/bindings/python/tests/test_datafusion_table_provider.py @@ -25,10 +25,13 @@ import pyarrow as pa from pathlib import Path import datafusion +from packaging.version import Version -assert ( - datafusion.__version__ >= "45" -) # iceberg table provider only works for datafusion >= 45 +if Version(datafusion.__version__) < Version("52.0.0"): + pytest.skip( + "Iceberg table provider requires datafusion>=52 for FFI compatibility", + allow_module_level=True, + ) @pytest.fixture(scope="session") @@ -142,12 +145,12 @@ def test_register_pyiceberg_table( iceberg_table.append(arrow_table_with_null) # monkey patch the __datafusion_table_provider__ method to the iceberg table - def __datafusion_table_provider__(self): + def __datafusion_table_provider__(self, session): return IcebergDataFusionTable( identifier=self.name(), metadata_location=self.metadata_location, file_io_properties=self.io.properties, - ).__datafusion_table_provider__() + ).__datafusion_table_provider__(session) iceberg_table.__datafusion_table_provider__ = MethodType( __datafusion_table_provider__, iceberg_table diff --git a/bindings/python/tests/test_manifest.py b/bindings/python/tests/test_manifest.py index 096e549937..62d9145e0b 100644 --- a/bindings/python/tests/test_manifest.py +++ b/bindings/python/tests/test_manifest.py @@ -34,7 +34,27 @@ @pytest.fixture(autouse=True) def clear_global_manifests_cache() -> None: # Clear the global cache before each test - _manifests.cache_clear() # type: ignore + cache_clear = getattr(_manifests, "cache_clear", None) + if callable(cache_clear): + # pyiceberg<=0.10: _manifests was lru_cache-decorated. + cache_clear() + return + + # pyiceberg>=0.11: manifests are cached in module-level LRUCache. + from pyiceberg import manifest as manifest_module + + manifest_cache = getattr(manifest_module, "_manifest_cache", None) + manifest_cache_lock = getattr(manifest_module, "_manifest_cache_lock", None) + + if manifest_cache is None: + return + + if manifest_cache_lock is None: + manifest_cache.clear() + return + + with manifest_cache_lock: + manifest_cache.clear() def fetch_manifest_entry( @@ -49,33 +69,43 @@ def fetch_manifest_entry( # but this is the easiest for now until we # have the write part in there as well def _convert_entry(entry: Any) -> ManifestEntry: - data_file = DataFile( - content=DataFileContent(entry.data_file.content), - file_path=entry.data_file.file_path, - file_format=FileFormat(entry.data_file.file_format), - partition=[ + data_file_args = { + "content": DataFileContent(entry.data_file.content), + "file_path": entry.data_file.file_path, + "file_format": FileFormat(entry.data_file.file_format), + "partition": [ p.value() if p is not None else None for p in entry.data_file.partition ], - record_count=entry.data_file.record_count, - file_size_in_bytes=entry.data_file.file_size_in_bytes, - column_sizes=entry.data_file.column_sizes, - value_counts=entry.data_file.value_counts, - null_value_counts=entry.data_file.null_value_counts, - nan_value_counts=entry.data_file.nan_value_counts, - lower_bounds=entry.data_file.lower_bounds, - upper_bounds=entry.data_file.upper_bounds, - key_metadata=entry.data_file.key_metadata, - split_offsets=entry.data_file.split_offsets, - equality_ids=entry.data_file.equality_ids, - sort_order_id=entry.data_file.sort_order_id, + "record_count": entry.data_file.record_count, + "file_size_in_bytes": entry.data_file.file_size_in_bytes, + "column_sizes": entry.data_file.column_sizes, + "value_counts": entry.data_file.value_counts, + "null_value_counts": entry.data_file.null_value_counts, + "nan_value_counts": entry.data_file.nan_value_counts, + "lower_bounds": entry.data_file.lower_bounds, + "upper_bounds": entry.data_file.upper_bounds, + "key_metadata": entry.data_file.key_metadata, + "split_offsets": entry.data_file.split_offsets, + "equality_ids": entry.data_file.equality_ids, + "sort_order_id": entry.data_file.sort_order_id, + } + data_file = ( + DataFile.from_args(**data_file_args) + if hasattr(DataFile, "from_args") + else DataFile(**data_file_args) ) - return ManifestEntry( - ManifestEntryStatus(entry.status), - entry.snapshot_id, - entry.sequence_number, - entry.file_sequence_number, - data_file, + manifest_entry_args = { + "status": ManifestEntryStatus(entry.status), + "snapshot_id": entry.snapshot_id, + "sequence_number": entry.sequence_number, + "file_sequence_number": entry.file_sequence_number, + "data_file": data_file, + } + return ( + ManifestEntry.from_args(**manifest_entry_args) + if hasattr(ManifestEntry, "from_args") + else ManifestEntry(**manifest_entry_args) ) return [ diff --git a/bindings/python/uv.lock b/bindings/python/uv.lock index a298afb63c..964538511b 100644 --- a/bindings/python/uv.lock +++ b/bindings/python/uv.lock @@ -1,6 +1,10 @@ version = 1 revision = 3 requires-python = ">=3.10, <4" +resolution-markers = [ + "python_full_version >= '3.14'", + "python_full_version < '3.14'", +] [[package]] name = "annotated-types" @@ -13,11 +17,11 @@ wheels = [ [[package]] name = "cachetools" -version = "5.5.2" +version = "6.2.1" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/6c/81/3747dad6b14fa2cf53fcf10548cf5aea6913e96fab41a3c198676f8948a5/cachetools-5.5.2.tar.gz", hash = "sha256:1a661caa9175d26759571b2e19580f9d6393969e5dfca11fdb1f947a23e640d4", size = 28380, upload-time = "2025-02-20T21:01:19.524Z" } +sdist = { url = "https://files.pythonhosted.org/packages/cc/7e/b975b5814bd36faf009faebe22c1072a1fa1168db34d285ef0ba071ad78c/cachetools-6.2.1.tar.gz", hash = "sha256:3f391e4bd8f8bf0931169baf7456cc822705f4e2a31f840d218f445b9a854201", size = 31325, upload-time = "2025-10-12T14:55:30.139Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/72/76/20fa66124dbe6be5cafeb312ece67de6b61dd91a0247d1ea13db4ebb33c2/cachetools-5.5.2-py3-none-any.whl", hash = "sha256:d26a22bcc62eb95c3beabd9f1ee5e820d3d2704fe2967cbe350e20c8ffcd3f0a", size = 10080, upload-time = "2025-02-20T21:01:16.647Z" }, + { url = "https://files.pythonhosted.org/packages/96/c5/1e741d26306c42e2bf6ab740b2202872727e0f606033c9dd713f8b93f5a8/cachetools-6.2.1-py3-none-any.whl", hash = "sha256:09868944b6dde876dfd44e1d47e18484541eaf12f26f29b7af91b26cc892d701", size = 11280, upload-time = "2025-10-12T14:55:28.382Z" }, ] [[package]] @@ -141,19 +145,19 @@ wheels = [ [[package]] name = "datafusion" -version = "50.1.0" +version = "52.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pyarrow" }, { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/fa/cc/e8e8f7c472e93e7a560203ac40ac319b926029007c0dad873dbba97f9f2d/datafusion-50.1.0.tar.gz", hash = "sha256:d8b8f027c7ce2498cda1589d3ce6d8720798963e031660fbe4d2e26e172442ec", size = 188103, upload-time = "2025-10-20T12:39:23.802Z" } +sdist = { url = "https://files.pythonhosted.org/packages/58/04/4dabd255e04801b942221bf7eeea661f540d8c116e6b4a783fe2479410f0/datafusion-52.0.0.tar.gz", hash = "sha256:842cf9cdb523d04a053c5408da24645e3b2adce5d6c42ddc80a8c5edf9013ff3", size = 204988, upload-time = "2026-02-23T12:22:50.919Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/1f/6e/f9e2d5d935024a79fd549b5ce1d05549d26a027aab800727d492ac036504/datafusion-50.1.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:aeaa3c7bcf630bbea962b8fe75d300d98eaf7e2a5edf98e6a0130a1bec3543ea", size = 29280689, upload-time = "2025-10-20T12:39:06.913Z" }, - { url = "https://files.pythonhosted.org/packages/db/58/2dc473240f552d3620186b527c04397f82b36f02243afaf49f0813c84a17/datafusion-50.1.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:85727df82c818103092c3ee18d198365833d3e44c2921d2b378d4d682798e511", size = 26140751, upload-time = "2025-10-20T12:39:09.95Z" }, - { url = "https://files.pythonhosted.org/packages/00/ba/8d8aa1df96e0666752e5c9d406d440495df2014d315b2a95bbef9856b23e/datafusion-50.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49f5bd0edb2bf2d00625beeb46a115e1421db2e1b14b535f7c17cc0927f36b8a", size = 32165290, upload-time = "2025-10-20T12:39:13.713Z" }, - { url = "https://files.pythonhosted.org/packages/11/9a/afce9586145b3ed153d75364b21102a6a95260940352e06b7c6709e9d2db/datafusion-50.1.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:5c9c2f70922ddedf54d8abd4ba9585a5026c3409438f5aafc1ad0428a67a4d1f", size = 29982398, upload-time = "2025-10-20T12:39:16.823Z" }, - { url = "https://files.pythonhosted.org/packages/51/a3/41ef1c565770ef0c4060ee3fd50367dd06816f70a5be1ef41fbd7c3975e8/datafusion-50.1.0-cp39-abi3-win_amd64.whl", hash = "sha256:145c8f2e969c9cc51dc6af8a185ec39739ebeb5d680f9fe0020e005564ed40a8", size = 31258359, upload-time = "2025-10-20T12:39:21.731Z" }, + { url = "https://files.pythonhosted.org/packages/77/38/66b2f2fd77d3fb66ff48a8922130379dece3ba6d2e29fc86fbb4298a874b/datafusion-52.0.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:999881df12ab78b6c8f04dd2056b24389374e93775a649ed20c5e35db2f42f65", size = 31473623, upload-time = "2026-02-23T12:22:30.437Z" }, + { url = "https://files.pythonhosted.org/packages/d0/b5/ce6c6030fa8e4fc38d10d5c4aa9cc6fe1cda625e409a18eb08ea09a87c8d/datafusion-52.0.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:fd58e64158152f5c4a5836a3ce3bcca2a109d600c9ce7efdcf82e61c1ab0fbc8", size = 28108736, upload-time = "2026-02-23T12:22:33.5Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c1/d7ac9ddc9f54a8f178900f529a723d6121361111f0d0d2527bb47f86f6ce/datafusion-52.0.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:ab3591904f32ce290ff7161fb804e1c7bf323de16e3ddc8cf1f76310e994208e", size = 30699663, upload-time = "2026-02-23T12:22:37.193Z" }, + { url = "https://files.pythonhosted.org/packages/b0/2f/14cffc5305abe05d56f3e99e8054c96bd94411185de059a98fc1ca0e5ec0/datafusion-52.0.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:ac4b364937c277bbfcac032dbc49d08c078b13ba3f8bfda117da5fda4ea328bc", size = 33050161, upload-time = "2026-02-23T12:22:40.858Z" }, + { url = "https://files.pythonhosted.org/packages/24/ae/3fdea50fa88f304db96728a67deb6e07bb0d9a02f665ca09db4237a9a199/datafusion-52.0.0-cp310-abi3-win_amd64.whl", hash = "sha256:67e252ef20b918537c8fdb47e6c825c0bd639795e19715a85fedde331a83d2e1", size = 33717685, upload-time = "2026-02-23T12:22:44.463Z" }, ] [[package]] @@ -480,44 +484,59 @@ wheels = [ [[package]] name = "pyarrow" -version = "19.0.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/7f/09/a9046344212690f0632b9c709f9bf18506522feb333c894d0de81d62341a/pyarrow-19.0.1.tar.gz", hash = "sha256:3bf266b485df66a400f282ac0b6d1b500b9d2ae73314a153dbe97d6d5cc8a99e", size = 1129437, upload-time = "2025-02-18T18:55:57.027Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/36/01/b23b514d86b839956238d3f8ef206fd2728eee87ff1b8ce150a5678d9721/pyarrow-19.0.1-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:fc28912a2dc924dddc2087679cc8b7263accc71b9ff025a1362b004711661a69", size = 30688914, upload-time = "2025-02-18T18:51:37.575Z" }, - { url = "https://files.pythonhosted.org/packages/c6/68/218ff7cf4a0652a933e5f2ed11274f724dd43b9813cb18dd72c0a35226a2/pyarrow-19.0.1-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:fca15aabbe9b8355800d923cc2e82c8ef514af321e18b437c3d782aa884eaeec", size = 32102866, upload-time = "2025-02-18T18:51:44.358Z" }, - { url = "https://files.pythonhosted.org/packages/98/01/c295050d183014f4a2eb796d7d2bbfa04b6cccde7258bb68aacf6f18779b/pyarrow-19.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad76aef7f5f7e4a757fddcdcf010a8290958f09e3470ea458c80d26f4316ae89", size = 41147682, upload-time = "2025-02-18T18:51:49.481Z" }, - { url = "https://files.pythonhosted.org/packages/40/17/a6c3db0b5f3678f33bbb552d2acbc16def67f89a72955b67b0109af23eb0/pyarrow-19.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d03c9d6f2a3dffbd62671ca070f13fc527bb1867b4ec2b98c7eeed381d4f389a", size = 42179192, upload-time = "2025-02-18T18:51:56.265Z" }, - { url = "https://files.pythonhosted.org/packages/cf/75/c7c8e599300d8cebb6cb339014800e1c720c9db2a3fcb66aa64ec84bac72/pyarrow-19.0.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:65cf9feebab489b19cdfcfe4aa82f62147218558d8d3f0fc1e9dea0ab8e7905a", size = 40517272, upload-time = "2025-02-18T18:52:02.969Z" }, - { url = "https://files.pythonhosted.org/packages/ef/c9/68ab123ee1528699c4d5055f645ecd1dd68ff93e4699527249d02f55afeb/pyarrow-19.0.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:41f9706fbe505e0abc10e84bf3a906a1338905cbbcf1177b71486b03e6ea6608", size = 42069036, upload-time = "2025-02-18T18:52:10.173Z" }, - { url = "https://files.pythonhosted.org/packages/54/e3/d5cfd7654084e6c0d9c3ce949e5d9e0ccad569ae1e2d5a68a3ec03b2be89/pyarrow-19.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:c6cb2335a411b713fdf1e82a752162f72d4a7b5dbc588e32aa18383318b05866", size = 25277951, upload-time = "2025-02-18T18:52:15.459Z" }, - { url = "https://files.pythonhosted.org/packages/a0/55/f1a8d838ec07fe3ca53edbe76f782df7b9aafd4417080eebf0b42aab0c52/pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:cc55d71898ea30dc95900297d191377caba257612f384207fe9f8293b5850f90", size = 30713987, upload-time = "2025-02-18T18:52:20.463Z" }, - { url = "https://files.pythonhosted.org/packages/13/12/428861540bb54c98a140ae858a11f71d041ef9e501e6b7eb965ca7909505/pyarrow-19.0.1-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:7a544ec12de66769612b2d6988c36adc96fb9767ecc8ee0a4d270b10b1c51e00", size = 32135613, upload-time = "2025-02-18T18:52:25.29Z" }, - { url = "https://files.pythonhosted.org/packages/2f/8a/23d7cc5ae2066c6c736bce1db8ea7bc9ac3ef97ac7e1c1667706c764d2d9/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0148bb4fc158bfbc3d6dfe5001d93ebeed253793fff4435167f6ce1dc4bddeae", size = 41149147, upload-time = "2025-02-18T18:52:30.975Z" }, - { url = "https://files.pythonhosted.org/packages/a2/7a/845d151bb81a892dfb368bf11db584cf8b216963ccce40a5cf50a2492a18/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f24faab6ed18f216a37870d8c5623f9c044566d75ec586ef884e13a02a9d62c5", size = 42178045, upload-time = "2025-02-18T18:52:36.859Z" }, - { url = "https://files.pythonhosted.org/packages/a7/31/e7282d79a70816132cf6cae7e378adfccce9ae10352d21c2fecf9d9756dd/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:4982f8e2b7afd6dae8608d70ba5bd91699077323f812a0448d8b7abdff6cb5d3", size = 40532998, upload-time = "2025-02-18T18:52:42.578Z" }, - { url = "https://files.pythonhosted.org/packages/b8/82/20f3c290d6e705e2ee9c1fa1d5a0869365ee477e1788073d8b548da8b64c/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:49a3aecb62c1be1d822f8bf629226d4a96418228a42f5b40835c1f10d42e4db6", size = 42084055, upload-time = "2025-02-18T18:52:48.749Z" }, - { url = "https://files.pythonhosted.org/packages/ff/77/e62aebd343238863f2c9f080ad2ef6ace25c919c6ab383436b5b81cbeef7/pyarrow-19.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:008a4009efdb4ea3d2e18f05cd31f9d43c388aad29c636112c2966605ba33466", size = 25283133, upload-time = "2025-02-18T18:52:54.549Z" }, - { url = "https://files.pythonhosted.org/packages/78/b4/94e828704b050e723f67d67c3535cf7076c7432cd4cf046e4bb3b96a9c9d/pyarrow-19.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:80b2ad2b193e7d19e81008a96e313fbd53157945c7be9ac65f44f8937a55427b", size = 30670749, upload-time = "2025-02-18T18:53:00.062Z" }, - { url = "https://files.pythonhosted.org/packages/7e/3b/4692965e04bb1df55e2c314c4296f1eb12b4f3052d4cf43d29e076aedf66/pyarrow-19.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee8dec072569f43835932a3b10c55973593abc00936c202707a4ad06af7cb294", size = 32128007, upload-time = "2025-02-18T18:53:06.581Z" }, - { url = "https://files.pythonhosted.org/packages/22/f7/2239af706252c6582a5635c35caa17cb4d401cd74a87821ef702e3888957/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d5d1ec7ec5324b98887bdc006f4d2ce534e10e60f7ad995e7875ffa0ff9cb14", size = 41144566, upload-time = "2025-02-18T18:53:11.958Z" }, - { url = "https://files.pythonhosted.org/packages/fb/e3/c9661b2b2849cfefddd9fd65b64e093594b231b472de08ff658f76c732b2/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3ad4c0eb4e2a9aeb990af6c09e6fa0b195c8c0e7b272ecc8d4d2b6574809d34", size = 42202991, upload-time = "2025-02-18T18:53:17.678Z" }, - { url = "https://files.pythonhosted.org/packages/fe/4f/a2c0ed309167ef436674782dfee4a124570ba64299c551e38d3fdaf0a17b/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d383591f3dcbe545f6cc62daaef9c7cdfe0dff0fb9e1c8121101cabe9098cfa6", size = 40507986, upload-time = "2025-02-18T18:53:26.263Z" }, - { url = "https://files.pythonhosted.org/packages/27/2e/29bb28a7102a6f71026a9d70d1d61df926887e36ec797f2e6acfd2dd3867/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b4c4156a625f1e35d6c0b2132635a237708944eb41df5fbe7d50f20d20c17832", size = 42087026, upload-time = "2025-02-18T18:53:33.063Z" }, - { url = "https://files.pythonhosted.org/packages/16/33/2a67c0f783251106aeeee516f4806161e7b481f7d744d0d643d2f30230a5/pyarrow-19.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:5bd1618ae5e5476b7654c7b55a6364ae87686d4724538c24185bbb2952679960", size = 25250108, upload-time = "2025-02-18T18:53:38.462Z" }, - { url = "https://files.pythonhosted.org/packages/2b/8d/275c58d4b00781bd36579501a259eacc5c6dfb369be4ddeb672ceb551d2d/pyarrow-19.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e45274b20e524ae5c39d7fc1ca2aa923aab494776d2d4b316b49ec7572ca324c", size = 30653552, upload-time = "2025-02-18T18:53:44.357Z" }, - { url = "https://files.pythonhosted.org/packages/a0/9e/e6aca5cc4ef0c7aec5f8db93feb0bde08dbad8c56b9014216205d271101b/pyarrow-19.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:d9dedeaf19097a143ed6da37f04f4051aba353c95ef507764d344229b2b740ae", size = 32103413, upload-time = "2025-02-18T18:53:52.971Z" }, - { url = "https://files.pythonhosted.org/packages/6a/fa/a7033f66e5d4f1308c7eb0dfcd2ccd70f881724eb6fd1776657fdf65458f/pyarrow-19.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6ebfb5171bb5f4a52319344ebbbecc731af3f021e49318c74f33d520d31ae0c4", size = 41134869, upload-time = "2025-02-18T18:53:59.471Z" }, - { url = "https://files.pythonhosted.org/packages/2d/92/34d2569be8e7abdc9d145c98dc410db0071ac579b92ebc30da35f500d630/pyarrow-19.0.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2a21d39fbdb948857f67eacb5bbaaf36802de044ec36fbef7a1c8f0dd3a4ab2", size = 42192626, upload-time = "2025-02-18T18:54:06.062Z" }, - { url = "https://files.pythonhosted.org/packages/0a/1f/80c617b1084fc833804dc3309aa9d8daacd46f9ec8d736df733f15aebe2c/pyarrow-19.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:99bc1bec6d234359743b01e70d4310d0ab240c3d6b0da7e2a93663b0158616f6", size = 40496708, upload-time = "2025-02-18T18:54:12.347Z" }, - { url = "https://files.pythonhosted.org/packages/e6/90/83698fcecf939a611c8d9a78e38e7fed7792dcc4317e29e72cf8135526fb/pyarrow-19.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1b93ef2c93e77c442c979b0d596af45e4665d8b96da598db145b0fec014b9136", size = 42075728, upload-time = "2025-02-18T18:54:19.364Z" }, - { url = "https://files.pythonhosted.org/packages/40/49/2325f5c9e7a1c125c01ba0c509d400b152c972a47958768e4e35e04d13d8/pyarrow-19.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:d9d46e06846a41ba906ab25302cf0fd522f81aa2a85a71021826f34639ad31ef", size = 25242568, upload-time = "2025-02-18T18:54:25.846Z" }, - { url = "https://files.pythonhosted.org/packages/3f/72/135088d995a759d4d916ec4824cb19e066585b4909ebad4ab196177aa825/pyarrow-19.0.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:c0fe3dbbf054a00d1f162fda94ce236a899ca01123a798c561ba307ca38af5f0", size = 30702371, upload-time = "2025-02-18T18:54:30.665Z" }, - { url = "https://files.pythonhosted.org/packages/2e/01/00beeebd33d6bac701f20816a29d2018eba463616bbc07397fdf99ac4ce3/pyarrow-19.0.1-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:96606c3ba57944d128e8a8399da4812f56c7f61de8c647e3470b417f795d0ef9", size = 32116046, upload-time = "2025-02-18T18:54:35.995Z" }, - { url = "https://files.pythonhosted.org/packages/1f/c9/23b1ea718dfe967cbd986d16cf2a31fe59d015874258baae16d7ea0ccabc/pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f04d49a6b64cf24719c080b3c2029a3a5b16417fd5fd7c4041f94233af732f3", size = 41091183, upload-time = "2025-02-18T18:54:42.662Z" }, - { url = "https://files.pythonhosted.org/packages/3a/d4/b4a3aa781a2c715520aa8ab4fe2e7fa49d33a1d4e71c8fc6ab7b5de7a3f8/pyarrow-19.0.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a9137cf7e1640dce4c190551ee69d478f7121b5c6f323553b319cac936395f6", size = 42171896, upload-time = "2025-02-18T18:54:49.808Z" }, - { url = "https://files.pythonhosted.org/packages/23/1b/716d4cd5a3cbc387c6e6745d2704c4b46654ba2668260d25c402626c5ddb/pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:7c1bca1897c28013db5e4c83944a2ab53231f541b9e0c3f4791206d0c0de389a", size = 40464851, upload-time = "2025-02-18T18:54:57.073Z" }, - { url = "https://files.pythonhosted.org/packages/ed/bd/54907846383dcc7ee28772d7e646f6c34276a17da740002a5cefe90f04f7/pyarrow-19.0.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:58d9397b2e273ef76264b45531e9d552d8ec8a6688b7390b5be44c02a37aade8", size = 42085744, upload-time = "2025-02-18T18:55:08.562Z" }, +version = "22.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/30/53/04a7fdc63e6056116c9ddc8b43bc28c12cdd181b85cbeadb79278475f3ae/pyarrow-22.0.0.tar.gz", hash = "sha256:3d600dc583260d845c7d8a6db540339dd883081925da2bd1c5cb808f720b3cd9", size = 1151151, upload-time = "2025-10-24T12:30:00.762Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/9b/cb3f7e0a345353def531ca879053e9ef6b9f38ed91aebcf68b09ba54dec0/pyarrow-22.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:77718810bd3066158db1e95a63c160ad7ce08c6b0710bc656055033e39cdad88", size = 34223968, upload-time = "2025-10-24T10:03:31.21Z" }, + { url = "https://files.pythonhosted.org/packages/6c/41/3184b8192a120306270c5307f105b70320fdaa592c99843c5ef78aaefdcf/pyarrow-22.0.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:44d2d26cda26d18f7af7db71453b7b783788322d756e81730acb98f24eb90ace", size = 35942085, upload-time = "2025-10-24T10:03:38.146Z" }, + { url = "https://files.pythonhosted.org/packages/d9/3d/a1eab2f6f08001f9fb714b8ed5cfb045e2fe3e3e3c0c221f2c9ed1e6d67d/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b9d71701ce97c95480fecb0039ec5bb889e75f110da72005743451339262f4ce", size = 44964613, upload-time = "2025-10-24T10:03:46.516Z" }, + { url = "https://files.pythonhosted.org/packages/46/46/a1d9c24baf21cfd9ce994ac820a24608decf2710521b29223d4334985127/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:710624ab925dc2b05a6229d47f6f0dac1c1155e6ed559be7109f684eba048a48", size = 47627059, upload-time = "2025-10-24T10:03:55.353Z" }, + { url = "https://files.pythonhosted.org/packages/3a/4c/f711acb13075c1391fd54bc17e078587672c575f8de2a6e62509af026dcf/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f963ba8c3b0199f9d6b794c90ec77545e05eadc83973897a4523c9e8d84e9340", size = 47947043, upload-time = "2025-10-24T10:04:05.408Z" }, + { url = "https://files.pythonhosted.org/packages/4e/70/1f3180dd7c2eab35c2aca2b29ace6c519f827dcd4cfeb8e0dca41612cf7a/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:bd0d42297ace400d8febe55f13fdf46e86754842b860c978dfec16f081e5c653", size = 50206505, upload-time = "2025-10-24T10:04:15.786Z" }, + { url = "https://files.pythonhosted.org/packages/80/07/fea6578112c8c60ffde55883a571e4c4c6bc7049f119d6b09333b5cc6f73/pyarrow-22.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:00626d9dc0f5ef3a75fe63fd68b9c7c8302d2b5bbc7f74ecaedba83447a24f84", size = 28101641, upload-time = "2025-10-24T10:04:22.57Z" }, + { url = "https://files.pythonhosted.org/packages/2e/b7/18f611a8cdc43417f9394a3ccd3eace2f32183c08b9eddc3d17681819f37/pyarrow-22.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:3e294c5eadfb93d78b0763e859a0c16d4051fc1c5231ae8956d61cb0b5666f5a", size = 34272022, upload-time = "2025-10-24T10:04:28.973Z" }, + { url = "https://files.pythonhosted.org/packages/26/5c/f259e2526c67eb4b9e511741b19870a02363a47a35edbebc55c3178db22d/pyarrow-22.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:69763ab2445f632d90b504a815a2a033f74332997052b721002298ed6de40f2e", size = 35995834, upload-time = "2025-10-24T10:04:35.467Z" }, + { url = "https://files.pythonhosted.org/packages/50/8d/281f0f9b9376d4b7f146913b26fac0aa2829cd1ee7e997f53a27411bbb92/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:b41f37cabfe2463232684de44bad753d6be08a7a072f6a83447eeaf0e4d2a215", size = 45030348, upload-time = "2025-10-24T10:04:43.366Z" }, + { url = "https://files.pythonhosted.org/packages/f5/e5/53c0a1c428f0976bf22f513d79c73000926cb00b9c138d8e02daf2102e18/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:35ad0f0378c9359b3f297299c3309778bb03b8612f987399a0333a560b43862d", size = 47699480, upload-time = "2025-10-24T10:04:51.486Z" }, + { url = "https://files.pythonhosted.org/packages/95/e1/9dbe4c465c3365959d183e6345d0a8d1dc5b02ca3f8db4760b3bc834cf25/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8382ad21458075c2e66a82a29d650f963ce51c7708c7c0ff313a8c206c4fd5e8", size = 48011148, upload-time = "2025-10-24T10:04:59.585Z" }, + { url = "https://files.pythonhosted.org/packages/c5/b4/7caf5d21930061444c3cf4fa7535c82faf5263e22ce43af7c2759ceb5b8b/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1a812a5b727bc09c3d7ea072c4eebf657c2f7066155506ba31ebf4792f88f016", size = 50276964, upload-time = "2025-10-24T10:05:08.175Z" }, + { url = "https://files.pythonhosted.org/packages/ae/f3/cec89bd99fa3abf826f14d4e53d3d11340ce6f6af4d14bdcd54cd83b6576/pyarrow-22.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:ec5d40dd494882704fb876c16fa7261a69791e784ae34e6b5992e977bd2e238c", size = 28106517, upload-time = "2025-10-24T10:05:14.314Z" }, + { url = "https://files.pythonhosted.org/packages/af/63/ba23862d69652f85b615ca14ad14f3bcfc5bf1b99ef3f0cd04ff93fdad5a/pyarrow-22.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:bea79263d55c24a32b0d79c00a1c58bb2ee5f0757ed95656b01c0fb310c5af3d", size = 34211578, upload-time = "2025-10-24T10:05:21.583Z" }, + { url = "https://files.pythonhosted.org/packages/b1/d0/f9ad86fe809efd2bcc8be32032fa72e8b0d112b01ae56a053006376c5930/pyarrow-22.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:12fe549c9b10ac98c91cf791d2945e878875d95508e1a5d14091a7aaa66d9cf8", size = 35989906, upload-time = "2025-10-24T10:05:29.485Z" }, + { url = "https://files.pythonhosted.org/packages/b4/a8/f910afcb14630e64d673f15904ec27dd31f1e009b77033c365c84e8c1e1d/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:334f900ff08ce0423407af97e6c26ad5d4e3b0763645559ece6fbf3747d6a8f5", size = 45021677, upload-time = "2025-10-24T10:05:38.274Z" }, + { url = "https://files.pythonhosted.org/packages/13/95/aec81f781c75cd10554dc17a25849c720d54feafb6f7847690478dcf5ef8/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:c6c791b09c57ed76a18b03f2631753a4960eefbbca80f846da8baefc6491fcfe", size = 47726315, upload-time = "2025-10-24T10:05:47.314Z" }, + { url = "https://files.pythonhosted.org/packages/bb/d4/74ac9f7a54cfde12ee42734ea25d5a3c9a45db78f9def949307a92720d37/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c3200cb41cdbc65156e5f8c908d739b0dfed57e890329413da2748d1a2cd1a4e", size = 47990906, upload-time = "2025-10-24T10:05:58.254Z" }, + { url = "https://files.pythonhosted.org/packages/2e/71/fedf2499bf7a95062eafc989ace56572f3343432570e1c54e6599d5b88da/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ac93252226cf288753d8b46280f4edf3433bf9508b6977f8dd8526b521a1bbb9", size = 50306783, upload-time = "2025-10-24T10:06:08.08Z" }, + { url = "https://files.pythonhosted.org/packages/68/ed/b202abd5a5b78f519722f3d29063dda03c114711093c1995a33b8e2e0f4b/pyarrow-22.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:44729980b6c50a5f2bfcc2668d36c569ce17f8b17bccaf470c4313dcbbf13c9d", size = 27972883, upload-time = "2025-10-24T10:06:14.204Z" }, + { url = "https://files.pythonhosted.org/packages/a6/d6/d0fac16a2963002fc22c8fa75180a838737203d558f0ed3b564c4a54eef5/pyarrow-22.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e6e95176209257803a8b3d0394f21604e796dadb643d2f7ca21b66c9c0b30c9a", size = 34204629, upload-time = "2025-10-24T10:06:20.274Z" }, + { url = "https://files.pythonhosted.org/packages/c6/9c/1d6357347fbae062ad3f17082f9ebc29cc733321e892c0d2085f42a2212b/pyarrow-22.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:001ea83a58024818826a9e3f89bf9310a114f7e26dfe404a4c32686f97bd7901", size = 35985783, upload-time = "2025-10-24T10:06:27.301Z" }, + { url = "https://files.pythonhosted.org/packages/ff/c0/782344c2ce58afbea010150df07e3a2f5fdad299cd631697ae7bd3bac6e3/pyarrow-22.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:ce20fe000754f477c8a9125543f1936ea5b8867c5406757c224d745ed033e691", size = 45020999, upload-time = "2025-10-24T10:06:35.387Z" }, + { url = "https://files.pythonhosted.org/packages/1b/8b/5362443737a5307a7b67c1017c42cd104213189b4970bf607e05faf9c525/pyarrow-22.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:e0a15757fccb38c410947df156f9749ae4a3c89b2393741a50521f39a8cf202a", size = 47724601, upload-time = "2025-10-24T10:06:43.551Z" }, + { url = "https://files.pythonhosted.org/packages/69/4d/76e567a4fc2e190ee6072967cb4672b7d9249ac59ae65af2d7e3047afa3b/pyarrow-22.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:cedb9dd9358e4ea1d9bce3665ce0797f6adf97ff142c8e25b46ba9cdd508e9b6", size = 48001050, upload-time = "2025-10-24T10:06:52.284Z" }, + { url = "https://files.pythonhosted.org/packages/01/5e/5653f0535d2a1aef8223cee9d92944cb6bccfee5cf1cd3f462d7cb022790/pyarrow-22.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:252be4a05f9d9185bb8c18e83764ebcfea7185076c07a7a662253af3a8c07941", size = 50307877, upload-time = "2025-10-24T10:07:02.405Z" }, + { url = "https://files.pythonhosted.org/packages/2d/f8/1d0bd75bf9328a3b826e24a16e5517cd7f9fbf8d34a3184a4566ef5a7f29/pyarrow-22.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:a4893d31e5ef780b6edcaf63122df0f8d321088bb0dee4c8c06eccb1ca28d145", size = 27977099, upload-time = "2025-10-24T10:08:07.259Z" }, + { url = "https://files.pythonhosted.org/packages/90/81/db56870c997805bf2b0f6eeeb2d68458bf4654652dccdcf1bf7a42d80903/pyarrow-22.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:f7fe3dbe871294ba70d789be16b6e7e52b418311e166e0e3cba9522f0f437fb1", size = 34336685, upload-time = "2025-10-24T10:07:11.47Z" }, + { url = "https://files.pythonhosted.org/packages/1c/98/0727947f199aba8a120f47dfc229eeb05df15bcd7a6f1b669e9f882afc58/pyarrow-22.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:ba95112d15fd4f1105fb2402c4eab9068f0554435e9b7085924bcfaac2cc306f", size = 36032158, upload-time = "2025-10-24T10:07:18.626Z" }, + { url = "https://files.pythonhosted.org/packages/96/b4/9babdef9c01720a0785945c7cf550e4acd0ebcd7bdd2e6f0aa7981fa85e2/pyarrow-22.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:c064e28361c05d72eed8e744c9605cbd6d2bb7481a511c74071fd9b24bc65d7d", size = 44892060, upload-time = "2025-10-24T10:07:26.002Z" }, + { url = "https://files.pythonhosted.org/packages/f8/ca/2f8804edd6279f78a37062d813de3f16f29183874447ef6d1aadbb4efa0f/pyarrow-22.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:6f9762274496c244d951c819348afbcf212714902742225f649cf02823a6a10f", size = 47504395, upload-time = "2025-10-24T10:07:34.09Z" }, + { url = "https://files.pythonhosted.org/packages/b9/f0/77aa5198fd3943682b2e4faaf179a674f0edea0d55d326d83cb2277d9363/pyarrow-22.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a9d9ffdc2ab696f6b15b4d1f7cec6658e1d788124418cb30030afbae31c64746", size = 48066216, upload-time = "2025-10-24T10:07:43.528Z" }, + { url = "https://files.pythonhosted.org/packages/79/87/a1937b6e78b2aff18b706d738c9e46ade5bfcf11b294e39c87706a0089ac/pyarrow-22.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ec1a15968a9d80da01e1d30349b2b0d7cc91e96588ee324ce1b5228175043e95", size = 50288552, upload-time = "2025-10-24T10:07:53.519Z" }, + { url = "https://files.pythonhosted.org/packages/60/ae/b5a5811e11f25788ccfdaa8f26b6791c9807119dffcf80514505527c384c/pyarrow-22.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:bba208d9c7decf9961998edf5c65e3ea4355d5818dd6cd0f6809bec1afb951cc", size = 28262504, upload-time = "2025-10-24T10:08:00.932Z" }, + { url = "https://files.pythonhosted.org/packages/bd/b0/0fa4d28a8edb42b0a7144edd20befd04173ac79819547216f8a9f36f9e50/pyarrow-22.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:9bddc2cade6561f6820d4cd73f99a0243532ad506bc510a75a5a65a522b2d74d", size = 34224062, upload-time = "2025-10-24T10:08:14.101Z" }, + { url = "https://files.pythonhosted.org/packages/0f/a8/7a719076b3c1be0acef56a07220c586f25cd24de0e3f3102b438d18ae5df/pyarrow-22.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e70ff90c64419709d38c8932ea9fe1cc98415c4f87ea8da81719e43f02534bc9", size = 35990057, upload-time = "2025-10-24T10:08:21.842Z" }, + { url = "https://files.pythonhosted.org/packages/89/3c/359ed54c93b47fb6fe30ed16cdf50e3f0e8b9ccfb11b86218c3619ae50a8/pyarrow-22.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:92843c305330aa94a36e706c16209cd4df274693e777ca47112617db7d0ef3d7", size = 45068002, upload-time = "2025-10-24T10:08:29.034Z" }, + { url = "https://files.pythonhosted.org/packages/55/fc/4945896cc8638536ee787a3bd6ce7cec8ec9acf452d78ec39ab328efa0a1/pyarrow-22.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:6dda1ddac033d27421c20d7a7943eec60be44e0db4e079f33cc5af3b8280ccde", size = 47737765, upload-time = "2025-10-24T10:08:38.559Z" }, + { url = "https://files.pythonhosted.org/packages/cd/5e/7cb7edeb2abfaa1f79b5d5eb89432356155c8426f75d3753cbcb9592c0fd/pyarrow-22.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:84378110dd9a6c06323b41b56e129c504d157d1a983ce8f5443761eb5256bafc", size = 48048139, upload-time = "2025-10-24T10:08:46.784Z" }, + { url = "https://files.pythonhosted.org/packages/88/c6/546baa7c48185f5e9d6e59277c4b19f30f48c94d9dd938c2a80d4d6b067c/pyarrow-22.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:854794239111d2b88b40b6ef92aa478024d1e5074f364033e73e21e3f76b25e0", size = 50314244, upload-time = "2025-10-24T10:08:55.771Z" }, + { url = "https://files.pythonhosted.org/packages/3c/79/755ff2d145aafec8d347bf18f95e4e81c00127f06d080135dfc86aea417c/pyarrow-22.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:b883fe6fd85adad7932b3271c38ac289c65b7337c2c132e9569f9d3940620730", size = 28757501, upload-time = "2025-10-24T10:09:59.891Z" }, + { url = "https://files.pythonhosted.org/packages/0e/d2/237d75ac28ced3147912954e3c1a174df43a95f4f88e467809118a8165e0/pyarrow-22.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7a820d8ae11facf32585507c11f04e3f38343c1e784c9b5a8b1da5c930547fe2", size = 34355506, upload-time = "2025-10-24T10:09:02.953Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/733dfffe6d3069740f98e57ff81007809067d68626c5faef293434d11bd6/pyarrow-22.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:c6ec3675d98915bf1ec8b3c7986422682f7232ea76cad276f4c8abd5b7319b70", size = 36047312, upload-time = "2025-10-24T10:09:10.334Z" }, + { url = "https://files.pythonhosted.org/packages/7c/2b/29d6e3782dc1f299727462c1543af357a0f2c1d3c160ce199950d9ca51eb/pyarrow-22.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:3e739edd001b04f654b166204fc7a9de896cf6007eaff33409ee9e50ceaff754", size = 45081609, upload-time = "2025-10-24T10:09:18.61Z" }, + { url = "https://files.pythonhosted.org/packages/8d/42/aa9355ecc05997915af1b7b947a7f66c02dcaa927f3203b87871c114ba10/pyarrow-22.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:7388ac685cab5b279a41dfe0a6ccd99e4dbf322edfb63e02fc0443bf24134e91", size = 47703663, upload-time = "2025-10-24T10:09:27.369Z" }, + { url = "https://files.pythonhosted.org/packages/ee/62/45abedde480168e83a1de005b7b7043fd553321c1e8c5a9a114425f64842/pyarrow-22.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f633074f36dbc33d5c05b5dc75371e5660f1dbf9c8b1d95669def05e5425989c", size = 48066543, upload-time = "2025-10-24T10:09:34.908Z" }, + { url = "https://files.pythonhosted.org/packages/84/e9/7878940a5b072e4f3bf998770acafeae13b267f9893af5f6d4ab3904b67e/pyarrow-22.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:4c19236ae2402a8663a2c8f21f1870a03cc57f0bef7e4b6eb3238cc82944de80", size = 50288838, upload-time = "2025-10-24T10:09:44.394Z" }, + { url = "https://files.pythonhosted.org/packages/7b/03/f335d6c52b4a4761bcc83499789a1e2e16d9d201a58c327a9b5cc9a41bd9/pyarrow-22.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0c34fe18094686194f204a3b1787a27456897d8a2d62caf84b61e8dfbc0252ae", size = 29185594, upload-time = "2025-10-24T10:09:53.111Z" }, ] [[package]] @@ -633,7 +652,7 @@ wheels = [ [[package]] name = "pyiceberg" -version = "0.9.1" +version = "0.11.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cachetools" }, @@ -642,38 +661,49 @@ dependencies = [ { name = "mmh3" }, { name = "pydantic" }, { name = "pyparsing" }, + { name = "pyroaring" }, { name = "requests" }, { name = "rich" }, - { name = "sortedcontainers" }, { name = "strictyaml" }, { name = "tenacity" }, + { name = "zstandard" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/bd/6a/6c1ac381ff0b8e03a9abc2f05722f6002d7452a2c05118697b3f3910e171/pyiceberg-0.9.1.tar.gz", hash = "sha256:3634134ce33859a441768b39df179b2c6f3de2bbbf506622884f553b013ee799", size = 617629, upload-time = "2025-04-30T14:59:34.306Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ea/5d/bb10c86b85895d4ba471b8a0e187031d4aaa82592a639242b83dd9354861/pyiceberg-0.9.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0a183d9217eb82159c01b23c683057f96c8b2375f592b921721d1c157895e2df", size = 527097, upload-time = "2025-04-30T14:58:52.39Z" }, - { url = "https://files.pythonhosted.org/packages/ec/b9/1d6f0d334bc51cd64a58b7320d521e54af3810a6bd748fe2e89db1ad8d5f/pyiceberg-0.9.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:57030bb15c397b0379242907c5611f5b4338fb799e972353fd0edafde6cfd2ef", size = 523267, upload-time = "2025-04-30T14:58:53.978Z" }, - { url = "https://files.pythonhosted.org/packages/02/f5/bd43a9c1d2cd3aeb987cbf2b7f25e2b10306fa81522ea00df250fb23cc84/pyiceberg-0.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3ba4cd9a8f6a04cfbc68e0c83f2db3ffd14244da8601a142cc05965d4b343645", size = 838616, upload-time = "2025-04-30T14:58:55.252Z" }, - { url = "https://files.pythonhosted.org/packages/d0/01/c68f9e03413dc983ddadc2c471038af2ff792449fc451731f58a958a7696/pyiceberg-0.9.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:d5a48c6a2016d0dcde8c9079cc5e6b6d2e2ac663eddfe4697e7ea03a0edc40b7", size = 838290, upload-time = "2025-04-30T14:58:56.412Z" }, - { url = "https://files.pythonhosted.org/packages/ab/80/b7cba54a33b8b7be3655ff656d6bb8594fec0316eec5cafa231ec7f6ff74/pyiceberg-0.9.1-cp310-cp310-win_amd64.whl", hash = "sha256:8bebfa5a804a95a9f3d98d88cbeb37430b09add04592238bba2a2b2e0466d60d", size = 523612, upload-time = "2025-04-30T14:58:59.507Z" }, - { url = "https://files.pythonhosted.org/packages/f6/75/c8b4ebba7d345b5e736ebf4976121b97dd7091dcad401a17ca57152704c5/pyiceberg-0.9.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0e75c502dd56ac3d77036ce8a3b2566348da5ff4367c7c671981616ef6dcc883", size = 566274, upload-time = "2025-04-30T14:59:00.626Z" }, - { url = "https://files.pythonhosted.org/packages/e0/a0/9494c7930e5e4dc951d95abba584d8ffdb7403368398796ede21ff25c26f/pyiceberg-0.9.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0a8189c9b3ba81dd12493d6bb874a656a4d4909904552b97a629d1d43b3a0e90", size = 560157, upload-time = "2025-04-30T14:59:02.082Z" }, - { url = "https://files.pythonhosted.org/packages/4a/d4/351776b1ae83de187d7cf37b100f4f124c7a71e35337182d3aef308156d1/pyiceberg-0.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7c03065d5c5b704444ab8fb18cdd232ec43994db95b9e53444008ebc2cf9dc2c", size = 1052290, upload-time = "2025-04-30T14:59:03.232Z" }, - { url = "https://files.pythonhosted.org/packages/40/17/d8fea681afb52f20bf6a640f9044dcf621a47165f67cc5320bf3c6e82e4e/pyiceberg-0.9.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:93f2586a5da737de6e4643bf096a01772f068d1eedb7ffde6b36c60b6b9e6bd3", size = 1047503, upload-time = "2025-04-30T14:59:04.38Z" }, - { url = "https://files.pythonhosted.org/packages/d0/e0/d173fc2aa8dc252d7aac71703ba2c0491e4988b3a160cf5abb531cfb9086/pyiceberg-0.9.1-cp311-cp311-win_amd64.whl", hash = "sha256:94e45c10051110ba7a43b85a1f0a680b4a31d1d6cee593c8e62e14d22d18c47d", size = 559491, upload-time = "2025-04-30T14:59:05.615Z" }, - { url = "https://files.pythonhosted.org/packages/52/26/77983c2884b4a5f13f8a35e5c5e762ae699f6c511efd16730ab883000c1b/pyiceberg-0.9.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b8a958e3bbe919026533cee1f0fb6b7040928fce8d42c2ecea228de7c17578fa", size = 605755, upload-time = "2025-04-30T14:59:07.087Z" }, - { url = "https://files.pythonhosted.org/packages/6d/67/e6ea7fcc43aebc85aea5a67a69d01c9015283478061c3121b6b8aa158ce4/pyiceberg-0.9.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b7e956b35c6822600c45fd8f3ea8cfea328cc406fefa534afeb6fdb325d05406", size = 597325, upload-time = "2025-04-30T14:59:08.644Z" }, - { url = "https://files.pythonhosted.org/packages/7f/cf/178a9f63fac1bfdd13bc85169e7ab903955d082e2cd80507b1921a6f64dc/pyiceberg-0.9.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1e4e585164d7d86f5c9a609a1bc2abeae2f0ea0680a11a2064d3a945866b5311", size = 1277399, upload-time = "2025-04-30T14:59:10.193Z" }, - { url = "https://files.pythonhosted.org/packages/d1/6b/78d1739eb1d5b18529ee438aed75dac3e0b246f5e4d800931f9d1e37cda2/pyiceberg-0.9.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5fee08dac30e8524526f7d18468f9670f8606905b850b261314c597c6633f3b4", size = 1269083, upload-time = "2025-04-30T14:59:11.964Z" }, - { url = "https://files.pythonhosted.org/packages/67/69/c0087d19c8d8e8530acee3ba485d54aedeebf2963784a16692ca4b439566/pyiceberg-0.9.1-cp312-cp312-win_amd64.whl", hash = "sha256:124793c54a0c2fb5ac4ab19c38da116c068e277c85cbaa7e4064e635a70b595e", size = 595512, upload-time = "2025-04-30T14:59:14.464Z" }, - { url = "https://files.pythonhosted.org/packages/aa/62/0153ed3a39d6f4b3235d430123703d4684eec7ba780404bbc118ace7406a/pyiceberg-0.9.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:77aec1c77d675603e0c5358e74adcae8d13b323753d702011be3f309d26af355", size = 668261, upload-time = "2025-04-30T14:59:21.751Z" }, - { url = "https://files.pythonhosted.org/packages/24/bd/c4cec142686dd8124032c69b6b02ba3703abc114ce787d0f02088b1f43d8/pyiceberg-0.9.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:cf567438bf6267bbb67fdfdfc72ac500d523725fca9a6a38f93e8acd4146190e", size = 657439, upload-time = "2025-04-30T14:59:23.304Z" }, - { url = "https://files.pythonhosted.org/packages/ae/74/bbfc70bb1857f9d55d06fee1330a0236876b8ae4aa6fc5d815e2c4fef4f7/pyiceberg-0.9.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5992db7c00d789a33ff117700d453126803e769507a5edeb79bb6510ff72fc00", size = 1352983, upload-time = "2025-04-30T14:59:25.023Z" }, - { url = "https://files.pythonhosted.org/packages/90/20/e33e1716d1368b2471b80d9f1e338110f1e781b34ebffc5e320523102ffc/pyiceberg-0.9.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:c9e460fca26162a3822c0e8d50b49c80928a0e35cb41698748d7a26f8c016215", size = 657563, upload-time = "2025-04-30T14:59:27.004Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/bd/22/3d02ad39710bf51834d108e6d548cee9c1916850460ccba80db47a982567/pyiceberg-0.11.0.tar.gz", hash = "sha256:095bbafc87d204cf8d3ffc1c434e07cf9a67a709192ac0b11dcb0f8251f7ad4e", size = 1074873, upload-time = "2026-02-10T02:28:20.762Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/f4/928cba4c8dff6e096fb44dd870c9d9fc4e1cebe6d9e7ac3c90ef1709b6fe/pyiceberg-0.11.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2f73e476108b4fe25db4a4a9e0e3fa5ac0aef7a180c276f7dff45e7fa4e79603", size = 532117, upload-time = "2026-02-10T02:27:38.186Z" }, + { url = "https://files.pythonhosted.org/packages/2c/80/71cbc31747cb78c28aa12176c1e861ca82e63e0161b300416e051dfa4c30/pyiceberg-0.11.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:918277e5c1c58dfe9efdf4b55bac3f08e3218181c30d51f44d311b535071c5a8", size = 532993, upload-time = "2026-02-10T02:27:40.427Z" }, + { url = "https://files.pythonhosted.org/packages/d2/16/42ea2c9cbfe56c4af4c00c8c2128211f5b11a3395d40964295265a1e55ea/pyiceberg-0.11.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d33c44139e2e07a0e9d3e3c13cbac9060e6b5b7692880b0d32b5c20082c2b4d3", size = 708334, upload-time = "2026-02-10T02:27:42.235Z" }, + { url = "https://files.pythonhosted.org/packages/03/f3/787d5838abc50e02c12fb36e49fca736716a63ca55503ccd5ebb9453e01b/pyiceberg-0.11.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7996d498e33b952c63cc8ef3ec60ea6647c79289738a0ba1b681131778de5252", size = 705296, upload-time = "2026-02-10T02:27:43.885Z" }, + { url = "https://files.pythonhosted.org/packages/ba/d2/a1a9839015b0a974942054031180f0bac3f9296554ff8ee629e692efff35/pyiceberg-0.11.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0c50cedaf98f5e671e14f46e5904a3243003bb08980cd6207cc1f3d9e34bb854", size = 703911, upload-time = "2026-02-10T02:27:45.177Z" }, + { url = "https://files.pythonhosted.org/packages/a8/d7/4321a49d497eaacb7cfb15cc6e1a818cb2e0c02c1884557c2a488b9c3c6e/pyiceberg-0.11.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:a436858753195bc0087083f4e15d5a051faa1ac9ec9cd72476da8c4a5c03ecf9", size = 703696, upload-time = "2026-02-10T02:27:46.391Z" }, + { url = "https://files.pythonhosted.org/packages/8c/76/69ebc14943cdd1c8d6a67d560a0a5a157b6fd12487646cce63f5f751148f/pyiceberg-0.11.0-cp310-cp310-win_amd64.whl", hash = "sha256:483914a79b5f0a8732ccd9928f204571b98250e2e3876135fcb2123fab0aa589", size = 530429, upload-time = "2026-02-10T02:27:48.019Z" }, + { url = "https://files.pythonhosted.org/packages/bf/7c/a002ceef5611a85ee03854c7c2976b3a82d2b0c1344ed11e75721565a852/pyiceberg-0.11.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:9f6f31b5e7656ff53a4938a4f961d2f79d88ea14d7eaf4009f9baa06d35caf12", size = 532116, upload-time = "2026-02-10T02:27:49.483Z" }, + { url = "https://files.pythonhosted.org/packages/95/32/61846df3bb7dc410ee3adbaf3c5ab73dc9081d11fb47bac563ff4ff52637/pyiceberg-0.11.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0c651bf7ab26b73f75b7cc63e387b416dee9822643b08ca9fcba4ae9547a577d", size = 533078, upload-time = "2026-02-10T02:27:51.681Z" }, + { url = "https://files.pythonhosted.org/packages/d7/7a/7e62e8b92516a4ae5d5992e959f50acb11af2427e3dd71a602df6668447d/pyiceberg-0.11.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1fbe5fc7eb99d8efad74449ee7ba7edd30910f81dcc2d4a9cd4efaf6292caccf", size = 722601, upload-time = "2026-02-10T02:27:53.355Z" }, + { url = "https://files.pythonhosted.org/packages/dc/3b/a43a21e1389be215d602327875f243bd488e9925a8c91fd0e8bafb5fda47/pyiceberg-0.11.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fe20129bfe408173c1a4483891bedcbf3378dcfac7d0dfe369e209fcb13674cd", size = 720930, upload-time = "2026-02-10T02:27:54.96Z" }, + { url = "https://files.pythonhosted.org/packages/41/1d/10052ccaa9d8caeebba290b2eb80f92116c0a723d887b5496977345f7955/pyiceberg-0.11.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:839688576a1d275b89fa2baab34f771c4d6ecf59f2a1f5ce5fa8634948e03366", size = 717693, upload-time = "2026-02-10T02:27:56.89Z" }, + { url = "https://files.pythonhosted.org/packages/d8/2c/725663b206c2cc13c7bb18c70e89fdf8badd72483879e408a0b0c4515743/pyiceberg-0.11.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6f6a9df9aa862217e2039cbbeac18c6ab9cba8acfe78a873f31041201f77333e", size = 718463, upload-time = "2026-02-10T02:27:58.729Z" }, + { url = "https://files.pythonhosted.org/packages/15/8f/8bc25e2b43bb5d303ab3b97f570d7cabb476ef988be980b946df41a3b5af/pyiceberg-0.11.0-cp311-cp311-win_amd64.whl", hash = "sha256:79092b327cdb969c218e91bedcbadb63876f1a5b86e899a77d2051cd86e8631b", size = 530791, upload-time = "2026-02-10T02:28:00.032Z" }, + { url = "https://files.pythonhosted.org/packages/c6/37/b5a818444f5563ee2dacac93cc690e63396ab60308be353502dc7008168b/pyiceberg-0.11.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6fc89c9581d42ff2383cc9ba3f443ab9f175d8e85216ecbd819e955e9069bc46", size = 532694, upload-time = "2026-02-10T02:28:01.298Z" }, + { url = "https://files.pythonhosted.org/packages/7d/f9/ef76d6cf62a7ba9d61a5e20216000d4b366d8eac3be5c89c2ce5c8eb38f9/pyiceberg-0.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e2dfdf5438cc5ad8eb8b2e3f7a41ab6f286fe8b6fd6f5c1407381f627097e2e0", size = 532901, upload-time = "2026-02-10T02:28:02.517Z" }, + { url = "https://files.pythonhosted.org/packages/15/2a/bcec7d0ca75259cdb83ddceee1c59cdad619d2dfe36cee802c7e7207d96a/pyiceberg-0.11.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4543e93c78bb4fd78da7093c8232d62487a68661ba6bff0bafc0b346b34ca38c", size = 729261, upload-time = "2026-02-10T02:28:03.694Z" }, + { url = "https://files.pythonhosted.org/packages/99/ff/db75a2062a0b4b64ad0a6c677cab5b6e3ac19e0820584c597e1822f2cf7c/pyiceberg-0.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8dda2ad8d57e3af743ab67d976a23ca1cd54a4849110b5c2375f5d9466a4ae80", size = 729979, upload-time = "2026-02-10T02:28:04.878Z" }, + { url = "https://files.pythonhosted.org/packages/d8/eb/453e8c4a7e6eb698bf1402337e3cd3516f20c4bbe0f06961d3e6c5031cca/pyiceberg-0.11.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b5999fb41ea0b4b153a5c80d56512ef0596f95fdd62512d1806b8db89fd4a5f9", size = 723778, upload-time = "2026-02-10T02:28:06.573Z" }, + { url = "https://files.pythonhosted.org/packages/c8/7b/4f38016722ecc04f97000f7b7f80ba1d74e66dcbf630a4c2b620b5393ce0/pyiceberg-0.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:63c76f882ad30bda5b5fc685c6ab053e5b5585eadab04d1afc515eec4e272b14", size = 726955, upload-time = "2026-02-10T02:28:08.684Z" }, + { url = "https://files.pythonhosted.org/packages/56/14/dc689c0637d7f6716cae614afcce5782903cc87a781dfd47e6d6e72ce104/pyiceberg-0.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:4bb26a9308e8bb97c1d3518209d221f2a790a37b9806b8b91fee4c47be4919a6", size = 531019, upload-time = "2026-02-10T02:28:10.333Z" }, + { url = "https://files.pythonhosted.org/packages/c6/72/ef1e816d79d703eec1182398947a6b72f502eefeee01c4484bd5e1493b07/pyiceberg-0.11.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c707f4463dd9c1ca664d41d5ddd38babadf1bf5fa1946cb591c033a6a2827eb4", size = 532359, upload-time = "2026-02-10T02:28:11.473Z" }, + { url = "https://files.pythonhosted.org/packages/1f/41/ec85279b1b8ed57d0d27d4675203d314b8f5d69383e1df68f615f45e9dda/pyiceberg-0.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f1c944969fda799a2d26dc6f57448ace44ee07e334306ba6f5110df1aadeeef1", size = 532496, upload-time = "2026-02-10T02:28:13.19Z" }, + { url = "https://files.pythonhosted.org/packages/b9/b4/02861c450057c9a6e2f2e1eb0ef735c2e28473cff60b2747c50d0427ec1c/pyiceberg-0.11.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1be075b9ecc175b8dd76822b081b379ce33cda33d6403eaf607268f6061f3275", size = 721917, upload-time = "2026-02-10T02:28:14.484Z" }, + { url = "https://files.pythonhosted.org/packages/16/cf/924b7b14267d47f5055bb5d032c7d24eb9542ac3631b460e1398fe9935ea/pyiceberg-0.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3507d079d43d724bffb80e75201f2995822af844b674642dcf73c19d5303994", size = 723754, upload-time = "2026-02-10T02:28:15.77Z" }, + { url = "https://files.pythonhosted.org/packages/24/a1/df2d73af6dc3ee301e727d0bef4421c57de02b5030cf38e39ed25ef36154/pyiceberg-0.11.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eb3719cd61a0512596b4306283072de443d84ec7b68654f565b0d7c2d7cdeeeb", size = 715749, upload-time = "2026-02-10T02:28:17.034Z" }, + { url = "https://files.pythonhosted.org/packages/8e/0a/c3cdcd5ed417aceb2f73e8463d97e8dd7e3f7021015d0c8d51394a5c5a63/pyiceberg-0.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b9a71fd6b1c3c625ed2a9ca2cecf0dc8713acc5814e78c9becde3b1f42315c35", size = 720600, upload-time = "2026-02-10T02:28:18.275Z" }, + { url = "https://files.pythonhosted.org/packages/01/b8/29ec7281fb831ab983f953b00924c1cc3ebc21e9f67a1466af9b63767ba4/pyiceberg-0.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:bed2df9eb7e1496af22fa2307dbd13f29865b98ba5851695ffd1f4436edc05f9", size = 530631, upload-time = "2026-02-10T02:28:19.561Z" }, ] [package.optional-dependencies] pyarrow = [ { name = "pyarrow" }, + { name = "pyiceberg-core" }, ] sql-sqlite = [ { name = "sqlalchemy" }, @@ -696,7 +726,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "datafusion", specifier = "==50.*" }, + { name = "datafusion", specifier = "==52.*" }, { name = "fastavro", specifier = ">=1.11.1" }, { name = "maturin", specifier = ">=1.0,<2.0" }, { name = "pyiceberg", extras = ["sql-sqlite", "pyarrow"] }, @@ -712,6 +742,70 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/5e/1aa9a93198c6b64513c9d7752de7422c06402de6600a8767da1524f9570b/pyparsing-3.2.5-py3-none-any.whl", hash = "sha256:e38a4f02064cf41fe6593d328d0512495ad1f3d8a91c4f73fc401b3079a59a5e", size = 113890, upload-time = "2025-09-21T04:11:04.117Z" }, ] +[[package]] +name = "pyroaring" +version = "1.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0f/e4/975f0fa77fc3590820b4a3ac49704644b389795409bc12eb91729f845812/pyroaring-1.0.3.tar.gz", hash = "sha256:cd7392d1c010c9e41c11c62cd0610c8852e7e9698b1f7f6c2fcdefe50e7ef6da", size = 188688, upload-time = "2025-10-09T09:08:22.448Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/87/f19328d35f29b1d634f8f2127941927b37c6b68890912a4668e1a272d54d/pyroaring-1.0.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:c10e4cfbe203a578c78808406af491e3615d5e46cf69a7709050243346cd68bc", size = 670434, upload-time = "2025-10-09T09:06:34.89Z" }, + { url = "https://files.pythonhosted.org/packages/2f/fa/d933448844925728990e2256bbfa4f18cc3956d43d548270317128038015/pyroaring-1.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:cc329c62e504f2531c4008240f31736bcd2dee4339071f1eac0648068e6d17fa", size = 367332, upload-time = "2025-10-09T09:06:36.549Z" }, + { url = "https://files.pythonhosted.org/packages/39/e4/ab74296c3aac8f3ceb800354c2f811de2c4c518b3ca3a7a28e0599740cff/pyroaring-1.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8c7fb6ddf6ef31148f0939bc5c26b681d63df301ee1e372525012dd7bfe4a30a", size = 311443, upload-time = "2025-10-09T09:06:37.617Z" }, + { url = "https://files.pythonhosted.org/packages/6b/71/13ff623f3bba340ea7cc841883d7a0eaba1bec7e2d4e0d6759b89a4ce754/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cd18446832ea04a7d33bd6b78270b0be14eabcda5937af3428d6cb3d2bf98e54", size = 1853845, upload-time = "2025-10-09T09:06:38.706Z" }, + { url = "https://files.pythonhosted.org/packages/2b/05/40c0b37d78b16842c924b87b4fa491f4b20ed0e40c6255c872df45314247/pyroaring-1.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f0cbc766df2a24e28f23d69b66bbec64e691799219fd82c2f2236f03fc88e2e", size = 2045170, upload-time = "2025-10-09T09:06:40.229Z" }, + { url = "https://files.pythonhosted.org/packages/78/c6/6560c61d2f5c30fbb8f7b9a1c7d02068e78a7e8cd336eb294ec70896a80a/pyroaring-1.0.3-cp310-cp310-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:96a51e96f8f473381615f0f852f7238ad0a47f28e4a35e9f082468c5cfe4e9c3", size = 1791073, upload-time = "2025-10-09T09:06:42.092Z" }, + { url = "https://files.pythonhosted.org/packages/33/9e/0c91d4dbc4ec7bea9dcd3c203cfb8d96ed9df3c46981c0b22e9f17e98296/pyroaring-1.0.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:654af38b1f1c9bdc27b4f6d331fc5d91599df96e72a6df1886f4d95eea60ab29", size = 1789048, upload-time = "2025-10-09T09:06:43.392Z" }, + { url = "https://files.pythonhosted.org/packages/e9/48/51af418321cda2a1cfa64a48397ea0b73da74afe5c53a862525476d8a42c/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6721036afa31c07bdcbb4fcafa166660cf9c2eac695dcd495f8778549fa55899", size = 2838811, upload-time = "2025-10-09T09:06:44.588Z" }, + { url = "https://files.pythonhosted.org/packages/b2/69/7bc7070b35f72706e3870b5856e73656b9065bedae90268da5d77be00b15/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:0caa10f20329d09233fac6550b2adce4d9f173f748a9a9a5ea3b7033827dfe2d", size = 2640299, upload-time = "2025-10-09T09:06:46.159Z" }, + { url = "https://files.pythonhosted.org/packages/d0/87/5f2f590973d454e79ee8729aca888f9bb2d6018f7c9816bf66000cbc5e88/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:f109be8af937e85c52cb920d3fd120db52b172f59460852d2e3d2e3d13a4f52a", size = 2965427, upload-time = "2025-10-09T09:06:47.558Z" }, + { url = "https://files.pythonhosted.org/packages/e4/c8/1b425503141681db94941d9d3f41333bbd61975c3fc62d95122c372da85a/pyroaring-1.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:ddc80bfcd313c7c524a2742d263e73cae088b6a611b77dcc46fa90c306f6dace", size = 3084447, upload-time = "2025-10-09T09:06:48.727Z" }, + { url = "https://files.pythonhosted.org/packages/51/b9/2ac712ea90bd1e0d7e49e5e26c0c5aad1d77f21cf520a7edf46a477f5217/pyroaring-1.0.3-cp310-cp310-win32.whl", hash = "sha256:5a183f5ec069757fe5b60e37f7c6fa8a53178eacf0d76601b739e2890edee036", size = 204956, upload-time = "2025-10-09T09:06:49.801Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b9/1d4859c74d05f72b86dc0b308e6221e814a178459301cea9bcd084b4a92b/pyroaring-1.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:051bd9a66ce855a1143faa2b879ea6c6ca2905209e172ce9eedf79834897c730", size = 253778, upload-time = "2025-10-09T09:06:50.68Z" }, + { url = "https://files.pythonhosted.org/packages/64/20/b421100bd14b6a1074945af1418671630e1b8c4996ef000ac4e363785ead/pyroaring-1.0.3-cp310-cp310-win_arm64.whl", hash = "sha256:3043ff5c85375310ca3cd3e01944e03026e0ec07885e52dfabcfcd9dc303867f", size = 219330, upload-time = "2025-10-09T09:06:52.011Z" }, + { url = "https://files.pythonhosted.org/packages/39/ed/5e555dd99b12318ea1c7666b773fc4f097aeb609eeb1c1b3da519d445f71/pyroaring-1.0.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:755cdac1f9a1b7b5c621e570d4f6dbcf3b8e4a1e35a66f976104ecb35dce4ed2", size = 675916, upload-time = "2025-10-09T09:06:53.174Z" }, + { url = "https://files.pythonhosted.org/packages/da/06/dd8a9a87b90c4560f8384ab1dbafcd40c2a16f6777a07334a8e341bd7383/pyroaring-1.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ebab073db620f26f0ba11e13fa2f35e3b1298209fba47b6bc8cb6f0e2c9627f9", size = 369743, upload-time = "2025-10-09T09:06:54.421Z" }, + { url = "https://files.pythonhosted.org/packages/35/aa/da882011045ddacffe818a4fcbdd7e609a15f9c83d536222ec5b17af4aa9/pyroaring-1.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:684fb8dffe19bdb7f91897c65eac6eee23b1e46043c47eb24288f28a1170fe04", size = 313981, upload-time = "2025-10-09T09:06:55.514Z" }, + { url = "https://files.pythonhosted.org/packages/ed/3c/f6534844b02e2505ccdc9aae461c9838ab96f72b5688c045448761735512/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:678d31fc24e82945a1bfb14816c77823983382ffea76985d494782aa2f058427", size = 1923181, upload-time = "2025-10-09T09:06:56.897Z" }, + { url = "https://files.pythonhosted.org/packages/ea/82/9f1a85ba33e3d89b9cdb8183fb2fd2f25720d10742dd8827508ccccc13ae/pyroaring-1.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7d815f624e0285db3669f673d1725cb754b120ec70d0032d7c7166103a96c96d", size = 2113222, upload-time = "2025-10-09T09:06:58.388Z" }, + { url = "https://files.pythonhosted.org/packages/a7/f8/4d4340971cbc1379f987c847080bcb7f9765a57e122f392c3a3485c9587e/pyroaring-1.0.3-cp311-cp311-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:57fd5b80dacb8e888402b6b7508a734c6a527063e4e24e882ff2e0fd90721ada", size = 1837385, upload-time = "2025-10-09T09:06:59.449Z" }, + { url = "https://files.pythonhosted.org/packages/c6/58/d14cc561685e4c224af26b4fdb4f6c7e643294ac5a4b29f178b5cbb71af1/pyroaring-1.0.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ab26a7a45a0bb46c00394d1a60a9f2d57c220f84586e30d59b39784b0f94aee6", size = 1856170, upload-time = "2025-10-09T09:07:00.608Z" }, + { url = "https://files.pythonhosted.org/packages/d1/d2/d2d9790c373f6438d4d0958bc4c79f3dc77826d8553743ff3f64acdc9ab3/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9232f3f606315d59049c128154100fd05008d5c5c211e48b21848cd41ee64d26", size = 2909282, upload-time = "2025-10-09T09:07:02.124Z" }, + { url = "https://files.pythonhosted.org/packages/bc/28/4b2277982302b5b406998064ca1eaef1a79e4ea87185f511e33e7a7e3511/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f34b44b3ec3df97b978799f2901fefb2a48d367496fd1cde3cc5fe8b3bc13510", size = 2701034, upload-time = "2025-10-09T09:07:03.403Z" }, + { url = "https://files.pythonhosted.org/packages/d2/91/b2340193825fa2431cf735f0ecb23206fb31f386fecca38336935a294513/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:25a83ec6bac3106568bd3fdd316f0fee52aa0be8c72da565ad02b10ae7905924", size = 3028962, upload-time = "2025-10-09T09:07:05.558Z" }, + { url = "https://files.pythonhosted.org/packages/07/ea/ad79073cc5d8dcca35d1a955bb886d96905e9dacc58d1971fda012a5ad18/pyroaring-1.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c17d4ec53b5b6b333d9a9515051213a691293ada785dc8c025d3641482597ed3", size = 3152109, upload-time = "2025-10-09T09:07:06.887Z" }, + { url = "https://files.pythonhosted.org/packages/9a/de/f55a1093acb16d25ff9811546823e59078e4a3e56d2eb0ff5d10f696933d/pyroaring-1.0.3-cp311-cp311-win32.whl", hash = "sha256:d54024459ace600f1d1ffbc6dc3c60eb47cca3b678701f06148f59e10f6f8d7b", size = 204246, upload-time = "2025-10-09T09:07:08.036Z" }, + { url = "https://files.pythonhosted.org/packages/c6/e5/36bf3039733b8e00732892c9334b2f5309f38e72af0b3b40b8729b5857a3/pyroaring-1.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:c28750148ef579a7447a8cb60b39e5943e03f8c29bce8f2788728f6f23d1887a", size = 254637, upload-time = "2025-10-09T09:07:09.103Z" }, + { url = "https://files.pythonhosted.org/packages/d6/e8/e2b78e595b5a82a6014af327614756a55f17ec4120a2ab197f1762641316/pyroaring-1.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:535d8deccbd8db2c6bf38629243e9646756905574a742b2a72ff51d6461d616c", size = 219597, upload-time = "2025-10-09T09:07:10.38Z" }, + { url = "https://files.pythonhosted.org/packages/dd/09/a5376d55672e0535019ba1469888909d0046cea0cfb969a4aa1f99caaf22/pyroaring-1.0.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:add3e4c78eb590a76526ecce8d1566eecdd5822e351c36b3697997f4a80ed808", size = 681056, upload-time = "2025-10-09T09:07:11.497Z" }, + { url = "https://files.pythonhosted.org/packages/23/dd/78f59d361bd9ebf8de3660408b0c48664ade0a057ebcf4b207d99ac1a698/pyroaring-1.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ebaffe846cf4ba4f00ce6b8a9f39613f24e2d09447e77be4fa6e898bc36451b6", size = 375111, upload-time = "2025-10-09T09:07:12.597Z" }, + { url = "https://files.pythonhosted.org/packages/bf/03/10dc93f83a5453eb40a69c79106a8385b40aa12cf4531ca72bd9d7f45cb2/pyroaring-1.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a9459f27498f97d08031a34a5ead230b77eb0ab3cc3d85b7f54faa2fd548acd6", size = 314319, upload-time = "2025-10-09T09:07:13.579Z" }, + { url = "https://files.pythonhosted.org/packages/86/9e/b00c38a7e62a73e152055f593595c37152e61fc2896fd11538a7c71fbe4e/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2b2eb8bd1c35c772994889be9f7dda09477475d7aa1e2af9ab4ef18619326f6", size = 1869251, upload-time = "2025-10-09T09:07:14.584Z" }, + { url = "https://files.pythonhosted.org/packages/4f/33/f32d00ca105b66303deab43d027c3574c8ade8525dac0e5b50a9fb4d1b76/pyroaring-1.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d31f4c1c906f1af14ce61a3959d04a14a64c594f8a768399146a45bbd341f21f", size = 2071551, upload-time = "2025-10-09T09:07:15.713Z" }, + { url = "https://files.pythonhosted.org/packages/5d/89/e953cae181ba4c7523334855a1ca0ae8eeea3cee8d7cd39c56bd99709d3f/pyroaring-1.0.3-cp312-cp312-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53be988fc86698d56c11049bfe5113a2f6990adb1fa2782b29636509808b6aa7", size = 1781071, upload-time = "2025-10-09T09:07:17.19Z" }, + { url = "https://files.pythonhosted.org/packages/fa/db/65d4be532e68b62a84a9c89b24d0a1394f452f484fa29392142d9a3b9c48/pyroaring-1.0.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7df84d223424523b19a23781f4246cc247fd6d821e1bc0853c2f25669136f7d0", size = 1795670, upload-time = "2025-10-09T09:07:18.524Z" }, + { url = "https://files.pythonhosted.org/packages/f5/9e/684ea0568ce7d30fc4e01ad1c666e9ce1a5b1702fa630231f4f6bdb96539/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:34a781f1f9766897f63ef18be129827340ae37764015b83fdcff1efb9e29136d", size = 2849305, upload-time = "2025-10-09T09:07:20.388Z" }, + { url = "https://files.pythonhosted.org/packages/7c/fd/d7773a2adf91f45d8924197954c66b1694325afd2f27e02edaac07338402/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:1f414343b4ed0756734328cdf2a91022fc54503769e3f8d79bd0b672ea815a16", size = 2692843, upload-time = "2025-10-09T09:07:22.042Z" }, + { url = "https://files.pythonhosted.org/packages/13/72/b8a99ba138eebd8ff9bf8d15f3942e9e43e8e45723e2e6b7b09e542b7448/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:d16ae185c72dc64f76335dbe53e53a892e78115adc92194957d1b7ef74d230b9", size = 2983440, upload-time = "2025-10-09T09:07:23.419Z" }, + { url = "https://files.pythonhosted.org/packages/ca/94/e6ed1f682d850e039c71b2032bacdefc5082dc809796cf34b9e6f24c604d/pyroaring-1.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f888447bf22dde7759108bfe6dfbeb6bbb61b14948de9c4cb6843c4dd57e2215", size = 3117542, upload-time = "2025-10-09T09:07:25.104Z" }, + { url = "https://files.pythonhosted.org/packages/8f/89/d55b0ed3e098ef89c421b43b748afe3d90eb250cab50b9e53e3a3449ac58/pyroaring-1.0.3-cp312-cp312-win32.whl", hash = "sha256:fbbdc44c51a0a3efd7be3dbe04466278ce098fcd101aa1905849319042159770", size = 205118, upload-time = "2025-10-09T09:07:26.532Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e1/b71fef6a73efb50110d33d714235ff7059f4ebae98dc474b6549b322f48f/pyroaring-1.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:3b217c4b3ad953b4c759a0d2f9bd95316f0c345b9f7adb49e6ded7a1f5106bd4", size = 260629, upload-time = "2025-10-09T09:07:27.528Z" }, + { url = "https://files.pythonhosted.org/packages/57/33/66ee872079c9c47512d6e17d374bcad8d91350c24dc20fbe678c34b33745/pyroaring-1.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:e6bcf838564c21bab8fe6c2748b4990d4cd90612d8c470c04889def7bb5114ea", size = 219032, upload-time = "2025-10-09T09:07:28.754Z" }, + { url = "https://files.pythonhosted.org/packages/1f/95/97142ee32587ddda9e2cd614b865eeb5c0ee91006a51928f4074cd6e8e5f/pyroaring-1.0.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:20bc947054b197d1baa76cd05d70b8e04f95b82e698266e2f8f2f4b36d764477", size = 678813, upload-time = "2025-10-09T09:07:29.936Z" }, + { url = "https://files.pythonhosted.org/packages/70/5e/cff22be3a76a80024bdf00a9decdffedc6e80f037328a58b58c1b521442d/pyroaring-1.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ba5909b4c66bb85cab345e2f3a87e5ce671509c94b8c9823d8db64e107cbe854", size = 373661, upload-time = "2025-10-09T09:07:30.983Z" }, + { url = "https://files.pythonhosted.org/packages/86/73/fc406a67cd49e1707d1c3d08214458959dd579eff88c28587b356dfa068b/pyroaring-1.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:b744746ba5da27fad760067f12633f5d384db6a1e65648d00244ceacbbd87731", size = 313559, upload-time = "2025-10-09T09:07:32.099Z" }, + { url = "https://files.pythonhosted.org/packages/f9/64/c7fe510523445f27e2cb04de6ffd3137f9d72db438b62db2bfa3dafcf4fc/pyroaring-1.0.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5b16c2a2791a5a09c4b59c0e1069ac1c877d0df25cae3155579c7eac8844676e", size = 1875926, upload-time = "2025-10-09T09:07:33.701Z" }, + { url = "https://files.pythonhosted.org/packages/47/74/da9b8ad2ca9ce6af1377f2cffdad6582a51a5f5df4f26df5c41810c9de5b/pyroaring-1.0.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e7f68dfcf8d01177267f4bc06c4960fe8e39577470d1b52c9af8b61a72ca8767", size = 2064377, upload-time = "2025-10-09T09:07:35.273Z" }, + { url = "https://files.pythonhosted.org/packages/99/e3/8a70c5a5f7821c63709e2769aeccda8ae87a192198374bc475cbee543a22/pyroaring-1.0.3-cp313-cp313-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:dba4e4700030182a981a3c887aa73887697145fc9ffb192f908aa59b718fbbdd", size = 1778320, upload-time = "2025-10-09T09:07:36.782Z" }, + { url = "https://files.pythonhosted.org/packages/04/4c/08159a07c3723a2775064887543766b6115b4975e7baaa4d51e5580701a4/pyroaring-1.0.3-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e26dd1dc1edba02288902914bdb559e53e346e9155defa43c31fcab831b55342", size = 1786569, upload-time = "2025-10-09T09:07:38.473Z" }, + { url = "https://files.pythonhosted.org/packages/e5/ff/55a18d0e7e0dc4cd9f43988b746e788234a8d660fa17367c5ed9fa799348/pyroaring-1.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6eb98d2cacfc6d51c6a69893f04075e07b3df761eac71ba162c43b9b4c4452ad", size = 2852766, upload-time = "2025-10-09T09:07:39.633Z" }, + { url = "https://files.pythonhosted.org/packages/24/3c/419e25c51843dd40975ae37d67dea4f2f256554b5bec32237f607ec8ef21/pyroaring-1.0.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:a967e9eddb9485cbdd95d6371e3dada67880844d836c0283d3b11efe9225d1b7", size = 2683904, upload-time = "2025-10-09T09:07:41.139Z" }, + { url = "https://files.pythonhosted.org/packages/75/64/8d91f1b85b42925af632fc2c1047bb314be622dce890a4181a0a8d6e498d/pyroaring-1.0.3-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b12ef7f992ba7be865f91c7c098fd8ac6c413563aaa14d5b1e2bcb8cb43a4614", size = 2973884, upload-time = "2025-10-09T09:07:42.34Z" }, + { url = "https://files.pythonhosted.org/packages/61/6d/c867625549df0dc9ad675424ecf989fa2f08f0571bd46dfc4f7218737dd2/pyroaring-1.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:82ca5be174b85c40be7b00bc6bf39b2931a1b4a465f3af17ec6b9c48e9aa6fe0", size = 3103671, upload-time = "2025-10-09T09:07:44.055Z" }, + { url = "https://files.pythonhosted.org/packages/59/b1/d47c5ec2b2580d0b94f42575be8f49907a0f4aa396fdc18660f3b5060d54/pyroaring-1.0.3-cp313-cp313-win32.whl", hash = "sha256:f758c681e63ffe74b20423695e71f0410920f41b075cee679ffb5bc2bf38440b", size = 205153, upload-time = "2025-10-09T09:07:45.496Z" }, + { url = "https://files.pythonhosted.org/packages/c4/92/3600486936eebab747ae1462d231d7f87d234da24a04e82e1915c00f4427/pyroaring-1.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:428c3bb384fe4c483feb5cf7aa3aef1621fb0a5c4f3d391da67b2c4a43f08a10", size = 260349, upload-time = "2025-10-09T09:07:46.524Z" }, + { url = "https://files.pythonhosted.org/packages/77/96/8dde074f1ad2a1c3d2091b22de80d1b3007824e649e06eeeebded83f4d48/pyroaring-1.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:9c0c856e8aa5606e8aed5f30201286e404fdc9093f81fefe82d2e79e67472bb2", size = 218775, upload-time = "2025-10-09T09:07:47.558Z" }, +] + [[package]] name = "pytest" version = "8.4.2" @@ -759,16 +853,15 @@ wheels = [ [[package]] name = "rich" -version = "13.9.4" +version = "14.2.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "markdown-it-py" }, { name = "pygments" }, - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ab/3a/0316b28d0761c6734d6bc14e770d85506c986c85ffb239e688eeaab2c2bc/rich-13.9.4.tar.gz", hash = "sha256:439594978a49a09530cff7ebc4b5c7103ef57baf48d5ea3184f21d9a2befa098", size = 223149, upload-time = "2024-11-01T16:43:57.873Z" } +sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/19/71/39c7c0d87f8d4e6c020a393182060eaefeeae6c01dab6a84ec346f2567df/rich-13.9.4-py3-none-any.whl", hash = "sha256:6049d5e6ec054bf2779ab3358186963bac2ea89175919d699e378b99738c2a90", size = 242424, upload-time = "2024-11-01T16:43:55.817Z" }, + { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" }, ] [[package]] @@ -780,15 +873,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] -[[package]] -name = "sortedcontainers" -version = "2.4.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, -] - [[package]] name = "sqlalchemy" version = "2.0.44" @@ -933,3 +1017,93 @@ sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6 wheels = [ { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" }, ] + +[[package]] +name = "zstandard" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/7a/28efd1d371f1acd037ac64ed1c5e2b41514a6cc937dd6ab6a13ab9f0702f/zstandard-0.25.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e59fdc271772f6686e01e1b3b74537259800f57e24280be3f29c8a0deb1904dd", size = 795256, upload-time = "2025-09-14T22:15:56.415Z" }, + { url = "https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4d441506e9b372386a5271c64125f72d5df6d2a8e8a2a45a0ae09b03cb781ef7", size = 640565, upload-time = "2025-09-14T22:15:58.177Z" }, + { url = "https://files.pythonhosted.org/packages/9d/1b/4fdb2c12eb58f31f28c4d28e8dc36611dd7205df8452e63f52fb6261d13e/zstandard-0.25.0-cp310-cp310-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:ab85470ab54c2cb96e176f40342d9ed41e58ca5733be6a893b730e7af9c40550", size = 5345306, upload-time = "2025-09-14T22:16:00.165Z" }, + { url = "https://files.pythonhosted.org/packages/73/28/a44bdece01bca027b079f0e00be3b6bd89a4df180071da59a3dd7381665b/zstandard-0.25.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e05ab82ea7753354bb054b92e2f288afb750e6b439ff6ca78af52939ebbc476d", size = 5055561, upload-time = "2025-09-14T22:16:02.22Z" }, + { url = "https://files.pythonhosted.org/packages/e9/74/68341185a4f32b274e0fc3410d5ad0750497e1acc20bd0f5b5f64ce17785/zstandard-0.25.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:78228d8a6a1c177a96b94f7e2e8d012c55f9c760761980da16ae7546a15a8e9b", size = 5402214, upload-time = "2025-09-14T22:16:04.109Z" }, + { url = "https://files.pythonhosted.org/packages/8b/67/f92e64e748fd6aaffe01e2b75a083c0c4fd27abe1c8747fee4555fcee7dd/zstandard-0.25.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:2b6bd67528ee8b5c5f10255735abc21aa106931f0dbaf297c7be0c886353c3d0", size = 5449703, upload-time = "2025-09-14T22:16:06.312Z" }, + { url = "https://files.pythonhosted.org/packages/fd/e5/6d36f92a197c3c17729a2125e29c169f460538a7d939a27eaaa6dcfcba8e/zstandard-0.25.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4b6d83057e713ff235a12e73916b6d356e3084fd3d14ced499d84240f3eecee0", size = 5556583, upload-time = "2025-09-14T22:16:08.457Z" }, + { url = "https://files.pythonhosted.org/packages/d7/83/41939e60d8d7ebfe2b747be022d0806953799140a702b90ffe214d557638/zstandard-0.25.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9174f4ed06f790a6869b41cba05b43eeb9a35f8993c4422ab853b705e8112bbd", size = 5045332, upload-time = "2025-09-14T22:16:10.444Z" }, + { url = "https://files.pythonhosted.org/packages/b3/87/d3ee185e3d1aa0133399893697ae91f221fda79deb61adbe998a7235c43f/zstandard-0.25.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:25f8f3cd45087d089aef5ba3848cd9efe3ad41163d3400862fb42f81a3a46701", size = 5572283, upload-time = "2025-09-14T22:16:12.128Z" }, + { url = "https://files.pythonhosted.org/packages/0a/1d/58635ae6104df96671076ac7d4ae7816838ce7debd94aecf83e30b7121b0/zstandard-0.25.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:3756b3e9da9b83da1796f8809dd57cb024f838b9eeafde28f3cb472012797ac1", size = 4959754, upload-time = "2025-09-14T22:16:14.225Z" }, + { url = "https://files.pythonhosted.org/packages/75/d6/57e9cb0a9983e9a229dd8fd2e6e96593ef2aa82a3907188436f22b111ccd/zstandard-0.25.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:81dad8d145d8fd981b2962b686b2241d3a1ea07733e76a2f15435dfb7fb60150", size = 5266477, upload-time = "2025-09-14T22:16:16.343Z" }, + { url = "https://files.pythonhosted.org/packages/d1/a9/ee891e5edf33a6ebce0a028726f0bbd8567effe20fe3d5808c42323e8542/zstandard-0.25.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:a5a419712cf88862a45a23def0ae063686db3d324cec7edbe40509d1a79a0aab", size = 5440914, upload-time = "2025-09-14T22:16:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/58/08/a8522c28c08031a9521f27abc6f78dbdee7312a7463dd2cfc658b813323b/zstandard-0.25.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:e7360eae90809efd19b886e59a09dad07da4ca9ba096752e61a2e03c8aca188e", size = 5819847, upload-time = "2025-09-14T22:16:20.559Z" }, + { url = "https://files.pythonhosted.org/packages/6f/11/4c91411805c3f7b6f31c60e78ce347ca48f6f16d552fc659af6ec3b73202/zstandard-0.25.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:75ffc32a569fb049499e63ce68c743155477610532da1eb38e7f24bf7cd29e74", size = 5363131, upload-time = "2025-09-14T22:16:22.206Z" }, + { url = "https://files.pythonhosted.org/packages/ef/d6/8c4bd38a3b24c4c7676a7a3d8de85d6ee7a983602a734b9f9cdefb04a5d6/zstandard-0.25.0-cp310-cp310-win32.whl", hash = "sha256:106281ae350e494f4ac8a80470e66d1fe27e497052c8d9c3b95dc4cf1ade81aa", size = 436469, upload-time = "2025-09-14T22:16:25.002Z" }, + { url = "https://files.pythonhosted.org/packages/93/90/96d50ad417a8ace5f841b3228e93d1bb13e6ad356737f42e2dde30d8bd68/zstandard-0.25.0-cp310-cp310-win_amd64.whl", hash = "sha256:ea9d54cc3d8064260114a0bbf3479fc4a98b21dffc89b3459edd506b69262f6e", size = 506100, upload-time = "2025-09-14T22:16:23.569Z" }, + { url = "https://files.pythonhosted.org/packages/2a/83/c3ca27c363d104980f1c9cee1101cc8ba724ac8c28a033ede6aab89585b1/zstandard-0.25.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:933b65d7680ea337180733cf9e87293cc5500cc0eb3fc8769f4d3c88d724ec5c", size = 795254, upload-time = "2025-09-14T22:16:26.137Z" }, + { url = "https://files.pythonhosted.org/packages/ac/4d/e66465c5411a7cf4866aeadc7d108081d8ceba9bc7abe6b14aa21c671ec3/zstandard-0.25.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a3f79487c687b1fc69f19e487cd949bf3aae653d181dfb5fde3bf6d18894706f", size = 640559, upload-time = "2025-09-14T22:16:27.973Z" }, + { url = "https://files.pythonhosted.org/packages/12/56/354fe655905f290d3b147b33fe946b0f27e791e4b50a5f004c802cb3eb7b/zstandard-0.25.0-cp311-cp311-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:0bbc9a0c65ce0eea3c34a691e3c4b6889f5f3909ba4822ab385fab9057099431", size = 5348020, upload-time = "2025-09-14T22:16:29.523Z" }, + { url = "https://files.pythonhosted.org/packages/3b/13/2b7ed68bd85e69a2069bcc72141d378f22cae5a0f3b353a2c8f50ef30c1b/zstandard-0.25.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:01582723b3ccd6939ab7b3a78622c573799d5d8737b534b86d0e06ac18dbde4a", size = 5058126, upload-time = "2025-09-14T22:16:31.811Z" }, + { url = "https://files.pythonhosted.org/packages/c9/dd/fdaf0674f4b10d92cb120ccff58bbb6626bf8368f00ebfd2a41ba4a0dc99/zstandard-0.25.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5f1ad7bf88535edcf30038f6919abe087f606f62c00a87d7e33e7fc57cb69fcc", size = 5405390, upload-time = "2025-09-14T22:16:33.486Z" }, + { url = "https://files.pythonhosted.org/packages/0f/67/354d1555575bc2490435f90d67ca4dd65238ff2f119f30f72d5cde09c2ad/zstandard-0.25.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:06acb75eebeedb77b69048031282737717a63e71e4ae3f77cc0c3b9508320df6", size = 5452914, upload-time = "2025-09-14T22:16:35.277Z" }, + { url = "https://files.pythonhosted.org/packages/bb/1f/e9cfd801a3f9190bf3e759c422bbfd2247db9d7f3d54a56ecde70137791a/zstandard-0.25.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9300d02ea7c6506f00e627e287e0492a5eb0371ec1670ae852fefffa6164b072", size = 5559635, upload-time = "2025-09-14T22:16:37.141Z" }, + { url = "https://files.pythonhosted.org/packages/21/88/5ba550f797ca953a52d708c8e4f380959e7e3280af029e38fbf47b55916e/zstandard-0.25.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:bfd06b1c5584b657a2892a6014c2f4c20e0db0208c159148fa78c65f7e0b0277", size = 5048277, upload-time = "2025-09-14T22:16:38.807Z" }, + { url = "https://files.pythonhosted.org/packages/46/c0/ca3e533b4fa03112facbe7fbe7779cb1ebec215688e5df576fe5429172e0/zstandard-0.25.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:f373da2c1757bb7f1acaf09369cdc1d51d84131e50d5fa9863982fd626466313", size = 5574377, upload-time = "2025-09-14T22:16:40.523Z" }, + { url = "https://files.pythonhosted.org/packages/12/9b/3fb626390113f272abd0799fd677ea33d5fc3ec185e62e6be534493c4b60/zstandard-0.25.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6c0e5a65158a7946e7a7affa6418878ef97ab66636f13353b8502d7ea03c8097", size = 4961493, upload-time = "2025-09-14T22:16:43.3Z" }, + { url = "https://files.pythonhosted.org/packages/cb/d3/23094a6b6a4b1343b27ae68249daa17ae0651fcfec9ed4de09d14b940285/zstandard-0.25.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:c8e167d5adf59476fa3e37bee730890e389410c354771a62e3c076c86f9f7778", size = 5269018, upload-time = "2025-09-14T22:16:45.292Z" }, + { url = "https://files.pythonhosted.org/packages/8c/a7/bb5a0c1c0f3f4b5e9d5b55198e39de91e04ba7c205cc46fcb0f95f0383c1/zstandard-0.25.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:98750a309eb2f020da61e727de7d7ba3c57c97cf6213f6f6277bb7fb42a8e065", size = 5443672, upload-time = "2025-09-14T22:16:47.076Z" }, + { url = "https://files.pythonhosted.org/packages/27/22/503347aa08d073993f25109c36c8d9f029c7d5949198050962cb568dfa5e/zstandard-0.25.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:22a086cff1b6ceca18a8dd6096ec631e430e93a8e70a9ca5efa7561a00f826fa", size = 5822753, upload-time = "2025-09-14T22:16:49.316Z" }, + { url = "https://files.pythonhosted.org/packages/e2/be/94267dc6ee64f0f8ba2b2ae7c7a2df934a816baaa7291db9e1aa77394c3c/zstandard-0.25.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:72d35d7aa0bba323965da807a462b0966c91608ef3a48ba761678cb20ce5d8b7", size = 5366047, upload-time = "2025-09-14T22:16:51.328Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a3/732893eab0a3a7aecff8b99052fecf9f605cf0fb5fb6d0290e36beee47a4/zstandard-0.25.0-cp311-cp311-win32.whl", hash = "sha256:f5aeea11ded7320a84dcdd62a3d95b5186834224a9e55b92ccae35d21a8b63d4", size = 436484, upload-time = "2025-09-14T22:16:55.005Z" }, + { url = "https://files.pythonhosted.org/packages/43/a3/c6155f5c1cce691cb80dfd38627046e50af3ee9ddc5d0b45b9b063bfb8c9/zstandard-0.25.0-cp311-cp311-win_amd64.whl", hash = "sha256:daab68faadb847063d0c56f361a289c4f268706b598afbf9ad113cbe5c38b6b2", size = 506183, upload-time = "2025-09-14T22:16:52.753Z" }, + { url = "https://files.pythonhosted.org/packages/8c/3e/8945ab86a0820cc0e0cdbf38086a92868a9172020fdab8a03ac19662b0e5/zstandard-0.25.0-cp311-cp311-win_arm64.whl", hash = "sha256:22a06c5df3751bb7dc67406f5374734ccee8ed37fc5981bf1ad7041831fa1137", size = 462533, upload-time = "2025-09-14T22:16:53.878Z" }, + { url = "https://files.pythonhosted.org/packages/82/fc/f26eb6ef91ae723a03e16eddb198abcfce2bc5a42e224d44cc8b6765e57e/zstandard-0.25.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b", size = 795738, upload-time = "2025-09-14T22:16:56.237Z" }, + { url = "https://files.pythonhosted.org/packages/aa/1c/d920d64b22f8dd028a8b90e2d756e431a5d86194caa78e3819c7bf53b4b3/zstandard-0.25.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00", size = 640436, upload-time = "2025-09-14T22:16:57.774Z" }, + { url = "https://files.pythonhosted.org/packages/53/6c/288c3f0bd9fcfe9ca41e2c2fbfd17b2097f6af57b62a81161941f09afa76/zstandard-0.25.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64", size = 5343019, upload-time = "2025-09-14T22:16:59.302Z" }, + { url = "https://files.pythonhosted.org/packages/1e/15/efef5a2f204a64bdb5571e6161d49f7ef0fffdbca953a615efbec045f60f/zstandard-0.25.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea", size = 5063012, upload-time = "2025-09-14T22:17:01.156Z" }, + { url = "https://files.pythonhosted.org/packages/b7/37/a6ce629ffdb43959e92e87ebdaeebb5ac81c944b6a75c9c47e300f85abdf/zstandard-0.25.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb", size = 5394148, upload-time = "2025-09-14T22:17:03.091Z" }, + { url = "https://files.pythonhosted.org/packages/e3/79/2bf870b3abeb5c070fe2d670a5a8d1057a8270f125ef7676d29ea900f496/zstandard-0.25.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a", size = 5451652, upload-time = "2025-09-14T22:17:04.979Z" }, + { url = "https://files.pythonhosted.org/packages/53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902", size = 5546993, upload-time = "2025-09-14T22:17:06.781Z" }, + { url = "https://files.pythonhosted.org/packages/85/c7/3483ad9ff0662623f3648479b0380d2de5510abf00990468c286c6b04017/zstandard-0.25.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f", size = 5046806, upload-time = "2025-09-14T22:17:08.415Z" }, + { url = "https://files.pythonhosted.org/packages/08/b3/206883dd25b8d1591a1caa44b54c2aad84badccf2f1de9e2d60a446f9a25/zstandard-0.25.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b", size = 5576659, upload-time = "2025-09-14T22:17:10.164Z" }, + { url = "https://files.pythonhosted.org/packages/9d/31/76c0779101453e6c117b0ff22565865c54f48f8bd807df2b00c2c404b8e0/zstandard-0.25.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6", size = 4953933, upload-time = "2025-09-14T22:17:11.857Z" }, + { url = "https://files.pythonhosted.org/packages/18/e1/97680c664a1bf9a247a280a053d98e251424af51f1b196c6d52f117c9720/zstandard-0.25.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91", size = 5268008, upload-time = "2025-09-14T22:17:13.627Z" }, + { url = "https://files.pythonhosted.org/packages/1e/73/316e4010de585ac798e154e88fd81bb16afc5c5cb1a72eeb16dd37e8024a/zstandard-0.25.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708", size = 5433517, upload-time = "2025-09-14T22:17:16.103Z" }, + { url = "https://files.pythonhosted.org/packages/5b/60/dd0f8cfa8129c5a0ce3ea6b7f70be5b33d2618013a161e1ff26c2b39787c/zstandard-0.25.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512", size = 5814292, upload-time = "2025-09-14T22:17:17.827Z" }, + { url = "https://files.pythonhosted.org/packages/fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa", size = 5360237, upload-time = "2025-09-14T22:17:19.954Z" }, + { url = "https://files.pythonhosted.org/packages/ff/8d/0309daffea4fcac7981021dbf21cdb2e3427a9e76bafbcdbdf5392ff99a4/zstandard-0.25.0-cp312-cp312-win32.whl", hash = "sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd", size = 436922, upload-time = "2025-09-14T22:17:24.398Z" }, + { url = "https://files.pythonhosted.org/packages/79/3b/fa54d9015f945330510cb5d0b0501e8253c127cca7ebe8ba46a965df18c5/zstandard-0.25.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01", size = 506276, upload-time = "2025-09-14T22:17:21.429Z" }, + { url = "https://files.pythonhosted.org/packages/ea/6b/8b51697e5319b1f9ac71087b0af9a40d8a6288ff8025c36486e0c12abcc4/zstandard-0.25.0-cp312-cp312-win_arm64.whl", hash = "sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9", size = 462679, upload-time = "2025-09-14T22:17:23.147Z" }, + { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" }, + { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" }, + { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" }, + { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" }, + { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" }, + { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" }, + { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" }, + { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" }, + { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" }, + { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" }, + { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" }, + { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" }, + { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" }, + { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" }, + { url = "https://files.pythonhosted.org/packages/8e/e0/69a553d2047f9a2c7347caa225bb3a63b6d7704ad74610cb7823baa08ed7/zstandard-0.25.0-cp313-cp313-win32.whl", hash = "sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09", size = 436936, upload-time = "2025-09-14T22:17:52.658Z" }, + { url = "https://files.pythonhosted.org/packages/d9/82/b9c06c870f3bd8767c201f1edbdf9e8dc34be5b0fbc5682c4f80fe948475/zstandard-0.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5", size = 506232, upload-time = "2025-09-14T22:17:50.402Z" }, + { url = "https://files.pythonhosted.org/packages/d4/57/60c3c01243bb81d381c9916e2a6d9e149ab8627c0c7d7abb2d73384b3c0c/zstandard-0.25.0-cp313-cp313-win_arm64.whl", hash = "sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049", size = 462671, upload-time = "2025-09-14T22:17:51.533Z" }, + { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" }, + { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" }, + { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" }, + { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" }, + { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" }, + { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" }, + { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" }, + { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" }, + { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" }, + { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" }, + { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" }, + { url = "https://files.pythonhosted.org/packages/c2/38/f249a2050ad1eea0bb364046153942e34abba95dd5520af199aed86fbb49/zstandard-0.25.0-cp314-cp314-win32.whl", hash = "sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12", size = 444513, upload-time = "2025-09-14T22:18:20.61Z" }, + { url = "https://files.pythonhosted.org/packages/3a/43/241f9615bcf8ba8903b3f0432da069e857fc4fd1783bd26183db53c4804b/zstandard-0.25.0-cp314-cp314-win_amd64.whl", hash = "sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2", size = 516118, upload-time = "2025-09-14T22:18:17.849Z" }, + { url = "https://files.pythonhosted.org/packages/f0/ef/da163ce2450ed4febf6467d77ccb4cd52c4c30ab45624bad26ca0a27260c/zstandard-0.25.0-cp314-cp314-win_arm64.whl", hash = "sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d", size = 476940, upload-time = "2025-09-14T22:18:19.088Z" }, +] diff --git a/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt index d3d1db1b67..54d74f5e70 100644 --- a/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt +++ b/crates/sqllogictest/testdata/slts/df_test/binary_predicate_pushdown.slt @@ -26,11 +26,10 @@ logical_plan 01)Filter: default.default.test_binary_table.data = LargeBinary("1,2") 02)--TableScan: default.default.test_binary_table projection=[id, data], partial_filters=[default.default.test_binary_table.data = LargeBinary("1,2")] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: data@1 = 0102 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,data] predicate:[data = 0102] +01)FilterExec: data@1 = 0102 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,data] predicate:[data = 0102] # Verify empty result from empty table query I? diff --git a/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt index c7269b68b9..466a45f8c6 100644 --- a/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt +++ b/crates/sqllogictest/testdata/slts/df_test/boolean_predicate_pushdown.slt @@ -37,11 +37,10 @@ logical_plan 01)Filter: default.default.test_boolean_table.is_active 02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[default.default.test_boolean_table.is_active] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: is_active@1 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = true] +01)FilterExec: is_active@1 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = true] # Query with is_active = true query ITT rowsort @@ -59,11 +58,10 @@ logical_plan 01)Filter: NOT default.default.test_boolean_table.is_active 02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[NOT default.default.test_boolean_table.is_active] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: NOT is_active@1 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] +01)FilterExec: NOT is_active@1 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] # Query with is_active = false query ITT rowsort @@ -80,11 +78,10 @@ logical_plan 01)Filter: NOT default.default.test_boolean_table.is_active 02)--TableScan: default.default.test_boolean_table projection=[id, is_active, description], partial_filters=[NOT default.default.test_boolean_table.is_active] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: NOT is_active@1 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] +01)FilterExec: NOT is_active@1 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,is_active,description] predicate:[is_active = false] # Query with is_active != true (includes false and NULL) query ITT rowsort diff --git a/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt index 41d8ec0276..a160ca77f4 100644 --- a/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt +++ b/crates/sqllogictest/testdata/slts/df_test/like_predicate_pushdown.slt @@ -35,11 +35,10 @@ logical_plan 01)Filter: default.default.test_unpartitioned_table.name LIKE Utf8("Al%") 02)--TableScan: default.default.test_unpartitioned_table projection=[id, name], partial_filters=[default.default.test_unpartitioned_table.name LIKE Utf8("Al%")] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: name@1 LIKE Al% -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,name] predicate:[name STARTS WITH "Al"] +01)FilterExec: name@1 LIKE Al% +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,name] predicate:[name STARTS WITH "Al"] # Test LIKE filtering with case-sensitive match query IT rowsort @@ -56,11 +55,10 @@ logical_plan 01)Filter: default.default.test_unpartitioned_table.name NOT LIKE Utf8("Al%") 02)--TableScan: default.default.test_unpartitioned_table projection=[id, name], partial_filters=[default.default.test_unpartitioned_table.name NOT LIKE Utf8("Al%")] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: name@1 NOT LIKE Al% -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,name] predicate:[name NOT STARTS WITH "Al"] +01)FilterExec: name@1 NOT LIKE Al% +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,name] predicate:[name NOT STARTS WITH "Al"] # Test NOT LIKE filtering query IT rowsort diff --git a/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt b/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt index 255731c046..3427625291 100644 --- a/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt +++ b/crates/sqllogictest/testdata/slts/df_test/timestamp_predicate_pushdown.slt @@ -48,11 +48,10 @@ logical_plan 01)Filter: default.default.test_timestamp_table.ts = TimestampNanosecond(1672921800000000000, None) 02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts = TimestampNanosecond(1672921800000000000, None)] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: ts@1 = 1672921800000000000 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,ts] predicate:[ts = 2023-01-05 12:30:00] +01)FilterExec: ts@1 = 1672921800000000000 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,ts] predicate:[ts = 2023-01-05 12:30:00] # Verify timestamp equality filtering works query I? @@ -68,11 +67,10 @@ logical_plan 01)Filter: default.default.test_timestamp_table.ts > TimestampNanosecond(1673308800000000000, None) 02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts > TimestampNanosecond(1673308800000000000, None)] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: ts@1 > 1673308800000000000 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-10 00:00:00] +01)FilterExec: ts@1 > 1673308800000000000 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-10 00:00:00] # Verify timestamp greater than filtering query I? rowsort @@ -99,11 +97,10 @@ logical_plan 01)Filter: default.default.test_timestamp_table.ts >= TimestampNanosecond(1672876800000000000, None) AND default.default.test_timestamp_table.ts <= TimestampNanosecond(1673827199000000000, None) 02)--TableScan: default.default.test_timestamp_table projection=[id, ts], partial_filters=[default.default.test_timestamp_table.ts >= TimestampNanosecond(1672876800000000000, None), default.default.test_timestamp_table.ts <= TimestampNanosecond(1673827199000000000, None)] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: ts@1 >= 1672876800000000000 AND ts@1 <= 1673827199000000000 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,ts] predicate:[(ts >= 2023-01-05 00:00:00) AND (ts <= 2023-01-15 23:59:59)] +01)FilterExec: ts@1 >= 1672876800000000000 AND ts@1 <= 1673827199000000000 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,ts] predicate:[(ts >= 2023-01-05 00:00:00) AND (ts <= 2023-01-15 23:59:59)] # Test timestamp range predicate filtering query I? rowsort @@ -166,11 +163,10 @@ logical_plan 01)Filter: default.default.test_timestamp_micros.ts > TimestampMicrosecond(1672531200000000, None) 02)--TableScan: default.default.test_timestamp_micros projection=[id, ts], partial_filters=[default.default.test_timestamp_micros.ts > TimestampMicrosecond(1672531200000000, None)] physical_plan -01)CoalesceBatchesExec: target_batch_size=8192 -02)--FilterExec: ts@1 > 1672531200000000 -03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 -04)------CooperativeExec -05)--------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-01 00:00:00] +01)FilterExec: ts@1 > 1672531200000000 +02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 +03)----CooperativeExec +04)------IcebergTableScan projection:[id,ts] predicate:[ts > 2023-01-01 00:00:00] query I? SELECT * FROM default.default.test_timestamp_micros WHERE ts > CAST('2023-01-01 00:00:00' AS TIMESTAMP) @@ -189,4 +185,3 @@ CREATE TABLE default.default.test_timestamp_millis (id INT NOT NULL, ts TIMESTAM # This should fail because Iceberg doesn't support second precision statement error DataFusion error: External error: DataInvalid => Unsupported Arrow data type: Timestamp\(s\) CREATE TABLE default.default.test_timestamp_seconds (id INT NOT NULL, ts TIMESTAMP(0)) - From 01a6a431d641202d6428f7d3af1e8ce27985e3a2 Mon Sep 17 00:00:00 2001 From: Junwang Zhao Date: Sat, 28 Feb 2026 02:53:04 +0800 Subject: [PATCH 226/891] chore: use ubuntu-slim for lightweight jobs (#2187) ubuntu-slim is a new cost-efficient runner to fit lightweight jobs. We can use this to save ASF infra usage (if possible). By *lightweight* I mean jobs that run for less than a minute and are not easily affected by other concurrently running jobs on the same host. This topic was raised in the Apache ORC project and later adopted by iceberg-cpp. We believe it could also benefit other projects under the Apache Iceberg umbrella. Refs: - https://github.blog/changelog/2025-10-28-1-vcpu-linux-runner-now-available-in-github-actions-in-public-preview/ - https://github.com/actions/runner-images/blob/main/images/ubuntu-slim/ubuntu-slim-Readme.md --- .github/workflows/bindings_python_ci.yml | 2 +- .github/workflows/ci_typos.yml | 2 +- .github/workflows/codeql.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/bindings_python_ci.yml b/.github/workflows/bindings_python_ci.yml index a78f27b0b1..78e1a9615c 100644 --- a/.github/workflows/bindings_python_ci.yml +++ b/.github/workflows/bindings_python_ci.yml @@ -56,7 +56,7 @@ jobs: run: cargo clippy --all-targets --all-features -- -D warnings check-python: - runs-on: ubuntu-latest + runs-on: ubuntu-slim steps: - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@v7 diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 38541bffbc..1dd42ed257 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -38,7 +38,7 @@ permissions: jobs: typos-check: name: typos check - runs-on: ubuntu-latest + runs-on: ubuntu-slim timeout-minutes: 10 env: FORCE_COLOR: 1 diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 7fef7973e5..17bfd8bf3d 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -33,7 +33,7 @@ permissions: jobs: analyze: name: Analyze Actions - runs-on: ubuntu-latest + runs-on: ubuntu-slim permissions: contents: read security-events: write From 647773ac0c9671735890b681aa32c70f6cd0f85e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 09:56:14 +0800 Subject: [PATCH 227/891] chore(deps): Bump tempfile from 3.24.0 to 3.25.0 (#2198) Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.24.0 to 3.25.0.
Changelog

Sourced from tempfile's changelog.

3.25.0

  • Allow getrandom 0.4.x while retaining support for getrandom 0.3.x.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tempfile&package-manager=cargo&previous-version=3.24.0&new-version=3.25.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index f3f32a2457..994bfff7ff 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -109,7 +109,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -120,7 +120,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -2408,7 +2408,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -2561,7 +2561,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -3753,7 +3753,7 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -4282,7 +4282,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.59.0", ] [[package]] @@ -5471,7 +5471,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -6452,15 +6452,15 @@ checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" [[package]] name = "tempfile" -version = "3.24.0" +version = "3.25.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "655da9c7eb6305c55742045d5a8d2037996d61d8de95806335c7c86ce0f82e9c" +checksum = "0136791f7c95b1f6dd99f9cc786b91bb81c3800b639b3478e561ddb7be95e5f1" dependencies = [ "fastrand", "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] From 12d1ec1c6c2a33c07ab2b5779708202474e43623 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 09:56:35 +0800 Subject: [PATCH 228/891] chore(deps): Bump fs-err from 3.2.2 to 3.3.0 (#2197) Bumps [fs-err](https://github.com/andrewhickman/fs-err) from 3.2.2 to 3.3.0.
Changelog

Sourced from fs-err's changelog.

fs-err Changelog

3.3.2

  • Cleanup rustdoc links to make them clickable in rust-analyzer (#88)
  • Add wrappers for several os::unix::fs functions (#87):
    • std::os::unix::fs::FileExt::read_exact_at
    • std::os::unix::fs::FileExt::write_all_at
    • std::os::unix::fs::chroot
    • std::os::unix::fs::chown
    • std::os::unix::fs::lchown
  • Replace '/' operator in SPDX license expression with 'OR' for readability (#86)
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fs-err&package-manager=cargo&previous-version=3.2.2&new-version=3.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 994bfff7ff..ef6a8133fb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2738,9 +2738,9 @@ checksum = "28dd6caf6059519a65843af8fe2a3ae298b14b80179855aeb4adc2c1934ee619" [[package]] name = "fs-err" -version = "3.2.2" +version = "3.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "baf68cef89750956493a66a10f512b9e58d9db21f2a573c079c0bdf1207a54a7" +checksum = "73fde052dbfc920003cfd2c8e2c6e6d4cc7c1091538c3a24226cec0665ab08c0" dependencies = [ "autocfg", ] From c47ae699a4c433c64005dba2ac2269d53909b4be Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 09:57:28 +0800 Subject: [PATCH 229/891] chore(deps): Bump crate-ci/typos from 1.43.5 to 1.44.0 (#2190) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.43.5 to 1.44.0.
Release notes

Sourced from crate-ci/typos's releases.

v1.44.0

[1.44.0] - 2026-02-27

Features

Changelog

Sourced from crate-ci/typos's changelog.

[1.44.0] - 2026-02-27

Features

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.43.5&new-version=1.44.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci_typos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_typos.yml b/.github/workflows/ci_typos.yml index 1dd42ed257..8031cd8ca9 100644 --- a/.github/workflows/ci_typos.yml +++ b/.github/workflows/ci_typos.yml @@ -45,4 +45,4 @@ jobs: steps: - uses: actions/checkout@v6 - name: Check typos - uses: crate-ci/typos@v1.43.5 + uses: crate-ci/typos@v1.44.0 From df239a814aa9e6a5b6d0f30a75ccd21a55b45112 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 09:57:55 +0800 Subject: [PATCH 230/891] chore(deps): Bump actions/download-artifact from 7 to 8 (#2191) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8.
Release notes

Sourced from actions/download-artifact's releases.

v8.0.0

v8 - What's new

Direct downloads

To support direct uploads in actions/upload-artifact, the action will no longer attempt to unzip all downloaded files. Instead, the action checks the Content-Type header ahead of unzipping and skips non-zipped files. Callers wishing to download a zipped file as-is can also set the new skip-decompress parameter to false.

Enforced checks (breaking)

A previous release introduced digest checks on the download. If a download hash didn't match the expected hash from the server, the action would log a warning. Callers can now configure the behavior on mismatch with the digest-mismatch parameter. To be secure by default, we are now defaulting the behavior to error which will fail the workflow run.

ESM

To support new versions of the @actions/* packages, we've upgraded the package to ESM.

What's Changed

Full Changelog: https://github.com/actions/download-artifact/compare/v7...v8.0.0

Commits
  • 70fc10c Merge pull request #461 from actions/danwkennedy/digest-mismatch-behavior
  • f258da9 Add change docs
  • ccc058e Fix linting issues
  • bd7976b Add a setting to specify what to do on hash mismatch and default it to error
  • ac21fcf Merge pull request #460 from actions/danwkennedy/download-no-unzip
  • 15999bf Add note about package bumps
  • 974686e Bump the version to v8 and add release notes
  • fbe48b1 Update test names to make it clearer what they do
  • 96bf374 One more test fix
  • b8c4819 Fix skip decompress test
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/download-artifact&package-manager=github_actions&previous-version=7&new-version=8)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release_python.yml | 2 +- .github/workflows/release_python_nightly.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index f4cf4a761a..a241a2315f 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -212,7 +212,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: pattern: wheels-* merge-multiple: true diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index 32614b2d0e..f1d312fca7 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -122,7 +122,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: pattern: wheels-* merge-multiple: true From ed6b6956ceac1faa99e253388c6d7c5d40fe1eb6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 10:01:30 +0800 Subject: [PATCH 231/891] chore(deps): Bump uuid from 1.19.0 to 1.21.0 (#2193) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.19.0 to 1.21.0.
Release notes

Sourced from uuid's releases.

v1.21.0

What's Changed

New Contributors

Full Changelog: https://github.com/uuid-rs/uuid/compare/v1.20.0...v1.21.0

v1.20.0

What's Changed

New Contributors

Full Changelog: https://github.com/uuid-rs/uuid/compare/v1.19.0...v1.20.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=uuid&package-manager=cargo&previous-version=1.19.0&new-version=1.21.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 166 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index ef6a8133fb..cd55121eb1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2888,6 +2888,19 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "getrandom" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec" +dependencies = [ + "cfg-if", + "libc", + "r-efi", + "wasip2", + "wasip3", +] + [[package]] name = "glob" version = "0.3.3" @@ -3597,6 +3610,12 @@ dependencies = [ "zerovec", ] +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + [[package]] name = "ident_case" version = "1.0.1" @@ -3826,6 +3845,12 @@ dependencies = [ "spin", ] +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + [[package]] name = "lexical-core" version = "1.0.6" @@ -7000,6 +7025,12 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + [[package]] name = "unit-prefix" version = "0.5.2" @@ -7050,11 +7081,11 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.19.0" +version = "1.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a" +checksum = "b672338555252d43fd2240c714dc444b8c6fb0a5c5335e65a07bba7742735ddb" dependencies = [ - "getrandom 0.3.4", + "getrandom 0.4.1", "js-sys", "serde_core", "wasm-bindgen", @@ -7171,7 +7202,16 @@ version = "1.0.1+wasi-0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.46.0", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", ] [[package]] @@ -7238,6 +7278,28 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap 2.12.1", + "wasm-encoder", + "wasmparser", +] + [[package]] name = "wasm-streams" version = "0.4.2" @@ -7251,6 +7313,18 @@ dependencies = [ "web-sys", ] +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap 2.12.1", + "semver", +] + [[package]] name = "web-sys" version = "0.3.83" @@ -7613,6 +7687,94 @@ version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap 2.12.1", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap 2.12.1", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap 2.12.1", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + [[package]] name = "writeable" version = "0.6.2" From ce00c9cd3aec655646bc97be8fc094083b5dcf99 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 10:03:43 +0800 Subject: [PATCH 232/891] chore(deps): Bump serde_with from 3.16.1 to 3.17.0 (#2194) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [serde_with](https://github.com/jonasbb/serde_with) from 3.16.1 to 3.17.0.
Release notes

Sourced from serde_with's releases.

serde_with v3.17.0

Added

  • Support OneOrMany with smallvec v1 (#920, #922)

Changed

  • Switch to yaml_serde for a maintained yaml dependency by @​kazan417 (#921)
  • Bump MSRV to 1.82, since that is required for yaml_serde dev-dependency.
Commits
  • 4031878 Bump version to v3.17.0 (#924)
  • 204ae56 Bump version to v3.17.0
  • 7812b5a serde_yaml 0.9 to yaml_serde 0.10 (#921)
  • 614bd89 Bump MSRV to 1.82 as required by yaml_serde
  • 518d0ed Suppress RUSTSEC-2026-0009 since we don't have untrusted time input in tests ...
  • a6579a8 Suppress RUSTSEC-2026-0009 since we don't have untrusted time input in tests
  • 9d4d069 Implement OneOrMany for smallvec_1::SmallVec (#922)
  • fc78243 Add changelog
  • 2b8c30b Implement OneOrMany for smallvec_1::SmallVec
  • 2d9b9a1 Carg.lock update
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=serde_with&package-manager=cargo&previous-version=3.16.1&new-version=3.17.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index cd55121eb1..4ddbe554e4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5873,9 +5873,9 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.16.1" +version = "3.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fa237f2807440d238e0364a218270b98f767a00d3dada77b1c53ae88940e2e7" +checksum = "381b283ce7bc6b476d903296fb59d0d36633652b633b27f64db4fb46dcbfc3b9" dependencies = [ "base64", "chrono", @@ -5892,9 +5892,9 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.16.1" +version = "3.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52a8e3ca0ca629121f70ab50f95249e5a6f925cc0f6ffe8256c45b728875706c" +checksum = "a6d4e30573c8cb306ed6ab1dca8423eec9a463ea0e155f45399455e0368b27e0" dependencies = [ "darling 0.21.3", "proc-macro2", From f8843979d220b38cb686f70b674109e72b9c03a3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 10:28:38 +0800 Subject: [PATCH 233/891] chore(deps): Bump actions/upload-artifact from 6 to 7 (#2189) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7.
Release notes

Sourced from actions/upload-artifact's releases.

v7.0.0

v7 What's new

Direct Uploads

Adds support for uploading single files directly (unzipped). Callers can set the new archive parameter to false to skip zipping the file during upload. Right now, we only support single files. The action will fail if the glob passed resolves to multiple files. The name parameter is also ignored with this setting. Instead, the name of the artifact will be the name of the uploaded file.

ESM

To support new versions of the @actions/* packages, we've upgraded the package to ESM.

What's Changed

New Contributors

Full Changelog: https://github.com/actions/upload-artifact/compare/v6...v7.0.0

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/upload-artifact&package-manager=github_actions&previous-version=6&new-version=7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release_python.yml | 4 ++-- .github/workflows/release_python_nightly.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release_python.yml b/.github/workflows/release_python.yml index a241a2315f..5d97e23a2e 100644 --- a/.github/workflows/release_python.yml +++ b/.github/workflows/release_python.yml @@ -136,7 +136,7 @@ jobs: command: sdist args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: wheels-sdist path: bindings/python/dist @@ -193,7 +193,7 @@ jobs: command: build args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist diff --git a/.github/workflows/release_python_nightly.yml b/.github/workflows/release_python_nightly.yml index f1d312fca7..6f9655e29c 100644 --- a/.github/workflows/release_python_nightly.yml +++ b/.github/workflows/release_python_nightly.yml @@ -53,7 +53,7 @@ jobs: args: -o dist - name: Upload sdist - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: wheels-sdist path: bindings/python/dist @@ -104,7 +104,7 @@ jobs: args: --release -o dist - name: Upload wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: name: wheels-${{ matrix.os }}-${{ matrix.target }} path: bindings/python/dist From 3974fe901c1feb759b369038d1e56008e842c25d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 10:28:59 +0800 Subject: [PATCH 234/891] chore(deps): Bump aws-sdk-glue from 1.133.0 to 1.134.0 (#2196) Bumps [aws-sdk-glue](https://github.com/awslabs/aws-sdk-rust) from 1.133.0 to 1.134.0.
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aws-sdk-glue&package-manager=cargo&previous-version=1.133.0&new-version=1.134.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- Cargo.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 4ddbe554e4..24d157bd79 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -109,7 +109,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -120,7 +120,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -593,9 +593,9 @@ dependencies = [ [[package]] name = "aws-sdk-glue" -version = "1.133.0" +version = "1.134.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46e5b45757688d5c99ef0ab38d064fefcc46ee7923fb6d7b23b035323e300c7d" +checksum = "ef38f029c1e39b96e47df528c90c3c8965e0cae883c43d7e0232f3444f6cd251" dependencies = [ "aws-credential-types", "aws-runtime", @@ -2561,7 +2561,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.52.0", + "windows-sys 0.59.0", ] [[package]] @@ -3772,7 +3772,7 @@ dependencies = [ "portable-atomic", "portable-atomic-util", "serde_core", - "windows-sys 0.52.0", + "windows-sys 0.59.0", ] [[package]] @@ -5091,7 +5091,7 @@ dependencies = [ "once_cell", "socket2 0.6.1", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.59.0", ] [[package]] @@ -5496,7 +5496,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.52.0", + "windows-sys 0.59.0", ] [[package]] @@ -6485,7 +6485,7 @@ dependencies = [ "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.52.0", + "windows-sys 0.59.0", ] [[package]] From 95ec8484f85dcfbc9360f689a0e2fccdb28a0bcc Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 3 Mar 2026 10:30:48 +0800 Subject: [PATCH 235/891] chore(deps): Bump the arrow-parquet group with 7 updates (#2192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps the arrow-parquet group with 7 updates: | Package | From | To | | --- | --- | --- | | [arrow-array](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-buffer](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-ord](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-schema](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-select](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [arrow-string](https://github.com/apache/arrow-rs) | `57.2.0` | `57.3.0` | | [parquet](https://github.com/apache/arrow-rs) | `57.1.0` | `57.2.0` | Updates `arrow-array` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-array's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-array's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-buffer` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-buffer's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-buffer's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-ord` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-ord's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-ord's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-schema` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-schema's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-schema's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-select` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-select's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-select's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `arrow-string` from 57.2.0 to 57.3.0
Release notes

Sourced from arrow-string's releases.

arrow 57.3.0

Changelog

57.3.0 (2026-02-02)

Full Changelog

Breaking changes:

Fixed bugs:

Changelog

Sourced from arrow-string's changelog.

Changelog

58.0.0 (2026-02-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Avoid allocating a Vec in StructBuilder #9427
  • Zstd context reuse #9401
  • Optimize from_bitwise_unary_op #9364
  • Support RunEndEncoded in ord comparator #9360
  • Support RunEndEncoded arrays in arrow-json #9359
  • Support BinaryView in bit_length kernel #9351
  • Remove support for List types in bit_length kernel #9349
  • Support roundtrip ListView in parquet arrow writer #9344
  • Support ListView in length kernel #9343
  • Support ListView in sort kernel #9341
  • Add some way to create a Timestamp from a DateTime #9337
  • Introduce DataType::is_list and DataType::IsBinary #9326
  • Performance of creating all null dictionary array can be improved #9321

... (truncated)

Commits
  • 7505005 [57_maintenance] Update version to 57.3.0, add changelog (#9333)
  • 6bbfb99 [maintenance_57] Fix string array equality when the values buffer is the same...
  • 505eb8e [57_maintenance] Revert "Seal Array trait (#9092)", mark Array as unsafe ...
  • 74cf914 [57_maintenance] Mark BufferBuilder::new_from_buffer as unsafe (#9292) (#9312)
  • 25cc1ac [57_maintenance] fix: ensure BufferBuilder::truncate doesn't overset length...
  • 9fc2fbb [57_maintenance[Parquet] Provide only encrypted column stats in plaintext foo...
  • 3df3157 [57_maintenance] [regression] Error with adaptive predicate pushdown: "Invali...
  • See full diff in compare view

Updates `parquet` from 57.1.0 to 57.2.0
Release notes

Sourced from parquet's releases.

arrow 57.2.0

Changelog

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946

... (truncated)

Changelog

Sourced from parquet's changelog.

57.2.0 (2026-01-07)

Full Changelog

Breaking changes:

Implemented enhancements:

  • [parquet] further relax LevelInfoBuilder::types_compatible for ArrowWriter #9098
  • Update arrow-row documentation with Union encoding #9084
  • Add code examples for min and max compute functions #9055
  • Add append_n to bytes view builder API #9034 [arrow]
  • Move RunArray::get_physical_indices to RunEndBuffer #9025 [arrow]
  • Allow quote style in csv writer #9003 [arrow]
  • IPC support for ListView #9002 [arrow]
  • Implement BinaryArrayType for &FixedSizeBinaryArrays #8992 [arrow]
  • arrow-buffer: implement num-traits for i256 #8976 [arrow]
  • Support for Arc<str> in ParquetRecordWriter derive macro #8972
  • [arrow-avro] suggest switching from xz to liblzma #8970 [arrow]
  • arrow-buffer: add i256::trailing_zeros #8968 [arrow]
  • arrow-buffer: make i256::leading_zeros public #8965 [arrow]
  • Add spark like ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options to the csv writer #8961 [arrow]
  • Add round trip benchmark for Parquet writer/reader #8955 [parquet]
  • Support performant interleave for List/LargeList #8952 [arrow]
  • [Variant] Support array access when parsing VariantPath #8946
  • Some panic!s could be represented as unimplemented!s #8932 [arrow]
  • [Variant] easier way to construct a shredded schema #8922
  • Support DataType::ListView and DataType::LargeListView in ArrayData::new_null #8908 [arrow]
  • Add GenericListViewArray::from_iter_primitive #8906 [arrow]
  • [Variant] Unify the cast option usage in ParquentVariant #8873
  • Blog post about efficient filter representation in Parquet filter pushdown #8843 [parquet]
  • Add comparison support for Union arrays in the cmp kernel #8837 [arrow]
  • [Variant] Support array shredding into List/LargeList/ListView/LargeListView #8830
  • Support Union data types for row format #8828 [arrow]
  • FFI support for ListView #8819 [arrow]
  • [Variant] Support more Arrow Datatypes from Variant primitive types #8805
  • FixedSizeBinaryBuilder supports append_array #8750 [arrow]
  • Implement special case zip with scalar for Utf8View #8724 [arrow]
  • [geometry] Wire up arrow reader/writer for GEOMETRY and GEOGRAPHY #8717 [parquet]

Fixed bugs:

  • Soundness Bug in try_binary when Array is implemented incorrectly in external crate #9106
  • casting Dict(_, LargeUtf8) to Utf8View (StringViewArray) panics #9101
  • wrong results for null count of nullif kernel #9085 [parquet] [arrow]
  • Empty first line in some code examples #9063
  • GenericByteViewArray::slice is not zero-copy but ought to be #9014

... (truncated)

Commits