Skip to content

Commit 0d2c72b

Browse files
authored
chore(deps): bump DataFusion to 54.1.0 (#114)
## Which issue does this PR close? N/A — routine dependency bump; no tracking issue was filed. ## Rationale for this change Keeps the binding current with upstream DataFusion. 54.1.0 is the latest release line, and staying close to it keeps the next bump small and makes upstream fixes available to Java callers. ## What changes are included in this PR? Bumps `datafusion`, `datafusion-proto`, `datafusion-spark` and `datafusion-substrait` from 53.1.0 to 54.1.0. The `datafusion.version` Maven property moves in lock step, since it selects the upstream tag the `datafusion.proto` / `datafusion_common.proto` definitions are downloaded from — the generated Java protobuf classes must match what `datafusion-proto` 54.1.0 decodes. The pinned sha512 digests for both protos are updated accordingly; each was verified to match the copy vendored in the published `datafusion-proto` / `datafusion-proto-common` 54.1.0 crates, independently of the GitHub download. `arrow` (58) and `object_store` (0.13) are unchanged — 54.1.0 resolves to the same majors, so the `object_store` pin comment still holds. Adapting to the upstream API changes the bump requires: - `TableProvider`, `ExecutionPlan` and `ScalarUDFImpl` now take `Any` as a supertrait, so the manual `as_any` overrides are no longer trait members and are removed. - `MemoryPool` gained a `name()` method and a `Display` supertrait. `TrackingMemoryPool` implements both, following upstream's wrapper convention: name the wrapper, add the counters it exists to expose, and defer to the inner pool for the usage detail. - `CacheManagerConfig::table_files_statistics_cache` is renamed to `file_statistics_cache`, and the accompanying limit is now the on/off switch — `CacheManager::try_new` installs a default statistics cache whenever `file_statistics_cache_limit > 0`, even when the cache slot is `None`, and the default limit is non-zero. An explicit `fileStatisticsCache(false)` from the Java surface therefore has to zero the limit as well; otherwise upstream would install the very cache the caller asked us to skip. This is the one place the bump would have silently changed observable Java behavior. - The `cache_unit` module is gone; the default cache impls now live in `cache::file_statistics_cache` and `cache`. - `DataFusionError::AvroError` is gone — DataFusion 54 reads Avro through `arrow-avro` rather than `apache-avro`. Avro decode failures now arrive as `ArrowError::AvroError`, which the exception classifier already routed to `ExecutionException` alongside the `CsvError` / `JsonError` decoder variants, so the mapping stays coherent. The dead arm is dropped, along with the `avro` feature on `datafusion-jni-common` that existed only to gate it. One test fixture is also corrected. `SessionContextSubstraitTest` built plans whose base schema declared both columns `NULLABILITY_REQUIRED`, while the tests register a CSV — whose inferred schema is always nullable. DataFusion 54's Substrait consumer now validates that a field a plan declares non-nullable really is non-nullable in the table, and rejects the mismatch. That check is correct: a plan built around a "never null" assumption must not run against data that can contain nulls. The fixture is fixed to declare nullable columns; it only passed before because 53 did not check. ## Are these changes tested? Covered by the existing suites — this is a dependency bump, so the value is in the current tests continuing to pass against the new version rather than in new assertions. - `./mvnw test` — 349 tests, 0 failures. Run with the `substrait` Cargo feature enabled (`cargo build -p datafusion-jni --features substrait`) so the Substrait tests execute rather than skip. - `cargo test --workspace` — all passing. - `cargo clippy --workspace --all-targets -- -D warnings` — clean. - `cargo fmt --all -- --check` and `./mvnw -q spotless:check` — clean. - `cargo build --workspace --all-features` with `RUSTFLAGS="--cfg tokio_unstable"`, to cover the optional `substrait` and `runtime-metrics` features that the default build does not compile. ## Are there any user-facing changes? No API changes. Two behavioral notes, both inherited from upstream: - Avro decode failures now surface as `ExecutionException` rather than `IoException`, following the move to `arrow-avro`. The `IoException` javadoc is updated to match. - DataFusion 54 enables the file statistics and list files caches by default. Callers who never configured `CacheManagerOptions` pick up upstream's new defaults; an explicit `fileStatisticsCache(false)` continues to disable the cache, as described above.
1 parent ab57cb9 commit 0d2c72b

14 files changed

Lines changed: 301 additions & 309 deletions

File tree

‎Cargo.lock‎

Lines changed: 233 additions & 257 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,13 @@ repository = "https://github.com/apache/datafusion-java"
3939
[workspace.dependencies]
4040
arrow = { version = "58", features = ["ffi"] }
4141
async-trait = "0.1"
42-
datafusion = { version = "53.1.0" }
43-
datafusion-proto = "53.1.0"
44-
datafusion-spark = "53.1.0"
45-
datafusion-substrait = "53.1.0"
42+
datafusion = { version = "54.1.0" }
43+
datafusion-proto = "54.1.0"
44+
datafusion-spark = "54.1.0"
45+
datafusion-substrait = "54.1.0"
4646
futures = "0.3"
4747
jni = "0.21"
48-
# Pinned to the major DataFusion 53.1 pulls in transitively (0.13.x) so we
48+
# Pinned to the major DataFusion 54.1 pulls in transitively (0.13.x) so we
4949
# share the same `dyn ObjectStore` vtable and don't double-link.
5050
object_store = { version = "0.13", default-features = false }
5151
prost = "0.14"

‎core/pom.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ under the License.
129129
<url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto-common/proto/datafusion_common.proto</url>
130130
<outputDirectory>${project.build.directory}/proto/datafusion/proto-common/proto</outputDirectory>
131131
<outputFileName>datafusion_common.proto</outputFileName>
132-
<sha512>d6f3368372ea277cc23e26f196994b81616d38599357bb374cbd7eb1760e649a789e4c133d86a395ac701049a500348da2ec039d3f978ac5d8112c2876dded1f</sha512>
132+
<sha512>e6a20115badec14f641b9237b0faa3f553162992444208322558ea3fcfaa24ca15334e4945827259dcb7c1de7f9aed659d496974eb8eeb1cfcf54ad4d7f6d20f</sha512>
133133
</configuration>
134134
</execution>
135135
<execution>
@@ -140,7 +140,7 @@ under the License.
140140
<url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto/proto/datafusion.proto</url>
141141
<outputDirectory>${project.build.directory}/proto/datafusion/proto/proto</outputDirectory>
142142
<outputFileName>datafusion.proto</outputFileName>
143-
<sha512>c3d162b8e2a418e03f74caceaccfd934af89bb95a12ede13d4cc1701d24c734d74b1e96372142b173db05938dab7f965ad60d476363308c441677a63ea5fbcf7</sha512>
143+
<sha512>bdae5361d32b4d4f0418677ce650ee8cc212fddfe9390b2c033ee4ab3bb3c969a6148b7b318b9bb268ee9384f18d4254ebe6e0701f15639059a8f2800d6e7847</sha512>
144144
</configuration>
145145
</execution>
146146
</executions>

‎core/src/main/java/org/apache/datafusion/IoException.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,8 @@
2121

2222
/**
2323
* IO-shaped failure: a local filesystem read failed, an object store request failed, or a parquet /
24-
* arrow / avro decoder reported a malformed file. Surfaces upstream {@code
25-
* DataFusionError::IoError}, {@code ObjectStore}, {@code ArrowError}, {@code ParquetError}, and
26-
* {@code AvroError}.
24+
* arrow decoder reported a malformed file. Surfaces upstream {@code DataFusionError::IoError},
25+
* {@code ObjectStore}, {@code ParquetError}, and the IO-shaped {@code ArrowError} variants.
2726
*
2827
* <p>Note: this is {@code org.apache.datafusion.IoException} (lowercase {@code o}), distinct from
2928
* {@code java.io.IOException}. The {@code IoException} spelling matches the orthography of the

‎core/src/test/java/org/apache/datafusion/DataFrameIntrospectionTest.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ void describeReturnsSummaryStats() throws Exception {
195195
}
196196
}
197197
}
198-
// DataFusion 53.1 reports these seven labels.
198+
// DataFusion 54.1 reports these seven labels.
199199
assertTrue(seen.contains("count"), () -> "labels=" + seen);
200200
assertTrue(seen.contains("null_count"), () -> "labels=" + seen);
201201
assertTrue(seen.contains("mean"), () -> "labels=" + seen);

‎core/src/test/java/org/apache/datafusion/SessionContextSubstraitTest.java‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,14 @@ static void checkFeatureEnabled() {
7373

7474
/**
7575
* Build a minimal Substrait {@code Plan} that scans a registered named table {@code tableName}
76-
* with two columns {@code (id int32, v int32)} and projects them through unchanged.
76+
* with two columns {@code (id int64, v int64)} and projects them through unchanged.
77+
*
78+
* <p>The columns are declared {@code NULLABILITY_NULLABLE} to match the schema DataFusion infers
79+
* for the CSV these tests register — CSV inference always yields nullable fields. DataFusion's
80+
* Substrait consumer verifies that a field the plan declares non-nullable really is non-nullable
81+
* in the table, and rejects the plan otherwise: a plan built around a "never null" assumption
82+
* must not silently run against data that can contain nulls. The enclosing struct stays {@code
83+
* REQUIRED} — the row itself is always present.
7784
*/
7885
private static Plan namedTableScanPlan(String tableName) {
7986
NamedStruct schema =
@@ -87,13 +94,13 @@ private static Plan namedTableScanPlan(String tableName) {
8794
.setI64(
8895
Type.I64
8996
.newBuilder()
90-
.setNullability(Type.Nullability.NULLABILITY_REQUIRED)))
97+
.setNullability(Type.Nullability.NULLABILITY_NULLABLE)))
9198
.addTypes(
9299
Type.newBuilder()
93100
.setI64(
94101
Type.I64
95102
.newBuilder()
96-
.setNullability(Type.Nullability.NULLABILITY_REQUIRED)))
103+
.setNullability(Type.Nullability.NULLABILITY_NULLABLE)))
97104
.setNullability(Type.Nullability.NULLABILITY_REQUIRED))
98105
.build();
99106
ReadRel read =

‎native-common/Cargo.toml‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,6 @@ publish = false
2727
readme = "README.md"
2828
description = "Shared JNI plumbing for DataFusion Java native crates: error-to-exception mapping, the per-cdylib Tokio runtime singleton, and the async-stream-to-FFI_ArrowArrayStream bridge."
2929

30-
[features]
31-
# `datafusion-jni` builds DataFusion with `avro`, which adds the
32-
# `DataFusionError::AvroError` variant our classifier maps to IoException.
33-
# Feature-forwarded so consumers that don't read Avro (the Spark helper)
34-
# don't pull the apache-avro stack into their cdylib.
35-
avro = ["datafusion/avro"]
36-
3730
[dependencies]
3831
datafusion = { workspace = true }
3932
futures = { workspace = true }

‎native-common/src/errors.rs‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -97,10 +97,6 @@ fn classify(err: &DataFusionError) -> &'static str {
9797
DataFusionError::IoError(_)
9898
| DataFusionError::ObjectStore(_)
9999
| DataFusionError::ParquetError(_) => "org/apache/datafusion/IoException",
100-
// The AvroError variant only exists when DataFusion is built with its
101-
// `avro` feature, forwarded by this crate's own `avro` feature.
102-
#[cfg(feature = "avro")]
103-
DataFusionError::AvroError(_) => "org/apache/datafusion/IoException",
104100
// ArrowError is a 21-variant grab bag -- only some of those variants
105101
// are actually IO-shaped. DivideByZero / ArithmeticOverflow / Compute
106102
// / Cast / InvalidArgument / Memory etc. are execution-time failures
@@ -125,9 +121,15 @@ fn classify(err: &DataFusionError) -> &'static str {
125121
/// Map an [`ArrowError`] variant onto the Java exception class to throw.
126122
/// Only the genuinely IO-shaped variants (`IoError`, `IpcError`) land on
127123
/// `IoException`; everything else is execution-time and routes through
128-
/// `ExecutionException`. Schema/parse-shaped variants route through
129-
/// `PlanException` so a malformed IPC schema or a parse error surfaces as a
130-
/// query problem rather than an execution failure.
124+
/// `ExecutionException` -- including `AvroError`, which is where Avro decode
125+
/// failures land now that DataFusion 54 reads Avro through `arrow-avro`
126+
/// instead of `apache-avro` (the old `DataFusionError::AvroError` variant is
127+
/// gone). That puts Avro alongside the `CsvError` / `JsonError` decoder
128+
/// variants, which this arm already classified as execution failures.
129+
///
130+
/// Schema/parse-shaped variants route through `PlanException` so a malformed
131+
/// IPC schema or a parse error surfaces as a query problem rather than an
132+
/// execution failure.
131133
///
132134
/// Variants without a clean caller-facing category (`CDataInterface`, the
133135
/// various overflow/index-overflow markers) fall through to the parent.

‎native/Cargo.toml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,8 @@ arrow = { workspace = true }
8282
async-trait = { workspace = true }
8383
datafusion = { workspace = true, features = ["avro"] }
8484
# Shared JNI plumbing (error->exception mapping, runtime singleton,
85-
# StreamingReader). `avro` keeps the classifier's AvroError->IoException arm
86-
# in sync with the `avro` feature on `datafusion` above.
87-
datafusion-jni-common = { path = "../native-common", features = ["avro"] }
85+
# StreamingReader).
86+
datafusion-jni-common = { path = "../native-common" }
8887
datafusion-proto = { workspace = true }
8988
# Apache Spark-compatible functions + expression planners. Optional and
9089
# gated behind the `spark` feature (in the default set). The `core` feature

‎native/src/cache_manager.rs‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,8 @@ use std::sync::Arc;
2929
use std::time::Duration;
3030

3131
use datafusion::execution::cache::cache_manager::CacheManagerConfig;
32-
use datafusion::execution::cache::cache_unit::{
33-
DefaultFileStatisticsCache, DefaultFilesMetadataCache,
34-
};
35-
use datafusion::execution::cache::DefaultListFilesCache;
32+
use datafusion::execution::cache::file_statistics_cache::DefaultFileStatisticsCache;
33+
use datafusion::execution::cache::{DefaultFilesMetadataCache, DefaultListFilesCache};
3634

3735
use crate::proto_gen::CacheManagerOptionsProto;
3836
use datafusion_jni_common::errors::JniResult;
@@ -76,8 +74,20 @@ pub(crate) fn build_config(
7674
config.list_files_cache_ttl = ttl;
7775
}
7876

79-
if opts.file_statistics_cache_enabled.unwrap_or(false) {
80-
config.table_files_statistics_cache = Some(Arc::new(DefaultFileStatisticsCache::default()));
77+
// DataFusion 54 renamed `table_files_statistics_cache` to
78+
// `file_statistics_cache` and made the *limit* the on/off switch:
79+
// `CacheManager::try_new` now builds a `DefaultFileStatisticsCache`
80+
// whenever `file_statistics_cache_limit > 0`, even when the cache slot is
81+
// `None`. The default limit is non-zero, so an explicit `false` from the
82+
// Java surface has to zero the limit -- otherwise upstream would install a
83+
// stats cache the caller just asked us not to.
84+
if let Some(enabled) = opts.file_statistics_cache_enabled {
85+
if enabled {
86+
config.file_statistics_cache = Some(Arc::new(DefaultFileStatisticsCache::default()));
87+
} else {
88+
config.file_statistics_cache = None;
89+
config.file_statistics_cache_limit = 0;
90+
}
8191
}
8292

8393
Ok(Some(config))

0 commit comments

Comments
 (0)