11# Apache DataFusion Java
22
3- Java bindings for [ Apache DataFusion] , the Rust-based query engine. SQL queries
4- run in native code and results are returned to the JVM as [ Apache Arrow] record
5- batches via the Arrow C Data Interface — no per-row JNI calls, no row-by-row
6- copies.
3+ Java bindings for [ Apache DataFusion] . Queries run in native Rust and results
4+ return to the JVM as [ Apache Arrow] batches via the Arrow C Data Interface.
75
86[ Apache DataFusion ] : https://datafusion.apache.org/
97[ Apache Arrow ] : https://arrow.apache.org/
108
11- > ** Project status: early development.** This is a brand-new project. The API
12- > is small, will change without notice, and there is no published release. Do
13- > not depend on it from production code yet. Bug reports, design feedback, and
14- > contributions are very welcome.
9+ > Early development: no releases yet, API will change. Bug reports and
10+ > contributions welcome.
1511
1612## Quickstart
1713
@@ -32,97 +28,55 @@ try (var allocator = new RootAllocator();
3228 ArrowReader reader = df. collect(allocator)) {
3329 while (reader. loadNextBatch()) {
3430 var batch = reader. getVectorSchemaRoot();
35- // ... consume batch ...
31+ // ...
3632 }
3733 }
3834}
3935```
4036
41- The current public surface mirrors a small slice of the Rust DataFusion API:
37+ ` SessionContext ` and ` DataFrame ` are ` AutoCloseable ` and not thread-safe.
4238
43- - ` SessionContext.sql(String query) ` — parse and plan a SQL query, returning a
44- lazy ` DataFrame ` . No execution happens yet.
45- - ` DataFrame.collect(BufferAllocator allocator) ` — execute the plan and return
46- the result batches as an ` ArrowReader ` . Consumes the DataFrame; the caller
47- closes the reader, and the allocator must outlive it.
48- - ` SessionContext.registerParquet(String name, String path) ` — register a local
49- Parquet file as a SQL table.
39+ ## Project status
5040
51- Both ` SessionContext ` and ` DataFrame ` are ` AutoCloseable ` and ** not
52- thread-safe** .
41+ Query interfaces:
5342
54- ## Prerequisites
43+ - [x] SQL: ` SessionContext.sql(String) `
44+ - [x] DataFrame: ` select ` , ` filter ` (other transformations TBD)
45+ - [x] DataFusion-Proto ` LogicalPlanNode ` : ` SessionContext.fromProto(byte[]) ` .
46+ The ` datafusion-proto ` Java classes are generated by the build.
5547
56- - JDK 17 or newer
57- - Rust toolchain (stable, installed via [ rustup] )
58- - [ ` tpchgen-cli ` ] — only needed to generate test data for the Parquet
59- integration test (` cargo install tpchgen-cli ` )
48+ Data sources:
6049
61- Maven is bundled via the ` ./mvnw ` wrapper; no separate Maven install required.
50+ - [x] Parquet via ` registerParquet ` / ` readParquet ` , with ` ParquetReadOptions `
51+ - [ ] CSV, JSON, Avro
52+ - [ ] Custom catalog and table providers
6253
63- [ rustup ] : https://rustup.rs/
64- [ `tpchgen-cli` ] : https://github.com/clflushopt/tpchgen-rs
54+ Results:
6555
66- ## Build & test
56+ - [x] ` DataFrame.collect(allocator) ` — Arrow C Data Interface stream
57+ - [x] ` DataFrame.count() ` , ` show() ` , ` show(int) `
58+ - [x] ` SessionContext.tableSchema(String) `
6759
68- make test
60+ Not yet:
6961
70- This builds the native Rust crate and runs the JUnit tests. The steps can be
71- run individually:
62+ - [ ] ` SessionConfig ` / ` RuntimeEnv ` knobs
63+ - [ ] Java UDFs
64+ - [ ] ` write_* ` outputs
7265
73- cd native && cargo build
74- ./mvnw test
66+ ## Requirements
7567
76- The native library must be built before running JVM tests .
68+ JDK 17+. Building from source: see [ CONTRIBUTING.md ] ( CONTRIBUTING.md ) .
7769
78- ## Test data
79-
80- The Parquet integration test reads TPC-H SF1 data (~ 345 MB across 8 tables in
81- Snappy-compressed Parquet). Generate it once with:
82-
83- make tpch-data
84-
85- Tests that need this data skip cleanly if it is missing. ` make clean ` does
86- ** not** remove ` tpch-data/ ` — delete it manually to reclaim the disk space.
87-
88- ## Repository layout
70+ ## Layout
8971
9072- ` src/ ` — Java sources and tests
91- - ` native/ ` — Rust crate that exposes DataFusion over JNI and the Arrow C Data
92- Interface
93-
94- ## Roadmap
95-
96- Near-term priorities, roughly in order:
97-
98- - ** Session configuration.** Expose ` SessionConfig ` and ` RuntimeEnv ` settings
99- (target partitions, batch size, memory pool, default catalog, …) so callers
100- can tune execution from the JVM.
101- - ** Full ` SessionContext ` and ` DataFrame ` APIs.** Expand beyond ` sql ` and
102- ` registerParquet ` to mirror the Rust API: table registration variants,
103- ` read_* ` / ` write_* ` entry points, and DataFrame transformations such as
104- ` select ` , ` filter ` , ` join ` , ` aggregate ` , ` sort ` , plus result-materialization
105- variants (` show ` , ` count ` , streaming collection).
106- - ** JVM-side plan construction via Protobuf.** Build DataFusion logical and
107- physical plans on the JVM using the existing DataFusion Protobuf
108- representation, then ship them to the native side for execution. This lets
109- plans assembled by other JVM tools (Spark, Flink, Beam, custom planners) run
110- on DataFusion without having to go through SQL.
111- - ** Java-defined expressions.** Allow user expressions / UDFs to be implemented
112- in Java and invoked from the native plan, operating on Apache Arrow vectors
113- (Java side) so evaluation stays vectorized end-to-end and avoids row-by-row
114- JNI crossings.
115-
116- These are intentionally large items — design discussion via GitHub issues
117- before implementation is welcome.
73+ - ` native/ ` — Rust crate (JNI + Arrow C Data Interface)
11874
11975## Contributing
12076
121- This project follows the Apache DataFusion contribution model. Issues and pull
122- requests are welcome — please open a GitHub issue to discuss any significant
123- change before sending a PR.
77+ Open an issue to discuss non-trivial changes before sending a PR.
78+ See [ CONTRIBUTING.md] ( CONTRIBUTING.md ) .
12479
12580## License
12681
127- Licensed under the [ Apache License, Version 2.0] ( LICENSE.txt ) . See
128- [ NOTICE.txt] ( NOTICE.txt ) for required attributions.
82+ Apache License 2.0. See [ LICENSE.txt] ( LICENSE.txt ) and [ NOTICE.txt] ( NOTICE.txt ) .
0 commit comments