Skip to content

Commit 82ef7ad

Browse files
committed
docs: replace roadmap with project-status checklist
Rewrite the README around a feature checklist covering the three query interfaces (SQL, limited DataFrame, DataFusion-Proto logical plans), data sources, result materialization, and not-yet items. Move build prerequisites, build/test commands, and the TPC-H test-data section from README into CONTRIBUTING.md so the README focuses on using the library rather than building it.
1 parent 757a078 commit 82ef7ad

2 files changed

Lines changed: 65 additions & 77 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,40 @@ to fetch the DataFusion protobuf schemas (see *Updating the DataFusion /
4141
protobuf schema version* below). Subsequent builds are offline — the
4242
`download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them.
4343

44+
## Build prerequisites
45+
46+
- JDK 17 or newer.
47+
- Rust toolchain (stable, installed via [rustup]).
48+
- [`tpchgen-cli`] — only needed to generate test data for the Parquet
49+
integration test (`cargo install tpchgen-cli`).
50+
51+
Maven is bundled via the `./mvnw` wrapper; no separate Maven install required.
52+
53+
[rustup]: https://rustup.rs/
54+
[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs
55+
56+
## Build and test
57+
58+
make test
59+
60+
This builds the native Rust crate and runs the JUnit tests. The steps can be
61+
run individually:
62+
63+
cd native && cargo build
64+
./mvnw test
65+
66+
The native library must be built before running JVM tests.
67+
68+
## Test data
69+
70+
The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in
71+
Snappy-compressed Parquet). Generate it once with:
72+
73+
make tpch-data
74+
75+
Tests that need this data skip cleanly if it is missing. `make clean` does
76+
**not** remove `tpch-data/` — delete it manually to reclaim the disk space.
77+
4478
## Code style
4579

4680
- Java: run `./mvnw spotless:apply` before committing. CI fails the build if

‎README.md‎

Lines changed: 31 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,13 @@
11
# Apache DataFusion Java
22

3-
Java bindings for [Apache DataFusion], the Rust-based query engine. SQL queries
4-
run in native code and results are returned to the JVM as [Apache Arrow] record
5-
batches via the Arrow C Data Interface — no per-row JNI calls, no row-by-row
6-
copies.
3+
Java bindings for [Apache DataFusion]. Queries run in native Rust and results
4+
return to the JVM as [Apache Arrow] batches via the Arrow C Data Interface.
75

86
[Apache DataFusion]: https://datafusion.apache.org/
97
[Apache Arrow]: https://arrow.apache.org/
108

11-
> **Project status: early development.** This is a brand-new project. The API
12-
> is small, will change without notice, and there is no published release. Do
13-
> not depend on it from production code yet. Bug reports, design feedback, and
14-
> contributions are very welcome.
9+
> Early development: no releases yet, API will change. Bug reports and
10+
> contributions welcome.
1511
1612
## Quickstart
1713

@@ -32,97 +28,55 @@ try (var allocator = new RootAllocator();
3228
ArrowReader reader = df.collect(allocator)) {
3329
while (reader.loadNextBatch()) {
3430
var batch = reader.getVectorSchemaRoot();
35-
// ... consume batch ...
31+
// ...
3632
}
3733
}
3834
}
3935
```
4036

41-
The current public surface mirrors a small slice of the Rust DataFusion API:
37+
`SessionContext` and `DataFrame` are `AutoCloseable` and not thread-safe.
4238

43-
- `SessionContext.sql(String query)` — parse and plan a SQL query, returning a
44-
lazy `DataFrame`. No execution happens yet.
45-
- `DataFrame.collect(BufferAllocator allocator)` — execute the plan and return
46-
the result batches as an `ArrowReader`. Consumes the DataFrame; the caller
47-
closes the reader, and the allocator must outlive it.
48-
- `SessionContext.registerParquet(String name, String path)` — register a local
49-
Parquet file as a SQL table.
39+
## Project status
5040

51-
Both `SessionContext` and `DataFrame` are `AutoCloseable` and **not
52-
thread-safe**.
41+
Query interfaces:
5342

54-
## Prerequisites
43+
- [x] SQL: `SessionContext.sql(String)`
44+
- [x] DataFrame: `select`, `filter` (other transformations TBD)
45+
- [x] DataFusion-Proto `LogicalPlanNode`: `SessionContext.fromProto(byte[])`.
46+
The `datafusion-proto` Java classes are generated by the build.
5547

56-
- JDK 17 or newer
57-
- Rust toolchain (stable, installed via [rustup])
58-
- [`tpchgen-cli`] — only needed to generate test data for the Parquet
59-
integration test (`cargo install tpchgen-cli`)
48+
Data sources:
6049

61-
Maven is bundled via the `./mvnw` wrapper; no separate Maven install required.
50+
- [x] Parquet via `registerParquet` / `readParquet`, with `ParquetReadOptions`
51+
- [ ] CSV, JSON, Avro
52+
- [ ] Custom catalog and table providers
6253

63-
[rustup]: https://rustup.rs/
64-
[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs
54+
Results:
6555

66-
## Build & test
56+
- [x] `DataFrame.collect(allocator)` — Arrow C Data Interface stream
57+
- [x] `DataFrame.count()`, `show()`, `show(int)`
58+
- [x] `SessionContext.tableSchema(String)`
6759

68-
make test
60+
Not yet:
6961

70-
This builds the native Rust crate and runs the JUnit tests. The steps can be
71-
run individually:
62+
- [ ] `SessionConfig` / `RuntimeEnv` knobs
63+
- [ ] Java UDFs
64+
- [ ] `write_*` outputs
7265

73-
cd native && cargo build
74-
./mvnw test
66+
## Requirements
7567

76-
The native library must be built before running JVM tests.
68+
JDK 17+. Building from source: see [CONTRIBUTING.md](CONTRIBUTING.md).
7769

78-
## Test data
79-
80-
The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in
81-
Snappy-compressed Parquet). Generate it once with:
82-
83-
make tpch-data
84-
85-
Tests that need this data skip cleanly if it is missing. `make clean` does
86-
**not** remove `tpch-data/` — delete it manually to reclaim the disk space.
87-
88-
## Repository layout
70+
## Layout
8971

9072
- `src/` — Java sources and tests
91-
- `native/` — Rust crate that exposes DataFusion over JNI and the Arrow C Data
92-
Interface
93-
94-
## Roadmap
95-
96-
Near-term priorities, roughly in order:
97-
98-
- **Session configuration.** Expose `SessionConfig` and `RuntimeEnv` settings
99-
(target partitions, batch size, memory pool, default catalog, …) so callers
100-
can tune execution from the JVM.
101-
- **Full `SessionContext` and `DataFrame` APIs.** Expand beyond `sql` and
102-
`registerParquet` to mirror the Rust API: table registration variants,
103-
`read_*` / `write_*` entry points, and DataFrame transformations such as
104-
`select`, `filter`, `join`, `aggregate`, `sort`, plus result-materialization
105-
variants (`show`, `count`, streaming collection).
106-
- **JVM-side plan construction via Protobuf.** Build DataFusion logical and
107-
physical plans on the JVM using the existing DataFusion Protobuf
108-
representation, then ship them to the native side for execution. This lets
109-
plans assembled by other JVM tools (Spark, Flink, Beam, custom planners) run
110-
on DataFusion without having to go through SQL.
111-
- **Java-defined expressions.** Allow user expressions / UDFs to be implemented
112-
in Java and invoked from the native plan, operating on Apache Arrow vectors
113-
(Java side) so evaluation stays vectorized end-to-end and avoids row-by-row
114-
JNI crossings.
115-
116-
These are intentionally large items — design discussion via GitHub issues
117-
before implementation is welcome.
73+
- `native/` — Rust crate (JNI + Arrow C Data Interface)
11874

11975
## Contributing
12076

121-
This project follows the Apache DataFusion contribution model. Issues and pull
122-
requests are welcome — please open a GitHub issue to discuss any significant
123-
change before sending a PR.
77+
Open an issue to discuss non-trivial changes before sending a PR.
78+
See [CONTRIBUTING.md](CONTRIBUTING.md).
12479

12580
## License
12681

127-
Licensed under the [Apache License, Version 2.0](LICENSE.txt). See
128-
[NOTICE.txt](NOTICE.txt) for required attributions.
82+
Apache License 2.0. See [LICENSE.txt](LICENSE.txt) and [NOTICE.txt](NOTICE.txt).

0 commit comments

Comments
 (0)