Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions .asf.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# This file controls the settings of this repository
#
# See more details at
# https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features

notifications:
commits: commits@datafusion.apache.org
issues: github@datafusion.apache.org
pullrequests: github@datafusion.apache.org
discussions: github@datafusion.apache.org
jira_options: link label worklog

github:
description: "Java bindings for Apache DataFusion"
homepage: https://datafusion.apache.org/
labels:
- apache
- arrow
- datafusion
- java
- jni
- jvm
- query-engine
- sql
enabled_merge_buttons:
squash: true
squash_commit_message: PR_TITLE_AND_DESC
merge: false
rebase: false
features:
issues: true
discussions: true
protected_branches:
main: {}
pull_requests:
allow_update_branch: true
allow_auto_merge: true
del_branch_on_merge: true
59 changes: 59 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: Build

on:
push:
branches: [main]
pull_request:
branches: [main]

permissions:
contents: read

jobs:
build:
name: make test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up JDK 17
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
cache: maven

- name: Set up Rust toolchain
run: |
rustup update stable
rustup default stable

- name: Cache cargo
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
key: ${{ runner.os }}-cargo-${{ hashFiles('native/Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-

- name: Build native and run JVM tests
run: make test
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
target/
*.class
.idea/
.vscode/
*.iml
.DS_Store
tpch-data/
.claude
docs/superpowers
3 changes: 3 additions & 0 deletions .mvn/wrapper/maven-wrapper.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
wrapperVersion=3.3.4
distributionType=only-script
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.zip
126 changes: 126 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Contributing to Apache DataFusion Java

Bug reports, design discussion, and patches are welcome. This project follows
the Apache DataFusion contribution model.

## Filing issues and discussing changes

- File bugs and feature requests on [GitHub issues](https://github.com/apache/datafusion-java/issues).
- For larger or design-level discussion, the mailing list is
[dev@datafusion.apache.org](mailto:dev@datafusion.apache.org).
- Please open an issue before sending a PR for any significant change so the
approach can be agreed on first.

## Development workflow

Branch from `main`, write changes with [conventional commit](https://www.conventionalcommits.org/)
messages in the imperative mood (e.g. `feat: add foo`, `fix(native): handle bar`),
and open a pull request targeting `main`.

The first build in a fresh checkout reaches out to `raw.githubusercontent.com`
to fetch the DataFusion protobuf schemas (see *Updating the DataFusion /
protobuf schema version* below). Subsequent builds are offline — the
`download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them.

## Build prerequisites

- JDK 17 or newer.
- Rust toolchain (stable, installed via [rustup]).
- [`tpchgen-cli`] — only needed to generate test data for the Parquet
integration test (`cargo install tpchgen-cli`).

Maven is bundled via the `./mvnw` wrapper; no separate Maven install required.

[rustup]: https://rustup.rs/
[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs

## Build and test

make test

This builds the native Rust crate and runs the JUnit tests. The steps can be
run individually:

cd native && cargo build
./mvnw test

The native library must be built before running JVM tests.

## Test data

The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in
Snappy-compressed Parquet). Generate it once with:

make tpch-data

Tests that need this data skip cleanly if it is missing. `make clean` does
**not** remove `tpch-data/` — delete it manually to reclaim the disk space.

## Code style

- Java: run `./mvnw spotless:apply` before committing. CI fails the build if
formatting drifts.
- Rust: run `cargo fmt` and `cargo clippy --all-targets -- -D warnings` inside
`native/`.
- New source files need the Apache 2.0 license header. Apache RAT enforces this
during `verify`.

## Updating the DataFusion / protobuf schema version

Three things must move together when bumping DataFusion:

1. `native/Cargo.toml` — the `datafusion` crate dependency.
2. `pom.xml` — the `<datafusion.version>` Maven property. **Must equal the
Cargo version**; a mismatch means JVM-built protobuf plans won't deserialize
on the native side.
3. `pom.xml` — the `<sha512>` checksums on the two `download-maven-plugin`
executions. These pin the downloaded `.proto` files; the build fails if
upstream silently re-tags them, which is the desired behavior.

Recipe:

```sh
# 1. Bump the Cargo dep
$EDITOR native/Cargo.toml # set datafusion = "<new>"
(cd native && cargo update -p datafusion)

# 2. Bump the Maven property to match
$EDITOR pom.xml # set <datafusion.version>

# 3. Compute the new SHA-512 hashes for both `.proto` files from the upstream
# tag you just set in step 2, then paste them into the two <sha512> elements
# in pom.xml.
NEW=$(grep -m1 -oE '<datafusion.version>[^<]+' pom.xml | cut -d'>' -f2)
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}'
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}'
$EDITOR pom.xml # paste the two hashes into the <sha512> elements

# Drop the local download cache so the next build re-downloads against the new hashes.
rm -rf ~/.m2/repository/.cache/download-maven-plugin target/proto

# 4. Verify
make && make test
```

The protobuf runtime version (`<protobuf.version>` in `pom.xml`) tracks the
Java ecosystem (security and JDK compatibility), not DataFusion. Bump it
independently when there is a reason.
Loading
Loading