Skip to content

Commit 5292e05

Browse files
authored
feat(proto): generate datafusion-proto Java classes at build time (#9)
1 parent e412ab5 commit 5292e05

3 files changed

Lines changed: 191 additions & 0 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Contributing to Apache DataFusion Java
21+
22+
Bug reports, design discussion, and patches are welcome. This project follows
23+
the Apache DataFusion contribution model.
24+
25+
## Filing issues and discussing changes
26+
27+
- File bugs and feature requests on [GitHub issues](https://github.com/apache/datafusion-java/issues).
28+
- For larger or design-level discussion, the mailing list is
29+
[dev@datafusion.apache.org](mailto:dev@datafusion.apache.org).
30+
- Please open an issue before sending a PR for any significant change so the
31+
approach can be agreed on first.
32+
33+
## Development workflow
34+
35+
Branch from `main`, write changes with [conventional commit](https://www.conventionalcommits.org/)
36+
messages in the imperative mood (e.g. `feat: add foo`, `fix(native): handle bar`),
37+
and open a pull request targeting `main`.
38+
39+
The first build in a fresh checkout reaches out to `raw.githubusercontent.com`
40+
to fetch the DataFusion protobuf schemas (see *Updating the DataFusion /
41+
protobuf schema version* below). Subsequent builds are offline — the
42+
`download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them.
43+
44+
## Code style
45+
46+
- Java: run `./mvnw spotless:apply` before committing. CI fails the build if
47+
formatting drifts.
48+
- Rust: run `cargo fmt` and `cargo clippy --all-targets -- -D warnings` inside
49+
`native/`.
50+
- New source files need the Apache 2.0 license header. Apache RAT enforces this
51+
during `verify`.
52+
53+
## Updating the DataFusion / protobuf schema version
54+
55+
Three things must move together when bumping DataFusion:
56+
57+
1. `native/Cargo.toml` — the `datafusion` crate dependency.
58+
2. `pom.xml` — the `<datafusion.version>` Maven property. **Must equal the
59+
Cargo version**; a mismatch means JVM-built protobuf plans won't deserialize
60+
on the native side.
61+
3. `pom.xml` — the `<sha512>` checksums on the two `download-maven-plugin`
62+
executions. These pin the downloaded `.proto` files; the build fails if
63+
upstream silently re-tags them, which is the desired behavior.
64+
65+
Recipe:
66+
67+
```sh
68+
# 1. Bump the Cargo dep
69+
$EDITOR native/Cargo.toml # set datafusion = "<new>"
70+
(cd native && cargo update -p datafusion)
71+
72+
# 2. Bump the Maven property to match
73+
$EDITOR pom.xml # set <datafusion.version>
74+
75+
# 3. Compute the new SHA-512 hashes for both `.proto` files from the upstream
76+
# tag you just set in step 2, then paste them into the two <sha512> elements
77+
# in pom.xml.
78+
NEW=$(grep -m1 -oE '<datafusion.version>[^<]+' pom.xml | cut -d'>' -f2)
79+
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}'
80+
curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}'
81+
$EDITOR pom.xml # paste the two hashes into the <sha512> elements
82+
83+
# Drop the local download cache so the next build re-downloads against the new hashes.
84+
rm -rf ~/.m2/repository/.cache/download-maven-plugin target/proto
85+
86+
# 4. Verify
87+
make && make test
88+
```
89+
90+
The protobuf runtime version (`<protobuf.version>` in `pom.xml`) tracks the
91+
Java ecosystem (security and JDK compatibility), not DataFusion. Bump it
92+
independently when there is a reason.

‎pom.xml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ under the License.
3333
<maven.compiler.target>17</maven.compiler.target>
3434
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
3535
<junit.version>5.11.3</junit.version>
36+
<datafusion.version>53.1.0</datafusion.version>
37+
<protobuf.version>3.25.5</protobuf.version>
3638
</properties>
3739

3840
<dependencies>
@@ -58,9 +60,21 @@ under the License.
5860
<version>19.0.0</version>
5961
<scope>runtime</scope>
6062
</dependency>
63+
<dependency>
64+
<groupId>com.google.protobuf</groupId>
65+
<artifactId>protobuf-java</artifactId>
66+
<version>${protobuf.version}</version>
67+
</dependency>
6168
</dependencies>
6269

6370
<build>
71+
<extensions>
72+
<extension>
73+
<groupId>kr.motd.maven</groupId>
74+
<artifactId>os-maven-plugin</artifactId>
75+
<version>1.7.1</version>
76+
</extension>
77+
</extensions>
6478
<plugins>
6579
<plugin>
6680
<groupId>org.apache.maven.plugins</groupId>
@@ -118,6 +132,7 @@ under the License.
118132
<exclude>NOTICE.txt</exclude>
119133
<!-- Project documentation that does not require headers -->
120134
<exclude>README.md</exclude>
135+
<exclude>CONTRIBUTING.md</exclude>
121136
<exclude>docs/**</exclude>
122137
<!-- VCS and editor metadata -->
123138
<exclude>.gitignore</exclude>
@@ -138,6 +153,49 @@ under the License.
138153
</excludes>
139154
</configuration>
140155
</plugin>
156+
<plugin>
157+
<groupId>com.googlecode.maven-download-plugin</groupId>
158+
<artifactId>download-maven-plugin</artifactId>
159+
<version>1.9.0</version>
160+
<executions>
161+
<execution>
162+
<id>fetch-datafusion-common-proto</id>
163+
<phase>generate-sources</phase>
164+
<goals><goal>wget</goal></goals>
165+
<configuration>
166+
<url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto-common/proto/datafusion_common.proto</url>
167+
<outputDirectory>${project.build.directory}/proto/datafusion/proto-common/proto</outputDirectory>
168+
<outputFileName>datafusion_common.proto</outputFileName>
169+
<sha512>d6f3368372ea277cc23e26f196994b81616d38599357bb374cbd7eb1760e649a789e4c133d86a395ac701049a500348da2ec039d3f978ac5d8112c2876dded1f</sha512>
170+
</configuration>
171+
</execution>
172+
<execution>
173+
<id>fetch-datafusion-proto</id>
174+
<phase>generate-sources</phase>
175+
<goals><goal>wget</goal></goals>
176+
<configuration>
177+
<url>https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto/proto/datafusion.proto</url>
178+
<outputDirectory>${project.build.directory}/proto/datafusion/proto/proto</outputDirectory>
179+
<outputFileName>datafusion.proto</outputFileName>
180+
<sha512>c3d162b8e2a418e03f74caceaccfd934af89bb95a12ede13d4cc1701d24c734d74b1e96372142b173db05938dab7f965ad60d476363308c441677a63ea5fbcf7</sha512>
181+
</configuration>
182+
</execution>
183+
</executions>
184+
</plugin>
185+
<plugin>
186+
<groupId>org.xolstice.maven.plugins</groupId>
187+
<artifactId>protobuf-maven-plugin</artifactId>
188+
<version>0.6.1</version>
189+
<configuration>
190+
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
191+
<protoSourceRoot>${project.build.directory}/proto</protoSourceRoot>
192+
</configuration>
193+
<executions>
194+
<execution>
195+
<goals><goal>compile</goal></goals>
196+
</execution>
197+
</executions>
198+
</plugin>
141199
</plugins>
142200
</build>
143201
</project>
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.datafusion.proto;
21+
22+
import static org.junit.jupiter.api.Assertions.assertNotNull;
23+
24+
import org.apache.datafusion.protobuf.LogicalPlanNode;
25+
import org.junit.jupiter.api.Test;
26+
27+
/**
28+
* Smoke test: confirms the datafusion-proto schema was downloaded, protoc generated Java sources,
29+
* those sources landed on the compile classpath, and the {@code protobuf-java} runtime resolves.
30+
*
31+
* <p>Does not exercise any DataFusion plan semantics — those tests arrive with JVM-side plan
32+
* construction.
33+
*/
34+
class ProtoGenerationTest {
35+
36+
@Test
37+
void generatedClassIsLoadableAndConstructible() {
38+
LogicalPlanNode node = LogicalPlanNode.newBuilder().build();
39+
assertNotNull(node);
40+
}
41+
}

0 commit comments

Comments
 (0)