Skip to content

Commit 3b27c05

Browse files
committed
chore(release): add ASF release scripts for source tarballs
Adds dev/release/ with create-tarball.sh, release-tarball.sh, and verify-release-candidate.sh, adapted from apache/datafusion. Also includes a release README, RAT exclude list, and the check-rat-report.py helper. These scripts produce, sign, vote on, promote, and verify the apache-datafusion-java-<version>.tar.gz source distribution published under https://dist.apache.org/repos/dist/{dev,release}/datafusion/.
1 parent 7f49541 commit 3b27c05

6 files changed

Lines changed: 615 additions & 0 deletions

File tree

‎dev/release/README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Release Process
21+
22+
This guide is for maintainers creating release candidates of Apache
23+
DataFusion Java and running the release process.
24+
25+
## Release Prerequisites
26+
27+
### Add a git remote for the `apache` repo
28+
29+
The instructions below assume the upstream git repo
30+
`git@github.com:apache/datafusion-java.git` is configured as the remote
31+
named `apache`:
32+
33+
```shell
34+
git remote add apache git@github.com:apache/datafusion-java.git
35+
```
36+
37+
### Add your GPG public key to the SVN `KEYS` file
38+
39+
If you will be releasing the final tarball, your GPG public key must be
40+
present in:
41+
42+
- https://dist.apache.org/repos/dist/dev/datafusion/KEYS
43+
- https://dist.apache.org/repos/dist/release/datafusion/KEYS
44+
45+
See https://infra.apache.org/release-signing.html#generate for generating
46+
keys.
47+
48+
Committers can add signing keys via Subversion using their ASF account:
49+
50+
```shell
51+
svn co https://dist.apache.org/repos/dist/dev/datafusion
52+
cd datafusion
53+
(gpg --list-sigs "Your Name" && gpg --armor --export "Your Name") >> KEYS
54+
svn commit KEYS -m "Add key for Your Name"
55+
```
56+
57+
## Release Process: Step by Step
58+
59+
As part of the Apache governance model, official releases consist of
60+
signed source tarballs approved by the PMC.
61+
62+
### 1. Pick a Release Candidate (RC) number
63+
64+
Pick numbers in sequential order, with `1` for `rc1`, `2` for `rc2`, and
65+
so on.
66+
67+
### 2. Create the Git tag for the Release Candidate
68+
69+
Release tags look like `0.1.0`, and release candidate tags look like
70+
`0.1.0-rc1`.
71+
72+
```shell
73+
git fetch apache
74+
git tag 0.1.0-rc1 apache/main
75+
git push apache 0.1.0-rc1
76+
```
77+
78+
### 3. Create, sign, and upload the tarball
79+
80+
Run `create-tarball.sh` with the version and RC number:
81+
82+
```shell
83+
./dev/release/create-tarball.sh 0.1.0 1
84+
```
85+
86+
The script:
87+
88+
1. Creates and uploads the release candidate artifacts to the
89+
[`dist/dev/datafusion`](https://dist.apache.org/repos/dist/dev/datafusion)
90+
location on the Apache SVN distribution server, under
91+
`apache-datafusion-java-<version>-rc<rc>/`.
92+
2. Prints an email template to send to `dev@datafusion.apache.org` for
93+
the release vote.
94+
95+
### 4. Vote on the Release Candidate
96+
97+
Send the email printed by the script to `dev@datafusion.apache.org`.
98+
99+
The vote must remain open for at least 72 hours and requires at least
100+
three PMC `+1` votes and no `-1` votes to pass.
101+
102+
### Verifying Release Candidates
103+
104+
`dev/release/verify-release-candidate.sh` downloads the candidate
105+
tarball, verifies its signature and checksums, and builds and tests the
106+
source distribution:
107+
108+
```shell
109+
./dev/release/verify-release-candidate.sh 0.1.0 1
110+
```
111+
112+
The script requires `curl`, `git`, `gpg`, a C toolchain, `java`, and
113+
either `shasum` or `sha256sum`/`sha512sum` to be on the `PATH`. It will
114+
install a temporary Rust toolchain into the sandbox directory.
115+
116+
### 5. If the Vote Passes: Promote the Tarball
117+
118+
Move artifacts from the `dev` SVN area to the `release` area using
119+
`release-tarball.sh`:
120+
121+
```shell
122+
./dev/release/release-tarball.sh 0.1.0 1
123+
```
124+
125+
### 6. Create the Release Git Tag
126+
127+
Tag the same release candidate commit with the final release tag:
128+
129+
```shell
130+
git checkout 0.1.0-rc1
131+
git tag 0.1.0
132+
git push apache 0.1.0
133+
```
134+
135+
### 7. Add the Release to Apache Reporter
136+
137+
Add the release at https://reporter.apache.org/addrelease.html?datafusion
138+
so it appears in the project's board report.
139+
140+
### 8. Delete Old RCs and Releases
141+
142+
See the ASF documentation on [when to
143+
archive](https://www.apache.org/legal/release-policy.html#when-to-archive).
144+
145+
Release candidates should be deleted once the release is published:
146+
147+
```shell
148+
svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep datafusion-java
149+
svn delete -m "delete old DataFusion Java RC" \
150+
https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-java-0.1.0-rc1/
151+
```
152+
153+
Only the latest release should remain in the `release` SVN area. Delete
154+
older releases after publishing a new one:
155+
156+
```shell
157+
svn ls https://dist.apache.org/repos/dist/release/datafusion | grep datafusion-java
158+
svn delete -m "delete old DataFusion Java release" \
159+
https://dist.apache.org/repos/dist/release/datafusion/datafusion-java-0.1.0
160+
```

‎dev/release/check-rat-report.py‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/usr/bin/python
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
19+
import fnmatch
20+
import re
21+
import sys
22+
import xml.etree.ElementTree as ET
23+
24+
if len(sys.argv) != 3:
25+
sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" %
26+
sys.argv[0])
27+
sys.exit(1)
28+
29+
exclude_globs_filename = sys.argv[1]
30+
xml_filename = sys.argv[2]
31+
32+
globs = [line.strip() for line in open(exclude_globs_filename, "r")]
33+
34+
tree = ET.parse(xml_filename)
35+
root = tree.getroot()
36+
resources = root.findall('resource')
37+
38+
all_ok = True
39+
for r in resources:
40+
approvals = r.findall('license-approval')
41+
if not approvals or approvals[0].attrib['name'] == 'true':
42+
continue
43+
clean_name = re.sub('^[^/]+/', '', r.attrib['name'])
44+
excluded = False
45+
for g in globs:
46+
if fnmatch.fnmatch(clean_name, g):
47+
excluded = True
48+
break
49+
if not excluded:
50+
sys.stdout.write("NOT APPROVED: %s (%s): %s\n" % (
51+
clean_name, r.attrib['name'], approvals[0].attrib['name']))
52+
all_ok = False
53+
54+
if not all_ok:
55+
sys.exit(1)
56+
57+
print('OK')
58+
sys.exit(0)

‎dev/release/create-tarball.sh‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
# Adapted from https://github.com/apache/datafusion/blob/main/dev/release/create-tarball.sh
22+
23+
# This script creates a signed tarball in
24+
# dev/dist/apache-datafusion-java-<version>-rc<rc>/apache-datafusion-java-<version>.tar.gz
25+
# and uploads it to the "dev" area of the dist.apache.datafusion repository,
26+
# then prints an email template for the dev@datafusion.apache.org list for a
27+
# formal vote.
28+
#
29+
# See dev/release/README.md for full release instructions.
30+
#
31+
# Requirements:
32+
#
33+
# 1. gpg setup for signing and have uploaded your public
34+
# signature to https://pgp.mit.edu/ and the project KEYS file.
35+
#
36+
# 2. Logged into the Apache SVN server with the appropriate
37+
# credentials.
38+
#
39+
# Based in part on 02-source.sh from apache/arrow.
40+
41+
set -e
42+
43+
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
44+
SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)"
45+
46+
if [ "$#" -ne 2 ]; then
47+
echo "Usage: $0 <version> <rc>"
48+
echo "ex. $0 0.1.0 1"
49+
exit
50+
fi
51+
52+
version=$1
53+
rc=$2
54+
tag="${version}-rc${rc}"
55+
56+
release=apache-datafusion-java-${version}
57+
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
58+
tarname=${release}.tar.gz
59+
tarball=${distdir}/${tarname}
60+
url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"
61+
62+
echo "Attempting to create ${tarball} from tag ${tag}"
63+
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})
64+
65+
if [ -z "$release_hash" ]; then
66+
echo "Cannot continue: unknown git tag: ${tag}"
67+
exit 1
68+
fi
69+
70+
echo "Draft email for dev@datafusion.apache.org mailing list"
71+
echo ""
72+
echo "---------------------------------------------------------"
73+
cat <<MAIL
74+
To: dev@datafusion.apache.org
75+
Subject: [VOTE] Release Apache DataFusion Java ${version} RC${rc}
76+
Hi,
77+
78+
I would like to propose a release of Apache DataFusion Java version ${version}.
79+
80+
This release candidate is based on commit: ${release_hash} [1]
81+
The proposed release tarball and signatures are hosted at [2].
82+
The changelog is located at [3].
83+
84+
Please download, verify checksums and signatures, run the unit tests, and vote
85+
on the release. The vote will be open for at least 72 hours.
86+
87+
Only votes from PMC members are binding, but all members of the community are
88+
encouraged to test the release and vote with "(non-binding)".
89+
90+
The standard verification procedure is documented at
91+
https://github.com/apache/datafusion-java/blob/main/dev/release/README.md#verifying-release-candidates.
92+
93+
[ ] +1 Release this as Apache DataFusion Java ${version}
94+
[ ] +0
95+
[ ] -1 Do not release this as Apache DataFusion Java ${version} because...
96+
97+
Here is my vote:
98+
99+
+1
100+
101+
[1]: https://github.com/apache/datafusion-java/tree/${release_hash}
102+
[2]: ${url}
103+
[3]: https://github.com/apache/datafusion-java/blob/${release_hash}/dev/changelog/${version}.md
104+
MAIL
105+
echo "---------------------------------------------------------"
106+
107+
108+
# create <tarball> containing the files in git at $release_hash
109+
# the files in the tarball are prefixed with apache-datafusion-java-<version>/
110+
mkdir -p ${distdir}
111+
(cd "${SOURCE_TOP_DIR}" && git archive ${release_hash} --prefix ${release}/ | gzip > ${tarball})
112+
113+
echo "Signing tarball and creating checksums"
114+
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
115+
# create checksums with relative path of tarball so that they can be verified
116+
# with a command such as
117+
# shasum --check apache-datafusion-java-0.1.0.tar.gz.sha512
118+
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
119+
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512
120+
121+
122+
echo "Uploading to datafusion dist/dev to ${url}"
123+
svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
124+
svn add ${distdir}
125+
svn ci -m "Apache DataFusion Java ${version} RC${rc}" ${distdir}

‎dev/release/rat_exclude_files.txt‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
*.gitignore
2+
*.dockerignore
3+
.github/pull_request_template.md
4+
.github/ISSUE_TEMPLATE/bug_report.yml
5+
.github/ISSUE_TEMPLATE/feature_request.yml
6+
.gitmodules
7+
.mvn/wrapper/maven-wrapper.properties
8+
mvnw
9+
mvnw.cmd
10+
native/Cargo.lock
11+
dev/release/rat_exclude_files.txt
12+
docs/source/_static/**
13+
docs/source/conf.py
14+
rust-toolchain
15+
target/*
16+
**/target/*
17+
tpch-data/*

0 commit comments

Comments
 (0)