Skip to content

chore: drop support for JDK 11 #5897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sunchao merged 1 commit into from
Sep 13, 2026
Merged

chore: drop support for JDK 11 #5897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ jobs:
uses: actions/setup-java@v6
with:
distribution: temurin
java-version: 11
java-version: 17

- name: Apache RAT license check
run: ./mvnw -B -N apache-rat:check
Expand Down Expand Up @@ -261,7 +261,7 @@ jobs:
with:
spark-short: '3.4'
spark-full: '3.4.3'
java: 11
java: 17

spark_3_5:
name: Spark SQL Tests (Spark 3.5)
Expand Down Expand Up @@ -317,7 +317,7 @@ jobs:
iceberg-full: '1.8.1'
spark-short: '3.4'
spark-full: '3.4.3'
java: 11
java: 17

iceberg_1_9:
name: Iceberg Spark SQL Tests (Iceberg 1.9)
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/pr_build_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,12 @@ jobs:
container:
image: amd64/rust
env:
JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
JAVA_TOOL_OPTIONS: '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED'
strategy:
matrix:
profile:
- name: "Spark 3.4, JDK 11, Scala 2.12"
java_version: "11"
- name: "Spark 3.4, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.4 -Pscala-2.12"
- name: "Spark 3.5, JDK 17, Scala 2.12"
java_version: "17"
Expand Down Expand Up @@ -347,8 +347,8 @@ jobs:
# the goal with these profiles is to get coverage of all Java, Scala, and Spark
# versions without testing all possible combinations, which would be overkill
profile:
- name: "Spark 3.4, JDK 11, Scala 2.12"
java_version: "11"
- name: "Spark 3.4, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.4 -Pscala-2.12"

- name: "Spark 3.5, JDK 17, Scala 2.13"
Expand Down Expand Up @@ -514,7 +514,7 @@ jobs:
container:
image: amd64/rust
env:
JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
JAVA_TOOL_OPTIONS: '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED'

steps:
- uses: actions/checkout@v7
Expand Down
15 changes: 4 additions & 11 deletions docs/source/contributor-guide/benchmarking_micro_ec2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,8 @@ java -version
javac -version
```

JDK 17 or later is required. Comet targets Java 17 bytecode whenever the JDK in use is 17 or
newer, through the `jdk17` profile in the pom, so no profile needs to be passed by hand. `run.py`
detects `JAVA_HOME` when it is not set, preferring the newest JDK under `/usr/lib/jvm`, and refuses
JDK 17 or later is required. Comet targets Java 17 bytecode, so no JDK profile needs to be passed
by hand. `run.py` detects `JAVA_HOME` when it is not set, preferring the newest JDK under `/usr/lib/jvm`, and refuses
to run with anything older than 17.

### Maven
Expand Down Expand Up @@ -352,21 +351,15 @@ sudo dnf install -y gcc gcc-c++ make
**`JAVA_HOME could not be determined`.** Install a JDK and export `JAVA_HOME`, or pass a checkout
that has one configured.

**`Class java.lang.Record not found - continuing with a stub`.** The Scala compiler is targeting the
Java 11 API, which has no `Record`, while the Spark 4.x sources need Java 17. This happens on a JDK
older than 17, where the `jdk17` profile does not activate and `java.version` stays at its default
of 11. Use JDK 17 or later:
**`Comet requires JDK 17 or later to build`.** The Maven enforcer plugin rejects JDKs older than 17.
Point `JAVA_HOME` at JDK 17 or later and rerun the build:

```shell
export JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto
export PATH=$JAVA_HOME/bin:$PATH
./mvnw clean
make release
```

Run `./mvnw clean` first: classes compiled against the wrong API stay in `target/` and break the
next build.

**`cargo: command not found` after `setup`.** `rustup` installs into `~/.cargo/bin`. Run
`source "$HOME/.cargo/env"`, or start a new shell.

Expand Down
6 changes: 2 additions & 4 deletions docs/source/contributor-guide/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -407,10 +407,8 @@ from a clean IntelliJ configuration:
PROFILES="-Pspark-4.0" make release
```

The `spark-4.0` profile sets Scala 2.13 and Java 17 properties. If you need to be explicit, use
`PROFILES="-Pspark-4.0 -Pscala-2.13 -Pjdk17" make release`.

The Maven profile is named `jdk17` in this project.
The `spark-4.0` profile sets the Scala 2.13 properties, and every profile targets Java 17. If you
need to be explicit, use `PROFILES="-Pspark-4.0 -Pscala-2.13" make release`.

If the native build previously used a different JDK, clear Cargo's cached JNI link path before
rebuilding:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributor-guide/iceberg-spark-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ diff must be generated against its own tag.
## Running Tests in CI

The `iceberg_spark_test_<version>.yml` workflows apply these diffs and run the three Gradle targets above
against each Iceberg version. Iceberg 1.8.1 runs against Spark 3.4.3 with Java 11; Iceberg 1.9.1 and 1.10.0
run against Spark 3.5.9 with Java 17; Iceberg 1.11.0 runs against Spark 4.1.3 with Java 17. Iceberg 1.11
against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against Spark 3.4.3; Iceberg 1.9.1 and 1.10.0
run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. Iceberg 1.11
(the only version testing Spark 4.1) runs on every pull request and in the merge queue; the older versions
(1.8, 1.9, 1.10) run only in the merge queue, or on a pull request labeled `run-iceberg-tests`. All caller
workflows delegate to `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See
Expand Down
9 changes: 2 additions & 7 deletions docs/source/user-guide/latest/compatibility/spark-versions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ of the versioning policy.

## Spark 3.4

Spark 3.4.3 is supported with Java 11/17 and Scala 2.12/2.13.
Spark 3.4.3 is supported with Java 17 and Scala 2.12/2.13.

```{warning}
Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release.
Expand All @@ -47,12 +47,7 @@ Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in t

## Spark 3.5

Spark 3.5.9 is supported with Java 11/17 and Scala 2.12/2.13.

```{warning}
JDK 11 support is deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release.
We recommend moving to JDK 17 or later.
```
Spark 3.5.9 is supported with Java 17 and Scala 2.12/2.13.

### Known Limitations

Expand Down
10 changes: 6 additions & 4 deletions docs/source/user-guide/latest/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,17 @@ in the [Compatibility Guide] for more information, such as known limitations per
We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI.
Other versions may work well enough for development and evaluation purposes.

Comet requires JDK 17 or later. JDK 11 is no longer supported as of the 1.1.0 release.

```{warning}
JDK 11 and Spark 3.4 support are deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release.
We recommend moving to JDK 17 or later and Spark 3.5 or later.
Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in the 1.1.0 release.
We recommend moving to Spark 3.5 or later.
```

| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |
| ------------- | ------------ | ------------- | ----------------- | --------------------- |
| 3.4.3 | 11/17 | 2.12/2.13 | Yes | Yes |
| 3.5.9 | 11/17 | 2.12/2.13 | Yes | Yes |
| 3.4.3 | 17 | 2.12/2.13 | Yes | Yes |
| 3.5.9 | 17 | 2.12/2.13 | Yes | Yes |
| 4.0.4 | 17/21 | 2.13 | Yes | Yes |
| 4.1.3 | 17/21 | 2.13 | Yes | Yes |

Expand Down
48 changes: 17 additions & 31 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ under the License.
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>11</java.version>
<java.version>17</java.version>
<!--
Default Delta dep version (only consumed when `-Pcontrib-delta` is also
active). Spark profiles override this: spark-3.5 -> 3.3.2, spark-4.1 ->
Expand Down Expand Up @@ -674,7 +674,7 @@ under the License.
<shims.majorVerSrc>spark-3.x</shims.majorVerSrc>
<shims.minorVerSrc>spark-3.4</shims.minorVerSrc>
<shims.minorPlusVerSrc>spark-none</shims.minorPlusVerSrc>
<java.version>11</java.version>
<java.version>17</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
Expand All @@ -693,7 +693,7 @@ under the License.
<shims.majorVerSrc>spark-3.x</shims.majorVerSrc>
<shims.minorVerSrc>spark-3.5</shims.minorVerSrc>
<shims.minorPlusVerSrc>spark-none</shims.minorPlusVerSrc>
<java.version>11</java.version>
<java.version>17</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
Expand Down Expand Up @@ -780,34 +780,6 @@ under the License.
</properties>
</profile>

<profile>
<id>jdk11</id>
<activation>
<jdk>11</jdk>
</activation>
<properties>
<java.version>11</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
</profile>

<profile>
<id>jdk17</id>
<activation>
<!-- A range, not an exact match. `<jdk>17</jdk>` is a prefix match, so it
activates for 17.x but not for 21 or later, leaving java.version at the
default of 11 and compiling the Spark 4.x sources against the Java 11 API
("Class java.lang.Record not found"). -->
<jdk>[17,)</jdk>
</activation>
<properties>
<java.version>17</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
</profile>

<profile>
<id>semanticdb</id>
<properties>
Expand Down Expand Up @@ -1210,6 +1182,20 @@ under the License.
<artifactId>maven-enforcer-plugin</artifactId>
<version>${maven-enforcer-plugin.version}</version>
<executions>
<execution>
<id>require-java-17</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireJavaVersion>
<version>[17,)</version>
<message>Comet requires JDK 17 or later to build.</message>
</requireJavaVersion>
</rules>
</configuration>
</execution>
<execution>
<id>no-duplicate-declared-dependencies</id>
<goals>
Expand Down
4 changes: 2 additions & 2 deletions spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ under the License.
<version>1.5.2</version>
<scope>test</scope>
</dependency>
<!-- Jetty 9.4.x for Spark 3.4 (JDK 11, javax.* packages) -->
<!-- Jetty 9.4.x for Spark 3.4 (javax.* packages) -->
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
Expand Down Expand Up @@ -299,7 +299,7 @@ under the License.
<version>1.8.1</version>
<scope>test</scope>
</dependency>
<!-- Jetty 9.4.x for Spark 3.5 (JDK 11, javax.* packages) -->
<!-- Jetty 9.4.x for Spark 3.5 (javax.* packages) -->
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
Expand Down
Loading