chore: drop support for JDK 11

[manuzhang]

Which issue does this PR close?

Closes #4844.

Rationale for this change

JDK 11 support was deprecated in 1.0.0 (#4857), with removal scheduled for 1.1.0.

Spark 3.4 and 3.5 themselves still run on JDK 11, as noted on #4844. However, Comet's published artifacts already require JDK 17:

• dev/release/build-release-comet.sh refuses to build on anything older than JDK 17.
• On JDK 17+, the jdk17 profile is declared after the Spark profiles, so it overrode their java.version=11.
• As a result, the 1.0.0 comet-spark-spark3.4_2.12 and comet-spark-spark3.5_2.12 jars contain 610 Comet classes with class-file major version 61 (Java 17), including NativeBase and CometShuffleBlockIterator. Those jars cannot load on JDK 11 today.

JDK 11 only worked when building Comet from source on JDK 11 for Spark 3.x. This PR makes the JDK 17 requirement explicit and drops the JDK 11 CI coverage.

What changes are included in this PR?

Build

• pom.xml: the default java.version and the spark-3.4 / spark-3.5 profiles now target 17. The jdk11 and jdk17 profiles are removed; with 17 as the default, jdk17 would be a no-op. Passing -Pjdk17 now only logs Maven's "profile does not exist" warning.
• A new require-java-17 enforcer execution fails the validate phase on older JDKs with Comet requires JDK 17 or later to build., rather than a later compiler error.

CI

• ci.yml: Preflight (RAT check) uses JDK 17. The Spark 3.4 Spark SQL tests and Iceberg 1.8 (Spark 3.4.3) run on JDK 17.
• pr_build_linux.yml: the Spark 3.4 lint and test profiles move to JDK 17. JAVA_TOOL_OPTIONS is no longer conditional, since every profile is now JDK 17 or 21. The job names change from Spark 3.4, JDK 11, Scala 2.12 to Spark 3.4, JDK 17, Scala 2.12. Nothing in .asf.yaml or dev/ci refers to the old names, because merges are gated on the Required Checks aggregator.

Docs

• installation.md and compatibility/spark-versions.md: Spark 3.4 and 3.5 are listed with Java 17. The JDK 11 deprecation warning is replaced by a statement that JDK 17 or later is required. The Spark 3.4 deprecation warning is unchanged.
• Contributor guide:
  • iceberg-spark-tests.md: all Iceberg jobs now run on Java 17.
  • development.md: the -Pjdk17 guidance is removed.
  • benchmarking_micro_ec2.md: the Class java.lang.Record not found troubleshooting entry, which only happened on JDK < 17, is replaced by the new enforcer message.

Not changed

• kubernetes.md still references an apache/datafusion-comet:...-java11 image tag. No workflow in this repo publishes Comet Docker images anymore, so I did not switch it to a tag that may not exist.
• The remaining "JDK 11+" mentions describe JFR or JDK behavior, not Comet support.

#5885 edits the same Spark 3.4 warning lines in installation.md and spark-versions.md, so whichever PR lands second will need a small rebase.

How are these changes tested?

• ./mvnw help:effective-pom on JDK 21: maven.compiler.source and maven.compiler.target are both 17 with no profile, -Pspark-3.4, -Pspark-3.5 and -Pspark-4.0.
• ./mvnw -N validate on JDK 21: the require-java-17 rule passes. I did not check the failure path on JDK 11, because no JDK 11 was available locally.
• ./mvnw -Pspark-3.4 -Pscala-2.12 -pl common,spark -am -DskipTests test-compile on JDK 21 succeeds, and the enforcer rule runs in each module.
• dev/ci/check-ci-config.py, actionlint and prettier --check pass on the changed files.

Running Spark 3.4 on JDK 17 in the Spark SQL and Iceberg 1.8 jobs is only exercised in the merge queue or with the run-spark-3.4-tests / run-iceberg-tests labels. It would be good to add those labels to this PR.

🤖 Generated with Claude Code

[andygrove]

Thanks @manuzhang

[rich7420]

@manuzhang thanks for the patch