Skip to content

chore(spark): Update patch versions for 25.11.0 #1273

Closed
Closed
chore(spark): Update patch versions for 25.11.0#1273
Assignees
razvan
Labels
release-noteDenotes a PR that will be considered when it comes time to generate release notes.release/25.11.0

Description

@Techassi
Techassi
opened on Sep 25, 2025

Part of #1267.

  • Add: 3.5.7 (LTS)
  • Deprecate: 3.5.6
  • Remove: 3.5.5
  • Add: 4.0.1 (experimental)
  • Remove: 4.0.0

Tip

Please add the scheduled-for/25.11.0 label, and add to the Stackable Engineering project.

Update tasks

  • Update boil-config.toml to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
  • Upload new version (see spark/upload_new_spark_version.sh).
  • Update boil-config.toml to the latest supported version of JVM (base and devel).
  • Ensure that the transitive dependencies in boil-config.toml are compatible with the Spark version(s).
  • Update other dependencies if applicable (eg: python, jmx_exporter, aws_java_sdk_bundle, etc).
  • Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below.
  • Update the version in demos. Add the PR(s) to the list below.

JDK Notes

Spark 3.5.7 still builds and runs on JDK 17.

This is because:

Related Pull Requests

Tip

Delete any items that do not apply so that all applicable items can be checked.
For example, if you add release notes to the documentation repository, you do not need the latter two criteria.

Acceptance

Tip

This list should be completed by the assignee(s), once respective PRs have been merged. Once all items have been
checked, the issue can be moved into Development: Done.

  • Can build image (either locally, or in CI)
  • Kuttl smoke tests passes (either locally, or in CI)
  • Release notes added to documentation and linked as a PR above
  • Release notes written in a comment below
  • Applicable release-note label added to this issue
Testing instructions 
# Where x.y.z is the new version added in this PR
boil build spark-k8s=x.y.z --strip-architecture --load

kind load docker-image oci.stackable.tech/sdp/spark-k8s:x.y.z-stackable0.0.0-dev

# Change directory into the spark-k8s-operator repository and update the
# product version in tests/test-definition.yaml
./scripts/run-tests --test-suite smoke-latest # or similar

Please consider updating this template if these instructions are wrong, or
could be made clearer.

Activity

  1. Techassi
    added
    scheduled-for/25.11.0
    on Sep 25, 2025
  2. razvan
    self-assigned this
    on Sep 25, 2025
  3. razvan
    added this to Stackable Engineeringon Sep 26, 2025
  4. razvan
    moved this to Development: In Progress in Stackable Engineeringon Sep 26, 2025

  5. razvan commented on Sep 30, 2025

    @razvan
    razvan
    on Sep 30, 2025
    Member

    New versions

    The following product versions have been added:

    Apache Spark 3.5.7
    Apache Spark 4.0.1

    Deprecated versions

    The following product versions have been deprecated:

    Apache Spark 3.5.6

    Removed versions

    The following product versions have been removed:

    Apache Spark 3.5.5
    Apache Spark 4.0.0

  6. razvan
    moved this from Development: In Progress to Development: Waiting for Review in Stackable Engineeringon Sep 30, 2025
  7. razvan
    moved this from Development: Waiting for Review to Development: Done in Stackable Engineeringon Sep 30, 2025
  8. razvan
    closed this as completedon Sep 30, 2025

  9. sbernauer commented on Oct 2, 2025

    @sbernauer
    sbernauer
    on Oct 2, 2025
    Member

    @razvan I just noticed that we should bump to HDFS 3.4.2 in

    docker-images/spark-k8s/boil-config.toml

    Line 40 in a9fea67

    "hadoop/hadoop" = "3.4.1"

    Watch out, this might cause problems because of AWS libs removal. Currently figuring this out for Hive.

    The versions in

    docker-images/spark-k8s/boil-config.toml

    Lines 48 to 53 in a9fea67

    aws-java-sdk-bundle-version = "2.24.6"
    azure-storage-version = "7.0.1" # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.4
    azure-keyvault-core-version = "1.0.0" # https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
    jackson-dataformat-xml-version = "2.15.2" # https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.5.1
    stax2-api-version = "4.2.1" # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
    woodstox-core-version = "6.5.1" # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2

    also need updating (the comments and the versions) as they are currently wrong

  10. sbernauer
    moved this from Development: Done to Development: In Progress in Stackable Engineeringon Oct 2, 2025
  11. sbernauer
    reopened this on Oct 2, 2025
  12. Techassi
    added
    release-noteDenotes a PR that will be considered when it comes time to generate release notes.
    on Oct 9, 2025
  13. razvan
    closed this as completedon Oct 9, 2025
  14. razvan
    moved this from Development: In Progress to Development: Done in Stackable Engineeringon Oct 9, 2025
  15. lfrancke
    added
    release/25.11.0
    and removed
    scheduled-for/25.11.0
    on Oct 14, 2025
  16. lfrancke
    moved this from Development: Done to Done in Stackable Engineeringon Oct 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

release-noteDenotes a PR that will be considered when it comes time to generate release notes.release/25.11.0

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions