Skip to content

bump: upgrade to PySpark 4.2.0 - #161

Merged
Tushar11Varshney merged 4 commits into
mainfrom
bump/pyspark-4.2.0
Sep 15, 2026
Merged

Tushar11Varshney merged 4 commits into
mainfrom
bump/pyspark-4.2.0

Conversation

@Tushar11Varshney

Copy link
Copy Markdown

Summary

  • PySpark 3.5.5 -> 4.2.0: Moved pyspark to extras_require["spark"] with version 4.2.0; removed standalone py4j pin (bundled in pyspark 4.x)
  • Ecosystem alignment: sparkautomapper 3.0.4 -> 3.0.7, sparkdataframecomparer 2.0.14 -> 2.0.16
  • Docker base image: helix.spark 3.5.5.0-slim -> 4.2.0.0-slim
  • Spark 4.x source compat: replaced deprecated .sql_ctx with .sparkSession, added spark.sql.ansi.enabled=false to test SparkSessions
  • Scala/Hadoop versions: Updated Makefile.spark from Scala 2.12 -> 2.13, Hadoop 3.2 -> 3.4

Part of the bwell Spark ecosystem PySpark 4.2.0 upgrade (SparkAutoMapper 3.0.7, SparkPipelineFramework 4.0.26, sparkdataframecomparer 2.0.16, sparkfhirschemas 2.0.4 already done).

Test plan

  • CI build passes with PySpark 4.2.0 Docker image
  • All existing unit tests pass (especially test_period.py which used removed .sql_ctx API)
  • Verify no remaining usages of removed Spark 4.x private APIs (_jdf, _jc, _sc)

🤖 Generated with Claude Code

- pyspark 3.5.5 -> 4.2.0, remove standalone py4j pin (bundled in pyspark 4.x)
- sparkautomapper 3.0.4 -> 3.0.7, sparkdataframecomparer 2.0.14 -> 2.0.16
- helix.spark base image 3.5.5.0-slim -> 4.2.0.0-slim
- Replace deprecated .sql_ctx with .sparkSession in test_period.py
- Add spark.sql.ansi.enabled=false to test SparkSessions for backward compat
- Update Makefile.spark: Scala 2.12 -> 2.13, Spark 3.1.1 -> 4.2.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@gecko-security gecko-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gecko PR Review

No vulnerabilities found, LGTM.

Summary

This PR upgrades PySpark from 3.5.5 to 4.2.0 in Pipfile and removes the explicit py4j dependency, along with bumping SparkAutoMapper to 3.0.7 and SparkDataFrameComparer to 2.0.16. The changes are primarily dependency updates with corresponding Pipfile.lock regeneration; however, PySpark 4.2.0 represents a major version jump that may introduce breaking changes in Spark API behavior, serialization, or JVM interop that could affect FHIR data mapping operations.

Important Files Changed

5 important files out of 7 changed

File Path Why it matters
Pipfile Removes py4j pinning and upgrades PySpark from 3.5.5 to 4.2.0—a major version bump that requires verification of API compatibility with SparkAutoMapper and FHIR schema generation.
spark.Dockerfile Base image imranq2/helix.spark:3.5.5.0-slim specifies Spark 3.5.5; upgrading PySpark to 4.2.0 in Pipfile may cause version mismatch between container Spark runtime and installed Python package.
tests/conftest.py Test configuration and fixtures must be verified to ensure they initialize Spark sessions and contexts correctly with PySpark 4.2.0's potentially different initialization behavior.
tests/complex_types/test_period.py FHIR data type test should be run against PySpark 4.2.0 to verify that period serialization, dataframe operations, and schema validation remain functional.
Pipfile.lock Regenerated lock file changes numpy and deprecated package versions; verify transitive dependency tree does not introduce security or compatibility issues with the Spark upgrade.

Omitted 2 lower-signal changed files.

Sequence Diagram

sequenceDiagram
  participant Reviewer
  participant PR as Changed Files
  Reviewer->>PR: Inspect changed files and control flow
  Reviewer->>Reviewer: Validate security assumptions
Loading

Edit Gecko PR Settings | Reviewed commit: bff39c5

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🔴 spark_auto_mapper_fhir/Makefile.spark — Contributors running make -f Makefile.spark spark now get Spark 4.2.0/Hadoop 3.4 but still Java 11 (sdk install java 11.0.8.hs-adpt at line 19), which Spark 4.x no longer supports (Java 17+ required). On the base branch Java 11 matched Spark 3.1.1 fine; now the JVM launch fails or PySpark raises at startup on any freshly provisioned machine using this target. Fix: bump the java target here (e.g. to a Java 17 adoptium/temurin version) to match SPARK_VER=4.2.0, the same way scala and SPARK_VER/HADOOP_VER were bumped.

    Extended reasoning...

    Makefile.spark line 8-9 bumps SPARK_VER to 4.2.0 and HADOOP_VER to 3.4, and line 24 bumps scala to 2.13.15, but the java target at lines 16-19 is untouched and still installs java 11.0.8.hs-adpt via sdkman. Apache Spark 4.0+ dropped support for Java 8/11 and requires Java 17 or newer. A developer who runs make -f spark_auto_mapper_fhir/Makefile.spark installspark (or spark) on a clean machine to set up a local Spark matching the new Docker image gets Java 11 installed, then spark-submit/pyspark with Spark 4.2.0 fails to start the JVM (UnsupportedClassVersionError or similar) because Spark 4.2.0 requires a newer JDK. This target is not invoked by the repo's own CI (.github/workflows), so it only breaks local/manual setup, but it's the file this diff explicitly edited for the version bump.

    Verification: normal. In spark_auto_mapper_fhir/Makefile.spark the spark target (line 49: spark: sdkman java scala brew wget download_spark) runs the java target, which at line 19 still executes sdk install java 11.0.8.hs-adpt. The same diff bumped the surrounding versions: SPARK_VER=4.2.0 (line 8), HADOOP_VER=3.4 (line 9), and scala 2.13.15 (line 24). Apache Spark 4.x requires Java 17+ (Java…

Comment on lines 21 to 27
.PHONY:scala
scala:
source "$(HOME)/.sdkman/bin/sdkman-init.sh" && \
sdk install scala 2.12.12 || echo "scala installed"
sdk install scala 2.13.15 || echo "scala installed"

.PHONY:brew
brew:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Contributors running make -f spark_auto_mapper_fhir/Makefile.spark spark (or installspark) now get a broken download_spark step because the URL is built from HADOOP_VER=3.4 (bumped from 3.2 at line 9), but Apache Spark distributions from 3.3 onward (including the 4.x line) only publish a generic hadoop3 build, not a versioned hadoop3.4 build. The old SPARK_VER=3.1.1/HADOOP_VER=3.2 combo at line 38 resolved to a real archive file; the new combo does not. Fix: build the tarball name to match what archive.apache.org actually publishes for SPARK_VER=4.2.0 (e.g. spark-4.2.0-bin-hadoop3.tgz), and add a curl -f/exit-code check so a bad URL fails the target loudly instead of writing an HTML error page into the tgz.

Extended reasoning...

Line 38 builds the URL as http://archive.apache.org/dist/spark/spark-$(SPARK_VER)/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz. With SPARK_VER=4.2.0 and HADOOP_VER=3.4 (lines 8-9) this resolves to spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark stopped publishing per-minor-Hadoop-version binaries after 3.2; 3.3+ and 4.x only ship a single 'hadoop3' variant, so this path 404s. sudo curl -o ... url has no -f flag, so curl exits 0 and writes the 404 HTML body to the .tgz path. The subsequent sudo tar -zxvf then fails on the corrupt archive, breaking make spark/make installspark for anyone provisioning Spark locally with this Makefile.

Verification: normal. Makefile.spark line 8-9 bumps SPARK_VER=4.2.0 and HADOOP_VER=3.4. Line 38 builds the download URL as spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz → spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark dropped per-Hadoop-minor artifact naming after 3.2; 3.3+ and all 4.x releases publish only spark-<ver>-bin-hadoop3.tgz, so this URL 404s. The base combo (3.1.1/3.2) resolved to the real…

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already fixed in b0b8acd — HADOOP_VER changed from 3.4 to 3 (Spark 4.x only publishes hadoop3 builds) and curl -fL added so 404s fail loudly.

Tushar11Varshney and others added 3 commits September 15, 2026 16:38
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…y), add curl -f

Spark 3.3+ and 4.x only publish a single "hadoop3" build variant, not
per-minor like "hadoop3.4". Also adds -fL to curl so a 404 fails loudly
instead of writing HTML into the tgz.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Tushar11Varshney
Tushar11Varshney merged commit d9cb8e5 into main Sep 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants