Repository navigation
bump: upgrade to PySpark 4.2.0 - #161
Conversation
- pyspark 3.5.5 -> 4.2.0, remove standalone py4j pin (bundled in pyspark 4.x) - sparkautomapper 3.0.4 -> 3.0.7, sparkdataframecomparer 2.0.14 -> 2.0.16 - helix.spark base image 3.5.5.0-slim -> 4.2.0.0-slim - Replace deprecated .sql_ctx with .sparkSession in test_period.py - Add spark.sql.ansi.enabled=false to test SparkSessions for backward compat - Update Makefile.spark: Scala 2.12 -> 2.13, Spark 3.1.1 -> 4.2.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Gecko PR Review
No vulnerabilities found, LGTM.
Summary
This PR upgrades PySpark from 3.5.5 to 4.2.0 in Pipfile and removes the explicit py4j dependency, along with bumping SparkAutoMapper to 3.0.7 and SparkDataFrameComparer to 2.0.16. The changes are primarily dependency updates with corresponding Pipfile.lock regeneration; however, PySpark 4.2.0 represents a major version jump that may introduce breaking changes in Spark API behavior, serialization, or JVM interop that could affect FHIR data mapping operations.
Important Files Changed
5 important files out of 7 changed
| File Path | Why it matters |
|---|---|
Pipfile |
Removes py4j pinning and upgrades PySpark from 3.5.5 to 4.2.0—a major version bump that requires verification of API compatibility with SparkAutoMapper and FHIR schema generation. |
spark.Dockerfile |
Base image imranq2/helix.spark:3.5.5.0-slim specifies Spark 3.5.5; upgrading PySpark to 4.2.0 in Pipfile may cause version mismatch between container Spark runtime and installed Python package. |
tests/conftest.py |
Test configuration and fixtures must be verified to ensure they initialize Spark sessions and contexts correctly with PySpark 4.2.0's potentially different initialization behavior. |
tests/complex_types/test_period.py |
FHIR data type test should be run against PySpark 4.2.0 to verify that period serialization, dataframe operations, and schema validation remain functional. |
Pipfile.lock |
Regenerated lock file changes numpy and deprecated package versions; verify transitive dependency tree does not introduce security or compatibility issues with the Spark upgrade. |
Omitted 2 lower-signal changed files.
Sequence Diagram
sequenceDiagram
participant Reviewer
participant PR as Changed Files
Reviewer->>PR: Inspect changed files and control flow
Reviewer->>Reviewer: Validate security assumptions
Edit Gecko PR Settings | Reviewed commit: bff39c5
There was a problem hiding this comment.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
spark_auto_mapper_fhir/Makefile.spark— Contributors runningmake -f Makefile.spark sparknow get Spark 4.2.0/Hadoop 3.4 but still Java 11 (sdk install java 11.0.8.hs-adptat line 19), which Spark 4.x no longer supports (Java 17+ required). On the base branch Java 11 matched Spark 3.1.1 fine; now the JVM launch fails or PySpark raises at startup on any freshly provisioned machine using this target. Fix: bump the java target here (e.g. to a Java 17 adoptium/temurin version) to match SPARK_VER=4.2.0, the same way scala and SPARK_VER/HADOOP_VER were bumped.Extended reasoning...
Makefile.spark line 8-9 bumps SPARK_VER to 4.2.0 and HADOOP_VER to 3.4, and line 24 bumps scala to 2.13.15, but the
javatarget at lines 16-19 is untouched and still installs java 11.0.8.hs-adpt via sdkman. Apache Spark 4.0+ dropped support for Java 8/11 and requires Java 17 or newer. A developer who runsmake -f spark_auto_mapper_fhir/Makefile.spark installspark(orspark) on a clean machine to set up a local Spark matching the new Docker image gets Java 11 installed, then spark-submit/pyspark with Spark 4.2.0 fails to start the JVM (UnsupportedClassVersionError or similar) because Spark 4.2.0 requires a newer JDK. This target is not invoked by the repo's own CI (.github/workflows), so it only breaks local/manual setup, but it's the file this diff explicitly edited for the version bump.Verification: normal. In spark_auto_mapper_fhir/Makefile.spark the
sparktarget (line 49:spark: sdkman java scala brew wget download_spark) runs thejavatarget, which at line 19 still executessdk install java 11.0.8.hs-adpt. The same diff bumped the surrounding versions:SPARK_VER=4.2.0(line 8),HADOOP_VER=3.4(line 9), andscala 2.13.15(line 24). Apache Spark 4.x requires Java 17+ (Java…
| .PHONY:scala | ||
| scala: | ||
| source "$(HOME)/.sdkman/bin/sdkman-init.sh" && \ | ||
| sdk install scala 2.12.12 || echo "scala installed" | ||
| sdk install scala 2.13.15 || echo "scala installed" | ||
|
|
||
| .PHONY:brew | ||
| brew: |
There was a problem hiding this comment.
🔴 Contributors running make -f spark_auto_mapper_fhir/Makefile.spark spark (or installspark) now get a broken download_spark step because the URL is built from HADOOP_VER=3.4 (bumped from 3.2 at line 9), but Apache Spark distributions from 3.3 onward (including the 4.x line) only publish a generic hadoop3 build, not a versioned hadoop3.4 build. The old SPARK_VER=3.1.1/HADOOP_VER=3.2 combo at line 38 resolved to a real archive file; the new combo does not. Fix: build the tarball name to match what archive.apache.org actually publishes for SPARK_VER=4.2.0 (e.g. spark-4.2.0-bin-hadoop3.tgz), and add a curl -f/exit-code check so a bad URL fails the target loudly instead of writing an HTML error page into the tgz.
Extended reasoning...
Line 38 builds the URL as http://archive.apache.org/dist/spark/spark-$(SPARK_VER)/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz. With SPARK_VER=4.2.0 and HADOOP_VER=3.4 (lines 8-9) this resolves to spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark stopped publishing per-minor-Hadoop-version binaries after 3.2; 3.3+ and 4.x only ship a single 'hadoop3' variant, so this path 404s. sudo curl -o ... url has no -f flag, so curl exits 0 and writes the 404 HTML body to the .tgz path. The subsequent sudo tar -zxvf then fails on the corrupt archive, breaking make spark/make installspark for anyone provisioning Spark locally with this Makefile.
Verification: normal. Makefile.spark line 8-9 bumps SPARK_VER=4.2.0 and HADOOP_VER=3.4. Line 38 builds the download URL as spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz → spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark dropped per-Hadoop-minor artifact naming after 3.2; 3.3+ and all 4.x releases publish only spark-<ver>-bin-hadoop3.tgz, so this URL 404s. The base combo (3.1.1/3.2) resolved to the real…
There was a problem hiding this comment.
Already fixed in b0b8acd — HADOOP_VER changed from 3.4 to 3 (Spark 4.x only publishes hadoop3 builds) and curl -fL added so 404s fail loudly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…y), add curl -f Spark 3.3+ and 4.x only publish a single "hadoop3" build variant, not per-minor like "hadoop3.4". Also adds -fL to curl so a 404 fails loudly instead of writing HTML into the tgz. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
pysparktoextras_require["spark"]with version 4.2.0; removed standalonepy4jpin (bundled in pyspark 4.x)sparkautomapper3.0.4 -> 3.0.7,sparkdataframecomparer2.0.14 -> 2.0.16helix.spark3.5.5.0-slim -> 4.2.0.0-slim.sql_ctxwith.sparkSession, addedspark.sql.ansi.enabled=falseto test SparkSessionsPart of the bwell Spark ecosystem PySpark 4.2.0 upgrade (SparkAutoMapper 3.0.7, SparkPipelineFramework 4.0.26, sparkdataframecomparer 2.0.16, sparkfhirschemas 2.0.4 already done).
Test plan
test_period.pywhich used removed.sql_ctxAPI)_jdf,_jc,_sc)🤖 Generated with Claude Code