Repository navigation
fix(deps): bump pyspark to 3.5.5, closing CVE-2025-55039 - #194
vanshikajtg wants to merge 2 commits into
Conversation
Closes Aikido 304138193 (CVE-2025-55039, "Apache Spark has Inadequate Encryption Strength", sev 29/low). Patched in 3.5.2; going to 3.5.5 rather than 3.5.2 because that is the version this repo already tests against. This also fixes a real drift. Everything else in the repo is on 3.5.5 and only the Pipfile was left behind: spark.Dockerfile icanbwell/helix.spark:3.5.5.0-slim (Spark JVM 3.5.5) setup.py extra pyspark==3.5.5 Pipfile pyspark==3.5.1 <-- odd one out Pipfile.lock ==3.5.1 So the test environment has been running a 3.5.1 client against a 3.5.5 JVM, in violation of the pin's own comment ("should match the version of spark we use for testing"). Sibling packages agree with 3.5.5 too: SparkAutoMapper.FHIR, SparkDataFrameComparer and SparkFhirSchemas all pin ==3.5.5. Why this does not re-break what PR #180 (LR-503) fixed ------------------------------------------------------ #180 moved this pin 3.5.5 -> 3.5.1 to resolve a helix.pipelines conflict. That constraint is real but does not act through this file: - helix.pipelines pins `pyspark==3.5.1` directly in its own pyproject.toml (line 23) and its uv.lock resolves 3.5.1. Its pyspark comes from that pin, not from us. - It depends on `sparkautomapper==3.0.2` WITHOUT the `[spark]` extra, and this repo's pyspark lives in extras_require["spark"] -- so it never resolves pyspark through this package at all. - Pipfile.lock is not a published artifact (absent from MANIFEST.in and setup.py), so nothing downstream reads it. #180 changed only Pipfile and Pipfile.lock, leaving setup.py at 3.5.5. Since setup.py is the only one of the three a consumer can see, the revert could not have altered what helix.pipelines resolved. Note also the base image had already moved to 3.5.5.0-slim on 2025-05-15 (4d59b75), four months before #180 merged, so the mismatch has been latent since then. If a helix.pipelines conflict does resurface, the lever is setup.py's `spark` extra -- still 3.5.5, untouched here -- not this pin. py4j stays ==0.10.9.7: Spark's own python/setup.py pins exactly that for both v3.5.1 and v3.5.5, so no companion change is needed. The single sdist hash is verified against PyPI, and `_meta.hash` recomputed (ad3c856c -> a67976ab). Lock diff is 3 lines. VERSION -> 0.1.31. CI is the real test: no Docker or JVM available locally to run the suite against the new pyspark. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment-only; `_meta.hash` unchanged and re-verified in sync. Verification refuted the reason the previous comment gave. It said consumers are unaffected because SAM's pyspark lives in `extras_require["spark"]`. That is only true from 3.0.4 onward (PR #178). helix.pipelines is pinned to `sparkautomapper==3.0.2`, whose published wheel METADATA declares `Requires-Dist: pyspark==3.5.1` with no `Provides-Extra` and no `; extra ==` marker -- i.e. unconditional install_requires. So SAM 3.0.2 *does* supply pyspark to helix.pipelines. The change is still safe, for a stronger reason the comment now gives instead: setup.py is byte-identical to tag 3.0.6, and neither Pipfile nor Pipfile.lock appears in the released sdist or wheel, so nothing a consumer resolves can change. Corroborating: SAM 3.0.6 was published 8 minutes after #180 merged and still declared pyspark==3.5.5, which makes #180 a provable no-op on published metadata. And 3.0.2 is immutable, so this branch cannot alter it. Also corrected: the base image at 4d59b75 was `imranq2/helix.spark:3.5.5.0-slim`, not `icanbwell/` -- that namespace only appeared 2026-09-08. And the comment now records *why* the JVM argument holds: helix.spark builds on `spark:3.5.5-java17-python3` with SPARK_HOME=/opt/spark, and pyspark honours SPARK_HOME over its bundled jars, so CI's Catalyst is 3.5.5 regardless of the pip pin. Pinning 3.5.1 created the skew rather than preventing it. Added a warning that `make pipenv-setup` (via `make update`) flattens [packages] into install_requires and would push this pin to consumers, undoing #178 -- that is how 3.0.3 shipped an unconditional pyspark==3.5.5 and triggered #180. No CI job runs it, so setup.py must stay hand-maintained. Two notes on the original commit message, not worth a rewrite but recorded here: the VERSION bump is cosmetic (python-publish.yml overwrites VERSION from the git tag before building, so the 0.1.x series is unrelated to the published 3.0.x); and while Aikido scores CVE-2025-55039 at 29/low, NVD rates it CVSS 3.1 6.5 MEDIUM. Real exposure here is still nil -- the flaw needs spark.network.crypto.enabled=true, which defaults false and is set nowhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Looks good — a straightforward, mechanical dependency version bump. Reviewed the Pipfile pyspark pin change (3.5.1 to 3.5.5), the corresponding Pipfile.lock hash/version updates, and the VERSION bump; no source, test, or doc logic changed, and the lock file hashes are internally consistent with the new version.
Extended reasoning...
Overview
The diff touches only three files: Pipfile (bumps pyspark from ==3.5.1 to ==3.5.5 and adds a long explanatory comment), Pipfile.lock (updates the pyspark hash/version entries and the top-level _meta.hash), and VERSION (0.1.30 to 0.1.31). No application source code, tests, or documentation logic was changed.
Security risks
The change itself introduces no new security risk — it is a dependency bump that, per the PR description, closes CVE-2025-55039 (a Spark encryption-strength issue that requires spark.network.crypto.enabled=true, which is not set anywhere in this repo). I was unable to reach PyPI to independently verify the new sha256 hash against the published pyspark 3.5.5 wheel/sdist due to network restrictions in this environment, but the hash strings are well-formed (64 hex characters) and the lock file structure is internally consistent (index, markers, and version fields all updated together).
Level of scrutiny
This warrants a light level of scrutiny: a pinned dependency version bump plus lock file regeneration and a version file increment is a mechanical, low-risk change with no logic modifications. There is no CODEOWNERS/ANTOWNERS file constraining who must review these paths, and the bug-hunting system reported no findings.
Other factors
The PR description provides substantial justification (CI Docker image already runs Spark 3.5.5 JVM, sibling packages already pin 3.5.5, private pyspark API surfaces used by this repo are byte-identical between 3.5.1 and 3.5.5) which is consistent with what's visible in the diff. There's no outstanding reviewer objection in the conversation timeline, and the change is small and self-contained enough that a human does not need to double-check it.
What this does
Bumps
pyspark==3.5.1→==3.5.5inPipfile/Pipfile.lock, closingAikido 304138193
(CVE-2025-55039, "Apache Spark has Inadequate Encryption Strength").
Patched in 3.5.2; going to 3.5.5 rather than 3.5.2 because that is the version this repo already tests against.
It also removes a real skew
Everything else in the repo was already on 3.5.5 — only the Pipfile was left behind:
spark.Dockerfileicanbwell/helix.spark:3.5.5.0-slim→ Spark JVM 3.5.5setup.pyextras_require["spark"]pyspark==3.5.5Pipfilepyspark==3.5.1← odd one outPipfile.lock==3.5.1helix.sparkbuilds onspark:3.5.5-java17-python3and setsSPARK_HOME=/opt/spark. pyspark honoursSPARK_HOMEover its bundled jars, so CI's Catalyst has been Spark 3.5.5 all along while the Python client was 3.5.1. This bump removes that mismatch rather than creating one. Sibling packages agree: SparkAutoMapper.FHIR, SparkDataFrameComparer and SparkFhirSchemas all pin==3.5.5.Why this is safe to undo PR #180
#180 ("LR-503 Reverting pyspark version to 3.5.1") moved this pin the other way to resolve a helix.pipelines conflict. That constraint does not act through these files:
PipfileandPipfile.lock, and neither is published — verified absent from the released sdist and wheel.setup.pyon this branch is byte-identical to tag3.0.6, so a release cut from here produces metadata indistinguishable from 3.0.6 for every consumer. The diff touches zero library source.pyspark==3.5.5. LR-503 Reverting pyspark version to 3.5.1 #180 was a provable no-op on published metadata.sparkautomapper==3.0.2— immutable, unaffected by this branch — and pinspyspark==3.5.1in its ownpyproject.tomlregardless.extras_requireso consumers never see it." That only became true in 3.0.4 (#178). The 3.0.2 wheel helix.pipelines consumes declaresRequires-Dist: pyspark==3.5.1unconditionally. The safety comes fromsetup.pybeing unchanged, not from the extra.Compatibility
Low risk. SAM reaches into private pyspark API in four places —
first_valid_column.py:60(_jc.expr().sql()),if_column_exists.py:76(_jc.toString()),automapper.py:232,284— andpython/pyspark/sql/column.pyis byte-identical between v3.5.1 and v3.5.5, so those bridges are provably unchanged.py4jstays==0.10.9.7: Spark's ownpython/setup.pypins exactly that for both versions.Pipfile.lockdiff is 3 lines. The single sdist sha256 is verified against PyPI, and_meta.hashwas recomputed (ad3c856c→a67976ab) withmainpassing as a control.Testing
CI is the real test — no Docker, JVM or Python 3.12 available locally, so the suite was not run here.
Worth a reviewer's attention: pyspark 3.5.5 + numpy 2.5.2 has never been CI'd together. #178 was green on 3.5.5 + numpy 1.x, #193 on 3.5.1 + numpy 2.5.2 — this exact pairing is new.
Note on severity
Aikido scores this 29/low; NVD rates it CVSS 3.1 6.5 MEDIUM. Practical exposure here is nil either way — the flaw needs
spark.network.crypto.enabled=true, which defaults false and is set nowhere in the repo. This is primarily about clearing the scanner and fixing the JVM alignment.Pre-existing hazard, not introduced here
make pipenv-setup(invoked bymake update) runspipenv-setup sync --pipfile, which flattensPipfile [packages]intoinstall_requires— that is how 3.0.3 shipped an unconditionalpyspark==3.5.5and triggered #180. No CI job runs it, sosetup.pymust stay hand-maintained. A warning is now recorded in the Pipfile. NotePipfile/setup.pyhave already drifted on numpy (>=2.5.2vs>=1.15, with helix.pipelines pinning<2), so this is worth a follow-up independent of pyspark.🤖 Generated with Claude Code