Skip to content

fix(deps): bump pyspark to 3.5.5, closing CVE-2025-55039 - #194

Open
vanshikajtg wants to merge 2 commits into
mainfrom
fix/aikido-pyspark-3.5.5
Open

vanshikajtg wants to merge 2 commits into
mainfrom
fix/aikido-pyspark-3.5.5

Conversation

@vanshikajtg

Copy link
Copy Markdown
Contributor

What this does

Bumps pyspark ==3.5.1 → ==3.5.5 in Pipfile / Pipfile.lock, closing
Aikido 304138193
(CVE-2025-55039, "Apache Spark has Inadequate Encryption Strength").

Patched in 3.5.2; going to 3.5.5 rather than 3.5.2 because that is the version this repo already tests against.

It also removes a real skew

Everything else in the repo was already on 3.5.5 — only the Pipfile was left behind:

version
spark.Dockerfile icanbwell/helix.spark:3.5.5.0-slim → Spark JVM 3.5.5
setup.py extras_require["spark"] pyspark==3.5.5
Pipfile pyspark==3.5.1 ← odd one out
Pipfile.lock ==3.5.1

helix.spark builds on spark:3.5.5-java17-python3 and sets SPARK_HOME=/opt/spark. pyspark honours SPARK_HOME over its bundled jars, so CI's Catalyst has been Spark 3.5.5 all along while the Python client was 3.5.1. This bump removes that mismatch rather than creating one. Sibling packages agree: SparkAutoMapper.FHIR, SparkDataFrameComparer and SparkFhirSchemas all pin ==3.5.5.

Why this is safe to undo PR #180

#180 ("LR-503 Reverting pyspark version to 3.5.1") moved this pin the other way to resolve a helix.pipelines conflict. That constraint does not act through these files:

  • LR-503 Reverting pyspark version to 3.5.1 #180 changed only Pipfile and Pipfile.lock, and neither is published — verified absent from the released sdist and wheel.
  • setup.py on this branch is byte-identical to tag 3.0.6, so a release cut from here produces metadata indistinguishable from 3.0.6 for every consumer. The diff touches zero library source.
  • SAM 3.0.6 was published 8 minutes after LR-503 Reverting pyspark version to 3.5.1 #180 merged and still declared pyspark==3.5.5. LR-503 Reverting pyspark version to 3.5.1 #180 was a provable no-op on published metadata.
  • helix.pipelines is exact-pinned to sparkautomapper==3.0.2 — immutable, unaffected by this branch — and pins pyspark==3.5.1 in its own pyproject.toml regardless.

⚠️ Note the tempting-but-wrong version of this argument: "pyspark is in extras_require so consumers never see it." That only became true in 3.0.4 (#178). The 3.0.2 wheel helix.pipelines consumes declares Requires-Dist: pyspark==3.5.1 unconditionally. The safety comes from setup.py being unchanged, not from the extra.

Compatibility

Low risk. SAM reaches into private pyspark API in four places — first_valid_column.py:60 (_jc.expr().sql()), if_column_exists.py:76 (_jc.toString()), automapper.py:232,284 — and python/pyspark/sql/column.py is byte-identical between v3.5.1 and v3.5.5, so those bridges are provably unchanged. py4j stays ==0.10.9.7: Spark's own python/setup.py pins exactly that for both versions.

Pipfile.lock diff is 3 lines. The single sdist sha256 is verified against PyPI, and _meta.hash was recomputed (ad3c856c → a67976ab) with main passing as a control.

Testing

CI is the real test — no Docker, JVM or Python 3.12 available locally, so the suite was not run here.

Worth a reviewer's attention: pyspark 3.5.5 + numpy 2.5.2 has never been CI'd together. #178 was green on 3.5.5 + numpy 1.x, #193 on 3.5.1 + numpy 2.5.2 — this exact pairing is new.

Note on severity

Aikido scores this 29/low; NVD rates it CVSS 3.1 6.5 MEDIUM. Practical exposure here is nil either way — the flaw needs spark.network.crypto.enabled=true, which defaults false and is set nowhere in the repo. This is primarily about clearing the scanner and fixing the JVM alignment.

Pre-existing hazard, not introduced here

make pipenv-setup (invoked by make update) runs pipenv-setup sync --pipfile, which flattens Pipfile [packages] into install_requires — that is how 3.0.3 shipped an unconditional pyspark==3.5.5 and triggered #180. No CI job runs it, so setup.py must stay hand-maintained. A warning is now recorded in the Pipfile. Note Pipfile/setup.py have already drifted on numpy (>=2.5.2 vs >=1.15, with helix.pipelines pinning <2), so this is worth a follow-up independent of pyspark.

🤖 Generated with Claude Code

vanshikajtg and others added 2 commits September 9, 2026 14:15
Closes Aikido 304138193 (CVE-2025-55039, "Apache Spark has Inadequate
Encryption Strength", sev 29/low).  Patched in 3.5.2; going to 3.5.5 rather
than 3.5.2 because that is the version this repo already tests against.

This also fixes a real drift.  Everything else in the repo is on 3.5.5 and
only the Pipfile was left behind:

  spark.Dockerfile   icanbwell/helix.spark:3.5.5.0-slim  (Spark JVM 3.5.5)
  setup.py extra     pyspark==3.5.5
  Pipfile            pyspark==3.5.1   <-- odd one out
  Pipfile.lock       ==3.5.1

So the test environment has been running a 3.5.1 client against a 3.5.5 JVM,
in violation of the pin's own comment ("should match the version of spark we
use for testing").  Sibling packages agree with 3.5.5 too: SparkAutoMapper.FHIR,
SparkDataFrameComparer and SparkFhirSchemas all pin ==3.5.5.

Why this does not re-break what PR #180 (LR-503) fixed
------------------------------------------------------
#180 moved this pin 3.5.5 -> 3.5.1 to resolve a helix.pipelines conflict.  That
constraint is real but does not act through this file:

- helix.pipelines pins `pyspark==3.5.1` directly in its own pyproject.toml
  (line 23) and its uv.lock resolves 3.5.1.  Its pyspark comes from that pin,
  not from us.
- It depends on `sparkautomapper==3.0.2` WITHOUT the `[spark]` extra, and this
  repo's pyspark lives in extras_require["spark"] -- so it never resolves
  pyspark through this package at all.
- Pipfile.lock is not a published artifact (absent from MANIFEST.in and
  setup.py), so nothing downstream reads it.

#180 changed only Pipfile and Pipfile.lock, leaving setup.py at 3.5.5.  Since
setup.py is the only one of the three a consumer can see, the revert could not
have altered what helix.pipelines resolved.  Note also the base image had
already moved to 3.5.5.0-slim on 2025-05-15 (4d59b75), four months before #180
merged, so the mismatch has been latent since then.

If a helix.pipelines conflict does resurface, the lever is setup.py's `spark`
extra -- still 3.5.5, untouched here -- not this pin.

py4j stays ==0.10.9.7: Spark's own python/setup.py pins exactly that for both
v3.5.1 and v3.5.5, so no companion change is needed.  The single sdist hash is
verified against PyPI, and `_meta.hash` recomputed (ad3c856c -> a67976ab).
Lock diff is 3 lines.

VERSION -> 0.1.31.  CI is the real test: no Docker or JVM available locally to
run the suite against the new pyspark.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment-only; `_meta.hash` unchanged and re-verified in sync.

Verification refuted the reason the previous comment gave. It said consumers are
unaffected because SAM's pyspark lives in `extras_require["spark"]`. That is
only true from 3.0.4 onward (PR #178). helix.pipelines is pinned to
`sparkautomapper==3.0.2`, whose published wheel METADATA declares
`Requires-Dist: pyspark==3.5.1` with no `Provides-Extra` and no `; extra ==`
marker -- i.e. unconditional install_requires. So SAM 3.0.2 *does* supply
pyspark to helix.pipelines.

The change is still safe, for a stronger reason the comment now gives instead:
setup.py is byte-identical to tag 3.0.6, and neither Pipfile nor Pipfile.lock
appears in the released sdist or wheel, so nothing a consumer resolves can
change. Corroborating: SAM 3.0.6 was published 8 minutes after #180 merged and
still declared pyspark==3.5.5, which makes #180 a provable no-op on published
metadata. And 3.0.2 is immutable, so this branch cannot alter it.

Also corrected: the base image at 4d59b75 was `imranq2/helix.spark:3.5.5.0-slim`,
not `icanbwell/` -- that namespace only appeared 2026-09-08. And the comment now
records *why* the JVM argument holds: helix.spark builds on
`spark:3.5.5-java17-python3` with SPARK_HOME=/opt/spark, and pyspark honours
SPARK_HOME over its bundled jars, so CI's Catalyst is 3.5.5 regardless of the
pip pin. Pinning 3.5.1 created the skew rather than preventing it.

Added a warning that `make pipenv-setup` (via `make update`) flattens [packages]
into install_requires and would push this pin to consumers, undoing #178 -- that
is how 3.0.3 shipped an unconditional pyspark==3.5.5 and triggered #180. No CI
job runs it, so setup.py must stay hand-maintained.

Two notes on the original commit message, not worth a rewrite but recorded here:
the VERSION bump is cosmetic (python-publish.yml overwrites VERSION from the git
tag before building, so the 0.1.x series is unrelated to the published 3.0.x);
and while Aikido scores CVE-2025-55039 at 29/low, NVD rates it CVSS 3.1 6.5
MEDIUM. Real exposure here is still nil -- the flaw needs
spark.network.crypto.enabled=true, which defaults false and is set nowhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — a straightforward, mechanical dependency version bump. Reviewed the Pipfile pyspark pin change (3.5.1 to 3.5.5), the corresponding Pipfile.lock hash/version updates, and the VERSION bump; no source, test, or doc logic changed, and the lock file hashes are internally consistent with the new version.

Extended reasoning...

Overview

The diff touches only three files: Pipfile (bumps pyspark from ==3.5.1 to ==3.5.5 and adds a long explanatory comment), Pipfile.lock (updates the pyspark hash/version entries and the top-level _meta.hash), and VERSION (0.1.30 to 0.1.31). No application source code, tests, or documentation logic was changed.

Security risks

The change itself introduces no new security risk — it is a dependency bump that, per the PR description, closes CVE-2025-55039 (a Spark encryption-strength issue that requires spark.network.crypto.enabled=true, which is not set anywhere in this repo). I was unable to reach PyPI to independently verify the new sha256 hash against the published pyspark 3.5.5 wheel/sdist due to network restrictions in this environment, but the hash strings are well-formed (64 hex characters) and the lock file structure is internally consistent (index, markers, and version fields all updated together).

Level of scrutiny

This warrants a light level of scrutiny: a pinned dependency version bump plus lock file regeneration and a version file increment is a mechanical, low-risk change with no logic modifications. There is no CODEOWNERS/ANTOWNERS file constraining who must review these paths, and the bug-hunting system reported no findings.

Other factors

The PR description provides substantial justification (CI Docker image already runs Spark 3.5.5 JVM, sibling packages already pin 3.5.5, private pyspark API surfaces used by this repo are byte-identical between 3.5.1 and 3.5.5) which is consistent with what's visible in the diff. There's no outstanding reviewer objection in the conversation timeline, and the change is small and self-contained enough that a human does not need to double-check it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant