From da8835346a18f243b3580b5916606f97bea21391 Mon Sep 17 00:00:00 2001 From: vanshikajtg Date: Wed, 9 Sep 2026 14:15:16 +0530 Subject: [PATCH 1/2] fix(deps): bump pyspark to 3.5.5, closing CVE-2025-55039 Closes Aikido 304138193 (CVE-2025-55039, "Apache Spark has Inadequate Encryption Strength", sev 29/low). Patched in 3.5.2; going to 3.5.5 rather than 3.5.2 because that is the version this repo already tests against. This also fixes a real drift. Everything else in the repo is on 3.5.5 and only the Pipfile was left behind: spark.Dockerfile icanbwell/helix.spark:3.5.5.0-slim (Spark JVM 3.5.5) setup.py extra pyspark==3.5.5 Pipfile pyspark==3.5.1 <-- odd one out Pipfile.lock ==3.5.1 So the test environment has been running a 3.5.1 client against a 3.5.5 JVM, in violation of the pin's own comment ("should match the version of spark we use for testing"). Sibling packages agree with 3.5.5 too: SparkAutoMapper.FHIR, SparkDataFrameComparer and SparkFhirSchemas all pin ==3.5.5. Why this does not re-break what PR #180 (LR-503) fixed ------------------------------------------------------ #180 moved this pin 3.5.5 -> 3.5.1 to resolve a helix.pipelines conflict. That constraint is real but does not act through this file: - helix.pipelines pins `pyspark==3.5.1` directly in its own pyproject.toml (line 23) and its uv.lock resolves 3.5.1. Its pyspark comes from that pin, not from us. - It depends on `sparkautomapper==3.0.2` WITHOUT the `[spark]` extra, and this repo's pyspark lives in extras_require["spark"] -- so it never resolves pyspark through this package at all. - Pipfile.lock is not a published artifact (absent from MANIFEST.in and setup.py), so nothing downstream reads it. #180 changed only Pipfile and Pipfile.lock, leaving setup.py at 3.5.5. Since setup.py is the only one of the three a consumer can see, the revert could not have altered what helix.pipelines resolved. Note also the base image had already moved to 3.5.5.0-slim on 2025-05-15 (4d59b75), four months before #180 merged, so the mismatch has been latent since then. If a helix.pipelines conflict does resurface, the lever is setup.py's `spark` extra -- still 3.5.5, untouched here -- not this pin. py4j stays ==0.10.9.7: Spark's own python/setup.py pins exactly that for both v3.5.1 and v3.5.5, so no companion change is needed. The single sdist hash is verified against PyPI, and `_meta.hash` recomputed (ad3c856c -> a67976ab). Lock diff is 3 lines. VERSION -> 0.1.31. CI is the real test: no Docker or JVM available locally to run the suite against the new pyspark. Co-Authored-By: Claude Opus 5 (1M context) --- Pipfile | 9 ++++++++- Pipfile.lock | 6 +++--- VERSION | 2 +- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Pipfile b/Pipfile index c28be929..f9c50048 100644 --- a/Pipfile +++ b/Pipfile @@ -7,7 +7,14 @@ name = "pypi" # py4j is needed for connecting to the JVM py4j = "==0.10.9.7" # https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies # pyspark is needed for working with Spark -pyspark = "==3.5.1" # should match the version of spark we use for testing +# Must match the Spark JVM in spark.Dockerfile, which is +# icanbwell/helix.spark:3.5.5.0-slim -- i.e. Spark 3.5.5. This also closes +# CVE-2025-55039 (patched in 3.5.2). It was briefly 3.5.1 (PR #180, LR-503) to +# avoid a helix.pipelines conflict, but that constraint does not act through +# this file: helix.pipelines pins pyspark==3.5.1 in its own pyproject.toml and +# depends on `sparkautomapper==3.0.2` without the `[spark]` extra, so it never +# resolves pyspark through this package. Pipfile.lock is not published either. +pyspark = "==3.5.5" # should match the version of spark we use for testing logger = ">=1.4" sparkdataframecomparer = "==2.0.16" Deprecated = ">=1.2.12" diff --git a/Pipfile.lock b/Pipfile.lock index 72da67c4..5f323bdc 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "ad3c856ce083eb9e226ce8300323063c4961ee862468d32cf9e5372742dbe373" + "sha256": "a67976abb0fcefe502a03c782483a17a5bce1e0900f31267730d670582c4fc56" }, "pipfile-spec": 6, "requires": { @@ -115,11 +115,11 @@ }, "pyspark": { "hashes": [ - "sha256:dd6569e547365eadc4f887bf57f153e4d582a68c4b490de475d55b9981664910" + "sha256:6effc9ce98edf231f4d683fd14f7270629bf8458c628d6a2620ded4bb34f3cb9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==3.5.1" + "version": "==3.5.5" }, "sparkdataframecomparer": { "hashes": [ diff --git a/VERSION b/VERSION index 013adb71..db7a4804 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.1.30 +0.1.31 From fefaeb6ef0879251d92f6ea47e1ae9dd8247441a Mon Sep 17 00:00:00 2001 From: vanshikajtg Date: Wed, 9 Sep 2026 15:04:26 +0530 Subject: [PATCH 2/2] docs: correct the rationale for the pyspark pin Comment-only; `_meta.hash` unchanged and re-verified in sync. Verification refuted the reason the previous comment gave. It said consumers are unaffected because SAM's pyspark lives in `extras_require["spark"]`. That is only true from 3.0.4 onward (PR #178). helix.pipelines is pinned to `sparkautomapper==3.0.2`, whose published wheel METADATA declares `Requires-Dist: pyspark==3.5.1` with no `Provides-Extra` and no `; extra ==` marker -- i.e. unconditional install_requires. So SAM 3.0.2 *does* supply pyspark to helix.pipelines. The change is still safe, for a stronger reason the comment now gives instead: setup.py is byte-identical to tag 3.0.6, and neither Pipfile nor Pipfile.lock appears in the released sdist or wheel, so nothing a consumer resolves can change. Corroborating: SAM 3.0.6 was published 8 minutes after #180 merged and still declared pyspark==3.5.5, which makes #180 a provable no-op on published metadata. And 3.0.2 is immutable, so this branch cannot alter it. Also corrected: the base image at 4d59b75 was `imranq2/helix.spark:3.5.5.0-slim`, not `icanbwell/` -- that namespace only appeared 2026-09-08. And the comment now records *why* the JVM argument holds: helix.spark builds on `spark:3.5.5-java17-python3` with SPARK_HOME=/opt/spark, and pyspark honours SPARK_HOME over its bundled jars, so CI's Catalyst is 3.5.5 regardless of the pip pin. Pinning 3.5.1 created the skew rather than preventing it. Added a warning that `make pipenv-setup` (via `make update`) flattens [packages] into install_requires and would push this pin to consumers, undoing #178 -- that is how 3.0.3 shipped an unconditional pyspark==3.5.5 and triggered #180. No CI job runs it, so setup.py must stay hand-maintained. Two notes on the original commit message, not worth a rewrite but recorded here: the VERSION bump is cosmetic (python-publish.yml overwrites VERSION from the git tag before building, so the 0.1.x series is unrelated to the published 3.0.x); and while Aikido scores CVE-2025-55039 at 29/low, NVD rates it CVSS 3.1 6.5 MEDIUM. Real exposure here is still nil -- the flaw needs spark.network.crypto.enabled=true, which defaults false and is set nowhere. Co-Authored-By: Claude Opus 5 (1M context) --- Pipfile | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/Pipfile b/Pipfile index f9c50048..3e83267e 100644 --- a/Pipfile +++ b/Pipfile @@ -7,13 +7,30 @@ name = "pypi" # py4j is needed for connecting to the JVM py4j = "==0.10.9.7" # https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies # pyspark is needed for working with Spark -# Must match the Spark JVM in spark.Dockerfile, which is -# icanbwell/helix.spark:3.5.5.0-slim -- i.e. Spark 3.5.5. This also closes -# CVE-2025-55039 (patched in 3.5.2). It was briefly 3.5.1 (PR #180, LR-503) to -# avoid a helix.pipelines conflict, but that constraint does not act through -# this file: helix.pipelines pins pyspark==3.5.1 in its own pyproject.toml and -# depends on `sparkautomapper==3.0.2` without the `[spark]` extra, so it never -# resolves pyspark through this package. Pipfile.lock is not published either. +# Must match the Spark JVM the tests actually run against: spark.Dockerfile is +# icanbwell/helix.spark:3.5.5.0-slim, which builds on `spark:3.5.5-java17-python3` +# and sets SPARK_HOME=/opt/spark. pyspark honours SPARK_HOME over its bundled +# jars, so CI's Catalyst is Spark 3.5.5 either way -- pinning 3.5.1 here created +# a Python/JVM skew rather than preventing one. Also closes CVE-2025-55039 +# (patched in 3.5.2). +# +# PR #180 (LR-503) set this to 3.5.1 to avoid a helix.pipelines conflict. That +# is safe to undo because #180 never reached a consumer: it changed only Pipfile +# and Pipfile.lock, and neither is published (verified absent from the released +# sdist and wheel). SAM 3.0.6 was published 8 minutes after #180 merged and +# still declared pyspark==3.5.5. helix.pipelines is exact-pinned to +# sparkautomapper==3.0.2, an immutable artifact this cannot alter, and it pins +# pyspark==3.5.1 in its own pyproject.toml anyway. +# +# Do NOT justify this by "pyspark is in extras_require so consumers never see +# it" -- that only became true in 3.0.4 (PR #178). The 3.0.2 wheel that +# helix.pipelines consumes declares `Requires-Dist: pyspark==3.5.1` +# unconditionally. +# +# WARNING: `make pipenv-setup` (run by `make update`) flattens [packages] into +# setup.py's install_requires, which WOULD push this version to consumers and +# undo #178. That is how 3.0.3 shipped an unconditional pyspark==3.5.5. No CI +# job runs it, so keep setup.py hand-maintained. pyspark = "==3.5.5" # should match the version of spark we use for testing logger = ">=1.4" sparkdataframecomparer = "==2.0.16"