Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,9 @@ verify_ssl = true
name = "pypi"

[packages]
# py4j is needed for connecting to the JVM
py4j = "==0.10.9.7" # https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies
# pyspark is needed for working with Spark
pyspark = "==3.5.5" # should match the version of spark we use for testing
sparkautomapper = "==3.0.4"
pyspark = "==4.2.0" # should match the version of spark we use for testing
sparkautomapper = "==3.0.7"
sparkfhirschemas = "==2.0.4"
deprecated = "*"

Expand Down Expand Up @@ -45,7 +43,7 @@ jinja2 = ">=3.0.3"
pycln = ">=2.4.0"
lxml = ">=4.6.3"

sparkdataframecomparer = "==2.0.14"
sparkdataframecomparer = "==2.0.16"

# These dependencies are required for pipenv-setup. They conflict with ones above, so we install these
# only when running pipenv-setup
Expand Down
2,201 changes: 1,296 additions & 905 deletions Pipfile.lock

Large diffs are not rendered by default.

5 changes: 2 additions & 3 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,13 @@
url="https://github.com/icanbwell/SparkAutoMapper.FHIR",
packages=find_packages(),
install_requires=[
"sparkautomapper==3.0.4",
"sparkautomapper==3.0.7",
"sparkfhirschemas==2.0.4",
"deprecated",
],
extras_require={
"spark": [
"py4j==0.10.9.7",
"pyspark==3.5.5",
"pyspark==4.2.0",
]
},
classifiers=[
Expand Down
2 changes: 1 addition & 1 deletion spark.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM icanbwell/helix.spark:3.5.5.0-slim
FROM icanbwell/helix.spark:4.2.0.0-slim
# https://github.com/icanbwell/helix.spark
USER root

Expand Down
11 changes: 6 additions & 5 deletions spark_auto_mapper_fhir/Makefile.spark
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ export LANG
BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
VERSION=$(shell cat VERSION)
GIT_HASH=${CIRCLE_SHA1}
SPARK_VER=3.1.1
HADOOP_VER=3.2
SPARK_VER=4.2.0
# Spark 4.x only publishes a single "hadoop3" variant (no minor version)
HADOOP_VER=3

.PHONY:sdkman
sdkman:
Expand All @@ -16,12 +17,12 @@ sdkman:
.PHONY:java
java:
source "$(HOME)/.sdkman/bin/sdkman-init.sh" && \
sdk install java 11.0.8.hs-adpt || echo "java installed"
sdk install java 17.0.13-tem || echo "java installed"

.PHONY:scala
scala:
source "$(HOME)/.sdkman/bin/sdkman-init.sh" && \
sdk install scala 2.12.12 || echo "scala installed"
sdk install scala 2.13.15 || echo "scala installed"

.PHONY:brew
brew:
Comment on lines 22 to 28

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Contributors running make -f spark_auto_mapper_fhir/Makefile.spark spark (or installspark) now get a broken download_spark step because the URL is built from HADOOP_VER=3.4 (bumped from 3.2 at line 9), but Apache Spark distributions from 3.3 onward (including the 4.x line) only publish a generic hadoop3 build, not a versioned hadoop3.4 build. The old SPARK_VER=3.1.1/HADOOP_VER=3.2 combo at line 38 resolved to a real archive file; the new combo does not. Fix: build the tarball name to match what archive.apache.org actually publishes for SPARK_VER=4.2.0 (e.g. spark-4.2.0-bin-hadoop3.tgz), and add a curl -f/exit-code check so a bad URL fails the target loudly instead of writing an HTML error page into the tgz.

Extended reasoning...

Line 38 builds the URL as http://archive.apache.org/dist/spark/spark-$(SPARK_VER)/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz. With SPARK_VER=4.2.0 and HADOOP_VER=3.4 (lines 8-9) this resolves to spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark stopped publishing per-minor-Hadoop-version binaries after 3.2; 3.3+ and 4.x only ship a single 'hadoop3' variant, so this path 404s. sudo curl -o ... url has no -f flag, so curl exits 0 and writes the 404 HTML body to the .tgz path. The subsequent sudo tar -zxvf then fails on the corrupt archive, breaking make spark/make installspark for anyone provisioning Spark locally with this Makefile.

Verification: normal. Makefile.spark line 8-9 bumps SPARK_VER=4.2.0 and HADOOP_VER=3.4. Line 38 builds the download URL as spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz → spark-4.2.0-bin-hadoop3.4.tgz. Apache Spark dropped per-Hadoop-minor artifact naming after 3.2; 3.3+ and all 4.x releases publish only spark-<ver>-bin-hadoop3.tgz, so this URL 404s. The base combo (3.1.1/3.2) resolved to the real…

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already fixed in b0b8acd — HADOOP_VER changed from 3.4 to 3 (Spark 4.x only publishes hadoop3 builds) and curl -fL added so 404s fail loudly.

Expand All @@ -35,7 +36,7 @@ wget:

.PHONY:download_spark
download_spark:
sudo curl -o ${TMPDIR}/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz http://archive.apache.org/dist/spark/spark-$(SPARK_VER)/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz && \
sudo curl -fL -o ${TMPDIR}/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz http://archive.apache.org/dist/spark/spark-$(SPARK_VER)/spark-$(SPARK_VER)-bin-hadoop$(HADOOP_VER).tgz && \
sudo mkdir -p /usr/local/opt/spark && \
sudo rm -r /usr/local/opt/spark/ && \
sudo mkdir -p /usr/local/opt/spark && \
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/address.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.address import AddressSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/age.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.age import AgeSchema


if TYPE_CHECKING:
pass
# value (decimal)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/annotation.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.annotation import AnnotationSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/attachment.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.attachment import AttachmentSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/backbone_element.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@

from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/codeable_concept.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.codeableconcept import CodeableConceptSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/coding.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.coding import CodingSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/contact_detail.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.contactdetail import ContactDetailSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/contact_point.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.contactpoint import ContactPointSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/contributor.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.contributor import ContributorSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/count.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.count import CountSchema


if TYPE_CHECKING:
pass
# value (decimal)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/data_requirement.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.datarequirement import DataRequirementSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
DataRequirement_CodeFilterSchema,
)


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
DataRequirement_DateFilterSchema,
)


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
DataRequirement_SortSchema,
)


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/distance.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.distance import DistanceSchema


if TYPE_CHECKING:
pass
# value (decimal)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/duration.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.duration import DurationSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/element.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.element import ElementSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/expression.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.expression import ExpressionSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/human_name.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.humanname import HumanNameSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/identifier.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.identifier import IdentifierSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/meta.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.meta import MetaSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/money.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.money import MoneySchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/narrative.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.narrative import NarrativeSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
ParameterDefinitionSchema,
)


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/period.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.period import PeriodSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/quantity.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.quantity import QuantitySchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/range.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.range import RangeSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/ratio.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.ratio import RatioSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/reference.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.reference import ReferenceSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/related_artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.relatedartifact import RelatedArtifactSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/sampled_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.sampleddata import SampledDataSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/signature.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.signature import SignatureSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
TriggerDefinitionSchema,
)


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/complex_types/usage_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from spark_auto_mapper_fhir.base_types.fhir_complex_type_base import FhirComplexTypeBase
from spark_fhir_schemas.r4.complex_types.usagecontext import UsageContextSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
1 change: 0 additions & 1 deletion spark_auto_mapper_fhir/extensions/extension.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@

from spark_fhir_schemas.r4.complex_types.extension import ExtensionSchema


if TYPE_CHECKING:
pass
# id_ (string)
Expand Down
2 changes: 1 addition & 1 deletion tests/complex_types/test_period.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ def test_auto_mapper_hir_period_uses_date(spark_session: SparkSession) -> None:
assert isinstance(mapper, AutoMapper)
result_df: DataFrame = mapper.transform(df=df)
assert result_df
fhir_encounters_df = df.sql_ctx.table("fhir_encounters")
fhir_encounters_df = df.sparkSession.table("fhir_encounters")
assert isinstance(
fhir_encounters_df.select(fhir_encounters_df.period.start).collect()[0][0],
datetime.date,
Expand Down
2 changes: 2 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ def spark_session(request: Any) -> SparkSession:
.config("spark.sql.shuffle.partitions", "2")
.config("spark.default.parallelism", "4")
.config("spark.sql.broadcastTimeout", "2400")
.config("spark.sql.ansi.enabled", "false")
.enableHiveSupport()
.getOrCreate()
)
Expand All @@ -104,6 +105,7 @@ def spark_session_per_function(request: Any) -> SparkSession:
.config("spark.sql.shuffle.partitions", "2")
.config("spark.default.parallelism", "4")
.config("spark.sql.broadcastTimeout", "2400")
.config("spark.sql.ansi.enabled", "false")
.enableHiveSupport()
.getOrCreate()
)
Expand Down
Loading