fix: spark connect extra class path

[Maleware]

Description

Spark-connect: adds missing class patch of spark-connect in exectuer

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added/passed
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[Maleware]

=== NAME  kuttl
    harness.go:430: run tests finished
    harness.go:547: cleaning up
    harness.go:604: removing temp folder: ""
--- PASS: kuttl (338.31s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-false (338.26s)
PASS

[lfrancke]

a "drive by" review. I didn't actually test the change but it looks sane to me in general. Thanks for catching.

[adwk67]

Looks good, but there are test failures for spark-connect, probably due to the assertion at step 14 as there is now SPARK_CLASSPATH included as an env-var.

[Maleware]

Hey,

to me all local tests succeed. Can you link the pipeline run not running? Might be it's just image pull backoffs or something.

[adwk67]

Hey,

to me all local tests succeed. Can you link the pipeline run not running? Might be it's just image pull backoffs or something.

I am getting (different from what I posted earlier) the following when the spark-connect app starts (on a fresh kind cluster):

Connecting to Spark Connect server at sc://spark-connect-server
Traceback (most recent call last):
  File "/stackable/spark/python/pyspark/sql/pandas/utils.py", line 69, in require_minimum_pyarrow_version
    import pyarrow
  File "/stackable/.local/lib/python3.12/site-packages/pyarrow/__init__.py", line 59, in <module>
    from pyarrow.lib import (BuildInfo, CppBuildInfo, RuntimeInfo, set_timezone_db_path,
  File "pyarrow/lib.pyx", line 42, in init pyarrow.lib
ImportError: pyarrow requires NumPy 2.0 or newer, found 1.26.4

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/example.py", line 47, in <module>
    .getOrCreate()
     ^^^^^^^^^^^^^
  File "/stackable/spark/python/pyspark/sql/session.py", line 502, in getOrCreate
    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
  File "/stackable/spark/python/pyspark/sql/connect/session.py", line 20, in <module>
    check_dependencies(__name__)
  File "/stackable/spark/python/pyspark/sql/connect/utils.py", line 36, in check_dependencies
    require_minimum_pyarrow_version()
  File "/stackable/spark/python/pyspark/sql/pandas/utils.py", line 76, in require_minimum_pyarrow_version
    raise PySparkImportError(
pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] PyArrow >= 15.0.0 must be installed; however, it was not found.
stream closed: EOF for kuttl-test-notable-thrush/simple-connect-app-zbmwc (simple-connect-app)

That pod is running with oci.stackable.tech/stackable/spark-connect-client:4.1.2-stackable0.0.0-dev

[Maleware]

Okay, I'm looking into it but so far it doesn't seem to be related to this specific PR. It seems like there are some different failures across tests which appear either flaky or lead to a successful state either way.

Which means, I'll go after it but it's unrelated.

[adwk67]

I ran them all and they all failed, so I think it's more than flakiness. It's not caused by this PR but it sort of blocks approving it, though:

--- FAIL: kuttl (4923.77s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-false (966.86s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-false (1022.93s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-true (823.44s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-true (935.64s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-false (809.59s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-false (758.47s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_s3-use-tls-true (756.88s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (766.92s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_s3-use-tls-false (739.42s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-false (738.21s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-true (753.05s)
        --- FAIL: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-true (766.49s)
FAIL

I'll have a look to see if it's something with my setup.

Update
I'm building a local image to get the test to work: then I'll approve this PR and create a new issue to fix the image.

[adwk67]

LGTM!
I made local changes to fix the library pin issue:

--- PASS: kuttl (579.36s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_s3-use-tls-true (338.30s)
        --- PASS: kuttl/harness/spark-connect-kerberos_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.2_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_s3-use-tls-true (241.04s)
PASS

I'll open an issue to address this in docker-images

[adwk67]

Issue to fix the library conflict: #786

Changes in