Repository navigation
chore: remove remaining Spark 3.4 compatibility code - #7625
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks @manuzhang, this is a nice cleanup. I went through the removed paths to check that each was only reachable on Spark 3.4. That covers the skip tags set by the deleted rule, the ParallelCollectionRDD branch (only 3.4's DataSourceV2ScanExecBase reports SinglePartition for one partition), CometBase64 (RuntimeReplaceable on 3.5), the no-op CTAS shims, and ReplaceIcebergData. They all check out. I also diffed the promoted TPC-DS baselines against the old -spark3_5 overrides, and they're identical.
Two suggestions:
IcebergReflection.scalastill says the fields are present "across the Iceberg 1.8.1 / 1.10.0 / 1.11.0 versions Comet supports". CI tests 1.9.1, 1.10.0 and 1.11.0, and the 1.8.1 diff is deleted here, so it might be worth making that list match what we test.- The
audit-comet-expressionskill (and the references to it inimplement-comet-expressionandwire-datafusion-function) still studies Spark 3.4.3. Droppingv3.4.3there would stop audits from comparing against a version we no longer support.
I've added run-all-spark-profiles and run-iceberg-tests, since the Iceberg reflection, Delta, and 3.5/4.0 plan-stability changes only run in the nightly otherwise.
|
@manuzhang could you fix conflicts? |
eb869f4 to
26ebdff
Compare
comphead
left a comment
There was a problem hiding this comment.
Merging with current main conflicts only in dev/diffs/3.4.3.diff, which #4653 modified, so the resolution is to keep the deletion. I could not find any reference on main to the symbols this PR removes, including in the new native/local and comet/local code from #6883. One Spark 3.4 remnant is left: getJsonObjectNativeFunctionName returns "get_json_object" in all three CometExprShim files, so nothing selects get_json_object_spark34 anymore. The native spark_get_json_object_spark34, its registration in comet_scalar_funcs.rs, and the check_jackson_limits = false path in get_json_object.rs can be removed together with the shim method.
Co-authored-by: Codex <codex@openai.com>
Remove the unused scan-rule import, preserve table-schema metadata in the supported-version test fixture, and correct the source-scoped Scala 2.12 nowarn exception. Normalize the remaining Delta test import. Co-authored-by: Codex <codex@openai.com>
Match Iceberg reflection documentation to the tested version matrix and drop Spark 3.4.3 from expression audit skills and their callers. Co-authored-by: Codex <codex@openai.com>
Remove the unreachable JSON compatibility function and redundant version selector while retaining Jackson limits for supported Spark versions. Drop the redundant expected error-class alias and stale compatibility references. Co-authored-by: Codex <codex@openai.com>
26ebdff to
edd2ac5
Compare
Which issue does this PR close?
Part of #4329.
Supersedes #5358 after #7521 removed the Spark 3.4 build profile and CI coverage and #7560 removed the Spark 3.4 shim source sets.
Rationale for this change
Spark 3.4 is no longer a supported build target, but compatibility branches, fallbacks, tests, generated diffs, and version-specific test assets remained in the codebase. Keeping those dead paths makes the supported Spark 3.5 and 4.x implementations harder to follow and maintain.
What changes are included in this PR?
get_json_object_spark34native function, registration, Jackson-limit bypass, and redundant Scala selector while preserving the supported versions' JSON behavior.CometExec3_4PlusSuitetoCometExecCompatibilitySuiteand update CI suite lists and active comments to describe the supported versions.How are these changes tested?
Local validation after the latest rebase and review fixes:
./mvnw -B test-compile -Pspark-3.5,delta -Pstrict-warnings -DskipTests./mvnw -B package scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb,spark-4.1 -Dtest=none -Dsuites=org.apache.comet.SqlFileTestParserSuite,org.apache.comet.serde.operator.CometNativeScanSuite(25 tests and semantic checks)cargo +1.95.0 test --locked -p datafusion-comet-spark-expr --lib get_json_object(54 tests)cargo fmt --all --checkpython3 dev/ci/check-ci-config.pypython3 dev/ci/check-versioned-source-parity.pypython3 dev/ci/check-suites.pygit diff --checkHosted validation uses the existing
run-spark-3.5-tests,run-all-spark-profiles, andrun-iceberg-testslabels. The prior-head Spark-profile, Spark 3.5 SQL/Hive, and Iceberg 1.9/1.10 runs passed; the same coverage is requested for the updated head.