Skip to content

chore: remove remaining Spark 3.4 compatibility code - #7625

Merged
manuzhang merged 4 commits into
apache:mainfrom
manuzhang:codex/remove-spark-34-dead-code
Nov 18, 2026
Merged

manuzhang merged 4 commits into
apache:mainfrom
manuzhang:codex/remove-spark-34-dead-code

Conversation

@manuzhang

@manuzhang manuzhang commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #4329.

Supersedes #5358 after #7521 removed the Spark 3.4 build profile and CI coverage and #7560 removed the Spark 3.4 shim source sets.

Rationale for this change

Spark 3.4 is no longer a supported build target, but compatibility branches, fallbacks, tests, generated diffs, and version-specific test assets remained in the codebase. Keeping those dead paths makes the supported Spark 3.5 and 4.x implementations harder to follow and maintain.

What changes are included in this PR?

  • Remove the remaining Spark 3.4 compatibility gates and fallback code from Comet, Delta, Iceberg, scan, shuffle, expression, and test paths.
  • Delete obsolete Spark 3.4 and Iceberg 1.8.1 generated diffs and Spark 3.4-only tests and shims.
  • Remove the unreachable get_json_object_spark34 native function, registration, Jackson-limit bypass, and redundant Scala selector while preserving the supported versions' JSON behavior.
  • Promote the Spark 3.5 TPC-DS plan baselines to the shared baseline and remove the now-redundant Spark 3.5 override files.
  • Rename CometExec3_4PlusSuite to CometExecCompatibilitySuite and update CI suite lists and active comments to describe the supported versions.

How are these changes tested?

Local validation after the latest rebase and review fixes:

  • ./mvnw -B test-compile -Pspark-3.5,delta -Pstrict-warnings -DskipTests
  • ./mvnw -B package scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb,spark-4.1 -Dtest=none -Dsuites=org.apache.comet.SqlFileTestParserSuite,org.apache.comet.serde.operator.CometNativeScanSuite (25 tests and semantic checks)
  • cargo +1.95.0 test --locked -p datafusion-comet-spark-expr --lib get_json_object (54 tests)
  • cargo fmt --all --check
  • python3 dev/ci/check-ci-config.py
  • python3 dev/ci/check-versioned-source-parity.py
  • python3 dev/ci/check-suites.py
  • git diff --check

Hosted validation uses the existing run-spark-3.5-tests, run-all-spark-profiles, and run-iceberg-tests labels. The prior-head Spark-profile, Spark 3.5 SQL/Hive, and Iceberg 1.9/1.10 runs passed; the same coverage is requested for the updated head.

@manuzhang manuzhang added the run-spark-3.5-tests Run the Spark 3.5 SQL tests on this PR label Nov 14, 2026
@github-actions github-actions Bot added enhancement New feature or request area:writer Native Parquet writer area:shuffle Shuffle (JVM and native) area:aggregation Hash aggregates, aggregate expressions area:scan Parquet scan / data reading area:expressions Expression evaluation area:Iceberg area:Delta Delta Lake contrib (native scan, deletion vectors) labels Nov 14, 2026
@andygrove andygrove added run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-iceberg-tests labels Nov 16, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @manuzhang, this is a nice cleanup. I went through the removed paths to check that each was only reachable on Spark 3.4. That covers the skip tags set by the deleted rule, the ParallelCollectionRDD branch (only 3.4's DataSourceV2ScanExecBase reports SinglePartition for one partition), CometBase64 (RuntimeReplaceable on 3.5), the no-op CTAS shims, and ReplaceIcebergData. They all check out. I also diffed the promoted TPC-DS baselines against the old -spark3_5 overrides, and they're identical.

Two suggestions:

  1. IcebergReflection.scala still says the fields are present "across the Iceberg 1.8.1 / 1.10.0 / 1.11.0 versions Comet supports". CI tests 1.9.1, 1.10.0 and 1.11.0, and the 1.8.1 diff is deleted here, so it might be worth making that list match what we test.
  2. The audit-comet-expression skill (and the references to it in implement-comet-expression and wire-datafusion-function) still studies Spark 3.4.3. Dropping v3.4.3 there would stop audits from comparing against a version we no longer support.

I've added run-all-spark-profiles and run-iceberg-tests, since the Iceberg reflection, Delta, and 3.5/4.0 plan-stability changes only run in the nightly otherwise.

@andygrove

Copy link
Copy Markdown
Member

@manuzhang could you fix conflicts?

@manuzhang
manuzhang force-pushed the codex/remove-spark-34-dead-code branch from eb869f4 to 26ebdff Compare November 17, 2026 03:10
@andygrove
andygrove enabled auto-merge November 17, 2026 13:42
@andygrove
andygrove added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 17, 2026
@manuzhang
manuzhang enabled auto-merge November 17, 2026 23:27

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging with current main conflicts only in dev/diffs/3.4.3.diff, which #4653 modified, so the resolution is to keep the deletion. I could not find any reference on main to the symbols this PR removes, including in the new native/local and comet/local code from #6883. One Spark 3.4 remnant is left: getJsonObjectNativeFunctionName returns "get_json_object" in all three CometExprShim files, so nothing selects get_json_object_spark34 anymore. The native spark_get_json_object_spark34, its registration in comet_scalar_funcs.rs, and the check_jackson_limits = false path in get_json_object.rs can be removed together with the shim method.

manuzhang and others added 4 commits November 18, 2026 11:11
Co-authored-by: Codex <codex@openai.com>
Remove the unused scan-rule import, preserve table-schema metadata in the supported-version test fixture, and correct the source-scoped Scala 2.12 nowarn exception. Normalize the remaining Delta test import.

Co-authored-by: Codex <codex@openai.com>
Match Iceberg reflection documentation to the tested version matrix and drop Spark 3.4.3 from expression audit skills and their callers.

Co-authored-by: Codex <codex@openai.com>
Remove the unreachable JSON compatibility function and redundant version selector while retaining Jackson limits for supported Spark versions. Drop the redundant expected error-class alias and stale compatibility references.

Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang force-pushed the codex/remove-spark-34-dead-code branch from 26ebdff to edd2ac5 Compare November 18, 2026 03:13
@manuzhang
manuzhang added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit 4d98e5d Nov 18, 2026
74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:aggregation Hash aggregates, aggregate expressions area:Delta Delta Lake contrib (native scan, deletion vectors) area:expressions Expression evaluation area:Iceberg area:scan Parquet scan / data reading area:shuffle Shuffle (JVM and native) area:writer Native Parquet writer enhancement New feature or request run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-iceberg-tests run-spark-3.5-tests Run the Spark 3.5 SQL tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants