diff --git a/.ai/skills/audit-comet-expression/SKILL.md b/.ai/skills/audit-comet-expression/SKILL.md index e864d23fa88..f0435c0593f 100644 --- a/.ai/skills/audit-comet-expression/SKILL.md +++ b/.ai/skills/audit-comet-expression/SKILL.md @@ -1,6 +1,6 @@ --- name: audit-comet-expression -description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests. +description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests. argument-hint: --- @@ -29,7 +29,7 @@ Audit the Comet implementation of the `$ARGUMENTS` expression for correctness an This audit covers: -1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1 +1. Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1 2. Comet Scala serde implementation 3. Comet Rust / DataFusion implementation 4. Existing test coverage (Comet SQL Tests and Comet Scala Tests) @@ -43,7 +43,7 @@ Clone specific Spark version tags (use shallow clones to avoid polluting the wor ```bash set -eu -o pipefail -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" if [ ! -d "$dir" ]; then git clone --depth 1 --branch "$tag" https://github.com/apache/spark.git "$dir" @@ -56,7 +56,7 @@ done Search the Catalyst SQL expressions source: ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql/catalyst/src/main/scala" -name "*.scala" | \ @@ -67,7 +67,7 @@ done If the expression is not found in catalyst, also check core: ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql" -name "*.scala" | \ @@ -90,7 +90,6 @@ For each Spark version, read the expression file and note: Produce a concise diff summary of what changed between: -- 3.4.3 → 3.5.8 - 3.5.8 → 4.0.1 - 4.0.1 → 4.1.1 @@ -107,7 +106,7 @@ Pay attention to: ## Step 2: Locate the Spark Tests ```bash -for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do +for tag in v3.5.8 v4.0.1 v4.1.1; do dir="/tmp/spark-${tag}" echo "=== $tag ===" find "$dir/sql" -name "*.scala" -path "*/test/*" | \ @@ -732,7 +731,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`. other pages. - Add (or update) a `## ` section, keeping sections alphabetically ordered. - Under that heading, add one bullet per Spark version checked, each including: - - Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1) + - Spark version (e.g. 3.5.8, 4.0.1, 4.1.1) - Today's date - A brief note for any version-specific finding (behavioral difference, known incompatibility); omit the note if nothing notable. @@ -744,7 +743,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`. Present the audit as: 1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input/output types, and null behavior -2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 +2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.5.8, 4.0.1, and 4.1.1 3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns 4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps 5. **Recommendations** - Prioritized list from Step 6 diff --git a/.ai/skills/implement-comet-expression/SKILL.md b/.ai/skills/implement-comet-expression/SKILL.md index 1ee9eff091f..85daf081176 100644 --- a/.ai/skills/implement-comet-expression/SKILL.md +++ b/.ai/skills/implement-comet-expression/SKILL.md @@ -89,7 +89,7 @@ make ### 4. Run the audit skill -Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.4.3, 3.5.8, and 4.0.1 and produces a prioritized list of gaps. +Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.5.8, 4.0.1, and 4.1.1 and produces a prioritized list of gaps. ### 5. Implement audit-recommended tests and iterate diff --git a/.ai/skills/review-comet-pr/SKILL.md b/.ai/skills/review-comet-pr/SKILL.md index 988ba875f17..2c6bb6f0db3 100644 --- a/.ai/skills/review-comet-pr/SKILL.md +++ b/.ai/skills/review-comet-pr/SKILL.md @@ -183,8 +183,8 @@ such as `wrap_second_millisecond_overflow`, never the version, such as `spark_42 fork that backports a Spark fix then changes the one Scala check that sets the flag and leaves the protobuf and Rust alone. Flag a version-named field, argument, or function, and comments that say "Spark 4.2" for behavior later releases inherit or that leave out the Spark JIRA. -`Cast.is_spark4_plus` and `get_json_object_spark34` predate the rule and #6891 tracks renaming them, -so flag them only when a PR adds behavior behind them. +`Cast.is_spark4_plus` predates the rule and #6891 tracks renaming it, +so flag it only when a PR adds behavior behind it. When Spark changed the behavior in a patch release, such as SPARK-55969 or SPARK-54918, a check on the minor version is wrong for every earlier patch. CI builds only the newest patch of each line, so diff --git a/.ai/skills/wire-datafusion-function/SKILL.md b/.ai/skills/wire-datafusion-function/SKILL.md index 4688f076c15..0cfc6e0e313 100644 --- a/.ai/skills/wire-datafusion-function/SKILL.md +++ b/.ai/skills/wire-datafusion-function/SKILL.md @@ -141,7 +141,7 @@ make cd native && cargo clippy --all-targets --workspace -- -D warnings ``` -Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 / 3.5.8 / 4.0.1 and surface coverage gaps; iterate on tests. +Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.5.8 / 4.0.1 / 4.1.1 and surface coverage gaps; iterate on tests. The user generally runs tests themselves. If asked for a smoke test: diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml index f13101deeeb..3c48f07ae74 100644 --- a/.github/workflows/pr_build_linux.yml +++ b/.github/workflows/pr_build_linux.yml @@ -582,7 +582,7 @@ jobs: - name: "exec" value: | org.apache.comet.exec.CometAggregateSuite - org.apache.comet.exec.CometExec3_4PlusSuite + org.apache.comet.exec.CometExecCompatibilitySuite org.apache.comet.exec.CometExecSuite org.apache.comet.exec.CometMergeRowsSuite org.apache.comet.exec.CometTopKSuite diff --git a/.github/workflows/pr_build_macos.yml b/.github/workflows/pr_build_macos.yml index ee13480830e..eae6aef64ad 100644 --- a/.github/workflows/pr_build_macos.yml +++ b/.github/workflows/pr_build_macos.yml @@ -220,7 +220,7 @@ jobs: - name: "exec" value: | org.apache.comet.exec.CometAggregateSuite - org.apache.comet.exec.CometExec3_4PlusSuite + org.apache.comet.exec.CometExecCompatibilitySuite org.apache.comet.exec.CometExecSuite org.apache.comet.exec.CometMergeRowsSuite org.apache.comet.exec.CometTopKSuite diff --git a/benchmarks/README.md b/benchmarks/README.md index 0b055900946..9cbd64e59d4 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -62,7 +62,7 @@ docker push localhost:32000/apache/datafusion-comet-tpcbench:latest export SPARK_MASTER=k8s://https://127.0.0.1:16443 export COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest # Location of Comet JAR within the Docker image -export COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar +export COMET_JAR=/opt/spark/jars/comet-spark-spark3.5_2.12-0.5.0-SNAPSHOT.jar $SPARK_HOME/bin/spark-submit \ --master $SPARK_MASTER \ diff --git a/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala b/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala index 96c18a85de0..15684b47c06 100644 --- a/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala +++ b/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala @@ -435,10 +435,9 @@ object DeltaScanSupport { return providerReason } - // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on - // Spark 3.4, exec enabled, existence default values, the Variant read confs, and a proto - // representation for every serialized data and partition type); tags its own fallback - // reasons. + // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, exec enabled, + // existence default values, the Variant read confs, and a proto representation for every + // serialized data and partition type); tags its own fallback reasons. if (!CometNativeScan.isSupported(scanExec)) { return Some("Core native scan gates rejected the scan (see reasons above)") } diff --git a/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala b/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala index 766fc4a3c8f..a3c5830787c 100644 --- a/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala +++ b/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala @@ -38,7 +38,7 @@ import org.apache.spark.sql.types.{ByteType, LongType, StringType, StructField, import org.apache.spark.sql.util.QueryExecutionListener import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.ExtendedExplainInfo import org.apache.comet.serde.OperatorOuterClass import org.apache.comet.serde.operator.CometNativeScan @@ -1751,7 +1751,6 @@ class CometDeltaNativeScanSuite extends CometDeltaTestBase { test( "DPP stays executable when transition reversion restores the delta scan: " + s"AQE=$adaptive") { - assume(isSpark35Plus, "Comet AQE DPP query-stage optimizer rules require Spark 3.5+") withTempPath { factDir => withTempPath { dimDir => val factPath = factDir.getAbsolutePath diff --git a/dev/diffs/3.4.3.diff b/dev/diffs/3.4.3.diff deleted file mode 100644 index faf550c0809..00000000000 --- a/dev/diffs/3.4.3.diff +++ /dev/null @@ -1,3448 +0,0 @@ -diff --git a/pom.xml b/pom.xml -index d3544881af1..f079228da3d 100644 ---- a/pom.xml -+++ b/pom.xml -@@ -148,6 +148,8 @@ - 0.10.0 - 2.5.1 - 2.0.8 -+ 3.4 -+ 1.3.0-SNAPSHOT - - -Wconf:cat=unused-nowarn&site=org\.apache\.spark\.sql\.comet\.execution\.shuffle\.Comet(Celeborn)?ShuffleManager\.get(Reader|Writer):s + + -Wconf:cat=unused-nowarn&src=.*/org/apache/spark/sql/comet/execution/shuffle/Comet(Celeborn)?ShuffleManager\.scala:s -Wconf:cat=unused-params&site=org\.apache\.spark\.sql\.comet\.CometIcebergNativeScanExec\.apply\.session:s -Wconf:cat=unused-params&site=org\.apache\.spark\.sql\.comet\.CometNativeScanExec\.apply\.session:s @@ -1062,8 +1063,9 @@ under the License. -Wconf:cat=unused-params&site=org\.apache\.comet\.udf\.CometUDF\.evaluate\.(partitionIndex|planId):s -Wconf:cat=unused-params&site=org\.apache\.comet\.udf\.CometUDF\.releasePlan\.planId:s -Wconf:cat=unused-params&site=org\.apache\.spark\.sql\.comet\.CometBaseAggregate\.operatorSupportLevel\.op:s - - -Wconf:cat=unused-nowarn&site=org\.apache\.spark\.sql\.comet\.execution\.shuffle\.Comet(Celeborn)?ShuffleManager\.get(Reader|Writer):s + + -Wconf:cat=unused-nowarn&src=.*/org/apache/spark/sql/comet/execution/shuffle/Comet(Celeborn)?ShuffleManager\.scala:s -Wconf:cat=unused-params&site=org\.apache\.spark\.sql\.comet\.CometIcebergNativeScanExec\.apply\.session:s -Wconf:cat=unused-params&site=org\.apache\.spark\.sql\.comet\.CometNativeScanExec\.apply\.session:s diff --git a/spark/src/main/scala/org/apache/comet/CometConf.scala b/spark/src/main/scala/org/apache/comet/CometConf.scala index a4e54555246..4f0e022b52d 100644 --- a/spark/src/main/scala/org/apache/comet/CometConf.scala +++ b/spark/src/main/scala/org/apache/comet/CometConf.scala @@ -479,9 +479,9 @@ object CometConf extends ShimCometConf { "mergeRows", defaultValue = false, notes = Some( - "Only takes effect on Spark 3.5 and later. Spark 3.4 has no MergeRowsExec. On Spark " + - "4.1 and later, stock V2 writers retain Spark MergeRowsExec for MergeSummary; native " + - "MergeRows is used only where the enclosing Comet write path preserves that contract.")) + "On Spark 4.1 and later, stock V2 writers retain Spark MergeRowsExec for MergeSummary; " + + "native MergeRows is used only where the enclosing Comet write path preserves that " + + "contract.")) val COMET_EXEC_IN_MEMORY_CACHE_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.exec.inMemoryCache.enabled") @@ -491,9 +491,7 @@ object CometConf extends ShimCometConf { "Requires spark.comet.enabled=true. On Spark 4.0 and later, it also decides how an " + "adaptive cached plan is built: when the final stage ends in a Spark row operator, " + "Comet writes the rows straight into Arrow batches for the cache instead of " + - "having Spark fill its own column vectors first. On Spark 3.4, AQE cannot coalesce the " + - "shuffle partitions of a union that Comet runs with a cached relation in one branch. " + - "At startup, this setting also decides " + + "having Spark fill its own column vectors first. At startup, this setting also decides " + "whether CometDriverPlugin installs Comet's cache serializer, which stores cached " + "data in Arrow format. The plugin installs it only if spark.comet.enabled and " + "spark.comet.exec.enabled are also enabled at startup, and only with one of Comet's " + diff --git a/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala b/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala index de923f45511..ec800e30f4d 100644 --- a/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala +++ b/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala @@ -34,7 +34,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.comet.CometConf._ import org.apache.comet.iceberg.IcebergWriteStrategy -import org.apache.comet.rules.{CometCoalesceShufflePartitions, CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery, CometRule, CometSpark34AqeDppFallbackRule} +import org.apache.comet.rules.{CometCoalesceShufflePartitions, CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery, CometRule} import org.apache.comet.shims.ShimCometSparkSessionExtensions /** @@ -59,7 +59,7 @@ import org.apache.comet.shims.ShimCometSparkSessionExtensions * 5. ReuseExchangeAndSubquery -- Spark deduplicates subqueries (sees Comet nodes) * }}} * - * AQE (AdaptiveSparkPlanExec, Spark 3.5+): + * AQE (AdaptiveSparkPlanExec): * {{{ * Initial plan: * PlanAdaptiveSubqueries: creates SubqueryAdaptiveBroadcastExec (SAB) for AQE DPP @@ -86,11 +86,6 @@ import org.apache.comet.shims.ShimCometSparkSessionExtensions * EliminateRedundantTransitions, CometCacheBuildRule, * CometCacheColumnarRule * }}} - * - * On Spark 3.4, injectQueryStageOptimizerRule is unavailable. CometExecRule does not wrap SABs, - * and CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery and - * CometCoalesceShufflePartitions are not registered. AQE DPP scans fall back to Spark so that - * Spark's PlanAdaptiveDynamicPruningFilters handles them natively (with DPP). */ class CometSparkSessionExtensions extends (SparkSessionExtensions => Unit) @@ -105,10 +100,6 @@ class CometSparkSessionExtensions return } extensions.injectColumnar { session => CometColumnar(session) } - // Pre-3.5 only: tag AQE DPP regions so the conversion rules below leave them Spark-native. - // Registered before CometRule so tags are in place when conversion runs. - // No-op on Spark 3.5+; see CometSpark34AqeDppFallbackRule's class docstring. - injectPreSpark35QueryStagePrepRuleShim(extensions, CometSpark34AqeDppFallbackRule) extensions.injectQueryStagePrepRule { session => CometRule(session, queryStagePrep = true) } @@ -278,10 +269,6 @@ object CometSparkSessionExtensions extends Logging { op.isInstanceOf[CometBatchScanExec] || op.isInstanceOf[CometScanExec] } - def isSpark35Plus: Boolean = { - org.apache.spark.SPARK_VERSION >= "3.5" - } - def isSpark40Plus: Boolean = { org.apache.spark.SPARK_VERSION >= "4.0" } diff --git a/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala b/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala index 58278b488aa..4f252558377 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala @@ -74,9 +74,6 @@ object IcebergReflection extends Logging { val MANIFEST_FILE = "org.apache.iceberg.ManifestFile" val MANIFEST_FILES = "org.apache.iceberg.ManifestFiles" val DATA_FILE = "org.apache.iceberg.DataFile" - - // Iceberg 1.5.2 uses its own `ReplaceIcebergData` due to lack of `ReplaceData` in Spark 3.4. - val REPLACE_ICEBERG_DATA = "org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData" } /** @@ -238,9 +235,6 @@ object IcebergReflection extends Logging { } } - def isReplaceIcebergData(plan: Any): Boolean = - plan != null && plan.getClass.getName == ClassNames.REPLACE_ICEBERG_DATA - private def reflectField(plan: Any, fieldName: String): Option[AnyRef] = try { val field = plan.getClass.getDeclaredField(fieldName) @@ -253,19 +247,6 @@ object IcebergReflection extends Logging { None } - def extractReplaceIcebergDataFields(plan: Any): Option[(AnyRef, AnyRef, AnyRef, AnyRef)] = { - if (!isReplaceIcebergData(plan)) return None - for { - table <- reflectField(plan, "table") - query <- reflectField(plan, "query") - originalTable <- reflectField(plan, "originalTable") - write <- reflectField( - plan, - "write" - ) // Option[Write]; field can be Some(null) so kept AnyRef - } yield (table, query, originalTable, write) - } - /** * Loads a class using the thread context classloader first, then falls back to the system * classloader. @@ -1437,8 +1418,8 @@ object IcebergReflection extends Logging { /** * Reads a `private final` field from a `SparkWrite` instance via reflection. The field names * referenced by callers (`queryId`, `targetFileSize`, `useFanoutWriter`, `outputSpecId`, - * `writeSchema`) are present on `SparkWrite` across the Iceberg 1.5.2 / 1.8.1 / 1.10.0 versions - * Comet supports. + * `writeSchema`) are present on `SparkWrite` across the Iceberg 1.9.1 / 1.10.0 / 1.11.0 + * versions Comet supports. */ private def getSparkWriteField(sparkWrite: Any, fieldName: String): Option[Any] = sparkWriteClassOpt.flatMap { cls => @@ -1447,11 +1428,7 @@ object IcebergReflection extends Logging { field.setAccessible(true) Option(field.get(sparkWrite)) } catch { - case _: NoSuchFieldException => - // Field may have been renamed across Iceberg versions. Callers that probe multiple - // candidate names (e.g. `useFanoutWriter` / `partitionedFanoutEnabled`) should expect - // this and `.orElse` onto the alternative. - None + case _: NoSuchFieldException => None case e: Exception => logError( s"Iceberg reflection failure: Failed to read SparkWrite.$fieldName: ${e.getMessage}") @@ -1471,16 +1448,10 @@ object IcebergReflection extends Logging { /** * Whether the planner would use a fanout writer. * - * Field name changed between Iceberg releases: - * - 1.5.2 (Spark 3.4 profile): `partitionedFanoutEnabled` - * - 1.8.1+ (Spark 3.5 / 4.0 profiles): `useFanoutWriter` - * - * Same semantic in both versions. Try the newer name first; fall back to the older one so the - * helper resolves across all supported Iceberg versions without a per-version shim. + * Iceberg 1.8.1 and later store this as `SparkWrite.useFanoutWriter`. */ def getUseFanoutWriterFromSparkWrite(sparkWrite: Any): Option[Boolean] = getSparkWriteField(sparkWrite, "useFanoutWriter") - .orElse(getSparkWriteField(sparkWrite, "partitionedFanoutEnabled")) .map(_.asInstanceOf[java.lang.Boolean].booleanValue()) /** Output partition spec id; sourced from `SparkWrite.outputSpecId`. */ @@ -1528,8 +1499,7 @@ object IcebergReflection extends Logging { * Stamping the table sort order id unconditionally (the previous behaviour) writes the wrong * value when Java would have used `unsorted`. * - * Iceberg 1.5.2 (Spark 3.4 profile) lacks this method on `SparkWriteConf`; we return `None` and - * let the caller fall back to `Table.sortOrder().orderId()`. + * Returns `None` when the pinned Iceberg version does not expose the resolver. */ def getOutputSortOrderIdFromSparkWrite(sparkWrite: Any): Option[Int] = { val writeConf = @@ -1979,52 +1949,19 @@ object IcebergReflection extends Logging { } /** - * Construct a `GenericManifestFile` pointing at an in-memory data manifest. Two version-skew - * issues to handle: - * - * 1. Iceberg 1.8 added a constructor that also takes the snapshot id, and 1.10 removed the - * older one: - * - 1.5.2 (Spark 3.4 profile): `(InputFile, int)` -- 2-arg - * - 1.8+ (Spark 3.5 / 4.x profiles): `(InputFile, int, long)` -- 3-arg, the third being - * the snapshot id. Both forms are package-private, and both leave the manifest's - * `first_row_id` null, so the reader assigns no row ids to the decoded `DataFile`s. On a - * format-version 3 table the commit assigns them, as it does for iceberg-java's files. - * - * 2. `ManifestFiles.read` on Iceberg 1.5.2 refuses to read a `ManifestFile` whose - * `snapshotId()` is `null` (`InheritableMetadataFactory.fromManifest` throws "Cannot read from - * ManifestFile with null (unassigned) snapshot ID"). Iceberg 1.8+ relaxed that check. We set - * `snapshotId = 0L` via reflection so the read path is happy across versions; the real snapshot - * id is stamped onto the embedded `DataFile`s later by `BatchWrite.commit(messages)`, so the - * placeholder never reaches storage. + * Construct a `GenericManifestFile` pointing at an in-memory data manifest. Iceberg 1.8 and + * later use the package-private `(InputFile, int, long)` constructor. The placeholder snapshot + * id never reaches storage: `BatchWrite.commit(messages)` stamps the real snapshot id onto the + * embedded `DataFile`s. */ private def newDataManifestFile(inputFile: AnyRef, specId: Int): AnyRef = { val inputFileClass = loadClass(ClassNames.INPUT_FILE) val cls = loadClass(ClassNames.GENERIC_MANIFEST_FILE) - // `Constructor[AnyRef]` rather than `Constructor[_]`: the two `try`/`catch` branches - // would otherwise infer a top-level existential, which `-Xlint:existential` rejects. - val (ctor, args): (java.lang.reflect.Constructor[AnyRef], Array[Object]) = - try { - val c = cls - .getDeclaredConstructor(inputFileClass, classOf[Int], classOf[Long]) - .asInstanceOf[java.lang.reflect.Constructor[AnyRef]] - (c, Array[Object](inputFile, Integer.valueOf(specId), java.lang.Long.valueOf(0L))) - } catch { - case _: NoSuchMethodException => - val c = cls - .getDeclaredConstructor(inputFileClass, classOf[Int]) - .asInstanceOf[java.lang.reflect.Constructor[AnyRef]] - (c, Array[Object](inputFile, Integer.valueOf(specId))) - } + val ctor = cls.getDeclaredConstructor(inputFileClass, classOf[Int], classOf[Long]) ctor.setAccessible(true) - val manifest = ctor.newInstance(args: _*).asInstanceOf[AnyRef] - try { - val snapshotIdField = cls.getDeclaredField("snapshotId") - snapshotIdField.setAccessible(true) - snapshotIdField.set(manifest, java.lang.Long.valueOf(0L)) - } catch { - case _: NoSuchFieldException => () // field renamed in a future release; soft-fail - } - manifest + ctor + .newInstance(inputFile, Integer.valueOf(specId), java.lang.Long.valueOf(0L)) + .asInstanceOf[AnyRef] } private def readDataFilesFromManifest( diff --git a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteReportListener.scala b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteReportListener.scala index b9dba6c1e2d..13849e98768 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteReportListener.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteReportListener.scala @@ -118,10 +118,6 @@ object IcebergWriteReportListener { // A streaming micro-batch, which Comet's split operator never plans. case w: WriteToDataSourceV2Exec if isIcebergMicroBatch(w.batchWrite) => Seq(IcebergWrite(Spark, w.nodeName, Nil)) - // Spark 3.4 writes a CTAS or RTAS from the create or replace exec itself. Later versions run - // that write as a nested append or overwrite, which this listener sees as a query of its own. - case w if IcebergTableAsSelectShim.writeCatalog(w).exists(isIceberg) => - Seq(IcebergWrite(Spark, w.nodeName, Nil)) case p => p.children.flatMap(writes) } diff --git a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala index 49cc6519867..d010bcde116 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala @@ -69,17 +69,6 @@ case class IcebergWriteStrategy(session: SparkSession) extends SparkStrategy { .extractProjections(rd) .map(ReplaceDataWrite) .getOrElse(PlainIcebergWrite)).toList - case replace if IcebergReflection.isReplaceIcebergData(replace) => - IcebergReflection - .extractReplaceIcebergDataFields(replace) - .flatMap { case (_, query, originalTable, write) => - matchedSparkWrite( - originalTable.asInstanceOf[org.apache.spark.sql.catalyst.analysis.NamedRelation], - write.asInstanceOf[Option[Write]], - query.asInstanceOf[LogicalPlan], - PlainIcebergWrite) - } - .toList // Hit by AQE. case l @ IcebergWriteLogical(child, batchWrite, dispatch) => Seq(IcebergWriteExec(batchWrite, l.output, planLater(child), dispatch)) diff --git a/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala b/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala index 8f9274a9e2b..0c802d975bc 100644 --- a/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala +++ b/spark/src/main/scala/org/apache/comet/iceberg/ReplaceDataDispatchInfo.scala @@ -22,9 +22,9 @@ package org.apache.comet.iceberg import org.apache.spark.sql.catalyst.ProjectingInternalRow /** - * Mirror of Spark 4.x's `ReplaceDataProjections`, defined here so we can compile against Spark - * 3.4 / 3.5 source trees (where the class doesn't exist). Populated by `IcebergReplaceDataShim` - * on 4.x and left `None` on 3.x; consumed by `IcebergWriteExec`. + * Mirror of Spark 4.x's `ReplaceDataProjections`, defined here so we can compile against the + * Spark 3.5 source tree (where the class doesn't exist). Populated by `IcebergReplaceDataShim` on + * 4.x and left `None` on 3.x; consumed by `IcebergWriteExec`. */ case class ReplaceDataDispatchInfo( rowProjection: ProjectingInternalRow, diff --git a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala index 12b73d768aa..870e7d66f63 100644 --- a/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala +++ b/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala @@ -130,9 +130,9 @@ object CometExecRule { // EmptyRelationExec was introduced in Spark 4.0. ShimCometEmptyRelation.emptyRelationClass.map(_ -> CometEmptyRelationExec) ++ ShimCometArrowEvalPythonExec.nativeOperator ++ - // WindowGroupLimitExec exists only on Spark 3.5+; the shim returns None on 3.4. + // Resolve WindowGroupLimitExec through its version shim. ShimCometWindowGroupLimit.windowGroupLimitClass.map(_ -> CometWindowGroupLimitExec) ++ - // MergeRowsExec exists only on Spark 3.5+; the shim is empty on 3.4. + // Resolve MergeRowsExec through its version shim. ShimCometMergeRows.nativeExecs /** @@ -166,14 +166,6 @@ object CometExecRule { val SKIP_COMET_SHUFFLE_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometShuffle") - /** - * Tag set on a `BroadcastExchangeExec` that should be left as a plain Spark broadcast rather - * than converted to `CometBroadcastExchangeExec`. Written by [[CometSpark34AqeDppFallbackRule]] - * on Spark < 3.5. See that rule's class docstring for the rationale. - */ - val SKIP_COMET_BROADCAST_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = - org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometBroadcast") - /** * The operators `spark.comet.sparkToColumnar.supportedOperatorList` named by default, before * each had a `spark.comet.convert` config of its own. `spark.comet.sparkToColumnar.enabled` @@ -606,7 +598,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false) // Spark 3.x: `getWriteFilesOpt` matches the concrete `WriteFilesExec` case class, so a // Comet node can never stand in for the write node. Native writes instead replace the whole // DataWritingCommandExec and re-implement the write framework inside CometNativeWriteExec. - // This path is retained only for 3.4/3.5 and goes away with them. + // This path is retained only for Spark 3.5 and goes away with Spark 3.x. // // `originalPlan` is that command. This rule copies `originalPlan.logicalLink` onto the // Comet node, so AQE re-plans the write with the command rather than with whatever child @@ -636,11 +628,6 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false) // broadcast exchange is forced to be enabled by Comet config. case plan if plan.children.exists(_.isInstanceOf[BroadcastExchangeExec]) => val newChildren = plan.children.map { - // Tagged by CometSpark34AqeDppFallbackRule on Spark < 3.5 to keep the build-side - // broadcast Spark-native so Spark's PlanAdaptiveDynamicPruningFilters can match it. - case b: BroadcastExchangeExec - if b.getTagValue(CometExecRule.SKIP_COMET_BROADCAST_TAG).isDefined => - b case b: BroadcastExchangeExec if b.children.forall(_.isInstanceOf[CometNativeExec]) => convertToComet(b, CometBroadcastExchangeExec).getOrElse(b) case other => other @@ -986,17 +973,13 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false) } case _ => inSub } - case sab: SubqueryAdaptiveBroadcastExec if isSpark35Plus => + case sab: SubqueryAdaptiveBroadcastExec => // Wrap SABs to prevent Spark's PlanAdaptiveDynamicPruningFilters from // converting them to Literal.TrueLiteral. Spark's rule pattern-matches for // BroadcastHashJoinExec, which Comet replaced with CometBroadcastHashJoinExec. // Without wrapping, DPP is disabled for both Comet native scans and non-Comet // scans (e.g., V2 BatchScan). CometPlanAdaptiveDynamicPruningFilters - // (queryStageOptimizerRule, 3.5+) unwraps and converts them later. - // - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable. The isSpark35Plus - // guard leaves SABs unwrapped; CometSpark34AqeDppFallbackRule then tags the - // matching BHJ's build broadcast so Spark's rule can match it natively. + // (queryStageOptimizerRule) unwraps and converts them later. assert( sab.buildKeys.nonEmpty, s"SubqueryAdaptiveBroadcastExec '${sab.name}' has empty buildKeys") @@ -1782,12 +1765,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false) * the backstop that undoes any bridge left under a Spark `BroadcastExchangeExec` after the * transform. Skipping here still matters because a bridge under a Spark broadcast breaks * Spark's DPP broadcast reuse until it is torn down (the DPP subquery keeps an unbridged copy). - * We skip when: - * - the join's own serde is disabled (`joinEnabled`), so it will stay on Spark; or - * - the build-side `BroadcastExchangeExec` carries `SKIP_COMET_BROADCAST_TAG`, which - * `CometSpark34AqeDppFallbackRule` sets on Spark 3.4 to keep the broadcast Spark-native so - * `PlanAdaptiveDynamicPruningFilters` can match it (bridging it would turn DPP off). See - * issue #6008. + * We skip when the join's own serde is disabled (`joinEnabled`), so it will stay on Spark. */ private def tagBuildIfProbeNative( buildSide: BuildSide, @@ -1801,12 +1779,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false) case BuildLeft => (left, right) case BuildRight => (right, left) } - val skipForDpp = buildPlan match { - case b: BroadcastExchangeExec => - b.getTagValue(CometExecRule.SKIP_COMET_BROADCAST_TAG).isDefined - case _ => false - } - if (!skipForDpp && hasOnlyNativeScans(probePlan)) { + if (hasOnlyNativeScans(probePlan)) { tagBuildSideScans(buildPlan) } } diff --git a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala index 73f19b2cfb2..722ec1a4402 100644 --- a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala +++ b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala @@ -32,7 +32,7 @@ import scala.jdk.CollectionConverters._ import org.apache.hadoop.conf.Configuration import org.apache.spark.internal.Logging import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruningExpression, Expression, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName} +import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruningExpression, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName} import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.util.{sideBySide, ArrayBasedMapData, DateTimeUtils, GenericArrayData, MetadataColumnHelper} import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues @@ -47,7 +47,7 @@ import org.apache.spark.sql.types._ import org.apache.comet.{CometConf, DataTypeSupport, NativeBase} import org.apache.comet.CometConf._ -import org.apache.comet.CometSparkSessionExtensions.{isCometLoaded, isSpark35Plus, withFallbackReason, withFallbackReasons} +import org.apache.comet.CometSparkSessionExtensions.{isCometLoaded, withFallbackReason, withFallbackReasons} import org.apache.comet.iceberg.{CometIcebergNativeScanMetadata, IcebergReflection, IcebergStorageSchemes} import org.apache.comet.objectstore.NativeConfig import org.apache.comet.parquet.CometParquetUtils.{encryptionEnabled, isEncryptionConfigSupported, readFieldId} @@ -100,11 +100,6 @@ case class CometScanRule(session: SparkSession) NativeConfig.resolveLibhdfsSchemes(relationSession.sessionState.newHadoopConf())) def transformScan(scanNode: SparkPlan): SparkPlan = scanNode match { - // Tagged by CometSpark34AqeDppFallbackRule on Spark < 3.5 to keep a peer scan - // Spark-native for canonical symmetry in SMJ self-joins (SPARK-32509). - case scan if scan.getTagValue(CometScanRule.SKIP_COMET_SCAN_TAG).isDefined => - withFallbackReason(scan, "AQE DPP region fallback (Spark < 3.5)") - case scan if !CometConf.COMET_NATIVE_SCAN_ENABLED.get(conf) => withFallbackReason(scan, "Comet Scan is not enabled") @@ -199,19 +194,6 @@ case class CometScanRule(session: SparkSession) s"Metadata column(s) ${unsupportedMetadataColNames.mkString(", ")} is not supported") } - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable, so - // CometPlanAdaptiveDynamicPruningFilters cannot run. Fall back this scan to Spark so that - // Spark's PlanAdaptiveDynamicPruningFilters handles the SAB natively. Comet's narrower - // CometSpark34AqeDppFallbackRule (queryStagePrepRule on 3.4) then tags any matching BHJ's - // build-side broadcast so Spark's rule can match it via sameResult. See - // CometSpark34AqeDppFallbackRule's class docstring. - // - // On 3.5+, CometPlanAdaptiveDynamicPruningFilters rewrites SABs directly and this fallback - // is not needed. - if (!isSpark35Plus && scanExec.partitionFilters.exists(isAqeDynamicPruningFilter)) { - return withFallbackReason(scanExec, "AQE Dynamic Partition Pruning requires Spark 3.5+") - } - scanExec.relation match { case r: HadoopFsRelation => if (!CometScanExec.isFileFormatSupported(r.fileFormat)) { @@ -1358,21 +1340,6 @@ case class CometScanRule(session: SparkSession) } } - /** - * Detects AQE DPP (SubqueryAdaptiveBroadcastExec), as opposed to non-AQE DPP. - * - * Non-AQE DPP (PlanDynamicPruningFilters) runs before Comet rules and produces - * SubqueryBroadcastExec/SubqueryExec which Spark's execution framework resolves. AQE DPP - * (PlanAdaptiveDynamicPruningFilters) runs after Comet rules and searches for - * BroadcastHashJoinExec. It doesn't recognize Comet operators, so it can't create DPP filters - * correctly. - */ - private def isAqeDynamicPruningFilter(e: Expression): Boolean = - e.exists { - case sub: InSubqueryExec => sub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] - case _ => false - } - private def metadataCols(plan: SparkPlan): Seq[String] = { plan.expressions.collect { case a: Attribute if a.isMetadataCol => a.name @@ -1683,15 +1650,6 @@ object CometScanRule extends Logging { private[comet] def objectStoreAcceptsPath(uri: URI): Boolean = probeObjectStore(uri.toString) - /** - * Tag set on a scan (`FileSourceScanExec` or `BatchScanExec`) that should be left as a plain - * Spark scan rather than converted to a Comet scan. Written by - * [[CometSpark34AqeDppFallbackRule]] on Spark < 3.5. See that rule's class docstring for the - * rationale. - */ - val SKIP_COMET_SCAN_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] = - org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometScan") - /** * Schemes Comet's native Iceberg scan can open, loaded from the native storage factory over JNI * so this gate cannot drift from `storage_factory_for`. Lazy so that constructing the rule does diff --git a/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala deleted file mode 100644 index ea8653071dc..00000000000 --- a/spark/src/main/scala/org/apache/comet/rules/CometSpark34AqeDppFallbackRule.scala +++ /dev/null @@ -1,389 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.rules - -import org.apache.spark.internal.Logging -import org.apache.spark.sql.catalyst.expressions.DynamicPruningExpression -import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight} -import org.apache.spark.sql.catalyst.rules.Rule -import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SparkPlan, SubqueryAdaptiveBroadcastExec, SubqueryBroadcastExec} -import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper -import org.apache.spark.sql.execution.datasources.HadoopFsRelation -import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, ShuffleExchangeExec} -import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec - -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus - -/** - * Preserves AQE Dynamic Partition Pruning on Spark < 3.5 by tagging specific nodes to stay - * Spark-native, so that Spark's PlanAdaptiveDynamicPruningFilters can match them natively. - * - * On Spark 3.5+, CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule) rewrites AQE - * DPP SABs into CometSubqueryBroadcastExec after Spark's own rule has run. Spark 3.4 does not - * expose injectQueryStageOptimizerRule (SPARK-45785 added it in 3.5), so the rewrite cannot run - * at the correct time. Rewriting the SAB at queryStagePrepRule time does not work either: AQE - * rebuilds plan nodes between prep and execution in ways that drop the `@transient` inner scan we - * would need to update. See the dual-filter handling in - * CometPlanAdaptiveDynamicPruningFilters.convertNativeScanDPP for why both filters need updating. - * - * Instead, on 3.4 we arrange for Spark's PlanAdaptiveDynamicPruningFilters to succeed on its own - * by keeping the BHJ's build-side exchange (or the peer branch of a self-join SMJ) Spark-native. - * This rule only writes skip-tags on nodes; it never rewrites expressions or plan structure. Tags - * survive AQE per-stage re-entry, matching the contract of SKIP_COMET_SHUFFLE_TAG from PR #4010. - * - * Registered via injectPreSpark35QueryStagePrepRuleShim before CometScanRule/CometExecRule in - * CometSparkSessionExtensions, so tags are in place when conversion runs. No-op on Spark 3.5+. - * - * Four cases handled: - * - * 1. SAB + matching BHJ (non-V1 fact scans: Hive / V2 / V2Filter). The cascade up from a non-V1 - * scan reaches a CometBroadcastHashJoinExec + CometBroadcastExchangeExec build side; Spark's - * class-sensitive sameResult check in PlanAdaptiveDynamicPruningFilters.scala:50-57 fails to - * match. Tag the BHJ's build-side BroadcastExchangeExec with SKIP_COMET_BROADCAST_TAG. With - * the build exchange Spark-native, Comet's BHJ conversion fails its - * forall(_.isInstanceOf[CometNativeExec]) guard and the BHJ stays Spark. Spark's rule then - * matches and creates SubqueryBroadcastExec. - * - * 2. SAB + matching BHJ on V1. CometScanRule.transformV1Scan already rejects the V1 fact scan via - * isAqeDynamicPruningFilter; the cascade keeps the BHJ and its BroadcastExchangeExec - * Spark-native. No tagging needed; this rule is a no-op for V1 BHJ. TPC-DS Q7 on 3.4 stays the - * same shape it has today, including the Comet acceleration on dim scans below the Spark - * broadcast. - * - * 3. SAB with no matching BHJ (V1 SMJ self-join, SPARK-32509 with - * AUTO_BROADCASTJOIN_THRESHOLD=-1). The logical Partition-Pruning rule attaches the SAB to only - * one branch of the self-join. transformV1Scan falls back the SAB-bearing scan; the peer scan (no - * SAB) Cometizes, producing canonical asymmetry that breaks shuffle exchange reuse. Tag the peer - * scan with SKIP_COMET_SCAN_TAG and any shuffle whose subtree contains the peer scan with - * SKIP_COMET_SHUFFLE_TAG. Both branches end up Spark-native with matching canonical forms; - * Spark's rule replaces the SAB with TrueLiteral, and FileSourceScanExec.doCanonicalize strips it - * via filterUnusedDynamicPruningExpressions (DataSourceScanExec.scala:731,736), restoring - * canonical symmetry for reuse. - * - * 4. SubqueryBroadcastExec-bearing scans (AQE re-optimize). On re-optimize cycles, - * ASPE.preprocessingRules (PlanAdaptiveSubqueries) fills the DPP slot with the already- - * materialized SubqueryBroadcastExec (produced by Spark's PlanAdaptiveDynamicPruningFilters on a - * previous pass) rather than the original SAB. The freshly-planned BroadcastExchangeExec on the - * main BHJ's build side is a new instance with no SKIP_COMET_BROADCAST_TAG carried over, so - * CometExecRule would Cometize it and lose AQE stageCache broadcast reuse with the DPP subquery's - * Spark broadcast. The rule also scans for SubqueryBroadcastExec (descending into QueryStageExec - * via AdaptiveSparkPlanHelper since the fact scan is already inside a materialized stage), - * extracts its buildKeys, finds the matching BHJ by exprId, and tags the build BE. - * - * Non-AQE DPP (#4011) is untouched: it produces SubqueryBroadcastExec, not the adaptive variant, - * and is handled by CometExecRule.convertSubqueryBroadcasts. - * - * Known limitation on 3.4: cross-plan scalar-subquery DPP. An SAB in a scalar subquery cannot see - * a matching BHJ in the main query because at prep-rule time each AdaptiveSparkPlanExec sees only - * its own plan. When the match fails, Spark's own rule falls back to TrueLiteral or aggregate - * SubqueryExec (same behavior as Spark-without-Comet on 3.4). - * - * @see - * PlanAdaptiveDynamicPruningFilters (Spark's rule this code arranges to succeed) - * @see - * CometPlanAdaptiveDynamicPruningFilters (Spark 3.5+ equivalent via queryStageOptimizerRule) - */ -case object CometSpark34AqeDppFallbackRule - extends Rule[SparkPlan] - with AdaptiveSparkPlanHelper - with Logging { - - override def apply(plan: SparkPlan): SparkPlan = { - // Registered only on Spark < 3.5 via injectPreSpark35QueryStagePrepRuleShim. If the - // 3.5+ shim mis-registers this rule, fail loud rather than silently disable Comet. - assert( - !isSpark35Plus, - "CometSpark34AqeDppFallbackRule must only be registered on Spark < 3.5; " + - "see ShimCometSparkSessionExtensions.injectPreSpark35QueryStagePrepRuleShim") - - if (!conf.dynamicPartitionPruningEnabled) return plan - - val sabScans = findSabScans(plan) - val sbScans = findSubqueryBroadcastScans(plan) - if (sabScans.isEmpty && sbScans.isEmpty) return plan - - sabScans.foreach { case (scan, sab) => - tagForSab(plan, scan, sab) - } - - // AQE re-optimization path: see case 4 in the class-level docstring. On subsequent - // re-optimize cycles Spark's PlanAdaptiveSubqueries hands us a SubqueryBroadcastExec (not - // the original SAB), and the freshly-planned main-BHJ build BE has no tag carried over. - sbScans.foreach { case (_, sb) => - tagForSubqueryBroadcast(plan, sb) - } - - // This rule only tags; it never rewrites the plan structurally. - plan - } - - /** - * Find every scan whose `partitionFilters` contain a `SubqueryAdaptiveBroadcastExec`. - * - * Mirrors the SAB pattern matched by Spark's `PlanAdaptiveDynamicPruningFilters.apply` (lines - * 41-43): - * {{{ - * case DynamicPruningExpression(InSubqueryExec( - * value, SubqueryAdaptiveBroadcastExec(...), ...)) - * }}} - * - * Returns the scan node itself and the SAB found on it. - */ - private def findSabScans(plan: SparkPlan): Seq[(SparkPlan, SubqueryAdaptiveBroadcastExec)] = { - val buf = scala.collection.mutable.ArrayBuffer[(SparkPlan, SubqueryAdaptiveBroadcastExec)]() - foreach(plan) { node => - extractFirstSab(node).foreach(sab => buf += ((node, sab))) - } - buf.toSeq - } - - private def extractFirstSab(node: SparkPlan): Option[SubqueryAdaptiveBroadcastExec] = { - node.expressions - .flatMap(_.collect { - case DynamicPruningExpression(inSub: InSubqueryExec) - if inSub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] => - inSub.plan.asInstanceOf[SubqueryAdaptiveBroadcastExec] - }) - .headOption - } - - /** - * Find every scan whose DPP `partitionFilters` contain a `SubqueryBroadcastExec` - i.e. a DPP - * subquery that Spark's `PlanAdaptiveDynamicPruningFilters` has already materialized on a - * previous AQE pass. See the comment block in `apply` for why we need this in addition to the - * SAB path. - * - * Uses `AdaptiveSparkPlanHelper.foreach`, which descends through `QueryStageExec.plan` and - * `AdaptiveSparkPlanExec.executedPlan`. By the re-optimize pass where this matters, the fact - * scan is inside a `ShuffleQueryStageExec` whose `children` is `Seq.empty`, so a plain - * `plan.foreach` would stop there and miss the scan's `SubqueryBroadcastExec`. - */ - private def findSubqueryBroadcastScans( - plan: SparkPlan): Seq[(SparkPlan, SubqueryBroadcastExec)] = { - val buf = scala.collection.mutable.ArrayBuffer[(SparkPlan, SubqueryBroadcastExec)]() - foreach(plan) { node => - extractFirstSubqueryBroadcast(node).foreach(sb => buf += ((node, sb))) - } - buf.toSeq - } - - private def extractFirstSubqueryBroadcast(node: SparkPlan): Option[SubqueryBroadcastExec] = { - node.expressions - .flatMap(_.collect { - case DynamicPruningExpression(inSub: InSubqueryExec) - if inSub.plan.isInstanceOf[SubqueryBroadcastExec] => - inSub.plan.asInstanceOf[SubqueryBroadcastExec] - }) - .headOption - } - - /** - * Place tags for a single SAB-bearing scan. Behavior depends on whether a matching - * broadcast-hash join exists in the plan: - * - Matching BHJ found: tag its build-side `BroadcastExchangeExec` (case 1 above). - * - No matching BHJ: tag peer scans + their shuffles for canonical symmetry (case 3). - */ - private def tagForSab( - plan: SparkPlan, - scan: SparkPlan, - sab: SubqueryAdaptiveBroadcastExec): Unit = { - val sabKeyIds: Set[Any] = sab.buildKeys.flatMap(_.references.map(_.exprId)).toSet - if (sabKeyIds.isEmpty) { - logWarning(s"SAB '${sab.name}' has empty buildKeys; skipping") - return - } - - findMatchingBroadcastJoin(plan, sabKeyIds) match { - case Some(buildSide) => - tagBhjBuildBroadcast(buildSide, sab.name) - case None => - tagPeerScansAndShuffles(plan, scan, sab.name) - } - } - - /** - * Tag the matching BHJ's build-side `BroadcastExchangeExec` for a scan whose DPP filter holds a - * `SubqueryBroadcastExec` (post-PADPF form). Only the matching-BHJ case applies here: if PADPF - * already ran on a previous AQE cycle, it would have fallen back to aggregate `SubqueryExec` or - * `Literal.TrueLiteral` instead of producing a `SubqueryBroadcastExec` when no BHJ matched - so - * a `SubqueryBroadcastExec` always implies a BHJ with compatible build keys somewhere in the - * plan. No peer-scan tagging path is needed because the SMJ self-join case (SPARK-32509) never - * produces a `SubqueryBroadcastExec`. - */ - private def tagForSubqueryBroadcast(plan: SparkPlan, sb: SubqueryBroadcastExec): Unit = { - val keyIds: Set[Any] = sb.buildKeys.flatMap(_.references.map(_.exprId)).toSet - if (keyIds.isEmpty) { - logWarning(s"SubqueryBroadcast '${sb.name}' has empty buildKeys; skipping") - return - } - - findMatchingBroadcastJoin(plan, keyIds) match { - case Some(buildSide) => - tagBhjBuildBroadcast(buildSide, sb.name) - case None => - // Nothing to tag. Either the BHJ has a different buildKey exprId set (e.g. AQE - // rewrote attributes across stages) or the matching join isn't in this plan snapshot. - logDebug( - s"SubqueryBroadcast '${sb.name}': no matching BHJ on this plan snapshot; no tag placed") - } - } - - /** - * Walk from a BHJ's build-side subtree root to the first `BroadcastExchangeExec` and tag it - * with `SKIP_COMET_BROADCAST_TAG`. If the subtree root already IS a BroadcastExchangeExec (most - * common), that's the one. Otherwise we walk until we find one or give up. - * - * Tagging the exchange is enough: when it stays Spark-native, `CometBroadcastHashJoinExec`'s - * conversion guard (`forall(_.isInstanceOf[CometNativeExec])`) fails because a Spark - * `BroadcastExchangeExec` is not a `CometNativeExec`, so the BHJ stays Spark too. Spark's - * `PlanAdaptiveDynamicPruningFilters` can then match it via `sameResult` - * (`PlanAdaptiveDynamicPruningFilters.scala:50-57`) and create a `SubqueryBroadcastExec`. - */ - private def tagBhjBuildBroadcast(buildSide: SparkPlan, dppName: String): Unit = { - val found = buildSide.find { - case _: BroadcastExchangeExec => true - case _ => false - } - found match { - case Some(be: BroadcastExchangeExec) => - be.setTagValue(CometExecRule.SKIP_COMET_BROADCAST_TAG, ()) - logDebug(s"Tagged BroadcastExchangeExec for DPP '$dppName' (BHJ build side)") - case _ => - logWarning( - s"DPP '$dppName': matched BHJ but could not locate BroadcastExchangeExec on " + - "build side; skipping") - } - } - - /** - * For SMJ-shaped DPP (SPARK-32509 with AUTO_BROADCASTJOIN_THRESHOLD=-1), there is no BHJ. The - * logical Partition-Pruning rule attaches the SAB to only one branch of a self-join; the peer - * branch has no SAB, so `transformV1Scan`'s V1 AQE DPP fallback fires for the SAB-bearing scan - * only. Peer scans Cometize normally, leading to canonical asymmetry that breaks shuffle - * exchange reuse (0 `ReusedExchangeExec` instead of 1). - * - * Tag every peer scan (same relation as the SAB-bearing scan, different instance) with - * `SKIP_COMET_SCAN_TAG`, and every `ShuffleExchangeExec` whose subtree contains a peer scan - * with `SKIP_COMET_SHUFFLE_TAG`. Both branches end up Spark-native with matching canonical - * forms. - * - * For canonical equality the SAB-bearing scan's side is already handled: - * `FileSourceScanExec.doCanonicalize` strips `DynamicPruningExpression(Literal.TrueLiteral)` - * via `filterUnusedDynamicPruningExpressions` (`DataSourceScanExec.scala:731,736`), and Spark's - * rule replaces the SAB with `TrueLiteral` when it can't match. - */ - private def tagPeerScansAndShuffles( - plan: SparkPlan, - sabScan: SparkPlan, - sabName: String): Unit = { - val sabRelation = sabScan match { - case f: FileSourceScanExec => Some(f.relation) - case _ => None - } - if (sabRelation.isEmpty) { - logDebug( - s"SAB '$sabName': non-V1 scan with no BHJ match; no peer-tagging heuristic available") - return - } - val sabRel = sabRelation.get - - var taggedScans = 0 - foreach(plan) { - case peer: FileSourceScanExec if (peer ne sabScan) && sameRelation(peer.relation, sabRel) => - peer.setTagValue(CometScanRule.SKIP_COMET_SCAN_TAG, ()) - taggedScans += 1 - case _ => - } - - var taggedShuffles = 0 - foreach(plan) { - case sh: ShuffleExchangeExec if shuffleSubtreeContainsMatchingScan(sh, sabRel, sabScan) => - sh.setTagValue(CometExecRule.SKIP_COMET_SHUFFLE_TAG, ()) - taggedShuffles += 1 - case _ => - } - - logDebug( - s"SAB '$sabName' (no BHJ match): tagged $taggedScans peer scan(s) + " + - s"$taggedShuffles shuffle(s)") - } - - private def sameRelation(a: HadoopFsRelation, b: HadoopFsRelation): Boolean = { - (a eq b) || - (a.location.rootPaths == b.location.rootPaths && - a.dataSchema == b.dataSchema && - a.partitionSchema == b.partitionSchema) - } - - private def shuffleSubtreeContainsMatchingScan( - shuffle: ShuffleExchangeExec, - sabRelation: HadoopFsRelation, - sabScan: SparkPlan): Boolean = { - find(shuffle.child) { - case scan: FileSourceScanExec => - (scan eq sabScan) || sameRelation(scan.relation, sabRelation) - case _ => false - }.isDefined - } - - /** - * Mirrors `PlanAdaptiveDynamicPruningFilters.apply` lines 50-57: - * {{{ - * find(rootPlan) { - * case BroadcastHashJoinExec(_, _, _, BuildLeft, _, left, _, _) => - * left.sameResult(exchange) - * case BroadcastHashJoinExec(_, _, _, BuildRight, _, _, right, _) => - * right.sameResult(exchange) - * } - * }}} - * - * Our rule runs BEFORE `CometScanRule`/`CometExecRule`, so the plan is entirely Spark-native at - * this point. We only match `BroadcastHashJoinExec`. Instead of `sameResult` we match on - * join-side exprId equality with the SAB's buildKeys; this is semantically equivalent because - * SAB buildKeys originate from the same logical plan as the join's build-side keys. - * - * Returns the BHJ's build-side subtree (the one we want to keep Spark-native), or None if no - * matching join is found. - */ - private def findMatchingBroadcastJoin( - plan: SparkPlan, - sabKeyIds: Set[Any]): Option[SparkPlan] = { - var result: Option[SparkPlan] = None - find(plan) { - case j: BroadcastHashJoinExec => - val joinBuildKeys = j.buildSide match { - case BuildLeft => j.leftKeys - case BuildRight => j.rightKeys - } - val joinKeyIds: Set[Any] = joinBuildKeys.flatMap(_.references.map(_.exprId)).toSet - if (sabKeyIds == joinKeyIds) { - result = Some(j.buildSide match { - case BuildLeft => j.left - case BuildRight => j.right - }) - true - } else { - false - } - case _ => false - } - result - } -} diff --git a/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala b/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala index f9b8c2cf2f2..b596ac45675 100644 --- a/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala +++ b/spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala @@ -137,7 +137,7 @@ case class EliminateRedundantTransitions(session: SparkSession) // Replace MapInBatchExec (PythonMapInArrowExec / MapInArrowExec / MapInPandasExec) that has // a ColumnarToRow child with CometMapInBatchExec, eliminating the input and output // UnsafeProjection copies and keeping the stage columnar. The matchers are - // version-shimmed: Spark 3.4 / 3.5 return None (they lack the required APIs) and Spark + // version-shimmed: Spark 3.5 returns None (it lacks the required APIs) and Spark // 4.1+ matches the renamed `MapInArrowExec`. // // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled: diff --git a/spark/src/main/scala/org/apache/comet/serde/CometScalarFunction.scala b/spark/src/main/scala/org/apache/comet/serde/CometScalarFunction.scala index 87db1da040d..a282b4098d9 100644 --- a/spark/src/main/scala/org/apache/comet/serde/CometScalarFunction.scala +++ b/spark/src/main/scala/org/apache/comet/serde/CometScalarFunction.scala @@ -50,7 +50,7 @@ object CometScalarFunction { * Product / Java field names that indicate ANSI / eval-mode sensitive Spark expressions. * * Detection uses Java reflection (not `Product.productElementNames`) so the check compiles on - * Scala 2.12 used by Spark 3.4 / 3.5. + * Scala 2.12 used by Spark 3.5. */ private val AnsiSensitiveFields: Set[String] = Set("failOnError", "evalMode", "nullOnOverflow", "ansiEnabled", "evalContext") diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala index c8b89b0742c..2d1d355fc75 100644 --- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala +++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala @@ -58,7 +58,7 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim { private[comet] val arrayExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map( // ArrayAppend is a concrete expression only on Spark 3.x. Spark 4.0+ marks it // RuntimeReplaceable and rewrites it to ArrayInsert before serde, so this entry is - // unreachable there and is kept solely for Spark 3.4/3.5. + // unreachable there and is kept solely for Spark 3.5. classOf[ArrayAppend] -> CometArrayAppend, // ArrayCompact is RuntimeReplaceable in all supported Spark versions (rewritten to // ArrayFilter(arr, IsNotNull(...))), so it never reaches serde directly; dispatch flows @@ -260,7 +260,6 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim { classOf[Overlay] -> CometOverlay, classOf[SoundEx] -> CometSoundEx, classOf[StringLocate] -> CometStringLocate, - classOf[Base64] -> CometBase64, classOf[UnBase64] -> CometUnBase64, classOf[ToCharacter] -> CometToCharacter, classOf[ToNumber] -> CometToNumber, diff --git a/spark/src/main/scala/org/apache/comet/serde/SparkFixes.scala b/spark/src/main/scala/org/apache/comet/serde/SparkFixes.scala index 12023fa55a7..e48046ce6cf 100644 --- a/spark/src/main/scala/org/apache/comet/serde/SparkFixes.scala +++ b/spark/src/main/scala/org/apache/comet/serde/SparkFixes.scala @@ -101,8 +101,8 @@ private[comet] object SparkFixes { /** * SPARK-48719 made regr_slope and regr_intercept count VariancePop(x) only over rows where both - * y and x are non-null. Present from 3.5.2 and 4.0.0; earlier 3.5 patches and 3.4 count every - * row where x is non-null. + * y and x are non-null. Present from 3.5.2 and 4.0.0; earlier 3.5 patches count every row where + * x is non-null. */ def slopeFiltersVarByPairNulls(sparkVersion: String): Boolean = Utils.majorMinorPatchVersion(sparkVersion).exists { @@ -113,7 +113,7 @@ private[comet] object SparkFixes { /** * SPARK-55969 swapped regr_r2's degenerate cases: a constant dependent variable yields 1.0 (was * null) and a constant independent variable yields null (was 1.0). Present from 3.5.9, 4.0.3, - * 4.1.2 and 4.2.0; earlier patches of those lines and 3.4 keep the old cases. + * 4.1.2 and 4.2.0; earlier patches of those lines keep the old cases. */ def r2DegenerateCasesSwapped(sparkVersion: String): Boolean = Utils.majorMinorPatchVersion(sparkVersion).exists { diff --git a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala index 15318da2413..d62325783cb 100644 --- a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala +++ b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala @@ -1169,8 +1169,8 @@ object CometCollectSet extends CometAggregateExpressionSerde[CollectSet] { override def getSupportLevel(expr: CollectSet): SupportLevel = { // The native path always drops null inputs. Spark 4.2 added an `ignoreNulls` field to // CollectSet that `RESPECT NULLS` sets to false, preserving nulls in the result; Comet - // cannot match that, so fall back. This branch is only reachable on Spark 4.2+: on 3.4 - // through 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` are rejected at + // cannot match that, so fall back. This branch is only reachable on Spark 4.2+: through + // Spark 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` are rejected at // analysis time, and CometCollectShim.ignoreNulls hardcodes true, making this a no-op. if (!CometCollectShim.ignoreNulls(expr)) { Unsupported(Some("collect_set with RESPECT NULLS (ignoreNulls = false) is not supported")) @@ -1243,7 +1243,7 @@ object CometCollectList extends CometAggregateExpressionSerde[CollectList] { // The native path delegates to SparkCollectList, which always drops null inputs. Spark 4.2 // added an `ignoreNulls` field to CollectList that `RESPECT NULLS` sets to false, preserving // nulls in the result; Comet cannot match that, so fall back. This branch is only reachable - // on Spark 4.2+: on 3.4 through 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` + // on Spark 4.2+: through Spark 4.1 the field does not exist, `RESPECT NULLS`/`IGNORE NULLS` // are rejected at analysis time, and CometCollectShim.ignoreNulls hardcodes true, making this // a no-op. if (!CometCollectShim.ignoreNulls(expr)) { diff --git a/spark/src/main/scala/org/apache/comet/serde/arrays.scala b/spark/src/main/scala/org/apache/comet/serde/arrays.scala index 4e7b8186868..7c60df6b281 100644 --- a/spark/src/main/scala/org/apache/comet/serde/arrays.scala +++ b/spark/src/main/scala/org/apache/comet/serde/arrays.scala @@ -283,7 +283,7 @@ object CometSortArray extends CometExpressionSerde[SortArray] with CodegenDispat Unsupported(Some(s"Sort on array element type $elementType is not supported")) } else { expr.ascendingOrder match { - // Spark 3.x requires a boolean Literal; Spark 4.0+ widens ascendingOrder to any + // Spark 3.5 requires a boolean Literal; Spark 4.0+ widens ascendingOrder to any // foldable boolean. Accept both; convert evaluates the foldable expression. case ao if ao.foldable && ao.dataType == BooleanType => Compatible() case other => diff --git a/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala b/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala index 8d08e06d334..d006efa41a6 100644 --- a/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala +++ b/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala @@ -1500,34 +1500,9 @@ object CometIcebergNativeScan extends CometOperatorSerde[CometBatchScanExec] wit residualEntries += pools.residuals.size perPartition += pools.build(partitionBuilder) } - case other if other.getClass.getName == "org.apache.spark.rdd.ParallelCollectionRDD" => - // Spark 3.4's BatchScanExec.inputRDD returns sparkContext.parallelize(empty, 1) when - // DPP removes every split of a scan that reports SinglePartition. Among supported - // versions, only Spark 3.4 infers SinglePartition from one execution partition, - // which may be a single split or a group of splits. Other fully pruned scans take - // the DataSourceRDD branch above: ungrouped scans have no partitions, while grouped - // scans may retain empty partitions. Preserve Spark's empty - // execution partition so global aggregates still run and scans in the same native block - // retain matching partition counts when no exchange separates them. - // Re-querying scan.toBatch.planInputPartitions() to verify is unreliable because - // Iceberg's Scan state after filter() doesn't always reflect post-DPP partitions on - // a re-call (V2 scan state is one-shot for the materialized inputRDD). Matched by class - // name because ParallelCollectionRDD is private[spark]. - logDebug( - "BatchScanExec.inputRDD is ParallelCollectionRDD (DPP pruned all partitions); " + - "preserving empty execution partitions") - other.partitions.foreach { _ => - perPartition += - new IcebergPartitionPlanData( - Array.emptyByteArray, - Array.empty, - Array.empty, - Array.empty) - } case other => throw new IllegalStateException( - "Expected DataSourceRDD or ParallelCollectionRDD from BatchScanExec, " + - s"got ${other.getClass.getName}") + s"Expected DataSourceRDD from BatchScanExec, got ${other.getClass.getName}") } // Log deduplication summary diff --git a/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala b/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala index 141c6e3c474..a1c29887883 100644 --- a/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala +++ b/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala @@ -1386,8 +1386,8 @@ object CometIcebergNativeWrite extends CometOperatorSerde[IcebergWriteExec] { return None } // Iceberg's Spark `SparkWrite$WriterFactory` does NOT wire the table sort order into the - // per-file writer factory for batch appends in any Iceberg version Comet targets (1.5.2 / - // 1.8.1 / 1.10.0): it builds `SparkFileWriterFactory` without `.dataSortOrder(...)`, so every + // per-file writer factory for batch appends in Iceberg 1.8.1 / 1.10.0: it builds + // `SparkFileWriterFactory` without `.dataSortOrder(...)`, so every // committed data file is stamped `sort_order_id = 0` (unsorted) even when the table itself has // a non-default sort order. We match that exactly. The // `SparkWriteConf.outputSortOrderId(writeRequirements)` resolver (explicit option / table order diff --git a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala index e8efa37202c..14abbb69f8f 100644 --- a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala +++ b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala @@ -28,14 +28,14 @@ import org.apache.spark.sql.catalyst.analysis.Resolver import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, Expression, Literal} import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues import org.apache.spark.sql.comet.{CometNativeExec, CometNativeScanExec, CometScanExec, CometScanUtils} -import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SubqueryAdaptiveBroadcastExec} +import org.apache.spark.sql.execution.FileSourceScanExec import org.apache.spark.sql.execution.datasources.parquet.ParquetUtils import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.types.{MetadataBuilder, StructField, StructType} +import org.apache.spark.sql.types.{StructField, StructType} import org.apache.comet.{CometConf, ConfigEntry, DataTypeSupport} import org.apache.comet.CometConf.COMET_EXEC_ENABLED -import org.apache.comet.CometSparkSessionExtensions.{hasFallbackReason, isSpark35Plus, isSpark41Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{hasFallbackReason, isSpark41Plus, withFallbackReason} import org.apache.comet.objectstore.NativeConfig import org.apache.comet.parquet.CometParquetUtils import org.apache.comet.serde.{CometOperatorSerde, CometTimeZone, Compatible, OperatorOuterClass, SupportLevel} @@ -114,33 +114,11 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS dataSchema: StructType, requiredSchema: StructType, resolver: Resolver): StructType = { - // Ordinary fields keep their names, which projection below also matches exactly. Index - // only ID-bearing required fields so wide ordinary schemas do not need quadratic lookups. - val requiredFieldsWithIds = requiredSchema.fields - .filter(ParquetUtils.hasFieldId) - .map(field => field.name -> field) - .toMap StructType(dataSchema.fields.flatMap { field => if (containsVariantType(field.dataType)) { requiredSchema.fields.find(requiredField => resolver(requiredField.name, field.name)) } else { - // Nested schema pruning drops root metadata from the relation's data schema. Spark - // 3.4 still retains it in requiredSchema, which its Parquet reader uses for ID matching. - // Use that ID here too, without changing the table schema's types or projection indexes. - // When pruning also drops requiredSchema's ID (Spark 3.5+), leave the missing - // data-schema ID untouched so lookup remains name-based. - Some( - requiredFieldsWithIds - .get(field.name) - .map { required => - field.copy(metadata = new MetadataBuilder() - .withMetadata(field.metadata) - .putLong( - ParquetUtils.FIELD_ID_METADATA_KEY, - ParquetUtils.getFieldId(required).toLong) - .build()) - } - .getOrElse(field)) + Some(field) } }) } @@ -160,18 +138,10 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS } // AQE DPP (SubqueryAdaptiveBroadcastExec) is converted to CometSubqueryBroadcastExec - // by CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule, Spark 3.5+). + // by CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule). // Non-AQE DPP (SubqueryBroadcastExec/SubqueryExec) is converted by // CometExecRule.convertSubqueryBroadcasts. Both are resolved through the lazy // partition serialization path in CometNativeScanExec. - // - // On Spark 3.4, injectQueryStageOptimizerRule is unavailable, so the AQE DPP conversion - // rule can't run. CometScanRule.transformV1Scan rejects AQE DPP on 3.4, so this check - // is a safety net: if the scan somehow reached here with AQE DPP on 3.4, reject it. - if (!isSpark35Plus && scanExec.partitionFilters.exists(isAqeDynamicPruningFilter)) { - withFallbackReason(scanExec, "Native DataFusion scan does not support AQE DPP on Spark 3.4") - } - if (SQLConf.get.ignoreCorruptFiles || scanExec.relation.options .get("ignorecorruptfiles") // Spark sets this to lowercase. @@ -245,13 +215,6 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS !hasFallbackReason(scanExec) } - /** Detects AQE DPP (SubqueryAdaptiveBroadcastExec), as opposed to non-AQE DPP. */ - private def isAqeDynamicPruningFilter(e: Expression): Boolean = - e.exists { - case sub: InSubqueryExec => sub.plan.isInstanceOf[SubqueryAdaptiveBroadcastExec] - case _ => false - } - override def enabledConfig: Option[ConfigEntry[Boolean]] = None override def getSupportLevel(operator: CometScanExec): SupportLevel = { diff --git a/spark/src/main/scala/org/apache/comet/serde/predicates.scala b/spark/src/main/scala/org/apache/comet/serde/predicates.scala index ae13a419266..f848d5574ac 100644 --- a/spark/src/main/scala/org/apache/comet/serde/predicates.scala +++ b/spark/src/main/scala/org/apache/comet/serde/predicates.scala @@ -27,7 +27,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{BooleanType, DoubleType, FloatType} import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, withFallbackReason} import org.apache.comet.serde.ExprOuterClass.Expr import org.apache.comet.serde.QueryPlanSerde._ @@ -330,14 +330,14 @@ object ComparisonUtils { s"effect (`$legacyNullInEmptyListConfig`), because a `NULL` operand then evaluates to " + "`NULL` rather than `false`." - // Spark 3.4 has no config and always uses the legacy behavior, Spark 3.5 defaults the config to - // true, and Spark 4.0+ makes it optional with a default of `!spark.sql.ansi.enabled`. Read by - // string key so this compiles against every supported Spark version. + // Spark 3.5 defaults the config to true, and Spark 4.0+ makes it optional with a default of + // `!spark.sql.ansi.enabled`. Read by string key so this compiles against every supported Spark + // version. private def legacyNullInEmptyListBehavior: Boolean = - !isSpark35Plus || { - val default = !isSpark40Plus || !SQLConf.get.ansiEnabled - CometConf.getBooleanConf(legacyNullInEmptyListConfig, default, SQLConf.get) - } + CometConf.getBooleanConf( + legacyNullInEmptyListConfig, + !isSpark40Plus || !SQLConf.get.ansiEnabled, + SQLConf.get) /** * Support level shared by the `In` and `InSet` serdes: `list` is the set of values being tested diff --git a/spark/src/main/scala/org/apache/comet/serde/statics.scala b/spark/src/main/scala/org/apache/comet/serde/statics.scala index 27f140e6b26..098cf73da91 100644 --- a/spark/src/main/scala/org/apache/comet/serde/statics.scala +++ b/spark/src/main/scala/org/apache/comet/serde/statics.scala @@ -32,8 +32,7 @@ import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctio object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] { - // With Spark 3.4, CharVarcharCodegenUtils.readSidePadding gets called to pad spaces for - // char types. + // CharVarcharCodegenUtils.readSidePadding gets called to pad spaces for char types. // See https://github.com/apache/spark/pull/38151 /** * Handlers keyed by `(functionName, staticObject class name)`. Class names rather than classes @@ -53,9 +52,8 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] { // carrying the `legacyCharsets` / `legacyErrorAction` flags. Routing through the codegen // dispatcher runs Spark's own decoder so both flags are honored. See #4465. ("decode", classOf[StringDecode].getName) -> CometStaticInvokeCodegenDispatch, - // Spark 3.5+ makes `Base64` RuntimeReplaceable, lowering `base64(bin)` to - // `StaticInvoke(Base64.encode, Seq(child, chunkBase64), ...)`. On Spark 3.4 the `Base64` - // node survives and is handled directly (see CometBase64). + // `Base64` is RuntimeReplaceable, lowering `base64(bin)` to + // `StaticInvoke(Base64.encode, Seq(child, chunkBase64), ...)`. ("encode", classOf[Base64].getName) -> CometBase64StaticInvoke) private def handlerFor(expr: StaticInvoke): Option[CometExpressionSerde[StaticInvoke]] = @@ -123,7 +121,7 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] { // `StaticInvoke.doGenCode` emits a static method call, so the kernel matches Spark by // construction. Spark 4.x keeps lowering more `RuntimeReplaceable` functions this way // (`encode`, `is_valid_utf8`, the `TIME` family, ...) and `lpad` / `rpad` on binary - // has lowered to `StaticInvoke(ByteArray, ...)` since Spark 3.4. The dispatcher + // lowers to `StaticInvoke(ByteArray, ...)` on every supported version. The dispatcher // declines a call into any class that is not Spark's, such as a DataSource V2 // function, so that falls back (see [[CometInvokeTargets]]). // diff --git a/spark/src/main/scala/org/apache/comet/serde/strings.scala b/spark/src/main/scala/org/apache/comet/serde/strings.scala index bf5218f8f9b..5700564f574 100644 --- a/spark/src/main/scala/org/apache/comet/serde/strings.scala +++ b/spark/src/main/scala/org/apache/comet/serde/strings.scala @@ -19,7 +19,7 @@ package org.apache.comet.serde -import org.apache.spark.sql.catalyst.expressions.{Attribute, Base64, BitLength, BoundReference, Cast, Concat, ConcatWs, Contains, Elt, Empty2Null, EndsWith, Expression, FindInSet, FormatNumber, FormatString, GetJsonObject, InitCap, Left, Length, Levenshtein, Like, Literal, Lower, Mask, OctetLength, Overlay, RegExpExtract, RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike, SoundEx, StartsWith, StringInstr, StringLocate, StringLPad, StringRepeat, StringReplace, StringRPad, StringSplit, StringTranslate, StringTrim, StringTrimLeft, StringTrimRight, Substring, SubstringIndex, ToCharacter, ToNumber, TryToNumber, UnBase64, Upper} +import org.apache.spark.sql.catalyst.expressions.{Attribute, BitLength, BoundReference, Cast, Concat, ConcatWs, Contains, Elt, Empty2Null, EndsWith, Expression, FindInSet, FormatNumber, FormatString, GetJsonObject, InitCap, Left, Length, Levenshtein, Like, Literal, Lower, Mask, OctetLength, Overlay, RegExpExtract, RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike, SoundEx, StartsWith, StringInstr, StringLocate, StringLPad, StringRepeat, StringReplace, StringRPad, StringSplit, StringTranslate, StringTrim, StringTrimLeft, StringTrimRight, Substring, SubstringIndex, ToCharacter, ToNumber, TryToNumber, UnBase64, Upper} import org.apache.spark.sql.types.{ArrayType, BinaryType, DataTypes, IntegerType, LongType, StringType} import org.apache.spark.unsafe.types.UTF8String @@ -73,7 +73,7 @@ class CometCaseConversionBase[T <: Expression](function: String) super.convert(expr, inputs, binding) } else { // Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the - // Comet pipeline. This guarantees Spark-compatible behavior across 3.4 / 3.5 / 4.0. + // Comet pipeline. This guarantees Spark-compatible behavior across supported versions. // Falls through to Spark when the dispatcher is disabled. CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding) } @@ -177,7 +177,7 @@ object CometInitCap extends CometScalarFunction[InitCap]("initcap") with NativeO super.convert(expr, inputs, binding) } else { // Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the - // Comet pipeline. This guarantees Spark-compatible behavior across 3.4 / 3.5 / 4.0. + // Comet pipeline. This guarantees Spark-compatible behavior across supported versions. // Falls through to Spark when the dispatcher is disabled. CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding) } @@ -836,7 +836,7 @@ object CometGetJsonObject val jsonExpr = exprToProtoInternal(expr.json, inputs, binding) val pathExpr = exprToProtoInternal(expr.path, inputs, binding) val optExpr = scalarFunctionExprToProtoWithReturnType( - getJsonObjectNativeFunctionName, + "get_json_object", expr.dataType, false, jsonExpr, @@ -863,24 +863,6 @@ object CometSoundEx extends CometScalarFunction[SoundEx]("soundex") object CometStringLocate extends CometCodegenDispatch[StringLocate] -// On Spark 3.4 `Base64` is a plain expression node and always chunks the output (it uses -// `java.util.Base64.getMimeEncoder()` with no arguments). On Spark 3.5+ it is RuntimeReplaceable -// and lowers to a `StaticInvoke`, handled by CometBase64StaticInvoke instead. -object CometBase64 extends CometExpressionSerde[Base64] { - override def convert(expr: Base64, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = { - val childExpr = exprToProtoInternal(expr.child, inputs, binding) - val chunkExpr = exprToProtoInternal(Literal(true), inputs, binding) - val optExpr = - scalarFunctionExprToProtoWithReturnType( - "base64", - StringType, - failOnError = false, - childExpr, - chunkExpr) - optExpr - } -} - // Base64.getMimeDecoder() semantics: skips non-alphabet bytes and matches Spark's codegen // path. The native path handles the default UnBase64 (failOnError = false, reachable from SQL // `unbase64(...)`) only when the child is a column reference or literal, since native diff --git a/spark/src/main/scala/org/apache/spark/Plugins.scala b/spark/src/main/scala/org/apache/spark/Plugins.scala index 57e308ecf92..5236fcca48b 100644 --- a/spark/src/main/scala/org/apache/spark/Plugins.scala +++ b/spark/src/main/scala/org/apache/spark/Plugins.scala @@ -307,7 +307,7 @@ object CometDriverPlugin extends Logging { // spark.executor.memoryOverhead, which the JVM's own non-heap usage already draws on. // // Comet used to add an overhead of its own to it here, but a driver plugin cannot: on Spark - // 3.4, 3.5 and 4.0, SparkContext builds the default ResourceProfile before it creates the plugin + // 3.5 and 4.0, SparkContext builds the default ResourceProfile before it creates the plugin // container, and the cluster managers size executors from that profile rather than re-reading // the conf, so the new value never reached the container. Say so while the application is still // starting up instead, because this has to be set before the SparkContext is created. diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala index 175c46a4b14..729cd985ab1 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala @@ -183,7 +183,7 @@ case class CometBroadcastExchangeExec( val beforeBroadcast = System.nanoTime() longMetric("buildTime") += NANOSECONDS.toMillis(beforeBroadcast - beforeBuild) - // (3.4 only) SPARK-39983 - Broadcast the relation without caching the unserialized object. + // SPARK-39983 - Broadcast the relation without caching the unserialized object. val broadcasted = sparkContext .broadcastInternal(new CometBroadcastValue(batches), true) .asInstanceOf[broadcast.Broadcast[Any]] diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala index 2c067e7c46b..2b6c3f31e21 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala @@ -26,7 +26,7 @@ import org.apache.spark.rdd.RDD import org.apache.spark.sql.SparkSession import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, SortOrder} import org.apache.spark.sql.catalyst.plans.QueryPlan -import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, SinglePartition, UnknownPartitioning} +import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning} import org.apache.spark.sql.execution.{SparkPlan, SQLExecution} import org.apache.spark.sql.execution.datasources.v2.BatchScanExec import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} @@ -128,25 +128,13 @@ case class CometIcebergNativeScanExec( // Only accessed during execution, not planning def numPartitions: Int = perPartitionData.length - // Like BatchScanExec on Spark 3.5 and later without a key-grouped partitioning, the scan does - // not know its partition count while planning, because dynamic partition pruning decides which - // tasks survive and its subqueries resolve at execution. AQE reads outputPartitioning while it - // optimizes a stage (ValidateRequirements after CoalesceShufflePartitions), which is before + // Without a key-grouped partitioning, the scan does not know its partition count while + // planning because dynamic partition pruning decides which tasks survive and its subqueries + // resolve at execution. AQE reads outputPartitioning while it optimizes a stage, before // CometPlanAdaptiveDynamicPruningFilters converts the DPP placeholders, so serializing the // tasks here would run a placeholder subquery. At execution, CometNativeExec sizes the native // RDD from perPartitionData instead. - // - // Spark 3.4's BatchScanExec reports SinglePartition for one input partition, which it counts - // before any DPP filtering, and Spark plans no exchange above it on that basis. The scan reports - // the same. Otherwise EnsureRequirements, which AQE runs again over a DPP subquery's converted - // plan, inserts an exchange inside a native block that is already serialized (#7422). - // serializePartitions keeps that one partition when DPP prunes it, as Spark does. - override lazy val outputPartitioning: Partitioning = - if (originalPlan != null && originalPlan.outputPartitioning == SinglePartition) { - SinglePartition - } else { - UnknownPartitioning(0) - } + override lazy val outputPartitioning: Partitioning = UnknownPartitioning(0) override lazy val outputOrdering: Seq[SortOrder] = Nil diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala index 97f041033d0..abd3a08937a 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala @@ -59,7 +59,7 @@ case class CometInMemoryTableScanExec( extends CometExec with ShimCometInMemoryTableScanExec { - // Implements InMemoryTableScanLike on Spark 3.5+; Spark 3.4 has no such interface. + // Implements InMemoryTableScanLike on supported Spark versions. def isMaterialized: Boolean = cacheBuilder.isCachedColumnBuffersLoaded def baseCacheRDD(): RDD[CachedBatch] = cacheBuilder.cachedColumnBuffers @@ -223,7 +223,7 @@ object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanEx * Spark collects those metrics once a query finishes, with `CollectMetricsExec.collect`, and * that reaches the ones recorded inside a cached plan only through an `InMemoryTableScanExec` * over it. It does not know this node, so replacing the scan of such a relation leaves its - * metrics empty, and on Spark 3.4 leaves `Observation.get` waiting for good. The walk mirrors + * metrics empty and leaves `Observation.get` waiting for good. The walk mirrors * `CollectMetricsExec.collect`, through subqueries, adaptive plans and nested caches. */ def recordsObservedMetrics(relation: InMemoryRelation): Boolean = diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala index 99063e8370c..2e14ba6e227 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryAdaptiveBroadcastExec.scala @@ -55,8 +55,8 @@ case class CometSubqueryAdaptiveBroadcastExec( with UnaryExecNode { // This node must be converted to CometSubqueryBroadcastExec by - // CometPlanAdaptiveDynamicPruningFilters before execution. If we reach doExecute(), - // the rule didn't run (e.g., Spark 3.4 where injectQueryStageOptimizerRule is unavailable). + // CometPlanAdaptiveDynamicPruningFilters before execution. Reaching doExecute() means the + // conversion rule did not run. protected override def doExecute(): RDD[InternalRow] = { throw QueryExecutionErrors.executeCodePathUnsupportedError( "CometSubqueryAdaptiveBroadcastExec (should have been converted by " + diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala index 1242e549084..0bc6e933ac4 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala @@ -40,16 +40,14 @@ import org.apache.comet.shims.ShimCometWindowGroupLimit * the Spark-sorted child. Every other combination (ROW_NUMBER partitioned, RANK/DENSE_RANK with * or without PARTITION BY) maps onto Comet's streaming `PartitionedRankLimitExec`. * - * The Scala type parameter is `SparkPlan` (not `WindowGroupLimitExec`) so this file stays - * compilable against Spark 3.4, where the exec class does not exist. Field extraction is + * The Scala type parameter is `SparkPlan` (not `WindowGroupLimitExec`), and field extraction is * delegated to the per-Spark-minor `ShimCometWindowGroupLimit`. */ object CometWindowGroupLimitExec extends CometOperatorSerde[SparkPlan] { /** - * Fields extracted from a Spark `WindowGroupLimitExec` (Spark 3.5+). `mode` is a Spark-agnostic - * string ("Partial" or "Final") because Spark's `WindowGroupLimitMode` type does not exist on - * Spark 3.4, and the enclosing file must compile on that profile. + * Fields extracted from a Spark `WindowGroupLimitExec`. `mode` is a Spark-agnostic string + * ("Partial" or "Final") so the shared serde is independent of Spark's mode type. */ case class Fields( partitionSpec: Seq[Expression], @@ -65,9 +63,8 @@ object CometWindowGroupLimitExec extends CometOperatorSerde[SparkPlan] { op: SparkPlan, builder: Operator.Builder, childOp: OperatorOuterClass.Operator*): Option[OperatorOuterClass.Operator] = { - // Shim returns `None` for a Spark 3.4 plan (WGL does not exist) or if a future Spark - // introduces a rank-like function this shim does not know about. In both cases fall back - // to Spark rather than throwing, so a working query stays working across Spark upgrades. + // If a future Spark version introduces a rank-like function this shim does not know about, + // fall back to Spark rather than throwing so a working query stays working across upgrades. val fields = ShimCometWindowGroupLimit.extract(op) match { case Some(f) => f case None => diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala b/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala index 14ab3554776..cd6e0eb5d42 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala @@ -28,7 +28,7 @@ import org.apache.spark.sql.types.DecimalType * * Spark itself stopped wrapping these in `CheckOverflow` in 3.4 (SPARK-39316), but Comet's native * `CheckOverflow` only validates precision (it does not rescale), so the target type must equal - * the child's actual `dataType`. Always using `expr.dataType` is the safe choice: on Spark 3.4 - + * the child's actual `dataType`. Always using `expr.dataType` is the safe choice: on Spark 3.5 - * 4.0 it equals the value the rule would otherwise recompute from `SQLConf`, and on Spark 4.1+ * (SPARK-53968) it preserves the per-expression `allowDecimalPrecisionLoss` captured at view * creation time. Recomputing from the live `SQLConf` would re-label a stored DEC(38, 17) result diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala index d34e07582ce..bbfefd84bfc 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala @@ -783,7 +783,7 @@ object ArrowCachedBatchSerializer { classOf[java.math.BigInteger], // A relation whose schema this serializer cannot store is delegated to Spark's // DefaultCachedBatchSerializer, so its payload has to survive Kryo too. Spark registers - // DefaultCachedBatch itself only from 4.1 onwards, so on 3.4, 3.5 and 4.0 the delegated path + // DefaultCachedBatch itself only from 4.1 onwards, so on 3.5 and 4.0 the delegated path // fails without this. Registering it twice on 4.1 is a no-op. classOf[DefaultCachedBatch]) } diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManager.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManager.scala index 921994b543c..e361c492cc9 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManager.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManager.scala @@ -821,7 +821,7 @@ private[shuffle] final class CelebornShuffleGenerationCoordinator( taskAttempt: Int): Boolean = outputCommitCoordinator.synchronized { // Spark records task failures here even when no file-output commit was requested. Read that - // record without acquiring its commit slot: Spark 3.4 aborts the whole stage if an authorized + // record without acquiring its commit slot: Spark aborts the whole stage if an authorized // file committer fails, including executor loss before native map output has been written. val stageStatesField = classOf[OutputCommitCoordinator].getDeclaredField("stageStates") stageStatesField.setAccessible(true) diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala index a86e20cb751..8946d4a665c 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala @@ -502,9 +502,9 @@ object CometShuffleExchangeExec // A Comet shuffle wrapped around a stage that still contains a Spark FileSourceScanExec // with DPP produces inefficient row<->columnar transitions. This only happens when the - // scan fell back to Spark (e.g., AQE DPP on Spark 3.4, or unsupported scan type). - // On 3.5+ with AQE DPP, the scan converts to CometNativeScanExec and - // stageContainsDPPScan won't match (it checks FileSourceScanExec). + // scan fell back to Spark because of an unsupported scan type. AQE DPP scans convert to + // CometNativeScanExec, so stageContainsDPPScan won't match them (it checks + // FileSourceScanExec). if (stageContainsDPPScan(s)) { withFallbackReasons(s, Set("Stage contains a scan with Dynamic Partition Pruning")) return None diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala index ce12111dc77..490b7f39cb6 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala @@ -92,9 +92,8 @@ class CometShuffleManager(conf: SparkConf) extends ShuffleManager with Logging { } override val shuffleBlockResolver: IndexShuffleBlockResolver = { - // The patch versions of Spark 3.4 have different constructor signatures: - // See https://github.com/apache/spark/commit/5180694705be3508bd21dd9b863a59b8cb8ba193 - // We look for proper constructor by reflection. + // Supported Spark versions have different constructor signatures, so select the matching + // constructor by reflection. classOf[IndexShuffleBlockResolver].getDeclaredConstructors .filter(c => List(2, 3).contains(c.getParameterCount())) .map { c => diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala b/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala index 5e637c6858d..0bc8c25fce8 100644 --- a/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala +++ b/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala @@ -59,7 +59,7 @@ import com.google.common.base.Objects import com.google.protobuf.CodedOutputStream import org.apache.comet.{CometConf, CometExecIterator, CometRuntimeException, ConfigEntry, ContribServices} -import org.apache.comet.CometSparkSessionExtensions.{isCometShuffleEnabled, isSpark35Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{isCometShuffleEnabled, withFallbackReason} import org.apache.comet.parquet.{CometParquetUtils, HadoopFileOutput} import org.apache.comet.rules.CometExecRule import org.apache.comet.serde.{CometCollectBuffer, CometOperatorSerde, Compatible, OperatorOuterClass, QueryContextInterner, SupportLevel, Unsupported} @@ -2586,11 +2586,11 @@ object CometHashAggregateExec extends CometBaseAggregate[HashAggregateExec] { // decimal sum that leaves the precision, while the native accumulator keeps it unbounded. // Spark turns codegen off by config, for an imperative aggregate, for a non-leaf // CodegenFallback expression, or when the output or an input exceeds its field limit. The - // NO_CODEGEN factory mode turns whole-stage codegen off only from Spark 3.5 (SPARK-44236). + // NO_CODEGEN factory mode also turns whole-stage codegen off (SPARK-44236). val codegenOff = !op.conf.wholeStageEnabled || - (isSpark35Plus && op.conf + op.conf .getConfString(SQLConf.CODEGEN_FACTORY_MODE.key) - .equalsIgnoreCase(CodegenObjectFactoryMode.NO_CODEGEN.toString)) || + .equalsIgnoreCase(CodegenObjectFactoryMode.NO_CODEGEN.toString) || op.aggregateExpressions.exists(_.aggregateFunction.isInstanceOf[ImperativeAggregate]) || WholeStageCodegenExec.isTooManyFields(op.conf, op.schema) || op.children.exists(child => WholeStageCodegenExec.isTooManyFields(op.conf, child.schema)) || diff --git a/spark/src/main/spark-3.5/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala b/spark/src/main/spark-3.5/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala deleted file mode 100644 index ef02700a9c8..00000000000 --- a/spark/src/main/spark-3.5/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.iceberg - -import org.apache.spark.sql.connector.catalog.TableCatalog -import org.apache.spark.sql.execution.SparkPlan - -/** - * Spark 3.5+: CTAS and RTAS run their write as a nested `AppendData` or `OverwriteByExpression` - * query, which is planned and reported like any other write, so no create or replace exec writes - * a table itself. - */ -private[iceberg] object IcebergTableAsSelectShim { - - /** The catalog `plan` writes through, when `plan` is a CTAS or RTAS exec. */ - def writeCatalog(plan: SparkPlan): Option[TableCatalog] = None -} diff --git a/spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala b/spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala index 997321c0152..e9383ae280c 100644 --- a/spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala +++ b/spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala @@ -35,8 +35,6 @@ trait CometExprShim { def binaryOutputStyle: BinaryOutputStyle = BinaryOutputStyle.HEX_DISCRETE - def getJsonObjectNativeFunctionName: String = "get_json_object" - def sparkVersionSpecificStringExpressions : Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(classOf[StringDecode] -> CometStringDecode, classOf[Encode] -> CometEncode) diff --git a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala index e28bdfc55b5..15bf156f89f 100644 --- a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala +++ b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala @@ -45,10 +45,4 @@ trait ShimCometSparkSessionExtensions { rule: Rule[SparkPlan]): Unit = { extensions.injectQueryStageOptimizerRule(_ => rule) } - - // No-op on Spark >= 3.5. See the Spark 3.4 shim and - // CometSpark34AqeDppFallbackRule's class docstring for why this shim exists. - def injectPreSpark35QueryStagePrepRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = {} } diff --git a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometWindowGroupLimit.scala b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometWindowGroupLimit.scala index baca02147a7..4fa9f4c48df 100644 --- a/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometWindowGroupLimit.scala +++ b/spark/src/main/spark-3.5/org/apache/comet/shims/ShimCometWindowGroupLimit.scala @@ -27,8 +27,7 @@ import org.apache.spark.sql.execution.window.WindowGroupLimitExec import org.apache.comet.serde.OperatorOuterClass.RankLikeFunction /** - * Spark 3.5+ shim exposing `WindowGroupLimitExec` (SPARK-37099) to the shared Comet code without - * causing the 3.4 build to fail on a missing class reference. + * Spark 3.5 shim exposing `WindowGroupLimitExec` (SPARK-37099) to the shared Comet code. */ object ShimCometWindowGroupLimit { diff --git a/spark/src/main/spark-3.x/org/apache/comet/cloud/s3/SdkCredentialExtraction.java b/spark/src/main/spark-3.x/org/apache/comet/cloud/s3/SdkCredentialExtraction.java index c9f75b62002..403b43cf7b4 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/cloud/s3/SdkCredentialExtraction.java +++ b/spark/src/main/spark-3.x/org/apache/comet/cloud/s3/SdkCredentialExtraction.java @@ -23,8 +23,8 @@ import com.amazonaws.auth.AWSSessionCredentials; /** - * Maps AWS SDK v1 {@link AWSCredentials} onto {@link CometS3Credentials}. Compiled only into - * spark-3.4 / 3.5 builds (spark-3.x source set), directly against SDK v1. + * Maps AWS SDK v1 {@link AWSCredentials} onto {@link CometS3Credentials}. Compiled only into Spark + * 3.5 builds (spark-3.x source set), directly against SDK v1. */ final class SdkCredentialExtraction { diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala b/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala index 3198035446b..a1b41412657 100644 --- a/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala +++ b/spark/src/main/spark-3.x/org/apache/comet/shims/ShimCometShuffleExchangeExec.scala @@ -26,19 +26,14 @@ import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec import org.apache.spark.sql.types.{StructField, StructType} trait ShimCometShuffleExchangeExec { - // TODO: remove after dropping Spark 3.4 support def apply(s: ShuffleExchangeExec, shuffleType: ShuffleType): CometShuffleExchangeExec = { - val advisoryPartitionSize = s.getClass.getDeclaredMethods - .filter(_.getName == "advisoryPartitionSize") - .flatMap(_.invoke(s).asInstanceOf[Option[Long]]) - .headOption CometShuffleExchangeExec( s.outputPartitioning, s.child, s, s.shuffleOrigin, shuffleType, - advisoryPartitionSize) + s.advisoryPartitionSize) } // TODO: remove after dropping Spark 3.x support diff --git a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala index 9f163da9b2a..229dcb85ca7 100644 --- a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala +++ b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimCometBatchScanExec.scala @@ -25,7 +25,7 @@ import org.apache.spark.sql.execution.datasources.v2.BatchScanExec trait ShimCometBatchScanExec { protected def wrappedScan: BatchScanExec - // Spark 3.4 and 3.5 expose `partitions` as Seq[Seq[InputPartition]] (same as 4.0/4.1); + // Spark 3.5 exposes `partitions` as Seq[Seq[InputPartition]] (same as 4.0/4.1); // 4.2 changed it to Seq[Option[InputPartition]]. The concrete subclass overrides // `partitions` to delegate to `shimPartitions`, which carries the version-specific // element type. diff --git a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala index dd1d30e09ee..a7173c6d8e2 100644 --- a/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala +++ b/spark/src/main/spark-3.x/org/apache/spark/sql/comet/shims/ShimDataSourceRDDPartition.scala @@ -23,7 +23,7 @@ import org.apache.spark.sql.connector.read.InputPartition import org.apache.spark.sql.execution.datasources.v2.DataSourceRDDPartition object ShimDataSourceRDDPartition { - // In Spark 3.4, 3.5, 4.0, and 4.1 a DataSourceRDDPartition holds a Seq[InputPartition] + // In Spark 3.5, 4.0, and 4.1 a DataSourceRDDPartition holds a Seq[InputPartition] // under `inputPartitions`. In 4.2 it was renamed to a single `inputPartition`. def inputPartitions(p: DataSourceRDDPartition): Seq[InputPartition] = p.inputPartitions } diff --git a/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala b/spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala index f0a4c66fe7f..ca3c79930f3 100644 --- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala +++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala @@ -31,8 +31,6 @@ import org.apache.comet.serde.ExprOuterClass.BinaryOutputStyle */ trait CometExprShim extends Spark4xCometExprShim { - def getJsonObjectNativeFunctionName: String = "get_json_object" - def binaryOutputStyle: BinaryOutputStyle = { SQLConf.get .getConf(SQLConf.BINARY_OUTPUT_STYLE) diff --git a/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.1+/org/apache/comet/shims/CometExprShim.scala b/spark/src/main/spark-4.1+/org/apache/comet/shims/CometExprShim.scala index fee16b9f076..5573d045832 100644 --- a/spark/src/main/spark-4.1+/org/apache/comet/shims/CometExprShim.scala +++ b/spark/src/main/spark-4.1+/org/apache/comet/shims/CometExprShim.scala @@ -35,8 +35,6 @@ import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctio */ trait CometExprShim extends Spark4xCometExprShim { - def getJsonObjectNativeFunctionName: String = "get_json_object" - def binaryOutputStyle: BinaryOutputStyle = { // Since Spark 4.1, BINARY_OUTPUT_STYLE is an enumConf so getConf already returns the enum value. SQLConf.get.getConf(SQLConf.BINARY_OUTPUT_STYLE) match { diff --git a/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala b/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala index 69f33a1b29a..f07245263b3 100644 --- a/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala +++ b/spark/src/main/spark-4.1/org/apache/comet/shims/CometCollectShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.catalyst.expressions.aggregate.{CollectList, CollectSet} /** - * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no + * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no * such field: these aggregates always drop null inputs, so this shim reports `true`. Spark 4.2 * added `ignoreNulls` (settable to `false` via `RESPECT NULLS`), handled by the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala b/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala index ff927120687..52cde14eeaa 100644 --- a/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala +++ b/spark/src/main/spark-4.1/org/apache/comet/shims/CometSampleShim.scala @@ -22,7 +22,7 @@ package org.apache.comet.shims import org.apache.spark.sql.execution.SampleExec /** - * Shim for the seed that `SampleExec` samples with. Spark 3.4 through 4.1 carry it as a plain + * Shim for the seed that `SampleExec` samples with. Spark 3.5 through 4.1 carry it as a plain * `Long`. Spark 4.2 made it an `Option[Long]` resolved into a `resolvedSeed` field, handled by * the spark-4.2 shim. */ diff --git a/spark/src/main/spark-4.x/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala b/spark/src/main/spark-4.x/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala deleted file mode 100644 index ef02700a9c8..00000000000 --- a/spark/src/main/spark-4.x/org/apache/comet/iceberg/IcebergTableAsSelectShim.scala +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.comet.iceberg - -import org.apache.spark.sql.connector.catalog.TableCatalog -import org.apache.spark.sql.execution.SparkPlan - -/** - * Spark 3.5+: CTAS and RTAS run their write as a nested `AppendData` or `OverwriteByExpression` - * query, which is planned and reported like any other write, so no create or replace exec writes - * a table itself. - */ -private[iceberg] object IcebergTableAsSelectShim { - - /** The catalog `plan` writes through, when `plan` is a CTAS or RTAS exec. */ - def writeCatalog(plan: SparkPlan): Option[TableCatalog] = None -} diff --git a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala index c78cc6b2d17..cac636c45cf 100644 --- a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala +++ b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala @@ -38,10 +38,4 @@ trait ShimCometSparkSessionExtensions { rule: Rule[SparkPlan]): Unit = { extensions.injectQueryStageOptimizerRule(_ => rule) } - - // No-op on Spark >= 3.5. See the Spark 3.4 shim and - // CometSpark34AqeDppFallbackRule's class docstring for why this shim exists. - def injectPreSpark35QueryStagePrepRuleShim( - extensions: SparkSessionExtensions, - rule: Rule[SparkPlan]): Unit = {} } diff --git a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala index 3157889b431..023340a5055 100644 --- a/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala +++ b/spark/src/main/spark-4.x/org/apache/comet/shims/ShimSQLConf.scala @@ -26,10 +26,9 @@ trait ShimSQLConf { protected val CORRECTED = LegacyBehaviorPolicy.CORRECTED /** - * Reads `spark.sql.execution.arrow.useLargeVarTypes`. Spark 4.x exposes a typed accessor; 3.4 - * lacks it (a 3.5 backport added it, but Comet's 3.x shim collapses both into a single string - * fallback). Forward to the accessor here so callers do not depend on which version they're - * compiled against. + * Reads `spark.sql.execution.arrow.useLargeVarTypes`. Spark 4.x exposes a typed accessor while + * Comet's 3.x shim uses a string fallback. Forward to the accessor here so callers do not + * depend on which version they're compiled against. */ protected def arrowUseLargeVarTypes(conf: SQLConf): Boolean = conf.arrowUseLargeVarTypes } diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql index 10e64c94d6a..647f8503b40 100644 --- a/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql +++ b/spark/src/test/resources/sql-tests/expressions/aggregate/collect_list.sql @@ -463,7 +463,7 @@ SELECT sort_array(collect_list(a)), sort_array(collect_list(b)) FROM cl_src_1764 -- FILTER (WHERE ...): only rows that pass the predicate are collected. -- The Spark analyzer routes this through `applyIgnoreNulls`, which the -- collect_list serde handler is expected to accept as long as ignoreNulls --- remains at its default (true on 3.4-4.1; explicit on 4.2). +-- remains at its default (true through 4.1; explicit on 4.2). -- ============================================================ query diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/mode.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/mode.sql index 6d984e623ee..8829e74f54c 100644 --- a/spark/src/test/resources/sql-tests/expressions/aggregate/mode.sql +++ b/spark/src/test/resources/sql-tests/expressions/aggregate/mode.sql @@ -115,7 +115,7 @@ SELECT grp, mode(b), mode(s), mode(l) FROM mode_nums GROUP BY grp ORDER BY grp -- ============================================================ -- Float / Double -- --- Whether `-0.0` and `0.0` share a frequency-map key is version-dependent: Spark 3.4-4.1 key on +-- Whether `-0.0` and `0.0` share a frequency-map key is version-dependent: Spark 3.5-4.1 key on -- `java.lang.Double.equals` and keep them apart, while Spark 4.2.0+ folds `-0.0` into `0.0` -- (SPARK-57329). Each group below still has a unique winner under both behaviours. -- diff --git a/spark/src/test/resources/sql-tests/expressions/array/array_append_eager_eval.sql b/spark/src/test/resources/sql-tests/expressions/array/array_append_eager_eval.sql index 93e9caa0b8c..ffaa1982292 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/array_append_eager_eval.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/array_append_eager_eval.sql @@ -112,7 +112,7 @@ query expect_native(array_append) SELECT _1, array_append(IF(_1 % 2 = 0, array(array(1)), CAST(NULL AS ARRAY>)), array(1 / (_1 + 1))) AS a FROM array_append_eager --- Without ANSI nothing raises, and the default on Spark 3.4 and 3.5 is without ANSI. A computed +-- Without ANSI nothing raises, and the default on Spark 3.5 is without ANSI. A computed -- item still goes through the extra evaluation for rows whose array is NULL, whose NULL has to be -- typed as the list the other branch returns, or the two cannot be merged. The division by zero -- returns NULL, and the second query has it land on a row whose array is not NULL, so [1, NULL] is diff --git a/spark/src/test/resources/sql-tests/expressions/array/explode.sql b/spark/src/test/resources/sql-tests/expressions/array/explode.sql index a958d9a3907..5cdb53e3f97 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/explode.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/explode.sql @@ -328,7 +328,7 @@ CREATE TABLE test_explode_empty(id int, arr array) USING parquet query SELECT id, explode_outer(arr) FROM test_explode_empty --- ===== TIMESTAMP_NTZ (Spark 3.4+): distinct Arrow layout (no tz) from LTZ ===== +-- ===== TIMESTAMP_NTZ: distinct Arrow layout (no tz) from LTZ ===== statement CREATE TABLE test_explode_ts_ntz(id int, arr array) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql index 024c6be5d32..20f0eb0e8b4 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql @@ -351,7 +351,7 @@ SELECT sort_array(arr) FROM test_sort_array_nested_struct query SELECT sort_array(arr, false) FROM test_sort_array_nested_struct --- Non-literal (foldable) ascendingOrder is exercised in sort_array_min_spark_4_0.sql: Spark 3.4/3.5 +-- Non-literal (foldable) ascendingOrder is exercised in sort_array_min_spark_4_0.sql: Spark 3.5 -- reject a non-literal sort order at analysis time, so those cases are gated to Spark 4.0+. -- literal arguments diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql index e3e0f6342c0..9b6aef10135 100644 --- a/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql +++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql @@ -17,7 +17,7 @@ -- ConfigMatrix: parquet.enable.dictionary=false,true --- Spark 3.4/3.5's SortArray requires the ascendingOrder argument to be a boolean literal and +-- Spark 3.5's SortArray requires the ascendingOrder argument to be a boolean literal and -- rejects any other expression at analysis time. Spark 4.0 relaxed this to accept any foldable -- boolean, so the non-literal-ascendingOrder cases below can only be exercised on Spark 4.0+. -- MinSparkVersion: 4.0 diff --git a/spark/src/test/resources/sql-tests/expressions/bitwise/bit_count_types.sql b/spark/src/test/resources/sql-tests/expressions/bitwise/bit_count_types.sql index 65329056a1f..f224f5d948a 100644 --- a/spark/src/test/resources/sql-tests/expressions/bitwise/bit_count_types.sql +++ b/spark/src/test/resources/sql-tests/expressions/bitwise/bit_count_types.sql @@ -17,11 +17,6 @@ -- ConfigMatrix: parquet.enable.dictionary=false,true --- Spark 3.4.3 emits invalid Java for bit_count(boolean). Use interpreted Spark --- reference execution; plain queries still assert Comet operator replacement. --- Config: spark.sql.codegen.wholeStage=false --- Config: spark.sql.codegen.factoryMode=NO_CODEGEN - statement CREATE TABLE test_bit_count_types(l bigint, i int, s smallint, b tinyint, flag boolean) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql b/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql index 5180f80f04c..53d891557f5 100644 --- a/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql +++ b/spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql @@ -30,7 +30,7 @@ CREATE TABLE cast_trim_pad(name string, pad string) USING parquet -- The padding is materialized here rather than in the queries, so that the cast operands below -- stay inside expressions Comet runs natively. The two multi-byte pads are spelled as casts of --- their UTF-8 bytes: `decode(bin, charset)` resolves to a RuntimeReplaceable on Spark 3.4 and +-- their UTF-8 bytes: `decode(bin, charset)` resolves to a RuntimeReplaceable on Spark 3.5 and -- 4.x, which is not foldable, and an inline table only accepts foldable expressions. statement INSERT INTO cast_trim_pad VALUES diff --git a/spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql b/spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql index 6bc210b83fb..8e69027f80a 100644 --- a/spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql +++ b/spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql @@ -48,7 +48,7 @@ SELECT to_csv(named_struct('a', a, 'b', b, 'c', c), map('sep', ';')) FROM test_t -- timestampFormat over date and timestamp fields: these are the #3232 types that were -- Incompatible before this change and now route through the dispatcher. --- BinaryType is the other #3232 type but is deliberately absent: on Spark 3.4 / 3.5 the CSV +-- BinaryType is the other #3232 type but is deliberately absent: on Spark 3.5 the CSV -- converter has no binary branch and renders it with Java's default Object.toString(), e.g. -- "[B@10bc15e4", an identity hash that differs between any two evaluations, so the value is not -- assertable by any engine including Spark itself. (Spark 4.0 added a real binary formatter, but @@ -75,7 +75,7 @@ SELECT to_csv(named_struct('d', d, 't', t), map('timestampFormat', 'yyyy/MM/dd H -- before the plan commits, so a runtime gap is an execute-time failure with no fallback left. -- -- Only non-nullness is asserted here, because the rendered value is not comparable on every --- supported Spark version. Spark 3.4 / 3.5's `UnivocityGenerator.makeConverter` has no branch for +-- supported Spark version. Spark 3.5's `UnivocityGenerator.makeConverter` has no branch for -- complex types and lands on `getter.get(ordinal, dataType).toString`, an identity string such as -- "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" whose hash differs between any two -- evaluations and whose class differs between Spark's converter input (`ColumnarArray` / diff --git a/spark/src/test/resources/sql-tests/expressions/csv/to_csv_nested.sql b/spark/src/test/resources/sql-tests/expressions/csv/to_csv_nested.sql index fe768ab16bc..44fdee289b8 100644 --- a/spark/src/test/resources/sql-tests/expressions/csv/to_csv_nested.sql +++ b/spark/src/test/resources/sql-tests/expressions/csv/to_csv_nested.sql @@ -16,7 +16,7 @@ -- under the License. -- Full-value coverage of `to_csv` over complex field types, which `to_csv.sql` can only assert for --- non-nullness because Spark 3.4 / 3.5 render a complex field as a Java identity string. Spark 4.0 +-- non-nullness because Spark 3.5 renders a complex field as a Java identity string. Spark 4.0 -- gave `UnivocityGenerator.makeConverter` real array / map / struct branches ("[1, 2, 3]", -- "{k -> 10}", "{5, z}"), built from the typed `getArray` / `getMap` / `getStruct` getters, so from -- 4.0 on the value is deterministic and Comet's kernel-side `CometInternalRow` / `CometArrayData` / diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql b/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql index 16ffa09d8b6..e693e5e917b 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/make_timestamp_ansi.sql @@ -23,7 +23,7 @@ -- class string `DATETIME_FIELD_OUT_OF_BOUNDS` is NOT preserved when the exception is -- thrown from a task on the executor side (only the wrapped `Job aborted ... Lost task -- ... SparkDateTimeException: ` form is preserved), so we match the JDK --- field names which are stable from Spark 3.4 through 4.x. +-- field names which are stable from Spark 3.5 through 4.x. -- Config: spark.sql.session.timeZone=UTC -- Config: spark.sql.ansi.enabled=true -- Config: spark.comet.exec.scalaUDF.codegen.enabled=true diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/timestamp_seconds.sql b/spark/src/test/resources/sql-tests/expressions/datetime/timestamp_seconds.sql index a7cd622119e..bd749e59752 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/timestamp_seconds.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/timestamp_seconds.sql @@ -114,7 +114,7 @@ SELECT c0, timestamp_seconds(c0) FROM test_ts_seconds_dec38_18 -- Spark converts decimal input with longValueExact, so a nonzero digit past microsecond precision -- raises rather than rounds. Spark raises a bare java.lang.ArithmeticException with no error class --- on 3.4 through 4.1, so the expect_error patterns below match the JDK message. +-- on 3.5 through 4.1, so the expect_error patterns below match the JDK message. statement CREATE TABLE test_ts_seconds_dec20_7(c0 decimal(20, 7)) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql b/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql deleted file mode 100644 index 4fed3ff6546..00000000000 --- a/spark/src/test/resources/sql-tests/expressions/datetime/to_unix_timestamp_ansi_spark34.sql +++ /dev/null @@ -1,39 +0,0 @@ --- Licensed to the Apache Software Foundation (ASF) under one --- or more contributor license agreements. See the NOTICE file --- distributed with this work for additional information --- regarding copyright ownership. The ASF licenses this file --- to you under the Apache License, Version 2.0 (the --- "License"); you may not use this file except in compliance --- with the License. You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, --- software distributed under the License is distributed on an --- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --- KIND, either express or implied. See the License for the --- specific language governing permissions and limitations --- under the License. - --- ANSI mode: to_unix_timestamp throws on parse failure. Same coverage as --- to_unix_timestamp_ansi.sql, but the expect_error substring targets the JDK --- DateTimeParseException message text because Spark 3.4 does not yet wrap these in the --- CANNOT_PARSE_TIMESTAMP error class (that classification arrived in Spark 3.5). The JDK --- message "Text 'not a date' could not be parsed" is stable on Spark 3.4. --- Config: spark.sql.session.timeZone=UTC --- Config: spark.sql.ansi.enabled=true --- Config: spark.sql.legacy.timeParserPolicy=CORRECTED --- Config: spark.comet.exec.scalaUDF.codegen.enabled=true --- MaxSparkVersion: 3.4 - -query expect_error(could not be parsed) -SELECT to_unix_timestamp('not a date', 'yyyy-MM-dd') - -query expect_error(could not be parsed) -SELECT to_unix_timestamp('2024-13-99', 'yyyy-MM-dd') - --- Sentinel: confirms Comet ran the expression natively. If the dispatcher silently rejects --- ToUnixTimestamp, the error queries above pass vacuously via Spark fallback. This valid --- query uses `checkSparkAnswerAndOperator` and fails if Comet did not execute it natively. -query -SELECT to_unix_timestamp('2024-06-15 10:30:45', 'yyyy-MM-dd HH:mm:ss') diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql index 99725e8b6f9..40477d6322b 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy.sql @@ -16,7 +16,6 @@ -- under the License. -- Convergent try_to_timestamp(string, format) behavior across all three policies. --- MinSparkVersion: 3.4 -- ConfigMatrix: spark.sql.legacy.timeParserPolicy=LEGACY,CORRECTED,EXCEPTION -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql index ee2b97968b1..c9a6b0dcdba 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_corrected.sql @@ -17,7 +17,6 @@ -- try_to_timestamp() under CORRECTED timeParserPolicy. -- Strict java.time parsing returns null for inputs that legacy would accept. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=CORRECTED -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql index ef4c4c464e2..9936f7992f3 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_exception.sql @@ -19,7 +19,6 @@ -- try_to_timestamp swallows DateTimeException/ParseException, but SparkUpgradeException -- raised by checkParsedDiff propagates through the catch -- so the EXCEPTION case still -- throws on legacy/new divergent inputs. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=EXCEPTION -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql index cee5ed4689d..59b2a7f061a 100644 --- a/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql +++ b/spark/src/test/resources/sql-tests/expressions/datetime/try_to_timestamp_time_parser_policy_legacy.sql @@ -17,7 +17,6 @@ -- try_to_timestamp() under LEGACY timeParserPolicy. -- Lenient SimpleDateFormat parsing accepts inputs that the new formatter rejects. --- MinSparkVersion: 3.4 -- Config: spark.sql.legacy.timeParserPolicy=LEGACY -- Config: spark.sql.session.timeZone=UTC diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql index c1cbfab3358..406ee1a15ca 100644 --- a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql +++ b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql @@ -16,7 +16,7 @@ -- under the License. -- With spark.sql.legacy.disableMapKeyNormalization, Spark 4.0+ leaves FLOAT and DOUBLE map keys as --- they are, as Spark 3.4 and 3.5 always do (they don't have the flag). So 0.0 and -0.0 are two +-- they are, as Spark 3.5 always does (it doesn't have the flag). So 0.0 and -0.0 are two -- keys and a -0.0 key stays -0.0. map_from_entries and map_from_arrays run Spark's code in the -- codegen dispatcher, which has to see the session's flag. diff --git a/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql b/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql index 7ce2b346461..e32d1e34073 100644 --- a/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql +++ b/spark/src/test/resources/sql-tests/expressions/misc/equal_null.sql @@ -15,8 +15,6 @@ -- specific language governing permissions and limitations -- under the License. --- MinSparkVersion: 3.4 - statement CREATE TABLE test_equal_null(a int, b int) USING parquet diff --git a/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql b/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql index 260e729f43a..be71c7cfe96 100644 --- a/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql +++ b/spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql @@ -69,7 +69,7 @@ SELECT uuid(42) FROM (SELECT id FROM test_uuid_parts DISTRIBUTE BY id) -- cannot be asserted here: this suite compares Comet against Spark, so if the plan ever collapsed -- to one partition both engines would collapse identically and any check would still pass. The -- partition count is asserted directly in CometUuidExpressionSuite ("across multiple partitions"), --- which checks getNumPartitions > 1 before comparing, and covers 3.4/3.5 as well. +-- which checks getNumPartitions > 1 before comparing. -- Aliased seeded projections: both nodes carry the same explicit seed, so freshCopyIfContainsStatefulExpression -- must not accidentally hoist a shared instance -- each row must satisfy uuid(0) = uuid(0). diff --git a/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql b/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql index 79a252bb868..d4c109b82eb 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/base64_unchunked.sql @@ -17,8 +17,6 @@ -- Exercises the unchunked base64 path (chunk = false in the native function). -- On Spark 3.5+ this config produces single-line output with no CRLF separators. --- On Spark 3.4 the config does not exist and is ignored, so both engines still chunk; --- the file is harmless there and Comet still matches Spark. -- Config: spark.sql.chunkBase64String.enabled=false statement diff --git a/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql b/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql index 5186a171db5..58bd98fb2c9 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/decode_invalid_utf8.sql @@ -17,7 +17,7 @@ -- decode() over invalid UTF-8 byte sequences with legacy / replacement-character semantics. -- --- On Spark 3.4 and 3.5 `decode(bin, charset)` always substitutes the Unicode replacement +-- On Spark 3.5 `decode(bin, charset)` always substitutes the Unicode replacement -- character for malformed sequences (it lowers to `new String(bytes, charset)`, which uses the -- JVM's default replace-on-error decoder). -- On Spark 4.0+ the same substitute behavior is selected by enabling both diff --git a/spark/src/test/resources/sql-tests/expressions/string/encode.sql b/spark/src/test/resources/sql-tests/expressions/string/encode.sql index 1d21864bdb6..438fc041a65 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/encode.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/encode.sql @@ -59,7 +59,7 @@ query SELECT encode(s, 'ISO-8859-1') FROM test_encode -- US-ASCII: use ASCII-only input so this file stays version-agnostic. Unmappable input under --- US-ASCII substitutes on Spark 3.4/3.5 and throws on Spark 4.0+, so it is covered by the paired +-- US-ASCII substitutes on Spark 3.5 and throws on Spark 4.0+, so it is covered by the paired -- fixtures encode_unmappable.sql and encode_unmappable_strict.sql instead. statement diff --git a/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset.sql b/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset.sql index 31f1e8fd710..4061baf26e1 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset.sql @@ -15,9 +15,9 @@ -- specific language governing permissions and limitations -- under the License. --- encode() charset validation on Spark 3.4 and 3.5. +-- encode() charset validation on Spark 3.5. -- --- On those versions `Encode` calls `String.getBytes(charset)` directly, so any charset the JVM +-- On that version `Encode` calls `String.getBytes(charset)` directly, so any charset the JVM -- knows is accepted (including ones outside Spark 4.0's allowlist, e.g. windows-1252) and an -- unknown name surfaces as `UnsupportedEncodingException` through `Platform.throwException`. -- Spark 4.0+ instead routes through `CharsetProvider.forName`, which rejects anything outside diff --git a/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset_strict.sql b/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset_strict.sql index 4c932b2cda6..8fc59a2bed4 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset_strict.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/encode_invalid_charset_strict.sql @@ -19,7 +19,7 @@ -- -- `CharsetProvider.forName` permits only us-ascii, iso-8859-1, utf-8, utf-16be, utf-16le, utf-16 -- and utf-32 unless `spark.sql.legacy.javaCharsets` is enabled, and otherwise raises --- `INVALID_PARAMETER_VALUE.CHARSET`. On Spark 3.4/3.5 a JVM charset outside that list is simply +-- `INVALID_PARAMETER_VALUE.CHARSET`. On Spark 3.5 a JVM charset outside that list is simply -- accepted; see encode_invalid_charset.sql. Enabling the legacy flag restores that behavior on -- 4.0+; see encode_legacy_charsets.sql. -- MinSparkVersion: 4.0 diff --git a/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable.sql b/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable.sql index 39b7ec580d5..f197a65585b 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable.sql @@ -15,9 +15,9 @@ -- specific language governing permissions and limitations -- under the License. --- encode() over characters the target charset cannot represent, on Spark 3.4 and 3.5. +-- encode() over characters the target charset cannot represent, on Spark 3.5. -- --- On those versions `Encode` is a plain BinaryExpression whose eval is +-- On that version `Encode` is a plain BinaryExpression whose eval is -- `input.toString.getBytes(charset)`, and `String.getBytes` uses an encoder configured with -- `CodingErrorAction.REPLACE`. Unmappable characters therefore become the charset's replacement -- byte (0x3F, '?') rather than raising. Spark 4.0+ builds the encoder with diff --git a/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable_strict.sql b/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable_strict.sql index b5f0bdcd72a..2b4b537dc36 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable_strict.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/encode_unmappable_strict.sql @@ -20,7 +20,7 @@ -- -- Spark 4.0 added `spark.sql.legacy.codingErrorAction` (default `false`), which builds the -- encoder with `CodingErrorAction.REPORT`. `Encode.encode` converts the resulting --- `CharacterCodingException` into `MALFORMED_CHARACTER_CODING`. On Spark 3.4/3.5 the same input +-- `CharacterCodingException` into `MALFORMED_CHARACTER_CODING`. On Spark 3.5 the same input -- is silently substituted; that branch is covered by the paired fixture encode_unmappable.sql. -- MinSparkVersion: 4.0 -- Config: spark.comet.exec.scalaUDF.codegen.enabled=true diff --git a/spark/src/test/resources/sql-tests/expressions/string/get_json_object.sql b/spark/src/test/resources/sql-tests/expressions/string/get_json_object.sql index 4e2fa2164c0..da5a7368498 100644 --- a/spark/src/test/resources/sql-tests/expressions/string/get_json_object.sql +++ b/spark/src/test/resources/sql-tests/expressions/string/get_json_object.sql @@ -243,10 +243,8 @@ SELECT get_json_object('{"a":[{"b":[1,2]},{"b":[3]}]}', '$.a[*].b[*]') query SELECT get_json_object('[[[1,2],[]]]', '$[*][*][*]'), get_json_object('[[[1,2]]]', '$[*][*][*]') --- Spark 3.5+ rejects numbers whose digit count exceeds Jackson's default --- 1000-digit limit, including values the path never selects. Spark 3.4's --- default Jackson version (2.14) does not impose this limit, but Comet follows --- whichever Jackson version is on the classpath. +-- Spark rejects numbers whose digit count exceeds Jackson's default +-- 1000-digit limit, including values the path never selects. query SELECT get_json_object(concat('[{"a":1,"b":', repeat('9', 1000), '}]'), '$[*].a'), get_json_object(concat('[{"a":1,"b":', repeat('9', 1001), '}]'), '$[*].a'), @@ -262,9 +260,8 @@ SELECT get_json_object(concat('[{"a":1,"b":', repeat('9', 1000), '}]'), '$[*].a' -- Spark's result can change when a previous parse leaves a larger buffer in -- Jackson's thread-local recycler. The opt-in native path uses a fixed rule. --- Spark 3.5+ rejects a document deeper than Jackson's default 1000-level --- nesting limit, including a subtree skipped by the wildcard path. Spark 3.4 --- has no default limit. +-- Spark rejects a document deeper than Jackson's default 1000-level +-- nesting limit, including a subtree skipped by the wildcard path. query SELECT get_json_object(concat('[{"a":1,"skip":', repeat('[', 998), '0', repeat(']', 998), '}]'), '$[*].a'), get_json_object(concat('[{"a":1,"skip":', repeat('[', 999), '0', repeat(']', 999), '}]'), '$[*].a'), diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q33/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q33/extended.txt deleted file mode 100644 index 759277aad55..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q33/extended.txt +++ /dev/null @@ -1,96 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.customer_address - : +- CometBroadcastExchange - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales - : : : : +- ReusedSubquery - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.customer_address - : +- CometBroadcastExchange - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.web_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.customer_address - +- CometBroadcastExchange - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 55 out of 55 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 8 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt deleted file mode 100644 index 84b0f44323b..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt +++ /dev/null @@ -1,59 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometSortMergeJoin - : : :- CometSort - : : : +- CometProject - : : : +- CometFilter - : : : +- CometWindowExec - : : : +- CometWindowGroupLimitExec - : : : +- CometSort - : : : +- CometExchange - : : : +- CometWindowGroupLimitExec - : : : +- CometSort - : : : +- CometFilter - : : : : +- Subquery - : : : : +- CometColumnarToRow - : : : : +- CometHashAggregate - : : : : +- CometExchange - : : : : +- CometHashAggregate - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSort - : : +- CometProject - : : +- CometFilter - : : +- CometWindowExec - : : +- CometWindowGroupLimitExec - : : +- CometSort - : : +- CometExchange - : : +- CometWindowGroupLimitExec - : : +- CometSort - : : +- CometFilter - : : : +- ReusedSubquery - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 44 out of 45 eligible operators (97%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt deleted file mode 100644 index fb2ca0d3972..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt +++ /dev/null @@ -1,111 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : : +- Subquery - : : : : : +- CometColumnarToRow - : : : : : +- CometProject - : : : : : +- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.item - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : : +- Subquery - : : : +- CometColumnarToRow - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometFilter - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : : +- Subquery - : : +- CometColumnarToRow - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometFilter - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.web_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometFilter - : +- Subquery - : +- CometColumnarToRow - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 54 out of 55 eligible operators (98%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt deleted file mode 100644 index 7eff14e00cb..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt +++ /dev/null @@ -1,40 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometFilter - +- CometWindowExec - +- CometWindowGroupLimitExec - +- CometSort - +- CometExchange - +- CometWindowGroupLimitExec - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt deleted file mode 100644 index e05ef30b1c8..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt +++ /dev/null @@ -1,56 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometWindowExec - +- CometWindowGroupLimitExec - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 44 out of 44 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt deleted file mode 100644 index d7249c9e3e3..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt +++ /dev/null @@ -1,104 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_returns - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometFilter - : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometBroadcastHashJoin - : : : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.item - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometBroadcastHashJoin - : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.item - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.web_returns - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt index 8e453677791..759277aad55 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt @@ -93,4 +93,4 @@ CometColumnarToRow +- CometFilter +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 63 out of 63 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 8 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 55 out of 55 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 8 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt index 75731d067a0..84b0f44323b 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt @@ -10,44 +10,43 @@ CometColumnarToRow : : : +- CometProject : : : +- CometFilter : : : +- CometWindowExec - : : : +- CometSort - : : : +- CometExchange - : : : +- CometFilter - : : : : +- Subquery - : : : : +- CometColumnarToRow - : : : : +- CometHashAggregate - : : : : +- CometExchange - : : : : +- CometHashAggregate - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometWindowGroupLimitExec + : : : +- CometSort + : : : +- CometExchange + : : : +- CometWindowGroupLimitExec + : : : +- CometSort + : : : +- CometFilter + : : : : +- Subquery + : : : : +- CometColumnarToRow + : : : : +- CometHashAggregate + : : : : +- CometExchange + : : : : +- CometHashAggregate + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometHashAggregate + : : : +- CometExchange + : : : +- CometHashAggregate + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales : : +- CometSort : : +- CometProject : : +- CometFilter : : +- CometWindowExec - : : +- CometSort - : : +- CometExchange - : : +- CometFilter - : : : +- Subquery - : : : +- CometColumnarToRow - : : : +- CometHashAggregate - : : : +- CometExchange - : : : +- CometHashAggregate - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometHashAggregate - : : +- CometExchange - : : +- CometHashAggregate - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- CometWindowGroupLimitExec + : : +- CometSort + : : +- CometExchange + : : +- CometWindowGroupLimitExec + : : +- CometSort + : : +- CometFilter + : : : +- ReusedSubquery + : : +- CometHashAggregate + : : +- CometExchange + : : +- CometHashAggregate + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales : +- CometBroadcastExchange : +- CometProject : +- CometFilter @@ -57,4 +56,4 @@ CometColumnarToRow +- CometFilter +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 35 out of 36 eligible operators (97%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 44 out of 45 eligible operators (97%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt index a1e2b95b2d0..fb2ca0d3972 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt @@ -58,17 +58,7 @@ CometColumnarToRow : : +- CometBroadcastHashJoin : : :- CometFilter : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : : +- ReusedSubquery - : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- ReusedSubquery : : +- CometBroadcastExchange : : +- CometProject : : +- CometFilter @@ -81,7 +71,11 @@ CometColumnarToRow : +- CometBroadcastExchange : +- CometProject : +- CometFilter - : : +- ReusedSubquery + : : +- Subquery + : : +- CometColumnarToRow + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometBroadcastExchange +- CometFilter @@ -107,7 +101,11 @@ CometColumnarToRow +- CometBroadcastExchange +- CometProject +- CometFilter - : +- ReusedSubquery + : +- Subquery + : +- CometColumnarToRow + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 64 out of 65 eligible operators (98%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 54 out of 55 eligible operators (98%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 16 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt index 34f837cc4d0..7eff14e00cb 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt @@ -2,36 +2,39 @@ CometColumnarToRow +- CometTakeOrderedAndProject +- CometFilter +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometExpand - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange + +- CometWindowGroupLimitExec + +- CometSort + +- CometExchange + +- CometWindowGroupLimitExec + +- CometSort + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometExpand +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 30 out of 30 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt index 0a9b1d2dbb7..e05ef30b1c8 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt @@ -32,24 +32,25 @@ CometColumnarToRow +- CometProject +- CometFilter +- CometWindowExec - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim + +- CometWindowGroupLimitExec + +- CometSort + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- ReusedSubquery + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 43 out of 43 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 44 out of 44 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt index 9dbc72b2944..d7249c9e3e3 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt @@ -54,20 +54,7 @@ CometColumnarToRow : : +- CometBroadcastHashJoin : : :- CometFilter : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- ReusedSubquery : : +- CometBroadcastExchange : : +- CometProject : : +- CometFilter @@ -114,4 +101,4 @@ CometColumnarToRow +- CometFilter +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 62 out of 62 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt deleted file mode 100644 index d32e1f42b9b..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt +++ /dev/null @@ -1,288 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometFilter - +- CometWindowExec - +- CometWindowGroupLimitExec - +- CometSort - +- CometExchange - +- CometWindowGroupLimitExec - +- CometSort - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item - -Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt deleted file mode 100644 index 185ce8a5247..00000000000 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt +++ /dev/null @@ -1,159 +0,0 @@ -CometColumnarToRow -+- CometTakeOrderedAndProject - +- CometProject - +- CometWindowExec - +- CometSort - +- CometExchange - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometWindowExec - : +- CometWindowGroupLimitExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometWindowExec - : +- CometWindowGroupLimitExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- CometSubqueryBroadcast - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim - +- CometBroadcastExchange - +- CometProject - +- CometBroadcastHashJoin - :- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometWindowExec - +- CometWindowGroupLimitExec - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim - -Comet accelerated 55 out of 55 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt index a6c75ac37db..d32e1f42b9b 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt @@ -2,284 +2,287 @@ CometColumnarToRow +- CometTakeOrderedAndProject +- CometFilter +- CometWindowExec - +- CometSort - +- CometExchange - +- CometUnion - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - :- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometProject - : : : +- CometBroadcastHashJoin - : : : :- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : : +- CometSubqueryBroadcast - : : : : +- CometBroadcastExchange - : : : : +- CometProject - : : : : +- CometFilter - : : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.item - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate + +- CometWindowGroupLimitExec + +- CometSort + +- CometExchange + +- CometWindowGroupLimitExec + +- CometSort + +- CometUnion + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item + :- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometProject + : : : +- CometBroadcastHashJoin + : : : :- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : : +- CometSubqueryBroadcast + : : : : +- CometBroadcastExchange + : : : : +- CometProject + : : : : +- CometFilter + : : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.item +- CometHashAggregate +- CometExchange +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- CometSubqueryBroadcast - : : : +- CometBroadcastExchange - : : : +- CometProject - : : : +- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.date_dim - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.date_dim - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.item + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- CometSubqueryBroadcast + : : : +- CometBroadcastExchange + : : : +- CometProject + : : : +- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.date_dim + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.item -Comet accelerated 62 out of 62 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt index 594ae116008..185ce8a5247 100644 --- a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt +++ b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt @@ -35,25 +35,26 @@ CometColumnarToRow : +- CometProject : +- CometFilter : +- CometWindowExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometWindowGroupLimitExec + : +- CometSort + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- ReusedSubquery + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim :- CometHashAggregate : +- CometExchange : +- CometHashAggregate @@ -84,25 +85,26 @@ CometColumnarToRow : +- CometProject : +- CometFilter : +- CometWindowExec - : +- CometSort - : +- CometHashAggregate - : +- CometExchange - : +- CometHashAggregate - : +- CometProject - : +- CometBroadcastHashJoin - : :- CometProject - : : +- CometBroadcastHashJoin - : : :- CometFilter - : : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : : +- ReusedSubquery - : : +- CometBroadcastExchange - : : +- CometProject - : : +- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.date_dim + : +- CometWindowGroupLimitExec + : +- CometSort + : +- CometHashAggregate + : +- CometExchange + : +- CometHashAggregate + : +- CometProject + : +- CometBroadcastHashJoin + : :- CometProject + : : +- CometBroadcastHashJoin + : : :- CometFilter + : : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : : +- ReusedSubquery + : : +- CometBroadcastExchange + : : +- CometProject + : : +- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.date_dim +- CometHashAggregate +- CometExchange +- CometHashAggregate @@ -133,24 +135,25 @@ CometColumnarToRow +- CometProject +- CometFilter +- CometWindowExec - +- CometSort - +- CometHashAggregate - +- CometExchange - +- CometHashAggregate - +- CometProject - +- CometBroadcastHashJoin - :- CometProject - : +- CometBroadcastHashJoin - : :- CometFilter - : : +- CometNativeScan parquet spark_catalog.default.store_sales - : : +- ReusedSubquery - : +- CometBroadcastExchange - : +- CometProject - : +- CometFilter - : +- CometNativeScan parquet spark_catalog.default.store - +- CometBroadcastExchange - +- CometProject - +- CometFilter - +- CometNativeScan parquet spark_catalog.default.date_dim + +- CometWindowGroupLimitExec + +- CometSort + +- CometHashAggregate + +- CometExchange + +- CometHashAggregate + +- CometProject + +- CometBroadcastHashJoin + :- CometProject + : +- CometBroadcastHashJoin + : :- CometFilter + : : +- CometNativeScan parquet spark_catalog.default.store_sales + : : +- ReusedSubquery + : +- CometBroadcastExchange + : +- CometProject + : +- CometFilter + : +- CometNativeScan parquet spark_catalog.default.store + +- CometBroadcastExchange + +- CometProject + +- CometFilter + +- CometNativeScan parquet spark_catalog.default.date_dim -Comet accelerated 54 out of 54 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file +Comet accelerated 55 out of 55 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch. \ No newline at end of file diff --git a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala index 63b40fe3e2e..a3f81b92df5 100644 --- a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala @@ -34,7 +34,7 @@ import org.apache.spark.sql.functions._ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{ArrayType, DataType, DoubleType, FloatType, IntegerType, LongType, StringType, StructType} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.DataTypeSupport.isComplexType import org.apache.comet.serde.{ArrayElementEqualitySupport, ArraySetSupport, CometArrayDistinct, CometArrayExcept, CometArrayJoin, CometArrayRemove, CometArrayReverse, CometArraysOverlap, CometArrayUnion, CometFlatten, CometSequence, Compatible, ExprOuterClass, Incompatible, SparkFixes} import org.apache.comet.testing.{DataGenOptions, ParquetGenerator, SchemaGenOptions} @@ -326,7 +326,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp } test("array_prepend") { - assume(isSpark35Plus) // in Spark 3.5 array_prepend is implemented via array_insert withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[ArrayInsert]) -> "true") { Seq(true, false).foreach { dictionaryEnabled => withTempDir { dir => @@ -1677,7 +1676,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp // drops the earlier entry under `LAST_WIN`, so `CometLiteral` declines expansion under either // policy and Spark evaluates the projection. test("folded map literal with duplicate keys falls back (multirow)") { - assume(isSpark35Plus) withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") { Seq("EXCEPTION", "LAST_WIN").foreach { policy => withSQLConf(SQLConf.MAP_KEY_DEDUP_POLICY.key -> policy) { @@ -1694,7 +1692,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp // check has to compare keys the way `ArrayBasedMapBuilder` does, through // `TypeUtils.getInterpretedOrdering`. test("folded map literal with duplicate binary keys falls back (multirow)") { - assume(isSpark35Plus) withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") { Seq("EXCEPTION", "LAST_WIN").foreach { policy => withSQLConf(SQLConf.MAP_KEY_DEDUP_POLICY.key -> policy) { @@ -1738,7 +1735,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp } test("array_insert on non-null element array from local table scan (#4789)") { - assume(isSpark35Plus) withLocalTableScanNoFold { import testImplicits._ val df = Seq(Seq(1, 2, 3), Seq(4, 5)).toDF("x") diff --git a/spark/src/test/scala/org/apache/comet/CometBroadcastBuildSideSuite.scala b/spark/src/test/scala/org/apache/comet/CometBroadcastBuildSideSuite.scala index a1cb9ffdec2..b4bf8afd1ff 100644 --- a/spark/src/test/scala/org/apache/comet/CometBroadcastBuildSideSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometBroadcastBuildSideSuite.scala @@ -32,8 +32,6 @@ import org.apache.spark.sql.execution.exchange.ReusedExchangeExec import org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoinExec import org.apache.spark.sql.internal.SQLConf -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus - // Tests for the broadcast build-side bridge, spark.comet.convert.broadcastBuildSide.enabled (issue // #6008). A build-side scan that Comet cannot read natively, such as ORC or an Iceberg table // the native reader declines, is converted to Arrow so that the broadcast join runs natively. Each @@ -214,7 +212,7 @@ class CometBroadcastBuildSideSuite extends CometTestBase with CometIcebergTestBa } } - test("Spark 3.4: native Iceberg fact, ORC dimension, AQE DPP (#6008)") { + test("native Iceberg fact, ORC dimension, AQE DPP (#6008)") { assume(icebergAvailable, "Iceberg not available in classpath") withTempIcebergDir { warehouseDir => withTempDir { dir => @@ -644,7 +642,6 @@ class CometBroadcastBuildSideSuite extends CometTestBase with CometIcebergTestBa // build (the ORC scan) is not bridged, so CometPlanAdaptiveDynamicPruningFilters must NOT wrap // it in a CometBroadcastExchangeExec (that crashed with "Comet execution only takes Arrow // Arrays"). It falls back to a Spark DPP broadcast while the join stays native. See issue #6008. - assume(isSpark35Plus, "Native AQE DPP requires Spark 3.5+") withSQLConf( SQLConf.USE_V1_SOURCE_LIST.key -> "parquet", SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", @@ -846,63 +843,11 @@ class CometBroadcastBuildSideSuite extends CometTestBase with CometIcebergTestBa } } - test("Spark 3.4 AQE DPP with a V1 Parquet fact and an ORC dimension (#6008)") { - // On Spark 3.4, CometScanRule.transformV1Scan has an "AQE Dynamic Partition Pruning requires - // Spark 3.5+" fallback that keeps a V1 scan with an AQE DPP filter on Spark. So this Parquet - // fact is not native, hasOnlyNativeScans is false, and the build side is never tagged or - // bridged. The feature stays out of the way on 3.4: DPP stays live and the answer matches - // Spark. The SKIP_COMET_BROADCAST_TAG path needs a native fact; the Iceberg test in - // CometBroadcastBuildSideSuite covers it. See #6008. - assume(!isSpark35Plus, "This test covers the Spark 3.4 V1-fact DPP fallback") - withSQLConf( - SQLConf.USE_V1_SOURCE_LIST.key -> "parquet", - SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", - SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true", - SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true") { - withTempDir { dir => - val factPath = s"${dir.getAbsolutePath}/fact.parquet" - val dimPath = s"${dir.getAbsolutePath}/dim.orc" - withSQLConf(CometConf.COMET_ENABLED.key -> "false") { - spark - .range(0, 100) - .selectExpr("CAST(id % 10 AS STRING) AS fk", "id AS v") - .write - .partitionBy("fk") - .parquet(factPath) - spark.range(0, 10).selectExpr("CAST(id AS STRING) AS value").write.orc(dimPath) - } - spark.read.parquet(factPath).createOrReplaceTempView("dpp34_fact_orc") - spark.read.orc(dimPath).createOrReplaceTempView("dpp34_dim_orc") - // CAST(d.value AS INT) % 2 = 0 is a filter Spark cannot infer onto f.fk, so the fact is - // pruned only via a live DPP subquery. - val df = sql(""" - |SELECT /*+ BROADCAST(d) */ f.v - |FROM dpp34_fact_orc f JOIN dpp34_dim_orc d ON f.fk = d.value - |WHERE CAST(d.value AS INT) % 2 = 0 - |""".stripMargin) - val (_, cometPlan) = checkSparkAnswer(df) - // The non-native V1 fact means the build side is never tagged, so nothing is bridged. - assert( - collect(cometPlan) { case s: CometSparkToColumnarExec => s }.isEmpty, - s"3.4: the ORC dimension must not be bridged:\n${cometPlan.treeString}") - // DPP is still live - the fact scan keeps a broadcast pruning subquery. - val dppSubqueries = collectWithSubqueries(cometPlan) { - case s: SubqueryBroadcastExec => s - case s: CometSubqueryBroadcastExec => s - } - assert( - dppSubqueries.nonEmpty, - s"3.4: expected a live DPP broadcast subquery on the fact scan:\n${cometPlan.treeString}") - } - } - } - test("AQE DPP reuses the broadcast for a UNION ALL dimension (#6008)") { // The DPP subquery's build is a CometUnionExec - a columnar CometPlan, not a CometNativeExec. // isNativeBuildSide must accept it so CometPlanAdaptiveDynamicPruningFilters converts the // subquery's SubqueryBroadcastExec and Spark reuses the join's CometBroadcastExchangeExec as a // ReusedExchangeExec. The earlier CometNativeExec-only check dropped the reuse. See #6008. - assume(isSpark35Plus, "Native AQE DPP requires Spark 3.5+") withSQLConf( SQLConf.USE_V1_SOURCE_LIST.key -> "parquet", SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", diff --git a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala index 46b0a94af33..4abd96100b4 100644 --- a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala @@ -34,7 +34,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.internal.SQLConf.SESSION_LOCAL_TIMEZONE import org.apache.spark.sql.types._ -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} import org.apache.comet.serde.{CometAdd, CometDivide, CometIntegralDivide, CometMultiply, CometRemainder, CometSubtract, Compatible, Unsupported} import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator} @@ -47,7 +47,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { private def arithmeticError(error: Throwable): SparkThrowable = causeChain(error) .collectFirst { - // SparkArithmeticException is private[spark] in Spark 3.4, so this cross-version test + // SparkArithmeticException is private[spark] in Spark 3.x, so this cross-version test // cannot pattern match on its type directly. case error: SparkThrowable if error.getClass.getName == "org.apache.spark.SparkArithmeticException" => @@ -2419,7 +2419,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { test("null IN empty list honours legacy null-in-empty behavior") { // Spark returns NULL for a NULL operand against an empty IN list when the legacy behavior is - // in effect (SPARK-44550): always on Spark 3.4, on by default on Spark 3.5, and whenever ANSI + // in effect (SPARK-44550): on by default on Spark 3.5 and whenever ANSI // mode is disabled on Spark 4.0+. Comet's native `in` kernel always returns false, so the // legacy case must leave the native path. Empty IN lists are not expressible in SQL and // `OptimizeIn` folds them away, so build the expression via the DataFrame API with that rule @@ -2448,7 +2448,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { col("a").isin(), !col("a").isin(), getColumnFromExpression(emptySet)) - val legacyEnabled = !CometSparkSessionExtensions.isSpark35Plus || + val legacyEnabled = legacy.map(_.toBoolean).getOrElse(!isSpark40Plus || !ansi.toBoolean) if (!legacyEnabled) { checkSparkAnswerAndImpl(df, native = Seq("in", "inset")) @@ -2466,7 +2466,6 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { } test("make_date is not evaluated by empty IN predicates") { - assume(isSpark35Plus, "Spark 3.4 always uses legacy empty-IN evaluation") withSQLConf( SQLConf.ANSI_ENABLED.key -> "true", "spark.sql.legacy.nullInEmptyListBehavior" -> "false", @@ -3020,7 +3019,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { CometConf.COMET_EXEC_ENABLED.key -> "true")(f) } - // Spark 3.4/3.5 throw `_LEGACY_ERROR_TEMP_2043` ("- caused overflow.") for byte and + // Spark 3.5 throws `_LEGACY_ERROR_TEMP_2043` ("- caused overflow.") for byte and // short. Spark 4.x routes them through `MathUtils.negateExact` and agrees with Comet, which // always renders `SparkError::ArithmeticOverflow` with the Spark type name. def sparkOverflowMsg(dtype: String): String = @@ -3637,12 +3636,6 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { ("INT", Int.MinValue.toLong, Int.MaxValue.toLong, "ARITHMETIC_OVERFLOW"), ("BIGINT", Long.MinValue, Long.MaxValue, "ARITHMETIC_OVERFLOW"))) { test(s"ANSI integral overflow fidelity - $sqlType") { - val expectedErrorClass = - if (errorClass == "BINARY_ARITHMETIC_OVERFLOW" && !isSpark35Plus) { - "_LEGACY_ERROR_TEMP_2044" - } else { - errorClass - } val cases = Seq( ("+", "try_add", max, 1L), ("+", "try_add", min, -1L), @@ -3678,7 +3671,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { } val expected = structured(sparkError) val actual = structured(cometError) - assert(expected.getErrorClass == expectedErrorClass, query) + assert(expected.getErrorClass == errorClass, query) assert(actual.getClass == expected.getClass, query) assert(actual.getErrorClass == expected.getErrorClass, query) assert(actual.getSqlState == expected.getSqlState, query) diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala index 85fd3ad3f43..46740ddb20e 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala @@ -48,7 +48,7 @@ import org.apache.spark.sql.functions.col import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{ArrayType, IntegerType, StringType, StructType, TimestampType} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} import org.apache.comet.IcebergScanPlanData.{fileScanTaskCount, partitionPools, partitionScans, storedPools} import org.apache.comet.cloud.s3.MinioLocationScopedCredentialProvider import org.apache.comet.iceberg.{IcebergReflection, RESTCatalogHelper} @@ -5258,14 +5258,12 @@ class CometIcebergNativeSuite assert(numPartitions == 1, s"Expected DPP to prune to 1 partition but got $numPartitions") // Verify AQE DPP used CometSubqueryBroadcastExec with broadcast reuse - if (isSpark35Plus) { - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } spark.sql("DROP TABLE runtime_cat.db.multi_dpp_fact") @@ -5353,11 +5351,9 @@ class CometIcebergNativeSuite s"Expected DPP to prune to 1 of 3 files but planned $plannedTasks tasks:\n$cometPlan") // Verify AQE DPP used CometSubqueryBroadcastExec with broadcast reuse - if (isSpark35Plus) { - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - assertCsbBroadcastReuse(subqueries, cometPlan) - } + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + assertCsbBroadcastReuse(subqueries, cometPlan) spark.sql("DROP TABLE runtime_cat.db.fact_table") } @@ -5912,7 +5908,6 @@ class CometIcebergNativeSuite test("AQE DPP remains executable when transition reversion restores an Iceberg scan") { assume(icebergAvailable, "Iceberg not available") - assume(isSpark35Plus, "Comet AQE DPP query-stage optimizer rules require Spark 3.5+") withTempIcebergDir { warehouseDir => val dimDir = new File(warehouseDir, "dim_parquet") @@ -6127,37 +6122,35 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Verify CometSubqueryBroadcastExec replaced SubqueryAdaptiveBroadcastExec - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - // Verify broadcast reuse: subquery child should be an ASPE that contains a - // BroadcastQueryStageExec (via AQE stageCache, possibly wrapped in - // ReusedExchangeExec). Reference equality (eq) on the join's BQS does not - // hold because convertSAB wraps a fresh exchange in a new ASPE; the actual - // reuse manifests as ReusedExchangeExec inside the ASPE's final plan. - assertCsbBroadcastReuse(subqueries, cometPlan) - - // Verify partition pruning. Iceberg packs these small files into one Spark partition - // whether or not DPP prunes, so count the planned file tasks and the splits read: only - // the file for 1970-01-02 of the 3 dates should be left. - val icebergScans = collectIcebergNativeScans(cometPlan) - assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") - val scan = icebergScans.head - val plannedTasks = fileScanTaskCount(scan) - val numSplits = scan.metrics("num_splits").value + // Verify CometSubqueryBroadcastExec replaced SubqueryAdaptiveBroadcastExec + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + subqueries.foreach { sub => assert( - plannedTasks == 1 && numSplits == 1, - s"Expected DPP to prune to 1 of 3 files, planned $plannedTasks tasks and read " + - s"$numSplits splits:\n${cometPlan.treeString}") + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } + // Verify broadcast reuse: subquery child should be an ASPE that contains a + // BroadcastQueryStageExec (via AQE stageCache, possibly wrapped in + // ReusedExchangeExec). Reference equality (eq) on the join's BQS does not + // hold because convertSAB wraps a fresh exchange in a new ASPE; the actual + // reuse manifests as ReusedExchangeExec inside the ASPE's final plan. + assertCsbBroadcastReuse(subqueries, cometPlan) + + // Verify partition pruning. Iceberg packs these small files into one Spark partition + // whether or not DPP prunes, so count the planned file tasks and the splits read: only + // the file for 1970-01-02 of the 3 dates should be left. + val icebergScans = collectIcebergNativeScans(cometPlan) + assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + val scan = icebergScans.head + val plannedTasks = fileScanTaskCount(scan) + val numSplits = scan.metrics("num_splits").value + assert( + plannedTasks == 1 && numSplits == 1, + s"Expected DPP to prune to 1 of 3 files, planned $plannedTasks tasks and read " + + s"$numSplits splits:\n${cometPlan.treeString}") + spark.sql("DROP TABLE aqe_cat.db.dpp_reuse_fact") } } @@ -6213,22 +6206,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Both DPP filters should use CometSubqueryBroadcastExec - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - // Both should reuse the dim broadcast. Each subquery child is an ASPE that - // contains a BroadcastQueryStageExec (or ReusedExchangeExec) - AQE stageCache - // dedupes via canonical form rather than Java reference identity. - assertCsbBroadcastReuse(subqueries, cometPlan) + // Both DPP filters should use CometSubqueryBroadcastExec + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.size == 2, s"Expected 2 DPP subqueries but got ${subqueries.size}") + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } + // Both should reuse the dim broadcast. Each subquery child is an ASPE that + // contains a BroadcastQueryStageExec (or ReusedExchangeExec) - AQE stageCache + // dedupes via canonical form rather than Java reference identity. + assertCsbBroadcastReuse(subqueries, cometPlan) + spark.sql("DROP TABLE aqe_cat.db.multi_dpp_reuse") } } @@ -6293,41 +6284,39 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Should have DPP subqueries for both joins - val subqueries = collectIcebergDPPSubqueries(cometPlan) - assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") + // Should have DPP subqueries for both joins + val subqueries = collectIcebergDPPSubqueries(cometPlan) + assert(subqueries.nonEmpty, s"Expected DPP subqueries in plan:\n$cometPlan") - // Each should be CometSubqueryBroadcastExec with an ASPE child wrapping the - // build-side broadcast (reuse manifests as ReusedExchangeExec inside the ASPE). - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } - - val subqueryCsbs = subqueries.collect { case csb: CometSubqueryBroadcastExec => csb } - assertCsbBroadcastReuse(subqueryCsbs, cometPlan) + // Each should be CometSubqueryBroadcastExec with an ASPE child wrapping the + // build-side broadcast (reuse manifests as ReusedExchangeExec inside the ASPE). + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") + } - // buildKeys disambiguation: the two DPP subqueries should not share canonical - // form (different join keys -> different broadcasts). Compare canonicalized - // plans rather than Java reference identity. - if (subqueryCsbs.size >= 2) { - val distinctCanonical = subqueryCsbs.map(_.child.canonicalized).distinct - assert( - distinctCanonical.size == subqueryCsbs.size, - s"Expected ${subqueryCsbs.size} distinct broadcasts (by canonical form) " + - s"but got ${distinctCanonical.size}. buildKeys disambiguation may not be working.") - } + val subqueryCsbs = subqueries.collect { case csb: CometSubqueryBroadcastExec => csb } + assertCsbBroadcastReuse(subqueryCsbs, cometPlan) - // Verify correct results: date=2024-01-02 AND category=A returns row (3, 2024-01-02, A, 30) - val icebergScans = collectIcebergNativeScans(cometPlan) - assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + // buildKeys disambiguation: the two DPP subqueries should not share canonical + // form (different join keys -> different broadcasts). Compare canonicalized + // plans rather than Java reference identity. + if (subqueryCsbs.size >= 2) { + val distinctCanonical = subqueryCsbs.map(_.child.canonicalized).distinct assert( - icebergScans.head.numPartitions == 1, - s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + distinctCanonical.size == subqueryCsbs.size, + s"Expected ${subqueryCsbs.size} distinct broadcasts (by canonical form) " + + s"but got ${distinctCanonical.size}. buildKeys disambiguation may not be working.") } + // Verify correct results: date=2024-01-02 AND category=A returns row (3, 2024-01-02, A, 30) + val icebergScans = collectIcebergNativeScans(cometPlan) + assert(icebergScans.nonEmpty, "Expected CometIcebergNativeScanExec in plan") + assert( + icebergScans.head.numPartitions == 1, + s"Expected DPP to prune to 1 partition but got ${icebergScans.head.numPartitions}") + spark.sql("DROP TABLE aqe_cat.db.two_join_fact") } } @@ -6410,8 +6399,7 @@ class CometIcebergNativeSuite .writeTo(factTable) .append() - // The regression requires one original split: Spark 3.4 can then omit the - // exchange between the partial and final global aggregates. + // The regression starts with one original split before DPP removes its file task. val unprunedPlan = spark.table(factTable).queryExecution.executedPlan assertSingleNativeScan(unprunedPlan) val unprunedScan = collectIcebergNativeScans(unprunedPlan).head @@ -6442,12 +6430,6 @@ class CometIcebergNativeSuite val plan = df.queryExecution.executedPlan assertSingleNativeScan(plan) val scan = collectIcebergNativeScans(plan).head - if (!isSpark35Plus) { - assert( - collect(plan) { case exchange: ShuffleExchangeLike => exchange }.isEmpty, - s"Expected the single-split aggregate to run without a shuffle:\n$plan") - assert(scan.numPartitions == 1, s"Expected one empty execution partition:\n$plan") - } assert( collectIcebergDPPSubqueries(plan).nonEmpty, s"Expected executable DPP on the native scan:\n$plan") @@ -6545,14 +6527,12 @@ class CometIcebergNativeSuite val plan = df.queryExecution.executedPlan val scans = collectIcebergNativeScans(plan) - if (isSpark35Plus && aqeEnabled && scans.isEmpty) { - // Newer Spark versions shuffle the join inputs, allowing AQE to replace - // the join and its scans with an empty relation after DPP prunes one input. + if (aqeEnabled && scans.isEmpty) { + // AQE can replace the join and its scans with an empty relation after DPP prunes + // one input. // Spark 4.0+ (SPARK-48466) plans it as EmptyRelationExec, which Comet // converts to CometEmptyRelationExec. Spark 3.5 has no EmptyRelation node // and plans an empty LocalTableScanExec instead. - // This checks result compatibility; the SinglePartition regression assertions - // below target Spark 3.4, where both scans must remain in one native block. val emptyRelations = if (isSpark40Plus) { collect(plan) { case empty: CometEmptyRelationExec => empty } } else { @@ -6579,27 +6559,6 @@ class CometIcebergNativeSuite scans.filterNot(_ eq prunedScan).map(fileTaskCount) == Seq(1), s"Expected the other scan to retain its file task:\n$plan") } - - if (!isSpark35Plus) { - val join = collectFirst(plan) { case j: CometSortMergeJoinExec => j } - .getOrElse(fail(s"Expected a native sort-merge join:\n$plan")) - assert( - collect(join) { case exchange: ShuffleExchangeLike => exchange }.isEmpty, - s"Expected no shuffle between the join and its scans:\n$plan") - val inputs = scala.collection.mutable.ArrayBuffer.empty[SparkPlan] - join.foreachUntilCometInput(join)(input => inputs += input) - val nativeScans = inputs.collect { case scan: CometIcebergNativeScanExec => scan } - assert(nativeScans.size == 2, s"Expected both scans in one native block:\n$plan") - assert( - nativeScans.forall(_.numPartitions == 1), - s"Expected both scans to retain one execution partition:\n$plan") - // Check execution input order, not just SQL aliases: pruning either input used - // to fail differently when its partition array was empty but the other was not. - val expectedTaskCounts = if (prunedSide == "a") Seq(0, 1) else Seq(1, 0) - assert( - nativeScans.map(fileTaskCount).toSeq == expectedTaskCounts, - s"Unexpected file tasks in native input order:\n$plan") - } } } finally { tables.foreach(table => spark.sql(s"DROP TABLE IF EXISTS $table")) @@ -6709,14 +6668,12 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Verify the rule still converted the SAB - val subqueries = collectIcebergDPPSubqueries(cometPlan) - subqueries.foreach { sub => - assert( - sub.isInstanceOf[CometSubqueryBroadcastExec], - s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") - } + // Verify the rule still converted the SAB + val subqueries = collectIcebergDPPSubqueries(cometPlan) + subqueries.foreach { sub => + assert( + sub.isInstanceOf[CometSubqueryBroadcastExec], + s"Expected CometSubqueryBroadcastExec but got ${sub.getClass.getSimpleName}") } spark.sql("DROP TABLE aqe_cat.db.empty_dpp_fact") @@ -6768,9 +6725,8 @@ class CometIcebergNativeSuite // Should produce correct results regardless of DPP path val (_, cometPlan) = checkSparkAnswer(query) assertIcebergNativeScanPresent(cometPlan) - // Even when no BHJ exists, no CSAB should survive the rule. On 3.5+ the rule - // emits TrueLiteral or aggregate SubqueryExec; on 3.4 the wrapper is never - // created in the first place. A leftover CSAB would crash at runtime. + // Even when no BHJ exists, no CSAB should survive the rule. It emits TrueLiteral or an + // aggregate SubqueryExec; a leftover CSAB would crash at runtime. assertNoLeftoverCSAB(cometPlan) spark.sql("DROP TABLE aqe_cat.db.smj_fact") @@ -6844,27 +6800,25 @@ class CometIcebergNativeSuite val (_, cometPlan) = checkSparkAnswer(query) assertIcebergNativeScanPresent(cometPlan) - // No CSAB should survive on either version (would crash at execution otherwise). + // No CSAB should survive (it would crash at execution otherwise). assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Subquery dedup: exactly 1 CometSubqueryBroadcastExec shared across both - // fact scans, plus at least 1 ReusedSubqueryExec pointer. - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.size == 1, - "Expected exactly 1 CometSubqueryBroadcastExec (shared), got " + - s"${cometSubqueries.size}:\n${cometPlan.treeString}") - val reusedCsbs = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r - } - assert( - reusedCsbs.nonEmpty, - "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec):" + - s"\n${cometPlan.treeString}") + // Subquery dedup: exactly 1 CometSubqueryBroadcastExec shared across both + // fact scans, plus at least 1 ReusedSubqueryExec pointer. + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + assert( + cometSubqueries.size == 1, + "Expected exactly 1 CometSubqueryBroadcastExec (shared), got " + + s"${cometSubqueries.size}:\n${cometPlan.treeString}") + val reusedCsbs = collectWithSubqueries(cometPlan) { + case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r } + assert( + reusedCsbs.nonEmpty, + "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec):" + + s"\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.fact1") spark.sql("DROP TABLE aqe_cat.db.fact2") @@ -6933,33 +6887,30 @@ class CometIcebergNativeSuite assertIcebergNativeScanPresent(cometPlan) assertNoLeftoverCSAB(cometPlan) - // On 3.4, Spark's own DPP rule plans the subquery before coalescing, so check results only. - if (isSpark35Plus) { - val dppScans = collectIcebergNativeScans(cometPlan).filter( - _.runtimeFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert(dppScans.size == 1, s"Expected one DPP Iceberg scan:\n${cometPlan.treeString}") - val scan = dppScans.head - // The test only covers the bug if coalescing changed the stage holding the DPP scan. - // That is the final stage, and TreeNode.collect stops at its query stage leaves. - val finalStage = stripAQEPlan(cometPlan) - assert( - finalStage.find(_ eq scan).nonEmpty && - finalStage.collect { - case r: AQEShuffleReadExec if r.isCoalescedRead => r - }.nonEmpty, - "Expected the DPP scan and a coalesced shuffle read in the final stage:\n" + - cometPlan.treeString) - assertCsbBroadcastReuse(collectIcebergDPPSubqueries(cometPlan), cometPlan) - // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 data files. - // Iceberg filters its already planned tasks at runtime, so count the serialized tasks - // and the splits read rather than Iceberg's planning metrics. - val plannedTasks = fileScanTaskCount(scan) - val numSplits = scan.metrics("num_splits").value - assert( - plannedTasks == 1 && numSplits == 1, - s"Expected DPP to prune to 1 of 10 files, planned $plannedTasks tasks and read " + - s"$numSplits splits:\n${cometPlan.treeString}") - } + val dppScans = collectIcebergNativeScans(cometPlan).filter( + _.runtimeFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert(dppScans.size == 1, s"Expected one DPP Iceberg scan:\n${cometPlan.treeString}") + val scan = dppScans.head + // The test only covers the bug if coalescing changed the stage holding the DPP scan. + // That is the final stage, and TreeNode.collect stops at its query stage leaves. + val finalStage = stripAQEPlan(cometPlan) + assert( + finalStage.find(_ eq scan).nonEmpty && + finalStage.collect { + case r: AQEShuffleReadExec if r.isCoalescedRead => r + }.nonEmpty, + "Expected the DPP scan and a coalesced shuffle read in the final stage:\n" + + cometPlan.treeString) + assertCsbBroadcastReuse(collectIcebergDPPSubqueries(cometPlan), cometPlan) + // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 data files. + // Iceberg filters its already planned tasks at runtime, so count the serialized tasks + // and the splits read rather than Iceberg's planning metrics. + val plannedTasks = fileScanTaskCount(scan) + val numSplits = scan.metrics("num_splits").value + assert( + plannedTasks == 1 && numSplits == 1, + s"Expected DPP to prune to 1 of 10 files, planned $plannedTasks tasks and read " + + s"$numSplits splits:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.coalesce_fact") } @@ -7031,25 +6982,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - // Cross-plan dedup is a known 3.4 limitation (each ASPE sees only its own - // plan at prep-rule time, so the scalar subquery's SAB cannot find the main - // query's BHJ). Strict shape checks only on 3.5+. - if (isSpark35Plus) { - val countBroadcasts = collectWithSubqueries(cometPlan) { - case _: CometSubqueryBroadcastExec => 1 - }.sum - val countReused = collectWithSubqueries(cometPlan) { - case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 - }.sum + val countBroadcasts = collectWithSubqueries(cometPlan) { + case _: CometSubqueryBroadcastExec => 1 + }.sum + val countReused = collectWithSubqueries(cometPlan) { + case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 + }.sum - assert( - countBroadcasts == 1, - "Expected 1 CometSubqueryBroadcastExec (shared across plans), " + - s"got $countBroadcasts:\n${cometPlan.treeString}") - assert( - countReused == 1, - s"Expected 1 ReusedSubqueryExec, got $countReused:\n${cometPlan.treeString}") - } + assert( + countBroadcasts == 1, + "Expected 1 CometSubqueryBroadcastExec (shared across plans), " + + s"got $countBroadcasts:\n${cometPlan.treeString}") + assert( + countReused == 1, + s"Expected 1 ReusedSubqueryExec, got $countReused:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.scalar_fact") } @@ -7172,17 +7118,15 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - // Two DPP filters from the same join must not duplicate the dim broadcast. - // Count broadcasts across the whole plan including subquery contexts. - val cometBroadcasts = collectWithSubqueries(cometPlan) { - case b: CometBroadcastExchangeExec => b - } - assert( - cometBroadcasts.size == 1, - "Expected exactly 1 CometBroadcastExchangeExec across whole plan, " + - s"got ${cometBroadcasts.size}:\n${cometPlan.treeString}") + // Two DPP filters from the same join must not duplicate the dim broadcast. + // Count broadcasts across the whole plan including subquery contexts. + val cometBroadcasts = collectWithSubqueries(cometPlan) { + case b: CometBroadcastExchangeExec => b } + assert( + cometBroadcasts.size == 1, + "Expected exactly 1 CometBroadcastExchangeExec across whole plan, " + + s"got ${cometBroadcasts.size}:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.exchg_fact") } @@ -7229,14 +7173,12 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected CometSubqueryBroadcastExec for DPP, got 0:\n${cometPlan.treeString}") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected CometSubqueryBroadcastExec for DPP, got 0:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.q34637_fact") } @@ -7251,14 +7193,9 @@ class CometIcebergNativeSuite // DynamicPruningExpression(TrueLiteral) - analogous to FileSourceScanExec.doCanonicalize // on V1. // - // The "exactly once" property manifests differently across Spark versions: - // - 3.5+: EnsureRequirements inserts a hash shuffle for SMJ; AQE recognizes the matching - // canonical form on the peer side and emits ReusedExchangeExec -> 1 shuffle, 1 reuse. - // Same shape as V1's CometExecSuite version of this test. - // - 3.4: planner doesn't insert shuffles for this query shape (V2-specific outputPartitioning - // interaction with EnsureRequirements). Result: 0 shuffles. Still "data read once" - just - // via a different mechanism. - // Either shape satisfies the invariant; assertion accepts both. + // EnsureRequirements inserts a hash shuffle for SMJ; AQE recognizes the matching canonical + // form on the peer side and emits ReusedExchangeExec -> 1 shuffle, 1 reuse. This is the same + // shape as V1's CometExecSuite version of this test. test("AQE DPP - unused DPP filter and exchange reuse (SPARK-32509)") { assume(icebergAvailable, "Iceberg not available") withTempIcebergDir { warehouseDir => @@ -7303,8 +7240,7 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - // Accept either shape (see test docstring): single shuffle with reuse, or no - // shuffles at all. Both prove the two scans canonicalize identically after DPP + // A single shuffle plus one reuse proves the two scans canonicalize identically after DPP // degrades to TrueLiteral, so fact data is read exactly once. val shuffleExchanges = collect(cometPlan) { case e: ShuffleExchangeExec => e @@ -7312,13 +7248,10 @@ class CometIcebergNativeSuite } val reusedExchanges = collect(cometPlan) { case r: ReusedExchangeExec => r } - val singleShuffleWithReuse = - shuffleExchanges.size == 1 && reusedExchanges.size == 1 - val noShuffles = shuffleExchanges.isEmpty && reusedExchanges.isEmpty assert( - singleShuffleWithReuse || noShuffles, - "Expected fact data read exactly once: either (1 shuffle + 1 ReusedExchange) " + - s"or (0 shuffles), got (${shuffleExchanges.size} shuffles, " + + shuffleExchanges.size == 1 && reusedExchanges.size == 1, + "Expected fact data read exactly once with 1 shuffle + 1 ReusedExchange, " + + s"got (${shuffleExchanges.size} shuffles, " + s"${reusedExchanges.size} reused):\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.q32509_fact") @@ -7392,8 +7325,6 @@ class CometIcebergNativeSuite assertIcebergNativeScanPresent(cometPlan) assertNoLeftoverCSAB(cometPlan) - // Also checked on 3.4: the Iceberg scan stays in Comet there and Spark's own DPP rule - // plans its filter. val dppScan: PartialFunction[SparkPlan, CometIcebergNativeScanExec] = { case s: CometIcebergNativeScanExec if s.runtimeFilters.nonEmpty => s } @@ -7466,22 +7397,20 @@ class CometIcebergNativeSuite assertNoLeftoverCSAB(cometPlan) - if (isSpark35Plus) { - val dpExprs = collect(cometPlan) { case s: CometIcebergNativeScanExec => s } - .flatMap(_.runtimeFilters.collect { case d: DynamicPruningExpression => - d.child - }) - val hasSubquery = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true - case _ => false - } - val hasBroadcast = dpExprs.exists { - case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true - case _ => false - } - assert(!hasSubquery, s"Should not have SubqueryExec DPP:\n${cometPlan.treeString}") - assert(hasBroadcast, s"Should have broadcast DPP:\n${cometPlan.treeString}") + val dpExprs = collect(cometPlan) { case s: CometIcebergNativeScanExec => s } + .flatMap(_.runtimeFilters.collect { case d: DynamicPruningExpression => + d.child + }) + val hasSubquery = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true + case _ => false + } + val hasBroadcast = dpExprs.exists { + case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true + case _ => false } + assert(!hasSubquery, s"Should not have SubqueryExec DPP:\n${cometPlan.treeString}") + assert(hasBroadcast, s"Should have broadcast DPP:\n${cometPlan.treeString}") spark.sql("DROP TABLE aqe_cat.db.large") } diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala index bb554ca095e..94bba27171d 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergRewriteActionSuite.scala @@ -32,7 +32,6 @@ import org.apache.spark.sql.{CometTestBase, SparkSession} import org.apache.spark.sql.execution.SparkPlanInfo import org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.iceberg.IcebergReflection /** @@ -244,7 +243,6 @@ class CometIcebergRewriteActionSuite extends CometTestBase with CometIcebergTest // native. Older Iceberg writes no lineage columns, so its rewrite stays native. test("binPack rewrite of a format-version=3 table keeps row lineage") { assume(icebergAvailable, "Iceberg not available in classpath") - assume(isSpark35Plus, "V3 tables require Iceberg 1.8.1+ (Spark 3.5 profile)") val writesLineage = icebergVersionAtLeast(1, 10) withTempIcebergDir { warehouseDir => diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala index f6b8d70669d..e03ef92b3f2 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala @@ -58,7 +58,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.streaming.Trigger import org.apache.spark.sql.types.{DoubleType, IntegerType, StringType, StructField, StructType} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark41Plus, isSpark42Plus} import org.apache.comet.iceberg.{IcebergReflection, IcebergWriteReportListener, PositionDeltaWrite} import org.apache.comet.serde.Unsupported import org.apache.comet.serde.operator.CometIcebergNativeWrite @@ -272,7 +272,6 @@ class CometIcebergWriteActionSuite // https://github.com/apache/datafusion-comet/issues/6240 test("a merge-on-read write keeps Spark's own write plan when only the write flag is set") { assume(icebergAvailable, "Iceberg not available in classpath") - assume(isSpark35Plus, "WriteDelta interception starts with Spark 3.5") withIcebergCatalog { _ => createTable( "mor_defaults", @@ -977,7 +976,6 @@ class CometIcebergWriteActionSuite Seq("DELETE", "UPDATE", "MERGE").foreach { command => test(s"WriteDelta $command is recognized and stays on Iceberg JVM DeltaWriter") { assume(icebergAvailable, "Iceberg not available in classpath") - assume(isSpark35Plus, "WriteDelta interception starts with Spark 3.5") withIcebergCatalog { _ => val suffix = command.toLowerCase(java.util.Locale.ROOT) val cometTable = s"write_delta_${suffix}_comet" @@ -1554,7 +1552,6 @@ class CometIcebergWriteActionSuite // the new rows their ids at commit time, so those writes stay native on a v3 table. test("native acceleration: format-version=3 appends and overwrites match iceberg-java") { assumeNativeAcceleration() - assume(isSpark35Plus, "V3 tables require Iceberg 1.8.1+ (Spark 3.5 profile)") withIcebergCatalog { warehouseDir => Seq("v3_native", "v3_jvm").foreach { t => createTable( @@ -1890,31 +1887,27 @@ class CometIcebergWriteActionSuite } withSQLConf(CometConf.COMET_EXEC_MERGE_ROWS_ENABLED.key -> "true") { - if (!isSpark35Plus) { - assertNativeWriteDoesNotEngage("native_cow_merge", Seq(1, 2, 3))(runMerge()) - } else { - val snapshot = withNativeEnabled { - captureWrite("native_cow_merge")(runMerge()) - } - assert( - snapshot.snapshotDelta == 1L, - s"expected exactly one Iceberg snapshot from native MERGE, got ${snapshot.snapshotDelta}") - val mergeExecs = snapshot.plans.flatMap { plan => - collectWithSubqueries(plan) { case e: CometMergeRowsExec => e } - } - assert( - mergeExecs.nonEmpty, - "expected Iceberg MERGE to execute through CometMergeRowsExec. Plans:\n" + - snapshot.plans.mkString("\n--\n")) - val nativeWrites = snapshot.plans.flatMap { plan => - collectWithSubqueries(plan) { case e: CometIcebergWriteExec => e } - } - assert( - nativeWrites.nonEmpty, - "expected Iceberg MERGE to feed CometIcebergWriteExec. Plans:\n" + - snapshot.plans.mkString("\n--\n")) - assertRows("native_cow_merge", expectedIds = Seq(1, 2, 3)) + val snapshot = withNativeEnabled { + captureWrite("native_cow_merge")(runMerge()) + } + assert( + snapshot.snapshotDelta == 1L, + s"expected exactly one Iceberg snapshot from native MERGE, got ${snapshot.snapshotDelta}") + val mergeExecs = snapshot.plans.flatMap { plan => + collectWithSubqueries(plan) { case e: CometMergeRowsExec => e } + } + assert( + mergeExecs.nonEmpty, + "expected Iceberg MERGE to execute through CometMergeRowsExec. Plans:\n" + + snapshot.plans.mkString("\n--\n")) + val nativeWrites = snapshot.plans.flatMap { plan => + collectWithSubqueries(plan) { case e: CometIcebergWriteExec => e } } + assert( + nativeWrites.nonEmpty, + "expected Iceberg MERGE to feed CometIcebergWriteExec. Plans:\n" + + snapshot.plans.mkString("\n--\n")) + assertRows("native_cow_merge", expectedIds = Seq(1, 2, 3)) } val updated = spark @@ -1926,7 +1919,6 @@ class CometIcebergWriteActionSuite test("MERGE matches Spark on partitioned Iceberg copy-on-write and merge-on-read") { assumeNativeAcceleration() - assume(isSpark35Plus, "native MergeRows requires Spark 3.5+") withIcebergCatalog { _ => spark .range(0, 20000, 1, 8) @@ -2083,7 +2075,6 @@ class CometIcebergWriteActionSuite test("native MergeRows Iceberg cardinality violation matches Spark") { assumeNativeAcceleration() - assume(isSpark35Plus, "native MergeRows requires Spark 3.5+") withIcebergCatalog { _ => val nativeTable = "merge_cardinality_native" val sparkTable = "merge_cardinality_spark" @@ -2449,7 +2440,6 @@ class CometIcebergWriteActionSuite test("native acceleration: CTAS runs its inner append through the native writer") { assumeNativeAcceleration() - assume(isSpark35Plus, "CTAS re-plans its inner append only on Spark 3.5+") withIcebergCatalog { _ => // A brand-new table has no metadata file yet, so this also pins the empty // metadata-location path in the proto builder. @@ -2472,7 +2462,6 @@ class CometIcebergWriteActionSuite test("native acceleration: RTAS replaces table contents through the native writer") { assumeNativeAcceleration() - assume(isSpark35Plus, "RTAS re-plans its inner append only on Spark 3.5+") withIcebergCatalog { _ => createTable("rtas_native", partitionSpec = "") coalesceInsert("rtas_native", Seq((1, "old", 1.0))) @@ -2500,13 +2489,7 @@ class CometIcebergWriteActionSuite // (apache/datafusion-comet#5694) -- and the second as uppercase hex. Comet renders the path // itself; this pins the result against the layout iceberg-java's own writer produces. // - // The byte-for-byte comparison against the JVM writer holds on Iceberg 1.8+ only. Iceberg 1.5.x - // (the Spark 3.4 profile) rendered a `timestamptz` partition value with - // `ChronoUnit.MICROS.addTo(EPOCH, micros).toString()`, i.e. `OffsetDateTime.toString()`, which - // spells the same instant `1969-12-31T23:59:58.500Z` rather than `1969-12-31T23:59:58.5+00:00`; - // 1.8 switched it to `microsToIsoTimestamptz` (and started escaping the field name too). Comet - // targets the 1.8+ spelling on every profile. The values Comet writes are the same either way, - // so the pinned expectations and the readback below stay unconditional. + // The byte-for-byte comparison targets Iceberg 1.8+'s `microsToIsoTimestamptz` spelling. test("native acceleration: timestamptz and binary partition paths match iceberg-java") { assumeNativeAcceleration() withIcebergCatalog { warehouseDir => @@ -4520,8 +4503,6 @@ class CometIcebergWriteActionSuite test("CoW MERGE with a NOT MATCHED BY SOURCE leg routes through two-op") { assume(icebergAvailable, "Iceberg not available in classpath") - // Iceberg 1.5.x (the Spark 3.4 pairing) rejects the clause in its extensions parser. - assume(isSpark35Plus, "NOT MATCHED BY SOURCE needs Iceberg 1.8+") withIcebergCatalog { _ => createTable( "merge_nmbs", @@ -4591,10 +4572,9 @@ class CometIcebergWriteActionSuite } } - // On Spark 3.5+ the staged CTAS/RTAS operators run their inner append as its own - // `AppendData` QueryExecution, which IcebergWriteStrategy intercepts like any other append. - // Spark 3.4 writes inline inside the exec (no re-planning), so nothing is intercepted there. - test("CTAS and RTAS write through the split operators on Spark 3.5+") { + // The staged CTAS/RTAS operators run their inner append as its own `AppendData` QueryExecution, + // which IcebergWriteStrategy intercepts like any other append. + test("CTAS and RTAS write through the split operators") { assume(icebergAvailable, "Iceberg not available in classpath") withIcebergCatalog { _ => val session = spark @@ -4606,15 +4586,9 @@ class CometIcebergWriteActionSuite def assertSplitUsage(plans: Seq[SparkPlan], statement: String): Unit = { val (commits, writes) = collectIcebergWriteOps(plans) - if (isSpark35Plus) { - assert( - commits.nonEmpty && writes.nonEmpty, - s"expected the $statement inner append to plan through the split operators: $plans") - } else { - assert( - commits.isEmpty && writes.isEmpty, - s"expected the $statement write to stay inside Spark's staged exec: $plans") - } + assert( + commits.nonEmpty && writes.nonEmpty, + s"expected the $statement inner append to plan through the split operators: $plans") } val ctasPlans = capturePlans(spark) { @@ -5317,14 +5291,12 @@ class CometIcebergWriteActionSuite "report_orc", partitionSpec = "", properties = Some("'write.format.default'='orc'")) - if (isSpark35Plus) { - createTable( - "report_delta", - partitionSpec = "PARTITIONED BY (region)", - properties = Some("'format-version'='2', 'write.delete.mode'='merge-on-read'")) - withSQLConf(CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> "false") { - coalesceInsert("report_delta", Seq((1, "us-east", 1.5), (2, "us-east", 2.5))) - } + createTable( + "report_delta", + partitionSpec = "PARTITIONED BY (region)", + properties = Some("'format-version'='2', 'write.delete.mode'='merge-on-read'")) + withSQLConf(CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> "false") { + coalesceInsert("report_delta", Seq((1, "us-east", 1.5), (2, "us-east", 2.5))) } val writes = reportedWrites { @@ -5335,16 +5307,13 @@ class CometIcebergWriteActionSuite .sql(s"INSERT INTO $catalog.$ns.report_parquet VALUES (1, 'us-east', 1.5)") .collect() spark.sql(s"INSERT INTO $catalog.$ns.report_orc VALUES (2, 'eu', 2.5)") - if (isSpark35Plus) { - spark.sql(s"DELETE FROM $catalog.$ns.report_delta WHERE id = 1") - } + spark.sql(s"DELETE FROM $catalog.$ns.report_delta WHERE id = 1") } withSQLConf(CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> "false") { spark.sql(s"INSERT INTO $catalog.$ns.report_parquet VALUES (3, 'eu', 3.5)") } } - val expectedWriters = - Seq("native", "jvm") ++ (if (isSpark35Plus) Seq("jvm") else Nil) ++ Seq("spark") + val expectedWriters = Seq("native", "jvm", "jvm", "spark") assert(writes.map(_.writer) == expectedWriters, writes.mkString("\n")) assert(writes(1).reasons.exists(_.contains("only parquet"))) assert(writes.last.node == "AppendData") @@ -5367,11 +5336,7 @@ class CometIcebergWriteActionSuite """) } } - // Spark 3.5+ runs each write as a nested append or overwrite, which Comet's split operator - // plans. Spark 3.4 writes from the create and replace execs, which it never sees. - val expected = - if (isSpark35Plus) Seq("native" -> "CometIcebergWrite", "native" -> "CometIcebergWrite") - else Seq("spark" -> "AtomicCreateTableAsSelect", "spark" -> "AtomicReplaceTableAsSelect") + val expected = Seq("native" -> "CometIcebergWrite", "native" -> "CometIcebergWrite") assert(writes.map(w => w.writer -> w.node) == expected, writes.mkString("\n")) assert(writes.forall(!_.failed)) assertRows("report_ctas", Seq(3)) @@ -5517,7 +5482,7 @@ class CometIcebergWriteActionSuite /** * Runs an [[IcebergCommitExec]] over `child` whose commit throws `commitFailure` and whose * abort fails. Asserts the failure is wrapped the way Spark's `V2TableWriteExec.writeWithV2` - * (3.4 through 4.2) wraps it, in `QueryExecutionErrors.writingJobFailedError` with the abort + * (3.5 through 4.2) wraps it, in `QueryExecutionErrors.writingJobFailedError` with the abort * failure suppressed on the original failure, and returns that original failure. */ private def failWithFailingAbort(child: SparkPlan, commitFailure: Throwable): Throwable = { diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala b/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala index 89e1be46891..0ab57a13bcb 100644 --- a/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala @@ -42,7 +42,7 @@ import org.apache.spark.sql.execution.{ApplyColumnarRulesAndInsertTransitions, C import org.apache.spark.sql.types.{BinaryType, IntegerType} import org.apache.spark.sql.vectorized.ColumnarBatch -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.iceberg.IcebergReflection import org.apache.comet.rules.EliminateRedundantTransitions import org.apache.comet.serde.{Compatible, SupportLevel, Unsupported} @@ -115,7 +115,7 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes withDetectionCatalog { _ => createTable("plan_only", partitionSpec = "") // IcebergWriteStrategy runs before CometRule, so it needs its own plan-only guard. - // withSQLConf returns Unit on Spark 3.4/3.5, hence the var. + // withSQLConf returns Unit on Spark 3.5, hence the var. var plan: SparkPlan = null withSQLConf(CometConf.COMET_EXPLAIN_PLAN_ONLY_ENABLED.key -> "true") { plan = captureWritePlan("plan_only", allowWriteFailure = false) { @@ -247,7 +247,6 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes } test("Compatible for a format-version=3 append") { - assume(isSpark35Plus, "V3 tables require Iceberg 1.8.1+ (Spark 3.5 profile)") withDetectionCatalog { _ => createTable("v3", partitionSpec = "", properties = Some("'format-version'='3'")) assertSupportLevelIs[Compatible]("v3") @@ -390,7 +389,7 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes createTable("brotli_session", partitionSpec = "") val tableWrite = planInsertWriteExec(s"$catalog.$ns.brotli_table") // The session setting reaches the gate through SparkWrite.writeProperties. withSQLConf - // returns Unit on Spark 3.4/3.5, hence the var. + // returns Unit on Spark 3.5, hence the var. var sessionWrite: IcebergWriteExec = null withSQLConf("spark.sql.iceberg.compression-codec" -> "brotli") { sessionWrite = planInsertWriteExec(s"$catalog.$ns.brotli_session") @@ -2050,7 +2049,7 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes values: String = "(1, 'us', 1.0)"): IcebergWriteExec = { val plan = spark.sessionState.sqlParser.parsePlan(s"INSERT INTO $qualifiedTable VALUES $values") - // withSQLConf returns Unit on Spark 3.4/3.5, hence the var. + // withSQLConf returns Unit on Spark 3.5, hence the var. var executed: SparkPlan = null withSplitWrite { executed = spark.sessionState.executePlan(plan, CommandExecutionMode.SKIP).executedPlan diff --git a/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala index fdad9e3d3df..a8c145d1036 100644 --- a/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometMathExpressionSuite.scala @@ -26,7 +26,6 @@ import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{DataTypes, StructField, StructType} -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator} class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper { @@ -95,7 +94,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame( @@ -108,7 +106,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - edge cases") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame(Seq( @@ -125,7 +122,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - NaN values") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .createDataFrame( @@ -137,7 +133,6 @@ class CometMathExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("width_bucket - with range data") { - assume(isSpark35Plus, "width_bucket was added in Spark 3.5") withSQLConf("spark.comet.exec.localTableScan.enabled" -> "true") { spark .range(10) diff --git a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala index 5aa1fad003f..894209204ac 100644 --- a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala @@ -3489,7 +3489,7 @@ class CometNativeCastSuite sparkMessage.startsWith( cometMessage.substring(0, math.min(40, cometMessage.length)))) } else { - // for Spark 3.4 we expect to reproduce the error message exactly + // For Spark 3.x we expect to reproduce the error message exactly. assert(cometMessage == sparkMessage) } } diff --git a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala index c9e97bbfddd..72cbee24ac8 100644 --- a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala @@ -1056,7 +1056,7 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH val ntzSchema = StructType(Seq(StructField("ts", DataTypes.TimestampNTZType, true))) val ntzDF = FuzzDataGenerator.generateDataFrame(r, spark, ntzSchema, 1000, DataGenOptions()) - // Spark 3.4: unix_micros() does not accept TIMESTAMP_NTZ; baseline matches micros / hour. + // Baseline matches micros / hour. val _spark = spark import _spark.implicits._ val expectedDF = ntzDF diff --git a/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala index 56029cba282..0a4b164c1a9 100644 --- a/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala @@ -28,11 +28,10 @@ import org.apache.spark.sql.functions.col * Bit-for-bit comparisons of seeded `uuid` against Spark. * * The SQL `uuid(seed)` form only exists in Spark 4.0+, so `uuid_with_seed.sql` carries a - * `MinSparkVersion: 4.0` marker and is skipped on 3.4 and 3.5. Unseeded `uuid()` cannot be - * compared across engines. That left the Rust golden constants in - * `nondetermenistic_funcs/uuid.rs` as the only guard on those profiles, and their provenance is - * not checkable from the repo -- if they were ever regenerated from the Rust side the test would - * become circular. + * `MinSparkVersion: 4.0` marker and is skipped on 3.5. Unseeded `uuid()` cannot be compared + * across engines. That left the Rust golden constants in `nondetermenistic_funcs/uuid.rs` as the + * only guard on those profiles, and their provenance is not checkable from the repo -- if they + * were ever regenerated from the Rust side the test would become circular. * * `Uuid(Some(seed))` is constructible from Scala on every supported version even where the SQL * form is not, so building the expression directly gives all profiles a real cross-engine diff --git a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala index bd8971f59a8..e44978c1d56 100644 --- a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala +++ b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala @@ -105,7 +105,7 @@ class SparkErrorConverterSuite extends AnyFunSuite { .getOrElse(fail("Expected CannotReadFile to be converted to a Spark exception")) // `cannotReadFilesError` IS the FAILED_READ_FILE path. Assert on the version-stable message // ("Encountered error while reading file ...") rather than the `FAILED_READ_FILE` literal, - // which only Spark 4.x prepends to getMessage as the error-class tag (3.4/3.5 do not). + // which only Spark 4.x prepends to getMessage as the error-class tag (3.5 does not). assert(ex.getMessage.contains("Encountered error while reading file")) assert(ex.getMessage.contains("part-0.parquet")) } diff --git a/spark/src/test/scala/org/apache/comet/csv/CsvReadFromS3Suite.scala b/spark/src/test/scala/org/apache/comet/csv/CsvReadFromS3Suite.scala index 0e3b7e75e3a..67135052544 100644 --- a/spark/src/test/scala/org/apache/comet/csv/CsvReadFromS3Suite.scala +++ b/spark/src/test/scala/org/apache/comet/csv/CsvReadFromS3Suite.scala @@ -25,16 +25,13 @@ import java.nio.charset.StandardCharsets import scala.jdk.CollectionConverters._ import org.apache.spark.sql.{DataFrame, Row} -import org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning import org.apache.spark.sql.comet.CometCsvNativeScanExec -import org.apache.spark.sql.execution.SparkPlan import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper -import org.apache.spark.sql.execution.datasources.v2.BatchScanExec import org.apache.spark.sql.functions.{input_file_name, spark_partition_id} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{IntegerType, StringType, StructType} -import org.apache.comet.{CometConf, CometS3TestBase, ExtendedExplainInfo} +import org.apache.comet.{CometConf, CometS3TestBase} /** Native CSV scans over files in two MinIO buckets. Manual: needs Docker for MinIO. */ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { @@ -84,34 +81,14 @@ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { spark.read.option("header", "false").schema(schema).csv(paths: _*) /** Whether Spark plans the scan of `paths` with a known partitioning, as 3.4 does for one. */ - private def isSparkPartitioningKnown(paths: String*): Boolean = { - var isKnown = false - withSQLConf(CometConf.COMET_ENABLED.key -> "false") { - val sparkScans = collect(readCsv(paths: _*).queryExecution.executedPlan) { - case scan: BatchScanExec => scan - } - isKnown = sparkScans.exists(!_.outputPartitioning.isInstanceOf[UnknownPartitioning]) - } - isKnown - } - - private def assertSplitFallback(plan: SparkPlan): Unit = { - val reasons = new ExtendedExplainInfo().getFallbackReasons(plan) - assert( - reasons.exists(_.contains("Native CSV scan would split a partition that mixes object")), - s"fallback reasons: $reasons") - } - /** * Reads `paths` with Comet off and on, and checks that Comet returns Spark's rows through a * native CSV scan whose partitions each read from one bucket. `sparkMixesBuckets` states - * whether Spark's own layout puts both buckets in one partition. Where Spark plans the scan - * with a known partitioning (one partition on Spark 3.4), Comet must leave it to Spark. + * whether Spark's own layout puts both buckets in one partition. */ private def assertMultiBucketRead(paths: Seq[String], sparkMixesBuckets: Boolean): Unit = { var expected: Seq[Row] = Nil var sparkLayout: Seq[Set[String]] = Nil - val isKnownPartitioning = isSparkPartitioningKnown(paths: _*) withSQLConf(CometConf.COMET_ENABLED.key -> "false") { expected = readCsv(paths: _*).collect().toSeq sparkLayout = readCsv(paths: _*) @@ -132,11 +109,6 @@ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { assert(rows.map(_.toString).sorted == expected.map(_.toString).sorted) val plan = df.queryExecution.executedPlan val scans = collect(plan) { case scan: CometCsvNativeScanExec => scan } - if (sparkMixesBuckets && isKnownPartitioning) { - assert(scans.isEmpty, s"expected Spark's CSV scan:\n$plan") - assertSplitFallback(plan) - return - } assert(scans.size == 1, s"expected one native CSV scan:\n$plan") val partitions = scans.head.nativeOp.getCsvScan.getFilePartitionsList.asScala.toSeq val cometLayout = @@ -176,7 +148,6 @@ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { Seq(false, true).foreach { isAdaptive => test(s"global aggregate over a one-partition CSV scan of two buckets, AQE $isAdaptive") { - // Spark 3.4 plans no exchange above a one-partition scan, so Comet must not split it. putTwoBucketFiles("csv-agg", "csv-agg", Seq("f1.csv", "f2.csv")) val paths = Seq(a("csv-agg"), b("csv-agg")) withSQLConf( @@ -191,13 +162,8 @@ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { assert(df.collect().head.getLong(0) == expected) val plan = df.queryExecution.executedPlan val scans = collect(plan) { case scan: CometCsvNativeScanExec => scan } - if (isSparkPartitioningKnown(paths: _*)) { - assert(scans.isEmpty, s"expected Spark's CSV scan:\n$plan") - assertSplitFallback(plan) - } else { - assert(scans.size == 1, s"expected one native CSV scan:\n$plan") - assert(scans.head.outputPartitioning.numPartitions == 2) - } + assert(scans.size == 1, s"expected one native CSV scan:\n$plan") + assert(scans.head.outputPartitioning.numPartitions == 2) } } } @@ -224,15 +190,12 @@ class CsvReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { } test("native CSV scan splits a mixed partition of a scan Spark plans as several partitions") { - // Spark plans more than one partition, so its partitioning is unknown on every version, - // Spark 3.4 included, and Comet may split the mixed one. createBucketIfNotExists(secondBucketName) putObject(testBucketName, "csv-mix-big/f.csv", csvBytes(100, 129, "A")) putObject(secondBucketName, "csv-mix-mid/f.csv", csvBytes(10, 14, "B")) putObject(testBucketName, "csv-mix-small/f.csv", csvBytes(1, 3, "A")) val paths = Seq(a("csv-mix-big"), b("csv-mix-mid"), a("csv-mix-small")) withSQLConf(csvNative ++ twoPartitionsOneMixed: _*) { - assert(!isSparkPartitioningKnown(paths: _*)) assertMultiBucketRead(paths, sparkMixesBuckets = true) } } diff --git a/spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala index f674a1a5e59..da5ed7e6ce6 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala @@ -44,7 +44,7 @@ import org.apache.spark.sql.types.{ArrayType, DataTypes, StructField, StructType import org.apache.comet.CometConf import org.apache.comet.CometConf.COMET_EXEC_STRICT_FLOATING_POINT -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} import org.apache.comet.rules.CometExecRule import org.apache.comet.serde.{CometCorr, CometRegrR2, ExprOuterClass, SparkFixes} import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator, ParquetGenerator, SchemaGenOptions} @@ -2601,17 +2601,7 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper { } withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> "NO_CODEGEN") { val df = sql("SELECT SUM(v) FROM dec_no_codegen") - if (isSpark35Plus) { - assertDecimalSumFallsBackLikeSpark(df, reason, ansiEnabled, Row(null)) - } else { - // Before SPARK-44236 (3.5) the factory mode leaves whole-stage codegen on, so - // Spark recovers the sum and the aggregate stays native. - checkSparkAnswerAndOperator(df) - val answer = df.collect().toSeq - assert( - answer == Seq(Row(recoveredSum.bigDecimal.setScale(38))), - s"NO_CODEGEN on Spark 3.4 returned $answer, expected $recoveredSum") - } + assertDecimalSumFallsBackLikeSpark(df, reason, ansiEnabled, Row(null)) } } } diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala similarity index 98% rename from spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala rename to spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala index 516ece40be0..18eec3f32f1 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometExec3_4PlusSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometExecCompatibilitySuite.scala @@ -36,10 +36,8 @@ import org.apache.spark.util.sketch.BloomFilter import org.apache.comet.CometConf import org.apache.comet.CometSparkSessionExtensions.isSpark42Plus -/** - * This test suite contains tests for only Spark 3.4+. - */ -class CometExec3_4PlusSuite extends CometTestBase { +/** Tests compatibility-sensitive execution paths shared by the supported Spark versions. */ +class CometExecCompatibilitySuite extends CometTestBase { import testImplicits._ val func_might_contain = new FunctionIdentifier("might_contain") @@ -83,7 +81,6 @@ class CometExec3_4PlusSuite extends CometTestBase { } } - // The syntax is only supported by Spark 3.4+. test("subquery limit: limit with offset should return correct results") { withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "jvm") { withTable("t1", "t2") { @@ -135,7 +132,6 @@ class CometExec3_4PlusSuite extends CometTestBase { } } - // Dataset.offset API is not available before Spark 3.4 test("offset") { withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "jvm") { checkSparkAnswer(testData.offset(90)) diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala index 7b30248b833..8e78885e6ce 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala @@ -53,7 +53,7 @@ import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.UTF8String import org.apache.comet.{CometConf, CometExecIterator, CometNativeException, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} import org.apache.comet.rules.CometCoalesceShufflePartitions import org.apache.comet.serde.Config.ConfigMap import org.apache.comet.testing.{DataGenOptions, ParquetGenerator, SchemaGenOptions} @@ -954,8 +954,8 @@ class CometExecSuite extends CometTestBase { } } - // On 3.5+, CometPlanAdaptiveDynamicPruningFilters converts SABs to - // CometSubqueryBroadcastExec with broadcast reuse. On 3.4, AQE DPP falls back to Spark. + // CometPlanAdaptiveDynamicPruningFilters converts SABs to + // CometSubqueryBroadcastExec with broadcast reuse. test("AQE DPP: BHJ works with CometNativeScanExec") { withTempDir { path => val factPath = s"${path.getAbsolutePath}/fact.parquet" @@ -982,61 +982,54 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) val infos = new ExtendedExplainInfo().generateExtendedInfo(cometPlan) - if (isSpark35Plus) { - // Verify native scan with DPP - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert( - dppScans.nonEmpty, - "Expected at least one CometNativeScanExec with DynamicPruningExpression") + // Verify native scan with DPP + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert( + dppScans.nonEmpty, + "Expected at least one CometNativeScanExec with DynamicPruningExpression") - // Verify CometSubqueryBroadcastExec with AdaptiveSparkPlanExec child - // (matches Spark's SubqueryBroadcastExec wrapping an ASPE that goes - // through AQE stageCache for broadcast reuse via ReusedExchangeExec) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } + // Verify CometSubqueryBroadcastExec with AdaptiveSparkPlanExec child + // (matches Spark's SubqueryBroadcastExec wrapping an ASPE that goes + // through AQE stageCache for broadcast reuse via ReusedExchangeExec) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + assert( + cometSubqueries.nonEmpty, + "Expected CometSubqueryBroadcastExec for broadcast reuse") + cometSubqueries.foreach { csb => assert( - cometSubqueries.nonEmpty, - "Expected CometSubqueryBroadcastExec for broadcast reuse") - cometSubqueries.foreach { csb => - assert( - csb.child.isInstanceOf[AdaptiveSparkPlanExec], - "Expected AdaptiveSparkPlanExec child but got " + - s"${csb.child.getClass.getSimpleName}") - } - - // Verify broadcast reuse: the subquery's ASPE final plan should contain - // ReusedExchangeExec (AQE stageCache matched the join's broadcast) - import org.apache.spark.sql.execution.exchange.ReusedExchangeExec - cometSubqueries.foreach { csb => - val aspe = csb.child.asInstanceOf[AdaptiveSparkPlanExec] - val hasReusedExchange = collect(aspe) { case r: ReusedExchangeExec => - r - }.nonEmpty || collect(aspe) { case b: BroadcastQueryStageExec => - b - }.nonEmpty - assert( - hasReusedExchange, - "DPP subquery's ASPE should contain ReusedExchangeExec or " + - "BroadcastQueryStageExec for broadcast reuse") - } - - // Verify no unconverted SABs remain - assertAqeDppShape(cometPlan) + csb.child.isInstanceOf[AdaptiveSparkPlanExec], + "Expected AdaptiveSparkPlanExec child but got " + + s"${csb.child.getClass.getSimpleName}") + } - // Verify no fallback - assert( - !infos.contains("AQE Dynamic Partition Pruning"), - s"Should not fall back for AQE DPP:\n$infos") - } else { - // 3.4: scan falls back to Spark so Spark handles DPP natively + // Verify broadcast reuse: the subquery's ASPE final plan should contain + // ReusedExchangeExec (AQE stageCache matched the join's broadcast) + import org.apache.spark.sql.execution.exchange.ReusedExchangeExec + cometSubqueries.foreach { csb => + val aspe = csb.child.asInstanceOf[AdaptiveSparkPlanExec] + val hasReusedExchange = collect(aspe) { case r: ReusedExchangeExec => + r + }.nonEmpty || collect(aspe) { case b: BroadcastQueryStageExec => + b + }.nonEmpty assert( - infos.contains("AQE Dynamic Partition Pruning requires Spark 3.5+"), - s"Expected 3.4 AQE DPP fallback message but got:\n$infos") + hasReusedExchange, + "DPP subquery's ASPE should contain ReusedExchangeExec or " + + "BroadcastQueryStageExec for broadcast reuse") } + + // Verify no unconverted SABs remain + assertAqeDppShape(cometPlan) + + // Verify no fallback + assert( + !infos.contains("AQE Dynamic Partition Pruning"), + s"Should not fall back for AQE DPP:\n$infos") } } } @@ -1076,14 +1069,12 @@ class CometExecSuite extends CometTestBase { // Verify no CometSubqueryBroadcastExec — Spark handles DPP with its own // SubqueryBroadcastExec since the join is Spark's BroadcastHashJoinExec - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.isEmpty, - "Should not have CometSubqueryBroadcastExec when Comet BHJ is disabled") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.isEmpty, + "Should not have CometSubqueryBroadcastExec when Comet BHJ is disabled") } } } @@ -1116,27 +1107,25 @@ class CometExecSuite extends CometTestBase { "on fact_date = dim_date where dim_id > 7") val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Verify native scan is used (DPP disabled via TrueLiteral, but scan still native) - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - - // No CometSubqueryBroadcastExec (DPP was disabled), no unconverted SABs - assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(0)) - - // Case 2 of CometPlanAdaptiveDynamicPruningFilters: SMJ with REUSE_BROADCAST_ONLY - // (Spark's default) sets onlyInBroadcast=true on the SAB. With no reusable - // broadcast, the rule replaces the DPP filter with DynamicPruningExpression( - // Literal.TrueLiteral). This distinguishes Case 2 from Case 3 (aggregate - // SubqueryExec), which would appear as a nested SubqueryExec in the filter. - val trueLiteralFilters = nativeScans.flatMap(_.partitionFilters).collect { - case DynamicPruningExpression(Literal.TrueLiteral) => true - } - assert( - trueLiteralFilters.nonEmpty, - "Expected DynamicPruningExpression(TrueLiteral) for onlyInBroadcast=true SMJ, " + - s"got partitionFilters: ${nativeScans.map(_.partitionFilters).mkString("; ")}") + // Verify native scan is used (DPP disabled via TrueLiteral, but scan still native) + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + + // No CometSubqueryBroadcastExec (DPP was disabled), no unconverted SABs + assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(0)) + + // Case 2 of CometPlanAdaptiveDynamicPruningFilters: SMJ with REUSE_BROADCAST_ONLY + // (Spark's default) sets onlyInBroadcast=true on the SAB. With no reusable + // broadcast, the rule replaces the DPP filter with DynamicPruningExpression( + // Literal.TrueLiteral). This distinguishes Case 2 from Case 3 (aggregate + // SubqueryExec), which would appear as a nested SubqueryExec in the filter. + val trueLiteralFilters = nativeScans.flatMap(_.partitionFilters).collect { + case DynamicPruningExpression(Literal.TrueLiteral) => true } + assert( + trueLiteralFilters.nonEmpty, + "Expected DynamicPruningExpression(TrueLiteral) for onlyInBroadcast=true SMJ, " + + s"got partitionFilters: ${nativeScans.map(_.partitionFilters).mkString("; ")}") } } } @@ -1181,17 +1170,15 @@ class CometExecSuite extends CometTestBase { |WHERE s.store_name = '1' AND r.region_name = '2'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert(dppScans.nonEmpty, "Expected DPP filters on native scan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert(dppScans.nonEmpty, "Expected DPP filters on native scan") - // Verify no unconverted SABs - assertAqeDppShape(cometPlan) - } + // Verify no unconverted SABs + assertAqeDppShape(cometPlan) } } } @@ -1265,22 +1252,20 @@ class CometExecSuite extends CometTestBase { |WHERE d.country = '3'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - val dppScans = nativeScans.filter( - _.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) - assert(dppScans.nonEmpty, "Expected DPP filter on native scan") + val dppScans = + nativeScans.filter(_.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression])) + assert(dppScans.nonEmpty, "Expected DPP filter on native scan") - // Verify CometSubqueryBroadcastExec is present (not TrueLiteral fallback) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - "Expected CometSubqueryBroadcastExec (DPP should be active, not TrueLiteral)") + // Verify CometSubqueryBroadcastExec is present (not TrueLiteral fallback) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + "Expected CometSubqueryBroadcastExec (DPP should be active, not TrueLiteral)") } } } @@ -1331,29 +1316,27 @@ class CometExecSuite extends CometTestBase { |FROM dpp_coalesce_other GROUP BY k""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val dppScans = collect(cometPlan) { - case s: CometNativeScanExec - if s.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]) => - s - } - assert(dppScans.size == 1, s"Expected one DPP native scan:\n${cometPlan.treeString}") - // The test only covers the bug if coalescing changed the stage holding the DPP scan. - // That is the final stage, and TreeNode.collect stops at its query stage leaves. - val finalStage = stripAQEPlan(cometPlan) - assert( - finalStage.find(_ eq dppScans.head).nonEmpty && - finalStage.collect { - case r: AQEShuffleReadExec if r.isCoalescedRead => r - }.nonEmpty, - "Expected the DPP scan and a coalesced shuffle read in the final stage:\n" + - cometPlan.treeString) - assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(1)) - // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 partitions. - assert( - dppScans.head.metrics("numPartitions").value == 1, - s"Expected DPP to prune to 1 partition:\n${cometPlan.treeString}") + val dppScans = collect(cometPlan) { + case s: CometNativeScanExec + if s.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]) => + s } + assert(dppScans.size == 1, s"Expected one DPP native scan:\n${cometPlan.treeString}") + // The test only covers the bug if coalescing changed the stage holding the DPP scan. + // That is the final stage, and TreeNode.collect stops at its query stage leaves. + val finalStage = stripAQEPlan(cometPlan) + assert( + finalStage.find(_ eq dppScans.head).nonEmpty && + finalStage.collect { + case r: AQEShuffleReadExec if r.isCoalescedRead => r + }.nonEmpty, + "Expected the DPP scan and a coalesced shuffle read in the final stage:\n" + + cometPlan.treeString) + assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(1)) + // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 partitions. + assert( + dppScans.head.metrics("numPartitions").value == 1, + s"Expected DPP to prune to 1 partition:\n${cometPlan.treeString}") } } } @@ -1412,22 +1395,18 @@ class CometExecSuite extends CometTestBase { shuffles.isEmpty, s"Expected no shuffle around the bucketed join:\n${cometPlan.treeString}") - if (isSpark35Plus) { - val dppScans = collect(cometPlan) { - case s: CometNativeScanExec - if s.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]) => - s - } - assert(dppScans.size == 1, s"Expected one DPP native scan:\n${cometPlan.treeString}") - assert( - dppScans.head.bucketedScan, - s"Expected a bucketed scan:\n${cometPlan.treeString}") - assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(1)) - // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 partitions. - assert( - dppScans.head.metrics("numPartitions").value == 1, - s"Expected DPP to prune to 1 partition:\n${cometPlan.treeString}") + val dppScans = collect(cometPlan) { + case s: CometNativeScanExec + if s.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]) => + s } + assert(dppScans.size == 1, s"Expected one DPP native scan:\n${cometPlan.treeString}") + assert(dppScans.head.bucketedScan, s"Expected a bucketed scan:\n${cometPlan.treeString}") + assertAqeDppShape(cometPlan, expectedCometSubqueryBroadcasts = Some(1)) + // The dimension filter keeps store 3 only, so DPP reads 1 of the 10 partitions. + assert( + dppScans.head.metrics("numPartitions").value == 1, + s"Expected DPP to prune to 1 partition:\n${cometPlan.treeString}") } } } @@ -1446,21 +1425,19 @@ class CometExecSuite extends CometTestBase { |ON f.store_id = s.store_id WHERE f.date_id <= 1030""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Verify no unconverted SABs remain - assertAqeDppShape(cometPlan) + // Verify no unconverted SABs remain + assertAqeDppShape(cometPlan) - // If DPP subqueries are present, verify they use CometSubqueryBroadcastExec - // with AdaptiveSparkPlanExec children (ASPE wrapping broadcast for stageCache reuse) - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - cometSubqueries.foreach { csb => - assert( - csb.child.isInstanceOf[AdaptiveSparkPlanExec], - "CometSubqueryBroadcastExec child should be AdaptiveSparkPlanExec, " + - s"got ${csb.child.getClass.getSimpleName}") - } + // If DPP subqueries are present, verify they use CometSubqueryBroadcastExec + // with AdaptiveSparkPlanExec children (ASPE wrapping broadcast for stageCache reuse) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + cometSubqueries.foreach { csb => + assert( + csb.child.isInstanceOf[AdaptiveSparkPlanExec], + "CometSubqueryBroadcastExec child should be AdaptiveSparkPlanExec, " + + s"got ${csb.child.getClass.getSimpleName}") } } } @@ -1518,17 +1495,15 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected DPP with CometSubqueryBroadcastExec for $dataType key:\n" + - cometPlan.treeString) - - assertAqeDppShape(cometPlan) + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected DPP with CometSubqueryBroadcastExec for $dataType key:\n" + + cometPlan.treeString) + + assertAqeDppShape(cometPlan) } } } @@ -1549,12 +1524,10 @@ class CometExecSuite extends CometTestBase { |JOIN dim_store s ON f.store_id = s.store_id AND s.country = 'NL'""".stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } - assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") + val nativeScans = collect(cometPlan) { case s: CometNativeScanExec => s } + assert(nativeScans.nonEmpty, "Expected CometNativeScanExec in plan") - assertAqeDppShape(cometPlan) - } + assertAqeDppShape(cometPlan) } } } @@ -1607,24 +1580,22 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val dpExprs = flatMap(cometPlan) { - case s: CometNativeScanExec => - s.partitionFilters.collect { case d: DynamicPruningExpression => d.child } - case _ => Nil - } - val hasSubquery = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true - case _ => false - } - val hasBroadcast = dpExprs.exists { - case InSubqueryExec(_, _: SubqueryBroadcastExec, _, _, _, _) => true - case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true - case _ => false - } - assert(!hasSubquery, "Should not have SubqueryExec DPP") - assert(hasBroadcast, "Should have broadcast DPP") + val dpExprs = flatMap(cometPlan) { + case s: CometNativeScanExec => + s.partitionFilters.collect { case d: DynamicPruningExpression => d.child } + case _ => Nil + } + val hasSubquery = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true + case _ => false + } + val hasBroadcast = dpExprs.exists { + case InSubqueryExec(_, _: SubqueryBroadcastExec, _, _, _, _) => true + case InSubqueryExec(_, _: CometSubqueryBroadcastExec, _, _, _, _) => true + case _ => false } + assert(!hasSubquery, "Should not have SubqueryExec DPP") + assert(hasBroadcast, "Should have broadcast DPP") } } } @@ -1649,19 +1620,17 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - val countBroadcasts = collectWithSubqueries(cometPlan) { - case _: SubqueryBroadcastExec => 1 - case _: CometSubqueryBroadcastExec => 1 - }.sum - val countReused = collectWithSubqueries(cometPlan) { - case ReusedSubqueryExec(_: SubqueryBroadcastExec) => 1 - case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 - }.sum - - assert(countBroadcasts == 1, s"Expected 1 SubqueryBroadcast, got $countBroadcasts") - assert(countReused == 1, s"Expected 1 ReusedSubquery, got $countReused") - } + val countBroadcasts = collectWithSubqueries(cometPlan) { + case _: SubqueryBroadcastExec => 1 + case _: CometSubqueryBroadcastExec => 1 + }.sum + val countReused = collectWithSubqueries(cometPlan) { + case ReusedSubqueryExec(_: SubqueryBroadcastExec) => 1 + case ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => 1 + }.sum + + assert(countBroadcasts == 1, s"Expected 1 SubqueryBroadcast, got $countBroadcasts") + assert(countReused == 1, s"Expected 1 ReusedSubquery, got $countReused") } } } @@ -1695,17 +1664,15 @@ class CometExecSuite extends CometTestBase { val (sparkPlan, cometPlan) = checkSparkAnswer(sql(query)) - if (isSpark35Plus) { - val sparkProjects = collectWithSubqueries(sparkPlan) { case p: ProjectExec => p } - val cometProjects = collectWithSubqueries(cometPlan) { - case p: ProjectExec => p - case p: CometProjectExec => p - } - assert( - cometProjects.size == sparkProjects.size, - s"Comet project count (${cometProjects.size}) should match " + - s"Spark (${sparkProjects.size})") + val sparkProjects = collectWithSubqueries(sparkPlan) { case p: ProjectExec => p } + val cometProjects = collectWithSubqueries(cometPlan) { + case p: ProjectExec => p + case p: CometProjectExec => p } + assert( + cometProjects.size == sparkProjects.size, + s"Comet project count (${cometProjects.size}) should match " + + s"Spark (${sparkProjects.size})") } } } @@ -1735,9 +1702,7 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Nil) - if (isSpark35Plus) { - assertAqeDppShape(cometPlan) - } + assertAqeDppShape(cometPlan) } } } @@ -1835,18 +1800,12 @@ class CometExecSuite extends CometTestBase { // Runs a UNION ALL of one CTE joined to dimension group 'a' and to group 'b'. private def checkDppBranchesKeepTheirRows(query: String): Unit = { withDppBranchConf { - if (isSpark35Plus) { - val (_, cometPlan) = checkSparkAnswerAndOperator(sql(query)) - assert( - collect(cometPlan)(dppScan).map(_.metrics("numPartitions").value).sorted == - Seq(1L, 3L), - "Expected one DPP scan per branch, pruned to 1 and 3 partitions:\n" + - cometPlan.treeString) - assertNoReusedExchangeOverDppScan(cometPlan)(dppScan) - } else { - // On 3.4, AQE DPP falls back to Spark. - checkSparkAnswer(sql(query)) - } + val (_, cometPlan) = checkSparkAnswerAndOperator(sql(query)) + assert( + collect(cometPlan)(dppScan).map(_.metrics("numPartitions").value).sorted == Seq(1L, 3L), + "Expected one DPP scan per branch, pruned to 1 and 3 partitions:\n" + + cometPlan.treeString) + assertNoReusedExchangeOverDppScan(cometPlan)(dppScan) } } @@ -1881,13 +1840,10 @@ class CometExecSuite extends CometTestBase { withDppBranchTables { withDppBranchConf { val df = sql(dppShuffleQuery("a", "a")) - val (sparkPlan, cometPlan) = - if (isSpark35Plus) checkSparkAnswerAndOperator(df) else checkSparkAnswer(df) + val (sparkPlan, cometPlan) = checkSparkAnswerAndOperator(df) assertSameExchangeReuseAsSpark(sparkPlan, cometPlan) - if (isSpark35Plus) { - assertExchangeReuseOver(cometPlan, "Expected the shuffle over a DPP scan to be reused")( - dppScan) - } + assertExchangeReuseOver(cometPlan, "Expected the shuffle over a DPP scan to be reused")( + dppScan) } } } @@ -1908,25 +1864,12 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Row(15, 15) :: Nil) - if (isSpark35Plus) { - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.nonEmpty, - s"Expected CometSubqueryBroadcastExec for DPP.\nPlan:\n${cometPlan.treeString}") - } else { - // On 3.4, fall back to Spark-native DPP: expect a SubqueryBroadcastExec - // (not Comet) indicating Spark's PlanAdaptiveDynamicPruningFilters ran. - val sparkSubqueries = collectWithSubqueries(cometPlan) { - case s: SubqueryBroadcastExec => s - } - assert( - sparkSubqueries.nonEmpty, - "Expected Spark SubqueryBroadcastExec for DPP on 3.4 fallback. " + - "If empty, Spark's rule killed DPP (likely because Comet BHJ was " + - s"not falling back).\nPlan:\n${cometPlan.treeString}") + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s } + assert( + cometSubqueries.nonEmpty, + s"Expected CometSubqueryBroadcastExec for DPP.\nPlan:\n${cometPlan.treeString}") } } } @@ -2010,12 +1953,10 @@ class CometExecSuite extends CometTestBase { val (_, cometPlan) = checkSparkAnswer(df) checkAnswer(df, Row(15, 15) :: Nil) - // SABs should have been unwrapped by CometPlanAdaptiveDynamicPruningFilters on 3.5+. - if (isSpark35Plus) { - assertAqeDppShape(cometPlan) - } + // SABs should have been unwrapped by CometPlanAdaptiveDynamicPruningFilters. + assertAqeDppShape(cometPlan) - if (isSpark35Plus && !isSpark41Plus) { + if (!isSpark41Plus) { // 3.5 - 4.0: CometPlanAdaptiveDynamicPruningFilters rewrites the SAB into // CometSubqueryBroadcastExec with the join's CometBroadcastExchange for native // broadcast reuse. @@ -2026,13 +1967,9 @@ class CometExecSuite extends CometTestBase { cometSubqueries.nonEmpty, s"V2 scan should have CometSubqueryBroadcastExec for DPP:\n$cometPlan") } else { - // 3.4 and 4.1+: DPP runs as Spark-native SubqueryBroadcastExec. - // - 3.4: CometSpark34AqeDppFallbackRule keeps the BHJ build broadcast Spark-native - // so Spark's PlanAdaptiveDynamicPruningFilters can create SubqueryBroadcastExec - // and AQE stageCache can dedupe with the DPP subquery's broadcast. - // - 4.1+: Partial/final aggregate shuffle is elided, which removes Comet's entry - // point for this query, so CometPlanAdaptiveDynamicPruningFilters falls into - // its Spark-native branch. + // On 4.1+, DPP runs as Spark-native SubqueryBroadcastExec. Partial/final aggregate + // shuffle is elided, which removes Comet's entry point for this query, so + // CometPlanAdaptiveDynamicPruningFilters falls into its Spark-native branch. val sparkSubqueries = collectWithSubqueries(cometPlan) { case s: SubqueryBroadcastExec => s } @@ -2123,99 +2060,53 @@ class CometExecSuite extends CometTestBase { """.stripMargin) val (_, cometPlan) = checkSparkAnswer(df) - if (isSpark35Plus) { - // Regression check: without the ReusedSubqueryExec unwrap in extractSABData, - // one CSAB survives the rule and trips CometSubqueryAdaptiveBroadcastExec.doExecute - // at runtime. assertAqeDppShape verifies no CSABs remain in the final plan. - assertAqeDppShape(cometPlan) + // Regression check: without the ReusedSubqueryExec unwrap in extractSABData, + // one CSAB survives the rule and trips CometSubqueryAdaptiveBroadcastExec.doExecute + // at runtime. assertAqeDppShape verifies no CSABs remain in the final plan. + assertAqeDppShape(cometPlan) - // Subquery reuse: exactly one canonical CometSubqueryBroadcastExec, plus at - // least one ReusedSubqueryExec(CometSubqueryBroadcastExec) pointer for the - // second fact scan. Without this dedup, both fact scans would evaluate the - // DPP subquery independently. - val cometSubqueries = collectWithSubqueries(cometPlan) { - case s: CometSubqueryBroadcastExec => s - } - assert( - cometSubqueries.size == 1, - "Expected exactly 1 CometSubqueryBroadcastExec (shared between fact scans), " + - s"got ${cometSubqueries.size}:\n${cometPlan.treeString}") - val reusedCsbs = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r - } - assert( - reusedCsbs.nonEmpty, - "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec) " + - s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") - - // Broadcast reuse via AQE stageCache: the DPP subquery's ASPE and the main - // BHJ should share the same underlying CometBroadcastExchange. Without this, - // we'd build two identical broadcasts of the dim. - val dppBroadcast = cometSubqueries.head.child match { - case aspe: AdaptiveSparkPlanExec => - val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } - assert( - bqs.isDefined, - "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + - cometPlan.treeString) - bqs.get.broadcast - case other => - fail( - s"Unexpected CometSubqueryBroadcastExec child: ${other.getClass.getSimpleName}") - } - val hasReuse = find(cometPlan) { - case ReusedExchangeExec(_, e) => e eq dppBroadcast - case b: BroadcastExchangeLike => b eq dppBroadcast - case _ => false - }.isDefined - assert( - hasReuse, - "DPP subquery's broadcast should be reused by the main BHJ " + - s"(via AQE stageCache):\n${cometPlan.treeString}") - } else { - // Spark 3.4: injectQueryStageOptimizerRule is unavailable, so - // CometPlanAdaptiveDynamicPruningFilters can't run. V1 fact scans are rejected - // to Spark by CometScanRule.transformV1Scan, and CometSpark34AqeDppFallbackRule - // tags the BHJ's build-side BroadcastExchange so Spark's own - // PlanAdaptiveDynamicPruningFilters handles DPP natively. Expected shape - // mirrors the 3.5+ assertions but with Spark-native node types. - val sparkSubqueries = collectWithSubqueries(cometPlan) { - case s: SubqueryBroadcastExec => s - } - assert( - sparkSubqueries.size == 1, - "Expected exactly 1 SubqueryBroadcastExec on 3.4 (Spark-native DPP, " + - s"shared between fact scans), got ${sparkSubqueries.size}. If 0, " + - "CometSpark34AqeDppFallbackRule didn't keep the BHJ Spark-native and " + - s"Spark's rule killed DPP:\n${cometPlan.treeString}") - val reusedSparkSubqueries = collectWithSubqueries(cometPlan) { - case r @ ReusedSubqueryExec(_: SubqueryBroadcastExec) => r - } - assert( - reusedSparkSubqueries.nonEmpty, - "Expected at least one ReusedSubqueryExec(SubqueryBroadcastExec) on 3.4 " + - s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") - val dppBroadcast = sparkSubqueries.head.child match { - case aspe: AdaptiveSparkPlanExec => - val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } - assert( - bqs.isDefined, - "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + - cometPlan.treeString) - bqs.get.broadcast - case other => - fail(s"Unexpected SubqueryBroadcastExec child: ${other.getClass.getSimpleName}") - } - val hasReuse = find(cometPlan) { - case ReusedExchangeExec(_, e) => e eq dppBroadcast - case b: BroadcastExchangeLike => b eq dppBroadcast - case _ => false - }.isDefined - assert( - hasReuse, - "DPP subquery's broadcast should be reused by the main BHJ on 3.4:\n" + - cometPlan.treeString) + // Subquery reuse: exactly one canonical CometSubqueryBroadcastExec, plus at + // least one ReusedSubqueryExec(CometSubqueryBroadcastExec) pointer for the + // second fact scan. Without this dedup, both fact scans would evaluate the + // DPP subquery independently. + val cometSubqueries = collectWithSubqueries(cometPlan) { + case s: CometSubqueryBroadcastExec => s + } + assert( + cometSubqueries.size == 1, + "Expected exactly 1 CometSubqueryBroadcastExec (shared between fact scans), " + + s"got ${cometSubqueries.size}:\n${cometPlan.treeString}") + val reusedCsbs = collectWithSubqueries(cometPlan) { + case r @ ReusedSubqueryExec(_: CometSubqueryBroadcastExec) => r } + assert( + reusedCsbs.nonEmpty, + "Expected at least one ReusedSubqueryExec(CometSubqueryBroadcastExec) " + + s"for the second fact scan's DPP filter:\n${cometPlan.treeString}") + + // Broadcast reuse via AQE stageCache: the DPP subquery's ASPE and the main + // BHJ should share the same underlying CometBroadcastExchange. Without this, + // we'd build two identical broadcasts of the dim. + val dppBroadcast = cometSubqueries.head.child match { + case aspe: AdaptiveSparkPlanExec => + val bqs = collectFirst(aspe) { case b: BroadcastQueryStageExec => b } + assert( + bqs.isDefined, + "Expected BroadcastQueryStageExec inside DPP subquery's ASPE:\n" + + cometPlan.treeString) + bqs.get.broadcast + case other => + fail(s"Unexpected CometSubqueryBroadcastExec child: ${other.getClass.getSimpleName}") + } + val hasReuse = find(cometPlan) { + case ReusedExchangeExec(_, e) => e eq dppBroadcast + case b: BroadcastExchangeLike => b eq dppBroadcast + case _ => false + }.isDefined + assert( + hasReuse, + "DPP subquery's broadcast should be reused by the main BHJ " + + s"(via AQE stageCache):\n${cometPlan.treeString}") } } } @@ -2407,8 +2298,6 @@ class CometExecSuite extends CometTestBase { } test("subquery execution under CometTakeOrderedAndProjectExec should not fail") { - assume(isSpark35Plus, "SPARK-45584 is fixed in Spark 3.5+") - withTable("t1") { sql(""" |CREATE TABLE t1 USING PARQUET @@ -3435,11 +3324,10 @@ class CometExecSuite extends CometTestBase { .map(_ => (Random.nextInt(), Random.nextInt() % 5)), "tbl") { - (if (isSpark35Plus) Seq("tinyint", "short", "int", "long", "string") else Seq("long")) - .foreach { input_type => - val df = sql(f"SELECT bloom_filter_agg(cast(_2 as $input_type)) FROM tbl") - checkSparkAnswerAndOperator(df) - } + Seq("tinyint", "short", "int", "long", "string").foreach { input_type => + val df = sql(f"SELECT bloom_filter_agg(cast(_2 as $input_type)) FROM tbl") + checkSparkAnswerAndOperator(df) + } } spark.sessionState.functionRegistry.dropFunction(funcId_bloom_filter_agg) @@ -3959,7 +3847,6 @@ class CometExecSuite extends CometTestBase { // Spark coalesces each child of a union as its own group, but its rule did not recognize // Comet's union, so the shuffled branch of a union with a scan kept every shuffle partition. // Comet's rule defers to Spark's, so the query should come out partitioned as it is on Spark. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withTempPath { dir => spark.range(0, 100, 1, 1).toDF("c").write.parquet(dir.getCanonicalPath) withSQLConf( @@ -4053,7 +3940,6 @@ class CometExecSuite extends CometTestBase { // Every leaf below the union is a shuffle stage, so Spark's rule coalesces its shuffles // together, and gives up because the aggregate's single-partition shuffle cannot be // coalesced. Below a Spark union it coalesces the join's shuffles as a group of their own. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1", @@ -4076,7 +3962,6 @@ class CometExecSuite extends CometTestBase { // aggregate above reads the union without a shuffle. Coalescing the branch that can be // coalesced, and not the other, would split each key between the branches' partitions, and // Comet's aggregate, which requires no distribution, would return each key twice. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key -> "1", @@ -4098,7 +3983,6 @@ class CometExecSuite extends CometTestBase { test("AQE coalesces the shuffle partitions of a union planned again over its stages") { // When AQE plans a query again after its stages materialize, a union directly over those // stages has the same children as the Spark union that Comet replaced. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withTempPath { dir => spark.range(0, 100, 1, 1).toDF("c").write.parquet(dir.getCanonicalPath) withSQLConf( @@ -4128,7 +4012,6 @@ class CometExecSuite extends CometTestBase { // group of its own, and from Spark 4.0 it targets the minimum partition size rather than the // advisory size below the latter two, which join every row with every row of the other side. // The shuffled branch of the union should keep as many partitions as it does on Spark. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withTempPath { dir => spark.range(0, 10, 1, 1).toDF("c").write.parquet(dir.getCanonicalPath) withSQLConf( @@ -4197,7 +4080,6 @@ class CometExecSuite extends CometTestBase { // Spark's rule coalesces the shuffle on the other side of a Cartesian product as a group of // its own before Comet's rule runs. The read it leaves there should not keep the union's // shuffled branch from being coalesced as it is on Spark. - assume(isSpark35Plus, "Comet's query-stage optimizer rules need Spark 3.5+") withTempPath { dir => spark.range(0, 10, 1, 1).toDF("c").write.parquet(dir.getCanonicalPath) withSQLConf( @@ -4800,8 +4682,8 @@ class CometExecSuite extends CometTestBase { df.select("*").groupBy("key", "value").count(), includeClasses = Seq(classOf[CometSparkToColumnarExec])) - // Verify that the BatchScanExec nodes supported columnar output when requested for Spark 3.4+. - // Earlier versions support columnar output for fewer type. + // Verify that the BatchScanExec nodes support columnar output when requested. + // Spark 3.5 supports columnar output for fewer types. val leaves = df.queryExecution.executedPlan.collectLeaves() if (parquetVectorized) { assert(leaves.forall(_.supportsColumnar)) @@ -4960,7 +4842,6 @@ class CometExecSuite extends CometTestBase { // https://github.com/apache/datafusion-comet/issues/6202 test("SparkToColumnar over InMemoryTableScanExec survives the AQE re-plan") { - assume(isSpark35Plus, "Table-cache query stages require Spark 3.5+") // CometInMemoryCacheSuite covers this too, but registers Comet's rules twice, through its // plugin and CometTestBase. This suite registers them once, as production does. Reset the // serializer so the relation is cached in Spark's format, as in the test above. diff --git a/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala index 338f1425fdb..29a788577ec 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala @@ -56,7 +56,7 @@ import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} import org.apache.spark.storage.StorageLevel import org.apache.comet.{CometArrowAllocator, CometConf, CometKryoRegistrator, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.rules.{CometCacheBuildRule, CometCacheColumnarRule} import org.apache.comet.vector.{CometPlainVector, CometVector} @@ -152,7 +152,6 @@ class CometInMemoryCacheSuite extends CometTestBase { // https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L3114-L3154 test("AQE SPARK-42101: cold and warm Comet cache materialization") { - assume(isSpark35Plus, "Table-cache query stages require Spark 3.5+") withAQECache { withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") { val left = spark.range(0, 10, 1, 2).selectExpr("cast(id as string) c1") @@ -187,7 +186,6 @@ class CometInMemoryCacheSuite extends CometTestBase { // https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L3156-L3176 test("AQE SPARK-42101: preserve shuffle partitions beside a table cache stage") { - assume(isSpark35Plus, "Table-cache query stages require Spark 3.5+") withAQECache { withSQLConf( SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1", @@ -207,7 +205,6 @@ class CometInMemoryCacheSuite extends CometTestBase { // https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L3178-L3191 test("AQE SPARK-42101: coalesce the shuffle partitions of a union with a table cache stage") { - assume(isSpark35Plus, "Table-cache query stages require Spark 3.5+") withAQECache { withSQLConf(SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key -> "1") { val cached = Seq(1).toDF("c").cache() @@ -281,7 +278,6 @@ class CometInMemoryCacheSuite extends CometTestBase { // https://github.com/apache/datafusion-comet/issues/6202 test("AQE keeps the operators above a table cache stage native once the stage materializes") { - assume(isSpark35Plus, "Table-cache query stages require Spark 3.5+") // `t` holds 1000 ids for each k, and the ids for a given k sum to 1000 * k + 4995000. val queries = Seq( "SELECT k, count(*) FROM t GROUP BY k" -> (0L until 10L).map(k => Row(k, 1000L)), @@ -796,7 +792,7 @@ class CometInMemoryCacheSuite extends CometTestBase { } assert(transitions.size == (if (enabled) 1 else 0), plan.toString) assert(collect(plan) { case s: CometInMemoryTableScanExec => s }.isEmpty) - if (adaptive && isSpark35Plus) { + if (adaptive) { assert(collect(plan) { case s: QueryStageExec if s.getClass.getSimpleName == "TableCacheQueryStageExec" => @@ -3161,10 +3157,7 @@ class CometInMemoryCacheSuite extends CometTestBase { // adaptively-cached relation that also finalizes the cached plan. EXPLAIN must launch nothing. withConversions( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true", - // Spark caches through a session with some configs forced off, and on 3.4 that list still - // includes AQE itself, so the cached plan comes back non-adaptive and there is nothing to - // finalize. This conf is what decides that list; 3.5 defaults it on, and 4.0 stopped - // disabling AQE either way. Setting it keeps the relation adaptive on every version. + // This setting keeps the cached relation adaptive on every supported version. SQLConf.CAN_CHANGE_CACHED_PLAN_OUTPUT_PARTITIONING.key -> "true", CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true") { @@ -3581,7 +3574,7 @@ class CometInMemoryCacheSuite extends CometTestBase { test("Comet in-memory cache keeps the observed metrics recorded in a cached plan") { // Spark collects the metrics of an observe() inside a cached plan only through an // InMemoryTableScanExec over it, so the scan of such a relation has to stay Spark's. Replaced, - // the metrics come back empty, and on Spark 3.4 Observation.get never returns. Nested the way + // the metrics come back empty and Observation.get never returns. Nested the way // SPARK-35695's test nests it, with a shuffle in the inner cached plan so that AQE plans it, // under one more cache that records no metrics of its own. The metrics read through that top // scan are only found by following it into the caches it reads. @@ -3621,7 +3614,7 @@ class CometInMemoryCacheSuite extends CometTestBase { val observation = Observation("cached_observation") val observed = spark.range(10).observe(observation, sum($"id").as("total")).persist() observed.collect() - // Bounded, so that a regression fails here rather than hanging the suite on Spark 3.4. + // Bounded, so that a regression fails here rather than hanging the suite. assert(Await.result(Future(observation.get), 1.minute) == Map("total" -> 45L)) val plain = spark.range(0, 100, 1, 2).persist() diff --git a/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala index b8a645a9eb6..e57d2eee864 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala @@ -50,7 +50,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{ArrayType, DoubleType, IntegerType, LongType, MetadataBuilder, StructField, StructType} import org.apache.comet.{CometConf, CometExecIterator, CometExplainInfo, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{hasFallbackReason, isSpark35Plus} +import org.apache.comet.CometSparkSessionExtensions.hasFallbackReason import org.apache.comet.rules.RewriteJoin import org.apache.comet.serde.OperatorOuterClass @@ -905,7 +905,7 @@ class CometJoinSuite extends CometTestBase { withParquetTable(Seq((Some(10), 1L), (Some(20), 2L), (None, 3L)), "dynamic_probe") { withParquetTable(Seq((Some(10), 4L), (None, 5L)), "dynamic_build") { val from = "FROM dynamic_probe p " - // Spark 3.4 requires building the left side for a right outer shuffled hash join. + // Spark 3.5 requires building the left side for a right outer shuffled hash join. val joins = Seq( "LEFT JOIN dynamic_build b ON p._1 = b._1" -> "b", "RIGHT JOIN dynamic_build b ON p._1 = b._1" -> "p", @@ -1057,12 +1057,10 @@ class CometJoinSuite extends CometTestBase { sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a FULL JOIN tbl_b ON tbl_a._2 = tbl_b._1") checkSparkAnswerAndOperator(df3) - // TODO: Spark 3.4 returns SortMergeJoin for this query even with SHUFFLE_HASH hint. - // Left join with build left and right join with build right in hash join is only supported - // in Spark 3.5 or above. See SPARK-36612. - // // Left join: build left - // sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a LEFT JOIN tbl_b ON tbl_a._2 = tbl_b._1") + val dfLeftBuildLeft = + sql("SELECT /*+ SHUFFLE_HASH(tbl_a) */ * FROM tbl_a LEFT JOIN tbl_b ON tbl_a._2 = tbl_b._1") + checkSparkAnswerAndOperator(dfLeftBuildLeft) // Inner join: build right val df4 = @@ -2723,21 +2721,17 @@ class CometJoinSuite extends CometTestBase { assertScansShareOneNativePlan(fullOuter) // Without filter pushdown the scans carry no data filters, and dynamic partition - // pruning picks a different partition for each side. Spark 3.4 only runs this with - // AQE off: CometScanRule.transformV1Scan leaves a V1 scan with AQE dynamic pruning in - // Spark there (case 2 in CometSpark34AqeDppFallbackRule). - if (!adaptive || isSpark35Plus) { - withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "false") { - val (_, pruned) = checkSparkAnswer( - "SELECT /*+ BROADCAST(d1), BROADCAST(d2) */ a.k, a.v, b.v " + - "FROM bucketed_self a JOIN bucketed_self b ON a.k = b.k " + - "JOIN bucketed_dim d1 ON a.p = d1.p JOIN bucketed_dim d2 ON b.p = d2.p " + - "WHERE d1.x = 'one' AND d2.x = 'two'") - assertScansShareOneNativePlan(pruned).foreach { scan => - assert( - scan.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]), - s"Expected each scan to be pruned dynamically:\n$pruned") - } + // pruning picks a different partition for each side. + withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "false") { + val (_, pruned) = checkSparkAnswer( + "SELECT /*+ BROADCAST(d1), BROADCAST(d2) */ a.k, a.v, b.v " + + "FROM bucketed_self a JOIN bucketed_self b ON a.k = b.k " + + "JOIN bucketed_dim d1 ON a.p = d1.p JOIN bucketed_dim d2 ON b.p = d2.p " + + "WHERE d1.x = 'one' AND d2.x = 'two'") + assertScansShareOneNativePlan(pruned).foreach { scan => + assert( + scan.partitionFilters.exists(_.isInstanceOf[DynamicPruningExpression]), + s"Expected each scan to be pruned dynamically:\n$pruned") } } } diff --git a/spark/src/test/scala/org/apache/comet/exec/CometMergeRowsNativeSuiteBase.scala b/spark/src/test/scala/org/apache/comet/exec/CometMergeRowsNativeSuiteBase.scala index c9459b1b742..e939940831f 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometMergeRowsNativeSuiteBase.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometMergeRowsNativeSuiteBase.scala @@ -29,7 +29,6 @@ import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.util.QueryExecutionListener import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus /** * `CometMergeRowsExec` converts Spark's `MergeRowsExec` and the `MergeRows` logical node that @@ -63,10 +62,7 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive .set("spark.sql.shuffle.partitions", "4") } - private def assumeMerge(): Unit = assume(isSpark35Plus, "MergeRowsExec requires Spark 3.5+") - test("MERGE all row-routing groups engage CometMergeRowsExec and match Spark") { - assumeMerge() val target = s"$catalog.default.rowlevel_target" val source = s"$catalog.default.rowlevel_source" @@ -135,7 +131,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive } test("MERGE cardinality violation matches Spark's structured exception") { - assumeMerge() val target = s"$catalog.default.rowlevel_target_card" val source = s"$catalog.default.rowlevel_source_card" @@ -254,7 +249,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive } test("MERGE with a scalar-subquery assignment runs natively and matches Spark") { - assumeMerge() val target = s"$catalog.default.subq_target" val source = s"$catalog.default.subq_source" @@ -310,7 +304,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive // a Split instruction, so the test cannot silently stop exercising delete+reinsert semantics. private def deltaMergeCase(name: String, splitUpdates: Boolean, matchedClause: String): Unit = { test(s"delta-backed MERGE: $name") { - assumeMerge() val target = s"$catalog.default.delta_t_${name.replaceAll("\\W", "_")}" val source = s"$catalog.default.delta_s_${name.replaceAll("\\W", "_")}" @@ -372,7 +365,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive names.foreach(name => sql(s"DROP TABLE IF EXISTS $name")) test("disabled MergeRows flag falls back cleanly") { - assumeMerge() val target = s"$catalog.default.flag_target" val source = s"$catalog.default.flag_source" dropTables(target, source) @@ -394,7 +386,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive } test("copy-on-write matched delete runs through native MergeRows") { - assumeMerge() val target = s"$catalog.default.delete_target" val source = s"$catalog.default.delete_source" dropTables(target, source) @@ -415,7 +406,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive } test("first matching clause prevents later ANSI predicate evaluation") { - assumeMerge() val target = s"$catalog.default.order_target" val source = s"$catalog.default.order_source" dropTables(target, source) @@ -439,7 +429,6 @@ abstract class CometMergeRowsNativeSuiteBase extends CometTestBase with Adaptive } test("CometMergeRowsExec semantic identity reflects instruction groups and cardinality flag") { - assumeMerge() val target = s"$catalog.default.eq_target" val source = s"$catalog.default.eq_source" sql(s"DROP TABLE IF EXISTS $target") diff --git a/spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala index bb06a09fbf3..f3ced4e3012 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala @@ -56,7 +56,7 @@ import org.apache.spark.sql.types.{ArrayType, DataType, IntegerType, LongType, M import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} import org.apache.comet.{CometConf, CometExecIterator, CometExplainInfo, CometShuffleBlockIterator, CometShuffleSizeLimitException, Native} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, withFallbackReason} import org.apache.comet.serde.{OperatorOuterClass, PartitioningOuterClass} import org.apache.comet.shuffle.ShufflePartitionPusher @@ -1989,7 +1989,6 @@ class CometNativeShuffleSuite extends CometTestBase with AdaptiveSparkPlanHelper } test("AQE DPP query with a broadcast side reads its shuffle stage through ShuffleScan") { - assume(isSpark35Plus, "Native AQE DPP requires Spark 3.5+") withTempDir { dir => withSQLConf(CometConf.COMET_ENABLED.key -> "false") { spark diff --git a/spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala index 5d09646d61e..62c87654b96 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala @@ -36,7 +36,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.DecimalType import org.apache.comet.{CometConf, CometExplainInfo, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus class CometWindowExecSuite extends CometTestBase { @@ -126,8 +126,6 @@ class CometWindowExecSuite extends CometTestBase { test( s"window group limit: $orderColumn NaN and signed zero peers at the cutoff " + s"(strictFloatingPoint=$strictFloatingPoint)") { - assume(isSpark35Plus, "WindowGroupLimit was added in Spark 3.5") - val positiveFloatNaN = java.lang.Float.intBitsToFloat(0x7fc00001) val negativeFloatNaN = java.lang.Float.intBitsToFloat(0xffc00002) val positiveDoubleNaN = java.lang.Double.longBitsToDouble(0x7ff8000000000001L) @@ -208,7 +206,6 @@ class CometWindowExecSuite extends CometTestBase { } test("window group limit: floating-point values nested in the order key") { - assume(isSpark35Plus, "WindowGroupLimit was added in Spark 3.5") // The native RANK and DENSE_RANK find ties by byte equality, on order keys that the native // planner has normalized, nested floats included. So [-0.0] and [0.0] are peers, and the // cutoff keeps both rows, as Spark does (#5507). These keys, read back from Parquet, can hold @@ -269,7 +266,6 @@ class CometWindowExecSuite extends CometTestBase { } test("window group limit: nested null peers with non-default root null ordering") { - assume(isSpark35Plus, "WindowGroupLimit was added in Spark 3.5") withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", SQLConf.SHUFFLE_PARTITIONS.key -> "1", @@ -340,8 +336,6 @@ class CometWindowExecSuite extends CometTestBase { } { test( s"window: floating partition keys ($partitionColumn, $function, group limit=$groupLimit)") { - assume(!groupLimit || isSpark35Plus, "WindowGroupLimit was added in Spark 3.5") - val partitionKeys = Seq( (Some(1.0f), Some(1.0d)), (Some(2.0f), Some(2.0d)), diff --git a/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala b/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala index 62e23e1a72e..5f4efe8ce57 100644 --- a/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala +++ b/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala @@ -54,7 +54,7 @@ import org.apache.spark.sql.types.{ArrayType, IntegerType, LongType, MapType, Me import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.hadoop.fs.RecordingFileSystem import org.apache.comet.rules.RevertNativeForTransitionHeavyStages import org.apache.comet.serde.operator.NativeWriteUtils @@ -633,7 +633,6 @@ class CometParquetWriterSuite extends CometParquetWriterTestBase { } test("parquet write with a mixed-case unsupported compression codec falls back to Spark") { - assume(isSpark35Plus, "lz4_raw was added in Spark 3.5") // The other half of the case bug, and the dangerous half: a case-sensitive read misses the // option entirely, finds `snappy` in the SQLConf, decides the codec is supported, and writes // SNAPPY bytes into a file Spark named `...-c000.lz4raw.parquet`. @@ -724,7 +723,6 @@ class CometParquetWriterSuite extends CometParquetWriterTestBase { } test("parquet write with unsupported compression codec falls back to Spark") { - assume(isSpark35Plus, "lz4_raw was added in Spark 3.5") withTempPath { dir => val outputPath = new File(dir, "output.parquet").getAbsolutePath val df = spark.range(0, 100).selectExpr("id", "cast(id as string) as name") diff --git a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromS3Suite.scala b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromS3Suite.scala index 4fbcec0ca0b..35b9b7daf31 100644 --- a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromS3Suite.scala +++ b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromS3Suite.scala @@ -41,7 +41,6 @@ import org.apache.spark.sql.functions.{col, expr, max, sum} import org.apache.spark.sql.internal.SQLConf import org.apache.comet.{CometConf, CometS3TestBase} -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus class ParquetReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelper { @@ -286,7 +285,6 @@ class ParquetReadFromS3Suite extends CometS3TestBase with AdaptiveSparkPlanHelpe test("dynamic partition pruning over a table whose partitions span two buckets") { // AQE dynamic partition pruning falls back to Spark's scan before Spark 3.5. - assume(isSpark35Plus) createBucketIfNotExists(secondBucketName) withTable("mb_fact", "mb_dim") { sql(s"""CREATE TABLE mb_fact (id INT, p STRING) USING parquet PARTITIONED BY (p) diff --git a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala index 56743367a4a..10772988d3a 100644 --- a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala +++ b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala @@ -53,7 +53,7 @@ import org.apache.spark.sql.types._ import com.google.common.primitives.UnsignedLong import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus} import org.apache.comet.vector.CometVector abstract class ParquetReadSuite extends CometTestBase { @@ -475,7 +475,7 @@ abstract class ParquetReadSuite extends CometTestBase { } test("TIMESTAMP_MILLIS overflow after the first batch of a file") { - // Spark 3.4 and 3.5 wrap a reader error in cannotReadFilesError only while FileScanRDD reads + // Spark 3.5 wraps a reader error in cannotReadFilesError only while FileScanRDD reads // the first batch of a file split. A later batch throws from hasNext without the wrapper: // https://github.com/apache/spark/blob/v3.5.9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L126-L132 // https://github.com/apache/spark/blob/v3.5.9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L285-L298 @@ -2553,10 +2553,10 @@ abstract class ParquetReadSuite extends CometTestBase { Seq(false, true).foreach { pruning => withSQLConf(SQLConf.NESTED_SCHEMA_PRUNING_ENABLED.key -> pruning.toString) { withClue(s"nested schema pruning $pruning: ") { - // Spark 3.4 keeps the root id in requiredSchema. From 3.5, pruning removes it - // while scan.output still carries it. With no id, the renamed root is absent - // from the file by name, so Spark reads null even though its output has an id. - val hasRootId = !pruning || !isSpark35Plus + // Pruning removes the root id from requiredSchema while scan.output still carries + // it. With no id, the renamed root is absent from the file by name, so Spark reads + // null even though its output has an id. + val hasRootId = !pruning def read(): DataFrame = spark.read.schema(readSchema).parquet(dir.getCanonicalPath) def checkRead(query: => DataFrame, expected: Seq[Row]): Unit = { diff --git a/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala b/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala index ffbe1d0e1b0..0ad4c50a2c1 100644 --- a/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala +++ b/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala @@ -51,7 +51,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{DataTypes, DoubleType, FloatType, StructField, StructType} import org.apache.comet.{CometConf, CometCoverageStats, CometExplainInfo, CometSparkSessionExtensions, ConfigEntry, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark42Plus, withFallbackReason} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark42Plus, withFallbackReason} import org.apache.comet.serde.{CometAggregateExpressionSerde, Compatible, ExprOuterClass, OperatorOuterClass, QueryPlanSerde, Unsupported} import org.apache.comet.shims.ShimCometWindowGroupLimit import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator} @@ -1678,7 +1678,6 @@ class CometExecRuleSuite extends CometTestBase { } test("AQE DPP broadcast roots retain temporary logical links after an unchanged replan") { - assume(isSpark35Plus, "Native AQE DPP requires Spark 3.5+") // With direct read, the aggregate under the broadcast reads the dim side's shuffle stage // through a ShuffleScan only once its stage is prepared, so each replan stays unchanged. Seq(false, true).foreach { directRead => @@ -1749,8 +1748,8 @@ class CometExecRuleSuite extends CometTestBase { // AQE adopts a replan that differs from its current plan, after which it cannot link a stage // built over a reused operator back into its logical plan. A broadcast stage it cannot link // no longer fixes the join's build side, which broke the exchange reuse that Spark's - // SPARK-34637 DPP test checks on Spark 3.4. The aggregate under the broadcast is the root of - // its native block, so a refreshed native plan there changes the plan that AQE compares. + // The aggregate under the broadcast is the root of its native block, so a refreshed native + // plan there changes the plan that AQE compares. def reusedOverStage(plan: SparkPlan): Seq[SparkPlan] = plan.collect { case native: CometNativeExec if native.children.exists(_.isInstanceOf[ShuffleQueryStageExec]) => @@ -1840,8 +1839,9 @@ class CometExecRuleSuite extends CometTestBase { Seq(0.0d, -0.0d) .flatMap(v => Seq(ordinary, infinities).map(list => (list :+ literal(v), false, 1))) ++ - Seq((ordinary ++ Seq(literal(0.0d), literal(-0.0d)), false, 0)) ++ - (if (isSpark35Plus) Seq((Seq.empty[Expression], false, 0)) else Nil) + Seq( + (ordinary ++ Seq(literal(0.0d), literal(-0.0d)), false, 0), + (Seq.empty[Expression], false, 0)) for ((list, needsNormalization, extraStaticCandidates) <- lists; asSet <- Seq(false, true) if !asSet || list.forall(_.isInstanceOf[Literal]); negate <- Seq(false, true); @@ -2593,7 +2593,6 @@ class CometExecRuleSuite extends CometTestBase { } test("CometExecRule should not allow try_sum mixed execution") { - assume(isSpark35Plus, "try_sum was added in Spark 3.5") withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") val sparkPlan = @@ -2919,7 +2918,7 @@ class CometExecRuleSuite extends CometTestBase { withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") - // Cast to bigint: Spark 3.4's bloom_filter_agg only accepts a long-typed first + // Cast to bigint so the test uses a type supported by bloom_filter_agg. // argument; later versions widened it to any integral type. val sparkPlan = createSparkPlan(spark, "SELECT bloom_filter_agg(CAST(id AS BIGINT)) FROM test_data") @@ -2958,7 +2957,7 @@ class CometExecRuleSuite extends CometTestBase { withTempView("test_data") { createTestDataFrame.createOrReplaceTempView("test_data") - // Cast to bigint: Spark 3.4's bloom_filter_agg only accepts a long-typed first + // Cast to bigint so the test uses a type supported by bloom_filter_agg. // argument; later versions widened it to any integral type. val sparkPlan = createSparkPlan(spark, "SELECT bloom_filter_agg(CAST(id AS BIGINT)) FROM test_data") @@ -3103,7 +3102,7 @@ class CometExecRuleSuite extends CometTestBase { } test("CometExecRule should not convert hash aggregate when grouping key contains map type") { - // Spark 3.4/3.5 reject `array>` as a grouping key in the analyzer (not orderable), + // Spark 3.5 rejects `array>` as a grouping key in the analyzer (not orderable), // so the plan never reaches CometExecRule on those versions. The guard we're exercising // (containsMapType) only matters on Spark 4.0+, which permits the GROUP BY to be analyzed. assume(isSpark40Plus) diff --git a/spark/src/test/scala/org/apache/comet/rules/CometMultiStoreScanSuite.scala b/spark/src/test/scala/org/apache/comet/rules/CometMultiStoreScanSuite.scala index 3db9f2fced3..a650ebcf2f5 100644 --- a/spark/src/test/scala/org/apache/comet/rules/CometMultiStoreScanSuite.scala +++ b/spark/src/test/scala/org/apache/comet/rules/CometMultiStoreScanSuite.scala @@ -30,7 +30,6 @@ import org.apache.commons.io.FileUtils import org.apache.spark.SparkConf import org.apache.spark.sql.{CometTestBase, DataFrame, SaveMode} import org.apache.spark.sql.catalyst.expressions.DynamicPruningExpression -import org.apache.spark.sql.catalyst.plans.physical.UnknownPartitioning import org.apache.spark.sql.comet.{CometCsvNativeScanExec, CometNativeScanExec, CometScanExec} import org.apache.spark.sql.execution.{ExtendedMode, FileSourceScanExec, FormattedMode, SparkPlan} import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper @@ -237,8 +236,6 @@ class CometMultiStoreScanSuite extends CometTestBase with AdaptiveSparkPlanHelpe } test("global aggregate over a csv scan Spark plans as one partition across name nodes") { - // Spark 3.4 reports a one-partition V2 scan as SinglePartition and plans no exchange below - // the final aggregate, so a split scan would run that aggregate on two partitions. val (a, b) = (hdfs("nn1", "csv-agg-a"), hdfs("nn2", "csv-agg-b")) writeIds(a, 0, "csv") writeIds(b, 10, "csv") @@ -257,20 +254,10 @@ class CometMultiStoreScanSuite extends CometTestBase with AdaptiveSparkPlanHelpe val plan = query.queryExecution.executedPlan val scans = collect(plan) { case scan: CometCsvNativeScanExec => scan } - if (sparkScan.outputPartitioning.isInstanceOf[UnknownPartitioning]) { - // Spark planned an exchange above the scan, so splitting the scan is safe. The query is - // not run because native cannot read the fake stores. - assert(scans.size == 1, s"expected one native CSV scan:\n$plan") - assert(scans.head.outputPartitioning.numPartitions == 2) - val exchanges = collect(plan) { case e: ShuffleExchangeLike => e } - assert(exchanges.nonEmpty, s"no exchange above the scan:\n$plan") - } else { - assert(scans.isEmpty, s"a split scan would break Spark's partitioning:\n$plan") - checkSparkAnswerAndFallbackReason( - query, - "Native CSV scan would split a partition that mixes object stores, but Spark " + - "planned the operators above it for the scan's own partitioning") - } + assert(scans.size == 1, s"expected one native CSV scan:\n$plan") + assert(scans.head.outputPartitioning.numPartitions == 2) + val exchanges = collect(plan) { case e: ShuffleExchangeLike => e } + assert(exchanges.nonEmpty, s"no exchange above the scan:\n$plan") } } diff --git a/spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala b/spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala index cb46516e5cc..573751dae65 100644 --- a/spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala +++ b/spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala @@ -41,7 +41,7 @@ import org.apache.spark.sql.types.BinaryType import org.apache.spark.sql.util.QueryExecutionListener import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus} +import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus import org.apache.comet.serde.OperatorOuterClass.Operator private case class AliasingFallbackCometExec( @@ -634,7 +634,6 @@ class RevertNativeForTransitionHeavyStagesSuite extends CometTestBase { } test("AQE DPP remains executable when transition reversion restores a V1 scan") { - assume(isSpark35Plus, "Comet AQE DPP query-stage optimizer rules require Spark 3.5+") import testImplicits._ withTempDir { dir => diff --git a/spark/src/test/scala/org/apache/comet/serde/SparkFixesSuite.scala b/spark/src/test/scala/org/apache/comet/serde/SparkFixesSuite.scala index c733785991c..40a6b12ee31 100644 --- a/spark/src/test/scala/org/apache/comet/serde/SparkFixesSuite.scala +++ b/spark/src/test/scala/org/apache/comet/serde/SparkFixesSuite.scala @@ -36,7 +36,6 @@ class SparkFixesSuite extends AnyFunSuite { test("fixes by Spark version") { // SPARK-58821, SPARK-60118 and SPARK-60119 are in 4.0.5, 4.1.4 and 4.2.1. Seq( - "3.4.3" -> false, "3.5.9" -> false, "4.0.4" -> false, "4.1.3" -> false, @@ -59,7 +58,6 @@ class SparkFixesSuite extends AnyFunSuite { // 4.2.0 normalizes the arguments in the plan (SPARK-54918), and 4.0.5+, 4.1.4+ and 4.2.1+ // normalize during evaluation (SPARK-59602). Earlier releases keep flat signed zeros apart. checkVersions("array set float normalization", SparkFixes.arraySetFunctionsNormalizeFloats)( - "3.4.3" -> false, "3.5.9" -> false, "4.0.0" -> false, "4.0.4" -> false, @@ -83,7 +81,6 @@ class SparkFixesSuite extends AnyFunSuite { // earlier patches of the same minor version. A vendor suffix parses and a missing patch // reads as 0. checkVersions("slope pair-null filtering", SparkFixes.slopeFiltersVarByPairNulls)( - "3.4.3" -> false, "3.5.0" -> false, "3.5.1" -> false, "3.5.2" -> true, @@ -96,7 +93,6 @@ class SparkFixesSuite extends AnyFunSuite { "3.5" -> false) checkVersions("regr_r2 degenerate-case swap", SparkFixes.r2DegenerateCasesSwapped)( - "3.4.3" -> false, "3.5.3" -> false, "3.5.8" -> false, "3.5.9" -> true, @@ -120,7 +116,6 @@ class SparkFixesSuite extends AnyFunSuite { test("percentile keeps signed-zero keys from the Spark release that fixed it") { // SPARK-45599 is in 3.5.2 and 4.0.0. checkVersions("percentile signed-zero keys", SparkFixes.keepsSignedZeroKeys)( - "3.4.3" -> false, "3.5.0" -> false, "3.5.1" -> false, "3.5" -> false, diff --git a/spark/src/test/scala/org/apache/comet/serde/operator/CometNativeScanSuite.scala b/spark/src/test/scala/org/apache/comet/serde/operator/CometNativeScanSuite.scala index a2bcde5d3c8..fba592852f6 100644 --- a/spark/src/test/scala/org/apache/comet/serde/operator/CometNativeScanSuite.scala +++ b/spark/src/test/scala/org/apache/comet/serde/operator/CometNativeScanSuite.scala @@ -35,22 +35,20 @@ class CometNativeScanSuite extends AnyFunSuite { import CometNativeScan.{constantMetadataFieldPrefix, uniqueConstantMetadataFields} - test("native data schema preserves required root field ids and table projection indexes") { + test("native data schema preserves table root field ids and projection indexes") { val nestedType = StructType(Seq(StructField("a", IntegerType), StructField("b", StringType))) - val root = StructField("renamed", nestedType, nullable = false) - val dataSchema = - StructType(Seq(StructField("unrequested", LongType), root, StructField("id", IntegerType))) val partitionSchema = StructType(Seq(StructField("part", StringType))) - Seq(true, false).foreach { hasRequiredId => + Seq(true, false).foreach { hasFieldId => val metadata = new MetadataBuilder() - if (hasRequiredId) { + if (hasFieldId) { metadata.putLong(ParquetUtils.FIELD_ID_METADATA_KEY, 2L) } - val requiredRoot = root.copy( - dataType = StructType(Seq(nestedType("a"))), - nullable = true, - metadata = metadata.build()) + val root = StructField("renamed", nestedType, nullable = false, metadata = metadata.build()) + val dataSchema = + StructType( + Seq(StructField("unrequested", LongType), root, StructField("id", IntegerType))) + val requiredRoot = root.copy(dataType = StructType(Seq(nestedType("a"))), nullable = true) val requiredSchema = StructType(Seq(dataSchema("id"), requiredRoot)) val output = (requiredSchema.fields ++ partitionSchema.fields).map { field => AttributeReference(field.name, field.dataType, field.nullable, field.metadata)() @@ -74,8 +72,8 @@ class CometNativeScanSuite extends AnyFunSuite { val serializedRoot = common.getDataSchema(1) val fieldIdKey = CometParquetUtils.PARQUET_FIELD_ID_META_KEY - assert(serializedRoot.getMetadataMap.containsKey(fieldIdKey) == hasRequiredId) - if (hasRequiredId) { + assert(serializedRoot.getMetadataMap.containsKey(fieldIdKey) == hasFieldId) + if (hasFieldId) { assert(serializedRoot.getMetadataMap.get(fieldIdKey) == "2") } // Replacing the whole field with requiredRoot would lose the table's type and nullability. diff --git a/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala b/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala index 6ebb7957c65..dc228b98916 100644 --- a/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala +++ b/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala @@ -157,7 +157,7 @@ class CometPluginsSuite extends CometTestBase { test("executor memory overhead is left alone") { // Comet does not adjust spark.executor.memoryOverhead. A driver plugin runs too late to - // influence the executor container on Spark 3.4, 3.5 and 4.0, so the value the application + // influence the executor container on Spark 3.5 and 4.0, so the value the application // set is the value that is used. val execMemOverhead1 = spark.conf.get("spark.executor.memoryOverhead") val execMemOverhead2 = spark.sessionState.conf.getConfString("spark.executor.memoryOverhead") @@ -560,7 +560,7 @@ class CometPluginsEventLogSuite extends SparkFunSuite { assert( pluginContext.ask( memoryUsageEvent("barrier", 0L, Array(0L, 0L, 0L, 0L), jvmArrow)) == null) - // The end as the listener sees it, which alone records what the driver holds on Spark 3.4 + // The end as the listener sees it, which records what the driver holds on Spark 3.5 // and 3.5. From Spark 4.0 the plugin's shutdown would record it too, when the context stops. postAndMark(sc, SparkListenerApplicationEnd(System.currentTimeMillis())) } diff --git a/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala b/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala index 2116f7506d1..129ed7370ec 100644 --- a/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala +++ b/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala @@ -612,7 +612,7 @@ abstract class CometTestBase // differs across architectures, so a native kernel and Spark can return NaNs that print // identically but compare unequal. Spark fixed this in `compare` itself for 4.1.2 and 4.2.0 // ("in some hardware NaN can be represented with different bits, so first check for it"), but - // 3.4, 3.5, 4.0 and 4.1.1 still compare raw bits. Canonicalize NaN here so every supported + // 3.5, 4.0 and 4.1.1 still compare raw bits. Canonicalize NaN here so every supported // version asserts NaN-ness without asserting the payload. Signed zero is deliberately left // alone: distinguishing it is the reason the raw-bit comparison exists. case f: java.lang.Float => if (f.isNaN) Float.NaN else f.floatValue @@ -700,7 +700,7 @@ abstract class CometTestBase (stripAQEPlan(plan) +: stagePlans).foreach(checkCometOperators(_, excluded: _*)) } - // Matched by name because Spark 3.4 has no TableCacheQueryStageExec. + // Matched by name so this shared helper does not bind to a minor-version implementation class. protected def isTableCacheStage(plan: SparkPlan): Boolean = plan.getClass.getSimpleName == "TableCacheQueryStageExec" diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCartesianProductBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCartesianProductBenchmark.scala index 6d49037855c..0b39523583e 100644 --- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCartesianProductBenchmark.scala +++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCartesianProductBenchmark.scala @@ -143,7 +143,7 @@ object CometCartesianProductBenchmark extends CometBenchmarkBase { ProductInSparkCaseName -> productInSparkConfigs, CometCaseName -> cometConfigs) val sums = arms.map { case (arm, configs) => - // A local, because `withSQLConf` returns `Unit` on Spark 3.4 and 3.5. + // A local, because `withSQLConf` returns `Unit` on Spark 3.5. var sum: Any = null withSQLConf(configs: _*) { val df = spark.sql(query) diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCodegenDispatchBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCodegenDispatchBenchmark.scala index 50984716f9e..e7579074e68 100644 --- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCodegenDispatchBenchmark.scala +++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometCodegenDispatchBenchmark.scala @@ -266,7 +266,7 @@ object CometCodegenDispatchBenchmark extends CometBenchmarkBase { */ private def verifyArmsAgree(c: DispatchCase, rows: Int): Unit = { def collect(configs: Seq[(String, String)]): Array[String] = { - // Assigned to a local rather than returned from the block: Spark 3.4 and 3.5 declare + // Assigned to a local rather than returned from the block: Spark 3.5 declares // `SQLHelper.withSQLConf` as returning `Unit`; only Spark 4 has the result-returning form. var collected: Array[Row] = Array.empty withSQLConf(configs: _*) { diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometIcebergSystemFunctionBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometIcebergSystemFunctionBenchmark.scala index c77339af329..52b8baa29a7 100644 --- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometIcebergSystemFunctionBenchmark.scala +++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometIcebergSystemFunctionBenchmark.scala @@ -96,7 +96,7 @@ object CometIcebergSystemFunctionBenchmark extends CometBenchmarkBase { */ private def verifyOutputsMatch(name: String, query: String): Unit = { // The rows are assigned to a local rather than returned from the `withSQLConf` block, because - // Spark 3.4 and 3.5 declare `SQLHelper.withSQLConf` as returning `Unit`; only Spark 4 has the + // Spark 3.5 declares `SQLHelper.withSQLConf` as returning `Unit`; only Spark 4 has the // generic result-returning form. def collect(cometEnabled: Boolean): Array[Row] = { var rows: Array[Row] = Array.empty diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometStringExpressionBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometStringExpressionBenchmark.scala index e28afe33748..178dc7da1f1 100644 --- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometStringExpressionBenchmark.scala +++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometStringExpressionBenchmark.scala @@ -126,7 +126,7 @@ object CometStringExpressionBenchmark extends CometBenchmarkBase { "from parquetV1Table")) } - // Collated cases are Spark 4.0+ only, since the COLLATE syntax does not parse on 3.4 and 3.5. + // Collated cases are Spark 4.0+ only, since the COLLATE syntax does not parse on 3.5. // CometLevenshtein reports collated input as Unsupported and CodegenDispatchFallback runs it // through the JVM codegen dispatcher, so these measure the dispatcher rather than the native // kernel. See https://github.com/apache/datafusion-comet/issues/5591. diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/CometCollectMetricsSuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/CometCollectMetricsSuite.scala index ad5aef6241d..0e50d55f40e 100644 --- a/spark/src/test/scala/org/apache/spark/sql/comet/CometCollectMetricsSuite.scala +++ b/spark/src/test/scala/org/apache/spark/sql/comet/CometCollectMetricsSuite.scala @@ -35,7 +35,6 @@ import org.apache.spark.sql.types.{ArrayType, IntegerType} import org.apache.spark.sql.util.QueryExecutionListener import org.apache.comet.CometConf -import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus import org.apache.comet.rules.CometExecRule class CometCollectMetricsSuite extends CometTestBase { @@ -44,7 +43,7 @@ class CometCollectMetricsSuite extends CometTestBase { private def observedMetrics(observation: Observation): Map[String, Any] = { // A missing observation in the executed plan must fail instead of blocking in get forever. - // Observation.get is available on every supported Spark version, including Spark 3.4. + // Observation.get is available on every supported Spark version. val result = new FutureTask[Map[String, Any]](() => observation.get) val reader = new Thread(result, "comet-observation-reader") reader.setDaemon(true) @@ -162,7 +161,6 @@ class CometCollectMetricsSuite extends CometTestBase { for (adaptive <- Seq(false, true)) { test(s"observation preserves widened array nullability, AQE=$adaptive") { - assume(isSpark35Plus) withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> adaptive.toString, SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> @@ -471,7 +469,6 @@ class CometCollectMetricsSuite extends CometTestBase { } test("observation below a window group limit preserves Spark row demand") { - assume(isSpark35Plus, "WindowGroupLimit was added in Spark 3.5") withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", "spark.sql.optimizer.windowGroupLimitThreshold" -> "1000", @@ -576,8 +573,7 @@ class CometCollectMetricsSuite extends CometTestBase { for ((consumeRows, sortBeforeConsume) <- Seq((1, false), (0, true))) { test(s"public foreachPartition preserves observation row demand, rows=$consumeRows") { - // Spark 3.4 runs foreachPartition without a name, so no listener callback reports its plan. - assume(isSpark35Plus) + // A listener callback may not report the foreachPartition plan. withSQLConf( SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true") { diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala index b63db32128b..903f123360d 100644 --- a/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala +++ b/spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala @@ -35,7 +35,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.TestSparkSession import org.apache.comet.{CometConf, ExtendedExplainInfo} -import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus, isSpark41Plus, isSpark42Plus} +import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, isSpark41Plus, isSpark42Plus} /** * Similar to [[org.apache.spark.sql.PlanStabilitySuite]], checks that TPC-DS Comet plans don't @@ -275,8 +275,7 @@ private object CometPlanStabilitySuite { Seq( isSpark42Plus -> s"approved-plans-${variant}-spark4_2", isSpark41Plus -> s"approved-plans-${variant}-spark4_1", - isSpark40Plus -> s"approved-plans-${variant}-spark4_0", - isSpark35Plus -> s"approved-plans-${variant}-spark3_5") + isSpark40Plus -> s"approved-plans-${variant}-spark4_0") .dropWhile(!_._1) .map(_._2) :+ s"approved-plans-${variant}" } diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala b/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala index 7789e13ed7f..f013b619657 100644 --- a/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala +++ b/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala @@ -305,8 +305,8 @@ object CometCachedBatchHelper { * The whole payload is rewritten, frame header included, so every zstd release rejects it * outright. Corrupting only a frame's tail is not enough: whether that is detected depends on * the zstd-jni each Spark version ships -- Comet takes it from Spark rather than from - * arrow-compression, and 1.5.5 (Spark 3.4, 3.5) decodes a frame whose last bytes have been - * zeroed that 1.5.7 (Spark 4.x) reports as corrupt. + * arrow-compression, and 1.5.5 (Spark 3.5) decodes a frame whose last bytes have been zeroed + * that 1.5.7 (Spark 4.x) reports as corrupt. */ private def scramble(batch: CachedBatch, start: Long, length: Long): Unit = { val from = start + 8 diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManagerSuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManagerSuite.scala index de3cc5de317..9c193a958ba 100644 --- a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManagerSuite.scala +++ b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManagerSuite.scala @@ -293,7 +293,7 @@ class CometCelebornShuffleManagerSuite extends AnyFunSuite { val original = coordinator.claimMapAttempt(ClaimCelebornMapAttempt(12, stageId, 0, 0, 0)) assert(original.authorized) - // Spark 3.4 calls stageFailed when taskCompleted matches an authorized file committer. + // Spark calls stageFailed when taskCompleted matches an authorized file committer. // Keeping that slot empty also covers executor loss, where no task-side cleanup can run. assert(sparkCommitOwners(sparkCoordinator, stageId).forall(_ == null)) completeFailedAttempt(sparkCoordinator, stageId, 0, 0, 0, reason)