Skip to content

Native Iceberg scan with AQE DPP fails when the DPP build side is a storage-partitioned join #7441

Open

Description

@mbutrovich

Describe the bug

With AQE on, a query fails with java.lang.AssertionError: assertion failed in AdaptiveSparkPlanExec.setLogicalLinkForNewQueryStage when the build side of a dynamic partition pruning (DPP) broadcast contains a storage-partitioned join over native Iceberg scans. A storage-partitioned join is a sort-merge join that Spark plans without a shuffle because both scans report KeyGroupedPartitioning.

org.apache.spark.SparkException: Exception thrown in awaitResult:
  at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:326)
  at org.apache.spark.sql.comet.CometSubqueryBroadcastExec.executeCollect(CometSubqueryBroadcastExec.scala:149)
  at org.apache.spark.sql.execution.InSubqueryExec.updateResult(subquery.scala:129)
  ...
Caused by: java.lang.AssertionError: assertion failed
  at scala.Predef$.assert(Predef.scala:208)
  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.setLogicalLinkForNewQueryStage(AdaptiveSparkPlanExec.scala:648)
  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.newQueryStage(AdaptiveSparkPlanExec.scala:622)
  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.createQueryStages(AdaptiveSparkPlanExec.scala:540)
  ...

The trace is from Spark 3.5. The same assertion fails on Spark 3.4, 4.0, and 4.1. On Spark 3.4 the DPP subquery is Spark's SubqueryBroadcastExec instead of CometSubqueryBroadcastExec. The spark-4.2 profile has no Iceberg runtime, so the test below does not run there.

It is a regression from #7231. The query passes at 7f50037, the parent of #7231, where the DPP subquery reuses the join's broadcast through a ReusedExchange. It fails at ce3e665.

Steps to reproduce

Add this test to CometIcebergNativeSuite:

test("AQE DPP - storage-partitioned join on the DPP build side") {
  assume(icebergAvailable, "Iceberg not available")
  withTempIcebergDir { warehouseDir =>
    withSQLConf(
      "spark.sql.catalog.aqe_cat" -> "org.apache.iceberg.spark.SparkCatalog",
      "spark.sql.catalog.aqe_cat.type" -> "hadoop",
      "spark.sql.catalog.aqe_cat.warehouse" -> warehouseDir.getAbsolutePath,
      SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
      SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
      SQLConf.V2_BUCKETING_ENABLED.key -> "true",
      SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
      "spark.sql.iceberg.planning.preserve-data-grouping" -> "true",
      CometConf.COMET_ENABLED.key -> "true",
      CometConf.COMET_EXEC_ENABLED.key -> "true",
      CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {

      spark.sql("""
        CREATE TABLE aqe_cat.db.probe_fact (amount INT, store_id INT)
        USING iceberg PARTITIONED BY (store_id)
      """)
      spark.sql("""
        CREATE TABLE aqe_cat.db.probe_dim_a (id INT, country STRING)
        USING iceberg PARTITIONED BY (id)
      """)
      spark.sql("""
        CREATE TABLE aqe_cat.db.probe_dim_b (id INT, name STRING)
        USING iceberg PARTITIONED BY (id)
      """)
      spark.sql(
        "INSERT INTO aqe_cat.db.probe_fact SELECT CAST(id AS INT), CAST(id % 10 AS INT) " +
          "FROM range(0, 100)")
      spark.sql(
        "INSERT INTO aqe_cat.db.probe_dim_a SELECT CAST(id AS INT), " +
          "IF(id < 3, 'X', 'Y') FROM range(0, 10)")
      spark.sql(
        "INSERT INTO aqe_cat.db.probe_dim_b SELECT CAST(id AS INT), concat('n', id) " +
          "FROM range(0, 10)")

      val query =
        """SELECT /*+ BROADCAST(d) */ f.store_id, f.amount, d.name
          |FROM aqe_cat.db.probe_fact f JOIN (
          |  SELECT /*+ MERGE(a, b) */ a.id, b.name
          |  FROM aqe_cat.db.probe_dim_a a JOIN aqe_cat.db.probe_dim_b b ON a.id = b.id
          |  WHERE a.country = 'X'
          |) d ON f.store_id = d.id""".stripMargin
      checkSparkAnswer(query)

      spark.sql("DROP TABLE aqe_cat.db.probe_fact")
      spark.sql("DROP TABLE aqe_cat.db.probe_dim_a")
      spark.sql("DROP TABLE aqe_cat.db.probe_dim_b")
    }
  }
}

Then run it:

make core
./mvnw test -Pspark-3.5 -Dtest=none -Dsuites="org.apache.comet.CometIcebergNativeSuite AQE DPP - storage-partitioned join on the DPP build side"

Expected behavior

The query returns the same rows as Spark, and the DPP subquery reuses the join's broadcast, as it does at 7f50037.

Additional context

To build the DPP subquery's plan, Comet wraps the build side it already converted in a new AdaptiveSparkPlanExec (CometPlanAdaptiveDynamicPruningFilters.scala#L262-L302). On Spark 3.4, Spark's own rule does the same (PlanAdaptiveDynamicPruningFilters.scala#L48-L61). The AdaptiveSparkPlanExec constructor runs the query stage preparation rules over that plan again, including EnsureRequirements (AdaptiveSparkPlanExec.scala#L113-L134, #L204-L208).

BatchScanExec reports KeyGroupedPartitioning for these scans (DataSourceV2ScanExecBase.scala#L93-L103), but CometIcebergNativeScanExec reports UnknownPartitioning(0) (CometIcebergNativeScanExec.scala#L138). Since #7231, CometSortMergeJoinExec declares Spark's required child distribution (operators.scala#L746-L749, #L3740-L3753). So EnsureRequirements inserts an exchange under each side of the join in the subquery's plan, although the main plan has none:

CometSubqueryBroadcast dynamicpruning#93, [0], [id#61], [id=#835]
+- AdaptiveSparkPlan isFinalPlan=false
   +- CometBroadcastExchange [id#61, name#64]
      +- CometProject [id#61, name#64], [id#61, name#64]
         +- CometSortMergeJoin [id#61], [id#63], Inner
            :- CometExchange hashpartitioning(id#61, 10), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=830]
            :  +- CometSort [id#61], [id#61 ASC NULLS FIRST]
            :     +- CometProject [id#61], [id#61]
            :        +- CometFilter [id#61, country#62], (isnotnull(country#62) AND (country#62 = X))
            :           +- CometIcebergNativeScan [id#61, country#62], ... [filters=country IS NOT NULL, country = 'X', id IS NOT NULL, groupedBy=id], 3
            +- CometExchange hashpartitioning(id#63, 10), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=820]
               +- CometSort [id#63, name#64], [id#63 ASC NULLS FIRST]
                  +- CometIcebergNativeScan [id#63, name#64], ... [filters=id IS NOT NULL, groupedBy=id], runtimeFilters=[dynamicpruningexpression(id#63 IN dynamicpruning#92)], 10

AQE finds no logical link on the inserted exchange or below it, and the assertion fails (AdaptiveSparkPlanExec.scala#L641-L650).

#7422 is the SinglePartition case of the same mismatch on Spark 3.4, and #7424 fixes it by having the native scan report what BatchScanExec reports. Could the native scan report originalPlan.outputPartitioning for KeyGroupedPartitioning too? #6268 moved it to UnknownPartitioning(0) because counting the scan's tasks went through BatchScanExec.inputRDD, which translates the DPP filter before its subquery has a result. As I read Spark 3.5.9, BatchScanExec.outputPartitioning reads groupedPartitions, which groups inputPartitions from planInputPartitions(), not the runtime-filtered partitions (BatchScanExec.scala#L59, #L116-L128, DataSourceV2ScanExecBase.scala#L106-L112).

Willingness to contribute

I cannot contribute a fix for this bug at this time

Activity

  1. added
    priority:mediumFunctional bugs, performance regressions, broken features
    area:scanParquet scan / data reading
    and removed on Nov 9, 2026
  2. snmvaughan commented on Nov 16, 2026

    @snmvaughan
    Contributor

    #7510 has merged, so branch-1.2 now has the same pieces: the native scan reports UnknownPartitioning(0), CometSortMergeJoinExec declares Spark's required distribution, and CometPlanAdaptiveDynamicPruningFilters wraps the converted build side in a new AdaptiveSparkPlanExec.

    Does the reproduction in this issue fail on branch-1.2 now? If it does, could the issue be labelled for a branch-1.2 backport of its fix and noted for the 1.2.x release notes in the meantime?

  3. snmvaughan commented on Nov 16, 2026

    @snmvaughan
    Contributor

    take

  4. snmvaughan commented on Nov 16, 2026

    @snmvaughan
    Contributor

    Thanks for the report and the suggestion. Having CometIcebergNativeScanExec report the replaced BatchScanExec's partitioning (as #7424 does for SinglePartition) is needed for an aggregate above the join, but it cannot stop the join's own exchanges. The DPP rule wraps the already converted build side in a new AdaptiveSparkPlanExec, whose constructor runs EnsureRequirements again. Since #7231 the Comet sort-merge and shuffled-hash joins declare the original join's ClusteredDistribution, and EnsureRequirements only treats key-grouped children as co-partitioned when the parent is Spark's own SortMergeJoinExec or ShuffledHashJoinExec (on 4.1 also over Spark's own KeyGroupedPartitionedScan leaves). A Comet join is neither, so each side gets a hash exchange whatever the scan reports, and AQE has no logical link for it. What works is for a Comet join whose two sides are scans Spark planned as key-grouped, with no shuffle below either side, to declare no required distribution, as it did before #7231. Joins with a shuffle below keep the #7231 behavior. The scan change stays for the aggregate case. #7662 makes both changes and adds a test for each shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions