diff --git a/docs/source/user-guide/latest/in-memory-cache.md b/docs/source/user-guide/latest/in-memory-cache.md index ef73632b925..c522ac38510 100644 --- a/docs/source/user-guide/latest/in-memory-cache.md +++ b/docs/source/user-guide/latest/in-memory-cache.md @@ -220,7 +220,8 @@ above it. Building the cache is measured the same way, from the same source, with its rows produced either by Comet operators or by Spark operators. Comet's format is written straight from the Arrow batches of Comet operators, while Spark's is always built from rows, so above Comet operators it first converts -their batches. Measured on an Apple M3 Max (JDK 17, Spark 4.1, release build): +their batches. On Spark 3.5, Comet's format is written from Arrow batches only with AQE off; see +[Limitations](#limitations). Measured on an Apple M3 Max (JDK 17, Spark 4.1, release build): | Rows produced by | Spark's cache format | Comet's cache format, `zstd` | Comet's cache format, `none` | | ---------------- | -------------------: | ---------------------------: | ---------------------------: | @@ -297,6 +298,14 @@ operator reads a relation cached in Comet's format depends on the scan below it: Whenever Spark's scan reads Comet's format, which also happens for a relation whose cached plan records `Dataset.observe` metrics, Comet records a fallback reason on the scan. +On **Spark 3.5**, a relation cached under AQE (`spark.sql.adaptive.enabled`, on by default) is built +from rows. Spark builds a cache from the batches of its plan only when the plan supports columnar +output, and a cached `AdaptiveSparkPlanExec` does not until Spark 4.0. So the cached plan runs +natively, then converts its batches to rows, which are written into Arrow again before they are +compressed, where Spark 4.0 and later, or any version without AQE, write the Arrow batches as they +are. Comet cannot change this without a hook in Spark. The relation is read the same way once it is +built, but the build timings above are from Spark 4.1 and do not apply to this case. + Measured by the same benchmark over the same 5M-row relation, with native execution off so that Spark operators consume the cached data, Comet disabled for the row reader and enabled for the fused reader (Apple M4, JDK 17, Spark 4.1; the average of two runs): diff --git a/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala index 338f1425fdb..eee0d1ccc22 100644 --- a/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala +++ b/spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala @@ -2991,6 +2991,47 @@ class CometInMemoryCacheSuite extends CometTestBase { assert(zstdSize == plainSize, s"the relation size should not depend on the codec: $sizes") } + test("Comet in-memory cache is built from rows under AQE before Spark 4.0") { + // Spark builds a cache from the batches of its plan only when the plan reports that it + // supports columnar output. A cached AdaptiveSparkPlanExec does not before Spark 4.0, so there + // a relation cached under AQE goes native plan, columnar-to-row, UnsafeRow and back to Arrow + // before it is compressed. Without AQE, or from 4.0 on, it is written from the Arrow batches. + // The user guide's build timings are from Spark 4.1 and say so. + Seq(true, false).foreach { adaptive => + withConversions( + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> adaptive.toString, + SQLConf.SHUFFLE_PARTITIONS.key -> "3", + CometConf.COMET_SHUFFLE_MODE.key -> "jvm", + CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true") { + spark.catalog.clearCache() + try { + spark + .range(0, 100, 1, 2) + .selectExpr("id % 7 AS k", "id AS v") + .groupBy("k") + .agg(sum("v").as("s")) + .createOrReplaceTempView("rows_build_cache") + spark.catalog.cacheTable("rows_build_cache") + assert(spark.table("rows_build_cache").count() == 7) + val cachedPlan = spark.sharedState.cacheManager + .lookupCachedData(spark.table("rows_build_cache")) + .get + .cachedRepresentation + .cacheBuilder + .cachedPlan + assert( + cachedPlan.isInstanceOf[AdaptiveSparkPlanExec] == adaptive, + s"AQE is $adaptive:\n$cachedPlan") + assert( + cachedPlan.supportsColumnar == (!adaptive || isSpark40Plus), + s"AQE is $adaptive:\n$cachedPlan") + } finally { + spark.catalog.clearCache() + } + } + } + } + test("Comet in-memory cache does not store the bytes before a sliced aggregate output") { // A grouped aggregate emits its groups as batch-size slices of one batch, and a slice of a // string array reaches the JVM with offsets that start past zero and the rows before it still