Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion docs/source/user-guide/latest/in-memory-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,8 @@ above it.
Building the cache is measured the same way, from the same source, with its rows produced either by
Comet operators or by Spark operators. Comet's format is written straight from the Arrow batches of
Comet operators, while Spark's is always built from rows, so above Comet operators it first converts
their batches. Measured on an Apple M3 Max (JDK 17, Spark 4.1, release build):
their batches. On Spark 3.5, Comet's format is written from Arrow batches only with AQE off; see
[Limitations](#limitations). Measured on an Apple M3 Max (JDK 17, Spark 4.1, release build):

| Rows produced by | Spark's cache format | Comet's cache format, `zstd` | Comet's cache format, `none` |
| ---------------- | -------------------: | ---------------------------: | ---------------------------: |
Expand Down Expand Up @@ -297,6 +298,14 @@ operator reads a relation cached in Comet's format depends on the scan below it:
Whenever Spark's scan reads Comet's format, which also happens for a relation whose cached plan
records `Dataset.observe` metrics, Comet records a fallback reason on the scan.

On **Spark 3.5**, a relation cached under AQE (`spark.sql.adaptive.enabled`, on by default) is built
from rows. Spark builds a cache from the batches of its plan only when the plan supports columnar
output, and a cached `AdaptiveSparkPlanExec` does not until Spark 4.0. So the cached plan runs
natively, then converts its batches to rows, which are written into Arrow again before they are
compressed, where Spark 4.0 and later, or any version without AQE, write the Arrow batches as they
are. Comet cannot change this without a hook in Spark. The relation is read the same way once it is
built, but the build timings above are from Spark 4.1 and do not apply to this case.

Measured by the same benchmark over the same 5M-row relation, with native execution off so that
Spark operators consume the cached data, Comet disabled for the row reader and enabled for the fused
reader (Apple M4, JDK 17, Spark 4.1; the average of two runs):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2991,6 +2991,47 @@ class CometInMemoryCacheSuite extends CometTestBase {
assert(zstdSize == plainSize, s"the relation size should not depend on the codec: $sizes")
}

test("Comet in-memory cache is built from rows under AQE before Spark 4.0") {
// Spark builds a cache from the batches of its plan only when the plan reports that it
// supports columnar output. A cached AdaptiveSparkPlanExec does not before Spark 4.0, so there
// a relation cached under AQE goes native plan, columnar-to-row, UnsafeRow and back to Arrow
// before it is compressed. Without AQE, or from 4.0 on, it is written from the Arrow batches.
// The user guide's build timings are from Spark 4.1 and say so.
Seq(true, false).foreach { adaptive =>
withConversions(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> adaptive.toString,
SQLConf.SHUFFLE_PARTITIONS.key -> "3",
CometConf.COMET_SHUFFLE_MODE.key -> "jvm",
CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true") {
spark.catalog.clearCache()
try {
spark
.range(0, 100, 1, 2)
.selectExpr("id % 7 AS k", "id AS v")
.groupBy("k")
.agg(sum("v").as("s"))
.createOrReplaceTempView("rows_build_cache")
spark.catalog.cacheTable("rows_build_cache")
assert(spark.table("rows_build_cache").count() == 7)
val cachedPlan = spark.sharedState.cacheManager
.lookupCachedData(spark.table("rows_build_cache"))
.get
.cachedRepresentation
.cacheBuilder
.cachedPlan
assert(
cachedPlan.isInstanceOf[AdaptiveSparkPlanExec] == adaptive,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the same reason, I expect this to fail on Spark 3.4 with adaptive = true, since the test does not set spark.sql.optimizer.canChangeCachedPlanOutputPartitioning and the cached plan comes back non-adaptive. withAQECache and the planning test set that conf to true for exactly this. The description says it passes on 3.4, and I could not find what makes it pass there. I have not run it. Could you recheck with run-spark-3.4-tests or run-all-spark-profiles, and set the conf here if needed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right: without the conf the test fails on 3.4 (false did not equal true AQE is true), and my claim that it passed there was wrong. It now sets CAN_CHANGE_CACHED_PLAN_OUTPUT_PARTITIONING to true, as withAQECache does, and passes on 3.4 locally. Fixed in f94fe70.

s"AQE is $adaptive:\n$cachedPlan")
assert(
cachedPlan.supportsColumnar == (!adaptive || isSpark40Plus),
s"AQE is $adaptive:\n$cachedPlan")
} finally {
spark.catalog.clearCache()
}
}
}
}

test("Comet in-memory cache does not store the bytes before a sliced aggregate output") {
// A grouped aggregate emits its groups as batch-size slices of one batch, and a slice of a
// string array reaches the JVM with offsets that start past zero and the rows before it still
Expand Down