Repository navigation
Conversation
… ExtendedExplainInfo `getFallbackReasons(df)` returns the fallback reasons of a DataFrame's executed plan. `planAccelerated(df)` is true when Comet is enabled for the session, using the same check as the Comet planner rules, and the plan has no fallback reasons.
andygrove
reviewed
Nov 18, 2026
| * manager is not set, even if `spark.comet.enabled` is true. Like `getFallbackReasons(df)`, | ||
| * this does not run the query. | ||
| */ | ||
| def planAccelerated(df: DataFrame): Boolean = { |
Member
There was a problem hiding this comment.
nit: the name is misleading, IMO. A plan can still be accelerated and have partial fallback. Maybe this should be planFullyAccelerated or noFallbacks or something?
Contributor
Author
There was a problem hiding this comment.
thats always tricky part :)
andygrove
approved these changes
Nov 18, 2026
The old name read as "some of the plan is accelerated". The method returns true only when Comet is enabled and the plan has no fallback reasons, so a plan with partial fallback is not fully accelerated. Add a user guide subsection that describes the check.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue was filed for this small addition.
Rationale for this change
Checking whether Comet ran a query natively currently takes
new ExtendedExplainInfo().getFallbackReasons(df.queryExecution.executedPlan)plus a separate checkthat Comet is enabled at all. About 20 suites repeat the first part, and applications that embed
Comet need the same check. A plan-only check is also misleading on its own: when Comet disables
itself (for example without off-heap memory or without the Comet shuffle manager), the plan has no
fallback reasons because Comet never tagged it, so "no fallback reasons" does not mean "accelerated".
What changes are included in this PR?
Two methods on
ExtendedExplainInfo, the class users already reach throughspark.sql.extendedExplainProviders:getFallbackReasons(df: DataFrame): Seq[String]returns the fallback reasons of the DataFrame'sexecuted plan. It delegates to the existing
getFallbackReasons(plan: SparkPlan).isFullyAccelerated(df: DataFrame): Booleanis true when Comet is enabled for the DataFrame'ssession and the plan has no fallback reasons. A plan with partial fallback is not fully
accelerated. "Enabled" uses
CometSparkSessionExtensions.isCometLoaded,the same check the Comet planner rules make, rather than
spark.comet.enabledalone.Neither method runs the query. For an adaptive query that has not run yet, the executed plan is the
initial plan. The scaladoc and the user guide say to run the query first to get the final plan. The user
guide in
understanding-comet-plans.mdgets a short subsection, "Checking That a Query Is FullyAccelerated", that describes the check.
The session conf is read through
df.queryExecution.sparkSession, because on Spark 4.xDataFrameis the API
Dataset, which hasqueryExecutionbut notsessionState.How are these changes tested?
A new test in
CometExecSuite, "getFallbackReasons and isFullyAccelerated for a DataFrame", covers:getFallbackReasons(df)contains thereason, matches the plan overload, and
isFullyAcceleratedis false.isFullyAcceleratedis true.spark.comet.enabled=false:isFullyAcceleratedis false.spotless:checkpassed locally on the first commit and has not been rerun after the rename. Thesuite has not been run locally yet.