Skip to content

feat: add DataFrame helpers getFallbackReasons and isFullyAccelerated to ExtendedExplainInfo - #7681

Open
comphead wants to merge 2 commits into
apache:mainfrom
comphead:df-fallback-helpers
Open

comphead wants to merge 2 commits into
apache:mainfrom
comphead:df-fallback-helpers

Conversation

@comphead

@comphead comphead commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No issue was filed for this small addition.

Rationale for this change

Checking whether Comet ran a query natively currently takes
new ExtendedExplainInfo().getFallbackReasons(df.queryExecution.executedPlan) plus a separate check
that Comet is enabled at all. About 20 suites repeat the first part, and applications that embed
Comet need the same check. A plan-only check is also misleading on its own: when Comet disables
itself (for example without off-heap memory or without the Comet shuffle manager), the plan has no
fallback reasons because Comet never tagged it, so "no fallback reasons" does not mean "accelerated".

What changes are included in this PR?

Two methods on ExtendedExplainInfo, the class users already reach through
spark.sql.extendedExplainProviders:

  • getFallbackReasons(df: DataFrame): Seq[String] returns the fallback reasons of the DataFrame's
    executed plan. It delegates to the existing getFallbackReasons(plan: SparkPlan).
  • isFullyAccelerated(df: DataFrame): Boolean is true when Comet is enabled for the DataFrame's
    session and the plan has no fallback reasons. A plan with partial fallback is not fully
    accelerated. "Enabled" uses CometSparkSessionExtensions.isCometLoaded,
    the same check the Comet planner rules make, rather than spark.comet.enabled alone.

Neither method runs the query. For an adaptive query that has not run yet, the executed plan is the
initial plan. The scaladoc and the user guide say to run the query first to get the final plan. The user
guide in understanding-comet-plans.md gets a short subsection, "Checking That a Query Is Fully
Accelerated", that describes the check.

The session conf is read through df.queryExecution.sparkSession, because on Spark 4.x DataFrame
is the API Dataset, which has queryExecution but not sessionState.

How are these changes tested?

A new test in CometExecSuite, "getFallbackReasons and isFullyAccelerated for a DataFrame", covers:

  • A query with a known fallback (sampling with replacement): getFallbackReasons(df) contains the
    reason, matches the plan overload, and isFullyAccelerated is false.
  • A fully native filter and project: no reasons, and isFullyAccelerated is true.
  • The same query with spark.comet.enabled=false: isFullyAccelerated is false.

spotless:check passed locally on the first commit and has not been rerun after the rename. The
suite has not been run locally yet.

… ExtendedExplainInfo

`getFallbackReasons(df)` returns the fallback reasons of a DataFrame's executed
plan. `planAccelerated(df)` is true when Comet is enabled for the session, using
the same check as the Comet planner rules, and the plan has no fallback reasons.
@github-actions github-actions Bot added the enhancement New feature or request label Nov 18, 2026
@comphead
comphead requested a review from andygrove November 18, 2026 20:51
* manager is not set, even if `spark.comet.enabled` is true. Like `getFallbackReasons(df)`,
* this does not run the query.
*/
def planAccelerated(df: DataFrame): Boolean = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the name is misleading, IMO. A plan can still be accelerated and have partial fallback. Maybe this should be planFullyAccelerated or noFallbacks or something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thats always tricky part :)

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead

The old name read as "some of the plan is accelerated". The method returns
true only when Comet is enabled and the plan has no fallback reasons, so a
plan with partial fallback is not fully accelerated. Add a user guide
subsection that describes the check.
@comphead comphead changed the title feat: add DataFrame helpers getFallbackReasons and planAccelerated to ExtendedExplainInfo feat: add DataFrame helpers getFallbackReasons and isFullyAccelerated to ExtendedExplainInfo Nov 19, 2026
@comphead
comphead added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants