Skip to content

fix: preserve current SQL configs when captured keys are unset - #7348

Merged
andygrove merged 1 commit into
apache:mainfrom
dajiaohuang:fix/comet-7312-sql-conf-20261030
Nov 9, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
dajiaohuang:fix/comet-7312-sql-conf-20261030

Conversation

@dajiaohuang

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7312.

Rationale for this change

A captured Comet setting may be unset on the session after a plan is cached. Inside a later SQL execution Spark propagates the current SQL configuration to the task, but if none of the captured keys remains set, CapturedSQLConf.withConf mistakes the absence of those keys for an unpropagated configuration and reinstalls the stale capture.

What changes are included in this PR?

As proposed in #7312, skip the captured configuration when the task has a SQL execution id. The existing captured-key check remains for callers that propagate SQL configuration without an execution id, and non-SQL tasks continue to use the capture when needed.

A regression test builds a capture with only one key, unsets that key, and checks a second session setting inside an actual SQL execution task. It also checks the non-SQL fallback and an existing task property without a SQL execution id.

How are these changes tested?

  • Before the fix the regression returned (stale, missing) instead of (missing, current).
  • The regression and its controls passed under Spark 4.1 and Spark 3.5, with one test executed on each profile.
  • make format, make and git diff --check passed in an isolated Linux environment using JDK 17.

The remaining Spark profiles and upstream Spark SQL suites were not run locally.

@github-actions github-actions Bot added the bug Something isn't working label Oct 30, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The execution-ID check preserves the active SQL configuration when a captured key has since been unset, and the plain RDD path still uses its captured settings. I ran all three tests in CometSessionExtensionsSuite on f3408660 with Spark 3.5.9 and 4.1.3. Both runs passed. As a control, the new test with the pre-fix implementation returned ("stale", "missing") where it expected ("missing", "current"). Upstream CI is awaiting workflow authorization. LGTM.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this matches what I proposed in #7312. Skipping withConf when the execution id is set is exact, since every Spark path that sets the id also calls withSQLConfPropagated, and the test builds the capture under an isolated SQLConf so the stale case doesn't pass vacuously.

Nit, optional: the withConf scaladoc still describes the skip as "Spark passed the task the session's configs (it passes all or none)". Could it mention the execution id too, so the next reader knows why the id check comes before the key check?

Nit, optional: spark.conf.unset(staleKey) in the test is a no-op, because staleKey only exists in the isolated capture and was never set on the session. Could it be dropped, or the test name reworded to say the key is absent from the session?

@andygrove
andygrove added this pull request to the merge queue Nov 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 7, 2026
@andygrove
andygrove added this pull request to the merge queue Nov 9, 2026
Merged via the queue into apache:main with commit 8386a34 Nov 9, 2026
45 checks passed
andygrove added a commit that referenced this pull request Nov 19, 2026
#7643)

(cherry picked from commit 8386a34)

Co-authored-by: Wu Shuwen <mikewushuwen@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CapturedSQLConf applies a stale capture inside a SQL execution when every captured key was unset

3 participants