Repository navigation
fix: preserve current SQL configs when captured keys are unset - #7348
Conversation
sunchao
left a comment
There was a problem hiding this comment.
The execution-ID check preserves the active SQL configuration when a captured key has since been unset, and the plain RDD path still uses its captured settings. I ran all three tests in CometSessionExtensionsSuite on f3408660 with Spark 3.5.9 and 4.1.3. Both runs passed. As a control, the new test with the pre-fix implementation returned ("stale", "missing") where it expected ("missing", "current"). Upstream CI is awaiting workflow authorization. LGTM.
andygrove
left a comment
There was a problem hiding this comment.
Thanks, this matches what I proposed in #7312. Skipping withConf when the execution id is set is exact, since every Spark path that sets the id also calls withSQLConfPropagated, and the test builds the capture under an isolated SQLConf so the stale case doesn't pass vacuously.
Nit, optional: the withConf scaladoc still describes the skip as "Spark passed the task the session's configs (it passes all or none)". Could it mention the execution id too, so the next reader knows why the id check comes before the key check?
Nit, optional: spark.conf.unset(staleKey) in the test is a no-op, because staleKey only exists in the isolated capture and was never set on the session. Could it be dropped, or the test name reworded to say the key is absent from the session?
Which issue does this PR close?
Closes #7312.
Rationale for this change
A captured Comet setting may be unset on the session after a plan is cached. Inside a later SQL execution Spark propagates the current SQL configuration to the task, but if none of the captured keys remains set,
CapturedSQLConf.withConfmistakes the absence of those keys for an unpropagated configuration and reinstalls the stale capture.What changes are included in this PR?
As proposed in #7312, skip the captured configuration when the task has a SQL execution id. The existing captured-key check remains for callers that propagate SQL configuration without an execution id, and non-SQL tasks continue to use the capture when needed.
A regression test builds a capture with only one key, unsets that key, and checks a second session setting inside an actual SQL execution task. It also checks the non-SQL fallback and an existing task property without a SQL execution id.
How are these changes tested?
(stale, missing)instead of(missing, current).make format,makeandgit diff --checkpassed in an isolated Linux environment using JDK 17.The remaining Spark profiles and upstream Spark SQL suites were not run locally.