You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
with_config() called after with_spark_features() drops the Spark configuration extension #300
Follow-up to the review discussion in #98. That PR documents the required
call order (with_config() before with_spark_features()); this issue tracks
addressing the underlying configuration-loss behavior separately.
Description
SessionStateBuilderSpark::with_spark_features() registers a SparkOptions configuration extension. However, calling SessionStateBuilder::with_config() afterward replaces the entire session configuration, removing that extension unless the replacement configuration also contains it.
This makes Spark configuration registration dependent on builder call order. The current documentation recommends calling with_config() before with_spark_features(), but this issue tracks a longer-term solution.
Reproduction
use datafusion::execution::SessionStateBuilder;use datafusion::prelude::SessionConfig;use datafusion_spark::{SessionStateBuilderSpark,SparkOptions};let state = SessionStateBuilder::new().with_spark_features().with_config(SessionConfig::new()).build();// The Spark configuration extension has been dropped.assert!(
state
.config_options().extensions
.get::<SparkOptions>().is_none());
Current Behavior
Spark functions remain registered, but the SparkOptions extension is absent. As a result, SET spark.map_key_dedup_policy = 'LAST_WIN' cannot configure the extension, and any previously configured extension values are lost.
Desired Outcome
Provide a clear, safe way to combine with_config() and with_spark_features() without silently losing Spark configuration.
The solution should define how existing Spark options interact with a replacement configuration, including which values take precedence when both configurations contain SparkOptions. It should respect the general replacement semantics of with_config() rather than implicitly changing them without consideration.
Current Workaround
Call with_config() first:
let state = SessionStateBuilder::new().with_config(SessionConfig::new()).with_spark_features().build();
To preserve custom Spark options, include them in the configuration passed to with_config() before calling with_spark_features().
Looking into this, SessionStateBuilder::with_config() replaces the entire SessionConfig, so this behavior can affect other configuration extensions too, not just SparkOptions. I understand that replacement may be intentional.
Should the solution stay specific to Spark, or should we discuss a broader extension-preservation mechanism in DataFusion core? For now, the PR documents calling with_config() before with_spark_features().
Activity
rizzabhdwaj commentedon Nov 16, 2026
Looking into this, SessionStateBuilder::with_config() replaces the entire SessionConfig, so this behavior can affect other configuration extensions too, not just SparkOptions. I understand that replacement may be intentional.
Should the solution stay specific to Spark, or should we discuss a broader extension-preservation mechanism in DataFusion core? For now, the PR documents calling with_config() before with_spark_features().
andygrove commentedon Nov 16, 2026
hmm, maybe this is fine and just needs to be documented well