Skip to content

with_config() called after with_spark_features() drops the Spark configuration extension #300

Open

Description

@rizzabhdwaj

Background

Follow-up to the review discussion in #98. That PR documents the required
call order (with_config() before with_spark_features()); this issue tracks
addressing the underlying configuration-loss behavior separately.

Description

SessionStateBuilderSpark::with_spark_features() registers a SparkOptions configuration extension. However, calling SessionStateBuilder::with_config() afterward replaces the entire session configuration, removing that extension unless the replacement configuration also contains it.

This makes Spark configuration registration dependent on builder call order. The current documentation recommends calling with_config() before with_spark_features(), but this issue tracks a longer-term solution.

Reproduction

use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::SessionConfig;
use datafusion_spark::{SessionStateBuilderSpark, SparkOptions};

let state = SessionStateBuilder::new()
    .with_spark_features()
    .with_config(SessionConfig::new())
    .build();

// The Spark configuration extension has been dropped.
assert!(
    state
        .config_options()
        .extensions
        .get::<SparkOptions>()
        .is_none()
);

Current Behavior

Spark functions remain registered, but the SparkOptions extension is absent. As a result, SET spark.map_key_dedup_policy = 'LAST_WIN' cannot configure the extension, and any previously configured extension values are lost.

Desired Outcome

Provide a clear, safe way to combine with_config() and with_spark_features() without silently losing Spark configuration.

The solution should define how existing Spark options interact with a replacement configuration, including which values take precedence when both configurations contain SparkOptions. It should respect the general replacement semantics of with_config() rather than implicitly changing them without consideration.

Current Workaround

Call with_config() first:

let state = SessionStateBuilder::new()
    .with_config(SessionConfig::new())
    .with_spark_features()
    .build();

To preserve custom Spark options, include them in the configuration passed to with_config() before calling with_spark_features().

Activity

  1. rizzabhdwaj commented on Nov 16, 2026

    @rizzabhdwaj
    ContributorAuthor

    Looking into this, SessionStateBuilder::with_config() replaces the entire SessionConfig, so this behavior can affect other configuration extensions too, not just SparkOptions. I understand that replacement may be intentional.

    Should the solution stay specific to Spark, or should we discuss a broader extension-preservation mechanism in DataFusion core? For now, the PR documents calling with_config() before with_spark_features().

  2. andygrove commented on Nov 16, 2026

    @andygrove
    Member

    hmm, maybe this is fine and just needs to be documented well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions