Skip to content

Move SparkOptions into datafusion-spark as a config extension #92

Closed

Description

@andygrove

This is step 1 of apache/datafusion#27342, done in this repository.

SparkOptions (datafusion.spark.*) and enable_ansi_mode are defined in datafusion-common, so every new Spark option needs a DataFusion PR and then a DataFusion release. Several functions need new options (see #16):

Proposal

  • Define SparkOptions here as a ConfigExtension with the prefix spark, so that keys mirror Spark's names (spark.map_key_dedup_policy, later spark.sql.legacy.*).
  • Register it in SessionStateBuilderSpark::with_spark_features().
  • Have functions read it from config_options.extensions, falling back to the defaults when it isn't registered. This builds against both DataFusion 56 and main.
  • Move map_from_arrays, map_from_entries and str_to_map to the new spark.map_key_dedup_policy key. Core's datafusion.spark.map_key_dedup_policy stays on branch-56 until moved SparkOptions to datafusion-spark datafusion#27371 removes it for DataFusion 57.

apache/datafusion#27371 implemented this in DataFusion before the crate moved here. Its author has been invited to port the datafusion-spark half to this repository.

Notes for users of these options

  • A config extension reaches scalar functions through ScalarFunctionArgs::config_options at execution time.
  • It does not reach return_field_from_args, coerce_types or aggregate accumulators. Functions whose return type or input types depend on an option need it when they are constructed, refreshed in with_updated_config.
  • Comet builds ScalarFunctionExpr with ConfigOptions::default(), so it sees only the defaults. Every flag therefore also needs a direct way in, through a constructor argument or a public function.

enable_ansi_mode stays in core until apache/datafusion#23929 decides whether core will read it.

Activity

  1. added a commit that references this issue on Nov 18, 2026
    4832699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions