You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Move SparkOptions into datafusion-spark as a config extension #92
SparkOptions (datafusion.spark.*) and enable_ansi_mode are defined in datafusion-common, so every new Spark option needs a DataFusion PR and then a DataFusion release. Several functions need new options (see #16):
Define SparkOptions here as a ConfigExtension with the prefix spark, so that keys mirror Spark's names (spark.map_key_dedup_policy, later spark.sql.legacy.*).
Register it in SessionStateBuilderSpark::with_spark_features().
Have functions read it from config_options.extensions, falling back to the defaults when it isn't registered. This builds against both DataFusion 56 and main.
Move map_from_arrays, map_from_entries and str_to_map to the new spark.map_key_dedup_policy key. Core's datafusion.spark.map_key_dedup_policy stays on branch-56 until moved SparkOptions to datafusion-spark datafusion#27371 removes it for DataFusion 57.
apache/datafusion#27371 implemented this in DataFusion before the crate moved here. Its author has been invited to port the datafusion-spark half to this repository.
Notes for users of these options
A config extension reaches scalar functions through ScalarFunctionArgs::config_options at execution time.
It does not reach return_field_from_args, coerce_types or aggregate accumulators. Functions whose return type or input types depend on an option need it when they are constructed, refreshed in with_updated_config.
Comet builds ScalarFunctionExpr with ConfigOptions::default(), so it sees only the defaults. Every flag therefore also needs a direct way in, through a constructor argument or a public function.
enable_ansi_mode stays in core until apache/datafusion#23929 decides whether core will read it.
Activity
fix(string): match Spark 4 encode charset and error behavior (#46) (#116
feat(spark): add Spark-compatible array_insert function (#70)
feat(spark): add SparkOptions config extension (apache#98)