You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Introduce version-specific behavior in Spark expressions #16
Is your feature request related to a problem or challenge?
Many Spark expressions have different behavior across Spark versions. This is especially the case when comparing Spark 3.x and 4.x where there are many breaking changes.
I think it is important to start addressing this in the Spark-compatible expressions in DataFusion.
fwiw, the approach we take in Comet is that each expression has a getSupportLevel method that can return Compatible, Incompatible(reason), or Unsupported. These methods are context-aware based on Spark version, Spark configuration (e.g. ANSI mode enabled/disabled), and the specific arguments being passed to the expression. Here is an example:
overridedefgetSupportLevel(expr: Reverse):SupportLevel= {
if (containsBinary(expr.child.dataType)) {
Incompatible(Some("reverse on array containing binary is not supported"))
} else {
Compatible(None)
}
}
We also have a shim layer where we can implement different code per Spark version. This was necessary for Comet because there are API changes in Spark between versions and we need to compile per-version. It is simpler for DataFusion because we can just pass the Spark version (or some other flags) into the expression constructor.
@andygrove This makes sense to me and would be valuable. I also wonder whether we should make the relevant Spark configuration available to all UDFs so they can make the same kind of context-aware compatibility decisions.
More broadly, I like the idea of a getSupportLevel mechanism where each Spark-compatible expression can report whether it is compatible, incompatible with a clear reason, or simply unsupported, based on Spark version, Spark configuration, and the specific arguments involved.
It is also worth thinking about whether compatibility should always be strictly version-locked. If a newer Spark version adds functionality to the same UDF, we might want to allow older versions to benefit from that functionality as well when it is safe to do so. That could even be exposed as a configuration option, letting users choose between strict version fidelity and newer function behavior. For example, a user targeting an older Spark version could still run a Spark SQL query that uses newer behavior for an existing function, rather than being restricted to the exact historical semantics of that older version.
I checked which functions in this crate need Spark-version or legacy-setting behavior today. To find them, I took the spark.sql.legacy.* and spark.sql.function.* settings that Spark 4.1.1's expressions and type coercion read, matched them against the functions here, and went through the open issues.
Inventory
Function
What decides it
Status here
Issue
size
spark.sql.legacy.sizeOfNull and ANSI mode
Wrong whenever ANSI mode is on (returns -1, Spark returns NULL)
ANSI mode is the most common of these, and its default depends on the version too: off in 3.x, on in 4.x.
Proposed design
Expressions take named behavior flags, never a version. A single module derives each flag's default from a full major.minor.patch Spark version. Where Spark has a setting for the behavior, the flag mirrors it, and SET can override it. Details are in #93.
Spark backports behavior changes to patch releases, so a version enum is not precise enough. negativeIndexInArrayInsert arrived in 3.4.2, and Comet's SparkFixes gates SPARK-60118 on 4.0.5, 4.1.4 and 4.2.1. Comet already works this way: its cast receives one flag per behavior, and the JVM decides each value from the running version or the real SQLConf.
Two practical constraints shape where the flags go:
Return types and type checks never see the config.return_field_from_args, coerce_types and aggregate accumulators don't receive it. About half of the cases above are decided there: the nullability of size, the return type of elt, concat and array(), and the map check in hash. Those flags have to be set when the function is constructed and refreshed in with_updated_config.
Comet doesn't pass a session config. It builds every ScalarFunctionExpr with ConfigOptions::default(), so a flag that is only read from the config never reaches Comet. Each flag also needs a direct way in, through a constructor argument or a public function. This is the same split as EvalMode in feat(spark): add EvalMode and use it for parse_url and try_parse_url #89, which has spark_parse_url(args, EvalMode) and EvalMode::from_config.
The settings themselves need SparkOptions to move into this crate as a config extension (#92, step 1 of apache/datafusion#27342).
Activity
andygrove commentedon Apr 17, 2026
@shehabgamin Would be good to get your thoughts on this
shehabgamin commentedon Apr 19, 2026
@andygrove This makes sense to me and would be valuable. I also wonder whether we should make the relevant Spark configuration available to all UDFs so they can make the same kind of context-aware compatibility decisions.
More broadly, I like the idea of a
getSupportLevelmechanism where each Spark-compatible expression can report whether it is compatible, incompatible with a clear reason, or simply unsupported, based on Spark version, Spark configuration, and the specific arguments involved.It is also worth thinking about whether compatibility should always be strictly version-locked. If a newer Spark version adds functionality to the same UDF, we might want to allow older versions to benefit from that functionality as well when it is safe to do so. That could even be exposed as a configuration option, letting users choose between strict version fidelity and newer function behavior. For example, a user targeting an older Spark version could still run a Spark SQL query that uses newer behavior for an existing function, rather than being restricted to the exact historical semantics of that older version.
andygrove commentedon Nov 13, 2026
I checked which functions in this crate need Spark-version or legacy-setting behavior today. To find them, I took the
spark.sql.legacy.*andspark.sql.function.*settings that Spark 4.1.1's expressions and type coercion read, matched them against the functions here, and went through the open issues.Inventory
sizespark.sql.legacy.sizeOfNulland ANSI modeeltspark.sql.function.eltOutputAsStringencodespark.sql.legacy.javaCharsets,spark.sql.legacy.codingErrorAction(added in 4.0.0)hash,xxhash64spark.sql.legacy.allowHashOnMapTypenext_dayarray_insertspark.sql.legacy.negativeIndexInArrayInsert(added in 3.4.2)map_from_arrays,map_from_entries,str_to_mapspark.sql.mapKeyDedupPolicydatafusion.spark.map_key_dedup_policyformat_string,concat,array()CASTis_spark4_plus,decimal_zero_any_exponent,decimal_skip_integral_digit_limitin CometANSI mode is the most common of these, and its default depends on the version too: off in 3.x, on in 4.x.
Proposed design
Expressions take named behavior flags, never a version. A single module derives each flag's default from a full
major.minor.patchSpark version. Where Spark has a setting for the behavior, the flag mirrors it, andSETcan override it. Details are in #93.Spark backports behavior changes to patch releases, so a version enum is not precise enough.
negativeIndexInArrayInsertarrived in 3.4.2, and Comet'sSparkFixesgates SPARK-60118 on 4.0.5, 4.1.4 and 4.2.1. Comet already works this way: its cast receives one flag per behavior, and the JVM decides each value from the running version or the realSQLConf.Two practical constraints shape where the flags go:
return_field_from_args,coerce_typesand aggregate accumulators don't receive it. About half of the cases above are decided there: the nullability ofsize, the return type ofelt,concatandarray(), and the map check inhash. Those flags have to be set when the function is constructed and refreshed inwith_updated_config.ScalarFunctionExprwithConfigOptions::default(), so a flag that is only read from the config never reaches Comet. Each flag also needs a direct way in, through a constructor argument or a public function. This is the same split asEvalModein feat(spark): add EvalMode and use it for parse_url and try_parse_url #89, which hasspark_parse_url(args, EvalMode)andEvalMode::from_config.The settings themselves need
SparkOptionsto move into this crate as a config extension (#92, step 1 of apache/datafusion#27342).