Repository navigation
Description
This implements the design proposed in #16.
Design
-
Expressions take named behavior flags, never a Spark version. For example,
SparkArrayInsert::with_legacy_negative_index(feat(spark): add Spark-compatible array_insert function #70),legacy_size_of_null([Bug] size returns -1 for NULL when ANSI mode is on, Spark returns NULL #90), ordecimal_zero_any_exponentin the cast port ([EPIC] Port Comet's Spark-compatible CAST #88). -
One module derives each flag's default from a full
major.minor.patchSpark version. Spark backports behavior changes to patch releases, so a3.4–4.2enum is not precise enough:spark.sql.legacy.negativeIndexInArrayInsertwas added in 3.4.2.- SPARK-60118 (casting a zero string with any exponent to decimal) first shipped in 4.0.5, 4.1.4 and 4.2.1.
Each entry cites its SPARK JIRA and the first release in each line that includes it, like Comet's
SparkFixes.scala. -
Where Spark has a setting for a behavior, the flag mirrors that setting. It becomes a key in
SparkOptions(Move SparkOptions into datafusion-spark as a config extension #92) whose default comes from the version, andSEToverrides it. For example,spark.sql.legacy.javaCharsetsdid not exist before 4.0, so its effective default istruefor 3.x andfalsefor 4.x ([Bug] encode accepts charsets and unmappable characters that Spark 4 rejects #46). -
Two ways in:
- In a DataFusion session, the UDF resolves its flags from
spark.versionplus any explicit setting, ininvoke_with_args, or inwith_updated_configwhen the return type depends on the flag. - Comet passes the flags directly, because its JVM side knows the exact running version and the real
SQLConf.
- In a DataFusion session, the UDF resolves its flags from
Tasks
-
SparkVersion { major, minor, patch }, parsed from a string such as4.0.5, defaulting to the crate's baseline (4.2.0) -
spark.versioninSparkOptions, which depends on Move SparkOptions into datafusion-spark as a config extension #92 - The flag-derivation module, with unit tests for each entry
- Document the convention for adding a version-dependent behavior in
CONTRIBUTING.mdor the audit skill - Use
SET spark.version = ...in sqllogictest files to cover non-default versions, which partly addresses Support Spark-version-specific expectations in datafusion-spark sqllogictest files #24
Behaviors waiting on this
| Function | Flag | Issue |
|---|---|---|
size |
spark.sql.legacy.sizeOfNull (with ANSI) |
#90 |
elt |
spark.sql.function.eltOutputAsString |
#91 |
encode |
spark.sql.legacy.javaCharsets, spark.sql.legacy.codingErrorAction (3.x vs 4.x) |
#46 |
hash, xxhash64 |
spark.sql.legacy.allowHashOnMapType |
#69 |
next_day |
error class differs between 3.5 and 4.x, with no setting | #25 |
array_insert |
spark.sql.legacy.negativeIndexInArrayInsert |
#70 |
format_string, concat, array() |
legacy settings whose defaults already match | #94 |
CAST |
is_spark4_plus, decimal_zero_any_exponent, decimal_skip_integral_digit_limit |
#88 |
Activity
fix(string): match Spark 4 encode charset and error behavior (#46) (#116