Skip to content

Implement Spark-compatible schema_of_variant #278

Open

Description

@andygrove

Part of #273 and apache/datafusion-comet#7300.

What the function does

schema_of_variant(v) returns a STRING describing the schema of a single VARIANT value. The format is Spark's
SQL type syntax, except that objects print as OBJECT<...> instead of STRUCT<...>. Variant objects are unordered
bags of fields, unlike structs. The argument must be VARIANT and the result uses the default string type (and
collation). SQL NULL input returns NULL through StaticInvoke null propagation. Added in Spark 4.0.0.

Type mapping (SchemaOfVariant.schemaOf):

Variant type Result
null VOID
boolean BOOLEAN
INT1, INT2, INT4, INT8 BIGINT
string STRING
double DOUBLE
float FLOAT
decimal DECIMAL(p,s), from the value after trailing zeros are stripped
date, timestamp, timestamp_ntz DATE, TIMESTAMP, TIMESTAMP_NTZ
binary BINARY
uuid UUID
object OBJECT<name: type, ...> in key order
array ARRAY<merged element type>

Array elements are merged with JsonInferSchema.compatibleType(t1, t2, VariantType), so incompatible types become
VARIANT.

Examples

SELECT schema_of_variant(parse_json('null'));
-- VOID
SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
-- ARRAY<OBJECT<a: BIGINT, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('1.0')), schema_of_variant(parse_json('10.10')), schema_of_variant(parse_json('-0.0'));
-- DECIMAL(1,0)  DECIMAL(3,1)  DECIMAL(1,0)
SELECT schema_of_variant(parse_json('9223372036854775808'));
-- DECIMAL(19,0)
SELECT schema_of_variant(parse_json('9999999999999999999.99999999999999999999'));
-- DOUBLE
SELECT schema_of_variant(parse_json('"2000-01-01"'));
-- STRING
SELECT schema_of_variant(parse_json('[]'));
-- ARRAY<VOID>
SELECT schema_of_variant(parse_json('[null, 1, 1.1]'));
-- ARRAY<DECIMAL(21,1)>
SELECT schema_of_variant(parse_json('[1.1, 11111111111111111111111111111111111111]'));
-- ARRAY<DOUBLE>
SELECT schema_of_variant(parse_json('[1, "1"]'));
-- ARRAY<VARIANT>
SELECT schema_of_variant(parse_json('[{"c": ""}, {"a": null}, {"b": 1}]'));
-- ARRAY<OBJECT<a: VOID, b: BIGINT, c: STRING>>
SELECT schema_of_variant(parse_json('[{"a": 1, "b": null}, {"b": true, "a": 1E0}]'));
-- ARRAY<OBJECT<a: DOUBLE, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('{"STRUCT": {"!special!": true}}'));
-- OBJECT<STRUCT: OBJECT<`!special!`: BOOLEAN>>

These come from the ExpressionDescription, VariantEndToEndSuite and VariantSuite.

Edge cases to test

  • Merge table from VariantExpressionSuite."schema_of_variant - schema merge":

    Pair Result
    BIGINT/DOUBLE DOUBLE
    BIGINT/FLOAT FLOAT
    BIGINT/DECIMAL(6,3) DECIMAL(23,3)
    DOUBLE/FLOAT DOUBLE
    DOUBLE/DECIMAL DOUBLE
    FLOAT/DECIMAL DOUBLE
    DATE/TIMESTAMP TIMESTAMP
    DATE/TIMESTAMP_NTZ TIMESTAMP_NTZ
    TIMESTAMP/TIMESTAMP_NTZ TIMESTAMP
    ARRAY<BIGINT>/ARRAY<DOUBLE> ARRAY<DOUBLE>
    OBJECT<a: STRING>/OBJECT<a: BOOLEAN, b: BIGINT> OBJECT<a: VARIANT, b: BIGINT>

    VOID merged with anything gives the other type, and every other pair gives VARIANT, including
    STRING/BIGINT and BINARY/STRING.

  • Decimal merging: the scale is max(s1, s2) and the integer range is max(p - s). When the total exceeds 38 the
    result is DOUBLE. An integral type is widened to DECIMAL(20,0) first ([null, 1, 1.0] gives
    DECIMAL(20,0), and [123456.789, 123.456789] gives DECIMAL(12,6)).

  • Decimal precision comes from stripTrailingZeros, with a negative scale normalized to 0: 1.00 gives
    DECIMAL(1,0) and 10.00 gives DECIMAL(2,0). It does not come from the stored scale.

  • Field names are quoted with backticks unless they match [a-zA-Z_][a-zA-Z0-9_]*, and embedded backticks are
    doubled (OBJECT<`it's`: ...>). Keywords like STRUCT are not quoted.

  • Struct merging inside arrays first goes through TypeCoercion.findTightestCommonType. For structs of equal
    length, that compares names with spark.sql.caseSensitive (default false) and keeps the first name. Only if it
    fails does Spark fall back to a case-sensitive sorted merge. By code reading, [{"a":1},{"A":2}] gives
    ARRAY<OBJECT<a: BIGINT>> by default. Verify this before asserting it.

  • Object fields must be strictly increasing by Java String.compareTo (UTF-16 order). Otherwise Spark throws
    MALFORMED_VARIANT, and duplicates hit the same check. Variants written by other tools that sort by UTF-8 bytes
    can trip this.

  • An unknown primitive type id raises UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT (id 25 in the suite).

  • UUID shows as UUID, and merging it with a variant null keeps UUID (ARRAY<UUID>). These values can only come
    from external data.

  • Test NULL input, column input, and deep nesting.

Differences across Spark versions

  • 3.x has no VARIANT.
  • No logic changes between 4.0.4, 4.1.3 and 4.2.0. 4.2 adds GEOMETRY/GEOGRAPHY cases to
    findTightestCommonType, but variants never produce those types.

Implementation notes

  • Spark: SchemaOfVariant (schemaOf, mergeSchema, printSchema) in
    sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala.
    The merge rules are JsonInferSchema.compatibleType in .../catalyst/json/JsonInferSchema.scala and
    TypeCoercion.findTightestCommonType / TypeCoercionHelper.findTypeForComplex in .../catalyst/analysis/.
    The printer is DataType.sql, with QuotingUtils.quoteIfNeeded for field names.
  • DataFusion core and arrow-rs have nothing equivalent. Reuse parquet_variant::Variant to walk values and write
    the Spark type lattice and printer directly. Share them with schema_of_variant_agg, and with
    schema_of_json / from_json's merge logic if they land.
  • This depends on the VARIANT Arrow representation in the parse_json issue.
  • [EPIC] Native Variant Support datafusion-comet#5438 tracks native Variant support in Comet.
  • Put it in crates/spark/src/function/variant/. Add crates/sqllogictest/test_files/variant/schema_of_variant.slt
    and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions