Repository navigation
Description
Part of #273 and apache/datafusion-comet#7300.
What the function does
schema_of_variant(v) returns a STRING describing the schema of a single VARIANT value. The format is Spark's
SQL type syntax, except that objects print as OBJECT<...> instead of STRUCT<...>. Variant objects are unordered
bags of fields, unlike structs. The argument must be VARIANT and the result uses the default string type (and
collation). SQL NULL input returns NULL through StaticInvoke null propagation. Added in Spark 4.0.0.
Type mapping (SchemaOfVariant.schemaOf):
| Variant type | Result |
|---|---|
| null | VOID |
| boolean | BOOLEAN |
| INT1, INT2, INT4, INT8 | BIGINT |
| string | STRING |
| double | DOUBLE |
| float | FLOAT |
| decimal | DECIMAL(p,s), from the value after trailing zeros are stripped |
| date, timestamp, timestamp_ntz | DATE, TIMESTAMP, TIMESTAMP_NTZ |
| binary | BINARY |
| uuid | UUID |
| object | OBJECT<name: type, ...> in key order |
| array | ARRAY<merged element type> |
Array elements are merged with JsonInferSchema.compatibleType(t1, t2, VariantType), so incompatible types become
VARIANT.
Examples
SELECT schema_of_variant(parse_json('null'));
-- VOID
SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
-- ARRAY<OBJECT<a: BIGINT, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('1.0')), schema_of_variant(parse_json('10.10')), schema_of_variant(parse_json('-0.0'));
-- DECIMAL(1,0) DECIMAL(3,1) DECIMAL(1,0)
SELECT schema_of_variant(parse_json('9223372036854775808'));
-- DECIMAL(19,0)
SELECT schema_of_variant(parse_json('9999999999999999999.99999999999999999999'));
-- DOUBLE
SELECT schema_of_variant(parse_json('"2000-01-01"'));
-- STRING
SELECT schema_of_variant(parse_json('[]'));
-- ARRAY<VOID>
SELECT schema_of_variant(parse_json('[null, 1, 1.1]'));
-- ARRAY<DECIMAL(21,1)>
SELECT schema_of_variant(parse_json('[1.1, 11111111111111111111111111111111111111]'));
-- ARRAY<DOUBLE>
SELECT schema_of_variant(parse_json('[1, "1"]'));
-- ARRAY<VARIANT>
SELECT schema_of_variant(parse_json('[{"c": ""}, {"a": null}, {"b": 1}]'));
-- ARRAY<OBJECT<a: VOID, b: BIGINT, c: STRING>>
SELECT schema_of_variant(parse_json('[{"a": 1, "b": null}, {"b": true, "a": 1E0}]'));
-- ARRAY<OBJECT<a: DOUBLE, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('{"STRUCT": {"!special!": true}}'));
-- OBJECT<STRUCT: OBJECT<`!special!`: BOOLEAN>>These come from the ExpressionDescription, VariantEndToEndSuite and VariantSuite.
Edge cases to test
-
Merge table from
VariantExpressionSuite."schema_of_variant - schema merge":Pair Result BIGINT/DOUBLEDOUBLEBIGINT/FLOATFLOATBIGINT/DECIMAL(6,3)DECIMAL(23,3)DOUBLE/FLOATDOUBLEDOUBLE/DECIMALDOUBLEFLOAT/DECIMALDOUBLEDATE/TIMESTAMPTIMESTAMPDATE/TIMESTAMP_NTZTIMESTAMP_NTZTIMESTAMP/TIMESTAMP_NTZTIMESTAMPARRAY<BIGINT>/ARRAY<DOUBLE>ARRAY<DOUBLE>OBJECT<a: STRING>/OBJECT<a: BOOLEAN, b: BIGINT>OBJECT<a: VARIANT, b: BIGINT>VOIDmerged with anything gives the other type, and every other pair givesVARIANT, including
STRING/BIGINTandBINARY/STRING. -
Decimal merging: the scale is
max(s1, s2)and the integer range ismax(p - s). When the total exceeds 38 the
result isDOUBLE. An integral type is widened toDECIMAL(20,0)first ([null, 1, 1.0]gives
DECIMAL(20,0), and[123456.789, 123.456789]givesDECIMAL(12,6)). -
Decimal precision comes from
stripTrailingZeros, with a negative scale normalized to 0:1.00gives
DECIMAL(1,0)and10.00givesDECIMAL(2,0). It does not come from the stored scale. -
Field names are quoted with backticks unless they match
[a-zA-Z_][a-zA-Z0-9_]*, and embedded backticks are
doubled (OBJECT<`it's`: ...>). Keywords likeSTRUCTare not quoted. -
Struct merging inside arrays first goes through
TypeCoercion.findTightestCommonType. For structs of equal
length, that compares names withspark.sql.caseSensitive(default false) and keeps the first name. Only if it
fails does Spark fall back to a case-sensitive sorted merge. By code reading,[{"a":1},{"A":2}]gives
ARRAY<OBJECT<a: BIGINT>>by default. Verify this before asserting it. -
Object fields must be strictly increasing by Java
String.compareTo(UTF-16 order). Otherwise Spark throws
MALFORMED_VARIANT, and duplicates hit the same check. Variants written by other tools that sort by UTF-8 bytes
can trip this. -
An unknown primitive type id raises
UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT(id 25 in the suite). -
UUID shows as
UUID, and merging it with a variant null keepsUUID(ARRAY<UUID>). These values can only come
from external data. -
Test NULL input, column input, and deep nesting.
Differences across Spark versions
- 3.x has no VARIANT.
- No logic changes between 4.0.4, 4.1.3 and 4.2.0. 4.2 adds
GEOMETRY/GEOGRAPHYcases to
findTightestCommonType, but variants never produce those types.
Implementation notes
- Spark:
SchemaOfVariant(schemaOf,mergeSchema,printSchema) in
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala.
The merge rules areJsonInferSchema.compatibleTypein.../catalyst/json/JsonInferSchema.scalaand
TypeCoercion.findTightestCommonType/TypeCoercionHelper.findTypeForComplexin.../catalyst/analysis/.
The printer isDataType.sql, withQuotingUtils.quoteIfNeededfor field names. - DataFusion core and arrow-rs have nothing equivalent. Reuse
parquet_variant::Variantto walk values and write
the Spark type lattice and printer directly. Share them withschema_of_variant_agg, and with
schema_of_json/from_json's merge logic if they land. - This depends on the VARIANT Arrow representation in the
parse_jsonissue. - [EPIC] Native Variant Support datafusion-comet#5438 tracks native Variant support in Comet.
- Put it in
crates/spark/src/function/variant/. Addcrates/sqllogictest/test_files/variant/schema_of_variant.slt
and document the function with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
Activity