Repository navigation
Description
Part of #273 and apache/datafusion-comet#7300.
What the function does
schema_of_variant_agg(v) is an aggregate. It merges the schemas of all non-NULL VARIANT values in the group and
returns the result as a STRING. The output format is the same as schema_of_variant: Spark SQL type syntax with
OBJECT<...> for objects. The argument must be VARIANT. The result is non-nullable and uses the default string
type. Added in Spark 4.0.0.
The buffer starts as VOID (NullType). Each row merges SchemaOfVariant.schemaOf(v) into it with
SchemaOfVariant.mergeSchema, and partial buffers are combined the same way. SQL NULL rows are skipped. The merge
and print rules are the ones in the schema_of_variant issue, so the two should share code.
Examples
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('1'), ('2'), ('3') AS tab(j);
-- BIGINT
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": 1}'), ('{"b": true}'), ('{"c": 1.23}') AS tab(j);
-- OBJECT<a: BIGINT, b: BOOLEAN, c: DECIMAL(3,2)>
SELECT schema_of_variant_agg(parse_json('{"a": [1, 2, 3]}'));
-- OBJECT<a: ARRAY<BIGINT>>
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": [1, 2, 3]}'), ('{"a": "banana"}') AS tab(j);
-- OBJECT<a: VARIANT>
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": "banana"}'), ('{"b": "apple"}') AS tab(j);
-- OBJECT<a: STRING, b: STRING>
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": "data"}'), (null) AS tab(j);
-- OBJECT<a: STRING>
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES (null), (null) AS tab(j);
-- VOID
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES ('{"a": null}'), ('{"a": null}') AS tab(j);
-- OBJECT<a: VOID>
SELECT schema_of_variant_agg(parse_json(j)) FROM VALUES
('{"hi":[]}'), ('{"hi":[{},{}]}'), ('{"hi":[{"it''s":[{"me":[{"a": 1}]}]}]}') AS tab(j);
-- OBJECT<hi: ARRAY<OBJECT<`it's`: ARRAY<OBJECT<me: ARRAY<OBJECT<a: BIGINT>>>>>>>These come from the ExpressionDescription and VariantEndToEndSuite."schema_of_variant_agg". The suite builds
the inputs as DataFrames, and the queries above are the equivalent SQL.
Edge cases to test
- Grouping.
VariantEndToEndSuiteuses rowsidin 0..100 with{"a": [id]}whenid % 4 == 0and
{"a": ["id"]}otherwise:GROUP BY id % 2givesOBJECT<a: ARRAY<STRING>>andOBJECT<a: ARRAY<VARIANT>>;% 3givesOBJECT<a: ARRAY<VARIANT>>three times;% 4givesOBJECT<a: ARRAY<STRING>>three times andOBJECT<a: ARRAY<BIGINT>>once.
These exercise the multi-partition merge path.
- If every input is NULL the result is
VOID. With no input rows the buffer is never updated, so by code reading
the result is alsoVOID, not NULL, because the function is non-nullable. - A merge across rows behaves like a merge across array elements. For example,
BIGINTwithDOUBLEgives
DOUBLE, objects are unioned with sorted field names, and incompatible types giveVARIANT. See the full table
in theschema_of_variantissue. - Order independence: decimal widening, struct union and
VOIDabsorption are commutative. One exception, by code
reading, is struct merging throughfindTightestCommonType, which keeps the first field name when names are
equal case-insensitively (spark.sql.caseSensitive=false). The result could depend on the order of rows or
partitions, so do not test that case for determinism. - Field-name quoting with backticks follows
QuotingUtils.quoteIfNeeded. For example,!special!is quoted and
STRUCTis not. - Malformed input raises
MALFORMED_VARIANT(unsorted object fields) orUNKNOWN_PRIMITIVE_TYPE_IN_VARIANT, the
same asschema_of_variant. - Test a literal argument, a column argument,
GROUP BY, and use as a window aggregate if datafusion-spark
supports that for UDAFs.
Differences across Spark versions
- 3.x has no VARIANT.
- No differences between 4.0.4, 4.1.3 and 4.2.0.
Implementation notes
- Spark:
SchemaOfVariantAgg(aTypedImperativeAggregate[DataType]) in
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala. Spark
serializes the buffer asDataType.json, which is internal. - In DataFusion, write an
AggregateUDFImpl/Accumulatorwhose state is the in-progress Spark type tree. The
state can be a singleUtf8field, with the type serialized in any internal form.merge_batchapplies the same
mergeSchemaasupdate_batch. Reuse the type lattice and printer fromschema_of_variant. - DataFusion core and arrow-rs have no equivalent. Use
parquet_variant::Variantto decode values. This depends on
the VARIANT Arrow representation proposed in theparse_jsonissue. - [EPIC] Native Variant Support datafusion-comet#5438 tracks native Variant support in Comet.
- Put it under
crates/spark/src/function/aggregate/or the newvariant/module. Add
crates/sqllogictest/test_files/variant/schema_of_variant_agg.sltand document the function with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
Activity