Repository navigation
Description
Part of #173 and apache/datafusion-comet#7300.
What the function does
inline(expr) is a generator that explodes an ARRAY<STRUCT<...>> into one output row per array element,
with one output column per struct field. The output column names and types are the struct's field names
and types (col1, col2, ... for struct(...) with unnamed fields). If the array's containsNull is
true, all output columns become nullable (st.asNullable). Since Spark 2.0.0 (table-valued function form
SELECT * FROM inline(...) since 3.4.0).
inline_outer(expr) is the same generator wrapped in GeneratorOuter: when the array is NULL or empty
it emits one row with every column NULL instead of no rows.
Only ARRAY<STRUCT> is accepted; there are no implicit casts. Anything else (including NULL
literals, array() which is ARRAY<VOID>, and a bare struct) fails analysis with
DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (requiredType "ARRAY<STRUCT>"). From 4.0 the argument can
be passed by name (input => ...).
Examples
SELECT inline(array(struct(1, 'a'), struct(2, 'b')));
-- 1 a
-- 2 b
SELECT * FROM inline(array(struct(1, 'a'), struct(2, 'b'))) t(x, y);
-- schema struct<x:int,y:string>
-- 1 a
-- 2 b
SELECT inline(array(struct(null, 'a'), struct(null, 'b')));
-- schema struct<col1:void,col2:string>
-- NULL a
-- NULL b
-- empty array: inline gives no rows, inline_outer gives one NULL row
SELECT * FROM inline(array_remove(array(struct(1, 'a')), struct(1, 'a')));
-- (no rows)
SELECT * FROM inline_outer(array_remove(array(struct(1, 'a')), struct(1, 'a')));
-- NULL NULL
-- NULL struct elements become a row of NULLs (GeneratorFunctionSuite, SPARK-39061)
SELECT inline(b) FROM VALUES (1, array(named_struct('c1', 0, 'c2', 1), null,
named_struct('c1', 2, 'c2', 3), null)) AS tbl(a, b);
-- 0 1
-- NULL NULL
-- 2 3
-- NULL NULL
-- LATERAL VIEW OUTER over arrays, an empty array and NULL (GeneratorFunctionSuite)
CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES
array(struct(0, 1), struct(3, 4)), array(struct(6, 7)), array(), null AS tbl(arr);
SELECT f1, f2 FROM t1 LATERAL VIEW inline_outer(arr) AS f1, f2;
-- 0 1
-- 3 4
-- 6 7
-- NULL NULL
-- NULL NULL
SELECT inline(1);
-- DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (inputType "INT", requiredType "ARRAY<STRUCT>")Edge cases to test
- NULL array:
inlineemits no rows;inline_outeremits one all-NULL row. - Empty array: same as NULL (no rows vs one all-NULL row).
- NULL struct element inside the array: emits a row whose fields are all NULL (both forms), and it must not
be confused with the outer padding row. - Struct with NULL fields, and with nested types (arrays, maps, structs) as fields.
- Struct with a single field, and with many fields; field names with dots or duplicates (Spark keeps the
struct's names, e.g.afromstruct(a, 'a')). - Output nullability: non-nullable struct fields become nullable when the array may contain NULLs.
- Other columns of the input row are repeated for each generated row (
SELECT a, inline(b)). - Literal array vs column input; many input rows with arrays of varying length, including length 0.
- Invalid input types:
inline(NULL)(VOID),inline(array())(ARRAY<VOID>),inline(array(1, 'a')),
inline(struct(1, 'a'))all raiseDATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPEat analysis. - Deterministic: output order follows array order within each input row.
Differences across Spark versions
Inlineitself (type check, null-struct handling, nullability) is the same in 3.4.3 through 4.2.0.- 3.4/3.5 register
inlineviaexpression[Inline]andinline_outerviaexpressionGeneratorOuter;
4.0+ useInlineExpressionBuilder/InlineGeneratorBuilder/InlineOuterGeneratorBuilder, which add
the named argumentinput =>. Results are unchanged.
Implementation notes
- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala, classes
Inline,InlineExpressionBuilder,InlineGeneratorBuilderBase,InlineGeneratorBuilder,
InlineOuterGeneratorBuilder; outer handling isGeneratorOuterplusGenerateExec(outer = true). - DataFusion core has no
inline. The closest isunneston aList<Struct>followed by struct
expansion. Its defaultNullHandling::Preservekeeps a NULL list as one NULL row, which matches neither
form:inlineneedsNullHandling::Dropandinline_outerneedsNullHandling::PreserveAndExpandEmpty
(datafusion/common/src/unnest.rs). A NULL struct element must expand to all-NULL fields. - In datafusion-spark this is a row-generating operation, like the existing
stack
(crates/spark/src/function/generator/stack.rs), which returns a list of structs for the caller to
expand. Sinceinline's input already is a list of structs, the useful pieces here are likely planning
helpers (output schema with Spark's names and nullability, and the outer/non-outer null policy) rather
than a scalar kernel; agree on the shape before writing code. Place it in thegeneratormodule. - Tests: add
crates/sqllogictest/test_files/generator/inline.slt(cover both functions), and document
with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
Activity