Skip to content

Implement Spark-compatible inline / inline_outer #191

Open

Description

@andygrove

Part of #173 and apache/datafusion-comet#7300.

What the function does

inline(expr) is a generator that explodes an ARRAY<STRUCT<...>> into one output row per array element,
with one output column per struct field. The output column names and types are the struct's field names
and types (col1, col2, ... for struct(...) with unnamed fields). If the array's containsNull is
true, all output columns become nullable (st.asNullable). Since Spark 2.0.0 (table-valued function form
SELECT * FROM inline(...) since 3.4.0).

inline_outer(expr) is the same generator wrapped in GeneratorOuter: when the array is NULL or empty
it emits one row with every column NULL instead of no rows.

Only ARRAY<STRUCT> is accepted; there are no implicit casts. Anything else (including NULL
literals, array() which is ARRAY<VOID>, and a bare struct) fails analysis with
DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (requiredType "ARRAY<STRUCT>"). From 4.0 the argument can
be passed by name (input => ...).

Examples

SELECT inline(array(struct(1, 'a'), struct(2, 'b')));
-- 1	a
-- 2	b

SELECT * FROM inline(array(struct(1, 'a'), struct(2, 'b'))) t(x, y);
-- schema struct<x:int,y:string>
-- 1	a
-- 2	b

SELECT inline(array(struct(null, 'a'), struct(null, 'b')));
-- schema struct<col1:void,col2:string>
-- NULL	a
-- NULL	b

-- empty array: inline gives no rows, inline_outer gives one NULL row
SELECT * FROM inline(array_remove(array(struct(1, 'a')), struct(1, 'a')));
-- (no rows)
SELECT * FROM inline_outer(array_remove(array(struct(1, 'a')), struct(1, 'a')));
-- NULL	NULL

-- NULL struct elements become a row of NULLs (GeneratorFunctionSuite, SPARK-39061)
SELECT inline(b) FROM VALUES (1, array(named_struct('c1', 0, 'c2', 1), null,
                                        named_struct('c1', 2, 'c2', 3), null)) AS tbl(a, b);
-- 0	1
-- NULL	NULL
-- 2	3
-- NULL	NULL

-- LATERAL VIEW OUTER over arrays, an empty array and NULL (GeneratorFunctionSuite)
CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES
  array(struct(0, 1), struct(3, 4)), array(struct(6, 7)), array(), null AS tbl(arr);
SELECT f1, f2 FROM t1 LATERAL VIEW inline_outer(arr) AS f1, f2;
-- 0	1
-- 3	4
-- 6	7
-- NULL	NULL
-- NULL	NULL

SELECT inline(1);
-- DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (inputType "INT", requiredType "ARRAY<STRUCT>")

Edge cases to test

  • NULL array: inline emits no rows; inline_outer emits one all-NULL row.
  • Empty array: same as NULL (no rows vs one all-NULL row).
  • NULL struct element inside the array: emits a row whose fields are all NULL (both forms), and it must not
    be confused with the outer padding row.
  • Struct with NULL fields, and with nested types (arrays, maps, structs) as fields.
  • Struct with a single field, and with many fields; field names with dots or duplicates (Spark keeps the
    struct's names, e.g. a from struct(a, 'a')).
  • Output nullability: non-nullable struct fields become nullable when the array may contain NULLs.
  • Other columns of the input row are repeated for each generated row (SELECT a, inline(b)).
  • Literal array vs column input; many input rows with arrays of varying length, including length 0.
  • Invalid input types: inline(NULL) (VOID), inline(array()) (ARRAY<VOID>), inline(array(1, 'a')),
    inline(struct(1, 'a')) all raise DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE at analysis.
  • Deterministic: output order follows array order within each input row.

Differences across Spark versions

  • Inline itself (type check, null-struct handling, nullability) is the same in 3.4.3 through 4.2.0.
  • 3.4/3.5 register inline via expression[Inline] and inline_outer via expressionGeneratorOuter;
    4.0+ use InlineExpressionBuilder / InlineGeneratorBuilder / InlineOuterGeneratorBuilder, which add
    the named argument input =>. Results are unchanged.

Implementation notes

  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala, classes
    Inline, InlineExpressionBuilder, InlineGeneratorBuilderBase, InlineGeneratorBuilder,
    InlineOuterGeneratorBuilder; outer handling is GeneratorOuter plus GenerateExec (outer = true).
  • DataFusion core has no inline. The closest is unnest on a List<Struct> followed by struct
    expansion. Its default NullHandling::Preserve keeps a NULL list as one NULL row, which matches neither
    form: inline needs NullHandling::Drop and inline_outer needs NullHandling::PreserveAndExpandEmpty
    (datafusion/common/src/unnest.rs). A NULL struct element must expand to all-NULL fields.
  • In datafusion-spark this is a row-generating operation, like the existing stack
    (crates/spark/src/function/generator/stack.rs), which returns a list of structs for the caller to
    expand. Since inline's input already is a list of structs, the useful pieces here are likely planning
    helpers (output schema with Spark's names and nullability, and the outer/non-outer null policy) rather
    than a scalar kernel; agree on the shape before writing code. Place it in the generator module.
  • Tests: add crates/sqllogictest/test_files/generator/inline.slt (cover both functions), and document
    with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions