You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(dataframe): expose withColumn and unnestColumns #41
Is your feature request related to a problem or challenge?
The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.
Describe the solution you'd like
DataFrame.withColumn(String name, String expr). The expression is
a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
convention already used by DataFrame.filter(String). Behaviour
mirrors DataFusion::DataFrame::with_column — replaces a column of
the same name in place, otherwise appends.
DataFrame.unnestColumns(String... columns) and an overload
accepting an UnnestOptions value (preserve-nulls, recursions). Maps
to unnest_columns / unnest_columns_with_options.
Tests in DataFrameTransformationsTest (round-trip on a small
fixture, schema check, error on unknown column).
Describe alternatives you've considered
SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.
Additional context
Follows the same JNI shape as #19's filter(String) — parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.
Activity