[SPARK-58035] Add try_variant_insert expression#57120
Conversation
try_variant_set expressiontry_variant_insert expression
harshmotw-db
left a comment
There was a problem hiding this comment.
LGTM! left minor comments
cloud-fan
left a comment
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
Clean, idiomatic addition of try_variant_insert following the established try_variant_get / try_parse_json pattern.
No findings. Structure (flag on the shared VariantInsert case class + parameterized builder objects), nullability, error taxonomy, registration, and test layout all match the existing try_* variant expressions. The prior inline comments (catch-clause ordering, "recoverable" terminology) both appear already addressed in the current head.
Verification
Checked the try-mode error contract: the shared try in insertAtPath catches only VARIANT_PATH_TYPE_MISMATCH and VARIANT_DUPLICATE_KEY (→ NULL); VARIANT_SIZE_LIMIT always rethrows, and a malformed path throws in parseVariantPath before the try, so both strict and try modes reject it — matching try_variant_get. The catch is precise, not over-broad: checkInputDataTypes restricts the value to allowStructsAndMaps=false, so castToVariant inside the same try cannot itself raise VARIANT_DUPLICATE_KEY — the only duplicate-key reachable there is the insertion collision. Nullability is also correct: !failOnError || children.exists(_.nullable) reduces to the inherited default for variant_insert (preserving its prior behavior exactly) and is unconditionally true for the try variant.
0378cf1 to
bd403be
Compare
What changes were proposed in this pull request?
Adds the SQL function
try_variant_insert(v, path, val), the error-tolerant counterpart ofvariant_insert(SPARK-57641). It inserts a value into a Variant value at a single JSONPath location, returning NULL instead of failing the query when the insertion hits a recoverable error.Details:
variant_insert: an object path (e.g.$.b) adds a new field; an array path (e.g.$[N]) inserts at indexN, shiftinglater elements right and padding gaps past the end with variant nulls; missing
intermediate keys/indices along the path are created;
(
VARIANT_DUPLICATE_KEY) and any path type mismatch (VARIANT_PATH_TYPE_MISMATCH,e.g. an object key applied to an array);
variant_insert: a malformed path isrejected with
INVALID_VARIANT_PATH(the path is parsed before the insert), and aresult exceeding the size limit raises
VARIANT_SIZE_LIMIT;Why are the changes needed?
Error-tolerant counterpart of
variant_insert, matchingtry_variant_getand othertry_*functions.Does this PR introduce any user-facing change?
Yes, a new SQL function
try_variant_insert(and the corresponding Scala/PythonfunctionsAPI).How was this patch tested?
Unit tests.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor with Claude Opus 4.8