[SPARK-57804][SQL] Add variant_set expression

[bojana-db]

What changes were proposed in this pull request?

Adds the SQL function variant_set(v, path, val[, create_if_missing]), which sets (inserts or replaces) a value in a Variant value at a single JSONPath location.

Details:

• Object path (e.g. $.a): replaces the field if it exists; creates it when create_if_missing is true (the default);
• Array path (e.g. $[N]): replaces the element at index N; when N is at or past the end and create_if_missing is true, the array is padded with variant nulls up to N;
• Missing intermediate keys/indices along the path are created when create_if_missing` is true;
• When create_if_missing is false, a missing leaf, a missing intermediate, or an out-of-range array index leaves v unchanged;
• Any NULL argument returns NULL;
• The value may be any expression castable to variant (primitives, arrays, or another variant; structs and maps are rejected);
• VARIANT_PATH_TYPE_MISMATCH is raised when a path segment is applied to a value of an incompatible type; the root path $ is rejected with INVALID_VARIANT_PATH, and results exceeding the size limit raise VARIANT_SIZE_LIMIT.

Why are the changes needed?

Without variant_set, updating a variant means converting it to another datatype (e.g. map), mutating, and converting back.

Does this PR introduce any user-facing change?

Yes, a new SQL function (and Scala/Python functions API).

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code with Claude Opus 4.8

[shrirangmhalgi]

Design question (non-blocking)

When createIfMissing is false, a type mismatch (e.g., object-key segment hitting a scalar) throws VARIANT_PATH_TYPE_MISMATCH. Should it instead leave the variant unchanged (same as missing-key behavior)?

PostgreSQL's jsonb_set(target, path, new_value, create_missing := false) returns target unchanged when a path is unreachable -- the docs state: "If these conditions are not met the target is returned unchanged." This covers both missing intermediates AND type mismatches.

The current behavior is internally consistent (createIfMissing=true also throws on type mismatch), but users migrating from PostgreSQL may expect silent passthrough when create_if_missing is false.

Not blocking - just flagging since it's a user-facing semantic difference from the ecosystem precedent.