[FLINK-39049][table] Add VARIANT_GET and TRY_VARIANT_GET

docs/content.zh/docs/sql/functions/built-in-functions.md

@@ -233,6 +233,36 @@ A document fails to parse if it:
 
 In all of these cases, `TRY_PARSE_XML` returns `NULL`.
 
+#### Variant paths
+
+`VARIANT_GET` and `TRY_VARIANT_GET` take a path that names one place in a Variant. A path starts
+with `$`, the whole value, followed by steps: `.name`, `['name']` or `["name"]` for a field, and
+`[n]` for an array element. For example, on
+`PARSE_JSON('{"user": {"name": "ada"}, "items": [{"qty": 2}]}')`:
+
+| Path                | Points at                          |
+|---------------------|------------------------------------|
+| `$`                 | the whole value                    |
+| `$.user.name`       | `"ada"`                            |
+| `$['user']["name"]` | `"ada"`                            |
+| `$.items[0].qty`    | `2`                                |
+| `$.items[1]`        | nothing, the index is past the end |
+| `$.user[0]`         | nothing, `user` is not an array    |
+
+- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts at 1. So
+  `VARIANT_GET(v, '$.items[0]')` and `v['items'][1]` are the same element.
+- Use brackets with single or double quotes for a name with whitespace or one of the characters
+  `. [ ] ( ) * ' " \`, for example `$['first name']` or `$["first name"]`. Inside the quotes, `\\`,
+  `\'` and `\"` stand for a backslash and a quote.
+- Wildcards, filters, slices, negative indices and the `lax` and `strict` modes are not supported,
+  so a path always names one place.
+- Trailing spaces are ignored.
+- A malformed path literal fails at planning time. Any other malformed path fails when the function
+  runs.
+
+A path points at nothing if a field is missing, an index is past the end, or a step does not match
+the kind of value, such as a field of an array. Each function describes what it does in that case.
+
 ### 值构建函数
 
 {{< sql_functions_zh "valueconstruction" >}}

docs/content.zh/docs/sql/reference/data-types.md

@@ -1551,7 +1551,10 @@ To store one of these values, cast the number to a `VARIANT` instead of parsing
 
 A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A cast succeeds only when
 the target holds the stored value without reinterpreting it, so a value is never wrapped or rounded
-to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`. `VARIANT_GET` and
+`TRY_VARIANT_GET` read the value at a
+[path]({{< ref "docs/sql/functions/built-in-functions" >}}#variant-paths) such as `'$.items[0].qty'`
+and cast it by the same rules, returning `NULL` when nothing is at the path.
 
 | Stored kind     | Succeeds for                                        |
 |-----------------|-----------------------------------------------------|

docs/content/docs/sql/functions/built-in-functions.md

@@ -236,6 +236,36 @@ A document fails to parse if it:
 
 In all of these cases, `TRY_PARSE_XML` returns `NULL`.
 
+#### Variant paths
+
+`VARIANT_GET` and `TRY_VARIANT_GET` take a path that names one place in a Variant. A path starts
+with `$`, the whole value, followed by steps: `.name`, `['name']` or `["name"]` for a field, and
+`[n]` for an array element. For example, on
+`PARSE_JSON('{"user": {"name": "ada"}, "items": [{"qty": 2}]}')`:
+
+| Path                | Points at                          |
+|---------------------|------------------------------------|
+| `$`                 | the whole value                    |
+| `$.user.name`       | `"ada"`                            |
+| `$['user']["name"]` | `"ada"`                            |
+| `$.items[0].qty`    | `2`                                |
+| `$.items[1]`        | nothing, the index is past the end |
+| `$.user[0]`         | nothing, `user` is not an array    |
+
+- Array indices start at 0, as in `JSON_VALUE`, while `variant[INT]` starts at 1. So
+  `VARIANT_GET(v, '$.items[0]')` and `v['items'][1]` are the same element.
+- Use brackets with single or double quotes for a name with whitespace or one of the characters
+  `. [ ] ( ) * ' " \`, for example `$['first name']` or `$["first name"]`. Inside the quotes, `\\`,
+  `\'` and `\"` stand for a backslash and a quote.
+- Wildcards, filters, slices, negative indices and the `lax` and `strict` modes are not supported,
+  so a path always names one place.
+- Trailing spaces are ignored.
+- A malformed path literal fails at planning time. Any other malformed path fails when the function
+  runs.
+
+A path points at nothing if a field is missing, an index is past the end, or a step does not match
+the kind of value, such as a field of an array. Each function describes what it does in that case.
+
 ### Value Construction Functions
 
 {{< sql_functions "valueconstruction" >}}

docs/content/docs/sql/reference/data-types.md

@@ -1559,7 +1559,10 @@ To store one of these values, cast the number to a `VARIANT` instead of parsing
 
 A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A cast succeeds only when
 the target holds the stored value without reinterpreting it, so a value is never wrapped or rounded
-to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`. `VARIANT_GET` and
+`TRY_VARIANT_GET` read the value at a
+[path]({{< ref "docs/sql/functions/built-in-functions" >}}#variant-paths) such as `'$.items[0].qty'`
+and cast it by the same rules, returning `NULL` when nothing is at the path.
 
 | Stored kind     | Succeeds for                                        |
 |-----------------|-----------------------------------------------------|

docs/data/sql_functions.yml

@@ -1451,13 +1451,69 @@ variant:
   - sql: variant '[' INT ']'
     table: VARIANT.at(INT)
     description: |
-      If the VARIANT is an ARRAY value, returns a VARIANT whose value is the element at
-      the specified index. The index starts from 1, If the index is out of range, it returns NULL.
+      If the VARIANT is an array, returns a VARIANT holding the element at the index. The index
+      starts at 1. If the index is out of range, returns NULL. If the VARIANT is not an array, an
+      error is thrown. To return NULL instead, use `VARIANT_GET`.
   - sql: variant '[' STRING ']'
     table: VARIANT.at(STRING)
     description: |
-      If the VARIANT is a MAP value that has an element with this key, a VARIANT holding
-      the associated value is returned. Otherwise, NULL is returned.
+      If the VARIANT is an object that has a field with this key, returns a VARIANT holding the
+      value of the field. If the object has no such field, returns NULL. If the VARIANT is not an
+      object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+  - sql: VARIANT_GET(variant, path[, returnType])
+    table: VARIANT.variantGet(path[, returnType])
+    description: |
+      Returns the value of a Variant at a path, cast to `returnType`. Without `returnType`, the
+      value is returned as a Variant. If the value cannot be cast to `returnType`, an error is
+      thrown. To return NULL instead of an error, use the `TRY_VARIANT_GET` function. See the
+      "Variant paths" section below the function table for the path syntax.
+
+      Returns NULL if:
+
+      * `variant` or `path` is NULL,
+      * nothing is at the path, or
+      * the value at the path is a variant null and `returnType` is not `'VARIANT'`.
+
+      Without `returnType`, or with `'VARIANT'`, a variant null stays a variant null. It is not SQL
+      NULL, so `IS NULL` is false and `JSON_STRING` returns `'null'`.
+
+      `returnType` is a string literal such as `'INT'` or `'ARRAY<ROW<sku STRING, qty INT>>'`, and
+      must not be NULL. The value is cast the way `CAST(v AS returnType)` casts a Variant. The
+      result is always nullable.
+
+      ```sql
+      -- 17 of type INT
+      VARIANT_GET(PARSE_JSON('{"order_id": 17}'), '$.order_id', 'INT')
+
+      -- 2 of type INT
+      VARIANT_GET(PARSE_JSON('{"items": [{"qty": 2}]}'), '$.items[0].qty', 'INT')
+
+      -- NULL of type STRING, nothing is at the path
+      VARIANT_GET(PARSE_JSON('{"order_id": 17}'), '$.user.name', 'STRING')
+
+      -- NULL of type INT, a variant null cast to INT
+      VARIANT_GET(PARSE_JSON('{"coupon": null}'), '$.coupon', 'INT')
+
+      -- null of type VARIANT, a variant null and not SQL NULL
+      VARIANT_GET(PARSE_JSON('{"coupon": null}'), '$.coupon')
+
+      -- {"name":"ada"} of type VARIANT
+      VARIANT_GET(PARSE_JSON('{"user": {"name": "ada"}}'), '$.user')
+
+      -- fails, a Variant string does not cast to INT
+      VARIANT_GET(PARSE_JSON('{"price": "12.50"}'), '$.price', 'INT')
+      ```
+  - sql: TRY_VARIANT_GET(variant, path[, returnType])
+    table: VARIANT.tryVariantGet(path[, returnType])
+    description: |
+      Returns the value of a Variant at a path like `VARIANT_GET`, but returns NULL instead of an
+      error if the value cannot be cast to `returnType`, or if a path that is not a literal is
+      malformed. A malformed path literal still fails at planning time.
+
+      ```sql
+      -- NULL of type INT, a Variant string does not cast to INT
+      TRY_VARIANT_GET(PARSE_JSON('{"price": "12.50"}'), '$.price', 'INT')
+      ```
 
 valueconstruction:
   - sql: |

docs/data/sql_functions_zh.yml

@@ -1549,12 +1549,70 @@ variant:
   - sql: variant '[' INT ']'
     table: VARIANT.at(INT)
     description: |
-      如果这是一个 ARRAY 类型的 VARIANT，则返回一个 VARIANT，其值为指定索引处的元素。索引从 1 开始，如果索引超出范围，则返回 NULL。
+      If the VARIANT is an array, returns a VARIANT holding the element at the index. The index
+      starts at 1. If the index is out of range, returns NULL. If the VARIANT is not an array, an
+      error is thrown. To return NULL instead, use `VARIANT_GET`.
 
   - sql: variant '[' STRING ']'
     table: VARIANT.at(STRING)
     description: |
-      如果这是一个 MAP 类型的 VARIANT，则返回一个 VARIANT，其值为指定 key 对应的值。否则返回 NULL。
+      If the VARIANT is an object that has a field with this key, returns a VARIANT holding the
+      value of the field. If the object has no such field, returns NULL. If the VARIANT is not an
+      object, an error is thrown. To return NULL instead, use `VARIANT_GET`.
+  - sql: VARIANT_GET(variant, path[, returnType])
+    table: VARIANT.variantGet(path[, returnType])
+    description: |
+      Returns the value of a Variant at a path, cast to `returnType`. Without `returnType`, the
+      value is returned as a Variant. If the value cannot be cast to `returnType`, an error is
+      thrown. To return NULL instead of an error, use the `TRY_VARIANT_GET` function. See the
+      "Variant paths" section below the function table for the path syntax.
+
+      Returns NULL if:
+
+      * `variant` or `path` is NULL,
+      * nothing is at the path, or
+      * the value at the path is a variant null and `returnType` is not `'VARIANT'`.
+
+      Without `returnType`, or with `'VARIANT'`, a variant null stays a variant null. It is not SQL
+      NULL, so `IS NULL` is false and `JSON_STRING` returns `'null'`.
+
+      `returnType` is a string literal such as `'INT'` or `'ARRAY<ROW<sku STRING, qty INT>>'`, and
+      must not be NULL. The value is cast the way `CAST(v AS returnType)` casts a Variant. The
+      result is always nullable.
+
+      ```sql
+      -- 17 of type INT
+      VARIANT_GET(PARSE_JSON('{"order_id": 17}'), '$.order_id', 'INT')
+
+      -- 2 of type INT
+      VARIANT_GET(PARSE_JSON('{"items": [{"qty": 2}]}'), '$.items[0].qty', 'INT')
+
+      -- NULL of type STRING, nothing is at the path
+      VARIANT_GET(PARSE_JSON('{"order_id": 17}'), '$.user.name', 'STRING')
+
+      -- NULL of type INT, a variant null cast to INT
+      VARIANT_GET(PARSE_JSON('{"coupon": null}'), '$.coupon', 'INT')
+
+      -- null of type VARIANT, a variant null and not SQL NULL
+      VARIANT_GET(PARSE_JSON('{"coupon": null}'), '$.coupon')
+
+      -- {"name":"ada"} of type VARIANT
+      VARIANT_GET(PARSE_JSON('{"user": {"name": "ada"}}'), '$.user')
+
+      -- fails, a Variant string does not cast to INT
+      VARIANT_GET(PARSE_JSON('{"price": "12.50"}'), '$.price', 'INT')
+      ```
+  - sql: TRY_VARIANT_GET(variant, path[, returnType])
+    table: VARIANT.tryVariantGet(path[, returnType])
+    description: |
+      Returns the value of a Variant at a path like `VARIANT_GET`, but returns NULL instead of an
+      error if the value cannot be cast to `returnType`, or if a path that is not a literal is
+      malformed. A malformed path literal still fails at planning time.
+
+      ```sql
+      -- NULL of type INT, a Variant string does not cast to INT
+      TRY_VARIANT_GET(PARSE_JSON('{"price": "12.50"}'), '$.price', 'INT')
+      ```
 
 
 valueconstruction:

flink-python/docs/reference/pyflink.table/expressions.rst

@@ -335,6 +335,17 @@ JSON functions
     Expression.json_length
     Expression.json_type
 
+variant functions
+-----------------
+
+.. currentmodule:: pyflink.table.expression
+
+.. autosummary::
+    :toctree: api/
+
+    Expression.variant_get
+    Expression.try_variant_get
+
 value modification functions
 ----------------------------
 

flink-python/pyflink/table/expression.py

@@ -2456,12 +2456,46 @@ def try_parse_xml(self, force_array=None) -> 'Expression':
         else:
             return _binary_op("tryParseXml")(self, force_array)
 
+    def variant_get(self, path, return_type: DataType = None) -> 'Expression':
+        """
+        Returns the value of a VARIANT at a path, cast to return_type. Without return_type, the
+        value is returned as a VARIANT. If the VARIANT or the path is NULL, or nothing is at the
+        path, NULL is returned. A variant null at the path gives NULL, unless return_type is
+        omitted or DataTypes.VARIANT(), which keeps it as a variant null. If the value cannot be
+        cast to return_type, an error is thrown. To return NULL instead, use
+        :func:`~Expression.try_variant_get`.
+
+        Array indices in a path such as '$.items[0].qty' start at 0. See the SQL function
+        VARIANT_GET for the path syntax.
+
+        Example:
+        ::
+
+            >>> lit('{"a": {"b": 1}}').parse_json().variant_get('$.a.b', DataTypes.INT()) # 1
+            >>> lit('{"a": 1}').parse_json().variant_get('$.b', DataTypes.INT()) # NULL
+        """
+        if return_type is None:
+            return _binary_op("variantGet")(self, path)
+        else:
+            return _ternary_op("variantGet")(self, path, _to_java_data_type(return_type))
+
+    def try_variant_get(self, path, return_type: DataType = None) -> 'Expression':
+        """
+        Returns the value of a VARIANT at a path, cast to return_type. Unlike
+        :func:`~Expression.variant_get`, NULL is returned instead of an error if the value cannot
+        be cast to return_type or a path that is not a literal is malformed.
+        """
+        if return_type is None:
+            return _binary_op("tryVariantGet")(self, path)
+        else:
+            return _ternary_op("tryVariantGet")(self, path, _to_java_data_type(return_type))
+
     def json_length(self, path=None) -> 'Expression':
         """
         Returns the number of elements in a JSON document, or the length of the value at the
         specified path if one is provided.
 
-        The input can be a JSON STRING or a VARIANT. Returns None if the argument is None,
+        The input can be a JSON STRING or a VARIANT. Returns NULL if the argument is NULL,
         the json is invalid, or the path is empty, malformed or does not locate a value.
 
         The path must be a plain path literal such as '$.a.b'. A path carrying a
@@ -2474,15 +2508,15 @@ def json_length(self, path=None) -> 'Expression':
         - Objects have a length equal to the number of their key-value pairs.
 
         When provided with a path that uses a wildcard and resolves to 2 or more paths,
-        'json_length' resolves to None.
+        'json_length' resolves to NULL.
 
         json_length also supports input of the VARIANT type; you can pass the output of
-        PARSE_JSON into json_length.
+        parse_json into json_length.
 
-        Because a None result can mean several different things (the input is not valid
+        Because a NULL result can mean several different things (the input is not valid
         JSON, the path does not match anything, or a wildcard path matched 2 or more
         nodes), it is recommended to pair json_length with a helper function so invalid
-        input is handled explicitly rather than silently returning None:
+        input is handled explicitly rather than silently returning NULL:
 
         - Without a path, guard the call with is_json to separate malformed input from a
           real result.
@@ -2491,10 +2525,10 @@ def json_length(self, path=None) -> 'Expression':
 
         ::
 
-            # returns the length only for valid JSON, otherwise None means "invalid input"
+            # returns the length only for valid JSON, otherwise NULL means "invalid input"
             >>> lit("[1,2]").is_json().then(lit("[1,2]").json_length(), null_of(DataTypes.INT()))
 
-            # path is present even when json_length is None due to a multi-match wildcard
+            # path is present even when json_length is NULL due to a multi-match wildcard
             >>> lit("{}").json_exists("$.items[*]")
             >>> lit("{}").json_length("$.items[*]")
 
@@ -2509,9 +2543,9 @@ def json_length(self, path=None) -> 'Expression':
             >>> lit('{"1": [1,2,3], "2": "bye bye"}').json_length('$.1') # 3
             >>> lit('[1,2,3,4,5]').json_length('$[3]') # 1
 
-            >>> lit('[1,2,3,4,5]').json_length('$[7]') # None
-            >>> lit('{"1": "bad", "2": "syntax here ->"').json_length('$.1') # None
-            >>> lit('[1,2,3,4,5]').json_length('$.[') # None
+            >>> lit('[1,2,3,4,5]').json_length('$[7]') # NULL
+            >>> lit('{"1": "bad", "2": "syntax here ->"').json_length('$.1') # NULL
+            >>> lit('[1,2,3,4,5]').json_length('$.[') # NULL
         """
         if path is None:
             return _unary_op("jsonLength")(self)

flink-python/pyflink/table/tests/test_expression.py

@@ -280,6 +280,14 @@ def test_expression(self):
         self.assertEqual("PARSE_XML(a, b)", str(expr1.parse_xml(expr2)))
         self.assertEqual("TRY_PARSE_XML(a, b)", str(expr1.try_parse_xml(expr2)))
 
+        # variant functions
+        self.assertEqual("VARIANT_GET(a, '$.x')", str(expr1.variant_get('$.x')))
+        self.assertEqual("VARIANT_GET(a, '$.x', 'INT')",
+                         str(expr1.variant_get('$.x', DataTypes.INT())))
+        self.assertEqual("TRY_VARIANT_GET(a, b)", str(expr1.try_variant_get(expr2)))
+        self.assertEqual("TRY_VARIANT_GET(a, '$.x', 'ARRAY<INT>')",
+                         str(expr1.try_variant_get('$.x', DataTypes.ARRAY(DataTypes.INT()))))
+
         # bitmap functions
         self.assertEqual("BITMAP_AND(a, b)", str(expr1.bitmap_and(expr2)))
         self.assertEqual("BITMAP_ANDNOT(a, b)", str(expr1.bitmap_andnot(expr2)))