Repository navigation
[FLINK-39049][table] Add VARIANT_GET and TRY_VARIANT_GET - #29617
Merged
Merged
Conversation
Collaborator
raminqaf
force-pushed
the
FLINK-39049-variant-get
branch
2 times, most recently
from
October 26, 2026 15:19
a7d0f3b to
9f30f42
Compare
raminqaf
commented
Oct 26, 2026
raminqaf
force-pushed
the
FLINK-39049-variant-get
branch
from
October 26, 2026 17:17
9f30f42 to
5b22983
Compare
twalthr
reviewed
Oct 28, 2026
twalthr
approved these changes
Oct 28, 2026
An expression gives a SQL NULL, not a Python None, and the Python method is `parse_json`, not `PARSE_JSON`. Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
`VARIANT_GET(variant, path[, returnType])` returns the value of a Variant at a path, cast to `returnType`, or as a Variant without `returnType`. `TRY_VARIANT_GET` returns NULL instead of an error when the value cannot be cast or a path that is not a literal is malformed. Paths follow FLIP-618: `$`, `.name`, `['name']`, `["name"]` and the 0-based `[n]`, without wildcards, filters, slices, negative indices or path modes. The internal `SimplePath` in flink-table-common parses them, so a malformed literal path fails at planning. A missing field, an index past the end, or a step on the wrong kind of value gives NULL. `returnType` is a string literal, as in Spark, parsed with `LogicalTypeParser`. The cast reuses `FromVariantConverter`, so `VARIANT_GET(v, p, 'T')` casts like `CAST`, and a variant null gives NULL unless `returnType` is `VARIANT`. The result is always nullable. The Table API and PyFlink get `variantGet` / `variant_get` and `tryVariantGet` / `try_variant_get`. `getLocalTimeZone(ReadableConfig)` moves from the planner's `TableConfigUtils` to `TableConfigValidation` in flink-table-api-java, so the runtime function can read the session time zone. The planner's method delegates to it. The docs add the function entries and a "Variant paths" section. They also correct `variant[INT]` and `variant[STRING]`, which throw on the wrong kind of value instead of returning NULL. Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
raminqaf
force-pushed
the
FLINK-39049-variant-get
branch
from
October 28, 2026 16:11
f914991 to
498f238
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Reading one value out of a VARIANT needs navigation plus a cast, and navigation fails as soon as a step meets the wrong kind of value. This adds
VARIANT_GET(variant, path[, type])andTRY_VARIANT_GET, with the path syntax proposed in FLIP-618.On
v = PARSE_JSON('{"order_id": 17, "user": {"name": "ada"}, "items": [{"sku": "A1", "qty": 2}], "coupon": null, "price": "12.50"}'), fromVariantFunctionsITCase:VARIANT_GET(v, '$.order_id', 'INT')17VARIANT_GET(v, '$.user.name', 'VARIANT')"ada", as a VARIANTVARIANT_GET(v, '$.missing', 'INT NOT NULL')NULL, the result type is always nullableVARIANT_GET(v, '$.coupon')JSON_STRINGgivesnullVARIANT_GET(v, '$.coupon', 'STRING')NULLVARIANT_GET(v, '$.price', 'DECIMAL(4, 2)')Failed to cast the VARIANT value at path '$.price' to DECIMAL(4, 2): Cannot cast a VARIANT STRING value to DECIMAL(4, 2).TRY_VARIANT_GET(v, '$.price', 'DECIMAL(4, 2)')NULLVARIANT_GET(v, '$.items[-1]')Invalid path '$.items[-1]': expected an index or a quoted name after '$.items['.Decisions a reviewer would otherwise ask about:
$,.name,['name'],["name"]and a 0-based[n], likeJSON_VALUEand Spark, whilevariant[INT]is 1-based. A missing field, an index past the end, or a step on the wrong kind of value gives NULL. A malformed literal path fails at planning. A malformed path from a column fails when the row runs, andTRY_VARIANT_GETreturns NULL. This differs from the ticket text, which said an invalid path gives NULL.returnTypeis a string literal such as'INT', as in Spark, notRETURNING INT. It needs no parser change and stays optional. The Table API takes aDataTypeand passes its logical type.FromVariantConverter, soVARIANT_GET(v, p, 'T')behaves likeCASTof the value atptoT. Only the top level is made nullable, soNOT NULLinsideTis still checked. Spark makes every level nullable.SimplePath, instead of Jayway.flink-table-commonhas no Jayway dependency, which planning-time validation needs. Jayway also accepts paths such as$.a[and$['a','b'], and the planned edit functions need the steps.CAST(v['n'] AS INT)makes the same per-row calls, andJSON_VALUEonly needs codegen for itsRETURNINGsyntax.Brief change log
[hotfix][docs]:variant[INT]andvariant[STRING]throw on the wrong kind of value, but the docs said they return NULLSimplePathinflink-table-commonparses the FLIP-618 path syntaxVARIANT_GETandTRY_VARIANT_GET: function definitions, input and output type strategies, runtime functions, Table API and PyFlink methodsjson_lengthdocstring says NULL instead of None, andgetLocalTimeZone(ReadableConfig)moves to flink-table-api-java so the runtime can use itVerifying this change
This change added tests and can be verified as follows:
SimplePathTest: one case per parse branch, which characters end a field name, error messages and serializationVariantFunctionsITCase: SQL and Table API results, NULL handling, the session time zone, and validation and runtime errorsVariantSemanticTest: a path that changes from row to row, including a malformed one, for both functionstest_expression.py: the PyFlink expressions. I did not run the Python tests locally.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes, new methods onBaseExpressionsDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)