Skip to content

[FLINK-39049][table] Add VARIANT_GET and TRY_VARIANT_GET - #29617

Merged
raminqaf merged 2 commits into
apache:masterfrom
raminqaf:FLINK-39049-variant-get
Oct 28, 2026
Merged

raminqaf merged 2 commits into
apache:masterfrom
raminqaf:FLINK-39049-variant-get

Conversation

@raminqaf

@raminqaf raminqaf commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Reading one value out of a VARIANT needs navigation plus a cast, and navigation fails as soon as a step meets the wrong kind of value. This adds VARIANT_GET(variant, path[, type]) and TRY_VARIANT_GET, with the path syntax proposed in FLIP-618.

On v = PARSE_JSON('{"order_id": 17, "user": {"name": "ada"}, "items": [{"sku": "A1", "qty": 2}], "coupon": null, "price": "12.50"}'), from VariantFunctionsITCase:

Call Result
VARIANT_GET(v, '$.order_id', 'INT') 17
VARIANT_GET(v, '$.user.name', 'VARIANT') "ada", as a VARIANT
VARIANT_GET(v, '$.missing', 'INT NOT NULL') NULL, the result type is always nullable
VARIANT_GET(v, '$.coupon') a variant null, JSON_STRING gives null
VARIANT_GET(v, '$.coupon', 'STRING') NULL
VARIANT_GET(v, '$.price', 'DECIMAL(4, 2)') fails: Failed to cast the VARIANT value at path '$.price' to DECIMAL(4, 2): Cannot cast a VARIANT STRING value to DECIMAL(4, 2).
TRY_VARIANT_GET(v, '$.price', 'DECIMAL(4, 2)') NULL
VARIANT_GET(v, '$.items[-1]') fails at planning: Invalid path '$.items[-1]': expected an index or a quoted name after '$.items['.

Decisions a reviewer would otherwise ask about:

  • Paths follow FLIP-618, which is still under discussion: $, .name, ['name'], ["name"] and a 0-based [n], like JSON_VALUE and Spark, while variant[INT] is 1-based. A missing field, an index past the end, or a step on the wrong kind of value gives NULL. A malformed literal path fails at planning. A malformed path from a column fails when the row runs, and TRY_VARIANT_GET returns NULL. This differs from the ticket text, which said an invalid path gives NULL.
  • returnType is a string literal such as 'INT', as in Spark, not RETURNING INT. It needs no parser change and stays optional. The Table API takes a DataType and passes its logical type.
  • The cast reuses FromVariantConverter, so VARIANT_GET(v, p, 'T') behaves like CAST of the value at p to T. Only the top level is made nullable, so NOT NULL inside T is still checked. Spark makes every level nullable.
  • An own path parser, SimplePath, instead of Jayway. flink-table-common has no Jayway dependency, which planning-time validation needs. Jayway also accepts paths such as $.a[ and $['a','b'], and the planned edit functions need the steps.
  • A runtime class, not codegen. The generated code for CAST(v['n'] AS INT) makes the same per-row calls, and JSON_VALUE only needs codegen for its RETURNING syntax.

Brief change log

  • [hotfix][docs]: variant[INT] and variant[STRING] throw on the wrong kind of value, but the docs said they return NULL
  • SimplePath in flink-table-common parses the FLIP-618 path syntax
  • VARIANT_GET and TRY_VARIANT_GET: function definitions, input and output type strategies, runtime functions, Table API and PyFlink methods
  • Docs: function entries and one "Variant paths" section that later path functions can point to
  • Review feedback, plus two hotfixes it led to: the json_length docstring says NULL instead of None, and getLocalTimeZone(ReadableConfig) moves to flink-table-api-java so the runtime can use it

Verifying this change

This change added tests and can be verified as follows:

  • SimplePathTest: one case per parse branch, which characters end a field name, error messages and serialization
  • VariantFunctionsITCase: SQL and Table API results, NULL handling, the session time zone, and validation and runtime errors
  • VariantSemanticTest: a path that changes from row to row, including a malformed one, for both functions
  • test_expression.py: the PyFlink expressions. I did not run the Python tests locally.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes, new methods on BaseExpressions
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes, the new function. A literal path is parsed once, and a path from a column is parsed when it changes from the previous row
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs, JavaDocs and PyFlink docstrings

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)

@flinkbot

flinkbot commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf
raminqaf force-pushed the FLINK-39049-variant-get branch 2 times, most recently from a7d0f3b to 9f30f42 Compare October 26, 2026 15:19
@raminqaf
raminqaf force-pushed the FLINK-39049-variant-get branch from 9f30f42 to 5b22983 Compare October 26, 2026 17:17

@twalthr twalthr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @raminqaf. I left some feedback.

Comment thread docs/data/sql_functions.yml Resolved
Comment thread docs/data/sql_functions.yml Resolved

@twalthr twalthr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @raminqaf

An expression gives a SQL NULL, not a Python None, and the Python method is `parse_json`, not `PARSE_JSON`.

Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
`VARIANT_GET(variant, path[, returnType])` returns the value of a Variant at a path, cast to `returnType`, or as a Variant without `returnType`. `TRY_VARIANT_GET` returns NULL instead of an error when the value cannot be cast or a path that is not a literal is malformed.

Paths follow FLIP-618: `$`, `.name`, `['name']`, `["name"]` and the 0-based `[n]`, without wildcards, filters, slices, negative indices or path modes. The internal `SimplePath` in flink-table-common parses them, so a malformed literal path fails at planning. A missing field, an index past the end, or a step on the wrong kind of value gives NULL.

`returnType` is a string literal, as in Spark, parsed with `LogicalTypeParser`. The cast reuses `FromVariantConverter`, so `VARIANT_GET(v, p, 'T')` casts like `CAST`, and a variant null gives NULL unless `returnType` is `VARIANT`. The result is always nullable. The Table API and PyFlink get `variantGet` / `variant_get` and `tryVariantGet` / `try_variant_get`.

`getLocalTimeZone(ReadableConfig)` moves from the planner's `TableConfigUtils` to `TableConfigValidation` in flink-table-api-java, so the runtime function can read the session time zone. The planner's method delegates to it.

The docs add the function entries and a "Variant paths" section. They also correct `variant[INT]` and `variant[STRING]`, which throw on the wrong kind of value instead of returning NULL.

Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
@raminqaf
raminqaf force-pushed the FLINK-39049-variant-get branch from f914991 to 498f238 Compare October 28, 2026 16:11
@raminqaf
raminqaf merged commit 469332b into apache:master Oct 28, 2026
1 check passed
@raminqaf
raminqaf deleted the FLINK-39049-variant-get branch October 28, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants