[FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation by haruki-830 · Pull Request #4474 · apache/flink-cdc

haruki-830 · 2026-07-15T09:57:54Z

Summary

This commit improves YAML Transform predicate support and nullable BOOLEAN logical evaluation. It adds support for several previously unsupported SQL predicates, and fixes runtime failures when logical expressions evaluate nullable BOOLEAN operands.

Key Changes

Additional Predicate Support

Added support for IS UNKNOWN / IS NOT UNKNOWN.
Added support for IS DISTINCT FROM / IS NOT DISTINCT FROM.
Added support for LIKE ... ESCAPE.
Added support for SIMILAR TO / NOT SIMILAR TO.

Nullable BOOLEAN Logical Evaluation

Added null-safe handling for AND, OR, and NOT with nullable BOOLEAN operands.
Added null-safe handling for IS TRUE / IS NOT TRUE.
Added null-safe handling for IS FALSE / IS NOT FALSE.
Kept boolean condition consumers null-safe so UNKNOWN does not cause Java unboxing failures.

Test Coverage

Added unit tests for distinct predicates, nullable BOOLEAN logical functions, legacy comparison behavior, legacy two-argument LIKE, LIKE ... ESCAPE, and SIMILAR TO.
Updated parser tests for generated Janino expressions.
Updated YAML transform specs for nullable BOOLEAN logic and the newly supported predicates.

JIRA Reference
https://issues.apache.org/jira/browse/FLINK-40151

haruki-830 · 2026-07-15T10:06:50Z

@lvyanquan Could u PTAL when u have time? Thanks!

lvyanquan · 2026-07-16T15:11:54Z

Thanks for the work here. The changes split into three buckets with very different compatibility impact. I'd suggest applying buckets 1–2 unconditionally and gating bucket 3 behind an explicit option.

1. New operators (currently unparseable) — keep always-on

SQL: IS [NOT] DISTINCT FROM, IS [NOT] UNKNOWN, SIMILAR TO / NOT SIMILAR TO, LIKE ... ESCAPE '<char>'.

These fail validation on the current release, so no existing pipeline can depend on them. Safe to enable unconditionally.

2. Crash → 3-valued logic, non-NULL behavior unchanged — keep always-on

SQL: AND, OR, NOT, IS [NOT] TRUE, IS [NOT] FALSE where the operand is a NULL boolean (nullable BOOLEAN column or CAST(NULL AS BOOLEAN)).

These previously threw an NPE on NULL; results for non-NULL operands are identical. Pure crash fix, safe to enable unconditionally.

3. Silent result changes — must be gated behind an explicit option

SQL:

Comparison / set predicates: =, <>, >, >=, <, <=, IN, NOT IN, BETWEEN, NOT BETWEEN. On a NULL operand the result changes from false/true to NULL. For negated forms (<>, NOT IN, NOT BETWEEN) this is true → NULL, which drops rows in WHERE and flips CASE / IF branches.
LIKE / NOT LIKE: pattern engine changes from Java regex (Pattern.find(), substring match) to SQL wildcards (%, _, whole-string match). This changes results for every row, not just NULL — e.g. 'xabcy' LIKE 'abc' goes true → false, col LIKE 'A%' goes false → true.

This is a backward-incompatible change to user-facing semantics, and the silent nature makes it dangerous: after the upgrade the YAML is unchanged, the job stays healthy, and no error or warning is emitted, yet the output data changes. For a CDC sync, altered filter results change which INSERT / UPDATE / DELETE events reach the sink, so the sink silently diverges from the source with no signal to detect or repair it. It therefore must not be the default on a minor/patch release.

Recommendation

Introduce pipeline.transform.expression.semantics (enum LEGACY | FLINK_SQL, default LEGACY). Keep the current behavior under LEGACY; apply bucket 3 only under FLINK_SQL. Buckets 1 and 2 remain always-on.

Also, to ease review, consider splitting this so buckets 1–2 and bucket 3 land as separate commits (or separate PRs) — the crash/parse fixes can then be merged quickly, while the semantic change is reviewed on its own.

haruki-830 · 2026-07-17T02:08:14Z

Also, to ease review, consider splitting this so buckets 1–2 and bucket 3 land as separate commits (or separate PRs) — the crash/parse fixes can then be merged quickly, while the semantic change is reviewed on its own.

Thanks for the detailed review and the compatibility analysis. I agree with the split.

I will separate this work into two PRs:

The first PR will only include the always-on changes with no silent behavior change:
- newly supported operators that are currently unparseable, such as IS [NOT] DISTINCT FROM, IS [NOT] UNKNOWN, SIMILAR TO / NOT SIMILAR TO, and LIKE ... ESCAPE
- NULL boolean crash fixes for AND, OR, NOT, IS [NOT] TRUE, and IS [NOT] FALSE
The second PR will handle the backward-incompatible semantic changes behind an explicit option:
- comparison predicates, IN, BETWEEN, and legacy LIKE / NOT LIKE
- add pipeline.transform.expression.semantics with LEGACY as the default and FLINK_SQL as the opt-in SQL-compatible behavior

This should allow the parse/crash fixes to move forward independently, while keeping the silent result-changing semantics isolated and explicitly gated.

…nullable BOOLEAN logical evaluation

lvyanquan · 2026-07-17T09:51:49Z

+            throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+        }
+        Java.Rvalue rightOperandSupplier =
+                new Java.AmbiguousName(Location.NOWHERE, new String[] {"() -> " + atoms[1]});


Could we add an end-to-end compilation test to JaninoCompilerTest?

The existing tests only verify the generated expression string or invoke
LogicalFunctions directly. They do not compile the generated expression with
Janino, so they cannot detect that Janino 3.1.10 does not support lambda
expressions.

Addressed in the latest commit.

lvyanquan · 2026-07-17T10:30:32Z

Please update both the English and Chinese transform documentation to reflect these changes

lvyanquan · 2026-07-20T06:18:48Z

Also cc’ing @yuxiqian.

lvyanquan · 2026-07-21T03:01:20Z

-| value IS NULL                        | null == value                                | 如果 value 为 NULL，返回 TRUE。                                      |
-| value IS NOT NULL                    | null != value                                | 如果 value 不为 NULL，返回 TRUE。                                     |
+| value IS NULL                        | isNull(value)                                | 如果 value 为 NULL，返回 TRUE。                                      |
+| value IS NOT NULL                    | isNotNull(value)                             | 如果 value 不为 NULL，返回 TRUE。                                     |


The following functions are not documented:

IS DISTINCT FROM

IS NOT DISTINCT FROM

LIKE ... ESCAPE

NOT LIKE ... ESCAPE

SIMILAR TO

NOT SIMILAR TO

SIMILAR TO ... ESCAPE

NOT SIMILAR TO ... ESCAPE

Moreover, the two forms of LIKE are handled differently:

Two-argument LIKE: uses Java regular expression semantics.

LIKE with ESCAPE: uses SQL % and _ wildcard semantics.

lvyanquan · 2026-07-21T03:23:09Z

+                new Java.AmbiguousName(
+                        Location.NOWHERE,
+                        new String[] {
+                            "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "


A possible optimization is to generate different code based on operand nullability:

If both operands are non-null, use native &&/||.

If only the left operand is non-null, use a conditional expression:
- AND: left ? right : Boolean.FALSE
- OR: left ? Boolean.TRUE : right

If the left operand is nullable, keep the current Supplier fallback to preserve three-valued logic and short-circuit evaluation.

This avoids unnecessary per-record Supplier allocations in common non-null cases.

We may add a test spec like true OR 1/0, false AND 1/0 to ensure the short-circuit logic is effective.

Added them in the latest commit. Thanks!

lvyanquan · 2026-07-21T03:27:12Z

+    DataChangeEvent{tableId=foo.bar.baz, before=[1, false, true, true, false], after=[-1, true, false, true, false], op=UPDATE, meta=()}
+    DataChangeEvent{tableId=foo.bar.baz, before=[-1, true, false, true, false], after=[], op=DELETE, meta=()}
+    DataChangeEvent{tableId=foo.bar.baz, before=[], after=[0, true, false, false, true], op=INSERT, meta=()}
+    DataChangeEvent{tableId=foo.bar.baz, before=[0, true, false, false, true], after=[], op=DELETE, meta=()}


Lacks end-to-end tests for:

NOT LIKE ... ESCAPE

SIMILAR TO ... ESCAPE

NOT SIMILAR TO ... ESCAPE

NULL inputs for these expressions

haruki-830 · 2026-07-21T08:41:09Z

@lvyanquan Thanks for the detailed feedback.

I split the follow-up work into separate JIRA issues:

For this PR, I kept the scope limited to the always-on changes that do not introduce silent behavior changes. The backward-incompatible semantic alignment work, such as comparison / IN / BETWEEN / two-argument LIKE behavior, will be handled under the explicit semantics option in a follow-up PR.

For nullable AND / OR, I applied the localized optimization suggested here: native && / || for non-null operands, conditional expressions when only the left operand is non-null, and the existing lazy fallback when the left operand is nullable. Avoiding that fallback completely would require statement-level codegen, so I tracked it separately instead of expanding this PR into an expression translator refactor.

yuxiqian

Thanks Haruki for the contribution, just left some minor comments.

yuxiqian · 2026-07-21T09:51:23Z

+    private static boolean isExpressionNullable(Context context, SqlNode sqlNode) {
+        if (sqlNode instanceof SqlIdentifier) {
+            return isIdentifierNullable(context, (SqlIdentifier) sqlNode);
+        }
+        if (sqlNode instanceof SqlLiteral) {
+            return ((SqlLiteral) sqlNode).getValue() == null;
+        }
+        if (sqlNode instanceof SqlBasicCall) {
+            return isBasicCallNullable(context, (SqlBasicCall) sqlNode);
+        }
+        return true;
+    }


Instead of deducing isIdentifierNullable at the Janino compiler level, can we check the corresponding RelNode nullability during Calcite validation?

Current implementation is OK though, as the compilation happens just once.

Thanks for the suggestion. I agree that deriving nullability from Calcite/RelNode would be cleaner.

Since the current check only runs once during expression compilation and is conservative for unknown expressions, I kept it local in this PR to avoid expanding the scope. I think it would be better to revisit this in the follow-up PR for statement-level Transform expression codegen.

haruki-830 marked this pull request as ready for review July 15, 2026 10:07

github-actions Bot added composer runtime labels Jul 15, 2026

haruki-830 changed the title ~~[FLINK-40151][runtime] Align YAML Transform BETWEEN, IN, and LIKE semantics with Flink SQL~~ [FLINK-40151][runtime] Support additional transform predicates and fix NULL boolean evaluation Jul 17, 2026

[FLINK-40151][runtime] Support missing YAML Transform predicates and …

9896f89

…nullable BOOLEAN logical evaluation

haruki-830 force-pushed the FLINK-40151 branch from e6db6ad to 9896f89 Compare July 17, 2026 03:17

haruki-830 changed the title ~~[FLINK-40151][runtime] Support additional transform predicates and fix NULL boolean evaluation~~ [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation Jul 17, 2026

春栖 added 2 commits July 17, 2026 13:40

fix

8ec3e6b

Preserve short-circuit evaluation for nullable AND/OR

3311c2e

lvyanquan reviewed Jul 17, 2026

View reviewed changes

address review comments

67ba8df

github-actions Bot added the docs Improvements or additions to documentation label Jul 19, 2026

lvyanquan reviewed Jul 21, 2026

View reviewed changes

fix

59c93a2

haruki-830 force-pushed the FLINK-40151 branch from 61c3306 to 59c93a2 Compare July 21, 2026 06:51

fix

9215a44

yuxiqian reviewed Jul 21, 2026

View reviewed changes

Add short-circuit logical operator spec

0f479aa

Uh oh!

Conversation

haruki-830 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Uh oh!

haruki-830 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lvyanquan commented Jul 16, 2026

Uh oh!

haruki-830 commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lvyanquan commented Jul 17, 2026

Uh oh!

lvyanquan commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haruki-830 commented Jul 21, 2026

Uh oh!

yuxiqian left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haruki-830 commented Jul 15, 2026 •

edited

Loading

haruki-830 commented Jul 15, 2026 •

edited

Loading