Verified on PR #220 after rebasing onto merged #222: commit 16ebb1e5a45cc5420abf90473b785d6c0b908a82, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each probe uses independent stock-Flink and native-enabled environments with bounded runtime DataStream input. Successful comparisons check output values and resolved result types before classifying execution.
Suggested priority: P2. This is a verified native support gap; the successful probes return equal results through explicit Flink fallback.
ENCODE/DECODE currently require a literal charset, so a runtime STRING charset column makes the Calc fall back even for standard supported charsets.
Runtime probe
Input type:
ROW<`f0` STRING, `f1` STRING, `f2` BIGINT, `f3` STRING, `f4` STRING, `f5` STRING, `f6` BIGINT> NOT NULL
Input: [+I[a,b,,c, ,, 1, UTF-16LE, 2024-02-29, yyyy-MM-dd, 1709164800]]
SELECT ENCODE(f0,f3) FROM src;
Flink result: [[61002c0062002c002c006300]]; resolved type: [BYTES]. Native substitutions: 0.
Calc: ENCODE requires a literal charset
Runtime probe
Input type:
ROW<`f0` STRING, `f1` STRING, `f2` BIGINT, `f3` STRING, `f4` STRING, `f5` STRING, `f6` BIGINT> NOT NULL
Input: [+I[a,b,,c, ,, 1, UTF-16LE, 2024-02-29, yyyy-MM-dd, 1709164800]]
SELECT DECODE(ENCODE(f0,'UTF-8'),f3) FROM src;
Flink result: [[ⱡⱢ挬]]; resolved type: [STRING]. Native substitutions: 0.
Calc: DECODE requires a literal charset
Verify UTF-8, UTF-16LE/BE, supported non-UTF charsets, NULLs, malformed byte sequences and invalid charset-name errors using the runtime JDK contract. Preserve exact bytes and Java replacement grouping; a charset conversion must not be implemented as generic UTF-8 decoding.
Acceptance
Preserve released Flink values, resolved types, NULL behavior and relevant error/short-circuit semantics. Add runtime-source SQL regressions that assert the specific native operator as well as result parity; do not use constant folding as proof of native support. Keep safe fallback for unverified forms and update the corresponding operator coverage documentation.
Verified on PR #220 after rebasing onto merged #222: commit
16ebb1e5a45cc5420abf90473b785d6c0b908a82, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each probe uses independent stock-Flink and native-enabled environments with bounded runtime DataStream input. Successful comparisons check output values and resolved result types before classifying execution.Suggested priority: P2. This is a verified native support gap; the successful probes return equal results through explicit Flink fallback.
ENCODE/DECODE currently require a literal charset, so a runtime STRING charset column makes the Calc fall back even for standard supported charsets.
Runtime probe
Input type:
Input:
[+I[a,b,,c, ,, 1, UTF-16LE, 2024-02-29, yyyy-MM-dd, 1709164800]]Flink result:
[[61002c0062002c002c006300]]; resolved type:[BYTES]. Native substitutions:0.Runtime probe
Input type:
Input:
[+I[a,b,,c, ,, 1, UTF-16LE, 2024-02-29, yyyy-MM-dd, 1709164800]]Flink result:
[[ⱡⱢ挬]]; resolved type:[STRING]. Native substitutions:0.Verify UTF-8, UTF-16LE/BE, supported non-UTF charsets, NULLs, malformed byte sequences and invalid charset-name errors using the runtime JDK contract. Preserve exact bytes and Java replacement grouping; a charset conversion must not be implemented as generic UTF-8 decoding.
Acceptance
Preserve released Flink values, resolved types, NULL behavior and relevant error/short-circuit semantics. Add runtime-source SQL regressions that assert the specific native operator as well as result parity; do not use constant folding as proof of native support. Keep safe fallback for unverified forms and update the corresponding operator coverage documentation.