Conversation
16 tasks
Evaluate exact unary functions and integral rounding inside the existing columnar Calc bridge. Flink owns resolved widths, decimal scales, minimum-integer overflow and dynamic truncation behavior; existing Rust decimal arithmetic stays in place. Validation: runtime SQL parity covers exact result schemas, numeric boundaries, NULLs, composed decimal consumers and all-filtered batches. Add release diagnostic cases for integral ABS and decimal SIGN. Closes #208
Use the batch expression bridge for mixed exact numerics, full-range timestamps and runtime strings from proven external Java-string conversions. Preserve existing Rust kernels and decline unknown or binary-backed string representations instead of changing Unicode ordering. Validation: four runtime SQL tests cover resolved schemas, supplementary Unicode, nullable mixed decimals, timestamp nanoseconds in UTC, Shanghai and Los Angeles, plus binary-backed input fallback. Include a release string-extrema diagnostic. Closes #209
Admit the released boolean overload through the existing searched-CASE lowering. Preserve null-condition selection, nullable branches and lazy evaluation without widening other result-type gates. Validation: runtime SQL/schema parity over 5003 rows covers nested branches, filter consumers, unselected division and all-filtered batches. Add a release diagnostic. Closes #210
Use Flink cast rules for BOOLEAN to STRING, VARCHAR and CHAR, including TRY_CAST. Preserve uppercase spelling, nulls, truncation and padding inside native Calc. Validation: two runtime SQL tests compare schemas and values for unbounded and bounded casts, TRY_CAST, filter consumers and empty batches. Add a release diagnostic. Closes #211
Evaluate explicit LIKE escapes and SIMILAR TO through Flink-generated batch expressions. Fuse AND/OR consumers to preserve row short-circuiting and keep unproven string-order representations outside the generated path. Validation: three runtime SQL tests cover dynamic escapes, negation, Unicode, NULLs, filters, empty batches and matching invalid-escape exception behavior. Add a release pattern diagnostic. Closes #212
Keep REGEXP, replacement, count, position and substring in the columnar Calc through Flink-generated expressions. Preserve Java patterns, UTF-16 positions, literal replacements, declared scalar types and invalid-pattern behavior. Validation: multi-batch SQL/schema parity exercises lookaround, backreferences, zero-width matches, Unicode, invalid dynamic patterns, NULLs, filters and empty results. Add a release count diagnostic. Closes #213
Use Flink-generated PARSE_URL evaluation for dynamic parts and optional query keys. Avoid normalization and decoding differences from alternate URL parsers. Validation: runtime SQL/schema parity covers all URL parts, spelling, explicit ports, dot segments, raw escaped queries, repeated and missing keys, invalid URLs, NULLs and empty batches. Add a release diagnostic. Closes #214
Generate PRINTF with the selected Flink runtime, retaining exact numeric formatting, locale and invalid-format behavior. Fuse consumers of formatted characters so isolated UTF-16 units are not encoded before comparisons; retain the existing operator-boundary guard. Validation: runtime SQL/schema parity covers dynamic formats, indexed arguments, precision, BIGINT boundaries, NULLs, invalid formats, surrogate comparisons, filters and empty batches. Include planner identity coverage and a release diagnostic. Closes #215
Use generated BTRIM, LTRIM and RTRIM for per-row sets from proven external Java-string sources. Keep literal kernels and fall back when binary-backed input behavior cannot be preserved across Arrow. Validation: runtime SQL/schema parity over 5003 rows covers whitespace-first sets, supplementary Unicode, empty and all-trimmed values, NULLs, filters and empty batches. Verify the binary-input fallback and add a release diagnostic. Closes #216
Evaluate Base64, classification, numeric split separators and dynamic SHA2 with released Flink code. Fuse operand computations to retain StringData representation and error behavior. Transport final decoded strings as Arrow Binary so arbitrary bytes never violate Utf8 invariants; keep unsafe relational and complex boundaries on Flink. Validation: 14 targeted tests pass, followed by all 5 string-helper tests including byte-exact RowData sink validation. Adds an IS_ALPHA release benchmark workload; no speedup is claimed. Closes #217.
Use released Flink-generated evaluation for binary STARTSWITH, ENDSWITH and integer-indexed ELT within native Calc. Keep byte arrays intact through the existing binary batch bridge, including repeated result columns and nested selection. Validation: 2 SQL integration tests pass over 5,003-row sources, covering empty and high-bit bytes, NULLs, index bounds, filtering and composition. Adds a release binary-predicate benchmark workload; no performance gain is claimed. Closes #218.
Generate REGEXP_EXTRACT_ALL and STR_TO_MAP with released Flink code, carrying array and character-map values through owned Arrow results. Preserve nested consumers, null keys and elements, regex delimiters, duplicate keys and filter-before-projection behavior. Retain sensitive string identity gates and prevent raw decoded STRING projections from feeding columnar sinks as Utf8. Complete cross-release regression guards for SQL absent from Flink 1.18 and document the final coverage and performance limits. Validation: 562 runtime SQL cases yield 525 native passes, 6 floating-point TRUNCATE fallback controls and 31 pre-existing both-error cases, with no mismatch or native-only failure. 268 Flink 2.2 tests pass; the 1.18 issue suites pass 24 tests with 11 host-unavailable cases marked N/A. Additional nullable-map tests pass on both released lines. Release diagnostics retain both transposes and matched-source controls: 13 representative expressions reach 0.630x to 0.913x stock Flink throughput at 200000 rows. This extends composable native-island coverage; it does not claim standalone scalar or full-pipeline speedups. Closes #219.
liuyongvs
force-pushed
the
feat/flink-function-parity-208-219
branch
from
September 23, 2026 03:05
cf27931 to
16ebb1e
Compare
This was referenced Sep 23, 2026
Native collection expressions: track remaining ARRAY/MAP functions, casts and aggregate results
#234
Open
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runtime SQL using common exact numeric, string, binary and collection functions previously fell back at Calc. This PR adds the verified overloads using released Flink expression generation through the existing batch JVM bridge, preserving result types, values, NULL behavior and error evaluation. BOOLEAN IF uses the existing lazy native CASE path.
The branch contains one commit for each issue:
Generated expressions keep intermediate computations and sensitive UTF-16 consumers together. Collection results use the whole-Calc JVM callback with owned Arrow outputs and filter-before-projection evaluation. Base64-decoded STRING results can contain arbitrary bytes: final scalar outputs use Arrow Binary and the row transpose preserves those bytes; they cannot feed a native columnar sink as Utf8. No Rust/JNI ABI or third-party dependency changes are required.
Admission remains conservative. Runtime string extrema and dynamic trim sets require proven Java-backed external DataStream inputs; unknown or binary-backed representations stay on Flink. Sensitive character results crossing relational operators and unverified complex boundaries retain fallback. DECIMAL FLOOR/CEIL and FLOAT/DOUBLE TRUNCATE are outside this scope. The operator remains columnar, while generated functions execute on the JVM.
Validation on canonical main
89da01cd, JDK 17 and released dependencies:docs/operators/calc-filter.md.Closes #208
Closes #209
Closes #210
Closes #211
Closes #212
Closes #213
Closes #214
Closes #215
Closes #216
Closes #217
Closes #218
Closes #219