Skip to content

Extend native Calc function coverage with released Flink semantics - #220

Open
liuyongvs wants to merge 12 commits into
mainfrom
feat/flink-function-parity-208-219
Open

liuyongvs wants to merge 12 commits into
mainfrom
feat/flink-function-parity-208-219

Conversation

@liuyongvs

Copy link
Copy Markdown
Collaborator

Runtime SQL using common exact numeric, string, binary and collection functions previously fell back at Calc. This PR adds the verified overloads using released Flink expression generation through the existing batch JVM bridge, preserving result types, values, NULL behavior and error evaluation. BOOLEAN IF uses the existing lazy native CASE path.

The branch contains one commit for each issue:

Issue Added coverage
#208 Exact unary minus, ABS/SIGN, integral FLOOR/CEIL/TRUNCATE
#209 STRING, mixed exact numeric and TIMESTAMP/LTZ extrema
#210 BOOLEAN IF results
#211 BOOLEAN to STRING/CHAR/VARCHAR casts
#212 LIKE ESCAPE and SIMILAR TO
#213 REGEXP and scalar regex helpers
#214 PARSE_URL
#215 PRINTF
#216 Dynamic BTRIM/LTRIM/RTRIM sets
#217 FROM_BASE64, numeric SPLIT_INDEX, dynamic SHA2 and classification
#218 Binary STARTSWITH/ENDSWITH/ELT
#219 REGEXP_EXTRACT_ALL and STR_TO_MAP

Generated expressions keep intermediate computations and sensitive UTF-16 consumers together. Collection results use the whole-Calc JVM callback with owned Arrow outputs and filter-before-projection evaluation. Base64-decoded STRING results can contain arbitrary bytes: final scalar outputs use Arrow Binary and the row transpose preserves those bytes; they cannot feed a native columnar sink as Utf8. No Rust/JNI ABI or third-party dependency changes are required.

Admission remains conservative. Runtime string extrema and dynamic trim sets require proven Java-backed external DataStream inputs; unknown or binary-backed representations stay on Flink. Sensitive character results crossing relational operators and unverified complex boundaries retain fallback. DECIMAL FLOOR/CEIL and FLOAT/DOUBLE TRUNCATE are outside this scope. The operator remains columnar, while generated functions execute on the JVM.

Validation on canonical main 89da01cd, JDK 17 and released dependencies:

  • Flink 2.2.1: 268 targeted integration/planner/bridge tests pass, including the twelve issue suites, JSON identity, temporal and UDF regressions, native Parquet boundaries and allocator checks.
  • Re-ran the same 562 runtime-source SQL cases used for the issue baseline: 525 native passes, 6 fallback passes, 31 pre-existing both-error cases, zero result/schema mismatches and zero native-only failures. All 305 newly admitted successful cases move from fallback to native. The six remaining fallbacks are floating-point TRUNCATE controls. Both-error cases are not counted as exception-equivalence proof.
  • Focused tests separately assert host/native root exception class, message and evaluation phase; exercise multi-batch NULL/Unicode/binary cases, invalid regex/format/escape/hash/Base64 inputs, short-circuiting and all-filtered batches. Raw decoded bytes are checked at the RowData sink.
  • Flink 1.18.1: 24 selected tests pass; 11 cases are explicitly N/A because the release lacks those SQL functions. Additional nullable-map boundary checks pass on both lines.
  • Release-only diagnostics use 200,000 rows, 64-byte payloads, parallelism 1, one warmup and three measured runs, matched-source controls and both transposes. The 13 representative expressions reach 0.630x–0.913x stock Flink throughput in this short row-fed workload. This is coverage work and does not claim standalone scalar or full-pipeline speedups; detailed results and retained gates are in docs/operators/calc-filter.md.

Closes #208
Closes #209
Closes #210
Closes #211
Closes #212
Closes #213
Closes #214
Closes #215
Closes #216
Closes #217
Closes #218
Closes #219

Evaluate exact unary functions and integral rounding inside the existing columnar Calc bridge. Flink owns resolved widths, decimal scales, minimum-integer overflow and dynamic truncation behavior; existing Rust decimal arithmetic stays in place.

Validation: runtime SQL parity covers exact result schemas, numeric boundaries, NULLs, composed decimal consumers and all-filtered batches. Add release diagnostic cases for integral ABS and decimal SIGN.

Closes #208
Use the batch expression bridge for mixed exact numerics, full-range timestamps and runtime strings from proven external Java-string conversions. Preserve existing Rust kernels and decline unknown or binary-backed string representations instead of changing Unicode ordering.

Validation: four runtime SQL tests cover resolved schemas, supplementary Unicode, nullable mixed decimals, timestamp nanoseconds in UTC, Shanghai and Los Angeles, plus binary-backed input fallback. Include a release string-extrema diagnostic.

Closes #209
Admit the released boolean overload through the existing searched-CASE lowering. Preserve null-condition selection, nullable branches and lazy evaluation without widening other result-type gates.

Validation: runtime SQL/schema parity over 5003 rows covers nested branches, filter consumers, unselected division and all-filtered batches. Add a release diagnostic.

Closes #210
Use Flink cast rules for BOOLEAN to STRING, VARCHAR and CHAR, including TRY_CAST. Preserve uppercase spelling, nulls, truncation and padding inside native Calc.

Validation: two runtime SQL tests compare schemas and values for unbounded and bounded casts, TRY_CAST, filter consumers and empty batches. Add a release diagnostic.

Closes #211
Evaluate explicit LIKE escapes and SIMILAR TO through Flink-generated batch expressions. Fuse AND/OR consumers to preserve row short-circuiting and keep unproven string-order representations outside the generated path.

Validation: three runtime SQL tests cover dynamic escapes, negation, Unicode, NULLs, filters, empty batches and matching invalid-escape exception behavior. Add a release pattern diagnostic.

Closes #212
Keep REGEXP, replacement, count, position and substring in the columnar Calc through Flink-generated expressions. Preserve Java patterns, UTF-16 positions, literal replacements, declared scalar types and invalid-pattern behavior.

Validation: multi-batch SQL/schema parity exercises lookaround, backreferences, zero-width matches, Unicode, invalid dynamic patterns, NULLs, filters and empty results. Add a release count diagnostic.

Closes #213
Use Flink-generated PARSE_URL evaluation for dynamic parts and optional query keys. Avoid normalization and decoding differences from alternate URL parsers.

Validation: runtime SQL/schema parity covers all URL parts, spelling, explicit ports, dot segments, raw escaped queries, repeated and missing keys, invalid URLs, NULLs and empty batches. Add a release diagnostic.

Closes #214
Generate PRINTF with the selected Flink runtime, retaining exact numeric formatting, locale and invalid-format behavior. Fuse consumers of formatted characters so isolated UTF-16 units are not encoded before comparisons; retain the existing operator-boundary guard.

Validation: runtime SQL/schema parity covers dynamic formats, indexed arguments, precision, BIGINT boundaries, NULLs, invalid formats, surrogate comparisons, filters and empty batches. Include planner identity coverage and a release diagnostic.

Closes #215
Use generated BTRIM, LTRIM and RTRIM for per-row sets from proven external Java-string sources. Keep literal kernels and fall back when binary-backed input behavior cannot be preserved across Arrow.

Validation: runtime SQL/schema parity over 5003 rows covers whitespace-first sets, supplementary Unicode, empty and all-trimmed values, NULLs, filters and empty batches. Verify the binary-input fallback and add a release diagnostic.

Closes #216
Evaluate Base64, classification, numeric split separators and dynamic SHA2 with released Flink code. Fuse operand computations to retain StringData representation and error behavior. Transport final decoded strings as Arrow Binary so arbitrary bytes never violate Utf8 invariants; keep unsafe relational and complex boundaries on Flink.

Validation: 14 targeted tests pass, followed by all 5 string-helper tests including byte-exact RowData sink validation. Adds an IS_ALPHA release benchmark workload; no speedup is claimed.

Closes #217.
Use released Flink-generated evaluation for binary STARTSWITH, ENDSWITH and integer-indexed ELT within native Calc. Keep byte arrays intact through the existing binary batch bridge, including repeated result columns and nested selection.

Validation: 2 SQL integration tests pass over 5,003-row sources, covering empty and high-bit bytes, NULLs, index bounds, filtering and composition. Adds a release binary-predicate benchmark workload; no performance gain is claimed.

Closes #218.
Generate REGEXP_EXTRACT_ALL and STR_TO_MAP with released Flink code, carrying array and character-map values through owned Arrow results. Preserve nested consumers, null keys and elements, regex delimiters, duplicate keys and filter-before-projection behavior. Retain sensitive string identity gates and prevent raw decoded STRING projections from feeding columnar sinks as Utf8.

Complete cross-release regression guards for SQL absent from Flink 1.18 and document the final coverage and performance limits. Validation: 562 runtime SQL cases yield 525 native passes, 6 floating-point TRUNCATE fallback controls and 31 pre-existing both-error cases, with no mismatch or native-only failure. 268 Flink 2.2 tests pass; the 1.18 issue suites pass 24 tests with 11 host-unavailable cases marked N/A. Additional nullable-map tests pass on both released lines.

Release diagnostics retain both transposes and matched-source controls: 13 representative expressions reach 0.630x to 0.913x stock Flink throughput at 200000 rows. This extends composable native-island coverage; it does not claim standalone scalar or full-pipeline speedups.

Closes #219.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment