Add an isolated Flink 1.18 build with host compatibility adapters - #198
Merged
Merged
Conversation
Build Flink 1.18.1 from explicit compatibility source roots and qualified Maven coordinates while keeping the existing 2.2 defaults. Reject mixed dependency lines and payload identities before execution, and preserve each host's deduplication, TTL emission, JSON decimal/array and Parquet map semantics. SQL/JSON still crosses to the host once per Arrow batch. Keep one native state encoding. Use a temporary heap projection for 1.18 canonical savepoints through released APIs; decline unverified stock RocksDB keyed paths at planning time. Preserve configured partitioner copies during checkpoint rescaling, and release failed snapshot uploads without registering an incomplete reuse base. Keep absent host constructs explicit in the parity matrix. Isolate upstream source builds and execution witnesses by Flink line, and add blocking Java/module/Paimon and optimized-artifact checks for the new development target. Production release and cross-line cluster validation remain separate gates. Validation so far: 3,469 default-line runtime/ORC invocations without failures; targeted 1.18 SQL, state, format and checkpoint/rescaling tests; 11 packaged-loader and payload-identity tests; a clean 14-project optimized release build with source/Javadoc attachments and exact artifact-boundary checks; deliberate wrong-line dependency rejection; native semantic regression tests and strict docs. Full 1.18 CI, upstream witnesses and release benchmarks continue on the draft PR. Refs #182, #183, #184, #185, #186, #187, #188, #189.
Flattening must retain the Arrow runtime dependencies inherited from the build parent so standalone consumers and untouched upstream test JVMs can load the same payload. Select inherited dependency flattening explicitly and share one lifecycle execution across source-suite and release profiles. Validate module identity, dependency line and inherited Arrow scopes beside the packaged JARs. The Flink 1.18 artifact audit and all three unchanged Calc smoke cases pass, with three confirmed native execution contracts.
…outes Flink 1.18 changelog replay derives key groups from serialized keys, so decline native keyed planning instead of bypassing the durability wrapper. Preserve randomized upstream settings and select explicit contracts for changelog and legacy lookup providers. Choose the native-state manifest at JVM startup and normalize old Surefire class names without losing per-invocation evidence. Verified six admission regressions, 24 harness tests, 42 unchanged upstream runtime invocations (20 native, 22 expected fallback), and 22 upstream state invocations (10 native, 12 expected fallback). Native RocksDB aggregation, Top-N and windows process real upstream rows.
…ries Select the official base image and qualified artifacts by Flink line, reject mismatched unpacked distributions before installation, and keep the submitted smoke job free of engine classes and native libraries. Both blocking image jobs submit native scalar and RocksDB aggregate SQL with the line's configuration key. The 1.18 release archive and installer were exercised locally, both image fixtures compile, and CI retains the optimized 1.18 Linux archive for inspection. Archive creation excludes macOS metadata sidecars. Real cross-line savepoint upgrade remains a separate release gate.
The new Flink 1.18 image gate exposed Linux symbols requiring GLIBC_2.38 from the Ubuntu 24.04 build, while the released image supplies 2.35. Build CI and published Linux payloads on Ubuntu 22.04 and use the released Bullseye Rust image for portable container builds. Use Flink 1.18’s provided SLF4J 1.7 API so Arrow cannot replace the host logging provider. Artifact checks reject bundled logging classes on that line and the image job verifies a real provider before SQL execution. The existing image check reproduced the ABI failure; the fixed optimized image run remains the validation gate on this draft port.
Capture the released planner’s proven legacy upsert keys before Arrow substitution so joins, ranks and aggregates retain the same sink contract. Count completed native join work for unchanged upstream variants and require explicit fallback evidence for unsupported state backends. Run the seven admitted Flink 1.18 suites alongside the existing eight Flink 2.2 suites, rejecting stale or mixed-line classpaths before execution. Kafka 3.2.0 uses its final candidate tag, verified against the published source archive. Full state runs require every contracted method and are included in the all command. Validation: 20 upstream runtime and 20 native-state invocations pass (8 and 13 native witnesses); 13 agent tests, 24 summarizer tests, 5 classpath tests and 7 local planning/admission tests pass. The unchanged ten-query Nexmark perimeter retains both transposes. Strict documentation build passes. The interrupted broad 1.18 runtime run is diagnostic only; full baselines remain gated in CI. Refs #184, #187, #189.
Moving a job to an older runner does not invalidate Rust caches by itself. Give Ubuntu 22.04 native builds a distinct ABI cache key and inspect every packaged Linux DSO before image execution, rejecting dependencies above GLIBC_2.35 and unknown private ABI requirements. Validation: all ten existing Linux ARM64 native payloads satisfy the check; three focused tests cover the released baseline, newer numeric versions and unknown requirements. Workflow parsing verifies every Ubuntu 22.04 native job has the separate key. Shell syntax and whitespace checks pass. Refs #183, #189.
Kafka 3.2 ships without a Maven wrapper. Paimon 2.0 compiles shared Flink-1 code on its released 1.20.1 baseline, then the audit selects 1.18.1 dependencies when executing its unchanged tests. Keep all selected state classes mandatory while requiring native evidence for the explicitly contracted methods; an uncontracted class must execute instead of failing an impossible contract lookup. The 25 suite guard tests and strict documentation build pass. Connector CI must validate the adjusted build/runtime split. The unchanged release+mimalloc Nexmark harness ran ten 2M-row queries on Flink 1.18 with both transposes: native/Flink speedups are 0.28x–0.88x, so this remains compatibility groundwork with no performance claim; full results and limitations are recorded.
Arrow brings SLF4J 2 transitively, but released Flink distributions own a SLF4J 1.7 provider. Bundling that newer API silently selects a no-operation logger in the deployed image. Compile against the host API as provided, reject packaged logging classes, and require an active provider before image SQL runs. Validated the runtime package with the packaged core first on a Flink-compatible logging classpath, eight native bridge tests, and a strict documentation build. Refs #189.
JNI loads native libraries after the JVM and other plugins have reserved thread-local storage. Mimalloc's initial-exec TLS can then fail at load time, especially when separate classloaders extract another copy. Enable its released local-dynamic TLS option while preserving library-local allocation and the JVM's allocator. Reject static TLS flags and x86_64/arm64 relocations in packaged Linux libraries. A compiled Linux regression reproduces the static TLS dlopen failure and loads four dynamic copies successfully. Optimized mimalloc build, eight native bridge tests, workspace validation and strict docs pass. Refs #189.
The isolated deployed planner sees a different Class instance for the final StreamFusion RocksDB backend than the host that creates it. Match that verified backend identity across the boundary so Flink 1.18 does not incorrectly reject native keyed plans; retain stock RocksDB and changelog-state exclusions. The packaged-loader regression reproduces the prior fallback with a normally configured host backend. All twelve 1.18 loader and identity tests now pass, including the native aggregate plan. Refs #183, #188.
Flink 1.18 copies custom ArrayData views into a reused serializer buffer. Taking an owned binary row before buffering prevents later Paimon rows from overwriting arrays retained across native/Java split restores. Keep Flink 2.2's corrected copy behavior unchanged. Run compaction fixtures with explicit table-level batch mode and portable positional arguments, retaining the same batch full-compaction behavior on both releases. All 352 selected snapshot/sink cases pass on 2.2; on 1.18, 341 pass in the broad run and the remaining 11 compaction cases pass in the focused rerun. Refs #187.
Paimon's common suite compiles against 1.20 APIs, while its versioned 1.18 connector supplies compatibility types absent from the Flink runtime. Add that already-resolved released connector artifact to the test classpath so CatalogMaterializedTable and related compatibility classes can link without installing a newer Flink runtime or changing upstream tests. The prior CI run reproduced missing CatalogMaterializedTable before SQL execution. Released artifact contents, line-selection tests, shell syntax and strict docs were checked; the corrected upstream SQL run remains required. Refs #187, #189.
The shared Paimon sources require Flink 1.20 to compile, but their managed-memory helper cannot run on Flink 1.18. Generate a test-only Maven model that keeps the untouched compiled tests and original working directory while placing the released 1.18 connector first and removing the incompatible common main output. Resolve the connector through canonical Maven coordinates; do not patch upstream code or add 1.20 runtime classes. Validation: the unchanged managed-memory regression and two native write/read witness tests pass against the packaged release, including native bundle, level-0 and snapshot-merge evidence. All 26 harness checks and strict docs pass. Refs #187 and #189.
Use the existing collection-source adapter for shared nested fixtures. Flink 1.18 cannot generate JSON_QUERY with a column-valued path, so require the same planning failure before exercising nested values with a literal selector. Preserve dynamic-path tests on 2.2 and keep nested ownership and changelog checks active on both lines. Validation: all 41 selected JSON/JNI cases pass on 2.2; 39 pass on 1.18 with two existing host-capability skips. All 15 nested cases execute on each line. Strict docs pass. Refs #184 and #189.
Flink 1.18 keeps statement hints outside the resolved catalog options. Merge the physical sink hints before table-scoped job overrides so unsupported writer options fall back and branch inserts cannot write into main. Preserve Paimon's precedence on both release lines. The new admission and branch regressions fail against the old 1.18 resolver. All three focused checks pass on 1.18 and 2.2, with native sink plan and native file-footer evidence on both branches. Both unchanged upstream branch tests also pass. This is a correctness repair, with no throughput claim. Refs #187, #189.
Preserve catalog metadata through the released factory API and translate moved checkpoint settings without changing upstream SQL or result assertions. Apply the legacy restore settings to stream graphs only inside the recovery and bucket-rescaling fixtures, since Flink 1.18 does not propagate mutable table restore options to the executor. Otherwise several tests falsely pass without restoring state. Retain per-test progress and delayed thread dumps across Paimon SQL fixtures. All 21 agent checks and strict docs pass. A combined unchanged upstream run passes 22 cases with actual savepoint restores and native bundle, level-zero file and snapshot-read evidence. The complete 262-case baseline is still not green; document named-procedure and overwrite-layout failures reproduced without StreamFusion and the remaining historical-schema reliability check. Refs #187, #189.
…e compatibility line
Retain the key-selector copy helper without requiring the interface method introduced after Flink 1.18. The newer host still dispatches the same method. All 153 selected SQL, checkpoint and rescaling tests pass on 1.18 with a fresh native build; the 32 affected fixture tests also pass on 2.2. Strict docs pass. Refs #184, #189.
Retain line-specific fixture names and required-class checks while adopting the complete execution denominator from main. Harness verification: 33 tests pass; strict documentation build passes.
Use the complete version-specific upstream module, as Paimon does in its own 1.x CI, plus eleven shared methods covering recovery, branch isolation and native reads and writes. The shared 1.20 corpus requires named procedures absent from the released 1.18 host; keep its failed stock-only probe documented rather than rewriting SQL or assertions. Keep both corpora in separate JVMs and report directories, require every selected method and class to execute, and preserve process failures and all native data-path gates. A fresh source build passes 71 cases with four upstream named-argument skips, with bundle, level-zero and snapshot witnesses. The summary and POM regression tests and strict documentation build pass.
Record the allocator and portable-glibc prerequisite while preserving the already-equivalent dual-line artifact checks and CI matrix.
Include the verified native and exact-fallback routes from the UDF audit while keeping Flink 1.18 contracts separately scoped. All 21 agent tests and the strict docs build pass on the combined branch.
jordepic
marked this pull request as ready for review
September 19, 2026 12:54
2 tasks
Both line-specific matrices report their own upstream results and execution contracts. Passing those checks establishes the tested compatibility boundary; real-cluster cross-line upgrades and publishing remain separate release requirements.
Kafka 3.2 loads its planner in an isolated child loader, leaving the injected native planner unable to resolve the Flink optimizer. Use the same unshaded planner as the other upstream suites and preserve Flink Calcite precedence without changing upstream source, SQL, or assertions. Validation: all 72 unchanged Kafka 3.2 SQL cases pass on Flink 1.18.1 with real Kafka containers, zero failures and zero skips. The full 1.18 runtime baseline passes 5661 cases with 25 upstream skips and all 65 execution contracts. Line-selection tests, shell syntax, and strict docs pass.
This was referenced Sep 19, 2026
This was referenced Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an experimental Flink 1.18.1 / Java 17 build alongside the unchanged Flink 2.2 defaults. The profile selects matching released dependencies, compatibility sources and line-qualified Maven coordinates together. Payload metadata and dependency enforcement reject mixed installations.
Compatibility adapters cover planner shapes, lookup lifecycle, state TTL, format semantics, Parquet map schemas and checkpoint recovery. SQL/JSON retains one JVM callback per Arrow batch. Native state keeps a shared encoding; 1.18 canonical savepoints use a temporary heap projection through released APIs. Unsupported stock RocksDB/changelog configurations fall back during planning. The Kernel-based Delta implementation remains 2.2-only; the separate legacy Delta audit tests the host connector on 1.18.
Both lines have blocking Java, format/connector, Paimon Parquet/ORC, optimized image and upstream CI. Builds, classpaths and execution evidence are isolated by line. Flink 1.18 Paimon uses its complete version-specific upstream module plus eleven shared methods covering catalog construction, native writes/reads, branch isolation, schema history, savepoint recovery and bucket rescaling. These run unchanged in separate JVMs with required class/method execution and native data-path evidence. Paimon's shared 1.20 corpus is a different contract: its named-procedure and overwrite-layout failures reproduce on stock 1.18, and the failed cross-version probe remains documented.
Validation completed:
Full unchanged 1.18 upstream runtime suite: 5,686 cases, 5,661 passed and 25 upstream skips, with zero failures/errors. All 65 execution contracts passed: 31 native and 34 expected fallback. Other cases remain explicitly unclassified.
Full unchanged 1.18 upstream Parquet: all eight cases pass with the required native-writer witness.
Full unchanged 1.18 upstream formats: 168 cases, 158 passed and ten upstream skips, zero failures/errors; the pinned Avro-Confluent module also compiles (it has no ITCase classes).
Full unchanged Kafka 3.2 SQL suite on 1.18: all 72 cases pass with real Kafka containers, zero failures and zero skips after correcting the legacy harness planner classpath. Upstream sources and assertions are unchanged.
Fresh 1.18 upstream Paimon build and combined run: 75 cases, 71 passed and four upstream named-argument skips; native bundle, level-zero writer and snapshot-reader witnesses present.
Full 1.18 upstream state CI: 1,120 passed and 16 upstream skips; 30 contracted executions, including 18 native and 12 expected-fallback witnesses.
Changing-bound Top-N on 1.18: 153 SQL/checkpoint/rescaling cases passed with a fresh native build; the affected fixture's 32 cases also pass on 2.2.
Fresh full 1.18 runtime/ORC run: 3,534 cases, 3,192 passed and 342 host-capability/opt-in skips, with zero failures/errors. Fresh full 2.2 runtime/ORC run: 3,549 cases, 3,515 passed and 34 skips, with zero failures/errors. Nested JSON: 41 selected cases pass on 2.2; 39 pass with two host-capability skips on 1.18.
Fresh 14-project qualified 1.18 release packaging, source/Javadoc attachments, exact artifact/consumer-POM boundaries, release-archive assembly and all 12 packaged-loader/identity tests pass. Archive installation and mixed-payload rejection were also validated locally. Deliberately inserting a 2.2 dependency fails the 1.18 enforcer.
Fresh 1.18 Paimon module runs: all 677 cases pass with Parquet and all 677 pass again with ORC. The other connector/format modules compile, with their tagged Java cases included in the full runtime run above; standalone CI package checks remain required.
Current combined suite agent: 21 tests pass. Summary/POM/line-selection regressions and strict documentation build pass.
All 47 checks passed on
24670e3ebefore mergec90f5a89, including both required aggregate gates. Linux CI confirms the complete 1.18 runtime baseline above (5,661 passed, 25 upstream skips; all 65 execution contracts), the 2.2 runtime baseline (8,571 passed, 48 upstream skips; all 77 contracts), and all 72 unchanged legacy Kafka SQL cases. A RAW module dependency-download failure passed its targeted retry; no test selection or merge requirement was weakened.This ships compatibility groundwork, not production support or a performance improvement. The committed release+mimalloc Nexmark comparison keeps both transposes and all ten queries are slower than stock 1.18 (0.28–0.88×). Real cross-line cluster upgrades, Podman validation and production publishing remain separate gates. No release is published by this PR.
Includes the allocator/glibc prerequisite from #204 and UDF execution audit from #206. Startup rejection before a container daemon launches is stacked separately in #200.
Refs #182, #183, #184, #185, #186, #187, #188, #189.