Flink 1.18: RocksDB state backend, recovery and rescaling

[Flanderzz]

Shipped

PR #198 ships the Flink 1.18 RocksDB package/factory adapters, native incremental checkpoint support and canonical savepoint projection through released host APIs. The native snapshot encoding remains shared across lines. Failed and aborted snapshot cleanup shipped in #196. Both lines pass the Java backend/operator suites and unchanged upstream state suites; the 1.18 baseline is 1,120 passed and 16 upstream skips, including 18 native and 12 expected-fallback contracted executions.

The 1.18 canonical path temporarily projects native partitions into a heap backend so explicit key-group identities survive the released API. Unwrapped stock RocksDB and changelog configurations that cannot preserve this contract decline at planning time. See backend coverage.

Remaining work

• Prove CLAIM and NO_CLAIM restore ownership explicitly on both lines, including a subsequent checkpoint and disposal of source/restored handles without deleting live shared files.
• Assert managed-memory leases are released on backend close and failure paths.
• Complete the per-stateful-operator rescaling matrix in both directions, at least 1 to 2 and 2 to 1, checking that every key group appears exactly once and native leak accounting stays clean.

Use released Flink APIs and preserve the native snapshot layout. Keep evidence and per-line limitations in docs/backends/rocksdb.md. Real cross-line cluster upgrades remain tracked by #188.

Split from #182. This issue stays open for the ownership, lease and complete rescaling proof above.

Reconciled verification

Verified on main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f: CI and upstream suites both completed successfully, with 47 successful jobs in total. Results were rechecked on September 20, 2026. Existing checkpoint, canonical restore, cleanup and selected rescaling regressions are shipped. They do not complete the explicit CLAIM/NO_CLAIM ownership and lease-disposal assertions or the complete bidirectional per-operator rescaling matrix above. This is a remaining validation gap, not a finding that those implementation paths are broken.

[Flanderzz]

The state backend adaptation is on main in #198, and the shared checkpoint-ownership fix in #196.

The package move and the positional versus parameter-object factory signature are both handled behind the per-line compatibility sources (FlinkStateBackendCompat, RocksDBOptionsCompat, StateCompat). Native state bytes, key-group partitioning and checkpoint ownership are unchanged and shared across lines, which was the condition this issue set.

Verified by the existing state suites running unmodified on both lines. RocksDBNativeStateBackendOperatorTest, RocksDBNativeStateBackendAllOperatorsTest and FlinkRocksDBNativeStateBackendSqlHarnessTest carry no line gating, and the CI Java matrix runs the whole suite under both the default profile and -Pflink-1.18. The upstream state suite also runs on both lines.

Two user-visible constraints apply to Flink 1.18 and are deliberate, not gaps to reopen later:

1. Canonical savepoints project through heap on 1.18. That release's RocksDB backend cannot set a serialized key and an independent key group through its public API, so native canonical state is written through a temporary heap-backed projection. The projection owns its copied state before live entries are cleared, and it temporarily consumes JVM heap proportional to the canonical serialized state. Native incremental checkpoints keep their Rust-owned RocksDB path, and ordinary host operators keep the RocksDB delegate.

2. A stock RocksDB delegate cannot run native keyed operators on 1.18. It cannot carry StreamFusion's synthetic canonical key-group keys on that line, so the planner keeps keyed operators on Flink for that backend and other unverified custom backends, with an explicit fallback reason. Stateless operators remain eligible. Flink 1.18's changelog state wrapper is likewise a planning fallback for keyed native operators even when it wraps heap state, because its log replay recomputes key groups from serialized keys and does not preserve the explicit partition and group pairing. Keep state.changelog.enabled=false for native keyed execution on 1.18.

So a 1.18 native stateful job must use StreamFusion's native memory or native RocksDB backend. That is a narrower supported surface than 2.2, and it is documented rather than silently enforced.

Documentation is at docs/flink-compatibility.md under "State and recovery" and docs/backends/rocksdb.md.

Recovery and rescaling here are proven at operator-harness level. Real-cluster recovery and the cross-line upgrade path remain tracked in #188 and are not claimed by this issue.

Complete.