Skip to content

Record the Kafka connector release each Flink line admits #207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Flanderzz wants to merge 1 commit into
base: main
Choose a base branch
from
+32 −0
Open

Record the Kafka connector release each Flink line admits #207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/connectors/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,34 @@ The `streamfusion-kafka` connector extension and the matching `streamfusion-*` f
must both be installed. A missing extension is a planner fallback, never a linkage failure. See
[Deployment](../deployment.md).

## Flink release lines

Each Flink line selects the Kafka connector release published for it, together with the Kafka
client that connector was built against.

| | Default build | `flink-1.18` build |
| --- | --- | --- |
| Flink | 2.2.1 | 1.18.1 |
| Kafka connector | `flink-connector-kafka:5.0.0-2.2` | `flink-connector-kafka:3.2.0-1.18` |
| Kafka client | 4.2.0 | 3.4.0 |

Pinning the client matters because a direct dependency is needed for source compilation, and
naming another version silently overrides the connector's transitive client and changes producer
semantics.

`3.2.0-1.18` is built against Flink **1.18.0** while the StreamFusion 1.18 build targets
**1.18.1**, and it is the final Kafka connector release published for the 1.18 line. Later
connector releases target 1.19 and newer, so Kafka fixes do not reach this line. That combination
is admitted deliberately rather than by default: it is exercised on every change by the
`streamfusion-kafka` module suite under `-Pflink-1.18`, and by Flink's own unchanged
`DynamicKafkaTableITCase`, `KafkaChangelogTableITCase`, `KafkaTableITCase` and
`UpsertKafkaTableITCase` running against real brokers in the upstream `kafka` suite on the 1.18
line. Both prove native execution rather than only passing.

Because Flink owns every broker interaction, a Kafka defect on this line is fixed by the
connector, not by StreamFusion. Deployments that need a newer Kafka connector need a newer Flink
line.

## Source: Flink consumption, native decode

Flink continues to own topic enumeration, split assignment, offsets, checkpoints, authentication,
Expand Down
4 changes: 4 additions & 0 deletions docs/flink-compatibility.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ Install one complete line; a mixed install is an error.
Both payload lines use the host's SLF4J 1.7 API and binding; it does not bundle Arrow's transitive
SLF4J 2 API into Flink's global classpath.

The 1.18 Kafka connector is the one dependency whose own target differs from the line it is used
on: `3.2.0-1.18` is built against Flink 1.18.0 and is the last release published for that line.
[Kafka](connectors/kafka.md) records what that admits and how it is verified.

The Kernel-based Delta implementation belongs only to the 2.2 source root and does not enter
the 1.18 compilation, Javadoc or source artifacts. There is no admitted native Delta connector on 1.18 yet. Do not build or install a 1.18 Delta payload;
this is unavailable functionality, not a verified host fallback.
Expand Down
Loading