Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ anything running.
| `sink-column-flattened` | a sink table names its columns as the source does while a `Flatten` transform on the way delivers the envelope as `after.<column>`, so every column stays at its default | v0.3 |
| `mv-column-match` | ClickHouse streaming materialized views match by name, refreshable ones by position. ClickHouse 25.4 and later reject a streaming view that writes a column the target lacks when it is created; a refreshable view's order mismatch was loud on 24.8 and is silent on 26.8 | v0.1 |
| `schema-before-connector` | this change adds a column to a captured table and leaves it out of the stream without deciding to, the trap itself, judged on the diff (warning: leaving PII off is right, so it asks for the decision) | v0.2 |
| `replica-identity` | a captured table's replica identity cannot supply what the sink reads | next |
| `replica-identity` | a captured Postgres table has no replica identity (`NOTHING`, or `DEFAULT` with no primary key, after the connector's `replica.identity.autoset.values`): through `pgoutput` Postgres rejects the application's `UPDATE` and `DELETE` on it, through `decoderbufs` Debezium streams only its inserts (warning: an insert-only table loses nothing, and a `cdclint:ignore` says so) | next |
| `migration-numbering` | duplicate or gapped migration prefixes | next |

Findings name the file, the column, the rule and the fix:
Expand Down
7 changes: 7 additions & 0 deletions corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,16 @@ The rest exercise one path each:
| `ignore-marker-mistakes` | markers used wrong: one with no reason, one naming a rule that does not exist, one covering nothing, each an `ignore-marker` warning; and one on the line above a finding, which acknowledges it |
| `golang-migrate-down-files` | Mattermost v10.11.0's `000092_add_createat_to_teammembers.down.sql` drops a column from the wrong table; read in name order it ran just before its up file and deleted `reactions.createat`. Down files are skipped |
| `goose-down-section` | a goose file's down section follows its up section; applied whole, every table was created and dropped again. Down sections are skipped |
| `postgres-glued-constraints` | Mattermost v10.11.0 (000006 emoji, 000037 sharedchannelremotes): `UNIQUE(name, deleteat)` and `PRIMARY KEY(id, channelid)` with no space before the parenthesis are table constraints, not columns named after them; the primary key is kept |
| `include-table-no-schema` | Stack Overflow 74103659: `table.include.list` names `ipaddrs` without its schema, the connector runs and no topic appears |
| `include-table-glob` | Stack Overflow 51345636: `public.bg_*` written as a shell glob; as a regular expression it names no table |
| `include-table-typo` | a misspelled `table.include.list` entry, which Debezium only logs as a warning (debezium/dbz#872) |
| `replica-identity-no-primary-key` | a captured table with no primary key on `REPLICA IDENTITY DEFAULT`, read through the default `decoderbufs`: Debezium emits its inserts only |
| `replica-identity-nothing-pgoutput` | a keyed table set to `REPLICA IDENTITY NOTHING`, read through `pgoutput`: Postgres rejects `UPDATE` and `DELETE` on a published table with no replica identity |
| `replica-identity-autoset-nothing` | the connector's `replica.identity.autoset.values` sets `NOTHING` on a keyed table; the finding points at the connector, which is what to edit |
| `replica-identity-autoset-full` | the same keyless table as `replica-identity-no-primary-key`, with the connector setting `FULL`: nothing to report |
| `replica-identity-key-added-later` | the primary key arrives in a later `ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY`: nothing to report |
| `replica-identity-key-in-do-block` | Mattermost v10.11.0 `000016_create_reactions.up.sql`, as written: the key is added inside a `DO` block, which cdclint does not run, so it makes no finding rather than guess |
| `flatten-bare-columns` | ClickHouse/clickhouse-kafka-connect discussion 182: a `Flatten` transform on the Debezium connector sends `after.<column>`, the ClickHouse table names its columns as Postgres does, and every change lands as a row of zeros |
| `flatten-envelope-columns` | ClickHouse's own Postgres CDC example (ClickHouse/examples `cdc/postgresql` at ae417ae): `before.` and `after.` columns resolve to the source columns, so the capture rules see `postcode2`, which the include list added for this entry leaves out. The sink connector config there has no `connector.class`; it is added so the file names its sink |
| `flatten-after-unwrap` | a guard: unwrapping first leaves a flat row, so a `Flatten` after it renames nothing |
Expand Down
5 changes: 5 additions & 0 deletions corpus/flatten-bare-columns/migrations/0001_testradiomeas.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ CREATE TABLE testradiomeas (
"measNetNoCoverage" INTEGER,
"measNetType" TEXT
);

-- The discussion's sample message carries a whole "before" row (measDate,
-- measLongitude, ...) on a table with no primary key, which only REPLICA
-- IDENTITY FULL produces: with DEFAULT the connector would emit inserts only.
ALTER TABLE testradiomeas REPLICA IDENTITY FULL;
Original file line number Diff line number Diff line change
@@ -1,10 +1,20 @@
-- Reduced from Mattermost v10.11.0 000002 and 000016, keys as Mattermost
-- declares them: teammembers' in its CREATE TABLE, reactions' in a DO block
-- (cdclint does not run DO bodies, so it does not judge reactions' key).
CREATE TABLE reactions (
userid VARCHAR(26) NOT NULL,
postid VARCHAR(26) NOT NULL,
createat BIGINT
userid VARCHAR(26) NOT NULL,
postid VARCHAR(26) NOT NULL,
emojiname VARCHAR(64) NOT NULL,
createat BIGINT
);

DO $$
BEGIN
ALTER TABLE reactions ADD PRIMARY KEY (postid, userid, emojiname);
END $$;

CREATE TABLE teammembers (
teamid VARCHAR(26) NOT NULL,
userid VARCHAR(26) NOT NULL
userid VARCHAR(26) NOT NULL,
PRIMARY KEY (teamid, userid)
);
10 changes: 10 additions & 0 deletions corpus/postgres-glued-constraints/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"name": "mattermost-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"topic.prefix": "mm",
"table.include.list": "public.emoji,public.sharedchannelremotes",
"column.include.list": "public.emoji\\.(id|createat|updateat|deleteat|creatorid|name),public.sharedchannelremotes\\.(id|channelid|creatorid|createat|updateat|isinviteaccepted|isinviteconfirmed|remoteid|lastpostupdateat|lastpostid)"
}
}
1 change: 1 addition & 0 deletions corpus/postgres-glued-constraints/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ok: source, connector and sink agree
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
-- Mattermost v10.11.0, server/channels/db/migrations/postgres/
-- 000006_create_emojis.up.sql, reduced to the table. The table constraint
-- is written UNIQUE(name, deleteat), with no space before the parenthesis.
CREATE TABLE IF NOT EXISTS emoji (
id VARCHAR(26) PRIMARY KEY,
createat bigint,
updateat bigint,
deleteat bigint,
creatorid VARCHAR(26),
name VARCHAR(64),
UNIQUE(name, deleteat)
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-- Mattermost v10.11.0, 000037_create_sharedchannelremotes.up.sql, as
-- written: both table constraints are glued to their parenthesis.
CREATE TABLE IF NOT EXISTS sharedchannelremotes (
id character varying(26) NOT NULL,
channelid character varying(26) NOT NULL,
creatorid character varying(26),
createat bigint,
updateat bigint,
isinviteaccepted boolean,
isinviteconfirmed boolean,
remoteid character varying(26),
PRIMARY KEY(id, channelid),
UNIQUE(channelid, remoteid)
);

ALTER TABLE sharedchannelremotes ADD COLUMN IF NOT EXISTS LastPostUpdateAt bigint;
ALTER TABLE sharedchannelremotes ADD COLUMN IF NOT EXISTS LastPostId character varying(26);
28 changes: 28 additions & 0 deletions corpus/postgres-glued-constraints/sink/0001_mattermost.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-- Invented sink for the entry: two Kafka-engine tables reading a few of the
-- captured columns. The entry is about the source reader, not the sink.
CREATE TABLE kafka_emoji
(
`id` String,
`name` String,
`deleteat` Int64
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'mm.public.emoji',
kafka_group_name = 'clickhouse-emoji',
kafka_format = 'JSONEachRow';

CREATE TABLE kafka_sharedchannelremotes
(
`id` String,
`channelid` String,
`remoteid` String,
`lastpostupdateat` Int64
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'mm.public.sharedchannelremotes',
kafka_group_name = 'clickhouse-sharedchannelremotes',
kafka_format = 'JSONEachRow';
9 changes: 9 additions & 0 deletions corpus/replica-identity-autoset-full/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "app",
"table.include.list": "public.audit_events",
"replica.identity.autoset.values": "public.audit_.*:FULL"
}
}
1 change: 1 addition & 0 deletions corpus/replica-identity-autoset-full/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ok: source, connector and sink agree
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Invented for the entry: a table created without a primary key, as tables
-- meant only to be appended to often are, then captured like the rest.
CREATE TABLE audit_events (
actor_id UUID NOT NULL,
action TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
14 changes: 14 additions & 0 deletions corpus/replica-identity-autoset-full/sink/0001_audit_events.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_audit_events
(
`actor_id` String,
`action` String,
`created_at` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.audit_events',
kafka_group_name = 'clickhouse-audit_events',
kafka_format = 'JSONEachRow';
9 changes: 9 additions & 0 deletions corpus/replica-identity-autoset-nothing/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "app",
"table.include.list": "public.orders",
"replica.identity.autoset.values": "public.orders:NOTHING"
}
}
5 changes: 5 additions & 0 deletions corpus/replica-identity-autoset-nothing/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
warning replica-identity replica-identity-autoset-nothing/connector.json
public.orders is captured with REPLICA IDENTITY NOTHING, set by replica.identity.autoset.values in replica-identity-autoset-nothing/connector.json, so it has no replica identity
Debezium's update and delete events for it carry no previous values, so a delete has no before to say which row went
fix: take public.orders out of replica.identity.autoset.values, or set it to DEFAULT (it has a primary key) or FULL; if it only ever takes inserts, say so on the line that creates it: -- cdclint:ignore replica-identity: append-only
0 error(s), 1 warning(s), 0 info
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
-- Invented for the entry: a keyed table whose replica identity the
-- connector overrides with replica.identity.autoset.values.
CREATE TABLE orders (
id UUID PRIMARY KEY,
status TEXT NOT NULL
);
13 changes: 13 additions & 0 deletions corpus/replica-identity-autoset-nothing/sink/0001_orders.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_orders
(
`id` String,
`status` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.orders',
kafka_group_name = 'clickhouse-orders',
kafka_format = 'JSONEachRow';
8 changes: 8 additions & 0 deletions corpus/replica-identity-key-added-later/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "app",
"table.include.list": "public.invoices"
}
}
1 change: 1 addition & 0 deletions corpus/replica-identity-key-added-later/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ok: source, connector and sink agree
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Invented for the entry: the primary key is added after the table, by
-- ALTER TABLE, as migrations that split constraints out often do.
CREATE TABLE invoices (
id UUID NOT NULL,
total NUMERIC NOT NULL
);
ALTER TABLE invoices ADD CONSTRAINT invoices_pkey PRIMARY KEY (id);
13 changes: 13 additions & 0 deletions corpus/replica-identity-key-added-later/sink/0001_invoices.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_invoices
(
`id` String,
`total` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.invoices',
kafka_group_name = 'clickhouse-invoices',
kafka_format = 'JSONEachRow';
8 changes: 8 additions & 0 deletions corpus/replica-identity-key-in-do-block/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "mm-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "mm",
"table.include.list": "public.reactions"
}
}
1 change: 1 addition & 0 deletions corpus/replica-identity-key-in-do-block/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ok: source, connector and sink agree
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
-- Mattermost v10.11.0, server/channels/db/migrations/postgres/
-- 000016_create_reactions.up.sql, as written: the table is created with no
-- primary key and the DO block below adds one. cdclint does not run DO
-- bodies, so it cannot know the key, and makes no replica-identity finding.
CREATE TABLE IF NOT EXISTS reactions(
userid VARCHAR(26) NOT NULL,
postid VARCHAR(26) NOT NULL,
emojiname VARCHAR(64) NOT NULL,
createat bigint
);

ALTER TABLE reactions ADD COLUMN IF NOT EXISTS updateat bigint;
ALTER TABLE reactions ADD COLUMN IF NOT EXISTS deleteat bigint;

DO $$
<<alter_pk>>
DECLARE
existing_index text;
BEGIN
SELECT string_agg(a.attname, ',') INTO existing_index
FROM pg_constraint AS c
CROSS JOIN
(SELECT unnest(conkey) FROM pg_constraint WHERE conrelid = 'reactions'::regclass AND contype='p') AS cols(colnum)
INNER JOIN pg_attribute AS a ON a.attrelid = c.conrelid AND cols.colnum = a.attnum
WHERE c.contype = 'p'
AND c.conrelid = 'reactions'::regclass;

IF COALESCE (existing_index, '') <> text('postid,userid,emojiname') THEN
ALTER TABLE reactions
DROP CONSTRAINT IF EXISTS reactions_pkey,
ADD PRIMARY KEY (postid, userid, emojiname);
END IF;
END alter_pk $$;

ALTER TABLE reactions ADD COLUMN IF NOT EXISTS remoteid VARCHAR(26);
14 changes: 14 additions & 0 deletions corpus/replica-identity-key-in-do-block/sink/0001_reactions.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_reactions
(
`userid` String,
`postid` String,
`emojiname` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'mm.public.reactions',
kafka_group_name = 'clickhouse-reactions',
kafka_format = 'JSONEachRow';
8 changes: 8 additions & 0 deletions corpus/replica-identity-no-primary-key/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "app",
"table.include.list": "public.audit_events"
}
}
5 changes: 5 additions & 0 deletions corpus/replica-identity-no-primary-key/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
warning replica-identity replica-identity-no-primary-key/migrations/0001_audit_events.sql:3
public.audit_events is captured with REPLICA IDENTITY DEFAULT and no primary key, so it has no replica identity
Debezium emits only its inserts: updates and deletes never reach the stream, and the sink keeps every row as it was first written
fix: ALTER TABLE public.audit_events REPLICA IDENTITY FULL; in a new migration, or give it a primary key; if it only ever takes inserts, say so on the line that creates it: -- cdclint:ignore replica-identity: append-only
0 error(s), 1 warning(s), 0 info
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Invented for the entry: a table created without a primary key, as tables
-- meant only to be appended to often are, then captured like the rest.
CREATE TABLE audit_events (
actor_id UUID NOT NULL,
action TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
14 changes: 14 additions & 0 deletions corpus/replica-identity-no-primary-key/sink/0001_audit_events.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_audit_events
(
`actor_id` String,
`action` String,
`created_at` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.audit_events',
kafka_group_name = 'clickhouse-audit_events',
kafka_format = 'JSONEachRow';
9 changes: 9 additions & 0 deletions corpus/replica-identity-nothing-pgoutput/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "app",
"table.include.list": "public.sessions",
"plugin.name": "pgoutput"
}
}
5 changes: 5 additions & 0 deletions corpus/replica-identity-nothing-pgoutput/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
warning replica-identity replica-identity-nothing-pgoutput/migrations/0001_sessions.sql:3
public.sessions is captured with REPLICA IDENTITY NOTHING, so it has no replica identity
Postgres rejects UPDATE and DELETE on a table with no replica identity while a publication replicates them, and this connector reads through one (plugin.name pgoutput): the application's own writes to it fail
fix: ALTER TABLE public.sessions REPLICA IDENTITY DEFAULT; in a new migration (it has a primary key), or FULL; if it only ever takes inserts, say so on the line that creates it: -- cdclint:ignore replica-identity: append-only
0 error(s), 1 warning(s), 0 info
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Invented for the entry: a keyed table later set to REPLICA IDENTITY
-- NOTHING, read through pgoutput, so through a publication.
CREATE TABLE sessions (
id UUID PRIMARY KEY,
user_id UUID NOT NULL,
expires_at TIMESTAMPTZ NOT NULL
);
ALTER TABLE sessions REPLICA IDENTITY NOTHING;
14 changes: 14 additions & 0 deletions corpus/replica-identity-nothing-pgoutput/sink/0001_sessions.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Invented sink for the entry: a Kafka-engine table reading the captured
-- table. The entry is about the source table's replica identity.
CREATE TABLE kafka_sessions
(
`id` String,
`user_id` String,
`expires_at` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.sessions',
kafka_group_name = 'clickhouse-sessions',
kafka_format = 'JSONEachRow';
Loading