Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ points at, with the reason:

```sql
ALTER TABLE users
ADD COLUMN ssn_hash TEXT, -- cdclint:ignore schema-before-connector: PII, never streamed
ADD COLUMN ssn_hash TEXT, -- cdclint:ignore schema-before-connector,source-column-not-captured: PII, never streamed
ADD COLUMN nickname TEXT;
```

Expand All @@ -213,10 +213,12 @@ below. It works in migrations and in sink DDL, with `--` or MySQL's `#`, and
names one or more rules separated by commas. The reason after the colon is
required, because it is the record of the decision. A marker with no reason,
with a rule name that does not exist, or that covers no finding is itself a
warning (`ignore-marker`). A `schema-before-connector` marker is expected to
go quiet once its pull request merges, since that rule judges the change;
to keep the column out of the inventory afterwards too, name both rules:
`-- cdclint:ignore schema-before-connector,source-column-not-captured: PII`.
warning (`ignore-marker`). Name both rules for a column left out of the
stream, as above and as the diff rule's own fix suggests:
`schema-before-connector` only exists in the pull request that adds the
column, and after the merge the same column is `source-column-not-captured`
on every run. A marker naming only the first goes quiet at the merge and the
column comes back into the inventory.

Files in, findings out, non-zero exit. No database, no daemon, no credentials.
Under a second on a laptop. `--fail-on warning` or `info` raises the bar;
Expand Down
2 changes: 2 additions & 0 deletions corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ The rest exercise one path each:
| `kafka-topic-typo` | a Kafka-engine table naming a topic nothing produces |
| `include-list-typo` | an include-list pattern that matches no column, and the read it silently breaks |
| `ignore-marker-pii` | the diff rule with a decision recorded: of two columns added and left off the include list, the one whose line says `cdclint:ignore schema-before-connector: PII` is acknowledged and the forgotten one is raised; a trailing marker does not reach the next line |
| `ignore-marker-both-rules` | the marker the diff rule's fix suggests, `schema-before-connector,source-column-not-captured`, in the pull request that adds the column: acknowledged, and the second rule is not reported as covering nothing |
| `ignore-marker-both-rules-merged` | the same migration after the merge (no base): the column stays out of the inventory of uncaptured columns, acknowledged as `source-column-not-captured` |
| `ignore-marker-mistakes` | markers used wrong: one with no reason, one naming a rule that does not exist, one covering nothing, each an `ignore-marker` warning; and one on the line above a finding, which acknowledges it |
| `golang-migrate-down-files` | Mattermost v10.11.0's `000092_add_createat_to_teammembers.down.sql` drops a column from the wrong table; read in name order it ran just before its up file and deleted `reactions.createat`. Down files are skipped |
| `goose-down-section` | a goose file's down section follows its up section; applied whole, every table was created and dropped again. Down sections are skipped |
Expand Down
2 changes: 1 addition & 1 deletion corpus/diff-adds-column-connector-untouched/expected.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ info source-column-not-captured diff-adds-column-connector-untouched/migrations/
warning schema-before-connector diff-adds-column-connector-untouched/migrations/0035_validation_response_distance.sql:5
this change adds public.report_validations.response_distance_m to a captured table without adding it to column.include.list in diff-adds-column-connector-untouched/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.report_validations.response_distance_m to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.report_validations.response_distance_m to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 1 warning(s), 1 info
2 changes: 1 addition & 1 deletion corpus/diff-connector-captures-one-of-two/expected.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
warning schema-before-connector diff-connector-captures-one-of-two/migrations/0141_movement_doubt_cleared.sql:7
this change adds public.reports.movement_cleared_by to a captured table without adding it to column.include.list in diff-connector-captures-one-of-two/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.reports.movement_cleared_by to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.reports.movement_cleared_by to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 1 warning(s), 0 info
6 changes: 3 additions & 3 deletions corpus/diff-connector-touched-other-table/expected.txt
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
warning schema-before-connector diff-connector-touched-other-table/migrations/0141_movement_doubt_cleared.sql:7
this change adds public.reports.movement_cleared_at to a captured table without adding it to column.include.list in diff-connector-touched-other-table/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.reports.movement_cleared_at to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.reports.movement_cleared_at to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
warning schema-before-connector diff-connector-touched-other-table/migrations/0141_movement_doubt_cleared.sql:8
this change adds public.reports.movement_cleared_by to a captured table without adding it to column.include.list in diff-connector-touched-other-table/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.reports.movement_cleared_by to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.reports.movement_cleared_by to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
warning schema-before-connector diff-connector-touched-other-table/migrations/0141_movement_doubt_cleared.sql:9
this change adds public.reports.movement_clear_reason to a captured table without adding it to column.include.list in diff-connector-touched-other-table/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.reports.movement_clear_reason to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.reports.movement_clear_reason to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 3 warning(s), 0 info
2 changes: 1 addition & 1 deletion corpus/diff-include-list-narrowed/expected.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ info source-column-not-captured diff-include-list-narrowed/migrations/0033_repor
warning schema-before-connector diff-include-list-narrowed/migrations/0035_validation_response_distance.sql:5
this change adds public.report_validations.response_distance_m to a captured table without adding it to column.include.list in diff-include-list-narrowed/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.report_validations.response_distance_m to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.report_validations.response_distance_m to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 1 warning(s), 1 info
12 changes: 12 additions & 0 deletions corpus/ignore-marker-both-rules-merged/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"topic.prefix": "app",
"table.include.list": "public.users",
"column.include.list": "public\\.users\\.(id|email)",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
}
}
2 changes: 2 additions & 0 deletions corpus/ignore-marker-both-rules-merged/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ok: nothing to report beyond what cdclint:ignore acknowledges
acknowledged (cdclint:ignore): source-column-not-captured ignore-marker-both-rules-merged/migrations/0002_users_ssn_hash.sql:4: PII, never streamed
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
-- The same migration after its pull request merged: with no base there is no
-- change to judge, and the column would be in the inventory of uncaptured
-- columns on every run if the marker named only schema-before-connector.
ALTER TABLE users ADD COLUMN ssn_hash TEXT; -- cdclint:ignore schema-before-connector,source-column-not-captured: PII, never streamed
11 changes: 11 additions & 0 deletions corpus/ignore-marker-both-rules-merged/sink/0001_users.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
CREATE TABLE kafka_users
(
`id` String,
`email` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.users',
kafka_group_name = 'clickhouse-users',
kafka_format = 'JSONEachRow';
12 changes: 12 additions & 0 deletions corpus/ignore-marker-both-rules/base/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"topic.prefix": "app",
"table.include.list": "public.users",
"column.include.list": "public\\.users\\.(id|email)",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL
);
12 changes: 12 additions & 0 deletions corpus/ignore-marker-both-rules/connector.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"name": "app-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"topic.prefix": "app",
"table.include.list": "public.users",
"column.include.list": "public\\.users\\.(id|email)",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
}
}
2 changes: 2 additions & 0 deletions corpus/ignore-marker-both-rules/expected.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ok: nothing to report beyond what cdclint:ignore acknowledges
acknowledged (cdclint:ignore): schema-before-connector ignore-marker-both-rules/migrations/0002_users_ssn_hash.sql:3: PII, never streamed
4 changes: 4 additions & 0 deletions corpus/ignore-marker-both-rules/migrations/0001_users.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- The change under review adds ssn_hash and leaves it off the include list on
-- purpose, with the marker the diff rule's own fix suggests: both rules named.
ALTER TABLE users ADD COLUMN ssn_hash TEXT; -- cdclint:ignore schema-before-connector,source-column-not-captured: PII, never streamed
11 changes: 11 additions & 0 deletions corpus/ignore-marker-both-rules/sink/0001_users.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
CREATE TABLE kafka_users
(
`id` String,
`email` String
)
ENGINE = Kafka
SETTINGS
kafka_broker_list = '${KAFKA_BROKERS}',
kafka_topic_list = 'app.public.users',
kafka_group_name = 'clickhouse-users',
kafka_format = 'JSONEachRow';
2 changes: 1 addition & 1 deletion corpus/ignore-marker-pii/expected.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
warning schema-before-connector ignore-marker-pii/migrations/0002_users_profile.sql:6
this change adds public.users.nickname to a captured table without adding it to column.include.list in ignore-marker-pii/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add public.users.nickname to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add public.users.nickname to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 1 warning(s), 0 info
acknowledged (cdclint:ignore): schema-before-connector ignore-marker-pii/migrations/0002_users_profile.sql:5: PII, never streamed
2 changes: 1 addition & 1 deletion corpus/mysql-diff-adds-column/expected.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
warning schema-before-connector mysql-diff-adds-column/migrations/V2__orders_coupon.sql:3
this change adds shop.orders.coupon_code to a captured table without adding it to column.include.list in mysql-diff-adds-column/connector.json (compared with base)
the column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot
fix: add shop.orders.coupon_code to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>
fix: add shop.orders.coupon_code to column.include.list in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>
0 error(s), 1 warning(s), 0 info
8 changes: 7 additions & 1 deletion internal/engine/diff.go
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,16 @@ func schemaBeforeConnector(in *Input, reads []Read) ([]model.Finding, map[string
}
q := fmt.Sprintf("%s.%s", t.Qualified(), c.Name)
raised[strings.ToLower(q)] = true
// The suggested marker names both rules. This rule only exists
// in the pull request that adds the column; once it merges the
// same column is source-column-not-captured on every run, so a
// marker naming only this rule put a deliberately excluded
// column back into the inventory (RefuseRadar #1142, round 1,
// where the pinned script's own example had made that mistake).
fs = append(fs, model.Finding{
Rule: "schema-before-connector", Severity: model.Warning, Pos: c.Pos,
Message: fmt.Sprintf("this change adds %s to a captured table without adding it to %s in %s (compared with %s)\nthe column will not be in the stream; if a sink is later given it, every row will be the default until a snapshot", q, in.Contract.ColumnListSetting(), in.Contract.Pos().File, short(in.Base.Ref)),
Fix: fmt.Sprintf("%s in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector: <why>", edit(in.Contract.ColumnListSetting(), q)),
Fix: fmt.Sprintf("%s in the same change, or, if it is left off on purpose, say so on the line that adds it: -- cdclint:ignore schema-before-connector,source-column-not-captured: <why>", edit(in.Contract.ColumnListSetting(), q)),
})
}
}
Expand Down
2 changes: 1 addition & 1 deletion internal/ignore/ignore.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
// where the pull request that made it shows it.
//
// ALTER TABLE users
// ADD COLUMN ssn_hash TEXT; -- cdclint:ignore schema-before-connector: PII, never streamed
// ADD COLUMN ssn_hash TEXT; -- cdclint:ignore schema-before-connector,source-column-not-captured: PII, never streamed
//
// A marker names one or more rules and gives a reason after a colon. After
// code on the same line it covers that line only; on a line of its own it
Expand Down