diff --git a/README.md b/README.md index 7f91126..e2547d8 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,14 @@ anything running. ## What it checks +![cdclint architecture: inputs read as files, one model, every sink column resolved to the source column it expects, three groups of rules, decisions recorded in the SQL, findings with an exit code](docs/diagrams/architecture.svg) + +cdclint reads the three schemas and the connector as files into one model, +resolves every sink column to the source column it expects, and asks three +questions of the result. The diagram's source is +[`docs/diagrams/architecture.svg`](docs/diagrams/architecture.svg), plain SVG +edited by hand. + | rule | catches | status | |---|---|---| | `sink-column-not-captured` | the sink reads a column the connector does not include | v0.1 | diff --git a/docs/diagrams/architecture.svg b/docs/diagrams/architecture.svg new file mode 100644 index 0000000..600d9bb --- /dev/null +++ b/docs/diagrams/architecture.svg @@ -0,0 +1,200 @@ + + + + + + + + + + your repository + three schemas and the connector between them, read as files: no database, no credentials + + + + + + + + + Source migrations + Postgres · MySQL · MariaDB + --migrations + + Debezium connector + source connector JSON + --connector + + Sink connector + Kafka Connect JSON, optional + --sink-connector + + Sink DDL + ClickHouse · BigQuery + Snowflake · Iceberg + --sink + + Git base ref + the pull request's target, optional + --base + + + + + + + + + + + + transforms + + + + cdclint one static binary, Go standard library only + + + + + + + + + + Source reader + source/postgres · source/mysql + replays up migrations in order + skips down migrations + + Capture reader + capture/debezium · smt + include and exclude lists · topics + transforms · replica identity + + Sink readers + sink/clickhouse · sink/sqlddl · sink/connect + Kafka-engine tables · materialized views + BigQuery · Snowflake · Iceberg · topic-to-table maps + + Base reader + gitread + the same files + as they were at --base + + + + + + + + + + + + + + + + + Source + tables · columns · keys + replica identity + + Contract + what is captured, + under which topic + + Sink + what each table reads, + from which topic + + + + + + + + + + + + + Resolve + every sink column to the source column it expects: by topic, by sink connector map, or by table name + + + + + + + + + + + + the diff + + + + + + + + + Does the stream carry what the sink reads? + + sink-column-not-captured · sink-table-not-captured + sink-column-unknown · topic-table-mapping + sink-column-flattened · mv-column-match + + + Is the connector right about the source? + + captured-column-missing · captured-table-missing + replica-identity + source-column-not-captured (info) + + + Did this change + forget the connector? + schema-before-connector + judged per column, on the diff + + + + + + + + + + + + Decisions recorded in the SQL + -- cdclint:ignore <rule>: <reason> on the line a finding points at: listed as acknowledged, with its reason, and it does not fail the run + + + + findings, sorted · --disable leaves a rule out, and says so + + + + output + each finding names the line to edit, the problem, what happens if it ships, and a fix to paste + + + + error sink-column-not-captured corpus/include-list-typo/sink/0020_report_validations.sql:12 + public.report_validations.responded_at is read by kafka_report_validations ... + every row will carry the column's default, with no error anywhere + fix: add public.report_validations.responded_at to column.include.list, ... + + + + + text or JSON · --fail-on error | warning | info + exit 1 fails the pull request + GitHub Action · Jenkins · any shell + +