From 3716b72f721963ef54cac5c2c15f5fec166c2dfc Mon Sep 17 00:00:00 2001 From: avison9 Date: Sat, 14 Nov 2026 17:49:20 +0100 Subject: [PATCH] The README shows how cdclint is put together in one diagram docs/diagrams/architecture.svg, embedded at the top of "What it checks": the five inputs read as files, the four readers in line with them, one model, the resolve step that matches every sink column to the source column it expects (the one accent, because that is where a dropped column is caught), the rules grouped by the question each answers, cdclint:ignore decisions, and the output with its exit code. Hand-written SVG in the style of the diagrams in avison9/warmbench, so it stays editable as text and renders the same on GitHub as anywhere else. Rejected: Mermaid. It renders natively on GitHub, but its automatic layout crossed edges and wrapped rule names at their hyphens, and the result did not read as cleanly. It lists replica-identity, which arrives in #29; merge this after #28 and #29. --- README.md | 8 ++ docs/diagrams/architecture.svg | 200 +++++++++++++++++++++++++++++++++ 2 files changed, 208 insertions(+) create mode 100644 docs/diagrams/architecture.svg diff --git a/README.md b/README.md index 7f91126..e2547d8 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,14 @@ anything running. ## What it checks +![cdclint architecture: inputs read as files, one model, every sink column resolved to the source column it expects, three groups of rules, decisions recorded in the SQL, findings with an exit code](docs/diagrams/architecture.svg) + +cdclint reads the three schemas and the connector as files into one model, +resolves every sink column to the source column it expects, and asks three +questions of the result. The diagram's source is +[`docs/diagrams/architecture.svg`](docs/diagrams/architecture.svg), plain SVG +edited by hand. + | rule | catches | status | |---|---|---| | `sink-column-not-captured` | the sink reads a column the connector does not include | v0.1 | diff --git a/docs/diagrams/architecture.svg b/docs/diagrams/architecture.svg new file mode 100644 index 0000000..600d9bb --- /dev/null +++ b/docs/diagrams/architecture.svg @@ -0,0 +1,200 @@ + + + + + + + + + + your repository + three schemas and the connector between them, read as files: no database, no credentials + + + + + + + + + Source migrations + Postgres · MySQL · MariaDB + --migrations + + Debezium connector + source connector JSON + --connector + + Sink connector + Kafka Connect JSON, optional + --sink-connector + + Sink DDL + ClickHouse · BigQuery + Snowflake · Iceberg + --sink + + Git base ref + the pull request's target, optional + --base + + + + + + + + + + + + transforms + + + + cdclint one static binary, Go standard library only + + + + + + + + + + Source reader + source/postgres · source/mysql + replays up migrations in order + skips down migrations + + Capture reader + capture/debezium · smt + include and exclude lists · topics + transforms · replica identity + + Sink readers + sink/clickhouse · sink/sqlddl · sink/connect + Kafka-engine tables · materialized views + BigQuery · Snowflake · Iceberg · topic-to-table maps + + Base reader + gitread + the same files + as they were at --base + + + + + + + + + + + + + + + + + Source + tables · columns · keys + replica identity + + Contract + what is captured, + under which topic + + Sink + what each table reads, + from which topic + + + + + + + + + + + + + Resolve + every sink column to the source column it expects: by topic, by sink connector map, or by table name + + + + + + + + + + + + the diff + + + + + + + + + Does the stream carry what the sink reads? + + sink-column-not-captured · sink-table-not-captured + sink-column-unknown · topic-table-mapping + sink-column-flattened · mv-column-match + + + Is the connector right about the source? + + captured-column-missing · captured-table-missing + replica-identity + source-column-not-captured (info) + + + Did this change + forget the connector? + schema-before-connector + judged per column, on the diff + + + + + + + + + + + + Decisions recorded in the SQL + -- cdclint:ignore <rule>: <reason> on the line a finding points at: listed as acknowledged, with its reason, and it does not fail the run + + + + findings, sorted · --disable leaves a rule out, and says so + + + + output + each finding names the line to edit, the problem, what happens if it ships, and a fix to paste + + + + error sink-column-not-captured corpus/include-list-typo/sink/0020_report_validations.sql:12 + public.report_validations.responded_at is read by kafka_report_validations ... + every row will carry the column's default, with no error anywhere + fix: add public.report_validations.responded_at to column.include.list, ... + + + + + text or JSON · --fail-on error | warning | info + exit 1 fails the pull request + GitHub Action · Jenkins · any shell + +