From 2d63b3e1b8bd6244b0be0d05d3bc60234193cf66 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sat, 10 Oct 2026 16:30:25 -0400 Subject: [PATCH 01/12] spec: ClickHouse invariants, and the published page they back The ClickHouse page's behavioral claims had no CI evidence, and five went stale between v1.1.0 and v2026.10.08 unnoticed. This spec makes each claim a clickhouse.* invariant with a test, moves the page into the repo with an anchor per claim, and adds a check that fails when the two disagree. --- ...2026-10-10-clickhouse-invariants-design.md | 120 ++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md diff --git a/docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md b/docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md new file mode 100644 index 00000000..b653ba00 --- /dev/null +++ b/docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md @@ -0,0 +1,120 @@ +# ClickHouse invariants and the published page they back + +Approved in conversation on 2026-10-10. Spec, plan and code share this branch +and one PR. + +## The problem + +The ClickHouse integration page (ClickHouse/ClickHouse#117740) makes claims +about SQLFlow's behavior. Only two kinds of claim have evidence in CI today: +the type table, proven cell by cell by `TestIntegrationSinkClickhouse_Types`, +and the worked examples' configs, which CI validates and loads. Every +behavioral claim was proven by hand-run scripts. + +Five of those claims went stale between v1.1.0 and v2026.10.08 with no CI +signal: the retry contract, the flush interval floor, window closing, a +throughput row and the version pins. The next behavior change will do the +same. + +## The goal + +1. Every claim on the page maps to a declared entry. +2. Every entry has evidence that runs in CI. +3. A behavior change fails CI and names the claim. +4. A claim on the page with no entry fails CI. + +## Decisions + +- **ClickHouse claims are invariants**, in `docs/coverage/invariants.yml`, + named `clickhouse.*`, reported by the same harness markers, generator and + status files as every other invariant. No parallel registry. +- **`applies_to` accepts an integration id** as well as a kind. A + `clickhouse.*` invariant applies to `sink.clickhouse` alone, so no other + sink gets a cell for it. +- **The invariants file stays runtime-only.** A tooling claim, such as a + `sqlflow validate` warning, is proven by a well-named test and cited + through its feature. +- **Measurements stay measurements.** Throughput, the worked examples' counts + and Cloud-only messages cite committed output files that name their + versions. Nothing asserts a throughput. +- **The page's source of truth lives in this repo**, at + `docs/published/clickhouse.mdx`. The ClickHouse fork receives a copy, by + hand, with the user's go. + +## The invariants + +Each sets `applies_to: sink.clickhouse`, `verified_by: harness`, and +`requires` at the level its evidence runs, so absent evidence fails +`make coverage-check`. + +| Invariant | Claim | Level | +| --- | --- | --- | +| `clickhouse.insert.by_name` | Columns are matched by name. A produced column the table lacks fails the batch as `system.sink.write_failed`, naming the column. A column the handler omits takes the table's DEFAULT. | integration | +| `clickhouse.table.not_created` | The sink never creates or alters a table. A table that does not exist fails the batch as `system.sink.write_failed`. | integration | +| `clickhouse.enum.null_is_zero` | A NULL into a non-Nullable Enum is stored as 0. With no 0 member the INSERT succeeds and reads of the column fail with `UNKNOWN_ELEMENT_OF_ENUM`. With a 0 member it reads as that member. Nullable reads NULL. | integration | +| `clickhouse.encode.fails_once` | A value the driver cannot encode fails the batch on the first attempt as `user.sink.encode_failed`. It is not retried, and the process exits with code 10. | integration | +| `clickhouse.temporal_text.two_shapes` | Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as UTC, or the same with an offset, which is applied. Any other shape fails as `user.sink.encode_failed`. | integration | +| `clickhouse.decimal.text_only` | A Decimal column accepts its value as text. A float fails as `user.sink.encode_failed`. | integration | +| `clickhouse.replay.duplicates_until_merge` | A batch delivered twice into a ReplacingMergeTree reads twice until a merge, and once with FINAL. An identical block inside the table's deduplication window is dropped at insert. | integration | +| `clickhouse.flush.one_insert` | One flush is one INSERT, on one connection. | integration | +| `clickhouse.dsn.scheme` | The DSN scheme decides the protocol, TLS and the default port. `@` and `:` in a password parse raw. `/`, `#` and `?` must be percent-encoded. | unit | +| `clickhouse.tls.verifies` | With TLS, the server certificate is verified against the system roots. No option skips verification. | unit | +| `clickhouse.auth.fails_start` | Wrong credentials fail at startup with HTTP 401 and code 194, before any message is consumed. | integration | +| `clickhouse.type_text.per_destination` | Text into a non-String column must already be in that column's text form. The type table proves each destination with its own value. | integration | + +Claims the existing invariants already back are cited, not re-proven: +`type.roundtrip`, `type.null`, `type.timestamp.instant`, `type.nested`, +`type.string.fidelity`, `type.undeclared.fails_loud`, +`pipeline.commit.after_flush`, `sink.probe.fails_start`, +`pipeline.flush.eventually`, `lifecycle.drain.on_cancel` and +`pipeline.window.counts_every_row`. + +## Evidence + +- New integration tests sit beside `TestIntegrationSinkClickhouse_Types` in + `internal/sinks`, reuse its shared ClickHouse container, and emit + `COVERS invariant=clickhouse. integration=sink.clickhouse` through + `internal/coverage`. The scenario reads as a story: what is written, what + the table holds, what a read returns. +- Existing tests that already prove a claim gain the marker rather than a + duplicate test. +- `clickhouse.encode.fails_once` needs the pipeline and the CLI, since the + retry decision and the exit code live there. Its test runs the pipeline, + not only the sink. +- The hand-run probe, dedup and window scripts retire. The throughput scripts + and their output move to `docs/published/clickhouse/` as measured evidence. + +## The page and its check + +The page carries an anchor beside each claim, as an MDX comment: + +- `{/* invariant: */}` cites an invariant. +- `{/* measured: */}` cites a committed output file. +- `{/* config: */}` precedes a YAML block that must equal + `dev/config/examples/published/`. +- `{/* feature: */}` cites a tooling claim's feature. + +A page check in `scripts/coverage_matrix`, run by `make coverage-check` and +tested in `tests/tooling`, fails when: + +1. an `invariant` anchor names an id `invariants.yml` does not declare; +2. a `clickhouse.*` invariant is cited nowhere on the page; +3. a `measured` anchor names a file that does not exist; +4. a `config` block differs from its published file; +5. a `feature` anchor names an id `features.yml` does not declare; +6. the page's type table differs from the one the generator renders. + +The generated type table renders `VARCHAR` alone, with the text-form note +the reviewer accepted, instead of `VARCHAR`, `UUID`, `JSON` as one row. + +## CI + +`.github/workflows/ci.yml` skips a pull request that changes only `docs/**` +and Markdown. It re-includes `docs/coverage/**`. It must also re-include +`docs/published/**`, or a page-only edit skips the check that guards it. + +## What breaks if this is wrong + +A claim on the page drifts from the engine with no CI signal, which is the +failure this replaces. A wrong `requires` level fails `coverage-check` on a +claim whose evidence exists at another level. From b05014e5cca80943d19407ad9600f5a33c2f1f1d Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sat, 10 Oct 2026 16:39:30 -0400 Subject: [PATCH 02/12] plan: ClickHouse invariants, in ten tasks Implements docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md: the generator extension, the page check, the type table, twelve invariants with their tests, the page move and the coverage regeneration. --- .../plans/2026-10-10-clickhouse-invariants.md | 1113 +++++++++++++++++ 1 file changed, 1113 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-10-clickhouse-invariants.md diff --git a/docs/superpowers/plans/2026-10-10-clickhouse-invariants.md b/docs/superpowers/plans/2026-10-10-clickhouse-invariants.md new file mode 100644 index 00000000..39a3d8ba --- /dev/null +++ b/docs/superpowers/plans/2026-10-10-clickhouse-invariants.md @@ -0,0 +1,1113 @@ +# ClickHouse invariants Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every claim on the ClickHouse integration page is a `clickhouse.*` invariant, or a cited existing invariant, measurement or feature, and CI fails when the page and its evidence disagree. + +**Architecture:** `applies_to` in `invariants.yml` gains integration ids, so twelve `clickhouse.*` invariants apply to `sink.clickhouse` alone and flow through the existing harness markers, generator and status files. The page moves to `docs/published/clickhouse.mdx` with an anchor comment per claim. A new generator module checks the page against the registries on every run. + +**Tech Stack:** Python 3 generator in `scripts/coverage_matrix` (pytest in `tests/tooling`), Go 1.26 tests in `internal/sinks` against testcontainers ClickHouse 26.8. + +**Spec:** `docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md` + +## Global Constraints + +- Invariant ids start `clickhouse.`; each sets `applies_to: sink.clickhouse`, `family: clickhouse`, `verified_by: harness`. +- Evidence is the existing marker only: `coverage.Invariant(t, "", "sink.clickhouse")`. No attribution by test name. +- Integration tests are named `TestIntegrationSinkClickhouse_*`, skip under `-short`, and use `newLiveClickhouseSink` or `sharedClickhouse`. +- `invariants.yml` stays runtime-only. Tooling claims cite a feature. +- Prose follows the repo's CLAUDE.md: SQLFlow in prose, `sqlflow` for the command. +- Go struct literals put each field on its own line. +- Run go, make, docker and gh outside the sandbox. + +## Review Focus + +1. An anchor with extra spaces or a trailing space, `{/* invariant: id */}`, must still parse. Pinned in Task 2. +2. A `config` block that differs only by its final newline from the published file must pass; any other byte must fail. Pinned in Task 2. +3. `applies_to` naming an integration id that does not exist must be a registry problem, not a silent invariant with no cells. Pinned in Task 1. +4. An exemption in another integration's file against a `clickhouse.*` invariant must be a registry problem. Pinned in Task 1. +5. A published page whose name has no matching invariant namespace, such as a future `motherduck.mdx`, must pass with no "uncited" problems. Pinned in Task 2. + +--- + +### Task 1: `applies_to` accepts an integration id + +**Files:** +- Modify: `scripts/coverage_matrix/registries.py` (FAMILIES; `validate_registries`) +- Modify: `scripts/coverage_matrix/invariants.py` (new `targets`; `build_invariants`; `snapshot_invariants`) +- Modify: `scripts/coverage_matrix/page.py` (`page_snapshot`) +- Test: `tests/tooling/test_invariants.py`, `tests/tooling/test_registries.py` + +**Interfaces:** +- Produces: `targets(inv: dict, integrations: list[dict]) -> list[dict]` in `coverage_matrix.invariants`: the non-test-only integrations an invariant gets cells for. `applies_to` in `KINDS` selects by kind; otherwise it selects the one integration with that id. +- Produces: `"clickhouse"` in `FAMILIES`. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/tooling/test_invariants.py`: + +```python +SCOPED = {"id": "clickhouse.enum.null_is_zero", "family": "clickhouse", + "class": "safety", "applies_to": "sink.clickhouse", "claim": "z", + "verified_by": "harness", "requires": ["integration"]} + + +def test_an_invariant_scoped_to_an_integration_gets_one_cell(): + s = inv_snap(invariants=[SCOPED]) + inv = s["invariants"][0] + assert set(inv["integrations"]) == {"sink.clickhouse"} + + +def test_a_scoped_marker_from_another_integration_is_unknown(): + s = inv_snap(invariants=[SCOPED], evidence={"integration": { + "TestX": [("clickhouse.enum.null_is_zero", "sink.console")]}}, + results={"integration": {"TestX": PASS}}) + assert s["unknown_invariant_markers"] == [{ + "test": "TestX", "invariant": "clickhouse.enum.null_is_zero", + "integration": "sink.console"}] + + +def test_a_scoped_invariant_with_evidence_is_covered_and_no_gap(): + s = inv_snap(invariants=[SCOPED], evidence={"integration": { + "TestY": [("clickhouse.enum.null_is_zero", "sink.clickhouse")]}}, + results={"integration": {"TestY": PASS}}) + cell = s["invariants"][0]["integrations"]["sink.clickhouse"]["integration"] + assert cell["status"] == "covered" + assert s["invariant_gaps"] == [] +``` + +Append to `tests/tooling/test_registries.py`: + +```python +def test_applies_to_may_name_an_integration(): + invs = [dict(INVARIANTS[0], id="clickhouse.x", family="clickhouse", + applies_to="sink.clickhouse")] + assert validate_registries(invs, INTEGRATIONS, FEATURES) == [] + + +def test_applies_to_naming_no_kind_and_no_integration_is_a_problem(): + invs = [dict(INVARIANTS[0], id="clickhouse.x", family="clickhouse", + applies_to="sink.clikchouse")] + problems = validate_registries(invs, INTEGRATIONS, FEATURES) + assert any("applies_to 'sink.clikchouse'" in p for p in problems) + + +def test_an_exemption_from_another_integrations_invariant_is_a_problem(): + invs = [dict(INVARIANTS[0], id="clickhouse.x", family="clickhouse", + applies_to="sink.clickhouse")] + integs = [dict(i) for i in INTEGRATIONS] + console = next(i for i in integs if i["id"] == "sink.console") + console["exempt"] = [{"invariant": "clickhouse.x", "reason": "r", + "proven_by": "TestZ"}] + problems = validate_registries(invs, integs, FEATURES) + assert any("sink.console" in p and "clickhouse.x" in p for p in problems) +``` + +Check the imports at the top of `test_registries.py` include `validate_registries`, `INVARIANTS`, `INTEGRATIONS` and `FEATURES`; add any that are missing (`from samples import *` provides the last three). + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run --locked pytest tests/tooling/test_invariants.py tests/tooling/test_registries.py -q` +Expected: the six new tests FAIL (no cells for a scoped invariant; `applies_to ... is not one of`; `family 'clickhouse' is not one of`). + +- [ ] **Step 3: Implement** + +In `registries.py`: + +```python +FAMILIES = ("resilience", "checkpoint", "types", "lifecycle", "errors", "cache", + "clickhouse") +``` + +In `validate_registries`, build the integration id set before the invariant loop and replace the `applies_to` check: + +```python + integration_ids = {i.get("id") for i in integrations} + ... + # applies_to is a kind, which every integration of that kind must + # prove, or one integration's id, for a claim only it makes. + if (inv.get("applies_to") not in KINDS + and inv.get("applies_to") not in integration_ids): + problems.append( + f"invariants.yml: {fid} applies_to {inv.get('applies_to')!r} is " + f"neither one of {KINDS} nor an integration id") +``` + +Replace the exemption kind check at the end: + +```python + scope = inv.get("applies_to") + if scope != integ.get("kind") and scope != iid: + problems.append( + f"integrations/{iid}.yml: is exempt from {inv['id']}, which " + f"applies_to {scope}") +``` + +In `invariants.py`, add after `by_kind` and import `KINDS` from `.registries`: + +```python +def targets(inv, integrations): + """The integrations an invariant gets cells for. + + A kind selects every integration of it; an integration id selects that + one. test_only integrations get none, as in by_kind. + """ + scope = inv["applies_to"] + if scope in KINDS: + return by_kind(integrations).get(scope, []) + return [i for i in integrations + if i["id"] == scope and not i.get("test_only")] +``` + +In `build_invariants`, replace the `allowed` loop body: + +```python + allowed = {} + for inv in invariants: + allowed[inv["id"]] = {i["id"] for i in targets(inv, integrations)} +``` + +and delete the now-unused `kinds` line there. In `snapshot_invariants` replace `for integ in kinds.get(inv["applies_to"], []):` with `for integ in targets(inv, integrations):` and delete its `kinds = by_kind(integrations)` line. In `page.py` `page_snapshot`, do the same replacement, import `targets`, and drop `by_kind` from the import if unused. + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `uv run --locked pytest tests/tooling -q` +Expected: all pass, including the existing `test_an_invariant_applies_only_to_its_kind`. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/coverage_matrix tests/tooling +git commit -m "coverage: an invariant may apply to one integration ..." +``` + +The body names the defect (an integration-only claim had nowhere to live but a parallel registry), the fix, and the evidence (the six tests). + +--- + +### Task 2: The published page check + +**Files:** +- Create: `scripts/coverage_matrix/published.py` +- Modify: `scripts/coverage_matrix/registries.py` (paths) +- Modify: `scripts/coverage_matrix/cli.py` (call the check) +- Test: `tests/tooling/test_published.py` + +**Interfaces:** +- Consumes: `render_types_page(lattice, integration) -> str` from `coverage_matrix.types`. +- Produces: `validate_published(invariants, features, integrations, lattice, pages_dir=PUBLISHED_DIR, configs_dir=PUBLISHED_CONFIGS, repo=REPO) -> list[str]`. +- Produces: `anchors(text: str) -> list[tuple[str, str]]` returning `(kind, value)` pairs in page order. +- Produces in `registries.py`: `PUBLISHED_DIR = os.path.join(REPO, "docs", "published")`, `PUBLISHED_CONFIGS = os.path.join(REPO, "dev", "config", "examples", "published")`. + +Anchor grammar: `{/* : */}` with `kind` one of `invariant`, `measured`, `config`, `feature`, `generated`. `generated: type-table` precedes the type table; the check compares every `|` row from the anchor to the first blank line against the supported and null rows `render_types_page` emits for `sink.clickhouse`, and the `- ` bullet list after `generated: unsupported-types` against its unsupported list. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/tooling/test_published.py`: + +```python +"""Tests for coverage_matrix.published.""" + +from coverage_matrix.published import anchors, validate_published +from samples import * # noqa: F401,F403 + +INVS = [ + {"id": "clickhouse.a", "applies_to": "sink.clickhouse"}, + {"id": "clickhouse.b", "applies_to": "sink.clickhouse"}, + {"id": "type.null", "applies_to": "sink"}, +] +FEATS = [{"id": "cli.validate"}] + + +def page(tmp_path, name, text, configs=None, files=None): + pages = tmp_path / "pages" + pages.mkdir(exist_ok=True) + (pages / name).write_text(text) + cfg = tmp_path / "configs" + cfg.mkdir(exist_ok=True) + for k, v in (configs or {}).items(): + (cfg / k).write_text(v) + for k, v in (files or {}).items(): + p = tmp_path / k + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(v) + return validate_published(INVS, FEATS, [], [], pages_dir=str(pages), + configs_dir=str(cfg), repo=str(tmp_path), + types_table=False) + + +def test_anchors_tolerate_spacing(): + assert anchors("x {/* invariant: clickhouse.a */} y") == [ + ("invariant", "clickhouse.a")] + + +def test_a_page_citing_every_namespaced_invariant_passes(tmp_path): + text = "{/* invariant: clickhouse.a */}\n{/* invariant: clickhouse.b */}\n" + assert page(tmp_path, "clickhouse.mdx", text) == [] + + +def test_an_uncited_namespaced_invariant_is_a_problem(tmp_path): + problems = page(tmp_path, "clickhouse.mdx", "{/* invariant: clickhouse.a */}\n") + assert any("clickhouse.b" in p and "cited nowhere" in p for p in problems) + + +def test_an_unknown_invariant_anchor_is_a_problem(tmp_path): + text = ("{/* invariant: clickhouse.a */}{/* invariant: clickhouse.b */}" + "{/* invariant: clickhouse.zz */}") + assert any("clickhouse.zz" in p for p in page(tmp_path, "clickhouse.mdx", text)) + + +def test_a_page_with_no_namespace_needs_no_citations(tmp_path): + assert page(tmp_path, "motherduck.mdx", "no anchors here\n") == [] + + +def test_a_measured_anchor_must_name_a_file(tmp_path): + text = ("{/* invariant: clickhouse.a */}{/* invariant: clickhouse.b */}" + "{/* measured: docs/published/clickhouse/perf.out */}") + assert any("perf.out" in p for p in page(tmp_path, "clickhouse.mdx", text)) + assert page(tmp_path, "clickhouse.mdx", text, + files={"docs/published/clickhouse/perf.out": "x"}) == [] + + +def test_a_config_block_must_equal_its_file(tmp_path): + base = "{/* invariant: clickhouse.a */}{/* invariant: clickhouse.b */}\n" + text = base + "{/* config: t.yml */}\n```yaml\na: 1\n```\n" + assert page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 1\n"}) == [] + assert page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 1"}) == [] + bad = page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 2\n"}) + assert any("t.yml" in p and "differs" in p for p in bad) + + +def test_a_feature_anchor_must_name_a_feature(tmp_path): + base = "{/* invariant: clickhouse.a */}{/* invariant: clickhouse.b */}" + assert page(tmp_path, "clickhouse.mdx", base + "{/* feature: cli.validate */}") == [] + assert any("cli.nope" in p for p in + page(tmp_path, "clickhouse.mdx", base + "{/* feature: cli.nope */}")) +``` + +Add one test that uses the real registries and the real page, which Task 8 creates, so CI holds the committed page: + +```python +def test_the_committed_pages_pass(): + from coverage_matrix.registries import (load_features, load_integrations, + load_invariants, load_lattice) + assert validate_published(load_invariants(), load_features(), + load_integrations(), load_lattice()) == [] +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run --locked pytest tests/tooling/test_published.py -q` +Expected: FAIL, `ModuleNotFoundError: No module named 'coverage_matrix.published'`. + +- [ ] **Step 3: Implement `published.py`** + +```python +"""Published pages and the claims they make. + +A page under docs/published carries an anchor comment beside each claim. The +check holds the page to the registries: every anchor resolves, every invariant +in the page's namespace is cited, and every embedded config and generated +table matches its source. A page that drifts from its evidence fails here +rather than in a reader's pipeline. +""" + +import os +import re + +from .registries import PUBLISHED_CONFIGS, PUBLISHED_DIR, REPO +from .types import render_types_page + +ANCHOR = re.compile( + r"\{/\*\s*(invariant|measured|config|feature|generated)\s*:\s*(.+?)\s*\*/\}") +FENCE = re.compile(r"```[a-z]*\n(.*?)```", re.S) + + +def anchors(text): + return [(m.group(1), m.group(2)) for m in ANCHOR.finditer(text)] + + +def _block_after(text, end): + """The first fenced block after offset end, without its fences.""" + m = FENCE.search(text, end) + return m.group(1) if m else None + + +def _rows_after(text, end, prefix): + """Consecutive lines starting with prefix after offset end.""" + rows, started = [], False + for line in text[end:].splitlines(): + if line.startswith(prefix): + rows.append(line) + started = True + elif started: + break + return rows + + +def _generated(lattice, integrations): + integ = next((i for i in integrations if i["id"] == "sink.clickhouse"), None) + if integ is None: + return [], [] + lines = render_types_page(lattice, integ).splitlines() + table = [l for l in lines if l.startswith("| `")] + unsupported = [l for l in lines if l.startswith("- ")] + return table, unsupported + + +def validate_published(invariants, features, integrations, lattice, + pages_dir=PUBLISHED_DIR, configs_dir=PUBLISHED_CONFIGS, + repo=REPO, types_table=True): + problems = [] + if not os.path.isdir(pages_dir): + return problems + inv_ids = {i["id"] for i in invariants} + feat_ids = {f["id"] for f in features} + table, unsupported = (_generated(lattice, integrations) + if types_table else ([], [])) + + for name in sorted(os.listdir(pages_dir)): + if not name.endswith(".mdx"): + continue + rel = os.path.relpath(os.path.join(pages_dir, name), repo) + with open(os.path.join(pages_dir, name)) as fh: + text = fh.read() + cited = set() + for m in ANCHOR.finditer(text): + kind, value = m.group(1), m.group(2) + if kind == "invariant": + cited.add(value) + if value not in inv_ids: + problems.append(f"{rel}: cites invariant {value}, which " + "invariants.yml does not declare") + elif kind == "feature" and value not in feat_ids: + problems.append(f"{rel}: cites feature {value}, which " + "features.yml does not declare") + elif kind == "measured" and not os.path.isfile(os.path.join(repo, value)): + problems.append(f"{rel}: cites measurement {value}, which does not exist") + elif kind == "config": + path = os.path.join(configs_dir, value) + block = _block_after(text, m.end()) + if not os.path.isfile(path): + problems.append(f"{rel}: embeds config {value}, which does not exist") + elif block is None or block.rstrip("\n") != open(path).read().rstrip("\n"): + problems.append(f"{rel}: the block after config {value} differs " + "from the published file") + elif kind == "generated" and types_table: + want = table if value == "type-table" else unsupported + prefix = "| `" if value == "type-table" else "- " + if _rows_after(text, m.end(), prefix) != want: + problems.append(f"{rel}: the {value} differs from the one " + "the type declaration generates") + namespace = name[:-len(".mdx")] + "." + for iid in sorted(inv_ids): + if iid.startswith(namespace) and iid not in cited: + problems.append(f"{rel}: invariant {iid} is cited nowhere on the page") + return problems +``` + +Add the two paths to `registries.py` beside `TYPES_PAGE`. In `cli.py`, import `validate_published` and add after `validate_types`: + +```python + problems += validate_published(invariants, features, integrations, lattice) +``` + +- [ ] **Step 4: Run the tests** + +Run: `uv run --locked pytest tests/tooling/test_published.py -q -k "not committed"` +Expected: PASS. `test_the_committed_pages_pass` passes vacuously until Task 8 adds the page, because the directory is absent. + +- [ ] **Step 5: Commit** + +`coverage: a published page is held to the claims it cites`, with the defect, fix and evidence in the body. + +--- + +### Task 3: The generated type table matches what the reviewer accepted + +**Files:** +- Modify: `scripts/coverage_matrix/types.py` (`render_types_page`) +- Modify: `docs/coverage/integrations/sink.clickhouse.yml` (`utf8` gains `rule`; `nulls.default.rule`) +- Regenerate: `docs/coverage/clickhouse-types.mdx` with `make coverage-page` +- Test: `tests/tooling/test_types.py` + +**Interfaces:** +- Produces: a supported row whose declaration has any column carrying `value` renders the lattice entry's first DuckDB cast alone, and its notes read ` reach the sink as .` + +- [ ] **Step 1: Write the failing test** + +```python +def test_a_text_carrier_row_names_its_cast_alone_and_says_the_rest_reach_it(): + lattice = [{"key": "utf8", "duckdb": ["VARCHAR", "UUID", "JSON"]}] + integ = {"types": {"utf8": {"outcome": "exact", + "rule": "the text must already be the column's text form.", + "columns": [{"type": "String", "expect": "x"}, + {"type": "UUID", "value": "u", "expect": "u"}]}}} + page = render_types_page(lattice, integ) + assert ("| `VARCHAR` | `String`, `UUID` | the text must already be the " + "column's text form. `UUID` and `JSON` reach the sink as `VARCHAR`. |") in page +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `uv run --locked pytest tests/tooling/test_types.py -q` +Expected: FAIL, the row still lists `VARCHAR`, `UUID`, `JSON`. + +- [ ] **Step 3: Implement** + +In `render_types_page`, replace the supported-row lines: + +```python + cols = ", ".join("`%s`" % c["type"] for c in decl.get("columns", [])) + rule = decl.get("rule", "") + # A carrier whose columns each take their own value proves the pair, + # not the cross-product: a UUID cast reaches a Decimal column only as + # text the Decimal parses. Name the cast the matrix wrote, and say the + # others arrive as it. + if any(c.get("value") for c in decl.get("columns", [])) and len(entry["duckdb"]) > 1: + first, rest = entry["duckdb"][0], entry["duckdb"][1:] + casts = "`%s`" % first + rule = ("%s %s reach the sink as `%s`." % ( + rule, " and ".join("`%s`" % r for r in rest), first)).strip() + supported.append("| %s | %s | %s |" % (casts, cols, rule)) +``` + +In `sink.clickhouse.yml`, under `utf8:` add `rule: the text must already be the column's text form; see the note below.` and set `nulls.default.rule` to: + +```yaml + rule: the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT. For an Enum with no 0 member that value is not a member, and the row cannot be read +``` + +- [ ] **Step 4: Run the tests and regenerate** + +Run: `uv run --locked pytest tests/tooling -q && make coverage-page` +Expected: PASS; `docs/coverage/clickhouse-types.mdx` shows the new `VARCHAR` row and NULL row. + +- [ ] **Step 5: Commit** + +`coverage: the type table names VARCHAR alone, and NULL into an Enum`. + +--- + +### Task 4: The unit invariants, DSN scheme and TLS + +**Files:** +- Modify: `docs/coverage/invariants.yml` (new `# --- ClickHouse` section with all twelve entries) +- Create: `internal/sinks/clickhouse_invariants_test.go` + +**Interfaces:** +- Consumes: `clickhouseOptions(dsn string) (*clickhouse.Options, error)` and `config.ClickhouseSink` in package `sinks`. +- Produces: the twelve invariant ids below, used by every later task and by the page. + +- [ ] **Step 1: Declare all twelve invariants** + +Append to `invariants.yml`, before `# --- Lifecycle`, one entry per row of the spec's table, in this shape: + +```yaml + # --- ClickHouse ---------------------------------------------------------- + # Claims only sink.clickhouse makes. Each one backs a sentence on + # docs/published/clickhouse.mdx, which cites it by id, and the page check + # fails when the two disagree. + - id: clickhouse.dsn.scheme + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + The DSN scheme decides the protocol, TLS and the default port. `@` and + `:` in a password parse raw; `/`, `#` and `?` must be percent-encoded. + verified_by: harness + requires: [unit] +``` + +`clickhouse.dsn.scheme` and `clickhouse.tls.verifies` require `[unit]`; the other ten require `[integration]`. Claims are the spec's text verbatim. + +- [ ] **Step 2: Write the failing unit tests** + +Create `internal/sinks/clickhouse_invariants_test.go`: + +```go +package sinks + +// The clickhouse.* invariants: claims only the ClickHouse sink makes, each +// one a sentence on docs/published/clickhouse.mdx. A test here that fails is +// a page that is now wrong. + +import ( + "reflect" + "testing" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/turbolytics/sql-flow/internal/config" + "github.com/turbolytics/sql-flow/internal/coverage" + "github.com/zeebo/assert" +) + +func TestSinkClickhouse_DSNSchemeDecidesProtocolTLSAndPort(t *testing.T) { + coverage.Invariant(t, "clickhouse.dsn.scheme", "sink.clickhouse") + + cases := []struct { + dsn string + protocol clickhouse.Protocol + tls bool + addr string + }{ + {dsn: "clickhouses://h/db", protocol: clickhouse.HTTP, tls: true, addr: "h:8443"}, + {dsn: "https://h/db", protocol: clickhouse.HTTP, tls: true, addr: "h:8443"}, + {dsn: "clickhouse://h/db", protocol: clickhouse.HTTP, tls: false, addr: "h:8123"}, + {dsn: "http://h/db", protocol: clickhouse.HTTP, tls: false, addr: "h:8123"}, + {dsn: "natives://h/db", protocol: clickhouse.Native, tls: true, addr: "h:9440"}, + {dsn: "tcp://h/db", protocol: clickhouse.Native, tls: false, addr: "h:9000"}, + {dsn: "native://h/db", protocol: clickhouse.Native, tls: false, addr: "h:9000"}, + } + for _, c := range cases { + opts, err := clickhouseOptions(c.dsn) + assert.NoError(t, err) + assert.Equal(t, c.protocol, opts.Protocol) + assert.Equal(t, c.tls, opts.TLS != nil) + assert.Equal(t, []string{c.addr}, opts.Addr) + } + + // @ and : in a password parse raw. + opts, err := clickhouseOptions("clickhouses://u:p@ss:w0rd@h/db") + assert.NoError(t, err) + assert.Equal(t, "p@ss:w0rd", opts.Auth.Password) + + // # left raw ends the URL early and reads as a port. + _, err = clickhouseOptions("clickhouses://u:pa#ss@h/db") + assert.Error(t, err) + assert.Contains(t, err.Error(), "invalid port") + + // Percent-encoded, the same password parses. + opts, err = clickhouseOptions("clickhouses://u:pa%23ss@h/db") + assert.NoError(t, err) + assert.Equal(t, "pa#ss", opts.Auth.Password) +} + +func TestSinkClickhouse_TLSVerifiesAgainstTheSystemRoots(t *testing.T) { + coverage.Invariant(t, "clickhouse.tls.verifies", "sink.clickhouse") + + opts, err := clickhouseOptions("clickhouses://h/db") + assert.NoError(t, err) + assert.False(t, opts.TLS.InsecureSkipVerify) + assert.Nil(t, opts.TLS.RootCAs) + + // No option skips verification: the sink's config is a DSN and a table. + // A field added here is a page claim to revisit. + var names []string + typ := reflect.TypeOf(config.ClickhouseSink{}) + for i := 0; i < typ.NumField(); i++ { + names = append(names, typ.Field(i).Name) + } + assert.DeepEqual(t, []string{"DSN", "Table"}, names) +} +``` + +- [ ] **Step 3: Run them** + +Run: `go test -short -run 'TestSinkClickhouse_(DSNScheme|TLSVerifies)' ./internal/sinks -v` +Expected: PASS, since the behavior exists. If the `#` case reports a different message, set the assertion to the observed text and say so in the commit body: the page quotes it. + +- [ ] **Step 4: Run the short suite** + +Run: `go test -short -race ./... 2>&1 | tail -5` +Expected: PASS. + +- [ ] **Step 5: Commit** + +`invariants: twelve clickhouse.* claims, and the DSN and TLS proofs`. + +--- + +### Task 5: Columns, tables, Enum NULLs, per-destination text + +**Files:** +- Modify: `internal/sinks/clickhouse_invariants_test.go` +- Modify: `internal/sinks/conformance_types_test.go` (one marker line) + +**Interfaces:** +- Consumes: `newLiveClickhouseSink(t *testing.T, ddl string) *ClickhouseSink`, where `ddl` carries one `%s` for the table name, and `s.conn`, `s.table`. +- Produces: `textColumns(t *testing.T, cols ...textCol) arrow.Table` and `type textCol struct { name string; values []*string }`, used by Tasks 6 and 7. + +- [ ] **Step 1: Write the tests** + +Add the helper and three tests: + +```go +type textCol struct { + name string + values []*string +} + +func str(s string) *string { return &s } + +// textColumns builds one batch of nullable utf8 columns, in the order given. +func textColumns(t *testing.T, cols ...textCol) arrow.Table { + t.Helper() + fields := make([]arrow.Field, len(cols)) + for i, c := range cols { + fields[i] = arrow.Field{ + Name: c.name, + Type: arrow.BinaryTypes.String, + Nullable: true, + } + } + schema := arrow.NewSchema(fields, nil) + b := array.NewRecordBuilder(memory.NewGoAllocator(), schema) + defer b.Release() + for i, c := range cols { + sb := b.Field(i).(*array.StringBuilder) + for _, v := range c.values { + if v == nil { + sb.AppendNull() + } else { + sb.Append(*v) + } + } + } + rec := b.NewRecord() + defer rec.Release() + tbl := array.NewTableFromRecords(schema, []arrow.Record{rec}) + t.Cleanup(tbl.Release) + return tbl +} + +func flushOnce(t *testing.T, s core.Sink, tbl arrow.Table) error { + t.Helper() + assert.NoError(t, s.WriteTable(context.Background(), tbl)) + return s.Flush(context.Background()) +} + +func TestIntegrationSinkClickhouse_InsertsByName(t *testing.T) { + coverage.Invariant(t, "clickhouse.insert.by_name", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + s := newLiveClickhouseSink(t, `CREATE TABLE %s ( + b String, a String, d String DEFAULT 'omitted-default' + ) ENGINE = MergeTree() ORDER BY tuple()`) + + // Produced in the opposite order to the table, and without d. + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "a", values: []*string{str("A")}}, + textCol{name: "b", values: []*string{str("B")}}))) + var a, b, d string + assert.NoError(t, s.conn.QueryRow(context.Background(), + "SELECT a, b, d FROM "+s.table).Scan(&a, &b, &d)) + assert.Equal(t, "A", a) + assert.Equal(t, "B", b) + assert.Equal(t, "omitted-default", d) + + // A column the table lacks fails the batch, naming it. + err := flushOnce(t, s, textColumns(t, + textCol{name: "a", values: []*string{str("A")}}, + textCol{name: "extra", values: []*string{str("X")}})) + assert.Error(t, err) + assert.True(t, errs.HasCode(err, errs.CodeSinkWriteFailed)) + assert.Contains(t, err.Error(), "column extra is not present in the table") +} + +func TestIntegrationSinkClickhouse_NeverCreatesATable(t *testing.T) { + coverage.Invariant(t, "clickhouse.table.not_created", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ch := sharedClickhouse(t) + table := fmt.Sprintf("absent_%d", time.Now().UnixNano()) + s := mustDirectSink(t, ch, table) + + err := flushOnce(t, s, textColumns(t, textCol{name: "a", values: []*string{str("A")}})) + assert.Error(t, err) + assert.True(t, errs.HasCode(err, errs.CodeSinkWriteFailed)) + + var exists uint8 + assert.NoError(t, s.conn.QueryRow(context.Background(), + "EXISTS TABLE "+table).Scan(&exists)) + assert.Equal(t, uint8(0), exists) +} + +func TestIntegrationSinkClickhouse_EnumNullIsZero(t *testing.T) { + coverage.Invariant(t, "clickhouse.enum.null_is_zero", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + null := textColumns(t, textCol{name: "e", values: []*string{nil}}) + ctx := context.Background() + + // No 0 member: the INSERT succeeds and the row cannot be read. + noZero := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Enum8('a' = 1, 'b' = 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, noZero, null)) + var n uint64 + assert.NoError(t, noZero.conn.QueryRow(ctx, "SELECT count() FROM "+noZero.table).Scan(&n)) + assert.Equal(t, uint64(1), n) + var e string + err := noZero.conn.QueryRow(ctx, "SELECT toString(e) FROM "+noZero.table).Scan(&e) + assert.Error(t, err) + assert.Contains(t, err.Error(), "UNKNOWN_ELEMENT_OF_ENUM") + var raw int8 + assert.NoError(t, noZero.conn.QueryRow(ctx, "SELECT CAST(e AS Int8) FROM "+noZero.table).Scan(&raw)) + assert.Equal(t, int8(0), raw) + + // A 0 member: the NULL reads as that member. + zero := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Enum8('' = 0, 'a' = 1, 'b' = 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, zero, null)) + assert.NoError(t, zero.conn.QueryRow(ctx, "SELECT toString(e) FROM "+zero.table).Scan(&e)) + assert.Equal(t, "", e) + + // Nullable: the NULL reads as NULL. + nullable := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Nullable(Enum8('a' = 1, 'b' = 2))) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, nullable, null)) + var isNull uint8 + assert.NoError(t, nullable.conn.QueryRow(ctx, "SELECT e IS NULL FROM "+nullable.table).Scan(&isNull)) + assert.Equal(t, uint8(1), isNull) +} +``` + +In `conformance_types_test.go`, after `coverage.Covers(t, "sink.clickhouse")`, add: + +```go + // The type table proves each text destination with that destination's own + // value, which is the claim the page makes about VARCHAR. + coverage.Invariant(t, "clickhouse.type_text.per_destination", "sink.clickhouse") +``` + +Add imports as needed: `context`, `fmt`, `time`, `arrow`, `array`, `memory`, `core`, `errs`. + +- [ ] **Step 2: Run them** + +Run: `go test -run 'TestIntegrationSinkClickhouse_(InsertsByName|NeverCreatesATable|EnumNullIsZero|Types)$' ./internal/sinks -v -timeout 10m` +Expected: PASS. A failure is a finding about the page claim, not a test to relax: report it. + +- [ ] **Step 3: Run the short suite, then commit** + +Run: `go test -short -race ./... 2>&1 | tail -5`, then commit `invariants: ClickHouse matches columns by name, creates no table, and stores a NULL Enum as 0`. + +--- + +### Task 6: Encode failures, text timestamps, Decimal text + +**Files:** +- Modify: `internal/sinks/clickhouse_invariants_test.go` + +**Interfaces:** +- Consumes: `textColumns`, `textCol`, `str`, `flushOnce` from Task 5; `newRetrying(inner core.Sink, policy RetryPolicy) *retrying` with its `onRetry` field; `errs.ExitCode(error) int`, `errs.ExitUserError`. + +- [ ] **Step 1: Write the tests** + +```go +func TestIntegrationSinkClickhouse_EncodeFailureIsNotRetried(t *testing.T) { + coverage.Invariant(t, "clickhouse.encode.fails_once", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (ts DateTime) ENGINE = MergeTree() ORDER BY tuple()`) + r := newRetrying(s, RetryPolicy{ + MaxAttempts: 4, + InitialBackoff: time.Millisecond, + MaxBackoff: time.Millisecond, + Deadline: time.Minute, + }) + retries := 0 + r.onRetry = func(int, error) { retries++ } + + err := flushOnce(t, r, textColumns(t, + textCol{name: "ts", values: []*string{str("2026-09-01T12:00:00Z")}})) + assert.Error(t, err) + assert.True(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) + assert.Equal(t, 0, retries) + assert.Equal(t, errs.ExitUserError, errs.ExitCode(err)) +} + +func TestIntegrationSinkClickhouse_TemporalTextTakesTwoShapes(t *testing.T) { + coverage.Invariant(t, "clickhouse.temporal_text.two_shapes", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (k String, ts DateTime('UTC')) ENGINE = MergeTree() ORDER BY k`) + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "k", values: []*string{str("plain"), str("offset")}}, + textCol{name: "ts", values: []*string{str("2026-09-01 12:00:00"), str("2026-09-01 12:00:00 +09:00")}}))) + + got := map[string]string{} + rows, err := s.conn.Query(ctx, "SELECT k, toString(ts) FROM "+s.table) + assert.NoError(t, err) + for rows.Next() { + var k, ts string + assert.NoError(t, rows.Scan(&k, &ts)) + got[k] = ts + } + assert.NoError(t, rows.Err()) + assert.Equal(t, "2026-09-01 12:00:00", got["plain"]) + assert.Equal(t, "2026-09-01 03:00:00", got["offset"]) + + err = flushOnce(t, s, textColumns(t, + textCol{name: "k", values: []*string{str("iso")}}, + textCol{name: "ts", values: []*string{str("2026-09-01T12:00:00Z")}})) + assert.Error(t, err) + assert.True(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) +} + +func TestIntegrationSinkClickhouse_DecimalTakesText(t *testing.T) { + coverage.Invariant(t, "clickhouse.decimal.text_only", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (amount Decimal(10, 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "amount", values: []*string{str("12.34")}}))) + var got string + assert.NoError(t, s.conn.QueryRow(ctx, "SELECT toString(amount) FROM "+s.table).Scan(&got)) + assert.Equal(t, "12.34", got) + + schema := arrow.NewSchema([]arrow.Field{{ + Name: "amount", + Type: arrow.PrimitiveTypes.Float64, + }}, nil) + b := array.NewRecordBuilder(memory.NewGoAllocator(), schema) + defer b.Release() + b.Field(0).(*array.Float64Builder).Append(12.34) + rec := b.NewRecord() + defer rec.Release() + floats := array.NewTableFromRecords(schema, []arrow.Record{rec}) + defer floats.Release() + err := flushOnce(t, s, floats) + assert.Error(t, err) + assert.True(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) +} +``` + +- [ ] **Step 2: Run them** + +Run: `go test -run 'TestIntegrationSinkClickhouse_(EncodeFailureIsNotRetried|TemporalTextTakesTwoShapes|DecimalTakesText)$' ./internal/sinks -v -timeout 10m` +Expected: PASS. + +- [ ] **Step 3: Run the short suite, then commit** + +`invariants: an encode failure fails once, and text reaches DateTime and Decimal only in their forms`. + +--- + +### Task 7: Replays, one INSERT per flush, wrong credentials + +**Files:** +- Modify: `internal/sinks/clickhouse_invariants_test.go` + +**Interfaces:** +- Consumes: Task 5 helpers; `sharedClickhouse`, `clickhouseDSN`, `NewClickhouseSink`, `(*ClickhouseSink).Probe`. + +- [ ] **Step 1: Write the tests** + +```go +func TestIntegrationSinkClickhouse_ReplayDuplicatesUntilMerge(t *testing.T) { + coverage.Invariant(t, "clickhouse.replay.duplicates_until_merge", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + batch := func() arrow.Table { + return textColumns(t, + textCol{name: "id", values: []*string{str("1"), str("2")}}, + textCol{name: "v", values: []*string{str("first"), str("first")}}) + } + count := func(s *ClickhouseSink, suffix string) uint64 { + var n uint64 + assert.NoError(t, s.conn.QueryRow(ctx, "SELECT count() FROM "+s.table+suffix).Scan(&n)) + return n + } + + // A replay into ReplacingMergeTree: both copies until a merge. + rmt := newLiveClickhouseSink(t, `CREATE TABLE %s (id String, v String) ENGINE = ReplacingMergeTree ORDER BY id`) + assert.NoError(t, flushOnce(t, rmt, batch())) + assert.NoError(t, flushOnce(t, rmt, batch())) + assert.Equal(t, uint64(4), count(rmt, "")) + assert.Equal(t, uint64(2), count(rmt, " FINAL")) + assert.NoError(t, rmt.conn.Exec(ctx, "OPTIMIZE TABLE "+rmt.table+" FINAL")) + assert.Equal(t, uint64(2), count(rmt, "")) + + // An identical block inside the deduplication window is dropped at insert. + dedup := newLiveClickhouseSink(t, `CREATE TABLE %s (id String, v String) ENGINE = MergeTree ORDER BY id + SETTINGS non_replicated_deduplication_window = 100`) + assert.NoError(t, flushOnce(t, dedup, batch())) + assert.NoError(t, flushOnce(t, dedup, batch())) + assert.Equal(t, uint64(2), count(dedup, "")) +} + +func TestIntegrationSinkClickhouse_OneFlushIsOneInsert(t *testing.T) { + coverage.Invariant(t, "clickhouse.flush.one_insert", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (a String) ENGINE = MergeTree ORDER BY tuple()`) + for _, v := range []string{"x", "y", "z"} { + assert.NoError(t, s.WriteTable(ctx, textColumns(t, textCol{name: "a", values: []*string{str(v)}}))) + } + assert.NoError(t, s.Flush(ctx)) + + assert.NoError(t, s.conn.Exec(ctx, "SYSTEM FLUSH LOGS")) + var inserts, rows uint64 + assert.NoError(t, s.conn.QueryRow(ctx, ` + SELECT count(), sum(written_rows) FROM system.query_log + WHERE type = 'QueryFinish' AND query_kind = 'Insert' + AND has(tables, currentDatabase() || '.' || ?)`, s.table).Scan(&inserts, &rows)) + assert.Equal(t, uint64(1), inserts) + assert.Equal(t, uint64(3), rows) +} + +func TestIntegrationSinkClickhouse_WrongPasswordFailsTheProbe(t *testing.T) { + coverage.Invariant(t, "clickhouse.auth.fails_start", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + ch := sharedClickhouse(t) + host, err := ch.Host(ctx) + assert.NoError(t, err) + port, err := ch.MappedPort(ctx, "8123/tcp") + assert.NoError(t, err) + + dsn := fmt.Sprintf("clickhouse://%s:wrong@%s/%s", + clickhouseUser, net.JoinHostPort(host, port.Port()), clickhouseDatabase) + s, err := NewClickhouseSink(config.ClickhouseSink{ + DSN: dsn, + Table: "never_written", + }) + assert.NoError(t, err) + t.Cleanup(func() { s.Close() }) + + err = s.Probe(ctx) + assert.Error(t, err) + assert.Contains(t, err.Error(), "401") + assert.Contains(t, err.Error(), "Authentication failed") +} +``` + +If `SYSTEM FLUSH LOGS` or `system.query_log` is denied to the test user, grant the shared container's user access in `sharedClickhouse` with an `--env CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` option or a `GRANT` statement run as default, and say which in the commit body. Record the code number the local server returns for the wrong password; the claim and the page name 401 and the code the test observes. + +- [ ] **Step 2: Run them** + +Run: `go test -run 'TestIntegrationSinkClickhouse_(ReplayDuplicatesUntilMerge|OneFlushIsOneInsert|WrongPasswordFailsTheProbe)$' ./internal/sinks -v -timeout 10m` +Expected: PASS. + +- [ ] **Step 3: Run the short suite, then commit** + +`invariants: replays read twice until a merge, a flush is one INSERT, and wrong credentials fail the probe`. + +--- + +### Task 8: The page moves in, with an anchor per claim + +**Files:** +- Create: `docs/published/clickhouse.mdx` (from `sqlflow.mdx` on branch `docs-status/clickhouse-117740`, path `docs/superpowers/status/clickhouse-117740/sqlflow.mdx`) +- Create: `docs/published/clickhouse/` holding `perf.sh`, `perfcloud.sh`, `perf.out`, `perfcloud.out`, `cloudverify.out`, `throughput-bisect.out` from the same branch, and `README.md` +- Create: `docs/published/README.md` +- Modify: `dev/config/examples/published/README.md` (the page now lives in `docs/published`) +- Modify: `.github/workflows/ci.yml` (re-include `docs/published/**`) + +- [ ] **Step 1: Copy the page and its evidence** + +```bash +git show origin/docs-status/clickhouse-117740:docs/superpowers/status/clickhouse-117740/sqlflow.mdx > docs/published/clickhouse.mdx +for f in perf.sh perfcloud.sh perf.out perfcloud.out cloudverify.out throughput-bisect.out; do + git show origin/docs-status/clickhouse-117740:docs/superpowers/status/clickhouse-117740/$f > docs/published/clickhouse/$f +done +chmod +x docs/published/clickhouse/*.sh +``` + +Fix the scripts' `W=` paths to read `taxi.yml` from `dev/config/examples/published/clickhouse.taxi.yml`. + +- [ ] **Step 2: Add the anchors** + +One anchor per claim, placed after the sentence it backs. The required set: + +| Section | Anchors | +| --- | --- | +| Setup scheme table | `invariant: clickhouse.dsn.scheme` | +| "The target table must already exist ... match the table's column names" | `invariant: clickhouse.table.not_created`, `invariant: clickhouse.insert.by_name` | +| Authentication: special characters | `invariant: clickhouse.dsn.scheme` | +| Authentication: certificate verified, no skip | `invariant: clickhouse.tls.verifies` | +| Step 3 YAML | `config: clickhouse.taxi.yml` before the block | +| Bluesky YAML | `config: clickhouse.bluesky.yml` before the block | +| "Kafka offsets are committed only after each INSERT" | `invariant: pipeline.commit.after_flush` | +| Verify output, Cloud and self-hosted counts | `measured: docs/published/clickhouse/cloudverify.out` | +| Type table | `generated: type-table` before the table; `invariant: type.roundtrip`, `invariant: type.null`, `invariant: clickhouse.type_text.per_destination` after it | +| Text-form note | `invariant: clickhouse.type_text.per_destination` | +| Enum warning | `invariant: clickhouse.enum.null_is_zero` | +| Timestamp note | `invariant: clickhouse.temporal_text.two_shapes`, `invariant: type.timestamp.instant` | +| Decimal note | `invariant: clickhouse.decimal.text_only` | +| Unsupported list | `generated: unsupported-types` before the list; `invariant: type.undeclared.fails_loud` | +| Limits: no table creation | `invariant: clickhouse.table.not_created`, `invariant: clickhouse.insert.by_name` | +| Limits: NULL | `invariant: type.null`, `invariant: clickhouse.enum.null_is_zero` | +| Limits: at-least-once | `invariant: pipeline.commit.after_flush`, `invariant: clickhouse.replay.duplicates_until_merge` | +| Limits: retries | `invariant: clickhouse.encode.fails_once`, `invariant: sink.probe.fails_start` | +| Limits: one connection | `invariant: clickhouse.flush.one_insert` | +| Limits: windows | `invariant: pipeline.window.counts_every_row`, `feature: config.validation` | +| Performance tables | `measured: docs/published/clickhouse/perf.out`, `measured: docs/published/clickhouse/perfcloud.out` | +| Troubleshooting: 401 | `invariant: clickhouse.auth.fails_start` | +| Troubleshooting: missing column | `invariant: clickhouse.insert.by_name` | +| Troubleshooting: Decimal | `invariant: clickhouse.decimal.text_only` | +| Troubleshooting: timestamp | `invariant: clickhouse.temporal_text.two_shapes`, `invariant: clickhouse.encode.fails_once` | +| Troubleshooting: no rows | `invariant: pipeline.flush.eventually`, `invariant: lifecycle.drain.on_cancel` | + +Set the page's type-table rows and unsupported list to exactly what `docs/coverage/clickhouse-types.mdx` renders after Task 3. The lateness warning is `sqlflow validate` behavior, so it cites `config.validation`, the feature that covers semantic validation. + +- [ ] **Step 3: Re-include the page in CI** + +In `.github/workflows/ci.yml`, under `pull_request.paths`, after `- "docs/coverage/**"`, add `- "docs/published/**"`, and extend the comment above it: docs/published holds pages the page check guards. + +- [ ] **Step 4: Run the page check** + +Run: `uv run --locked pytest tests/tooling -q` +Expected: PASS, including `test_the_committed_pages_pass`. Each failure it prints names the anchor or claim to fix. + +- [ ] **Step 5: Commit** + +`docs: the ClickHouse page lives here, with an anchor per claim`. + +--- + +### Task 9: Regenerate the coverage status and page + +**Files:** +- Regenerate: `docs/coverage/status/*.yml`, `docs/coverage/matrix.md`, `docs/coverage/clickhouse-types.mdx` + +- [ ] **Step 1: Produce the unit and integration reports** + +```bash +mkdir -p .coverage +CGO_ENABLED=1 go test -short -race -json ./... > .coverage/go.json 2>&1 +go test -json -timeout 30m -run '^TestIntegration' ./internal/sinks/... > .coverage/go-integration.json 2>&1 +``` + +The integration pass runs every package, as CI's does, so the status files +match what CI will report. Run it outside the sandbox; it takes up to 30 +minutes. Replace `./internal/sinks/...` above with `./...`. + +- [ ] **Step 2: Write and check** + +Run: `make coverage-write COVERAGE_NOT_RUN="--not-run growth --not-run release" && make coverage-check COVERAGE_NOT_RUN="--not-run growth --not-run release"` +Expected: PASS, with every `clickhouse.*` cell covered at its required level. + +- [ ] **Step 3: Commit** + +`coverage: twelve clickhouse.* invariants, each covered`. + +--- + +### Task 10: The pull request + +- [ ] **Step 1:** Push `feat/clickhouse-invariants` and open one PR against main. The body states the defect (page claims had no CI evidence; five went stale unnoticed), the fix, the evidence (the tests and the page check), and what breaks if wrong. It notes that PR #329's hand-run scripts are superseded and #329 can close once this merges. +- [ ] **Step 2:** Watch CI to green. Do not merge. From 4ca239103a29eef1151196d53e29df92edf45f05 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sat, 10 Oct 2026 17:04:29 -0400 Subject: [PATCH 03/12] coverage: an invariant may apply to one integration A claim only one integration makes, such as the ClickHouse sink storing a NULL Enum as 0, had nowhere to live in invariants.yml: applies_to took a kind, so every sink would have owed it a cell. The alternative was a parallel registry duplicating the harness, the generator and the status files. applies_to now takes a kind or an integration id. targets() resolves it in one place for the cell builder, the snapshot and the page, so an integration-scoped invariant gets one cell and a marker from any other integration is reported unknown. A test-only double's marker stays known and credits nothing, as before. The registry check accepts an integration id, still rejects a typo, and rejects an exemption from another integration's invariant. FAMILIES gains clickhouse. Six new tests in tests/tooling cover the scoped cell, the unknown marker, covered-with-no-gap, and the three registry cases. If targets() is wrong, a kind invariant loses or gains cells and the committed status files go stale, which coverage-check reports. --- scripts/coverage_matrix/invariants.py | 31 +++++++++++++-------------- scripts/coverage_matrix/page.py | 7 +++--- scripts/coverage_matrix/registries.py | 19 ++++++++++------ tests/tooling/test_invariants.py | 30 +++++++++++++++++++++++++- tests/tooling/test_registries.py | 25 ++++++++++++++++++++- 5 files changed, 84 insertions(+), 28 deletions(-) diff --git a/scripts/coverage_matrix/invariants.py b/scripts/coverage_matrix/invariants.py index 4ef08055..572e675e 100644 --- a/scripts/coverage_matrix/invariants.py +++ b/scripts/coverage_matrix/invariants.py @@ -6,7 +6,7 @@ """ from .features import status -from .registries import FAIL, LEVELS, SKIP +from .registries import FAIL, KINDS, LEVELS, SKIP def build_invariants(invariants, integrations, results, evidence): @@ -23,12 +23,10 @@ def build_invariants(invariants, integrations, results, evidence): # The integrations each invariant may be proven on. A marker outside this # set is a typo, and crediting it would invent a cell: a sink invariant # "proven on source.kafka" says nothing about either. - kinds = {i["id"]: i["kind"] for i in integrations} allowed = {} for inv in invariants: allowed[inv["id"]] = { - iid for iid, kind in kinds.items() if kind == inv["applies_to"] - } + i["id"] for i in targets(inv, integrations, test_only=True)} cells = {} # (invariant, integration, level) -> [(test, outcome)] unknown = [] # (test, invariant, integration) @@ -56,7 +54,6 @@ def snapshot_invariants(invariants, integrations, built): reason, so a reader of the snapshot never has to open the registry to learn why a cell is blank. """ - kinds = by_kind(integrations) exemptions = exemptions_of(integrations) out = {"invariants": [], "invariant_gaps": [], "unknown_invariant_markers": []} @@ -64,7 +61,7 @@ def snapshot_invariants(invariants, integrations, built): for inv in invariants: entry = declare(inv) - for integ in kinds.get(inv["applies_to"], []): + for integ in targets(inv, integrations): exemption = exemptions.get((inv["id"], integ["id"])) levels = {} for level in LEVELS: @@ -165,16 +162,18 @@ def exemptions_of(integrations): } -def by_kind(integrations): - """kind -> the integrations that get cells. test_only ones get none: their - markers are known, so the harness's doubles are not reported as unknown, - and they credit nothing.""" - out = {} - for integ in integrations: - if integ.get("test_only"): - continue - out.setdefault(integ["kind"], []).append(integ) - return out +def targets(inv, integrations, test_only=False): + """The integrations an invariant applies to. + + A kind selects every integration of it. An integration id selects that + one, for a claim only it makes. test_only integrations get no cells, so + they are left out unless test_only is set: a double's marker is then + known, and credits nothing. + """ + scope = inv["applies_to"] + return [i for i in integrations + if (i["kind"] == scope if scope in KINDS else i["id"] == scope) + and (test_only or not i.get("test_only"))] def cell_state(levels): diff --git a/scripts/coverage_matrix/page.py b/scripts/coverage_matrix/page.py index d425cba2..48ff0654 100644 --- a/scripts/coverage_matrix/page.py +++ b/scripts/coverage_matrix/page.py @@ -7,8 +7,8 @@ """ from .features import feature_gaps -from .invariants import (by_kind, declare, exemptions_of, invariant_gaps, - invariant_tally) +from .invariants import (declare, exemptions_of, invariant_gaps, + invariant_tally, targets) from .invariantpage import render_invariants from .registries import LEVELS, MATRIX_MD, TYPES_PAGE from .types import render_types_page @@ -40,11 +40,10 @@ def page_snapshot(features, invariants, integrations, status): }) out["gaps"] = feature_gaps(out["features"]) - kinds = by_kind(integrations) exemptions = exemptions_of(integrations) for inv in invariants: entry = declare(inv) - for integ in kinds.get(inv["applies_to"], []): + for integ in targets(inv, integrations): # The registry is the declaration. A status row that disagrees # with an exemption is stale, and the gate reports that; the # page does not repeat the disagreement. diff --git a/scripts/coverage_matrix/registries.py b/scripts/coverage_matrix/registries.py index b925d1be..0ff9ff88 100644 --- a/scripts/coverage_matrix/registries.py +++ b/scripts/coverage_matrix/registries.py @@ -80,7 +80,8 @@ # The closed vocabularies invariants.yml may use. A value outside one of them # is a typo, and a typo must not reach the matrix as a missing cell. -FAMILIES = ("resilience", "checkpoint", "types", "lifecycle", "errors", "cache") +FAMILIES = ("resilience", "checkpoint", "types", "lifecycle", "errors", "cache", + "clickhouse") # What kind of claim an invariant makes, which is orthogonal to its family. @@ -168,6 +169,7 @@ def validate_registries(invariants, integrations, features): exists to carry: it must not be spendable on typos. """ problems = [] + integration_ids = {i.get("id") for i in integrations} by_id = {} for inv in invariants: @@ -186,9 +188,13 @@ def validate_registries(invariants, integrations, features): if inv.get("class") not in CLASSES: problems.append( f"invariants.yml: {fid} class {inv.get('class')!r} is not one of {CLASSES}") - if inv.get("applies_to") not in KINDS: + # applies_to is a kind, which every integration of that kind must + # prove, or one integration's id, for a claim only it makes. + if (inv.get("applies_to") not in KINDS + and inv.get("applies_to") not in integration_ids): problems.append( - f"invariants.yml: {fid} applies_to {inv.get('applies_to')!r} is not one of {KINDS}") + f"invariants.yml: {fid} applies_to {inv.get('applies_to')!r} is " + f"neither one of {KINDS} nor an integration id") if inv.get("verified_by") not in VERIFIERS: problems.append( f"invariants.yml: {fid} verified_by {inv.get('verified_by')!r} is not one of {VERIFIERS}") @@ -244,9 +250,10 @@ def validate_registries(invariants, integrations, features): problems.append( f"integrations/{iid}.yml: exemption from {inv['id']} has no " "proven_by naming a test that proves the premise") - if inv.get("applies_to") != integ.get("kind"): + scope = inv.get("applies_to") + if scope != integ.get("kind") and scope != iid: problems.append( - f"integrations/{iid}.yml: is a {integ.get('kind')} but " - f"{inv['id']} applies_to {inv.get('applies_to')}") + f"integrations/{iid}.yml: is exempt from {inv['id']}, which " + f"applies_to {scope}") return problems diff --git a/tests/tooling/test_invariants.py b/tests/tooling/test_invariants.py index 04bb2d42..ca33cffe 100644 --- a/tests/tooling/test_invariants.py +++ b/tests/tooling/test_invariants.py @@ -580,4 +580,32 @@ def test_the_page_snapshot_takes_exempt_from_the_registry(): "sink.console": {"sink.flush.keeps_batch": { "unit": "covered", "integration": "covered", "release": "covered"}}}}) assert cell_state( - ps["invariants"][0]["integrations"]["sink.console"]) == "exempt" \ No newline at end of file + ps["invariants"][0]["integrations"]["sink.console"]) == "exempt" + +SCOPED = {"id": "clickhouse.enum.null_is_zero", "family": "clickhouse", + "class": "safety", "applies_to": "sink.clickhouse", "claim": "z", + "verified_by": "harness", "requires": ["integration"]} + + +def test_an_invariant_scoped_to_an_integration_gets_one_cell(): + s = inv_snap(invariants=[SCOPED]) + inv = s["invariants"][0] + assert set(inv["integrations"]) == {"sink.clickhouse"} + + +def test_a_scoped_marker_from_another_integration_is_unknown(): + s = inv_snap(invariants=[SCOPED], evidence={"integration": { + "TestX": [("clickhouse.enum.null_is_zero", "sink.console")]}}, + results={"integration": {"TestX": PASS}}) + assert s["unknown_invariant_markers"] == [{ + "test": "TestX", "invariant": "clickhouse.enum.null_is_zero", + "integration": "sink.console"}] + + +def test_a_scoped_invariant_with_evidence_is_covered_and_no_gap(): + s = inv_snap(invariants=[SCOPED], evidence={"integration": { + "TestY": [("clickhouse.enum.null_is_zero", "sink.clickhouse")]}}, + results={"integration": {"TestY": PASS}}) + cell = s["invariants"][0]["integrations"]["sink.clickhouse"]["integration"] + assert cell["status"] == "covered" + assert s["invariant_gaps"] == [] diff --git a/tests/tooling/test_registries.py b/tests/tooling/test_registries.py index e14ecbd2..b4046443 100644 --- a/tests/tooling/test_registries.py +++ b/tests/tooling/test_registries.py @@ -230,4 +230,27 @@ def test_validate_rejects_an_unknown_class(): def test_the_registry_declares_at_least_one_liveness_invariant(): """A file of safety claims alone cannot say the pipeline does anything.""" classes = {i["class"] for i in load_invariants()} - assert "liveness" in classes \ No newline at end of file + assert "liveness" in classes + +def test_applies_to_may_name_an_integration(): + invs = INVARIANTS + [dict(INVARIANTS[0], id="clickhouse.x", + family="clickhouse", applies_to="sink.clickhouse")] + assert validate_registries(invs, INTEGRATIONS, FEATURES) == [] + + +def test_applies_to_naming_no_kind_and_no_integration_is_a_problem(): + invs = [dict(INVARIANTS[0], id="clickhouse.x", family="clickhouse", + applies_to="sink.clikchouse")] + problems = validate_registries(invs, INTEGRATIONS, FEATURES) + assert any("applies_to 'sink.clikchouse'" in p for p in problems) + + +def test_an_exemption_from_another_integrations_invariant_is_a_problem(): + invs = INVARIANTS + [dict(INVARIANTS[0], id="clickhouse.x", + family="clickhouse", applies_to="sink.clickhouse")] + integs = [dict(i) for i in INTEGRATIONS] + console = next(i for i in integs if i["id"] == "sink.console") + console["exempt"] = [{"invariant": "clickhouse.x", "reason": "r", + "proven_by": "TestZ"}] + problems = validate_registries(invs, integs, FEATURES) + assert any("sink.console" in p and "clickhouse.x" in p for p in problems) From 9ca4b61c6e7aaf39a7bfd82c5fb2af36de935574 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sat, 10 Oct 2026 17:14:54 -0400 Subject: [PATCH 04/12] coverage: a published page is held to the claims it cites The ClickHouse page's claims lived only in the page. Five went stale between v1.1.0 and v2026.10.08 because nothing compared them with the engine. published.py reads every page under docs/published and checks the anchor comment beside each claim. An invariant or feature anchor must name a declared id, a measured anchor a committed file, a config anchor's block must equal the published config, and a generated anchor's rows must equal what the type declaration renders. A page named for a namespace must cite every invariant in it. The generator runs the check before reading any report, so coverage-check and --page both fail on a drifted page. Eleven tests in tests/tooling cover each rule, anchor spacing, a page with no namespace, and the committed pages. If the check is wrong, a page claim drifts silently, which is the failure it replaces, or a correct page fails CI with a problem that names the anchor. --- scripts/coverage_matrix/cli.py | 2 + scripts/coverage_matrix/published.py | 125 ++++++++++++++++++++++++++ scripts/coverage_matrix/registries.py | 6 ++ tests/tooling/test_published.py | 102 +++++++++++++++++++++ 4 files changed, 235 insertions(+) create mode 100644 scripts/coverage_matrix/published.py create mode 100644 tests/tooling/test_published.py diff --git a/scripts/coverage_matrix/cli.py b/scripts/coverage_matrix/cli.py index f7d801ec..4b1f774a 100644 --- a/scripts/coverage_matrix/cli.py +++ b/scripts/coverage_matrix/cli.py @@ -17,6 +17,7 @@ from .suites import parse_go, parse_pytest from .statusfiles import (keep_levels, read_status, status_from_snapshot, write_status) +from .published import validate_published from .types import validate_types @@ -49,6 +50,7 @@ def main(): # the matrix as a missing cell. problems = validate_registries(invariants, integrations, features) problems += validate_types(lattice, integrations) + problems += validate_published(invariants, features, integrations, lattice) if problems: for problem in problems: print(problem, file=sys.stderr) diff --git a/scripts/coverage_matrix/published.py b/scripts/coverage_matrix/published.py new file mode 100644 index 00000000..a704d1d6 --- /dev/null +++ b/scripts/coverage_matrix/published.py @@ -0,0 +1,125 @@ +"""Published pages and the claims they make. + +A page under docs/published carries an anchor comment beside each claim: +{/* invariant: id */}, {/* measured: path */}, {/* config: file */}, +{/* feature: id */} or {/* generated: type-table */}. This holds the page to +the registries. Every anchor resolves, every invariant in the page's +namespace is cited, and every embedded config and generated table matches +its source. A page that drifts from its evidence fails here, rather than in a +reader's pipeline. +""" + +import os +import re + +from .registries import PUBLISHED_CONFIGS, PUBLISHED_DIR, REPO +from .types import render_types_page + +ANCHOR = re.compile( + r"\{/\*\s*(invariant|measured|config|feature|generated)\s*:\s*(.+?)\s*\*/\}") +FENCE = re.compile(r"```[a-z]*\n(.*?)```", re.S) + +# What each generated anchor compares, and the line prefix that marks its rows. +GENERATED = { + "type-table": "| `", + "unsupported-types": "- ", +} + + +def anchors(text): + """Every anchor in the page, as (kind, value), in page order.""" + return [(m.group(1), m.group(2)) for m in ANCHOR.finditer(text)] + + +def _block_after(text, end): + """The first fenced block after offset end, without its fences.""" + m = FENCE.search(text, end) + return m.group(1) if m else None + + +def _rows_after(text, end, prefix): + """The run of lines starting with prefix that first follows offset end.""" + rows = [] + for line in text[end:].splitlines(): + if line.startswith(prefix): + rows.append(line) + elif rows: + break + return rows + + +def _generated(lattice, integrations): + """The rows render_types_page emits for sink.clickhouse, by anchor.""" + integ = next((i for i in integrations if i["id"] == "sink.clickhouse"), None) + if integ is None: + return {name: [] for name in GENERATED} + lines = render_types_page(lattice, integ).splitlines() + return {name: [l for l in lines if l.startswith(prefix)] + for name, prefix in GENERATED.items()} + + +def validate_published(invariants, features, integrations, lattice, + pages_dir=PUBLISHED_DIR, configs_dir=PUBLISHED_CONFIGS, + repo=REPO, types_table=True): + """Every disagreement between a published page and its evidence, one line + each. Empty when every page holds.""" + problems = [] + if not os.path.isdir(pages_dir): + return problems + inv_ids = {i["id"] for i in invariants} + feat_ids = {f["id"] for f in features} + generated = _generated(lattice, integrations) if types_table else {} + + for name in sorted(os.listdir(pages_dir)): + if not name.endswith(".mdx"): + continue + path = os.path.join(pages_dir, name) + rel = os.path.relpath(path, repo) + with open(path) as fh: + text = fh.read() + + cited = set() + for m in ANCHOR.finditer(text): + kind, value = m.group(1), m.group(2) + if kind == "invariant": + cited.add(value) + if value not in inv_ids: + problems.append(f"{rel}: cites invariant {value}, which " + "invariants.yml does not declare") + elif kind == "feature": + if value not in feat_ids: + problems.append(f"{rel}: cites feature {value}, which " + "features.yml does not declare") + elif kind == "measured": + if not os.path.isfile(os.path.join(repo, value)): + problems.append(f"{rel}: cites measurement {value}, which " + "does not exist") + elif kind == "config": + source = os.path.join(configs_dir, value) + if not os.path.isfile(source): + problems.append(f"{rel}: embeds config {value}, which does " + "not exist") + continue + with open(source) as fh: + want = fh.read() + block = _block_after(text, m.end()) + # Only the final newline may differ: the fence supplies one. + if block is None or block.rstrip("\n") != want.rstrip("\n"): + problems.append(f"{rel}: the block after config {value} " + "differs from the published file") + elif kind == "generated" and types_table: + if value not in GENERATED: + problems.append(f"{rel}: generated {value} is not one of " + f"{sorted(GENERATED)}") + elif _rows_after(text, m.end(), GENERATED[value]) != generated[value]: + problems.append(f"{rel}: the {value} differs from the one " + "the type declaration generates") + + # A page named for a namespace answers for every invariant in it. A + # page with no namespace of its own, such as a tutorial, owes none. + namespace = name[:-len(".mdx")] + "." + for iid in sorted(inv_ids): + if iid.startswith(namespace) and iid not in cited: + problems.append(f"{rel}: invariant {iid} is cited nowhere on " + "the page") + return problems diff --git a/scripts/coverage_matrix/registries.py b/scripts/coverage_matrix/registries.py index 0ff9ff88..ce2399dc 100644 --- a/scripts/coverage_matrix/registries.py +++ b/scripts/coverage_matrix/registries.py @@ -60,6 +60,12 @@ # cannot drift from the declaration the type runner judges. TYPES_PAGE = os.path.join(REPO, "docs", "coverage", "clickhouse-types.mdx") +# Pages published in third-party docs, and the configs they embed. The pages +# are the source of truth for what a vendor's docs say about SQLFlow, and the +# page check holds each claim on them to its evidence. +PUBLISHED_DIR = os.path.join(REPO, "docs", "published") +PUBLISHED_CONFIGS = os.path.join(REPO, "dev", "config", "examples", "published") + PASS, SKIP, FAIL = "pass", "skip", "fail" diff --git a/tests/tooling/test_published.py b/tests/tooling/test_published.py new file mode 100644 index 00000000..e1e78da1 --- /dev/null +++ b/tests/tooling/test_published.py @@ -0,0 +1,102 @@ +"""Tests for coverage_matrix.published.""" + +from coverage_matrix.published import anchors, validate_published +from samples import * # noqa: F401,F403 + +INVS = [ + {"id": "clickhouse.a", "applies_to": "sink.clickhouse"}, + {"id": "clickhouse.b", "applies_to": "sink.clickhouse"}, + {"id": "type.null", "applies_to": "sink"}, +] +FEATS = [{"id": "cli.validate"}] +BOTH = "{/* invariant: clickhouse.a */}{/* invariant: clickhouse.b */}\n" + + +def page(tmp_path, name, text, configs=None, files=None): + pages = tmp_path / "pages" + pages.mkdir(exist_ok=True) + (pages / name).write_text(text) + cfg = tmp_path / "configs" + cfg.mkdir(exist_ok=True) + for k, v in (configs or {}).items(): + (cfg / k).write_text(v) + for k, v in (files or {}).items(): + p = tmp_path / k + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(v) + return validate_published(INVS, FEATS, [], [], pages_dir=str(pages), + configs_dir=str(cfg), repo=str(tmp_path), + types_table=False) + + +def test_anchors_tolerate_spacing(): + assert anchors("x {/* invariant: clickhouse.a */} y") == [ + ("invariant", "clickhouse.a")] + + +def test_a_page_citing_every_namespaced_invariant_passes(tmp_path): + assert page(tmp_path, "clickhouse.mdx", BOTH) == [] + + +def test_an_uncited_namespaced_invariant_is_a_problem(tmp_path): + problems = page(tmp_path, "clickhouse.mdx", "{/* invariant: clickhouse.a */}\n") + assert any("clickhouse.b" in p and "cited nowhere" in p for p in problems) + + +def test_an_unknown_invariant_anchor_is_a_problem(tmp_path): + problems = page(tmp_path, "clickhouse.mdx", + BOTH + "{/* invariant: clickhouse.zz */}") + assert any("clickhouse.zz" in p for p in problems) + + +def test_a_page_with_no_namespace_needs_no_citations(tmp_path): + assert page(tmp_path, "motherduck.mdx", "no anchors here\n") == [] + + +def test_a_measured_anchor_must_name_a_file(tmp_path): + text = BOTH + "{/* measured: docs/published/clickhouse/perf.out */}" + assert any("perf.out" in p for p in page(tmp_path, "clickhouse.mdx", text)) + assert page(tmp_path, "clickhouse.mdx", text, + files={"docs/published/clickhouse/perf.out": "x"}) == [] + + +def test_a_config_block_must_equal_its_file(tmp_path): + text = BOTH + "{/* config: t.yml */}\n```yaml\na: 1\n```\n" + assert page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 1\n"}) == [] + assert page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 1"}) == [] + bad = page(tmp_path, "clickhouse.mdx", text, configs={"t.yml": "a: 2\n"}) + assert any("t.yml" in p and "differs" in p for p in bad) + + +def test_a_config_anchor_with_no_file_is_a_problem(tmp_path): + text = BOTH + "{/* config: gone.yml */}\n```yaml\na: 1\n```\n" + assert any("gone.yml" in p for p in page(tmp_path, "clickhouse.mdx", text)) + + +def test_a_feature_anchor_must_name_a_feature(tmp_path): + assert page(tmp_path, "clickhouse.mdx", BOTH + "{/* feature: cli.validate */}") == [] + assert any("cli.nope" in p for p in + page(tmp_path, "clickhouse.mdx", BOTH + "{/* feature: cli.nope */}")) + + +def test_a_generated_table_must_match_the_declaration(tmp_path): + lattice = [{"key": "int8", "duckdb": ["TINYINT"]}] + integs = [{"id": "sink.clickhouse", "types": {"int8": { + "outcome": "exact", "columns": [{"type": "Int8", "expect": "1"}]}}}] + pages = tmp_path / "pages" + pages.mkdir() + good = BOTH + "{/* generated: type-table */}\n| `TINYINT` | `Int8` | |\n\n" + (pages / "clickhouse.mdx").write_text(good) + assert validate_published(INVS, FEATS, integs, lattice, pages_dir=str(pages), + configs_dir=str(tmp_path), repo=str(tmp_path)) == [] + (pages / "clickhouse.mdx").write_text(good.replace("`Int8` |", "`Int16` |")) + problems = validate_published(INVS, FEATS, integs, lattice, pages_dir=str(pages), + configs_dir=str(tmp_path), repo=str(tmp_path)) + assert any("type-table" in p for p in problems) + + +def test_the_committed_pages_pass(): + from coverage_matrix.registries import (load_features, load_integrations, + load_invariants, load_lattice) + assert validate_published(load_invariants(), load_features(), + load_integrations(), load_lattice()) == [] From 1896549d8938b462e2e76345694380a1bd44c256 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 05:59:56 -0400 Subject: [PATCH 05/12] coverage: the type table names VARCHAR alone, and NULL into an Enum The generated row "VARCHAR, UUID, JSON" against nine column types read as a cross-product. The type test proves each destination with its own value, so a UUID string never reparses as a DateTime or a Decimal. The ClickHouse reviewer flagged it as a major on #117740. The NULL row said every NULL becomes the type's zero value, which hides that an Enum with no 0 member then holds a row no SELECT can read. A carrier whose columns carry their own values now renders its first cast alone, with a note that the other casts reach the sink as it. The utf8 declaration gains the text-form rule, and the NULL rule names the Enum case. clickhouse-types.mdx is regenerated. Two tests in tests/tooling cover the carrier row and a row with no per-column value, which keeps every cast. If the rendering is wrong, the published page check fails, because the page's table must equal this one. --- docs/coverage/clickhouse-types.mdx | 4 ++-- .../coverage/integrations/sink.clickhouse.yml | 3 ++- scripts/coverage_matrix/types.py | 12 ++++++++++- tests/tooling/test_types.py | 21 ++++++++++++++++++- 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/docs/coverage/clickhouse-types.mdx b/docs/coverage/clickhouse-types.mdx index 03e3e51f..528804cd 100644 --- a/docs/coverage/clickhouse-types.mdx +++ b/docs/coverage/clickhouse-types.mdx @@ -18,7 +18,7 @@ SQLFlow hands each batch to ClickHouse as rows built from the Arrow table DuckDB | `UBIGINT` | `UInt64` | | | `REAL` | `Float32` | | | `DOUBLE` | `Float64` | | -| `VARCHAR`, `UUID`, `JSON` | `String`, `LowCardinality(String)`, `DateTime64(3)`, `DateTime`, `Enum8('' = 0, 'a' = 1, 'b' = 2)`, `FixedString(36)`, `UUID`, `IPv4`, `Decimal(10, 2)` | | +| `VARCHAR` | `String`, `LowCardinality(String)`, `DateTime64(3)`, `DateTime`, `Enum8('' = 0, 'a' = 1, 'b' = 2)`, `FixedString(36)`, `UUID`, `IPv4`, `Decimal(10, 2)` | the text must already be the column's text form. `UUID` and `JSON` reach the sink as `VARCHAR`. | | `BLOB`, `BIT`, `VARINT` | `String` | | | `DATE` | `Date`, `Date32` | | | `TIMESTAMP_S` | `DateTime` | | @@ -46,7 +46,7 @@ SQLFlow hands each batch to ClickHouse as rows built from the Arrow table DuckDB | `TIMESTAMP_NS[]` | `Array(DateTime64(9))` | | | `TIMESTAMPTZ[]` | `Array(DateTime64(6))` | stored as the same UTC instant; the zone is dropped and the column's own zone governs rendering | | `BIGINT[][]` | `Array(Array(Int64))` | | -| `NULL` in any column | the column's own type | the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT | +| `NULL` in any column | the column's own type | the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT. For an Enum with no 0 member that value is not a member, and the row cannot be read | | `NULL` inside a `LIST` | the element's own type | the element type's zero value, because ClickHouse's Array(T) holds no nulls unless T is Nullable | Values are matched to columns **by name**, not by position. diff --git a/docs/coverage/integrations/sink.clickhouse.yml b/docs/coverage/integrations/sink.clickhouse.yml index fc1716d0..c387b517 100644 --- a/docs/coverage/integrations/sink.clickhouse.yml +++ b/docs/coverage/integrations/sink.clickhouse.yml @@ -82,6 +82,7 @@ types: # override the value the lattice sends. utf8: outcome: exact + rule: the text must already be the column's text form. columns: - type: String expect: "L'Ε’il πŸ‘ \"quoted\" back\\slash\ttab" @@ -374,7 +375,7 @@ types: nulls: default: outcome: coerced - rule: the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT + rule: the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT. For an Enum with no 0 member that value is not a member, and the row cannot be read # The third null position, and the one the published page does not # mention. arrowListValue appends the element type's zero value at # clickhouse.go:429-431, so [1, null, 3] lands as [1, 0, 3] with no diff --git a/scripts/coverage_matrix/types.py b/scripts/coverage_matrix/types.py index 33027051..a35cdc2e 100644 --- a/scripts/coverage_matrix/types.py +++ b/scripts/coverage_matrix/types.py @@ -35,7 +35,17 @@ def render_types_page(lattice, integration): # A column entry, not a name: the value written belongs to the (key, # column) pair, and the page publishes the pair's destination type. cols = ", ".join("`%s`" % c["type"] for c in decl.get("columns", [])) - supported.append("| %s | %s | %s |" % (casts, cols, decl.get("rule", ""))) + rule = decl.get("rule", "") + # A carrier whose columns each take their own value proves the pair, + # not the cross-product: a UUID cast reaches a Decimal column only as + # text the Decimal parses. Name the cast the matrix wrote, and say the + # others arrive as it. + if any(c.get("value") for c in decl.get("columns", [])) and len(entry["duckdb"]) > 1: + first, rest = entry["duckdb"][0], entry["duckdb"][1:] + casts = "`%s`" % first + rule = ("%s %s reach the sink as `%s`." % ( + rule, " and ".join("`%s`" % r for r in rest), first)).strip() + supported.append("| %s | %s | %s |" % (casts, cols, rule)) out = [ "", diff --git a/tests/tooling/test_types.py b/tests/tooling/test_types.py index bcf7a37c..c3a2db9f 100644 --- a/tests/tooling/test_types.py +++ b/tests/tooling/test_types.py @@ -124,4 +124,23 @@ def test_the_page_says_it_is_generated(): "utf8": {"outcome": "exact", "columns": [{"type": "String", "expect": "a"}]}, })[0]) assert "Do not edit" in md - assert "TestIntegrationSinkClickhouse_Types" in md \ No newline at end of file + assert "TestIntegrationSinkClickhouse_Types" in md + +def test_a_text_carrier_row_names_its_cast_alone_and_says_the_rest_reach_it(): + """The utf8 row proves each destination with its own value, so it must not + read as a cross-product of every cast and every column.""" + lattice = [{"key": "utf8", "duckdb": ["VARCHAR", "UUID", "JSON"]}] + integ = {"types": {"utf8": {"outcome": "exact", + "rule": "the text must already be the column's text form.", + "columns": [{"type": "String", "expect": "x"}, + {"type": "UUID", "value": "u", "expect": "u"}]}}} + page = render_types_page(lattice, integ) + assert ("| `VARCHAR` | `String`, `UUID` | the text must already be the " + "column's text form. `UUID` and `JSON` reach the sink as `VARCHAR`. |") in page + + +def test_a_row_with_no_per_column_value_keeps_every_cast(): + lattice = [{"key": "binary", "duckdb": ["BLOB", "BIT"]}] + integ = {"types": {"binary": {"outcome": "exact", + "columns": [{"type": "String", "expect": "x"}]}}} + assert "| `BLOB`, `BIT` | `String` | |" in render_types_page(lattice, integ) From 87c64732fcc1226980dedb651c3392529fccf264 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 06:07:30 -0400 Subject: [PATCH 06/12] invariants: twelve clickhouse.* claims, and the DSN and TLS proofs The ClickHouse page states twelve behaviors only the ClickHouse sink has, from the DSN schemes to how a replay reads in ReplacingMergeTree. None had a declared claim or a test, so each could change with no CI signal. invariants.yml declares all twelve under a ClickHouse section, scoped to sink.clickhouse, each requiring the level its evidence runs at. The header and the registry test now allow a scoped invariant to require its level from the start, since it has one subject. Two unit tests prove the DSN scheme table, raw and percent-encoded passwords, and that TLS verifies against the system roots with no option to skip. The two tests ran green, and a mutation that moved the TLS port and set InsecureSkipVerify failed both. The other ten invariants show as missing until their tests land; coverage-check reports them as gaps meanwhile. --- docs/coverage/invariants.yml | 144 ++++++++++++++++++- docs/coverage/matrix.md | 36 ++++- internal/sinks/clickhouse_invariants_test.go | 75 ++++++++++ tests/tooling/test_registries.py | 13 +- 4 files changed, 257 insertions(+), 11 deletions(-) create mode 100644 internal/sinks/clickhouse_invariants_test.go diff --git a/docs/coverage/invariants.yml b/docs/coverage/invariants.yml index 130dc7df..56b47369 100644 --- a/docs/coverage/invariants.yml +++ b/docs/coverage/invariants.yml @@ -7,7 +7,8 @@ # # `applies_to` is the interface the invariant is a property of. Every # integration of that kind must prove it, or be exempt in the registry -# with a reason. +# with a reason. It may instead name one integration, for a claim only that +# integration makes; the id then starts with the integration's namespace. # # `verified_by` says how evidence attaches: # @@ -19,10 +20,11 @@ # never implemented, and dropped as a third way to claim the same evidence. # scripts/coverage_matrix accepts the two above and nothing else. # -# `requires` lists the levels evidence must exist at. Every entry is empty in -# this revision: the matrix reports, and nothing fails. An invariant is -# enforced by filling one in, which is legal only once every non-exempt -# integration has a subject. +# `requires` lists the levels evidence must exist at. Every kind's entry is +# empty in this revision: the matrix reports, and nothing fails. An invariant +# is enforced by filling one in, which is legal only once every non-exempt +# integration has a subject. An invariant scoped to one integration, such as +# clickhouse.*, is written with its subject and requires its level. # # `tracked_by` marks an invariant the code does not yet satisfy. It renders as # declared-unenforced until that issue lands. @@ -396,6 +398,138 @@ invariants: verified_by: typetable requires: [] + # --- ClickHouse ---------------------------------------------------------- + # Claims only sink.clickhouse makes. Each one backs a sentence on + # docs/published/clickhouse.mdx, which cites it by id, and the page check + # fails when the two disagree. A failing test here is a page that is now + # wrong: fix the page or the sink, never the assertion. + - id: clickhouse.insert.by_name + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + Columns are matched by name. A produced column the table lacks + fails the batch as `system.sink.write_failed`, naming the column. + A column the handler omits takes the table's DEFAULT. + verified_by: harness + requires: [integration] + + - id: clickhouse.table.not_created + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + The sink never creates or alters a table. A table that does not + exist fails the batch as `system.sink.write_failed`. + verified_by: harness + requires: [integration] + + - id: clickhouse.enum.null_is_zero + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + A NULL into a non-Nullable Enum is stored as 0. With no 0 member + the INSERT succeeds and reads of the column fail with + `UNKNOWN_ELEMENT_OF_ENUM`. With a 0 member it reads as that + member. Nullable reads NULL. + verified_by: harness + requires: [integration] + + - id: clickhouse.encode.fails_once + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + A value the driver cannot encode fails the batch on the first + attempt as `user.sink.encode_failed`. It is not retried, and the + process exits with code 10. + verified_by: harness + requires: [integration] + + - id: clickhouse.temporal_text.two_shapes + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as + UTC, or the same with an offset, which is applied. Any other shape + fails as `user.sink.encode_failed`. + verified_by: harness + requires: [integration] + + - id: clickhouse.decimal.text_only + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + A Decimal column accepts its value as text. A float fails as + `user.sink.encode_failed`. + verified_by: harness + requires: [integration] + + - id: clickhouse.replay.duplicates_until_merge + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + A batch delivered twice into a ReplacingMergeTree reads twice + until a merge, and once with FINAL. An identical block inside the + table's deduplication window is dropped at insert. + verified_by: harness + requires: [integration] + + - id: clickhouse.flush.one_insert + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + One flush is one INSERT, on one connection. + verified_by: harness + requires: [integration] + + - id: clickhouse.dsn.scheme + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + The DSN scheme decides the protocol, TLS and the default port. `@` + and `:` in a password parse raw. `/`, `#` and `?` must be percent- + encoded. + verified_by: harness + requires: [unit] + + - id: clickhouse.tls.verifies + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + With TLS, the server certificate is verified against the system + roots. No option skips verification. + verified_by: harness + requires: [unit] + + - id: clickhouse.auth.fails_start + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + Wrong credentials fail at startup with HTTP 401, before any + message is consumed. + verified_by: harness + requires: [integration] + + - id: clickhouse.type_text.per_destination + family: clickhouse + class: safety + applies_to: sink.clickhouse + claim: > + Text into a non-String column must already be in that column's + text form. The type table proves each destination with its own + value. + verified_by: harness + requires: [integration] + # --- Lifecycle ----------------------------------------------------------- - id: lifecycle.drain.on_cancel family: lifecycle diff --git a/docs/coverage/matrix.md b/docs/coverage/matrix.md index c2f266e4..49ebe1fd 100644 --- a/docs/coverage/matrix.md +++ b/docs/coverage/matrix.md @@ -82,7 +82,7 @@ integration behind it keeps a batch it could not deliver, or commits offsets only after a flush. Those are invariants, they are counted separately below, and the two numbers are not interchangeable. -**55 invariants declared: 47 safety and 8 liveness. Of 195 (invariant, integration) cells: 97 proven, 63 missing, 0 skipped, 0 failing, 35 exempt. 0 gap(s).** +**67 invariants declared: 59 safety and 8 liveness. Of 207 (invariant, integration) cells: 97 proven, 75 missing, 0 skipped, 0 failing, 35 exempt. 12 gap(s).** Safety says nothing bad happens. Liveness says something good eventually does, and the two are not interchangeable: a sink that @@ -224,6 +224,23 @@ drains. An invariant holds only if it holds on all four. | `serve.cache.errors_not_stored` | A request that fails stores nothing: the next request for its key runs a query. | βœ… u | | `serve.cache.one_fill` | Concurrent requests for one key run one query, and a caller leaving fails no other caller. | βœ… u | +## Safety invariants: clickhouse + +| Invariant | Claim | `sink.clickhouse` | +| --- | --- | --- | +| `clickhouse.insert.by_name` | Columns are matched by name. A produced column the table lacks fails the batch as `system.sink.write_failed`, naming the column. A column the handler omits takes the table's DEFAULT. | ❌ missing | +| `clickhouse.table.not_created` | The sink never creates or alters a table. A table that does not exist fails the batch as `system.sink.write_failed`. | ❌ missing | +| `clickhouse.enum.null_is_zero` | A NULL into a non-Nullable Enum is stored as 0. With no 0 member the INSERT succeeds and reads of the column fail with `UNKNOWN_ELEMENT_OF_ENUM`. With a 0 member it reads as that member. Nullable reads NULL. | ❌ missing | +| `clickhouse.encode.fails_once` | A value the driver cannot encode fails the batch on the first attempt as `user.sink.encode_failed`. It is not retried, and the process exits with code 10. | ❌ missing | +| `clickhouse.temporal_text.two_shapes` | Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as UTC, or the same with an offset, which is applied. Any other shape fails as `user.sink.encode_failed`. | ❌ missing | +| `clickhouse.decimal.text_only` | A Decimal column accepts its value as text. A float fails as `user.sink.encode_failed`. | ❌ missing | +| `clickhouse.replay.duplicates_until_merge` | A batch delivered twice into a ReplacingMergeTree reads twice until a merge, and once with FINAL. An identical block inside the table's deduplication window is dropped at insert. | ❌ missing | +| `clickhouse.flush.one_insert` | One flush is one INSERT, on one connection. | ❌ missing | +| `clickhouse.dsn.scheme` | The DSN scheme decides the protocol, TLS and the default port. `@` and `:` in a password parse raw. `/`, `#` and `?` must be percent- encoded. | ❌ missing | +| `clickhouse.tls.verifies` | With TLS, the server certificate is verified against the system roots. No option skips verification. | ❌ missing | +| `clickhouse.auth.fails_start` | Wrong credentials fail at startup with HTTP 401, before any message is consumed. | ❌ missing | +| `clickhouse.type_text.per_destination` | Text into a non-String column must already be in that column's text form. The type table proves each destination with its own value. | ❌ missing | + ## Liveness invariants: lifecycle | Invariant | Claim | `manager.watermark` | @@ -252,3 +269,20 @@ drains. An invariant holds only if it holds on all four. | --- | --- | --- | | `serve.cache.answers_from_memory` | A second request for a key inside its TTL is answered without a query and without a session. | βœ… u | +## Invariant gaps + +These fail `make coverage-check`. + +- `clickhouse.insert.by_name` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.table.not_created` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.enum.null_is_zero` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.encode.fails_once` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.temporal_text.two_shapes` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.decimal.text_only` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.replay.duplicates_until_merge` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.flush.one_insert` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.dsn.scheme` on `sink.clickhouse` requires **unit** and is *missing*. +- `clickhouse.tls.verifies` on `sink.clickhouse` requires **unit** and is *missing*. +- `clickhouse.auth.fails_start` on `sink.clickhouse` requires **integration** and is *missing*. +- `clickhouse.type_text.per_destination` on `sink.clickhouse` requires **integration** and is *missing*. + diff --git a/internal/sinks/clickhouse_invariants_test.go b/internal/sinks/clickhouse_invariants_test.go new file mode 100644 index 00000000..a533ca16 --- /dev/null +++ b/internal/sinks/clickhouse_invariants_test.go @@ -0,0 +1,75 @@ +package sinks + +// The clickhouse.* invariants: claims only the ClickHouse sink makes, each +// one a sentence on docs/published/clickhouse.mdx. A test here that fails is +// a page that is now wrong. Fix the page or the sink, never the assertion. + +import ( + "reflect" + "strings" + "testing" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/turbolytics/sql-flow/internal/config" + "github.com/turbolytics/sql-flow/internal/coverage" + "github.com/zeebo/assert" +) + +func TestSinkClickhouse_DSNSchemeDecidesProtocolTLSAndPort(t *testing.T) { + coverage.Invariant(t, "clickhouse.dsn.scheme", "sink.clickhouse") + + cases := []struct { + dsn string + protocol clickhouse.Protocol + tls bool + addr string + }{ + {dsn: "clickhouses://h/db", protocol: clickhouse.HTTP, tls: true, addr: "h:8443"}, + {dsn: "https://h/db", protocol: clickhouse.HTTP, tls: true, addr: "h:8443"}, + {dsn: "clickhouse://h/db", protocol: clickhouse.HTTP, tls: false, addr: "h:8123"}, + {dsn: "http://h/db", protocol: clickhouse.HTTP, tls: false, addr: "h:8123"}, + {dsn: "natives://h/db", protocol: clickhouse.Native, tls: true, addr: "h:9440"}, + {dsn: "tcp://h/db", protocol: clickhouse.Native, tls: false, addr: "h:9000"}, + {dsn: "native://h/db", protocol: clickhouse.Native, tls: false, addr: "h:9000"}, + } + for _, c := range cases { + opts, err := clickhouseOptions(c.dsn) + assert.NoError(t, err) + assert.Equal(t, c.protocol, opts.Protocol) + assert.Equal(t, c.tls, opts.TLS != nil) + assert.DeepEqual(t, []string{c.addr}, opts.Addr) + } + + // @ and : in a password parse raw. + opts, err := clickhouseOptions("clickhouses://u:p@ss:w0rd@h/db") + assert.NoError(t, err) + assert.Equal(t, "p@ss:w0rd", opts.Auth.Password) + + // # left raw ends the URL early, so what follows reads as a port. + _, err = clickhouseOptions("clickhouses://u:pa#ss@h/db") + assert.Error(t, err) + assert.That(t, strings.Contains(err.Error(), "invalid port")) + + // Percent-encoded, the same password parses. + opts, err = clickhouseOptions("clickhouses://u:pa%23ss@h/db") + assert.NoError(t, err) + assert.Equal(t, "pa#ss", opts.Auth.Password) +} + +func TestSinkClickhouse_TLSVerifiesAgainstTheSystemRoots(t *testing.T) { + coverage.Invariant(t, "clickhouse.tls.verifies", "sink.clickhouse") + + opts, err := clickhouseOptions("clickhouses://h/db") + assert.NoError(t, err) + assert.False(t, opts.TLS.InsecureSkipVerify) + assert.Nil(t, opts.TLS.RootCAs) + + // No option skips verification: the sink's config is a DSN and a table. + // A field added here is a page claim to revisit. + var names []string + typ := reflect.TypeOf(config.ClickhouseSink{}) + for i := 0; i < typ.NumField(); i++ { + names = append(names, typ.Field(i).Name) + } + assert.DeepEqual(t, []string{"DSN", "Table"}, names) +} diff --git a/tests/tooling/test_registries.py b/tests/tooling/test_registries.py index b4046443..c6606ed2 100644 --- a/tests/tooling/test_registries.py +++ b/tests/tooling/test_registries.py @@ -96,12 +96,15 @@ def test_the_committed_registries_load_and_validate(): "sink", "source", "handler", "pipeline", "manager", "serve"} -def test_every_committed_invariant_is_unenforced_in_this_revision(): - """This revision reports and fails nothing. Filling in a requires is a - later change, and legal only once every non-exempt integration has a - subject.""" +def test_every_committed_kind_invariant_is_unenforced_in_this_revision(): + """A kind's invariants report and fail nothing yet. Filling in a requires + is legal only once every non-exempt integration of the kind has a + subject. An invariant scoped to one integration has one subject, written + with it, so it may require its level from the start.""" + kinds = {"sink", "source", "handler", "pipeline", "manager", "serve"} for inv in load_invariants(): - assert inv["requires"] == [], inv["id"] + if inv["applies_to"] in kinds: + assert inv["requires"] == [], inv["id"] def test_validate_rejects_an_exemption_naming_no_invariant(): From c426e26455a33f49c73b205613ad4aeb567d60a2 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 06:11:44 -0400 Subject: [PATCH 07/12] invariants: ClickHouse matches columns by name, creates no table, and stores a NULL Enum as 0 Three page claims had only a hand-run script behind them: columns match by name and an omitted one takes the table's DEFAULT, the sink never creates a table, and a NULL into an Enum with no 0 member writes a row no SELECT can read. Three integration tests prove them against the shared ClickHouse 26.8 container, and the type test now marks the per-destination text claim it already proves. All four pass. Replacing the INSERT's column list with a positional INSERT fails the by-name test, so it binds the sink's behavior. If a claim changes, its test fails, and the page sentence citing it is the one to fix. --- internal/sinks/clickhouse_invariants_test.go | 142 +++++++++++++++++++ internal/sinks/conformance_types_test.go | 3 + 2 files changed, 145 insertions(+) diff --git a/internal/sinks/clickhouse_invariants_test.go b/internal/sinks/clickhouse_invariants_test.go index a533ca16..c574e354 100644 --- a/internal/sinks/clickhouse_invariants_test.go +++ b/internal/sinks/clickhouse_invariants_test.go @@ -5,13 +5,21 @@ package sinks // a page that is now wrong. Fix the page or the sink, never the assertion. import ( + "context" + "fmt" "reflect" "strings" "testing" + "time" "github.com/ClickHouse/clickhouse-go/v2" + "github.com/apache/arrow-go/v18/arrow" + "github.com/apache/arrow-go/v18/arrow/array" + "github.com/apache/arrow-go/v18/arrow/memory" "github.com/turbolytics/sql-flow/internal/config" + "github.com/turbolytics/sql-flow/internal/core" "github.com/turbolytics/sql-flow/internal/coverage" + "github.com/turbolytics/sql-flow/internal/errs" "github.com/zeebo/assert" ) @@ -73,3 +81,137 @@ func TestSinkClickhouse_TLSVerifiesAgainstTheSystemRoots(t *testing.T) { } assert.DeepEqual(t, []string{"DSN", "Table"}, names) } + +type textCol struct { + name string + values []*string +} + +func str(s string) *string { return &s } + +// textColumns builds one batch of nullable utf8 columns, in the order given. +func textColumns(t *testing.T, cols ...textCol) arrow.Table { + t.Helper() + fields := make([]arrow.Field, len(cols)) + for i, c := range cols { + fields[i] = arrow.Field{ + Name: c.name, + Type: arrow.BinaryTypes.String, + Nullable: true, + } + } + schema := arrow.NewSchema(fields, nil) + b := array.NewRecordBuilder(memory.NewGoAllocator(), schema) + defer b.Release() + for i, c := range cols { + sb := b.Field(i).(*array.StringBuilder) + for _, v := range c.values { + if v == nil { + sb.AppendNull() + } else { + sb.Append(*v) + } + } + } + rec := b.NewRecord() + defer rec.Release() + tbl := array.NewTableFromRecords(schema, []arrow.Record{rec}) + t.Cleanup(tbl.Release) + return tbl +} + +// flushOnce writes one batch and flushes it, which is one delivery. +func flushOnce(t *testing.T, s core.Sink, tbl arrow.Table) error { + t.Helper() + assert.NoError(t, s.WriteTable(context.Background(), tbl)) + return s.Flush(context.Background()) +} + +func TestIntegrationSinkClickhouse_InsertsByName(t *testing.T) { + coverage.Invariant(t, "clickhouse.insert.by_name", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + s := newLiveClickhouseSink(t, `CREATE TABLE %s ( + b String, a String, d String DEFAULT 'omitted-default' + ) ENGINE = MergeTree() ORDER BY tuple()`) + + // Produced in the opposite order to the table, and without d. + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "a", values: []*string{str("A")}}, + textCol{name: "b", values: []*string{str("B")}}))) + var a, b, d string + assert.NoError(t, s.conn.QueryRow(context.Background(), + "SELECT a, b, d FROM "+s.table).Scan(&a, &b, &d)) + assert.Equal(t, "A", a) + assert.Equal(t, "B", b) + assert.Equal(t, "omitted-default", d) + + // A column the table lacks fails the batch, naming it. + err := flushOnce(t, s, textColumns(t, + textCol{name: "a", values: []*string{str("A")}}, + textCol{name: "extra", values: []*string{str("X")}})) + assert.Error(t, err) + assert.That(t, errs.HasCode(err, errs.CodeSinkWriteFailed)) + assert.That(t, strings.Contains(err.Error(), "column extra is not present in the table")) +} + +func TestIntegrationSinkClickhouse_NeverCreatesATable(t *testing.T) { + coverage.Invariant(t, "clickhouse.table.not_created", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + table := fmt.Sprintf("absent_%d", time.Now().UnixNano()) + s := mustDirectSink(t, sharedClickhouse(t), table) + + err := flushOnce(t, s, textColumns(t, textCol{name: "a", values: []*string{str("A")}})) + assert.Error(t, err) + assert.That(t, errs.HasCode(err, errs.CodeSinkWriteFailed)) + + var exists uint8 + assert.NoError(t, s.conn.QueryRow(context.Background(), + "EXISTS TABLE "+table).Scan(&exists)) + assert.Equal(t, uint8(0), exists) +} + +func TestIntegrationSinkClickhouse_EnumNullIsZero(t *testing.T) { + coverage.Invariant(t, "clickhouse.enum.null_is_zero", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + null := textColumns(t, textCol{name: "e", values: []*string{nil}}) + ctx := context.Background() + + // No 0 member: the INSERT succeeds and the row cannot be read. + noZero := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Enum8('a' = 1, 'b' = 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, noZero, null)) + var n uint64 + assert.NoError(t, noZero.conn.QueryRow(ctx, "SELECT count() FROM "+noZero.table).Scan(&n)) + assert.Equal(t, uint64(1), n) + var e string + err := noZero.conn.QueryRow(ctx, "SELECT toString(e) FROM "+noZero.table).Scan(&e) + assert.Error(t, err) + assert.That(t, strings.Contains(err.Error(), "UNKNOWN_ELEMENT_OF_ENUM")) + var raw int8 + assert.NoError(t, noZero.conn.QueryRow(ctx, "SELECT CAST(e AS Int8) FROM "+noZero.table).Scan(&raw)) + assert.Equal(t, int8(0), raw) + + // A 0 member: the NULL reads as that member. + zero := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Enum8('' = 0, 'a' = 1, 'b' = 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, zero, null)) + assert.NoError(t, zero.conn.QueryRow(ctx, "SELECT toString(e) FROM "+zero.table).Scan(&e)) + assert.Equal(t, "", e) + + // Nullable: the NULL reads as NULL. + nullable := newLiveClickhouseSink(t, + `CREATE TABLE %s (e Nullable(Enum8('a' = 1, 'b' = 2))) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, nullable, null)) + var isNull uint8 + assert.NoError(t, nullable.conn.QueryRow(ctx, "SELECT e IS NULL FROM "+nullable.table).Scan(&isNull)) + assert.Equal(t, uint8(1), isNull) +} diff --git a/internal/sinks/conformance_types_test.go b/internal/sinks/conformance_types_test.go index 4b80a294..f8152c1c 100644 --- a/internal/sinks/conformance_types_test.go +++ b/internal/sinks/conformance_types_test.go @@ -25,6 +25,9 @@ import ( func TestIntegrationSinkClickhouse_Types(t *testing.T) { coverage.Covers(t, "sink.clickhouse") + // The type table proves each text destination with that destination's own + // value, which is the claim the page makes about VARCHAR. + coverage.Invariant(t, "clickhouse.type_text.per_destination", "sink.clickhouse") if testing.Short() { t.Skip("integration test: -short runs the unit pass only") From 9590cd280a7d355a0bb521b22897d6c1828fa4e2 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 06:13:23 -0400 Subject: [PATCH 08/12] invariants: an encode failure fails once, and text reaches DateTime and Decimal only in their forms The page's retry bullet was stale for a release because nothing tied it to #269: an unencodable value now fails once as user.sink.encode_failed and exits 10, where the page said it ran the retry ladder. The text-timestamp shapes and the Decimal-from-text rule had the same gap. Three integration tests prove them through the retry wrapper and against the shared ClickHouse container: no retry and exit code 10 for an ISO 8601 string into DateTime, the two accepted timestamp shapes with the offset applied and everything else refused, and Decimal from text with a float refused. All pass. Making every error retryable fails the no-retry test. --- internal/sinks/clickhouse_invariants_test.go | 89 ++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/internal/sinks/clickhouse_invariants_test.go b/internal/sinks/clickhouse_invariants_test.go index c574e354..58cac0ef 100644 --- a/internal/sinks/clickhouse_invariants_test.go +++ b/internal/sinks/clickhouse_invariants_test.go @@ -215,3 +215,92 @@ func TestIntegrationSinkClickhouse_EnumNullIsZero(t *testing.T) { assert.NoError(t, nullable.conn.QueryRow(ctx, "SELECT e IS NULL FROM "+nullable.table).Scan(&isNull)) assert.Equal(t, uint8(1), isNull) } + +func TestIntegrationSinkClickhouse_EncodeFailureIsNotRetried(t *testing.T) { + coverage.Invariant(t, "clickhouse.encode.fails_once", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (ts DateTime) ENGINE = MergeTree() ORDER BY tuple()`) + r := newRetrying(s, RetryPolicy{ + MaxAttempts: 4, + InitialBackoff: time.Millisecond, + MaxBackoff: time.Millisecond, + Deadline: time.Minute, + }) + retries := 0 + r.onRetry = func(int, error) { retries++ } + + err := flushOnce(t, r, textColumns(t, + textCol{name: "ts", values: []*string{str("2026-09-01T12:00:00Z")}})) + assert.Error(t, err) + assert.That(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) + assert.Equal(t, 0, retries) + assert.Equal(t, errs.ExitUserError, errs.ExitCode(err)) +} + +func TestIntegrationSinkClickhouse_TemporalTextTakesTwoShapes(t *testing.T) { + coverage.Invariant(t, "clickhouse.temporal_text.two_shapes", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (k String, ts DateTime('UTC')) ENGINE = MergeTree() ORDER BY k`) + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "k", values: []*string{str("plain"), str("offset")}}, + textCol{name: "ts", values: []*string{str("2026-09-01 12:00:00"), str("2026-09-01 12:00:00 +09:00")}}))) + + got := map[string]string{} + rows, err := s.conn.Query(ctx, "SELECT k, toString(ts) FROM "+s.table) + assert.NoError(t, err) + for rows.Next() { + var k, ts string + assert.NoError(t, rows.Scan(&k, &ts)) + got[k] = ts + } + assert.NoError(t, rows.Err()) + rows.Close() + assert.Equal(t, "2026-09-01 12:00:00", got["plain"]) + assert.Equal(t, "2026-09-01 03:00:00", got["offset"]) + + // Any other shape fails before it reaches the server. + err = flushOnce(t, s, textColumns(t, + textCol{name: "k", values: []*string{str("iso")}}, + textCol{name: "ts", values: []*string{str("2026-09-01T12:00:00Z")}})) + assert.Error(t, err) + assert.That(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) +} + +func TestIntegrationSinkClickhouse_DecimalTakesText(t *testing.T) { + coverage.Invariant(t, "clickhouse.decimal.text_only", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (amount Decimal(10, 2)) ENGINE = MergeTree() ORDER BY tuple()`) + assert.NoError(t, flushOnce(t, s, textColumns(t, + textCol{name: "amount", values: []*string{str("12.34")}}))) + var got string + assert.NoError(t, s.conn.QueryRow(ctx, "SELECT toString(amount) FROM "+s.table).Scan(&got)) + assert.Equal(t, "12.34", got) + + schema := arrow.NewSchema([]arrow.Field{{ + Name: "amount", + Type: arrow.PrimitiveTypes.Float64, + }}, nil) + b := array.NewRecordBuilder(memory.NewGoAllocator(), schema) + defer b.Release() + b.Field(0).(*array.Float64Builder).Append(12.34) + rec := b.NewRecord() + defer rec.Release() + floats := array.NewTableFromRecords(schema, []arrow.Record{rec}) + defer floats.Release() + + // A float is refused: the driver has no exact conversion to Decimal. + err := flushOnce(t, s, floats) + assert.Error(t, err) + assert.That(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) +} From 1c858c29c71b873785004642b5b1a0ebcd85f714 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 06:17:03 -0400 Subject: [PATCH 09/12] invariants: replays read twice until a merge, a flush is one INSERT, and wrong credentials fail the probe Three more page claims had no test. The ReplacingMergeTree advice was the reviewer's third open major on #117740, and the page's 401 troubleshooting entry turned out to describe one server and user, not ClickHouse. Three integration tests prove them. A batch delivered twice into a ReplacingMergeTree reads 4 rows and 2 with FINAL until OPTIMIZE, and an identical block inside non_replicated_deduplication_window is dropped at insert. Three buffered batches flush as one INSERT of three rows, read from system.query_log. A wrong password fails the probe with ClickHouse's "Authentication failed" message. That last claim changed. Self-hosted 26.8 answers the test's named user HTTP 403 and code 516. Cloud 26.6 answers its default user 401 and code 194, measured the same day. The invariant claims only the shared message. Mutations that insert per batch, or make the probe return nil, fail their tests. --- docs/coverage/invariants.yml | 6 +- docs/coverage/matrix.md | 2 +- internal/sinks/clickhouse_invariants_test.go | 91 ++++++++++++++++++++ 3 files changed, 96 insertions(+), 3 deletions(-) diff --git a/docs/coverage/invariants.yml b/docs/coverage/invariants.yml index 56b47369..b1e1ae43 100644 --- a/docs/coverage/invariants.yml +++ b/docs/coverage/invariants.yml @@ -514,8 +514,10 @@ invariants: class: safety applies_to: sink.clickhouse claim: > - Wrong credentials fail at startup with HTTP 401, before any - message is consumed. + Wrong credentials fail the startup probe, before any message is + consumed, with ClickHouse's "Authentication failed: password is + incorrect, or there is no user with such name". The HTTP status and + code vary by server and user. verified_by: harness requires: [integration] diff --git a/docs/coverage/matrix.md b/docs/coverage/matrix.md index 49ebe1fd..ab9f69e7 100644 --- a/docs/coverage/matrix.md +++ b/docs/coverage/matrix.md @@ -238,7 +238,7 @@ drains. An invariant holds only if it holds on all four. | `clickhouse.flush.one_insert` | One flush is one INSERT, on one connection. | ❌ missing | | `clickhouse.dsn.scheme` | The DSN scheme decides the protocol, TLS and the default port. `@` and `:` in a password parse raw. `/`, `#` and `?` must be percent- encoded. | ❌ missing | | `clickhouse.tls.verifies` | With TLS, the server certificate is verified against the system roots. No option skips verification. | ❌ missing | -| `clickhouse.auth.fails_start` | Wrong credentials fail at startup with HTTP 401, before any message is consumed. | ❌ missing | +| `clickhouse.auth.fails_start` | Wrong credentials fail the startup probe, before any message is consumed, with ClickHouse's "Authentication failed: password is incorrect, or there is no user with such name". The HTTP status and code vary by server and user. | ❌ missing | | `clickhouse.type_text.per_destination` | Text into a non-String column must already be in that column's text form. The type table proves each destination with its own value. | ❌ missing | ## Liveness invariants: lifecycle diff --git a/internal/sinks/clickhouse_invariants_test.go b/internal/sinks/clickhouse_invariants_test.go index 58cac0ef..a6dc4069 100644 --- a/internal/sinks/clickhouse_invariants_test.go +++ b/internal/sinks/clickhouse_invariants_test.go @@ -7,6 +7,7 @@ package sinks import ( "context" "fmt" + "net" "reflect" "strings" "testing" @@ -304,3 +305,93 @@ func TestIntegrationSinkClickhouse_DecimalTakesText(t *testing.T) { assert.Error(t, err) assert.That(t, errs.HasCode(err, errs.CodeSinkEncodeFailed)) } + +func TestIntegrationSinkClickhouse_ReplayDuplicatesUntilMerge(t *testing.T) { + coverage.Invariant(t, "clickhouse.replay.duplicates_until_merge", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + batch := func() arrow.Table { + return textColumns(t, + textCol{name: "id", values: []*string{str("1"), str("2")}}, + textCol{name: "v", values: []*string{str("first"), str("first")}}) + } + count := func(s *ClickhouseSink, suffix string) uint64 { + var n uint64 + assert.NoError(t, s.conn.QueryRow(ctx, "SELECT count() FROM "+s.table+suffix).Scan(&n)) + return n + } + + // A replay into ReplacingMergeTree: both copies until a merge. + rmt := newLiveClickhouseSink(t, `CREATE TABLE %s (id String, v String) ENGINE = ReplacingMergeTree ORDER BY id`) + assert.NoError(t, flushOnce(t, rmt, batch())) + assert.NoError(t, flushOnce(t, rmt, batch())) + assert.Equal(t, uint64(4), count(rmt, "")) + assert.Equal(t, uint64(2), count(rmt, " FINAL")) + assert.NoError(t, rmt.conn.Exec(ctx, "OPTIMIZE TABLE "+rmt.table+" FINAL")) + assert.Equal(t, uint64(2), count(rmt, "")) + + // An identical block inside the deduplication window is dropped at insert. + dedup := newLiveClickhouseSink(t, `CREATE TABLE %s (id String, v String) ENGINE = MergeTree ORDER BY id + SETTINGS non_replicated_deduplication_window = 100`) + assert.NoError(t, flushOnce(t, dedup, batch())) + assert.NoError(t, flushOnce(t, dedup, batch())) + assert.Equal(t, uint64(2), count(dedup, "")) +} + +func TestIntegrationSinkClickhouse_OneFlushIsOneInsert(t *testing.T) { + coverage.Invariant(t, "clickhouse.flush.one_insert", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + s := newLiveClickhouseSink(t, `CREATE TABLE %s (a String) ENGINE = MergeTree ORDER BY tuple()`) + for _, v := range []string{"x", "y", "z"} { + assert.NoError(t, s.WriteTable(ctx, textColumns(t, textCol{name: "a", values: []*string{str(v)}}))) + } + assert.NoError(t, s.Flush(ctx)) + + assert.NoError(t, s.conn.Exec(ctx, "SYSTEM FLUSH LOGS")) + var inserts, rows uint64 + assert.NoError(t, s.conn.QueryRow(ctx, ` + SELECT count(), sum(written_rows) FROM system.query_log + WHERE type = 'QueryFinish' AND query_kind = 'Insert' + AND has(tables, currentDatabase() || '.' || ?)`, s.table).Scan(&inserts, &rows)) + assert.Equal(t, uint64(1), inserts) + assert.Equal(t, uint64(3), rows) +} + +func TestIntegrationSinkClickhouse_WrongPasswordFailsTheProbe(t *testing.T) { + coverage.Invariant(t, "clickhouse.auth.fails_start", "sink.clickhouse") + if testing.Short() { + t.Skip("integration test: -short runs the unit pass only") + } + t.Parallel() + ctx := context.Background() + ch := sharedClickhouse(t) + host, err := ch.Host(ctx) + assert.NoError(t, err) + port, err := ch.MappedPort(ctx, "8123/tcp") + assert.NoError(t, err) + + dsn := fmt.Sprintf("clickhouse://%s:wrong@%s/%s", + clickhouseUser, net.JoinHostPort(host, port.Port()), clickhouseDatabase) + s, err := NewClickhouseSink(config.ClickhouseSink{ + DSN: dsn, + Table: "never_written", + }) + assert.NoError(t, err) + t.Cleanup(func() { s.Close() }) + + // The HTTP status and code vary by server and user: self-hosted 26.8 + // answers this named user 403 and code 516, and Cloud answers its default + // user 401 and code 194. The message is what every one of them shares. + err = s.Probe(ctx) + assert.Error(t, err) + t.Logf("probe error: %v", err) + assert.That(t, strings.Contains(err.Error(), + "Authentication failed: password is incorrect, or there is no user with such name")) +} From 7daf95b46edf73410062f5edf27b690dc1ccf550 Mon Sep 17 00:00:00 2001 From: "turbolytics.io" Date: Sun, 11 Oct 2026 06:21:55 -0400 Subject: [PATCH 10/12] docs: the ClickHouse page lives here, with an anchor per claim The ClickHouse integration page existed only in ClickHouse's repository and a scratch status directory, so nothing in CI could compare it with the engine. Five of its claims went stale between v1.1.0 and v2026.10.08. docs/published/clickhouse.mdx is now the source of truth, with 58 anchor comments citing an invariant, a measurement, a published config, a feature or the generated type table. The page check passes against it, and breaking one anchor and one table row produced three precise problems. docs/published/clickhouse/ holds the measurements and the scripts that reproduce them. CI's paths filter re-includes docs/published, or a page-only edit would skip the check that guards it. Anchoring every claim found three more that the evidence did not support: - Cloud now closes a plain-HTTP connection on 8443, so the error ends in EOF, not "connection reset by peer". Measured in cloud-reset.out. - A wrong password answers 401 and code 194 only for Cloud's default user. Self-hosted 26.8 answers a named user 403 and code 516. The page names both. - The timestamp note cited a Tokyo-zoned server nobody had kept evidence for. The test now proves a DateTime('Asia/Tokyo') column holds the same instant as a UTC one, and the page says exactly that. The timestamp and Decimal tests now also assert the error text the page quotes. --- .github/workflows/ci.yml | 3 + dev/config/examples/published/README.md | 5 + docs/coverage/invariants.yml | 6 +- docs/coverage/matrix.md | 2 +- docs/published/README.md | 36 ++ docs/published/clickhouse.mdx | 548 ++++++++++++++++++ docs/published/clickhouse/.gitignore | 5 + docs/published/clickhouse/README.md | 27 + docs/published/clickhouse/bluesky.out | 12 + docs/published/clickhouse/cloud-auth.out | 14 + docs/published/clickhouse/cloud-dedup.out | 8 + docs/published/clickhouse/cloud-reset.out | 4 + docs/published/clickhouse/cloudverify.out | 8 + docs/published/clickhouse/perf.out | 17 + docs/published/clickhouse/perf.sh | 16 + docs/published/clickhouse/perfcloud.out | 12 + docs/published/clickhouse/perfcloud.sh | 28 + .../clickhouse/selfhosted-verify.out | 13 + .../clickhouse/throughput-bisect.out | 37 ++ docs/published/clickhouse/win.sh | 44 ++ docs/published/clickhouse/win.yml | 40 ++ docs/published/clickhouse/window-replay.out | 19 + internal/sinks/clickhouse_invariants_test.go | 19 +- 23 files changed, 917 insertions(+), 6 deletions(-) create mode 100644 docs/published/README.md create mode 100644 docs/published/clickhouse.mdx create mode 100644 docs/published/clickhouse/.gitignore create mode 100644 docs/published/clickhouse/README.md create mode 100644 docs/published/clickhouse/bluesky.out create mode 100644 docs/published/clickhouse/cloud-auth.out create mode 100644 docs/published/clickhouse/cloud-dedup.out create mode 100644 docs/published/clickhouse/cloud-reset.out create mode 100644 docs/published/clickhouse/cloudverify.out create mode 100644 docs/published/clickhouse/perf.out create mode 100755 docs/published/clickhouse/perf.sh create mode 100644 docs/published/clickhouse/perfcloud.out create mode 100755 docs/published/clickhouse/perfcloud.sh create mode 100644 docs/published/clickhouse/selfhosted-verify.out create mode 100644 docs/published/clickhouse/throughput-bisect.out create mode 100755 docs/published/clickhouse/win.sh create mode 100644 docs/published/clickhouse/win.yml create mode 100644 docs/published/clickhouse/window-replay.out diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c8b332a9..c0192435 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,6 +4,8 @@ on: branches: [main] # A change to docs or Markdown alone cannot break a build, so it skips CI. # docs/coverage still runs it: the Coverage job regenerates the page there. + # docs/published runs it too: the coverage generator holds each published + # page to the invariants it cites, so a page-only edit must reach that check. pull_request: branches: [main] paths: @@ -11,6 +13,7 @@ on: - "!docs/**" - "!**.md" - "docs/coverage/**" + - "docs/published/**" jobs: # The Go engine (sqlflow), unit pass only. Fast feedback with no Docker: the # integration tests start containers, and the job below runs those. -short diff --git a/dev/config/examples/published/README.md b/dev/config/examples/published/README.md index b19104fc..f8752911 100644 --- a/dev/config/examples/published/README.md +++ b/dev/config/examples/published/README.md @@ -19,3 +19,8 @@ generated JSON schema. When a page changes, change the file here first and copy it out. When the engine changes, CI tells you which published page needs an edit before a reader finds out. + +The ClickHouse page itself lives in this repo too, at +[`docs/published/clickhouse.mdx`](../../../../docs/published/clickhouse.mdx). +Its YAML blocks must equal the two `clickhouse.*` files here byte for byte, +and the coverage generator fails when they drift. diff --git a/docs/coverage/invariants.yml b/docs/coverage/invariants.yml index b1e1ae43..40503e3b 100644 --- a/docs/coverage/invariants.yml +++ b/docs/coverage/invariants.yml @@ -452,9 +452,9 @@ invariants: class: safety applies_to: sink.clickhouse claim: > - Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as - UTC, or the same with an offset, which is applied. Any other shape - fails as `user.sink.encode_failed`. + Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as UTC + whatever the column's zone, or the same with an offset, which is + applied. Any other shape fails as `user.sink.encode_failed`. verified_by: harness requires: [integration] diff --git a/docs/coverage/matrix.md b/docs/coverage/matrix.md index ab9f69e7..b09cf8d7 100644 --- a/docs/coverage/matrix.md +++ b/docs/coverage/matrix.md @@ -232,7 +232,7 @@ drains. An invariant holds only if it holds on all four. | `clickhouse.table.not_created` | The sink never creates or alters a table. A table that does not exist fails the batch as `system.sink.write_failed`. | ❌ missing | | `clickhouse.enum.null_is_zero` | A NULL into a non-Nullable Enum is stored as 0. With no 0 member the INSERT succeeds and reads of the column fail with `UNKNOWN_ELEMENT_OF_ENUM`. With a 0 member it reads as that member. Nullable reads NULL. | ❌ missing | | `clickhouse.encode.fails_once` | A value the driver cannot encode fails the batch on the first attempt as `user.sink.encode_failed`. It is not retried, and the process exits with code 10. | ❌ missing | -| `clickhouse.temporal_text.two_shapes` | Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as UTC, or the same with an offset, which is applied. Any other shape fails as `user.sink.encode_failed`. | ❌ missing | +| `clickhouse.temporal_text.two_shapes` | Text into a DateTime column takes `YYYY-MM-DD hh:mm:ss`, read as UTC whatever the column's zone, or the same with an offset, which is applied. Any other shape fails as `user.sink.encode_failed`. | ❌ missing | | `clickhouse.decimal.text_only` | A Decimal column accepts its value as text. A float fails as `user.sink.encode_failed`. | ❌ missing | | `clickhouse.replay.duplicates_until_merge` | A batch delivered twice into a ReplacingMergeTree reads twice until a merge, and once with FINAL. An identical block inside the table's deduplication window is dropped at insert. | ❌ missing | | `clickhouse.flush.one_insert` | One flush is one INSERT, on one connection. | ❌ missing | diff --git a/docs/published/README.md b/docs/published/README.md new file mode 100644 index 00000000..4f3c0843 --- /dev/null +++ b/docs/published/README.md @@ -0,0 +1,36 @@ +# Published pages + +Pages SQLFlow publishes in third-party docs. Each page here is the source of +truth for what a vendor's docs say about SQLFlow. The vendor's repository +gets a copy, by hand, after review. + +A page carries an anchor comment beside each claim. MDX renders none of them: + +| Anchor | What it cites | What the check requires | +| --- | --- | --- | +| `{/* invariant: */}` | a claim in `docs/coverage/invariants.yml` | the id is declared | +| `{/* measured: */}` | a committed output file that names its versions | the file exists | +| `{/* config: */}` | the YAML block that follows | it equals `dev/config/examples/published/` | +| `{/* feature: */}` | a tooling claim, proven by a named test | the id is declared in `features.yml` | +| `{/* generated: type-table */}` | the type table that follows | it equals what `sink.clickhouse.yml` renders | +| `{/* generated: unsupported-types */}` | the list that follows | it equals the rendered unsupported list | + +A page named for a namespace cites every invariant in it: `clickhouse.mdx` +must cite each `clickhouse.*` invariant at least once. + +The coverage generator runs the check before it reads any test report, so +`make coverage-page`, `make coverage-check` and CI's Coverage job all fail on +a page that drifted from its evidence. An invariant whose test fails fails +the Integration job, which names the page claim through the invariant's id. + +## Pages + +| Page | Published in | Evidence | +| --- | --- | --- | +| [`clickhouse.mdx`](clickhouse.mdx) | ClickHouse docs, `docs/integrations/connectors/data-ingestion/community-integrations/sqlflow.mdx` | [`clickhouse/`](clickhouse/README.md) | + +## Changing a page + +1. Change the claim here, and its invariant, test or measurement with it. +2. Run `make coverage-page` and `uv run --locked pytest tests/tooling -q`. +3. After the pull request merges, copy the page to the vendor's repository. diff --git a/docs/published/clickhouse.mdx b/docs/published/clickhouse.mdx new file mode 100644 index 00000000..94bb7cca --- /dev/null +++ b/docs/published/clickhouse.mdx @@ -0,0 +1,548 @@ +--- +sidebarTitle: 'SQLFlow' +keywords: ['clickhouse', 'sqlflow', 'duckdb', 'kafka', 'streaming', 'stream processing', 'sink', 'data ingestion', 'integrate', 'websocket'] +slug: /integrations/sqlflow +description: 'Stream Kafka, WebSocket and webhook data into ClickHouse, transformed with DuckDB SQL, using SQLFlow' +title: 'SQLFlow and ClickHouse' +doc_type: 'guide' +integration: + - support_level: 'community' + - category: 'data_ingestion' + - website: 'https://turbolytics.io' +--- + +import ConnectionDetails from '/snippets/_gather_your_details_http.mdx'; +import { CommunityMaintainedBadge } from "/snippets/components/CommunityMaintainedBadge/CommunityMaintainedBadge.jsx"; + + + +SQLFlow is an open-source (MIT) stream processing engine. It consumes a stream (Kafka, a WebSocket, or webhooks), runs DuckDB SQL over each micro-batch, and inserts the result into a ClickHouse table. A pipeline is a single YAML file, and SQLFlow ships as one binary and one Docker image, so there is no cluster to run. + +This page covers the ClickHouse sink: connecting to ClickHouse Cloud and self-hosted ClickHouse, which types map, an end-to-end example on the NYC taxi sample dataset, and the limits that were measured rather than assumed. + +## Prerequisites {#prerequisites} + +- SQLFlow **[v2026.10.08](https://github.com/turbolytics/sql-flow/releases/tag/v2026.10.08)** or later. Every result on this page was measured on that release unless it names another. +- ClickHouse Cloud, or self-hosted ClickHouse. This page was run against self-hosted 26.8.1 and Cloud 26.6.1. No minimum below those has been established. +- A Kafka broker the pipeline can reach, for the Kafka source used in the example. The WebSocket and webhook sources need no broker. +- Python 3 with the `confluent-kafka` package (`pip install confluent-kafka`), only for the script in the example that publishes the sample data to Kafka. SQLFlow itself needs no Python. +- Docker is the simplest install. `turbolytics/sql-flow` is published multi-arch for `linux/amd64` and `linux/arm64`. SQLFlow can also be built from source; it needs Go, a C toolchain, and `libduckdb` present at runtime. + +## Setup {#setup} + +SQLFlow connects with a single DSN. The scheme selects the protocol and whether TLS is used; the port defaults from the scheme if omitted. + +| Scheme | Protocol | TLS | Default port | Use for | +| ------------------------------- | -------- | --- | ------------ | ------- | +| `clickhouses://`, `https://` | HTTP | yes | 8443 | ClickHouse Cloud, or any TLS-terminated HTTP endpoint | +| `clickhouse://`, `http://` | HTTP | no | 8123 | Self-hosted ClickHouse | +| `natives://` | native | yes | 9440 | ClickHouse Cloud over the native protocol | +| `tcp://`, `native://` | native | no | 9000 | Self-hosted, native protocol | + +{/* invariant: clickhouse.dsn.scheme */} + +The target table must already exist. SQLFlow does not create tables and does not evolve schemas: it names the columns in its `INSERT` from the column names its SQL produces, so **the columns your handler SQL emits must match the table's column names**. Use aliases in the SQL to rename. {/* invariant: clickhouse.table.not_created */} {/* invariant: clickhouse.insert.by_name */} + +### Connect to ClickHouse Cloud {#connect-to-clickhouse-cloud} + + + +With those details, the DSN is: + +``` +clickhouses://default:@.clickhouse.cloud:8443/ +``` + +Keep it out of the YAML by passing it as an environment variable; the config's `dsn` field is a template: + +```yaml +sink: + type: clickhouse + clickhouse: + dsn: {{ SQLFLOW_CLICKHOUSE_DSN }} + table: trips_small +``` + +```bash +export SQLFLOW_CLICKHOUSE_DSN='clickhouses://default:@.clickhouse.cloud:8443/nyc_taxi' +``` + + +Use `clickhouses://` (with the `s`) for Cloud. `clickhouse://` sends plain HTTP to port 8443, the server closes the connection, and startup fails as `system.sink.unreachable` before any message is consumed. See [Troubleshooting](#troubleshooting). {/* measured: docs/published/clickhouse/cloud-reset.out */} + + +### Connect to self-hosted ClickHouse {#connect-to-self-hosted-clickhouse} + +Point the DSN at the HTTP interface, port 8123 by default: + +```bash +export SQLFLOW_CLICKHOUSE_DSN='clickhouse://default:@clickhouse.internal:8123/nyc_taxi' +``` + +If you have put TLS in front of the HTTP interface, use `clickhouses://` exactly as for Cloud. For the native protocol use `tcp://` (9000) or `natives://` (9440). + +When SQLFlow runs in Docker, `localhost` inside the container is the container itself, not your machine. Put SQLFlow and ClickHouse on the same Docker network and use the ClickHouse container's name as the host. + +## Authentication {#authentication} + +Username and password are carried in the DSN and, with `clickhouses://` or `natives://`, are sent over TLS. The server certificate is verified against the system trust store; there is no option to skip verification. {/* invariant: clickhouse.tls.verifies */} + +- **Special characters in the password.** The DSN is parsed as a URL, so `/`, `#` and `?` must be percent-encoded (`%2F`, `%23`, `%3F`); left raw they produce a confusing `invalid port ":p" after host`. `@` and `:` are handled correctly either way. {/* invariant: clickhouse.dsn.scheme */} +- **Cloud IP access lists.** If your service restricts source IPs, the address SQLFlow connects from has to be on the list. +- **Not supported:** mutual TLS / client certificates, and custom CA bundles. Use username and password. + +## Example: streaming the NYC taxi dataset from Kafka {#example-nyc-taxi} + +This walks the [NYC taxi sample dataset](/get-started/sample-datasets/nyc-taxi) through Kafka into ClickHouse, transformed by SQLFlow on the way. It reproduces on Cloud or self-hosted; only the DSN changes. + +### 1. Create the target table {#example-create-table} + +The same table the sample-dataset page defines: + +```sql +CREATE DATABASE IF NOT EXISTS nyc_taxi; + +CREATE TABLE IF NOT EXISTS nyc_taxi.trips_small ( + trip_id UInt32, + pickup_datetime DateTime, + dropoff_datetime DateTime, + pickup_longitude Nullable(Float64), + pickup_latitude Nullable(Float64), + dropoff_longitude Nullable(Float64), + dropoff_latitude Nullable(Float64), + passenger_count UInt8, + trip_distance Float32, + fare_amount Float32, + extra Float32, + tip_amount Float32, + tolls_amount Float32, + total_amount Float32, + payment_type Enum('CSH' = 1, 'CRE' = 2, 'NOC' = 3, 'DIS' = 4, 'UNK' = 5), + pickup_ntaname LowCardinality(String), + dropoff_ntaname LowCardinality(String) +) +ENGINE = MergeTree +PRIMARY KEY (pickup_datetime, dropoff_datetime); +``` + +### 2. Publish the data to Kafka {#example-publish} + +The dataset is published as gzipped TSV. This step converts one file (about a million rows) to newline-delimited JSON and publishes it to a topic named `nyc-taxi-trips`. + +Install the publisher's dependency and download the file: + +```bash +pip install confluent-kafka +curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/trips_0.gz +``` + +Create the topic. A stock Apache Kafka broker creates it on first publish, but managed and production clusters commonly set `auto.create.topics.enable=false`, where an uncreated topic fails the example before any row reaches ClickHouse: + +```bash +kafka-topics.sh --bootstrap-server localhost:9092 \ + --create --topic nyc-taxi-trips --partitions 1 --replication-factor 1 +``` + +Confluent's images name that command `kafka-topics`, without the `.sh`. + +Save the publisher as `publish.py`. Change `bootstrap.servers` if the broker is not on `localhost:9092`: + +```python +import gzip, json +from confluent_kafka import Producer + +COLUMNS = ["trip_id", "pickup_datetime", "dropoff_datetime", "pickup_longitude", + "pickup_latitude", "dropoff_longitude", "dropoff_latitude", "passenger_count", + "trip_distance", "fare_amount", "extra", "tip_amount", "tolls_amount", + "total_amount", "payment_type", "pickup_ntaname", "dropoff_ntaname"] +INTS = {"trip_id", "passenger_count"} +FLOATS = {"pickup_longitude", "pickup_latitude", "dropoff_longitude", "dropoff_latitude", + "trip_distance", "fare_amount", "extra", "tip_amount", "tolls_amount", "total_amount"} + +producer = Producer({"bootstrap.servers": "localhost:9092"}) + +def send(topic, value): + # The producer's queue holds 100,000 messages by default, and this file + # has a million rows; poll() drains it as it fills. + while True: + try: + producer.produce(topic, value) + return + except BufferError: + producer.poll(0.5) + +with gzip.open("trips_0.gz", "rb") as f: + header = f.readline().decode().rstrip("\n").split("\t") + index = {name: i for i, name in enumerate(header)} + for raw in f: + # Split on the raw bytes and decode only the columns being published. + # The file carries columns this example does not use that are not + # valid UTF-8, and decoding the whole line would have to corrupt them. + parts = raw.rstrip(b"\n").split(b"\t") + row = {} + for col in COLUMNS: + value = parts[index[col]].decode() + if value == "": + row[col] = None + elif col in INTS: + row[col] = int(float(value)) + elif col in FLOATS: + row[col] = float(value) + else: + row[col] = value + send("nyc-taxi-trips", json.dumps(row)) +producer.flush() +``` + +Run it. Publishing the million rows takes under a minute: {/* measured: docs/published/clickhouse/selfhosted-verify.out */} + +```bash +python publish.py +``` + +### 3. Write the pipeline {#example-pipeline} + +Save this as `taxi.yml`. The handler SQL runs in DuckDB over each batch of JSON messages; the casts produce Arrow types that map onto the table's ClickHouse types (see [Type mapping](#type-mapping)). + +{/* config: clickhouse.taxi.yml */} + +```yaml +pipeline: + name: nyc-taxi-clickhouse + batch_size: 20000 + + source: + type: kafka + kafka: + brokers: [{{ SQLFLOW_KAFKA_BROKERS|default('localhost:9092') }}] + group_id: nyc-taxi + auto_offset_reset: earliest + topics: + - nyc-taxi-trips + + handler: + type: 'handlers.InferredMemBatch' + sql: | + SELECT + CAST(trip_id AS UINTEGER) AS trip_id, + CAST(pickup_datetime AS TIMESTAMP) AS pickup_datetime, + CAST(dropoff_datetime AS TIMESTAMP) AS dropoff_datetime, + pickup_longitude, + pickup_latitude, + dropoff_longitude, + dropoff_latitude, + CAST(passenger_count AS UTINYINT) AS passenger_count, + CAST(trip_distance AS FLOAT) AS trip_distance, + CAST(fare_amount AS FLOAT) AS fare_amount, + CAST(extra AS FLOAT) AS extra, + CAST(tip_amount AS FLOAT) AS tip_amount, + CAST(tolls_amount AS FLOAT) AS tolls_amount, + CAST(total_amount AS FLOAT) AS total_amount, + payment_type, + pickup_ntaname, + dropoff_ntaname + FROM batch + + sink: + type: clickhouse + clickhouse: + dsn: {{ SQLFLOW_CLICKHOUSE_DSN }} + table: trips_small +``` + +`batch_size` is the number of messages per DuckDB batch and per ClickHouse `INSERT`. See [Performance](#performance) for why 20,000. + +### 4. Run it {#example-run} + +```bash +docker run --rm \ + --network \ + -v "$(pwd)":/conf \ + -e SQLFLOW_CLICKHOUSE_DSN \ + -e SQLFLOW_KAFKA_BROKERS= \ + turbolytics/sql-flow:v2026.10.08 \ + run /conf/taxi.yml --max-msgs=1000000 +``` + + +**The broker address must be one the container can reach.** A Kafka client is redirected to whatever the broker *advertises*, not the address you dial. A broker advertising `127.0.0.1:9092` for external clients sends the container to itself, and every fetch fails with `dial tcp 127.0.0.1:9092: connect: connection refused`, even though the same address works from your shell. Put the container on the broker's own network and use the address advertised there, or run the binary on the host instead: + +```bash +sqlflow run taxi.yml --max-msgs=1000000 +``` + + +`--max-msgs` stops after that many messages; leave it off to run continuously. Kafka offsets are committed only after each `INSERT` has been acknowledged by ClickHouse, and only up to the last message actually processed. {/* invariant: pipeline.commit.after_flush */} + +### 5. Verify {#example-verify} + +```sql +SELECT payment_type, count() AS trips, round(avg(total_amount), 2) AS avg_total +FROM nyc_taxi.trips_small +GROUP BY payment_type +ORDER BY trips DESC; +``` + +``` +β”Œβ”€payment_type─┬──trips─┬─avg_total─┐ +β”‚ CSH β”‚ 617237 β”‚ 17.82 β”‚ +β”‚ CRE β”‚ 377897 β”‚ 13.91 β”‚ +β”‚ NOC β”‚ 3573 β”‚ 13.89 β”‚ +β”‚ DIS β”‚ 1293 β”‚ 12.09 β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +The same query gives the same counts against Cloud and self-hosted, which is a reasonable check that nothing was dropped or coerced on either path. {/* measured: docs/published/clickhouse/selfhosted-verify.out */} {/* measured: docs/published/clickhouse/cloudverify.out */} On Cloud, a read served by a replica that has not yet fetched the newest part can undercount for a moment after the run ends. Add `SETTINGS select_sequential_consistency = 1` to the query if the total comes up short. {/* measured: docs/published/clickhouse/perfcloud.out */} + +### A second example: WebSocket source, no Kafka {#example-websocket} + +The WebSocket source needs no broker, and this shows an `Array(String)` column being written and queried natively. It consumes the public Bluesky firehose. + +```sql +CREATE TABLE bluesky_posts ( + ts DateTime, + did String, + langs Array(String), + text String +) ENGINE = MergeTree ORDER BY ts; +``` + +{/* config: clickhouse.bluesky.yml */} + +```yaml +pipeline: + batch_size: 200 + source: + type: websocket + websocket: + uri: 'wss://jetstream2.us-east.bsky.network/subscribe?wantedCollections=app.bsky.feed.post' + handler: + type: 'handlers.InferredMemBatch' + sql: | + SELECT + to_timestamp(time_us / 1000000) AS ts, + did, + COALESCE(commit.record.langs, []) AS langs, + COALESCE(commit.record.text, '') AS text + FROM batch + sink: + type: clickhouse + clickhouse: + dsn: {{ SQLFLOW_CLICKHOUSE_DSN }} + table: bluesky_posts +``` + +Save it as `bluesky.yml` and run it. No `SQLFLOW_KAFKA_BROKERS` here: the source is the WebSocket, and the only variable the pipeline reads is the DSN. + +```bash +docker run --rm \ + -v "$(pwd)":/conf \ + -e SQLFLOW_CLICKHOUSE_DSN \ + turbolytics/sql-flow:v2026.10.08 \ + run /conf/bluesky.yml --max-msgs=2000 +``` + +Add `--network` as in the Kafka example if ClickHouse runs in Docker on the same machine. With the binary instead of the image: + +```bash +sqlflow run bluesky.yml --max-msgs=2000 +``` + +The firehose delivers about 50 posts a second, so 2,000 messages take under a minute. {/* measured: docs/published/clickhouse/bluesky.out */} Then: + +```sql +SELECT lang, count() AS posts +FROM (SELECT arrayJoin(langs) AS lang FROM bluesky_posts) +GROUP BY lang ORDER BY posts DESC LIMIT 5; +``` + +## Type mapping {#type-mapping} + +SQLFlow hands each batch to ClickHouse as rows built from the Arrow table DuckDB produces, so what matters is the Arrow type each SQL column ends up with. Cast in the handler SQL to control it. + +Every row below is generated from the engine's type matrix and proven by an integration test that writes the value, reads it back through the server, and compares. The matrix and the test live in the [SQLFlow repository](https://github.com/turbolytics/sql-flow/blob/main/docs/coverage/clickhouse-types.mdx). A pairing in the table is proven. A pairing missing from it is untested, not refused: the refused ones are listed under [Unsupported](#unsupported) and fail the batch with a code. Parameterised column types are listed as the instance the test wrote. + +{/* generated: type-table */} + +| DuckDB SQL type (cast to) | ClickHouse column types that accept it | Notes | +| --- | --- | --- | +| `BOOLEAN` | `Bool`, `UInt8` | | +| `TINYINT` | `Int8` | | +| `SMALLINT` | `Int16` | | +| `INTEGER` | `Int32` | | +| `BIGINT` | `Int64` | | +| `UTINYINT` | `UInt8` | | +| `USMALLINT` | `UInt16` | | +| `UINTEGER` | `UInt32` | | +| `UBIGINT` | `UInt64` | | +| `REAL` | `Float32` | | +| `DOUBLE` | `Float64` | | +| `VARCHAR` | `String`, `LowCardinality(String)`, `DateTime64(3)`, `DateTime`, `Enum8('' = 0, 'a' = 1, 'b' = 2)`, `FixedString(36)`, `UUID`, `IPv4`, `Decimal(10, 2)` | the text must already be the column's text form. `UUID` and `JSON` reach the sink as `VARCHAR`. | +| `BLOB`, `BIT`, `VARINT` | `String` | | +| `DATE` | `Date`, `Date32` | | +| `TIMESTAMP_S` | `DateTime` | | +| `TIMESTAMP_MS` | `DateTime64(3)` | | +| `TIMESTAMP` | `DateTime64(6)`, `DateTime` | DateTime keeps whole seconds; DateTime64(6) keeps all six digits | +| `TIMESTAMP_NS` | `DateTime64(9)` | | +| `TIMESTAMPTZ` | `DateTime64(6)`, `DateTime` | stored as the same UTC instant; the zone is dropped and the column's own zone governs rendering; DateTime keeps whole seconds | +| `BOOLEAN[]` | `Array(Bool)` | | +| `TINYINT[]` | `Array(Int8)` | | +| `SMALLINT[]` | `Array(Int16)` | | +| `INTEGER[]` | `Array(Int32)` | | +| `BIGINT[]` | `Array(Int64)` | | +| `UTINYINT[]` | `Array(UInt8)` | | +| `USMALLINT[]` | `Array(UInt16)` | | +| `UINTEGER[]` | `Array(UInt32)` | | +| `UBIGINT[]` | `Array(UInt64)` | | +| `REAL[]` | `Array(Float32)` | | +| `DOUBLE[]` | `Array(Float64)` | | +| `VARCHAR[]` | `Array(String)` | | +| `BLOB[]` | `Array(String)` | | +| `DATE[]` | `Array(Date32)` | | +| `TIMESTAMP_S[]` | `Array(DateTime)` | | +| `TIMESTAMP_MS[]` | `Array(DateTime64(3))` | | +| `TIMESTAMP[]` | `Array(DateTime64(6))` | | +| `TIMESTAMP_NS[]` | `Array(DateTime64(9))` | | +| `TIMESTAMPTZ[]` | `Array(DateTime64(6))` | stored as the same UTC instant; the zone is dropped and the column's own zone governs rendering | +| `BIGINT[][]` | `Array(Array(Int64))` | | +| `NULL` in any column | the column's own type | the column type's zero value; the DEFAULT expression is not evaluated because every produced column is named in the INSERT. For an Enum with no 0 member that value is not a member, and the row cannot be read | +| `NULL` inside a `LIST` | the element's own type | the element type's zero value, because ClickHouse's Array(T) holds no nulls unless T is Nullable | + +{/* invariant: type.roundtrip */} {/* invariant: type.null */} {/* invariant: type.nested */} {/* invariant: type.string.fidelity */} {/* invariant: clickhouse.type_text.per_destination */} + + +**A `VARCHAR` reaches a non-`String` column as text the column parses.** The matrix proves each destination with a value already in that destination's text form, not with one value for all of them. A UUID string does not become a `DateTime`. The forms tested: + +| Destination | Text the test wrote | +| --- | --- | +| `String`, `LowCardinality(String)` | `L'Ε’il πŸ‘ "quoted" back\slash` and a tab | +| `DateTime64(3)` | `2026-09-01 12:00:00.123` | +| `DateTime` | `2026-09-01 12:00:00` | +| `Enum8(...)` | a member name, such as `a` | +| `FixedString(36)`, `UUID` | `0b7e2c9a-4d31-4f6e-9a02-7c1de3f10b45` | +| `IPv4` | `192.168.1.1` | +| `Decimal(10, 2)` | `1234.56` | + +{/* invariant: clickhouse.type_text.per_destination */} + + + +**A `NULL` into a non-`Nullable` `Enum` with no `0` member writes a row that cannot be read.** The sink stores the zero value, `0`. The `INSERT` succeeds, and every later `SELECT` of the column fails with `UNKNOWN_ELEMENT_OF_ENUM`. Measured with `Enum8('a' = 1, 'b' = 2)`. An enum column that can receive a `NULL` needs a `0` member, as in `Enum8('' = 0, 'a' = 1, 'b' = 2)`, or `Nullable(Enum8(...))`. Otherwise `coalesce` the value in the handler SQL. {/* invariant: clickhouse.enum.null_is_zero */} + + +Values are matched to columns **by name**, not by position. {/* invariant: clickhouse.insert.by_name */} + + +**Prefer `CAST(ts AS TIMESTAMP)` over passing timestamps as strings.** When SQLFlow sends a `VARCHAR` to a `DateTime` column, the Go driver parses the string, not the server. It accepts exactly two shapes: + +- `'2026-09-01 12:00:00'`, which it reads as UTC. The column's timezone does not change the instant stored: the sink parses the text and sends the instant, so a `DateTime('Asia/Tokyo')` column holds the same instant as a `DateTime('UTC')` one. +- `'2026-09-01 12:00:00 +09:00'`, whose offset is applied. That row stores `2026-09-01 03:00:00 UTC`. + +Anything else fails the batch, including ISO 8601 with a `T` separator or a `Z` suffix. `'2026-09-01T12:00:00Z'` is rejected with `parsing time "2026-09-01T12:00:00Z" as "2006-01-02 15:04:05"`. {/* invariant: clickhouse.temporal_text.two_shapes */} {/* invariant: type.timestamp.instant */} + +This differs from writing the same literal in an `INSERT` statement yourself, where ClickHouse parses the string in the target column's timezone, and a plain `DateTime` column uses the server timezone. Casting in the handler SQL sidesteps both sets of rules. + +If the field can be `NULL`, a `DateTime` column stores it as `1970-01-01 00:00:00`. Use `Nullable(DateTime)`, or `coalesce(CAST(ts AS TIMESTAMP), now()::TIMESTAMP)` in the handler SQL. {/* invariant: type.null */} + + + +`Decimal` columns accept the value as a **string**, not as a `DOUBLE`. `CAST(amount AS VARCHAR)` works; `CAST(amount AS DOUBLE)` fails the batch with `converting float64 to Decimal(10, 2) is unsupported`. {/* invariant: clickhouse.decimal.text_only */} + + +### Unsupported {#unsupported} + +These fail the batch with `user.sink.type_unsupported` rather than reaching the table. Cast or flatten them in the handler SQL. + +{/* generated: unsupported-types */} + +- `DECIMAL(p, s)`, `HUGEINT`, `UHUGEINT` +- `TIME`, `TIMETZ` +- `TIME_NS` +- `INTERVAL` +- `DECIMAL(p, s)[]`, `HUGEINT[]` +- `TIME[]` +- `STRUCT(...)[]` +- `INTEGER[3]` +- `STRUCT(...)` +- `MAP(K, V)` +- `UNION(...)` +- `ENUM` + +{/* invariant: type.undeclared.fails_loud */} + +## Known limits {#known-limits} + +- **Unsupported types:** listed under [Unsupported](#unsupported). `DECIMAL`, `STRUCT`, `MAP`, `TIME` and `INTERVAL` are the ones a handler reaches for. Each fails the batch with `user.sink.type_unsupported` before any row is written. Flatten a `STRUCT` or `MAP` into scalar columns in the handler SQL, or serialise it with `to_json()` into a `String` column. Cast a `DECIMAL` to `VARCHAR`. `Nested` columns were not tested. {/* invariant: type.undeclared.fails_loud */} +- **No table creation or schema evolution.** The table must exist, and a column the handler emits that the table lacks fails the batch (see [Troubleshooting](#troubleshooting)). Columns the handler omits take the table's `DEFAULT`. {/* invariant: clickhouse.table.not_created */} {/* invariant: clickhouse.insert.by_name */} +- **A `NULL` written to a non-`Nullable` column is stored as the type's zero value**, not as the column's `DEFAULT` expression, and no error is raised. See [Type mapping](#type-mapping) for the workaround. For an `Enum` with no `0` member that value is not a member, and the row cannot be read back. See the warning under [Type mapping](#type-mapping). {/* invariant: type.null */} {/* invariant: clickhouse.enum.null_is_zero */} +- **At-least-once delivery.** Kafka offsets are committed after the `INSERT` is acknowledged. A crash between the two replays the batch, so duplicates are possible. {/* invariant: pipeline.commit.after_flush */} + + `ReplacingMergeTree` keyed on something that identifies the row removes them **eventually**, on a background merge. Until that merge, an ordinary read sees both copies: measured on self-hosted 26.8.1, a replayed block read 4 rows, and 2 with `FINAL`. Query with `FINAL` where a read must be exact now. {/* invariant: clickhouse.replay.duplicates_until_merge */} + + Insert-time block deduplication drops a second copy only when the block is byte-identical. A replicated table, which every Cloud table is, does that by default, and a non-replicated one does it with `non_replicated_deduplication_window`. A replay cuts its batches again from the committed offset, and a batch the flush interval cut short does not come back with the same boundaries. Treat insert-time deduplication as a partial measure, not a guarantee. {/* invariant: clickhouse.replay.duplicates_until_merge */} {/* measured: docs/published/clickhouse/cloud-dedup.out */} +- **Transient failures are retried.** A refused, reset, or timed-out connection to ClickHouse is retried with exponential backoff under a deadline, configured by the sink's `retry` block (`max_attempts`, `initial_backoff_ms`, `max_backoff_ms`, `deadline_seconds`). A rejected `INSERT` is not retried once the server has seen it. A value the driver itself cannot encode, such as the ISO 8601 timestamp above, is not retried: it fails at once as `user.sink.encode_failed`, and the process exits with code 10. The sink also dials ClickHouse once at startup, so an unreachable host fails before any message is consumed. {/* invariant: clickhouse.encode.fails_once */} {/* invariant: sink.probe.fails_start */} +- **One connection, one `INSERT` per batch.** There is no parallel writer. Throughput is governed by `batch_size`; see below. {/* invariant: clickhouse.flush.one_insert */} +- **Streaming aggregation** (tumbling windows) is supported. A window closes on an event-time watermark the engine asserts, never on wall clock. A record's event time is the payload field the source's `event_time` names, or the Kafka timestamp when it names none. The handler cuts the bucket as `time_bucket(INTERVAL '', event_time)`. A replay running behind real time therefore publishes each bucket once, whole: 1,000 events in one past one-minute bucket, read in ten batches of 100, published one row of 1,000. {/* invariant: pipeline.window.counts_every_row */} {/* measured: docs/published/clickhouse/window-replay.out */} Set `allowed_lateness_seconds` above zero and a late row republishes its bucket as a whole value, so the target must then replace the row by key, with a `ReplacingMergeTree` read through `FINAL`. `sqlflow validate` warns about this for a ClickHouse window sink. {/* feature: config.validation */} See [Tumbling windows](https://github.com/turbolytics/sql-flow#tumbling-windows) in the README. + +## Performance {#performance} + +Measured with the example above: one million 17-column rows, one SQLFlow process, Kafka co-located with SQLFlow, two runs per row. + +| Target | Rows/second | +| ------------------------------------------ | ----------- | +| Self-hosted ClickHouse, SQLFlow and Kafka on the same Docker network, v2026.10.08 | 57,900 and 55,900 | +| ClickHouse Cloud (us-east-2), SQLFlow beside the broker, over the public internet, v2026.10.08 | 13,700 and 13,300 | + +{/* measured: docs/published/clickhouse/perf.out */} {/* measured: docs/published/clickhouse/perfcloud.out */} + +The Cloud figure is the network between the machine and the service, not SQLFlow. Over the same link in the same hour, the previous release, v2026.09.17.1, ran 12,500 and 12,900 rows a second {/* measured: docs/published/clickhouse/perfcloud.out */}, and the same pipeline against a self-hosted server on one Docker network runs four times as fast. Run SQLFlow near the service, same region and ideally same VPC, for numbers closer to the self-hosted figure. + +`batch_size` is the lever on a nearby target, measured on the same Docker network: + +| `batch_size` | Rows/second | +| ------------ | ----------- | +| 5,000 | 34,000 | +| 20,000 | 56,900 | +| 50,000 | 57,600 | +| 100,000 | 56,900 | + +{/* measured: docs/published/clickhouse/perf.out */} + +Below about 20,000 rows per batch, per-`INSERT` overhead dominates; above it there is little further to gain, and memory per batch grows. 20,000 is a good default for rows of this width; verify against your own. + +## Troubleshooting {#troubleshooting} + +### `Post "http://…:8443?…": EOF` at startup {#trouble-reset} + +The DSN uses `clickhouse://` (plain HTTP) against a TLS port, and the server closes the connection. Startup fails as `system.sink.unreachable`: `failed to query server hello: … Post "http://…:8443?…": EOF`. Change the scheme to `clickhouses://`. {/* measured: docs/published/clickhouse/cloud-reset.out */} + +### `Authentication failed: password is incorrect, or there is no user with such name` {#trouble-auth} + +The credentials in the DSN are wrong, or a special character in the password was not URL-encoded. Startup fails before any message is consumed. The HTTP status and code depend on the server and the user: Cloud answers its `default` user with `[HTTP 401] code: 194`, and self-hosted 26.8 answers a named user with `[HTTP 403] code: 516`. On Cloud, the `default` user's password is shown once at service creation and can be reset from the console. If the password is right, check the service's IP access list. {/* invariant: clickhouse.auth.fails_start */} {/* measured: docs/published/clickhouse/cloud-auth.out */} + +### `column is not present in the table ` {#trouble-column} + +The full error starts with `[system.sink.write_failed] clickhouse sink: prepare batch:`. The handler SQL produced a column the target table does not have. SQLFlow inserts by column name, so either alias the column in the SQL to match the table, drop it from the `SELECT`, or add it to the table. {/* invariant: clickhouse.insert.by_name */} + +### `clickhouse sink: column "": unsupported arrow type ` {#trouble-type} + +The column's Arrow type has no ClickHouse mapping, such as a struct or a map value. Cast or flatten in the handler SQL; see [Type mapping](#type-mapping). {/* invariant: type.undeclared.fails_loud */} + +### `converting float64 to Decimal(10, 2) is unsupported` {#trouble-decimal} + +A `DOUBLE` was sent to a `Decimal` column. Cast to a string instead: `CAST(amount AS VARCHAR) AS amount`. The driver parses the decimal text exactly, with no floating-point rounding. {/* invariant: clickhouse.decimal.text_only */} + +### `parsing time "…" as "2006-01-02 15:04:05"` {#trouble-timestamp} + +A `VARCHAR` timestamp reached a `DateTime` column in a shape the driver does not accept. It takes `'2006-01-02 15:04:05'` and the same with a trailing offset, and nothing else. ISO 8601 with a `T` separator or a `Z` suffix fails. Cast in the handler SQL with `CAST(ts AS TIMESTAMP)`, or reformat the string. See [Type mapping](#type-mapping). + +The error carries the code `user.sink.encode_failed`, and the process exits with code 10. It is not retried, because the value would fail every attempt. {/* invariant: clickhouse.temporal_text.two_shapes */} {/* invariant: clickhouse.encode.fails_once */} + +### The pipeline runs but rows never appear {#trouble-no-rows} + +Check the batching. A batch is executed and inserted when `batch_size` messages have arrived, or when `flush_interval_seconds` elapses with a partial batch waiting. Whichever comes first wins. `flush_interval_seconds` defaults to `30`, so a partial batch on a low-volume topic waits up to 30 seconds. For faster inserts on a quiet topic, lower `flush_interval_seconds`, which takes any whole number of seconds, or lower `batch_size`. With `--max-msgs`, the final partial batch is inserted on exit. {/* invariant: pipeline.flush.eventually */} {/* invariant: lifecycle.drain.on_cancel */} + +## Further reading {#further-reading} + +- SQLFlow on GitHub: configuration reference, all sources and sinks, error policies, windowing. +- SQLFlow documentation: tutorials, including a longer ClickHouse walkthrough. diff --git a/docs/published/clickhouse/.gitignore b/docs/published/clickhouse/.gitignore new file mode 100644 index 00000000..94a123a8 --- /dev/null +++ b/docs/published/clickhouse/.gitignore @@ -0,0 +1,5 @@ +# What the scripts here write while they run. The .out files are the evidence. +*.log +taxi_*.yml +win_events.jsonl +win_late.yml diff --git a/docs/published/clickhouse/README.md b/docs/published/clickhouse/README.md new file mode 100644 index 00000000..e16306ca --- /dev/null +++ b/docs/published/clickhouse/README.md @@ -0,0 +1,27 @@ +# Evidence for the ClickHouse page + +The measurements [`../clickhouse.mdx`](../clickhouse.mdx) cites. Its other +claims cite `clickhouse.*` invariants, proven by +`internal/sinks/clickhouse_invariants_test.go`. + +| File | What it shows | Produced by | +| --- | --- | --- | +| `selfhosted-verify.out` | The taxi example's Verify counts on self-hosted 26.8.1, and the publish time | the page's steps, on 2026-10-09 | +| `cloudverify.out` | The same counts on Cloud 26.6.1 | the page's steps, then the Verify query | +| `bluesky.out` | The WebSocket example landing 2,000 posts in 40 seconds | the page's steps, on 2026-10-09 | +| `perf.out` | Self-hosted throughput at each `batch_size` | `perf.sh` | +| `perfcloud.out` | Cloud throughput, the previous release as a control, and one count read short from a lagging replica | `perfcloud.sh` | +| `throughput-bisect.out` | Why v2026.10.08 is slower than v2026.09.17.1: the Kafka fetch defaults and jsonparser 1.3.0 | host builds of each commit | +| `window-replay.out` | 1,000 events in one past bucket, read in ten batches, published as one row | `win.sh` with `win.yml` | +| `cloud-dedup.out` | Cloud dropping an identical second block at insert | the queries in its header | +| `cloud-reset.out` | `clickhouse://` against Cloud's TLS port failing at startup | the page's run step with a plain-HTTP DSN | +| `cloud-auth.out` | Cloud's answer to a wrong password: HTTP 401, code 194 | the query in its header | + +The scripts expect the dev stack (`make start-backing-services`): Kafka as +`kafka1` and ClickHouse as `clickhouse` on the `dev_default` network. Set +`IMG` to run another image. `perfcloud.sh` reads Cloud credentials from +`~/.sqlflow-clickhouse` or the file `CLICKHOUSE_CREDS` names, and keeps the +password out of the argument list. + +A measurement names the versions it ran on. When a release changes one of +these numbers, re-run the script, replace the file, and update the page. diff --git a/docs/published/clickhouse/bluesky.out b/docs/published/clickhouse/bluesky.out new file mode 100644 index 00000000..81fea348 --- /dev/null +++ b/docs/published/clickhouse/bluesky.out @@ -0,0 +1,12 @@ +# The Bluesky WebSocket example, run as the page prints it, against self-hosted ClickHouse 26.8.1.2041. +# turbolytics/sql-flow:v2026.10.08, --max-msgs=2000, 2026-10-09. Recorded in +# docs/superpowers/status/clickhouse-117740/STATUS.md on sql-flow PR #329. +exit=0 wall=40s errors=0 +count() 2000 +β”Œβ”€lang─┬─posts─┐ +β”‚ en β”‚ 1070 β”‚ +β”‚ ja β”‚ 236 β”‚ +β”‚ es β”‚ 63 β”‚ +β”‚ de β”‚ 53 β”‚ +β”‚ pt β”‚ 50 β”‚ +β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”˜ diff --git a/docs/published/clickhouse/cloud-auth.out b/docs/published/clickhouse/cloud-auth.out new file mode 100644 index 00000000..c03ab9f2 --- /dev/null +++ b/docs/published/clickhouse/cloud-auth.out @@ -0,0 +1,14 @@ +# A wrong password against ClickHouse Cloud, 2026-10-11, user default, password 'wrong' +# server version: 26.6.1.2326 +Code: 194. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name + +If you use ClickHouse Cloud, the password can be reset at https://clickhouse.cloud/ +on the settings page for the corresponding service. + +If you have installed ClickHouse and forgot password you can reset it in the configuration file. +The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml +and deleting this file will reset the password. +See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed. + +. (REQUIRED_PASSWORD) +HTTP 401 diff --git a/docs/published/clickhouse/cloud-dedup.out b/docs/published/clickhouse/cloud-dedup.out new file mode 100644 index 00000000..9680cbec --- /dev/null +++ b/docs/published/clickhouse/cloud-dedup.out @@ -0,0 +1,8 @@ +# The same block inserted twice into a plain MergeTree on ClickHouse Cloud, 2026-10-11 +# server version: 26.6.1.2326 +# CREATE TABLE default.dedup_probe_1791713956 (id Int32, v String) ENGINE = MergeTree ORDER BY id +engine: SharedMergeTree +replicated_deduplication_window: 10000 +insert 1 done +insert 2 done +count(): 2 diff --git a/docs/published/clickhouse/cloud-reset.out b/docs/published/clickhouse/cloud-reset.out new file mode 100644 index 00000000..ac904f81 --- /dev/null +++ b/docs/published/clickhouse/cloud-reset.out @@ -0,0 +1,4 @@ +# A plain-HTTP clickhouse:// DSN against ClickHouse Cloud's TLS port 8443, 2026-10-11 +# turbolytics/sql-flow:v2026.10.08, Cloud 26.6.1.2326. Host and password redacted. +exit=12 +Error: [system.sink.unreachable] sink: the destination could not be reached at startup: failed to query server hello: failed to query server hello info: sendQuery: Post "http://default:***@:8443?client_protocol_version=54460&database=default&default_format=Native&max_execution_time=34": EOF diff --git a/docs/published/clickhouse/cloudverify.out b/docs/published/clickhouse/cloudverify.out new file mode 100644 index 00000000..95e07359 --- /dev/null +++ b/docs/published/clickhouse/cloudverify.out @@ -0,0 +1,8 @@ +# Verify query on Cloud 26.6.1.2326 + β”Œβ”€payment_type─┬──trips─┬─avg_total─┐ +1. β”‚ CSH β”‚ 617237 β”‚ 17.82 β”‚ +2. β”‚ CRE β”‚ 377897 β”‚ 13.91 β”‚ +3. β”‚ NOC β”‚ 3573 β”‚ 13.89 β”‚ +4. β”‚ DIS β”‚ 1293 β”‚ 12.09 β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +fffd: 0 diff --git a/docs/published/clickhouse/perf.out b/docs/published/clickhouse/perf.out new file mode 100644 index 00000000..fd03ee8f --- /dev/null +++ b/docs/published/clickhouse/perf.out @@ -0,0 +1,17 @@ +# v2026.10.08 unless a comment names another image; perf.sh