Repository navigation
invariants: the ClickHouse page's claims, each with a test, and a check that holds the page to them - #454
Merged
Conversation
The ClickHouse page's behavioral claims had no CI evidence, and five went stale between v1.1.0 and v2026.10.08 unnoticed. This spec makes each claim a clickhouse.* invariant with a test, moves the page into the repo with an anchor per claim, and adds a check that fails when the two disagree.
Implements docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md: the generator extension, the page check, the type table, twelve invariants with their tests, the page move and the coverage regeneration.
A claim only one integration makes, such as the ClickHouse sink storing a NULL Enum as 0, had nowhere to live in invariants.yml: applies_to took a kind, so every sink would have owed it a cell. The alternative was a parallel registry duplicating the harness, the generator and the status files. applies_to now takes a kind or an integration id. targets() resolves it in one place for the cell builder, the snapshot and the page, so an integration-scoped invariant gets one cell and a marker from any other integration is reported unknown. A test-only double's marker stays known and credits nothing, as before. The registry check accepts an integration id, still rejects a typo, and rejects an exemption from another integration's invariant. FAMILIES gains clickhouse. Six new tests in tests/tooling cover the scoped cell, the unknown marker, covered-with-no-gap, and the three registry cases. If targets() is wrong, a kind invariant loses or gains cells and the committed status files go stale, which coverage-check reports.
The ClickHouse page's claims lived only in the page. Five went stale between v1.1.0 and v2026.10.08 because nothing compared them with the engine. published.py reads every page under docs/published and checks the anchor comment beside each claim. An invariant or feature anchor must name a declared id, a measured anchor a committed file, a config anchor's block must equal the published config, and a generated anchor's rows must equal what the type declaration renders. A page named for a namespace must cite every invariant in it. The generator runs the check before reading any report, so coverage-check and --page both fail on a drifted page. Eleven tests in tests/tooling cover each rule, anchor spacing, a page with no namespace, and the committed pages. If the check is wrong, a page claim drifts silently, which is the failure it replaces, or a correct page fails CI with a problem that names the anchor.
The generated row "VARCHAR, UUID, JSON" against nine column types read as a cross-product. The type test proves each destination with its own value, so a UUID string never reparses as a DateTime or a Decimal. The ClickHouse reviewer flagged it as a major on #117740. The NULL row said every NULL becomes the type's zero value, which hides that an Enum with no 0 member then holds a row no SELECT can read. A carrier whose columns carry their own values now renders its first cast alone, with a note that the other casts reach the sink as it. The utf8 declaration gains the text-form rule, and the NULL rule names the Enum case. clickhouse-types.mdx is regenerated. Two tests in tests/tooling cover the carrier row and a row with no per-column value, which keeps every cast. If the rendering is wrong, the published page check fails, because the page's table must equal this one.
The ClickHouse page states twelve behaviors only the ClickHouse sink has, from the DSN schemes to how a replay reads in ReplacingMergeTree. None had a declared claim or a test, so each could change with no CI signal. invariants.yml declares all twelve under a ClickHouse section, scoped to sink.clickhouse, each requiring the level its evidence runs at. The header and the registry test now allow a scoped invariant to require its level from the start, since it has one subject. Two unit tests prove the DSN scheme table, raw and percent-encoded passwords, and that TLS verifies against the system roots with no option to skip. The two tests ran green, and a mutation that moved the TLS port and set InsecureSkipVerify failed both. The other ten invariants show as missing until their tests land; coverage-check reports them as gaps meanwhile.
… stores a NULL Enum as 0 Three page claims had only a hand-run script behind them: columns match by name and an omitted one takes the table's DEFAULT, the sink never creates a table, and a NULL into an Enum with no 0 member writes a row no SELECT can read. Three integration tests prove them against the shared ClickHouse 26.8 container, and the type test now marks the per-destination text claim it already proves. All four pass. Replacing the INSERT's column list with a positional INSERT fails the by-name test, so it binds the sink's behavior. If a claim changes, its test fails, and the page sentence citing it is the one to fix.
…nd Decimal only in their forms The page's retry bullet was stale for a release because nothing tied it to #269: an unencodable value now fails once as user.sink.encode_failed and exits 10, where the page said it ran the retry ladder. The text-timestamp shapes and the Decimal-from-text rule had the same gap. Three integration tests prove them through the retry wrapper and against the shared ClickHouse container: no retry and exit code 10 for an ISO 8601 string into DateTime, the two accepted timestamp shapes with the offset applied and everything else refused, and Decimal from text with a float refused. All pass. Making every error retryable fails the no-retry test.
…and wrong credentials fail the probe Three more page claims had no test. The ReplacingMergeTree advice was the reviewer's third open major on #117740, and the page's 401 troubleshooting entry turned out to describe one server and user, not ClickHouse. Three integration tests prove them. A batch delivered twice into a ReplacingMergeTree reads 4 rows and 2 with FINAL until OPTIMIZE, and an identical block inside non_replicated_deduplication_window is dropped at insert. Three buffered batches flush as one INSERT of three rows, read from system.query_log. A wrong password fails the probe with ClickHouse's "Authentication failed" message. That last claim changed. Self-hosted 26.8 answers the test's named user HTTP 403 and code 516. Cloud 26.6 answers its default user 401 and code 194, measured the same day. The invariant claims only the shared message. Mutations that insert per batch, or make the probe return nil, fail their tests.
The ClickHouse integration page existed only in ClickHouse's repository and
a scratch status directory, so nothing in CI could compare it with the
engine. Five of its claims went stale between v1.1.0 and v2026.10.08.
docs/published/clickhouse.mdx is now the source of truth, with 58 anchor
comments citing an invariant, a measurement, a published config, a
feature or the generated type table. The page check passes against it, and
breaking one anchor and one table row produced three precise problems.
docs/published/clickhouse/ holds the measurements and the scripts that
reproduce them. CI's paths filter re-includes docs/published, or a
page-only edit would skip the check that guards it.
Anchoring every claim found three more that the evidence did not support:
- Cloud now closes a plain-HTTP connection on 8443, so the error ends in
EOF, not "connection reset by peer". Measured in cloud-reset.out.
- A wrong password answers 401 and code 194 only for Cloud's default user.
Self-hosted 26.8 answers a named user 403 and code 516. The page names
both.
- The timestamp note cited a Tokyo-zoned server nobody had kept evidence
for. The test now proves a DateTime('Asia/Tokyo') column holds the same
instant as a UTC one, and the page says exactly that.
The timestamp and Decimal tests now also assert the error text the page
quotes.
…have evidence The branch review found claims on the ClickHouse page that the page check passed without real evidence: - The retry sentence (a failed connection is retried, a refused INSERT is not) had no anchor, and sink.error.classifies, the invariant that states it, read missing for ClickHouse. - "The process exits with code 10" was proven only as an error-to-exit mapping, not through the pipeline as the spec asked. - The flush interval's default and floor cited invariants that state neither. - "On one connection" was in a claim no test proved, and two page sentences had no anchor. - The replay test could flake if a background merge landed between its two inserts. TestIntegrationSinkClickhouse_ErrorsClassifyForTheRetryLadder proves sink.error.classifies for ClickHouse: a refused connection runs the ladder to its limit and ends unreachable, while a refused INSERT and an encode failure are not retried. TestCore_ASinkEncodeFailureEndsTheRunWithExitCode10 runs the pipeline and holds the exit. The flush interval cites cli.invocation, whose TestCliInvocation_FlushIntervalNeverZero asserts the default of 30 and any positive value. The replay test stops merges until it has counted. The one-connection clause is gone, and the two sentences are anchored. Making no error retryable fails the classification test, and wrapping a flush error as unreachable fails the exit-code test.
The status files and the matrix page carried no evidence for the twelve clickhouse.* invariants, so the page showed ten gaps. They are regenerated from a full local unit and integration pass at this branch's head, with growth and release marked not run: every clickhouse.* cell is covered at its required level, and sink.error.classifies is now covered for ClickHouse. CI's run on main rewrites the growth and release columns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The ClickHouse integration page's claims now have evidence in CI. A claim
that drifts from the engine fails a test, and a page that drifts from its
evidence fails the coverage check.
Before this, only the type table and the two worked configs had CI
evidence. Every behavioral claim had a hand-run script behind it, and five
went stale between v1.1.0 and v2026.10.08 with no signal: the retry
contract, the flush interval floor, window closing, a throughput row and
the version pins.
clickhouse.*invariants indocs/coverage/invariants.yml,scoped to
sink.clickhouse. They run through the same harness markers,generator and status files as every other invariant. There is no parallel
registry.
applies_toaccepts an integration id as well as a kind, resolved inone place,
targets(). A kind's cells are unchanged.internal/sinks/clickhouse_invariants_test.go, and one pipeline test ininternal/core. The ClickHouse type test now also marks theper-destination text claim, and ClickHouse now proves the kind invariant
sink.error.classifies, which the page's retry sentence cites.docs/published/clickhouse.mdx, with an anchorcomment beside each claim. Each one cites an invariant, a committed
measurement, a published config, a feature, or the generated type table.
scripts/coverage_matrix/published.pyfails on anunknown id, an uncited
clickhouse.*invariant, a missing measurement, aconfig block that differs from its file, or a type table that differs
from the declaration.
VARCHARalone.UUIDandJSONarrive as it, and the NULL row names the Enum case. The ClickHouse
reviewer raised both as majors on Docs: Add sqlflow integration page (data ingestion, community) ClickHouse/ClickHouse#117740.
docs/published/**, so a page-only editstill reaches the check.
Spec:
docs/superpowers/specs/2026-10-10-clickhouse-invariants-design.md.Plan:
docs/superpowers/plans/2026-10-10-clickhouse-invariants.md.What it found
Anchoring every claim against fresh evidence corrected three more:
so the error ends in
EOF, notconnection reset by peer.for its
defaultuser. Self-hosted 26.8 answers a named user with 403and code 516. The invariant claims the message both share.
evidence. The test now proves a
DateTime('Asia/Tokyo')column holds thesame instant as a UTC one.
Verification
uv run --locked pytest tests/tooling -q: all pass, including the pagecheck against the committed page.
go test -short -race ./...: pass.retryable, no error retryable, one INSERT per batch, a probe that returns
nil, a flush error rewrapped as unreachable, and a moved TLS port with
InsecureSkipVerify.
problems.
make coverage-checkpasses with 0 gaps. Proven cells rose from 97 to110.
6107a13: the retry sentence, the exit code through the pipeline, the
flush interval citation, a merge-race flake in the replay test, and two
unanchored sentences.
What breaks if this is wrong: a page claim drifts from the engine with no
CI signal, which is the failure this replaces.
Deferred minors, for a later change:
sink.clickhouse.applies_tonaming a test-only integration is accepted, and gets nocell.
The hand-run scripts on #329 are superseded. #329 can close once this
merges. The page still reaches ClickHouse's repository by hand, after
review.