diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1edde77..06ebd9c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -225,6 +225,22 @@ jobs: - name: rust — suite with the registry; the golden test must run run: scripts/check-suite.sh --require-artifacts rust + # The examples are advertised in README.md as four runnable tours, and + # until now nothing ran them: a signature change in any binding would + # break its tour with every gate green, and examples/rust/Cargo.lock sat + # at 0.1.0 through the whole 0.1.1 release because nothing resolved it. + # This is the only job with an artifact already in hand and all four + # toolchains already set up, so the marginal cost is one step. + # + # BOTH FLAGS ARE LOAD-BEARING. --require-all: chplay SKIPS a missing + # toolchain, so without it a run where all four skipped exits 0 and + # proves nothing. --locked: an unlocked resolve silently REWRITES a + # drifted lockfile and exits 0 — and on a runner that rewrite is thrown + # away, so the drift would pass forever. Verified by planting the 0.1.0 + # drift back into the lock: --locked exits 1, unlocked exits 0. + - name: examples — the four tours actually run, on their committed lockfiles + run: examples/chplay.sh --require-all --locked + # ── Lint / supply-chain jobs below ───────────────────────────────── # # All additive: none of the jobs above gained a `needs:` on any of these, diff --git a/CLAUDE.md b/CLAUDE.md index 8ea7ed9..f604661 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,17 +1,17 @@ # chtypes (SDK repository) — orientation for Claude sessions -This is the **SDK half** of chtypes, Apache 2.0, public: `go/ python/ ts/ rust/` over the frozen `chs_*` C ABI (`include/chtypes.h`, ABI revision 4, 28 functions), the normative `docs/reference/`, four side-by-side `examples/` tours, and `goldens/` — the public golden set every binding runs. The bindings contain no ClickHouse code; they `dlopen` per-version artifacts and speak the ABI. +This is the **SDK half** of chtypes, Apache 2.0, public: `go/ python/ ts/ rust/` over the frozen `chs_*` C ABI (`include/chtypes.h`, ABI revision 4, 28 functions), the normative `docs/reference/`, and four side-by-side `examples/` tours, which CI runs. The bindings contain no ClickHouse code; they `dlopen` per-version artifacts and speak the ABI. -The other half is the sibling core repository (wrapper, build, artifacts, rigs, corpus, runs of record; license pending) — private, and the housing folder's own `CLAUDE.md` is where its checkout location is recorded. Its `CLAUDE.md` carries the rules that were paid for; the ones that bind here: +The other half is the sibling core repository — private: the wrapper, the build, the artifacts, and the rigs and corpus that score every SDK. The artifacts it publishes are under the Elastic License 2.0 and this repository is Apache 2.0; `NOTICE` is the statement of that split. The rules from that half which bind here: - **Never rebuild a ClickHouse rule in an SDK.** A binding is a thin passthrough to the artifact; scalar, comparison, coercion and timestamp logic never live in Go/Python/TS/Rust. The one derived result is `Transformed`, per spec. - **The four bindings give one answer.** A behavior change lands in all four in one cycle (`docs/reference/bindings.md` is the shape), and the golden set must stay green in all four. The golden set is **served, not tracked**: core publishes `sdk-goldens.json` in the rolling release as a row in the signed `SHA256SUMS`, `scripts/fetch.sh` installs it at `/sdk-goldens.json`, and each binding's golden test reads it offline from there (`CHTYPES_GOLDENS` overrides). There is no cases file in this repository. A case runs only against the EXACT ClickHouse version `generated.exact` names for its line and skips loudly otherwise, and the set shrinks as lines are added, because a case that stops being version-agnostic is dropped rather than recorded twice. -- **The header is owned here.** An ABI change: `include/chtypes.h` + the four bindings' pinned constants (`ABIRevision` / `ABI_REVISION`) in one commit; the core repository then pulls the header (`ci/steps/header-sync.sh --pull`) and relinks every artifact. CI asserts the header and the bindings agree. +- **The header is owned here.** An ABI change: `include/chtypes.h` + the four bindings' pinned constants (`ABIRevision` / `ABI_REVISION`) in one commit; the core repository then pulls the header and relinks every artifact. CI asserts the header and the bindings agree. - **Never trust exit codes or self-reports.** Every test here skips LOUDLY without a registry and refuses a zero-run; `scripts/check-standalone.sh` reads its verdict off a `go test -json` census and `scripts/check-suite.sh` off each runner's own summary line, color stripped. - **The Go package is dlopen-only by default.** The linked path (package-level `CompileDDL`, `BuiltVersion`) is behind `-tags chtypes_linked` and needs a core build tree via `CGO_LDFLAGS`; `undefined: chtypes.CompileDDL` means the tag is missing. `go/chtypes/linked_abi_check.go` pins the hardcoded ABI numbers to the header at compile time (tagged build only). - **Artifacts live in the per-user cache**, `~/.cache/chtypes/artifacts/-/` (`$CHTYPES_REGISTRY` overrides): `scripts/fetch.sh` installs there, a core build lands there, every SDK's registry default resolves there. - macOS artifacts are a dev floor, not an oracle (float parses diverge). Float expectations come from Linux or a live server. -Gates: `scripts/check-standalone.sh` (Go, from a bare copy) and `scripts/check-suite.sh python|ts|rust`; per language `go test ./...`, `uv run pytest -q`, `pnpm test`, `cargo test`. With a registry they run everything; without one every artifact test skips by name and the rest still runs, and a suite that ran nothing fails. `.github/workflows/ci.yml` runs on hosted runners with no variable and no secret: once with no artifact (`--no-artifacts`), once with two published lines fetched by `scripts/fetch.sh` (`--require-artifacts`: the goldens must run). The server-truth suites for these SDKs live in the core repository and run from there against this tree as a sibling; the artifact-backed proof is that repository's `sdk-suites` workflow (named `certify` until 2026-09-11). +Gates: `scripts/check-standalone.sh` (Go, from a bare copy) and `scripts/check-suite.sh python|ts|rust`; per language `go test ./...`, `uv run pytest -q`, `pnpm test`, `cargo test`. With a registry they run everything; without one every artifact test skips by name and the rest still runs, and a suite that ran nothing fails. `.github/workflows/ci.yml` runs on hosted runners with no variable and no secret: once with no artifact (`--no-artifacts`), once with two published lines fetched by `scripts/fetch.sh` (`--require-artifacts`: the goldens must run). The server-truth suites for these SDKs live in the core repository and run from there against this tree as a sibling. The `artifacts` job also runs all four `examples/` tours (`examples/chplay.sh --require-all --locked`), so "runnable" is measured rather than claimed. Pre-1.0, and published: all four bindings are live — `github.com/wave-rf/chtypes/go` (lowercase, the Go norm), PyPI `chtypes`, npm `@wavehouse/chtypes`, crates.io `chtypes`. The module path and the function signatures froze at the first tag (`v0.1.0`). **Do not write the current version into this file or any other prose** — it rots within one release; the README's badges read it live from each registry, and each binding's CHANGELOG is the record. Nothing publishes without a `/v*` tag, and each release workflow refuses a tag whose version differs from the manifest's; the registry setup and the order are in `RELEASING.md` (the one manual first publish per registry is done). diff --git a/RELEASING.md b/RELEASING.md index 0fdda0e..3bc8d4f 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -20,12 +20,27 @@ It fetches anonymously on purpose. A registry can show a maintainer a version th ## Before the first tag of each package 1. Bump the manifest version (`python/pyproject.toml`, `ts/package.json`, `rust/Cargo.toml`; Go has none — the tag is the version). -2. `CHANGELOG` entry naming the ABI revision the release speaks (`4` today) and the ClickHouse lines the golden set was generated on. -3. Run that package's suite against a registry, and `scripts/check-standalone.sh` for Go. -4. Tag: `git tag go/v0.1.0 && git push origin go/v0.1.0`, etc. +2. **Regenerate the example lockfiles**, which record the bindings as path dependencies and therefore carry the version: `(cd examples/rust && cargo update -p chtypes)` and `(cd examples/python && uv lock)`. CI runs the four tours with `--locked`, so a stale one fails the `artifacts` job. This is deliberate: `examples/rust/Cargo.lock` sat at `0.1.0` through the whole `0.1.1` release, and `examples/python/uv.lock` missed `0.1.1` _and_ `0.1.2`, because nothing resolved them. +3. `CHANGELOG` entry naming the ABI revision the release speaks (`4` today) and the ClickHouse lines the golden set was generated on. +4. Run that package's suite against a registry, and `scripts/check-standalone.sh` for Go. +5. Tag: `git tag go/v0.1.0 && git push origin go/v0.1.0`, etc. The first tag freezes the `chs_*` signatures and the Go module path. +## Every release after the first: the order, and why Go is last + +Push the four tags **one at a time**, and wait for each workflow to go green before pushing the next: + +> **`rust` → `ts` → `python` → `go`** + +**Rust first** because it has the most failure surface — a `--locked` build, a `--locked` publish and an OIDC exchange — so a problem stops the sequence while nothing is public yet. + +⚠️ **Go last, and this is the one that matters.** A Go module publishes by **the tag existing on a public repository**. `release-go.yml` only _verifies_; it does not publish, and nothing gates it. The instant `git push origin go/v0.1.2` lands, proxy.golang.org can serve it, and it cannot be withdrawn. Every other registry has a workflow between the tag and the public artifact. Go has none, so it goes last, when the other three have already proved the release is good. + +None of the four can be taken back: crates.io and PyPI refuse to reuse a version number, and npm the same. A release is a one-way door on all four — the order only decides how much you know before you walk through the last one. + +⚠️ **Do not read `release-ts` or `release-rust` failing on an already-published version as broken publishing.** `error: crate chtypes@X already exists` and `[E403] You cannot publish over the previously published versions` are both reached _after_ authentication succeeds, so they are the check that the publishers are still configured. Read the error before reporting a problem. + ## Registry setup, once each (the owner's console; nothing is stored here) The repository has the deployment environments `pypi`, `npm` and `crates-io` — one per publishing workflow, named to match what each registry's trusted publisher is configured with — and no registry token exists anywhere: every publish authenticates by OIDC, with the one exception per registry noted below. Do these only once the repository is public — provenance links point at the source, and a package on a public registry whose source 404s is worse than no package. diff --git a/examples/chplay.sh b/examples/chplay.sh index e95a4db..5ab4534 100755 --- a/examples/chplay.sh +++ b/examples/chplay.sh @@ -5,6 +5,8 @@ # ./chplay.sh go run one (any of: go python ts rust) # ./chplay.sh go rust run a subset # ./chplay.sh --list show what would run, and with which toolchain +# ./chplay.sh --require-all a missing toolchain is a FAILURE, not a skip +# ./chplay.sh --locked every tour honors its committed lockfile # # Every tour is OFFLINE: it needs only that language's toolchain plus the # artifacts in the registry (scripts/fetch.sh, or a core-repository build). @@ -13,6 +15,16 @@ # A missing toolchain is a SKIP with instructions, never a failure. A tour # that crashes is a failure and makes this script exit nonzero. # +# THE TWO CI FLAGS, and why they exist. Interactively a skip is the right +# answer: you should not need four toolchains to see one tour. In CI it is +# the wrong answer twice over — a run where all four silently skipped exits +# 0 and proves nothing, and a tour resolved against a freshly-solved +# dependency set cannot catch a committed lockfile that has drifted. That +# is what let examples/rust/Cargo.lock sit at 0.1.0 through the whole 0.1.1 +# release with every gate green. --require-all turns a skip into a failure; +# --locked makes each tour use its lockfile as committed and fail if it no +# longer resolves. CI passes both; neither changes what a tour prints. +# # The one thing here that DOES want a server — go/ingest-demo — is not run by # this script at all; see go/ingest-demo/README.md (the optional online demo). set -u -o pipefail @@ -25,11 +37,15 @@ _arch="$(uname -m)"; case "$_arch" in x86_64|amd64) _arch=amd64 ;; arm64|aarch64 REGISTRY="${CHTYPES_REGISTRY:-${XDG_CACHE_HOME:-$HOME/.cache}/chtypes/artifacts/$(uname -s | tr '[:upper:]' '[:lower:]')-$_arch}" ALL_LANGS=(go python ts rust) +# Both default off: the interactive run is the forgiving one. +REQUIRE_ALL=0 +LOCKED=0 + bold() { printf '\033[1m%s\033[0m' "$1"; } say() { printf '%s\n' "$*"; } usage() { - sed -n '2,17p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + sed -n '2,29p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' exit 0 } @@ -84,7 +100,10 @@ run_go() { have_python() { command -v uv >/dev/null 2>&1 && return 0 SKIP_REASON="uv not found — brew install uv (or https://docs.astral.sh/uv); or skip it: ./chplay.sh go ts rust"; return 1; } -run_python() { (cd "$HERE/python" && uv run demo.py); } +run_python() { + if [ "$LOCKED" -eq 1 ]; then (cd "$HERE/python" && uv run --locked demo.py) + else (cd "$HERE/python" && uv run demo.py); fi +} have_ts() { command -v node >/dev/null 2>&1 || { SKIP_REASON="node not found — brew install node; or skip it: ./chplay.sh go python rust"; return 1; } @@ -92,14 +111,21 @@ have_ts() { return 0 } run_ts() { - (cd "$HERE/ts" && - { [ -d node_modules ] || pnpm install --silent; } && - node demo.mjs) + if [ "$LOCKED" -eq 1 ]; then + (cd "$HERE/ts" && pnpm install --frozen-lockfile --silent && node demo.mjs) + else + (cd "$HERE/ts" && + { [ -d node_modules ] || pnpm install --silent; } && + node demo.mjs) + fi } have_rust() { command -v cargo >/dev/null 2>&1 && return 0 SKIP_REASON="rust toolchain not found — install rustup (https://rustup.rs); or skip it: ./chplay.sh go python ts"; return 1; } -run_rust() { (cd "$HERE/rust" && cargo run --quiet); } +run_rust() { + if [ "$LOCKED" -eq 1 ]; then (cd "$HERE/rust" && cargo run --quiet --locked) + else (cd "$HERE/rust" && cargo run --quiet); fi +} # ------------------------------------------------------------------ driving @@ -107,6 +133,8 @@ langs=() for arg in "$@"; do case "$arg" in -h | --help) usage ;; + --require-all) REQUIRE_ALL=1 ;; + --locked) LOCKED=1 ;; --list) require_artifacts say "artifacts: $(artifact_count) version(s) under $REGISTRY" @@ -127,6 +155,8 @@ done require_artifacts say "chplay: $(artifact_count) artifact version(s) under $REGISTRY" +[ "$REQUIRE_ALL" -eq 1 ] && say "chplay: --require-all — a missing toolchain fails this run" +[ "$LOCKED" -eq 1 ] && say "chplay: --locked — every tour must resolve its committed lockfile" [ -n "${CHTYPES_VERSION:-}" ] && say "chplay: CHTYPES_VERSION=$CHTYPES_VERSION (tours will select it)" logdir="$(mktemp -d "${TMPDIR:-/tmp}/chplay.XXXXXX")" @@ -138,9 +168,18 @@ failed=0 for l in "${langs[@]}"; do SKIP_REASON="" if ! "have_$l"; then - say "" - say "$(bold "-- $l: SKIPPED") — $SKIP_REASON" - results+=("$l: skipped — $SKIP_REASON") + if [ "$REQUIRE_ALL" -eq 1 ]; then + # A gate that skipped everything exits 0 and proves nothing. + say "" + say "$(bold "-- $l: FAILED") — toolchain missing, and --require-all forbids a skip" + say " $SKIP_REASON" + results+=("$l: FAILED — toolchain missing under --require-all") + failed=1 + else + say "" + say "$(bold "-- $l: SKIPPED") — $SKIP_REASON" + results+=("$l: skipped — $SKIP_REASON") + fi continue fi say "" diff --git a/examples/go/README.md b/examples/go/README.md index 208c321..1916786 100644 --- a/examples/go/README.md +++ b/examples/go/README.md @@ -5,15 +5,16 @@ Sixteen sections over the whole chtypes surface, matching `../python`, `../ts` a ## Run it ```bash -go run -tags chtypes_linked . # or, with prerequisite checks: ../chplay.sh go -# (the tag adds the statically linked section — §static artifact — which -# needs a core build tree via CGO_LDFLAGS; the registry sections need only artifacts) +go run . # 16/16 sections, dlopen-only — what a public clone runs +../chplay.sh go # the same tour with prerequisite checks; this is what CI runs ``` +`chplay.sh` is the tested path: CI runs `chplay.sh --require-all --locked`, so the command above is verified on every pull request rather than merely written down. + ## Prerequisites - **Artifacts.** The tour reads `$CHTYPES_REGISTRY`, defaulting to the per-user cache (`~/.cache/chtypes/artifacts/-/`) or wherever `$CHTYPES_REGISTRY` points. No artifacts? `../../scripts/fetch.sh 25.8`. One version is enough; section 12's cross-version sweeps want several and degrade gracefully without them. -- **cgo**. Sections 9 and 14 (the statically linked shape) additionally need the core repository's `lib/build` on `CGO_LDFLAGS` and the `chtypes_linked` tag; `chplay.sh go` sets both when the sibling is present, and without the tag those two sections say so and skip. They use the static path it provides; everything else goes through the registry. +- **cgo — optional, and not available to a public clone.** Sections 9 and 14 (the statically linked shape) need a core build tree on `CGO_LDFLAGS` plus the `chtypes_linked` tag. That tree is not public: adding the tag without one fails at link time with `library 'chtypes' not found`, which is why it is not in the command above. Without the tag those two sections say so and skip, and the other fourteen run against the registry. `chplay.sh go` adds the tag only when `CHTYPES_CORE_DIR` or `CHTYPES_LIB_BUILD` points at such a tree. - Go 1.27+. ## Knobs diff --git a/examples/python/uv.lock b/examples/python/uv.lock index dde7961..c68cc2a 100644 --- a/examples/python/uv.lock +++ b/examples/python/uv.lock @@ -4,7 +4,7 @@ requires-python = ">=3.11" [[package]] name = "chtypes" -version = "0.1.0" +version = "0.1.2" source = { editable = "../../python" } [package.metadata] diff --git a/ts/CHANGELOG.md b/ts/CHANGELOG.md index 7a5aaee..1036eec 100644 --- a/ts/CHANGELOG.md +++ b/ts/CHANGELOG.md @@ -8,7 +8,7 @@ The four bindings in this repository are released together and give one answer, ### Changed — BREAKING -- **The transform reason for a materialized DEFAULT is now `default_materialized`** — previously the same word spelled with an `s` — and the exported constant naming it is now `DEFAULT_MATERIALIZED`. ClickHouse's own keyword is `MATERIALIZED`; its parser rejects the `s` spelling outright with a syntax error, so the reason naming that concept now matches the system it describes. Code comparing against the old string or the old constant name must be updated. +- **The transform reason for a materialized DEFAULT is now `default_materialized`** — previously the same word spelled with an `s` — and the exported constant naming it is now `Reason.DefaultMaterialized`. ClickHouse's own keyword is `MATERIALIZED`; its parser rejects the `s` spelling outright with a syntax error, so the reason naming that concept now matches the system it describes. Code comparing against the old string or the old constant name must be updated. - This is an SDK-only change and **the ABI is untouched: it remains revision 4, and no artifact needs relinking.** The reason is derived in the binding, not received from the artifact — the library emits `default_substituted`, which each binding translates. That wire value is unchanged. ### Changed