diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc06659..8b65074 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ chtypes' SDKs are thin, honest bindings over one frozen C ABI (`include/chtypes. - **Build and test locally.** `scripts/fetch.sh ` installs an artifact for this machine into the per-user cache; each binding's README says how to run its suite against it. Without one, every test that needs an artifact skips by name and the rest still runs — `scripts/check-suite.sh --no-artifacts ` and `scripts/check-standalone.sh --no-artifacts` are exactly what CI runs first; it then runs the same suites with two published lines. The artifact-backed proof beyond that lives in the core repository. - **Every binding follows the same contract** (`docs/reference/`). A change to what a call means belongs in the spec and in all four bindings, not one. -- **Goldens are served, not tracked.** The golden set is published in the rolling release as `sdk-goldens.json` and installed beside the artifacts by `scripts/fetch.sh`; there is no cases file in this repository to edit. Open an issue here for a missing case; it is routed to whoever owns the generator. +- **Goldens are served, not tracked.** The golden set is published in the rolling release as `sdk-goldens.json` and installed beside the artifacts by `scripts/fetch.sh`; there is no cases file in this repository to edit. Open an issue here for a missing case. - **Report security issues privately**: see `SECURITY.md`. By contributing you agree your work is licensed under the Apache License 2.0 (`LICENSE`). @@ -17,6 +17,8 @@ Each binding has its own linter, configured to be strict but green on the curren It runs two independent checks, because one is not enough: `misspell -locale US`, and a grep backstop for a family of words misspell's matcher has been proven not to fire on reliably even when they are in its own dictionary. The word list lives in the script and nowhere else, so this page cannot drift from it. Run `scripts/lint-prose.sh --selftest` for the proof: it builds a probe file, shows misspell missing most of it, and shows the backstop catching all of it. -Markdown formatting and structure (`dprint check`, `.markdownlint.json`) run in CI's `prose` job but do not block merges yet, for the same reason: this tree's markdown was written hard-wrapped and most of it hasn't been through a `dprint fmt` pass. Dependency vulnerability scanning (`govulncheck`, `pip-audit`, `pnpm audit`, `cargo audit`) runs in CI's `security` job and also does not block — a fresh advisory against a pinned dependency with no fix available yet should not stall every unrelated PR. Both jobs still report their findings on every run. +Markdown formatting and structure (`dprint check`, `.markdownlint.json`) run in CI's `prose` job and **do block merges** — `prose` is a required status check on `main`. It was report-only while the tree was still hard-wrapped; `dprint fmt` has since run over every markdown file, so the reason for the exemption is gone. `pnpm dlx dprint@0.57.4 fmt` fixes the formatting half automatically. Dependency vulnerability scanning (`govulncheck`, `pip-audit`, `pnpm audit`, `cargo audit`) runs in CI's `security` job and also does not block — a fresh advisory against a pinned dependency with no fix available yet should not stall every unrelated PR. Both jobs still report their findings on every run. + +This repository is public, and two more jobs block on that being true. `scripts/lint-public.sh` rejects any pointer to the private core repository **by name**; `scripts/lint-paths.sh` rejects any citation of a repository path that is not here. Both are in the `public` job, and the fix for either is the same: say what a thing **is** — "the C ABI contract", with `include/chtypes.h` as its public authority — rather than where it lives. Both take `--selftest`, which proves the rules fire before you trust a clean run. Run any of these locally with the same command CI uses; each job's step name in `.github/workflows/ci.yml` names the exact invocation. diff --git a/docs/index.md b/docs/index.md index 3da0b1d..959892a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -75,9 +75,10 @@ chtypes derives that report and it is not optional: every accepted row carries a ### Reference -| | | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -| [`reference/go.md`](reference/go.md) · [`reference/python.md`](reference/python.md) · [`reference/ts.md`](reference/ts.md) · [`reference/rust.md`](reference/rust.md) | every public symbol, the C entry point under it, and what it returns or raises | -| [`reference/bindings.md`](reference/bindings.md) | the normative shape every binding implements — read this when porting, or when two bindings seem to disagree | +| | | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| [`reference/go.md`](reference/go.md) · [`reference/python.md`](reference/python.md) · [`reference/ts.md`](reference/ts.md) · [`reference/rust.md`](reference/rust.md) | every public symbol, the C entry point under it, and what it returns or raises | +| [`reference/bindings.md`](reference/bindings.md) | the normative shape every binding implements — read this when porting, or when two bindings seem to disagree | +| [`reference/artifact.md`](reference/artifact.md) | the artifact and registry contract: file names, `manifest.json`, platform keys, and how a loader verifies one | The two normative pages are the deep layer. You should not need either to use chtypes; they are where a disagreement is settled, and where the answer is when a per-language reference page says "see the ABI contract". diff --git a/docs/install.md b/docs/install.md index 3416b2b..64d8788 100644 --- a/docs/install.md +++ b/docs/install.md @@ -41,6 +41,8 @@ npm install @wavehouse/chtypes ESM only. Native calls go through `ffi-rs`, prebuilt for darwin arm64/x64 and linux arm64/x64 (gnu and musl), so there is no build step. +⚠️ That is the FFI loader's matrix, not the artifact's. chtypes artifacts are published for **darwin-arm64, linux-amd64 and linux-arm64 only** ([support.md](support.md)). On an Intel Mac or a musl distribution the package installs and `ffi-rs` resolves, and then `chtypes fetch ` has nothing to give you. +
Rust diff --git a/docs/limitations.md b/docs/limitations.md index 48972cf..93002d3 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -46,7 +46,7 @@ The rules genuinely differ: `256` into a `UInt8` column stores `0`, while `x = 2 ## Filters are shadow-only for now -No read-side security may be enforced on the filter surface until the WHERE-truth rig gates green. Until then it is for shadow and replay: run it beside your existing enforcement and compare, do not replace. +No read-side security may be enforced on the filter surface until a release explicitly lifts this limitation — the CHANGELOG will say so, and until it does, assume it has not. Until then the surface is for shadow and replay: run it beside your existing enforcement and compare, do not replace. The parse-once block twin does not change that — it is a performance shape, not a maturity signal, and sits under the same gate. diff --git a/docs/reference/artifact.md b/docs/reference/artifact.md index 90c5d39..3038742 100644 --- a/docs/reference/artifact.md +++ b/docs/reference/artifact.md @@ -80,13 +80,14 @@ The reference algorithm (`chtypes.NewRegistry` in `go/chtypes/multiversion.go`): 1. `ReadDir(registry)`. For each entry that is a directory: 2. Read `manifest.json`. **If it is missing or unparseable, skip the directory silently** — a registry may legitimately contain scratch directories, and a `.DS_Store` is not a version. 3. `dlopen(dir/, RTLD_NOW | RTLD_LOCAL)`. A failure here IS an error and MUST abort with the path and the `dlerror()` text: a directory that has a manifest and does not load is broken, not absent. -4. Resolve the `chs_*` symbols by name. Four are **mandatory** — `chs_clickhouse_version`, `chs_init`, `chs_schema_compile`, `chs_rows`. If any is missing, `dlclose` and reject the library: it is not a chtypes artifact. -5. Resolve `chs_abi_revision`. **Absent** -> the artifact predates the probe; record revision `0` and continue with the rules below — absence is ignorance, not incompatibility. **Present** -> call it. If it returns a value that is neither `0` nor the revision the binding was written against (`CHS_ABI_REVISION`), **reject the library**, naming both numbers: the artifact has positively stated that the binding's declarations do not describe it, and calling through them is undefined. 6. Every other symbol is **optional**. A missing one means "this artifact predates the feature" and MUST degrade to `unsupported` at call time, never to a load failure. The reference returns a private `-3` from its C shims for a missing `chs_schema_engine` / `chs_schema_ttl` and turns it into `CodeUnsupported` with `"this artifact predates engine support (rebuild it)"`; a missing `chs_row` returns `NULL`, reported as `"this artifact predates chs_row (rebuild it)"`. -6. Column introspection is **all-or-nothing**: `chs_schema_column_count`, `_name`, `_type`, `_default_expr`, `_default_kind`, `_default_is_literal` shipped together. If any is missing, treat the whole group as absent and leave the schema's column list empty rather than partially populated. -7. Ask the library its own version: `chs_clickhouse_version()`. **The library names itself; nothing is inferred from the path.** Derive the minor line from that string. -8. `chs_init(timezone, unsafe_families, out_err)`, once per library, with the contents of that version's own `unsafe_families.txt`. Each library keeps its own DateLUT and its own refuse-list. `out_err` MAY be `NULL`; when it is not and the call fails, it carries ClickHouse's own message (free with `chs_free`) — the reachable failure is an unknown timezone. -9. Index the library under **both** its exact version and its minor line. -10. If zero libraries loaded, that is an error naming the directory — an empty registry is a configuration mistake, not an empty result. +4. Resolve the `chs_*` symbols by name. Four are **mandatory everywhere** — `chs_clickhouse_version`, `chs_init`, `chs_schema_compile`, `chs_rows`. If any is missing, `dlclose` and reject the library: it is not a chtypes artifact. **The four bindings do not agree beyond that**, and this is a known divergence rather than a rule. The Go reference named above also requires `chs_free`, `chs_validate_type` and `chs_schema_free` — seven in all — because its C shims call those three without NULL checks, so admitting a library that lacks one would trade a clean load error for a SIGSEGV on first use. Python, TypeScript and Rust treat all three as optional and degrade to `unsupported`. No artifact this repository builds is affected: every one exports all seven. Which number is the contract is open — see issue #13. +5. Resolve `chs_abi_revision`. **Absent** -> the artifact predates the probe; record revision `0` and continue with the rules below — absence is ignorance, not incompatibility. **Present** -> call it. If it returns a value that is neither `0` nor the revision the binding was written against (`CHS_ABI_REVISION`), **reject the library**, naming both numbers: the artifact has positively stated that the binding's declarations do not describe it, and calling through them is undefined. +6. Every other symbol is **optional**. A missing one means "this artifact predates the feature" and MUST degrade to `unsupported` at call time, never to a load failure. The reference returns a private `-3` from its C shims for a missing `chs_schema_engine` / `chs_schema_ttl` and turns it into `CodeUnsupported` with `"this artifact predates engine support (rebuild it)"`; a missing `chs_row` returns `NULL`, reported as `"this artifact predates chs_row (rebuild it)"`. +7. Column introspection is **all-or-nothing**: `chs_schema_column_count`, `_name`, `_type`, `_default_expr`, `_default_kind`, `_default_is_literal` shipped together. If any is missing, treat the whole group as absent and leave the schema's column list empty rather than partially populated. +8. Ask the library its own version: `chs_clickhouse_version()`. **The library names itself; nothing is inferred from the path.** Derive the minor line from that string. +9. `chs_init(timezone, unsafe_families, out_err)`, once per library, with the contents of that version's own `unsafe_families.txt`. Each library keeps its own DateLUT and its own refuse-list. `out_err` MAY be `NULL`; when it is not and the call fails, it carries ClickHouse's own message (free with `chs_free`) — the reachable failure is an unknown timezone. +10. Index the library under **both** its exact version and its minor line. +11. If zero libraries loaded, that is an error naming the directory — an empty registry is a configuration mistake, not an empty result. `RTLD_LOCAL` is not a detail: it is what keeps each library's ClickHouse symbols private, so two builds that both define `DB::DataTypeFactory` never collide. A loader that uses `RTLD_GLOBAL` will appear to work and answer with the wrong version's semantics. diff --git a/docs/reference/bindings.md b/docs/reference/bindings.md index 5d5dd66..1f909db 100644 --- a/docs/reference/bindings.md +++ b/docs/reference/bindings.md @@ -16,32 +16,32 @@ Registry ── For(version) ──▶ Library ── CompileDDL(ddl) ──▶ **Every row of this table is enforced.** `tests/parity/manifest.json` is the machine-readable half of this document — one entry per logical capability, its spelling in all four languages, and for the shared vocabularies the VALUE every binding must answer with — and each language's own suite checks its own column (`python/tests/test_parity.py`, `ts/test/parity.test.ts`, `go/chtypes/parity_test.go`, `rust/tests/parity.rs`). The manifest is not derived from this table at test time and never will be: markdown is for humans and a test that parses one breaks on a reflow. The two are kept in step by a check that every spelling named here appears there, and a gap a binding SHOULD have is written into the manifest as an `absent` carrying its reason — an absence nobody had to justify in writing fails the manifest's own integrity check. -| Concept | Go | Python | TypeScript | Rust | Notes | -| -------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Artifact directory loader | `NewRegistry(dir)` | `Registry(dir)` | `new Registry(dir)` | `Registry::new(dir)` | Scans one subdirectory per version. The directory is optional in Go, Python and TypeScript and then means the `docs/guides/fetch.md` §1 search path; Rust spells that `Registry::from_search_path()`, having no default arguments | -| One loaded version | `*Library` | `Library` | `Library` | `Library` | Carries `Version`, `Minor`, `Path` | -| ABI revision (binding) | `chtypes.ABIRevision` | `chtypes.ABI_REVISION` | `ABI_REVISION` | `chtypes::ABI_REVISION` | The revision the binding was written against; cgo reads the header macro, the rest mirror it by hand | -| ABI revision (artifact) | `lib.ABIRevision` | `lib.abi_revision` | `lib.abiRevision` | `lib.abi_revision()` | `0` = predates the probe. A different nonzero value is refused at load | -| Resolve a version | `r.For(v)` | `r.for_version(v)` / `r[v]` | `r.for(v)` | `r.for_version(v)` | Minor line **or** exact patch | -| List versions | `r.Versions()` | `r.versions()` | `r.versions()` | `r.versions()` | Minor lines, in numeric release order | -| List loaded libraries | `r.Libraries()` | `r.libraries()` | `r.libraries()` | `r.libraries()` | The same numeric order — §Version selection rule 2 governs _every_ ordered surface, and this is one of them. Python's additionally opens the lines it has only discovered; Go, TypeScript and Rust list what is already loaded and open nothing | -| The search path | — | `r.search_path` | `r.searchPath` | `r.search_path()` | The `docs/guides/fetch.md` §1 directories, in order. Go keeps it unexported and renders it into the not-found message instead | -| Compile a column list | `lib.CompileDDL(ddl)` | `lib.compile_ddl(ddl)` | `lib.compileDdl(ddl)` | `lib.compile(ddl).compile()` | Rust is a builder — see §One compile function | -| … under a declared settings profile | `lib.CompileDDL(ddl, WithCompileSettings(m), WithCompileMode(m))` | `lib.compile_ddl(ddl, settings=…, mode=…)` | `lib.compileDdl(ddl, {settings, mode})` | `lib.compile(ddl).settings(…).mode(…).compile()` | **ONE function per SDK, options optional** — see §One compile function. Declaring no settings behaves exactly as if the parameter did not exist | -| … does this artifact have that channel | `lib.HasCompileSettings()` | `lib.has_compile_settings()` | `lib.hasCompileSettings()` | `lib.has_compile_settings()` | An artifact linked before `chs_schema_compile_with_settings` answers false, and a caller that needs the profile honored must ask rather than assume | -| Canonicalize a type | `lib.ValidateType(expr)` | `lib.validate_type(expr)` | `lib.validateType(expr)` | `lib.validate_type(expr)` | | -| Declare the engine | `s.SetEngine(engine, orderBy)` | `s.set_engine(engine, order_by)` | `s.setEngine(...)` | `s.set_engine(engine, order_by, NO_SETTINGS)` | | -| … + MergeTree settings | `s.SetEngine(engine, orderBy, WithMergeTreeSettings(m))` | `s.set_engine(..., merge_tree_settings=…)` | `s.setEngine(..., {mergeTreeSettings})` | `s.set_engine(engine, order_by, settings)` | Same one function. Unknown name ⇒ the server's **115 rejection**; a non-default declared value ⇒ **-2 unsupported** (never silently ignored); a binding MUST NOT flatten those two into one verdict — see rule 12 | -| Declare the rows TTL | `s.SetTTL(ttl)` | `s.set_ttl(ttl)` | `s.setTtl(ttl)` | `s.set_ttl(ttl)` | | -| One row | `s.Row(format, raw)` | `s.row(format, raw)` | `s.row(format, raw)` | `s.row(format, raw)` | | -| One row + settings | `s.RowWithSettings(format, raw, settings)` | `s.row(..., settings=)` | `s.row(..., settings)` | `s.row_with_settings(format, raw, settings)` | A default argument is fine, and so is a second function: Python and TypeScript have named optional arguments and use them, Go and Rust do not and do not | -| A whole body | `s.Rows(format, body, settings)` | `s.rows(...)` | `s.rows(...)` | `s.rows(format, body, settings)` | | -| … with the export channel | `s.RowsExport(..., exportFormat, docFlags)` | `s.rows(..., export=…, doc_flags=…)` | `s.rows(..., {exportFormat, docFlags})` | `s.rows_export(..., export, flags)` | §Revision 3. Still ONE `chs_rows` call. Same split as the row above: an optional argument where the language has one, a second function where it does not | -| Parse a body into a block | `s.ParseBlock(format, body, settings)` | `s.parse_block(...)` | `s.parseBlock(...)` | `s.parse_block(...)` | §Revision 4 — parse once, evaluate K filters against it | -| Compile a filter | `s.CompileFilter(expr, WithFilterParams(p))` | `s.compile_filter(expr, params=…)` | `s.compileFilter(expr, {params})` | `s.compile_filter(expr, params)` | §Revision 3, §Revision 4. Params cross as STRINGS and are never hand-escaped into the expression text | -| Release | `s.Close()` | context manager / `close()` | `s.close()` / `Symbol.dispose` | `Drop` | Rust has no inherent `close()` on purpose: the borrow checker enforces the free-before-schema order the other three enforce at runtime | -| Process teardown | package-level only — `*Registry`/`*Library` expose none, deliberately | `Registry.close()` / `Library.close()`; `Registry` is a context manager | `registry.close()` / `library.shutdown()`; `Registry`, `Schema`, `Filter` and `Block` are `Symbol.dispose` | `Registry::shutdown()` / `Library::shutdown()` | **Required before `dlclose`** — see below. Note that in both Python and TypeScript it is the _Registry_ that carries the scope-based form and the `Library` that does not; that asymmetry is deliberate for now and §Teardown says why | -| Server timezone | `chtypes.Timezone` (package-level, set before the first call) | `Registry(…, timezone=…)` | `new Registry(dir, {timezone})` | `Registry::with_timezone(dir, tz)`, `RegistryOptions::timezone` | The timezone assumed for bare `DateTime`/`DateTime64` columns. **Never the host's `TZ`**, which would leak into every result; `UTC` is what a stock ClickHouse container runs. Go's is a process-global and the other three are per-registry | +| Concept | Go | Python | TypeScript | Rust | Notes | +| -------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Artifact directory loader | `NewRegistry(dir)` | `Registry(dir)` | `new Registry(dir)` | `Registry::new(dir)` | Scans one subdirectory per version. The directory is optional in Go, Python and TypeScript and then means the `docs/guides/fetch.md` §1 search path; Rust spells that `Registry::from_search_path()`, having no default arguments | +| One loaded version | `*Library` | `Library` | `Library` | `Library` | Carries `Version`, `Minor`, `Path` | +| ABI revision (binding) | `chtypes.ABIRevision` | `chtypes.ABI_REVISION` | `ABI_REVISION` | `chtypes::ABI_REVISION` | The revision the binding was written against; cgo reads the header macro, the rest mirror it by hand | +| ABI revision (artifact) | `lib.ABIRevision` | `lib.abi_revision` | `lib.abiRevision` | `lib.abi_revision()` | `0` = predates the probe. A different nonzero value is refused at load | +| Resolve a version | `r.For(v)` | `r.for_version(v)` / `r[v]` | `r.for(v)` | `r.for_version(v)` | Minor line **or** exact patch | +| List versions | `r.Versions()` | `r.versions()` | `r.versions()` | `r.versions()` | Minor lines, in numeric release order | +| List loaded libraries | `r.Libraries()` | `r.libraries()` | `r.libraries()` | `r.libraries()` | The same numeric order — §Version selection rule 2 governs _every_ ordered surface, and this is one of them. Python's additionally opens the lines it has only discovered; Go, TypeScript and Rust list what is already loaded and open nothing | +| The search path | — | `r.search_path` | `r.searchPath` | `r.search_path()` | The `docs/guides/fetch.md` §1 directories, in order. Go keeps it unexported and renders it into the not-found message instead | +| Compile a column list | `lib.CompileDDL(ddl)` | `lib.compile_ddl(ddl)` | `lib.compileDdl(ddl)` | `lib.compile(ddl).compile()` | Rust is a builder — see §One compile function | +| … under a declared settings profile | `lib.CompileDDL(ddl, WithCompileSettings(m), WithCompileMode(m))` | `lib.compile_ddl(ddl, settings=…, mode=…)` | `lib.compileDdl(ddl, {settings, mode})` | `lib.compile(ddl).settings(…).mode(…).compile()` | **ONE function per SDK, options optional** — see §One compile function. Declaring no settings behaves exactly as if the parameter did not exist | +| … does this artifact have that channel | `lib.HasCompileSettings()` | `lib.has_compile_settings()` | `lib.hasCompileSettings()` | `lib.has_compile_settings()` | Always true for any artifact this repository builds — `chs_schema_compile` is mandatory, and the separate settings-aware entry point was folded into it in the 2026-08-24 consolidation. The probe remains for a third-party artifact predating that, and a caller that needs the profile honored should still ask rather than assume | +| Canonicalize a type | `lib.ValidateType(expr)` | `lib.validate_type(expr)` | `lib.validateType(expr)` | `lib.validate_type(expr)` | | +| Declare the engine | `s.SetEngine(engine, orderBy)` | `s.set_engine(engine, order_by)` | `s.setEngine(...)` | `s.set_engine(engine, order_by, NO_SETTINGS)` | | +| … + MergeTree settings | `s.SetEngine(engine, orderBy, WithMergeTreeSettings(m))` | `s.set_engine(..., merge_tree_settings=…)` | `s.setEngine(..., {mergeTreeSettings})` | `s.set_engine(engine, order_by, settings)` | Same one function. Unknown name ⇒ the server's **115 rejection**; a non-default declared value ⇒ **-2 unsupported** (never silently ignored); a binding MUST NOT flatten those two into one verdict — see rule 12 | +| Declare the rows TTL | `s.SetTTL(ttl)` | `s.set_ttl(ttl)` | `s.setTtl(ttl)` | `s.set_ttl(ttl)` | | +| One row | `s.Row(format, raw)` | `s.row(format, raw)` | `s.row(format, raw)` | `s.row(format, raw)` | | +| One row + settings | `s.RowWithSettings(format, raw, settings)` | `s.row(..., settings=)` | `s.row(..., settings)` | `s.row_with_settings(format, raw, settings)` | A default argument is fine, and so is a second function: Python and TypeScript have named optional arguments and use them, Go and Rust do not and do not | +| A whole body | `s.Rows(format, body, settings)` | `s.rows(...)` | `s.rows(...)` | `s.rows(format, body, settings)` | | +| … with the export channel | `s.RowsExport(..., exportFormat, docFlags)` | `s.rows(..., export=…, doc_flags=…)` | `s.rows(..., {exportFormat, docFlags})` | `s.rows_export(..., export, flags)` | §Revision 3. Still ONE `chs_rows` call. Same split as the row above: an optional argument where the language has one, a second function where it does not | +| Parse a body into a block | `s.ParseBlock(format, body, settings)` | `s.parse_block(...)` | `s.parseBlock(...)` | `s.parse_block(...)` | §Revision 4 — parse once, evaluate K filters against it | +| Compile a filter | `s.CompileFilter(expr, WithFilterParams(p))` | `s.compile_filter(expr, params=…)` | `s.compileFilter(expr, {params})` | `s.compile_filter(expr, params)` | §Revision 3, §Revision 4. Params cross as STRINGS and are never hand-escaped into the expression text | +| Release | `s.Close()` | context manager / `close()` | `s.close()` / `Symbol.dispose` | `Drop` | Rust has no inherent `close()` on purpose: the borrow checker enforces the free-before-schema order the other three enforce at runtime | +| Process teardown | package-level only — `*Registry`/`*Library` expose none, deliberately | `Registry.close()` / `Library.close()`; `Registry` is a context manager | `registry.close()` / `library.shutdown()`; `Registry`, `Schema`, `Filter` and `Block` are `Symbol.dispose` | `Registry::shutdown()` / `Library::shutdown()` | **Required before `dlclose`** — see below. Note that in both Python and TypeScript it is the _Registry_ that carries the scope-based form and the `Library` that does not; that asymmetry is deliberate for now and §Teardown says why | +| Server timezone | `chtypes.Timezone` (package-level, set before the first call) | `Registry(…, timezone=…)` | `new Registry(dir, {timezone})` | `Registry::with_timezone(dir, tz)`, `RegistryOptions::timezone` | The timezone assumed for bare `DateTime`/`DateTime64` columns. **Never the host's `TZ`**, which would leak into every result; `UTC` is what a stock ClickHouse container runs. Go's is a process-global and the other three are per-registry | A binding MAY additionally expose the statically linked, single-version shape (the reference's package-level `ValidateType` / `CompileDDL`, linked against one artifact). It is the fast path and it is optional; the Registry path is the product. @@ -52,7 +52,7 @@ At ABI revision 3 the C `chs_rows` gained `export_format`, `doc_flags` and `out_ - **`Rows()` keeps today's behavior exactly**: one `chs_rows` call with `export_format = CHS_EXPORT_NONE` (`-1`) and `doc_flags = CHS_DOC_ALL` (`7`), `out_bytes = NULL`. The document that comes back is byte-identical to revision 2's, so nothing downstream moves. - **`RowsExport()`** (per `docs/proposals/rows-export.md`: flags 0 by default, the bitmask exposed, `Payload []byte` + `Spans` from `out_bytes` + `row_spans`) and the filter SDK surface (`CompileFilter` / verdict types) were specified for the SDK cycle that FOLLOWS the C surface — this revision's bindings changes were the mechanical pass-through above and the hand-kept `ABI_REVISION` mirrors bumping to 3 in the same cycle, nothing more. One C call per SDK method, always; never a second call, never re-parsing. **That follow-on cycle has landed**, in all four bindings, at some point before the repository split squashed its history: the export channel and the filter surface are in the object-model table above and `tests/parity/manifest.json` enforces both. This paragraph is kept as the statement of the sequencing rule — the C surface first, the SDK surface in the cycle after — not as a description of what is missing. - **Lean documents stay SDK-derivable.** Under `CHS_DOC_TRANSFORMS` without `CHS_DOC_VALUES` the C layer retains every `cols[]` entry any spec'd detector could fire on, with the full field set (the conservative byte-equality retention rule. A binding runs the SAME detectors of §Transformed over the retained entries — no new classifier exists on either side, and the reason vocabulary does not move into C. Under `CHS_DOC_VALUES` without `CHS_DOC_TRANSFORMS` the reference parse is skipped C-side (`ref` is `null`, no `wire`), so detectors 2 and 3 have nothing to run on — that is the caller's explicit choice, not data loss. -- **Filter verdicts map to a four-state type, and two of the states are fail-closed.** `'t'`/`'f'` are answers; `'e'` (the predicate threw on this row — the server would have failed the whole query) and `'d'` (this library declines) are NOT answers, and a caller enforcing visibility MUST hide the row / fail the request on both. A binding MUST NOT collapse `'e'` or `'d'` into `false`-the-answer: under `NOT`, a decline-read-as-false inverts fail-closed into fail-open — the measured leak class. Unknown verdict characters degrade to the decline state, mirroring the unknown- `outcome` rule below. **No SDK may offer filter-backed read-side enforcement until the WHERE-truth rig gates green**; until then the surface is shadow/replay. +- **Filter verdicts map to a four-state type, and two of the states are fail-closed.** `'t'`/`'f'` are answers; `'e'` (the predicate threw on this row — the server would have failed the whole query) and `'d'` (this library declines) are NOT answers, and a caller enforcing visibility MUST hide the row / fail the request on both. A binding MUST NOT collapse `'e'` or `'d'` into `false`-the-answer: under `NOT`, a decline-read-as-false inverts fail-closed into fail-open — the measured leak class. The four states and the fail-closed rule are identical in every binding; **the spelling is not.** Go, Python and Rust render the wire characters `'t'`/`'f'`/`'e'`/`'d'`; **TypeScript renders `'true'`/`'false'`/`'error'`/`'decline'`**. That is a real divergence, not an idiom difference — two SDKs cannot share a log format or a test fixture across it — and it is open as issue #13. Match on your own binding's spelling, never on the characters as though they were universal. Unknown verdict characters degrade to the decline state, mirroring the unknown- `outcome` rule below. **No SDK may offer filter-backed read-side enforcement until the WHERE-truth rig gates green**; until then the surface is shadow/replay. - A filter handle wraps BOTH pointers' lifetimes: the SDK object MUST keep its schema object alive (a reference, not a copy) and free the filter before the schema — the C layer does not refcount. ### Revision 4: filter query parameters, and the block-parse twin @@ -180,7 +180,7 @@ ttl_expired ttl_column_expired `Lossy()` is **false** for exactly four of them — `reformat`, `default_filled`, `zero_filled`, `default_materialized` — and true for everything else. All of them are still _reported_: a preview must show the tenant what the table will actually hold, and `1700000000` becoming `"2023-11-14 22:13:20"` is a visible change even though nothing was lost. Only the lossy ones are a warning. -**How a binding may implement it.** Two independent detectors run and their union is reported (`go/chtypes/transform.go`): +**How a binding may implement it.** Three independent detectors run and their union is reported (`go/chtypes/transform.go`): 1. _Supplied vs stored_ — compare the raw input text against ClickHouse's rendering of what it kept. Models nothing; catches changes a widened type makes identically (a calendar roll-over, `2024-02-30 → 2024-03-01`) and types with no wider type to compare against (`Float64`, `Int256`, `String`). 2. _Reference type_ — the C layer already parsed each field a second time through a structurally identical type with widened leaves and reported it as `ref` / `ref_type`. This one names the _reason_ precisely (an overflow wrap versus a decimal truncation) and still fires where the supplied text is not comparable (a CSV field, a base64 blob). diff --git a/docs/reference/python.md b/docs/reference/python.md index bbba113..42fc407 100644 --- a/docs/reference/python.md +++ b/docs/reference/python.md @@ -54,7 +54,7 @@ Row-level verdicts are **returned, never raised**: a rejected row is a `RowResul The export channel is keyword-only on `rows`: `export=Format.JSON_COMPACT_EACH_ROW` and `doc_flags=DOC_VALUES | DOC_TRANSFORMS`. There is no separate `rows_export` method. -`Schema`, `Filter` and `Block` are all context managers. +`Registry`, `Schema`, `Filter` and `Block` are all context managers. (`Library` deliberately is not — see `bindings.md` §Teardown: reopening after a full close segfaults, so a scope-based release on a `Library` is the mid-lifecycle teardown that section warns against.) ## Filter and Block