From 834a0935d141ce2e2f7b7db1ab7f9440d80c0e0e Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:24 -0400 Subject: [PATCH 01/59] feat(sdk): add Go client SDK with full API-tree parity Complete Go SDK at clients/go/ with zero third-party runtime dependencies. Covers ingest, query (structured + SQL), streaming (SSE with reconnect), live queries, pipes, DLQ, schema, policy, and health endpoints. - Immutable QueryBuilder with generics (FetchTyped[Row], SQL[Row]) - SSE StreamController with reconnect/backoff and client-side filtering - LiveQuery drain-then-switch backfill (stream-first, dedup, go live) - Reflect-based Insert handling any []T, not just []map[string]any - wavehouse-codegen CLI for generating typed row structs from /v1/schema - 42 unit tests + 44 cross-language wire-format conformance cases - E2E test scaffolding (build tag e2e, 9 tests against live server) - Cross-language conformance runner for TS SDK (tests/conformance/) - Makefile targets: verify-go-sdk, test-go-sdk, test-go-sdk-e2e, lint-go-sdk --- CHANGELOG.md | 1 + Makefile | 45 +- clients/go/README.md | 235 ++++++++++ clients/go/client_test.go | 100 ++++ clients/go/cmd/wavehouse-codegen/main.go | 338 ++++++++++++++ clients/go/conformance_test.go | 377 +++++++++++++++ clients/go/dlq.go | 42 ++ clients/go/e2e_test.go | 436 ++++++++++++++++++ clients/go/errors.go | 81 ++++ clients/go/errors_test.go | 145 ++++++ clients/go/example_test.go | 71 +++ clients/go/go.mod | 3 + clients/go/http.go | 213 +++++++++ clients/go/http_test.go | 218 +++++++++ clients/go/live_query.go | 158 +++++++ clients/go/namespaces_test.go | 193 ++++++++ clients/go/pipes.go | 99 ++++ clients/go/policy.go | 42 ++ clients/go/query_builder.go | 310 +++++++++++++ clients/go/query_builder_test.go | 259 +++++++++++ clients/go/schema.go | 33 ++ clients/go/stream.go | 561 +++++++++++++++++++++++ clients/go/sys.go | 17 + clients/go/table.go | 205 +++++++++ clients/go/table_test.go | 209 +++++++++ clients/go/testdata/wire_cases.json | 547 ++++++++++++++++++++++ clients/go/types.go | 245 ++++++++++ clients/go/wavehouse.go | 142 ++++++ tests/conformance/conformance_ts.mjs | 279 +++++++++++ 29 files changed, 5601 insertions(+), 3 deletions(-) create mode 100644 clients/go/README.md create mode 100644 clients/go/client_test.go create mode 100644 clients/go/cmd/wavehouse-codegen/main.go create mode 100644 clients/go/conformance_test.go create mode 100644 clients/go/dlq.go create mode 100644 clients/go/e2e_test.go create mode 100644 clients/go/errors.go create mode 100644 clients/go/errors_test.go create mode 100644 clients/go/example_test.go create mode 100644 clients/go/go.mod create mode 100644 clients/go/http.go create mode 100644 clients/go/http_test.go create mode 100644 clients/go/live_query.go create mode 100644 clients/go/namespaces_test.go create mode 100644 clients/go/pipes.go create mode 100644 clients/go/policy.go create mode 100644 clients/go/query_builder.go create mode 100644 clients/go/query_builder_test.go create mode 100644 clients/go/schema.go create mode 100644 clients/go/stream.go create mode 100644 clients/go/sys.go create mode 100644 clients/go/table.go create mode 100644 clients/go/table_test.go create mode 100644 clients/go/testdata/wire_cases.json create mode 100644 clients/go/types.go create mode 100644 clients/go/wavehouse.go create mode 100644 tests/conformance/conformance_ts.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 3db0d718..e0ba485d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Go SDK (`clients/go/`) — official Go client with full API-tree parity against the TypeScript SDK, zero third-party runtime dependencies, cross-language wire-format conformance tests - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. diff --git a/Makefile b/Makefile index de1ebea2..3174e5c6 100644 --- a/Makefile +++ b/Makefile @@ -362,6 +362,10 @@ lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint) + TS/ lint-go: $(GOLANGCI_LINT) go-mod-download $(call run,golangci-lint,$(GOLANGCI_LINT) run ./... --allow-parallel-runners,run make fix to auto-fix what is fixable) +.PHONY: lint-go-sdk +lint-go-sdk: $(GOLANGCI_LINT) + $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./...,) + .PHONY: lint-ts lint-ts: pnpm-install $(call run,Biome (lint + format + imports),$(PNPM) -s -w run check,run make fix to auto-fix what is fixable) @@ -424,6 +428,18 @@ endif tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) +# verify-go-sdk: static checks for clients/go/ — a nested Go module (its own +# go.mod), so it's invisible to `go list ./...` from the root and every leaf +# above (GO_DIRS, lint-go, vulncheck, tidy) silently skips it. Scoped and run +# explicitly here instead. `go vet` needs its own module context (cd +# clients/go), but gofumpt is a pure syntax formatter with no module +# resolution of its own, so the repo-pinned $(GOFUMPT) binary can format it +# directly by path from the root — no second tool pin needed. +.PHONY: verify-go-sdk +verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt + $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) + $(call run,gofumpt (Go SDK),$(GOFUMPT) -l clients/go | (! grep .),run make fix to apply formatting) + # fix: apply auto-fixes everywhere, fanned out into three tracks that touch # disjoint files — Go (.go + go.mod/sum), TS/JS/JSON (Biome), Markdown — so they # run in parallel safely. The Go track is itself a serial chain (tidy → gofumpt → @@ -467,7 +483,8 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (9): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck on the Go +# Leaves (10): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, +# verify-go-sdk (go vet + gofumpt on the nested clients/go module) on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) for JS/TS + Markdown + prose; # check-docs (astro check — the only leaf that writes, to docs/.astro/, and @@ -483,7 +500,7 @@ verify: ## Run all static checks across the repo (Go + TS + docs, parallelized) @printf "$(GREEN)$(BOLD)✔ All static checks passed$(RESET)\n" .PHONY: verify-parallel -verify-parallel: tidy fmt-go lint-go lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths vulncheck check-docs typecheck-ts +verify-parallel: tidy fmt-go lint-go lint-go-sdk lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths vulncheck check-docs typecheck-ts verify-go-sdk # typecheck-ts: tsc --noEmit on the SDK. Its own target (was inline in verify's # recipe) so it can run as a parallel leaf of verify-parallel. @@ -713,6 +730,28 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi +# test-go-sdk: unit tests for clients/go/ — a nested Go module (its own +# go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and +# needs its own leaf; a root `go test ./...` wouldn't reach it either. Zero +# third-party runtime or test deps (stdlib only, no go.sum), so no +# go-mod-download prereq. Not yet wired into the Go/TS coverage gate — see +# verify-go-sdk above for the same "nested module, own leaf" reasoning. +.PHONY: test-go-sdk +test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" + @cd clients/go && go test ./... + +# test-go-sdk-e2e: runs the Go SDK's E2E tests against a live WaveHouse +# instance. Requires a running server (e.g. `make dev` in the main repo). +# Env vars: +# WAVEHOUSE_URL base URL of the server (default: http://localhost:8080) +# WAVEHOUSE_AUTH bearer token for auth (optional; omit for default_role) +# The tests skip gracefully when the server is unreachable. +.PHONY: test-go-sdk-e2e +test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) + @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" + @cd clients/go && go test -tags e2e -v -count=1 -timeout 60s ./... + # Aggregator: recipe-based with $(MAKE) calls so suites run sequentially even # under `make -j N`. The suites bind ports / spin testcontainers / start the # release binary, so concurrent execution is unsafe. @@ -749,7 +788,7 @@ cov: ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-r # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage diff --git a/clients/go/README.md b/clients/go/README.md new file mode 100644 index 00000000..e3af7bd4 --- /dev/null +++ b/clients/go/README.md @@ -0,0 +1,235 @@ +# WaveHouse Go SDK + +Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. + +**Zero third-party runtime dependencies** — stdlib only. + +**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go/)** + +## Install + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + // Create an unauthenticated client (uses the server's default_role). + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check. + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } + + ctx := context.Background() + + // Insert a row. + _, err := client.From("clicks").Insert(ctx, map[string]any{ + "page": "/home", "button": "cta", + }) + if err != nil { + log.Fatal(err) + } + + // Query with the fluent builder. + page, err := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(ctx) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } +} +``` + +## Authentication + +```go +// Static token. +client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) + +// Dynamic token (e.g. rotated). +client = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: func(ctx context.Context) (string, error) { + return fetchFreshToken(ctx) + }, +}) +``` + +## Typed Queries (Generics) + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + client.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +## Batch Insert (NDJSON) + +```go +// Array of maps — serialized to NDJSON automatically. +result, _ := client.From("clicks").Insert(ctx, []map[string]any{ + {"page": "/a", "button": "cta"}, + {"page": "/b", "button": "nav"}, +}) +// result.OK, result.Total, result.Succeeded, result.Failed + +// Pre-formatted NDJSON string. +result, _ = client.From("clicks").InsertNDJSON(ctx, + `{"page":"/a"}`+"\n"+`{"page":"/b"}`, +) +``` + +## Streaming (SSE) + +```go +stream := client.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() + +// Channel-based consumption. +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) +} + +// Or callback-based. +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, + Status: func(s wavehouse.StreamStatus) { fmt.Println("status:", s) }, +}) +defer unsub() +``` + +## Live Queries + +```go +lq := client.From("clicks"). + SelectAll(). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Historical backfill. + fmt.Println("initial rows:", len(rows)) + }, + Next: func(e wavehouse.StreamEvent) { + // Live events after backfill. + fmt.Println("live:", e.Data) + }, + }, nil) +defer lq.Close() +``` + +## Named Pipes + +```go +// Execute a pipe. +rows, _ := wavehouse.Fetch[map[string]any](ctx, + client.Pipe("top_pages", map[string]any{"limit": 10}), +) + +// Admin: manage pipes. +client.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + AllowedRoles: []string{"viewer", "admin"}, +}) +pipes, _ := client.Pipes.List(ctx) +client.Pipes.Delete(ctx, "old_pipe") +``` + +## Admin + +```go +// Schema introspection (admin-only). +schemas, _ := client.Schema.List(ctx) +client.Schema.Refresh(ctx) + +// Policy management (admin-only). +policy, _ := client.Policy.Get(ctx) +client.Policy.Set(ctx, policy) +result, _ := client.Policy.Validate(ctx, policy) + +// DLQ stats (admin-only). +stats, _ := client.DLQ.List(ctx) + +// Raw SQL (admin-only). +rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM clicks") +``` + +## Codegen + +Generate Go structs from a running WaveHouse instance: + +```bash +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ + --url http://localhost:8080 \ + --auth \ + --out ./db_types.go \ + --package myapp +``` + +The CLI reads `/v1/schema` (admin-only) and maps ClickHouse types to Go types: + +| ClickHouse | Go | +|---|---| +| `String`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | +| `UInt8/16/32/64` | `uint8/16/32/64` | +| `Int8/16/32/64` | `int8/16/32/64` | +| `Float32/64` | `float32/64` | +| `Bool` | `bool` | +| `Nullable(T)` | `*T` | +| `Array(T)` | `[]T` | +| `Map(K,V)` | `map[K]V` | +| `UInt128/256`, `Int128/256`, `Decimal*` | `string` | + +## Error Handling + +All SDK operations return `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`): + +```go +page, err := client.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } +} +``` + +The HTTP layer retries 5xx and network errors with exponential backoff (default 2 retries). 503 with `Retry-After` is honored. Context cancellation returns immediately with code `ABORTED`. + +## License + +Apache-2.0 diff --git a/clients/go/client_test.go b/clients/go/client_test.go new file mode 100644 index 00000000..4f57507f --- /dev/null +++ b/clients/go/client_test.go @@ -0,0 +1,100 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewClient_Defaults(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + if c.ctx.maxRetries != 2 { + t.Fatalf("want default maxRetries=2, got %d", c.ctx.maxRetries) + } + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want baseURL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_StripsTrailingSlashes(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080///"}) + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want stripped URL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_CustomMaxRetries(t *testing.T) { + c := NewClient(Config{ + BaseURL: "http://localhost:8080", + Options: &ClientOptions{MaxRetries: 5}, + }) + if c.ctx.maxRetries != 5 { + t.Fatalf("want 5, got %d", c.ctx.maxRetries) + } +} + +func TestNewClient_HasNamespaces(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + if c.Schema == nil { + t.Fatal("Schema namespace nil") + } + if c.Policy == nil { + t.Fatal("Policy namespace nil") + } + if c.DLQ == nil { + t.Fatal("DLQ namespace nil") + } + if c.Sys == nil { + t.Fatal("Sys namespace nil") + } + if c.Pipes == nil { + t.Fatal("Pipes namespace nil") + } +} + +func TestClient_From(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // Verify the table name appears in the URL. + if r.URL.Query().Get("table") != "events" { + t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) + } + json.NewEncoder(w).Encode([]map[string]any{}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + _, _ = c.From("events").Fetch(context.Background()) +} + +func TestClient_SQL(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/admin/query" { + t.Errorf("want /v1/admin/query, got %s", r.URL.Path) + } + var body map[string]string + json.NewDecoder(r.Body).Decode(&body) + if body["sql"] != "SELECT 1" { + t.Errorf("want sql=SELECT 1, got %s", body["sql"]) + } + json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } +} + +func TestStaticToken(t *testing.T) { + fn := StaticToken("abc") + token, err := fn(context.Background()) + if err != nil { + t.Fatal(err) + } + if token != "abc" { + t.Fatalf("want abc, got %s", token) + } +} diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go new file mode 100644 index 00000000..1df138b9 --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -0,0 +1,338 @@ +// Command wavehouse-codegen reads a WaveHouse server's /v1/schema endpoint +// and generates Go struct definitions for use with the wavehouse SDK. +// +// Usage: +// +// wavehouse-codegen --url http://localhost:8080 --out ./db.go --auth +package main + +import ( + "context" + "encoding/json" + "fmt" + "go/format" + "net/http" + "os" + "slices" + "strings" + "unicode" +) + +type cliArgs struct { + url string + out string + auth string + pkg string +} + +func parseArgs() cliArgs { + args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} + for i := 1; i < len(os.Args); i++ { + switch os.Args[i] { + case "--url", "-u": + i++ + if i < len(os.Args) { + args.url = os.Args[i] + } + case "--out", "-o": + i++ + if i < len(os.Args) { + args.out = os.Args[i] + } + case "--auth", "-a": + i++ + if i < len(os.Args) { + args.auth = os.Args[i] + } + case "--package", "-p": + i++ + if i < len(os.Args) { + args.pkg = os.Args[i] + } + case "--help", "-h": + fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema + +Options: + --url, -u WaveHouse base URL (default: http://localhost:8080) + --out, -o Output .go file path (default: ./wavehouse_types.go) + --auth, -a Bearer token for authenticated /v1/schema endpoint + --package, -p Go package name (default: main) + --help, -h Show this help`) + os.Exit(0) + } + } + return args +} + +type column struct { + Name string `json:"name"` + Type string `json:"type"` + IsNullable bool `json:"is_nullable"` + HasDefault bool `json:"has_default"` +} + +type tableSchema struct { + Name string `json:"name"` + Columns []column `json:"columns"` +} + +func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { + url := strings.TrimRight(baseURL, "/") + "/v1/schema" + req, err := http.NewRequestWithContext(ctx, "GET", url, nil) + if err != nil { + return nil, err + } + if auth != "" { + req.Header.Set("Authorization", "Bearer "+auth) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) + } + + // Server returns either []tableSchema or map[string]tableSchema. + var raw json.RawMessage + if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { + return nil, err + } + // Try array first. + var arr []tableSchema + if err := json.Unmarshal(raw, &arr); err == nil { + m := make(map[string]tableSchema, len(arr)) + for _, t := range arr { + m[t.Name] = t + } + return m, nil + } + var m map[string]tableSchema + if err := json.Unmarshal(raw, &m); err != nil { + return nil, err + } + return m, nil +} + +// chTypeToGo maps a ClickHouse type string (as reported by /v1/schema) to a +// Go type name suitable for a JSON struct field. +// +// We deliberately don't import clickhouse-go's type catalog +// (github.com/ClickHouse/clickhouse-go/v2/lib/column) for this. It's public +// and does expose a real ClickHouse-type-string parser — +// column.Type(chType).Column(name, sc).ScanType() — but it answers a +// different question than the one we're asking. That catalog maps to the Go +// types the *driver* scans query results into over the native protocol +// (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for +// Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types +// that round-trip cleanly through the JSON the /v1/schema and query +// endpoints actually speak. ClickHouse's JSON output renders DateTime as +// "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default +// time.Time JSON unmarshaling; big integers and decimals are similarly +// rendered as JSON strings, not driver-native types. Adopting the driver's +// ScanType() as-is would produce generated structs that don't unmarshal the +// server's actual JSON, and would drag uuid/decimal/orb/net imports into +// generated output that today has zero non-stdlib dependencies. So we keep +// the hand-rolled JSON-oriented mapping below, informed by (but not bound +// to) the type set clickhouse-go's lib/column recognizes. +func chTypeToGo(chType string) string { + // Unwrap Nullable → pointer. + if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { + inner := chType[9 : len(chType)-1] + return "*" + chTypeToGo(inner) + } + // Unwrap LowCardinality. + if strings.HasPrefix(chType, "LowCardinality(") && strings.HasSuffix(chType, ")") { + return chTypeToGo(chType[15 : len(chType)-1]) + } + // Unwrap SimpleAggregateFunction(func, InnerType) — readable columns in + // AggregatingMergeTree/SummingMergeTree rollup tables. The value on the + // wire is just InnerType; the aggregate function name only describes how + // merges combine rows. + if strings.HasPrefix(chType, "SimpleAggregateFunction(") && strings.HasSuffix(chType, ")") { + inner := chType[len("SimpleAggregateFunction(") : len(chType)-1] + if comma := findTopLevelComma(inner); comma != -1 { + return chTypeToGo(strings.TrimSpace(inner[comma+1:])) + } + return "any" + } + // String-like. + switch { + case chType == "String", + strings.HasPrefix(chType, "FixedString("), + chType == "UUID", + strings.HasPrefix(chType, "DateTime"), + strings.HasPrefix(chType, "Date"), + // Time/Time64 are ClickHouse's newer time-of-day types (distinct + // from DateTime); same JSON-string-not-RFC3339 story applies. + strings.HasPrefix(chType, "Time"), + strings.HasPrefix(chType, "Enum8("), + strings.HasPrefix(chType, "Enum16("), + chType == "IPv4", + chType == "IPv6": + return "string" + case chType == "Bool", chType == "Boolean": + return "bool" + } + // Numeric — map widths honestly. + switch { + case chType == "UInt8": + return "uint8" + case chType == "UInt16": + return "uint16" + case chType == "UInt32": + return "uint32" + case chType == "UInt64": + return "uint64" + case chType == "Int8": + return "int8" + case chType == "Int16": + return "int16" + case chType == "Int32": + return "int32" + case chType == "Int64": + return "int64" + case chType == "Float32": + return "float32" + case chType == "Float64": + return "float64" + case chType == "BFloat16": + return "float32" + case strings.HasPrefix(chType, "Decimal"), + strings.HasPrefix(chType, "UInt128"), + strings.HasPrefix(chType, "UInt256"), + strings.HasPrefix(chType, "Int128"), + strings.HasPrefix(chType, "Int256"): + return "string" // big numbers are strings in JSON + } + // Array. + if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { + inner := chType[6 : len(chType)-1] + return "[]" + chTypeToGo(inner) + } + // Map. + if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { + inner := chType[4 : len(chType)-1] + comma := findTopLevelComma(inner) + if comma != -1 { + k := chTypeToGo(strings.TrimSpace(inner[:comma])) + v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) + return "map[" + k + "]" + v + } + return "map[string]any" + } + return "any" +} + +func findTopLevelComma(s string) int { + depth := 0 + for i := range len(s) { + switch s[i] { + case '(': + depth++ + case ')': + depth-- + case ',': + if depth == 0 { + return i + } + } + } + return -1 +} + +func pascalCase(s string) string { + parts := strings.FieldsFunc(s, func(r rune) bool { + return r == '_' || r == '-' || r == ' ' || r == '.' + }) + var sb strings.Builder + for _, p := range parts { + if len(p) == 0 { + continue + } + runes := []rune(p) + runes[0] = unicode.ToUpper(runes[0]) + sb.WriteString(string(runes)) + } + result := sb.String() + if result == "" { + return result + } + // Go identifiers can't start with a digit (e.g. a table named + // "2fa_events" would otherwise produce the invalid identifier + // "2faEvents"). Prefix with "X" to keep it a valid, exported name. + if unicode.IsDigit([]rune(result)[0]) { + result = "X" + result + } + return result +} + +func generate(schemas map[string]tableSchema, pkg string) string { + var sb strings.Builder + fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) + + names := make([]string, 0, len(schemas)) + for name := range schemas { + names = append(names, name) + } + slices.Sort(names) + + for _, name := range names { + schema := schemas[name] + typeName := pascalCase(name) + "Row" + fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) + for _, col := range schema.Columns { + goType := chTypeToGo(col.Type) + fieldName := pascalCase(col.Name) + jsonTag := col.Name + if col.HasDefault { + jsonTag += ",omitempty" + } + fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) + } + sb.WriteString("}\n\n") + } + + return sb.String() +} + +func main() { + args := parseArgs() + fmt.Printf("Fetching schema from %s...\n", args.url) + + schemas, err := fetchSchemas(context.Background(), args.url, args.auth) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + if len(schemas) == 0 { + fmt.Fprintln(os.Stderr, "No tables found. Is WaveHouse running with tables in ClickHouse?") + os.Exit(1) + } + + names := make([]string, 0, len(schemas)) + for name := range schemas { + names = append(names, name) + } + slices.Sort(names) + fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) + + output := generate(schemas, args.pkg) + + // gofmt the output. A failure here means the generated source is not + // valid Go (e.g. a table/column name produced an invalid identifier); + // don't write unusable output and claim success. + formatted, err := format.Source([]byte(output)) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: generated code is not valid Go: %v\n", err) + os.Exit(1) + } + + if err := os.WriteFile(args.out, formatted, 0o644); err != nil { + fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) + os.Exit(1) + } + + fmt.Printf("✓ Types written to %s\n", args.out) +} diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go new file mode 100644 index 00000000..5f5edf74 --- /dev/null +++ b/clients/go/conformance_test.go @@ -0,0 +1,377 @@ +package wavehouse + +import ( + "context" + _ "embed" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" +) + +// wireCasesJSON embeds the shared wire-format conformance fixture so the +// test binary is self-contained: it works from a module archive or a +// standalone checkout without depending on paths outside the Go module. +// +//go:embed testdata/wire_cases.json +var wireCasesJSON []byte + +// wireCase is one entry in the shared wire_cases.json fixture. +type wireCase struct { + Name string `json:"name"` + Endpoint string `json:"endpoint"` + Table string `json:"table"` + Operations []wireOp `json:"operations"` + PipeName string `json:"pipe_name"` + PipeParams map[string]any `json:"pipe_params"` + PipeDefBody json.RawMessage `json:"pipe_def"` + PolicyBody json.RawMessage `json:"policy_body"` + SQL string `json:"sql"` + ExpectedPath string `json:"expected_path"` + ExpectedMethod string `json:"expected_method"` + ExpectedContentType string `json:"expected_content_type"` + ExpectedBody json.RawMessage `json:"expected_body"` + ExpectedRawBody *string `json:"expected_raw_body"` +} + +type wireOp struct { + Method string `json:"method"` + Args []any `json:"args"` +} + +func loadWireCases(t *testing.T) []wireCase { + t.Helper() + var cases []wireCase + if err := json.Unmarshal(wireCasesJSON, &cases); err != nil { + t.Fatalf("parse wire_cases.json: %v", err) + } + return cases +} + +// captured holds the HTTP request details from a single SDK call. +type captured struct { + method string + path string // path + query string + contentType string + body string +} + +func TestConformance_WireFormat(t *testing.T) { + cases := loadWireCases(t) + + for _, tc := range cases { + t.Run(tc.Name, func(t *testing.T) { + var cap captured + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + cap.method = r.Method + cap.path = r.URL.RequestURI() + cap.contentType = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + cap.body = string(raw) + + // Return valid JSON so the SDK doesn't error on decode. + w.Header().Set("Content-Type", "application/json") + switch { + case strings.HasPrefix(r.URL.Path, "/v1/dlq"): + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) + case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": + json.NewEncoder(w).Encode([]TableSchema{}) + case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": + json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": + json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": + json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) + case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": + json.NewEncoder(w).Encode([]Pipe{}) + default: + json.NewEncoder(w).Encode([]map[string]any{}) + } + })) + defer srv.Close() + + c := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) + ctx := context.Background() + + // Execute the case. + switch tc.Endpoint { + case "query": + q := c.From(tc.Table).Select() + q = applyOps(t, q, tc.Table, c, tc.Operations) + _, _ = q.FetchUntyped(ctx) + + case "ingest": + if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + data := tc.Operations[0].Args[0] + _, _ = c.From(tc.Table).Insert(ctx, data) + } + + case "ingest_batch": + if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + rawArr, ok := tc.Operations[0].Args[0].([]any) + if !ok { + t.Fatalf("batch insert args[0] is not an array") + } + rows := make([]map[string]any, len(rawArr)) + for i, r := range rawArr { + rows[i] = toStringMap(r) + } + _, _ = c.From(tc.Table).Insert(ctx, rows) + } + + case "pipe": + p := c.Pipe(tc.PipeName, tc.PipeParams) + _, _ = p.FetchUntyped(ctx) + + case "sql": + _, _ = SQL[map[string]any](ctx, c, tc.SQL) + + case "health": + _ = c.Sys.Health(ctx) + + case "schema_list": + _, _ = c.Schema.List(ctx) + + case "schema_refresh": + _ = c.Schema.Refresh(ctx) + + case "policy_get": + _, _ = c.Policy.Get(ctx) + + case "policy_set": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + _ = c.Policy.Set(ctx, &pol) + + case "policy_validate": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + _, _ = c.Policy.Validate(ctx, &pol) + + case "dlq_list": + _, _ = c.DLQ.List(ctx) + + case "dlq_table": + _, _ = c.DLQ.Table(ctx, tc.Table) + + case "pipes_list": + _, _ = c.Pipes.List(ctx) + + case "pipes_get": + _, _ = c.Pipes.Get(ctx, tc.PipeName) + + case "pipes_set": + var def PipeDef + if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { + t.Fatalf("parse pipe_def: %v", err) + } + _ = c.Pipes.Set(ctx, tc.PipeName, def) + + case "pipes_delete": + _ = c.Pipes.Delete(ctx, tc.PipeName) + + default: + t.Skipf("unhandled endpoint: %s", tc.Endpoint) + } + + // Verify method. + if tc.ExpectedMethod != "" && cap.method != tc.ExpectedMethod { + t.Errorf("method: want %s, got %s", tc.ExpectedMethod, cap.method) + } + + // Verify path. + if tc.ExpectedPath != "" { + // Normalize: the SDK may use different encoding (+ vs %20). + wantPath := normalizePath(tc.ExpectedPath) + gotPath := normalizePath(cap.path) + if wantPath != gotPath { + t.Errorf("path: want %s, got %s", tc.ExpectedPath, cap.path) + } + } + + // Verify content type. + if tc.ExpectedContentType != "" && cap.contentType != tc.ExpectedContentType { + t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, cap.contentType) + } + + // Verify raw body (for NDJSON). + if tc.ExpectedRawBody != nil { + if cap.body != *tc.ExpectedRawBody { + t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, cap.body) + } + return + } + + // Verify JSON body. + if tc.ExpectedBody != nil && string(tc.ExpectedBody) != "null" { + var want, got any + if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { + t.Fatalf("parse expected_body: %v", err) + } + if err := json.Unmarshal([]byte(cap.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, cap.body) + } + if !deepEqualJSON(want, got) { + wantJSON, _ := json.MarshalIndent(want, "", " ") + gotJSON, _ := json.MarshalIndent(got, "", " ") + t.Errorf("body mismatch:\n want: %s\n got: %s", wantJSON, gotJSON) + } + } + }) + } +} + +// applyOps replays the operation chain from the fixture onto a QueryBuilder. +// Fixtures always put select first (mirroring real usage), so rebuilding on +// select is safe and keeps this simple. +func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wireOp) *QueryBuilder { + t.Helper() + q := c.From(table).Select() + + for _, op := range ops { + switch op.Method { + case "select": + q = c.From(table).Select(toStringSlice(op.Args)...) + case "selectAll": + q = q.SelectAll() + case "where": + if len(op.Args) != 3 { + t.Fatalf("where needs 3 args, got %d", len(op.Args)) + } + col := op.Args[0].(string) + opStr := FilterOp(op.Args[1].(string)) + val := op.Args[2] + q = q.Where(col, opStr, val) + case "count": + col, alias := stringArg(op.Args, 0, "*"), stringArg(op.Args, 1, "count") + q = q.Count(col, alias) + case "sum": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Sum(col, alias) + case "avg": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Avg(col, alias) + case "min": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Min(col, alias) + case "max": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Max(col, alias) + case "countDistinct": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.CountDistinct(col, alias) + case "aggregate": + fn := stringArg(op.Args, 0, "") + col := stringArg(op.Args, 1, "") + alias := stringArg(op.Args, 2, "") + q = q.Aggregate(fn, col, alias) + case "groupBy": + cols := toStringSlice(op.Args) + q = q.GroupBy(cols...) + case "orderBy": + col := stringArg(op.Args, 0, "") + dir := stringArg(op.Args, 1, "asc") + q = q.OrderBy(col, dir) + case "limit": + n := intArg(op.Args, 0) + q = q.Limit(n) + case "timeRange": + col := stringArg(op.Args, 0, "") + since := stringArg(op.Args, 1, "") + until := stringArg(op.Args, 2, "") + q = q.TimeRange(col, since, until) + case "cacheTTL": + n := intArg(op.Args, 0) + q = q.CacheTTL(n) + } + } + return q +} + +func stringArg(args []any, i int, fallback string) string { + if i >= len(args) { + return fallback + } + s, ok := args[i].(string) + if !ok { + return fallback + } + return s +} + +func intArg(args []any, i int) int { + if i >= len(args) { + return 0 + } + switch v := args[i].(type) { + case float64: + return int(v) + case int: + return v + default: + return 0 + } +} + +func toStringSlice(args []any) []string { + out := make([]string, len(args)) + for i, a := range args { + out[i], _ = a.(string) + } + return out +} + +func toStringMap(v any) map[string]any { + m, ok := v.(map[string]any) + if ok { + return m + } + return nil +} + +// deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal +// to ints (JSON numbers decode as float64 in Go). +func deepEqualJSON(a, b any) bool { + return reflect.DeepEqual(normalizeJSON(a), normalizeJSON(b)) +} + +func normalizeJSON(v any) any { + switch val := v.(type) { + case map[string]any: + m := make(map[string]any, len(val)) + for k, v := range val { + m[k] = normalizeJSON(v) + } + return m + case []any: + s := make([]any, len(val)) + for i, v := range val { + s[i] = normalizeJSON(v) + } + return s + case float64: + // Normalize integer-valued floats to int for comparison. + if val == float64(int64(val)) { + return int64(val) + } + return val + default: + return val + } +} + +func normalizePath(p string) string { + // Normalize URL encoding differences (+ vs %20 for spaces). + return strings.ReplaceAll(p, "+", "%20") +} diff --git a/clients/go/dlq.go b/clients/go/dlq.go new file mode 100644 index 00000000..01248b2c --- /dev/null +++ b/clients/go/dlq.go @@ -0,0 +1,42 @@ +package wavehouse + +import ( + "context" + "net/url" +) + +// DLQNamespace provides admin-only dead-letter-queue statistics. +type DLQNamespace struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController +} + +// List returns DLQ statistics (message counts per table). Admin-only. +func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { + var stats DLQStats + if err := doRequest(d.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/dlq/stats", + }, &stats); err != nil { + return nil, err + } + return &stats, nil +} + +// Table returns DLQ stats filtered by table name. Admin-only. +func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { + var stats DLQStats + if err := doRequest(d.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/dlq/stats", + params: url.Values{"table": {name}}, + }, &stats); err != nil { + return nil, err + } + return &stats, nil +} + +// Stream subscribes to live DLQ events. Not yet functional server-side (#197). +func (d *DLQNamespace) Stream(opts *StreamOptions) *StreamController { + return d.createStream("dlq", opts) +} diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go new file mode 100644 index 00000000..ea8aa7bc --- /dev/null +++ b/clients/go/e2e_test.go @@ -0,0 +1,436 @@ +//go:build e2e + +package wavehouse + +import ( + "context" + "fmt" + "net/http" + "os" + "strings" + "testing" + "time" +) + +// e2eClient builds a Client pointing at the live WaveHouse instance. +// It reads WAVEHOUSE_URL (default http://localhost:8080) and the optional +// WAVEHOUSE_AUTH bearer token. The test is skipped when the server is +// unreachable — so `go test -tags e2e` degrades gracefully on a dev +// machine that isn't running the stack. +func e2eClient(t *testing.T) *Client { + t.Helper() + + base := os.Getenv("WAVEHOUSE_URL") + if base == "" { + base = "http://localhost:8080" + } + + cfg := Config{ + BaseURL: base, + Options: &ClientOptions{MaxRetries: 1}, + } + if tok := os.Getenv("WAVEHOUSE_AUTH"); tok != "" { + cfg.Auth = StaticToken(tok) + } + + // Probe the server before committing to the test. + probe, err := http.NewRequestWithContext( + context.Background(), "GET", base+"/v1/health", nil, + ) + if err != nil { + t.Skipf("e2e: bad WAVEHOUSE_URL %q: %v", base, err) + } + resp, err := http.DefaultClient.Do(probe) + if err != nil { + t.Skipf("e2e: server unreachable at %s: %v", base, err) + } + resp.Body.Close() + + return NewClient(cfg) +} + +// marker returns a unique string for the running test, useful for +// inserting distinguishable rows that won't collide across parallel runs. +func marker(t *testing.T) string { + t.Helper() + // Replace slashes in subtest names so it's a clean string value. + safe := strings.ReplaceAll(t.Name(), "/", "_") + return fmt.Sprintf("%s_%d", safe, time.Now().UnixNano()) +} + +// firstTable discovers a usable table from the schema list. Many E2E tests +// need a real table to insert/query — this avoids hardcoding a name. +func firstTable(t *testing.T, c *Client) string { + t.Helper() + ctx := context.Background() + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Skipf("e2e: cannot list schemas (auth?): %v", err) + } + for name := range schemas { + return name + } + t.Skip("e2e: no tables found — server has an empty schema") + return "" +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +func TestE2E_HealthCheck(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + if err := c.Sys.Health(ctx); err != nil { + t.Fatalf("Health check failed: %v", err) + } +} + +func TestE2E_SchemaList(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List failed: %v", err) + } + if len(schemas) == 0 { + t.Fatal("Schema.List returned zero tables — expected at least one") + } + // Quick sanity: every table should have columns. + for name, ts := range schemas { + if len(ts.Columns) == 0 { + t.Errorf("table %q has no columns", name) + } + } +} + +func TestE2E_InsertAndQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + mk := marker(t) + + // Discover columns so we can build a valid row. We need at least one + // string-ish column to inject our marker. Fall back to skipping if the + // table's schema doesn't have one we can use. + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts, ok := schemas[table] + if !ok { + t.Skipf("table %q vanished between discovery and use", table) + } + + row := buildMarkerRow(t, ts, mk) + markerCol := markerColumn(t, ts) + + res, err := c.From(table).Insert(ctx, row) + if err != nil { + t.Fatalf("Insert into %s failed: %v", table, err) + } + if !res.OK { + t.Fatalf("Insert into %s: OK=false", table) + } + + // Allow a moment for async ingestion to settle. + time.Sleep(500 * time.Millisecond) + + // Query it back. + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(1). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("Query failed: %v", err) + } + if len(page.Data) == 0 { + t.Fatal("Query returned zero rows — expected the inserted marker row") + } + got, _ := page.Data[0][markerCol].(string) + if got != mk { + t.Errorf("marker mismatch: want %q, got %q", mk, got) + } +} + +func TestE2E_BatchInsert(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts := schemas[table] + + mk := marker(t) + markerCol := markerColumn(t, ts) + + // Build 3 rows, each with the same marker so we can count them. + rows := make([]map[string]any, 3) + for i := range rows { + rows[i] = buildMarkerRow(t, ts, mk) + } + + res, err := c.From(table).Insert(ctx, rows) + if err != nil { + t.Fatalf("Batch insert failed: %v", err) + } + if !res.OK { + t.Fatalf("Batch insert: OK=false") + } + + time.Sleep(500 * time.Millisecond) + + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(10). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("Query after batch insert failed: %v", err) + } + if len(page.Data) < 3 { + t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(page.Data)) + } +} + +func TestE2E_QueryBuilder(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts := schemas[table] + + // Pick two columns for a minimal projection. + var cols []string + for _, col := range ts.Columns { + cols = append(cols, col.Name) + if len(cols) >= 2 { + break + } + } + + page, err := c.From(table). + Select(cols...). + OrderBy(cols[0], "asc"). + Limit(5). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("QueryBuilder chain failed: %v", err) + } + // We can't assert exact data, but the chain should execute without error + // and return at most 5 rows. + if len(page.Data) > 5 { + t.Errorf("Limit(5) returned %d rows", len(page.Data)) + } +} + +func TestE2E_TypedFetch(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + q := c.From(table).SelectAll().Limit(3) + page, err := FetchTyped[map[string]any](ctx, q) + if err != nil { + t.Fatalf("FetchTyped failed: %v", err) + } + // If the table has data we should get rows; if it's empty that's still + // a valid result. The important thing is no error and correct type. + for i, row := range page.Data { + if row == nil { + t.Errorf("row %d is nil", i) + } + } +} + +func TestE2E_SQLQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") + if err != nil { + // SQL requires admin role — skip gracefully if forbidden. + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: SQL query requires admin auth: %v", err) + } + t.Fatalf("SQL query failed: %v", err) + } + if len(rows) != 1 { + t.Fatalf("expected 1 row, got %d", len(rows)) + } + // ClickHouse returns numbers as strings or floats depending on format; + // accept either. + n := rows[0]["n"] + switch v := n.(type) { + case float64: + if v != 1 { + t.Errorf("expected n=1, got %v", v) + } + case string: + if v != "1" { + t.Errorf("expected n=1, got %q", v) + } + default: + t.Errorf("unexpected type for n: %T = %v", n, n) + } +} + +func TestE2E_PolicyGetSet(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pol, err := c.Policy.Get(ctx) + if err != nil { + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: Policy.Get requires admin auth: %v", err) + } + t.Fatalf("Policy.Get failed: %v", err) + } + + // Round-trip: set the same policy back. + if err := c.Policy.Set(ctx, pol); err != nil { + t.Fatalf("Policy.Set (round-trip) failed: %v", err) + } + + // Read again and verify tables still match. + pol2, err := c.Policy.Get(ctx) + if err != nil { + t.Fatalf("Policy.Get (after set) failed: %v", err) + } + if len(pol2.Tables) != len(pol.Tables) { + t.Errorf("policy table count changed: %d -> %d", len(pol.Tables), len(pol2.Tables)) + } +} + +func TestE2E_PipesCRUD(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pipeName := fmt.Sprintf("e2e_test_%d", time.Now().UnixNano()) + + // Create + def := PipeDef{ + SQL: "SELECT 1 AS ok", + Description: "E2E test pipe — safe to delete", + } + if err := c.Pipes.Set(ctx, pipeName, def); err != nil { + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: Pipes.Set requires admin auth: %v", err) + } + t.Fatalf("Pipes.Set (create) failed: %v", err) + } + + // Cleanup: always attempt delete so we don't litter. + t.Cleanup(func() { + _ = c.Pipes.Delete(context.Background(), pipeName) + }) + + // Get + pipe, err := c.Pipes.Get(ctx, pipeName) + if err != nil { + t.Fatalf("Pipes.Get failed: %v", err) + } + if pipe.SQL != def.SQL { + t.Errorf("pipe SQL mismatch: want %q, got %q", def.SQL, pipe.SQL) + } + + // List — verify it appears + pipes, err := c.Pipes.List(ctx) + if err != nil { + t.Fatalf("Pipes.List failed: %v", err) + } + found := false + for _, p := range pipes { + if p.Name == pipeName { + found = true + break + } + } + if !found { + t.Errorf("Pipes.List: created pipe %q not found in list of %d pipes", pipeName, len(pipes)) + } + + // Delete + if err := c.Pipes.Delete(ctx, pipeName); err != nil { + t.Fatalf("Pipes.Delete failed: %v", err) + } + + // Verify gone — Get should fail. + _, err = c.Pipes.Get(ctx, pipeName) + if err == nil { + t.Error("Pipes.Get after delete: expected error, got nil") + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +// markerColumn finds the first String/LowCardinality(String) column in the +// schema that we can use to inject a test marker value. +func markerColumn(t *testing.T, ts TableSchema) string { + t.Helper() + for _, col := range ts.Columns { + ct := strings.ToLower(col.Type) + if ct == "string" || strings.Contains(ct, "string") { + return col.Name + } + } + t.Skipf("e2e: table %q has no string column for marker injection", ts.Name) + return "" +} + +// buildMarkerRow constructs a minimal valid row for the table, injecting the +// marker into the first string column and using sensible defaults for other +// required columns. +func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { + t.Helper() + row := make(map[string]any) + markerSet := false + for _, col := range ts.Columns { + if col.HasDefault { + continue // let the server fill defaults + } + ct := strings.ToLower(col.Type) + switch { + case !markerSet && strings.Contains(ct, "string"): + row[col.Name] = mk + markerSet = true + case strings.Contains(ct, "string"): + row[col.Name] = "e2e" + case strings.Contains(ct, "int"): + row[col.Name] = 0 + case strings.Contains(ct, "float") || strings.Contains(ct, "decimal"): + row[col.Name] = 0.0 + case strings.Contains(ct, "date") || strings.Contains(ct, "datetime"): + row[col.Name] = time.Now().UTC().Format(time.RFC3339) + case strings.Contains(ct, "bool"): + row[col.Name] = false + default: + row[col.Name] = "" + } + } + if !markerSet { + t.Skipf("e2e: table %q has no non-default string column for marker", ts.Name) + } + return row +} + +// isHTTPStatus checks whether err is a wavehouse.Error with the given status. +func isHTTPStatus(err error, status int) bool { + if err == nil { + return false + } + if e, ok := err.(*Error); ok { + return e.Status == status + } + return false +} diff --git a/clients/go/errors.go b/clients/go/errors.go new file mode 100644 index 00000000..1a161a1b --- /dev/null +++ b/clients/go/errors.go @@ -0,0 +1,81 @@ +package wavehouse + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "net/http" +) + +// Error is the structured error returned by all SDK operations. Use +// [errors.As] to extract it from wrapped errors. +type Error struct { + // Status is the HTTP status code (0 for network/abort errors). + Status int `json:"status"` + // Code is a machine-readable error code (e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED"). + Code string `json:"code"` + // Message is a human-readable description. + Message string `json:"message"` + // Details contains the full parsed error body, if available. + Details map[string]any `json:"details,omitempty"` + // Retryable indicates whether the request can be retried. + Retryable bool `json:"retryable"` +} + +func (e *Error) Error() string { + if e.Status > 0 { + return fmt.Sprintf("wavehouse: %s (%d): %s", e.Code, e.Status, e.Message) + } + return fmt.Sprintf("wavehouse: %s: %s", e.Code, e.Message) +} + +// IsRetryable reports whether err wraps a retryable [*Error]. +func IsRetryable(err error) bool { + var e *Error + if errors.As(err, &e) { + return e.Retryable + } + return false +} + +// parseErrorResponse creates an Error from an HTTP response. +func parseErrorResponse(res *http.Response) *Error { + var body map[string]any + if res.Body != nil { + raw, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) // cap at 1 MiB + _ = json.Unmarshal(raw, &body) + } + + msg := "" + if s, ok := body["error"].(string); ok { + msg = s + } else if s, ok := body["message"].(string); ok { + msg = s + } else { + msg = http.StatusText(res.StatusCode) + } + + retryable := res.StatusCode == http.StatusServiceUnavailable || res.StatusCode >= 500 + return &Error{ + Status: res.StatusCode, + Code: fmt.Sprintf("HTTP_%d", res.StatusCode), + Message: msg, + Details: body, + Retryable: retryable, + } +} + +// networkError creates an Error from a transport-level failure. +func networkError(cause error) *Error { + msg := "unknown network error" + if cause != nil { + msg = cause.Error() + } + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: msg, + Retryable: true, + } +} diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go new file mode 100644 index 00000000..0c9dd05f --- /dev/null +++ b/clients/go/errors_test.go @@ -0,0 +1,145 @@ +package wavehouse + +import ( + "errors" + "io" + "net/http" + "strings" + "testing" +) + +func TestParseErrorResponse_JSONError(t *testing.T) { + res := &http.Response{ + StatusCode: 404, + Body: io.NopCloser(strings.NewReader(`{"error":"unknown table: foo"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Status != 404 { + t.Fatalf("want status 404, got %d", e.Status) + } + if e.Code != "HTTP_404" { + t.Fatalf("want code HTTP_404, got %s", e.Code) + } + if e.Message != "unknown table: foo" { + t.Fatalf("want message 'unknown table: foo', got %s", e.Message) + } + if e.Retryable { + t.Fatal("4xx should not be retryable") + } +} + +func TestParseErrorResponse_MessageField(t *testing.T) { + res := &http.Response{ + StatusCode: 400, + Body: io.NopCloser(strings.NewReader(`{"message":"bad request"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "bad request" { + t.Fatalf("want 'bad request', got %s", e.Message) + } +} + +func TestParseErrorResponse_FallsBackToStatusText(t *testing.T) { + res := &http.Response{ + StatusCode: 500, + Body: io.NopCloser(strings.NewReader(`{"code":123}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "Internal Server Error" { + t.Fatalf("want status text fallback, got %s", e.Message) + } +} + +func TestParseErrorResponse_NonJSONBody(t *testing.T) { + res := &http.Response{ + StatusCode: 502, + Body: io.NopCloser(strings.NewReader("plain text")), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "Bad Gateway" { + t.Fatalf("want 'Bad Gateway', got %s", e.Message) + } + if e.Details != nil { + t.Fatal("details should be nil for non-JSON body") + } +} + +func TestParseErrorResponse_5xxRetryable(t *testing.T) { + tests := []struct { + status int + retryable bool + }{ + {400, false}, + {403, false}, + {500, true}, + {503, true}, + } + for _, tt := range tests { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Retryable != tt.retryable { + t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) + } + } +} + +func TestNetworkError(t *testing.T) { + e := networkError(errors.New("connection refused")) + if e.Code != "NETWORK_ERROR" { + t.Fatalf("want NETWORK_ERROR, got %s", e.Code) + } + if e.Message != "connection refused" { + t.Fatalf("want 'connection refused', got %s", e.Message) + } + if !e.Retryable { + t.Fatal("network errors should be retryable") + } + if e.Status != 0 { + t.Fatalf("want status 0, got %d", e.Status) + } +} + +func TestError_ErrorMethod(t *testing.T) { + e := &Error{Status: 404, Code: "HTTP_404", Message: "not found"} + got := e.Error() + if !strings.Contains(got, "HTTP_404") || !strings.Contains(got, "not found") { + t.Fatalf("unexpected Error() output: %s", got) + } + + e2 := &Error{Status: 0, Code: "NETWORK_ERROR", Message: "timeout"} + got2 := e2.Error() + if !strings.Contains(got2, "NETWORK_ERROR") { + t.Fatalf("unexpected Error() output: %s", got2) + } +} + +func TestIsRetryable(t *testing.T) { + if !IsRetryable(&Error{Retryable: true}) { + t.Fatal("want true for retryable error") + } + if IsRetryable(&Error{Retryable: false}) { + t.Fatal("want false for non-retryable error") + } + if IsRetryable(errors.New("plain error")) { + t.Fatal("want false for non-wavehouse error") + } +} + +func TestErrorsAs(t *testing.T) { + err := error(&Error{Status: 403, Code: "HTTP_403", Message: "forbidden"}) + var e *Error + if !errors.As(err, &e) { + t.Fatal("errors.As should find *Error") + } + if e.Status != 403 { + t.Fatalf("want 403, got %d", e.Status) + } +} diff --git a/clients/go/example_test.go b/clients/go/example_test.go new file mode 100644 index 00000000..03d8b518 --- /dev/null +++ b/clients/go/example_test.go @@ -0,0 +1,71 @@ +package wavehouse_test + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +// ExampleNewClient demonstrates creating an unauthenticated client and +// performing a health check. The Output assertion is omitted because the +// example needs a running server. +func ExampleNewClient() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check — returns nil when the server is reachable. + err := client.Sys.Health(context.Background()) + _ = err +} + +func ExampleNewClient_withAuth() { + _ = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("my-jwt-token"), + }) +} + +func ExampleClient_From() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Insert a row. + _, _ = client.From("clicks").Insert(context.Background(), map[string]any{ + "page": "/home", + "button": "cta", + }) + + // Query with the builder. + page, _ := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(context.Background()) + + for _, row := range page.Data { + fmt.Println(row["page"]) + } +} + +func ExampleSQL() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("admin-token"), + }) + + rows, err := wavehouse.SQL[map[string]any]( + context.Background(), client, + "SELECT page, count() as views FROM clicks GROUP BY page LIMIT 5", + ) + if err != nil { + log.Fatal(err) + } + for _, row := range rows { + fmt.Println(row["page"], row["views"]) + } +} diff --git a/clients/go/go.mod b/clients/go/go.mod new file mode 100644 index 00000000..84e065de --- /dev/null +++ b/clients/go/go.mod @@ -0,0 +1,3 @@ +module github.com/Wave-RF/WaveHouse/clients/go + +go 1.26.5 diff --git a/clients/go/http.go b/clients/go/http.go new file mode 100644 index 00000000..52aae6b4 --- /dev/null +++ b/clients/go/http.go @@ -0,0 +1,213 @@ +package wavehouse + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "net/http" + "net/url" + "strconv" + "time" +) + +// httpContext carries per-client state needed by every request. +type httpContext struct { + baseURL string + auth func(ctx context.Context) (string, error) + maxRetries int + httpClient *http.Client +} + +// requestOptions describes a single HTTP request. +type requestOptions struct { + method string + path string + body any // JSON-serialized if non-nil + rawBody string // sent verbatim if non-empty (takes precedence over body) + contentType string // overrides Content-Type (default "application/json") + params url.Values +} + +// doRequest is the internal fetch wrapper with auth, retry, and backoff. +// It decodes the response body into dst (unless dst is nil). +func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst any) error { + reqURL := buildURL(hctx.baseURL, opts.path, opts.params) + ct := opts.contentType + if ct == "" { + ct = "application/json" + } + + // Serialize body once so every retry sends identical bytes. + var bodyBytes []byte + if opts.rawBody != "" { + bodyBytes = []byte(opts.rawBody) + } else if opts.body != nil { + var err error + bodyBytes, err = json.Marshal(opts.body) + if err != nil { + return fmt.Errorf("wavehouse: marshal request body: %w", err) + } + } + + // Resolve auth once per request (not per attempt). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + return fmt.Errorf("wavehouse: auth provider: %w", err) + } + if token != "" { + authHeader = "Bearer " + token + } + } + + var lastErr error + maxAttempts := hctx.maxRetries + 1 + + // Retries below are not restricted by HTTP method — this matches the TS + // SDK's http.ts, which retries POST the same as GET on network errors, + // 503/Retry-After, and other retryable 5xx. For /v1/ingest, at-least-once + // delivery on retry is a documented contract (see docs/api.md's + // "At-least-once on retry" note); dedup is the prescribed server-side + // safety net when duplicate suppression matters. The only other mutation + // path, /v1/admin/query, is gated by admin_role, so repeated execution on + // retry is assumed to be an accepted risk for admin-only raw SQL. + for attempt := range maxAttempts { + var bodyReader io.Reader + if bodyBytes != nil { + bodyReader = bytes.NewReader(bodyBytes) + } + + req, err := http.NewRequestWithContext(ctx, opts.method, reqURL, bodyReader) + if err != nil { + return fmt.Errorf("wavehouse: build request: %w", err) + } + req.Header.Set("Content-Type", ct) + req.Header.Set("Accept", "application/json") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + res, err := hctx.httpClient.Do(req) + if err != nil { + // Context cancellation — return immediately, no retry. + if ctx.Err() != nil { + return &Error{ + Status: 0, + Code: "ABORTED", + Message: "Request aborted", + Retryable: false, + } + } + lastErr = networkError(err) + if attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + } + continue + } + + if res.StatusCode >= 200 && res.StatusCode < 300 { + defer res.Body.Close() + if dst == nil { + _, _ = io.Copy(io.Discard, res.Body) + return nil + } + raw, readErr := io.ReadAll(res.Body) + if readErr != nil { + return networkError(readErr) + } + if len(raw) == 0 { + return nil + } + if err := json.Unmarshal(raw, dst); err != nil { + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: fmt.Errorf("decode response: %w", err).Error(), + Retryable: false, + } + } + return nil + } + + apiErr := parseErrorResponse(res) + res.Body.Close() + + // 503 with Retry-After: wait the specified duration. + if res.StatusCode == http.StatusServiceUnavailable { + if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { + delay := 30 * time.Second + if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { + delay = time.Duration(secs) * time.Second + } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { + if d := time.Until(parsed); d > 0 { + delay = d + } + } + if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + lastErr = apiErr + continue + } + } + + // Retryable server errors (5xx). + if apiErr.Retryable && attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + lastErr = apiErr + continue + } + + return apiErr + } + + return lastErr +} + +func buildURL(base, path string, params url.Values) string { + u := base + path + if len(params) > 0 { + u += "?" + params.Encode() + } + return u +} + +func backoff(attempt int) time.Duration { + ms := 1000 * math.Pow(2, float64(attempt)) + if ms > 30000 { + ms = 30000 + } + return time.Duration(ms) * time.Millisecond +} + +func sleepWithContext(ctx context.Context, d time.Duration) error { + if ctx.Err() != nil { + return ctx.Err() + } + t := time.NewTimer(d) + defer t.Stop() + select { + case <-t.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// errIs checks if err wraps a *Error with the given code. +func errIs(err error, code string) bool { + var e *Error + if errors.As(err, &e) { + return e.Code == code + } + return false +} diff --git a/clients/go/http_test.go b/clients/go/http_test.go new file mode 100644 index 00000000..7ce4c964 --- /dev/null +++ b/clients/go/http_test.go @@ -0,0 +1,218 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +func testCtx(handler http.Handler) httpContext { + srv := httptest.NewServer(handler) + return httpContext{ + baseURL: srv.URL, + maxRetries: 0, + httpClient: srv.Client(), + } +} + +func TestDoRequest_SuccessfulGET(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatal(err) + } + if result["status"] != "ok" { + t.Fatalf("want ok, got %v", result) + } +} + +func TestDoRequest_POSTWithBody(t *testing.T) { + var gotBody map[string]string + var gotCT string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + json.NewDecoder(r.Body).Decode(&gotBody) + w.WriteHeader(200) + })) + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/ingest", + body: map[string]string{"page": "/home"}, + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/json" { + t.Fatalf("want application/json, got %s", gotCT) + } + if gotBody["page"] != "/home" { + t.Fatalf("want /home, got %v", gotBody) + } +} + +func TestDoRequest_RawBody(t *testing.T) { + var gotBody string + var gotCT string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw := make([]byte, 1024) + n, _ := r.Body.Read(raw) + gotBody = string(raw[:n]) + json.NewEncoder(w).Encode(map[string]int{"total": 1}) + })) + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/ingest", + rawBody: `{"page":"/a"}`, + contentType: "application/x-ndjson", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}` { + t.Fatalf("want raw body, got %s", gotBody) + } +} + +func TestDoRequest_AuthInjection(t *testing.T) { + var gotAuth string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAuth = r.Header.Get("Authorization") + w.WriteHeader(200) + })) + hctx.auth = StaticToken("my-token") + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/v1/schema", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer my-token" { + t.Fatalf("want 'Bearer my-token', got %s", gotAuth) + } +} + +func TestDoRequest_4xxNotRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + count.Add(1) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(404) + json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) + })) + hctx.maxRetries = 2 + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/v1/schema", + }, nil) + + if !errIs(err, "HTTP_404") { + t.Fatalf("want HTTP_404 error, got %v", err) + } + if count.Load() != 1 { + t.Fatalf("4xx should not retry, got %d attempts", count.Load()) + } +} + +func TestDoRequest_5xxRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + n := count.Add(1) + if n < 3 { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) + return + } + json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + })) + hctx.maxRetries = 2 + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatalf("want success after retries, got %v", err) + } + if count.Load() != 3 { + t.Fatalf("want 3 attempts, got %d", count.Load()) + } +} + +func TestDoRequest_AbortedOnCancel(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + time.Sleep(5 * time.Second) + })) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancel immediately + + err := doRequest(hctx, ctx, requestOptions{ + method: "GET", + path: "/health", + }, nil) + + if !errIs(err, "ABORTED") { + t.Fatalf("want ABORTED, got %v", err) + } +} + +func TestDoRequest_EmptyResponse(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(200) + })) + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/schema/refresh", + }, &result) + if err != nil { + t.Fatal(err) + } + // Empty body = no decode, result stays zero value. + if result != nil { + t.Fatalf("want nil, got %v", result) + } +} + +func TestBackoff(t *testing.T) { + tests := []struct { + attempt int + want time.Duration + }{ + {0, 1 * time.Second}, + {1, 2 * time.Second}, + {2, 4 * time.Second}, + {3, 8 * time.Second}, + {10, 30 * time.Second}, // capped at 30s + } + for _, tt := range tests { + got := backoff(tt.attempt) + if got != tt.want { + t.Errorf("backoff(%d) = %v, want %v", tt.attempt, got, tt.want) + } + } +} diff --git a/clients/go/live_query.go b/clients/go/live_query.go new file mode 100644 index 00000000..fad4e80a --- /dev/null +++ b/clients/go/live_query.go @@ -0,0 +1,158 @@ +package wavehouse + +import ( + "context" + "sync" +) + +// LiveQueryHandle controls a live query that combines historical backfill +// with a real-time stream. +type LiveQueryHandle struct { + stream *StreamController + cancel context.CancelFunc + closeOnce sync.Once +} + +// newLiveQuery starts a live query: opens the stream immediately, fetches +// historical data, deduplicates buffered events, then goes live. +func newLiveQuery( + stream *StreamController, + fetchFn func(ctx context.Context) ([]map[string]any, error), + sub *StreamSubscriber, + filters []QueryFilter, +) *LiveQueryHandle { + ctx, cancel := context.WithCancel(context.Background()) + lq := &LiveQueryHandle{ + stream: stream, + cancel: cancel, + } + + var ( + mu sync.Mutex + buffer []StreamEvent + buffering = true + closed = false + ) + + // Step 1: Subscribe to live events and buffer them. + stream.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + mu.Lock() + defer mu.Unlock() + if closed { + return + } + if buffering { + buffer = append(buffer, event) + } else if sub.Next != nil { + sub.Next(event) + } + }, + Status: func(s StreamStatus) { + if sub.Status != nil { + sub.Status(s) + } + }, + Error: func(err error) { + if sub.Error != nil { + sub.Error(err) + } + }, + }) + + // Step 2–5: Fetch historical and flush. + go func() { + rows, err := fetchFn(ctx) + if ctx.Err() != nil { + return + } + + // Step 3: Deliver initial snapshot. + if sub.Initial != nil { + sub.Initial(rows, err) + } + + if err != nil { + mu.Lock() + buffering = false + buffer = nil + mu.Unlock() + return + } + + // Step 4: Deduplicate buffered events. + var lastTimestamp string + if len(rows) > 0 { + lastRow := rows[len(rows)-1] + if ts, ok := lastRow["received_timestamp"].(string); ok { + lastTimestamp = ts + } + } + + // Step 5: Flush buffered events newer than the fetch. + // + // buffering stays true for the whole flush: events that arrive + // concurrently (after Subscribe's Next handler releases mu but + // before we're done here) must keep landing in buffer rather than + // being dispatched directly by the live path, or two goroutines + // could call sub.Next at once. We only flip buffering to false + // once a lock-protected check finds the buffer empty, which + // guarantees no event is ever handed to sub.Next by both paths + // and that delivery stays in arrival order. + for { + mu.Lock() + if closed { + mu.Unlock() + return + } + pending := buffer + buffer = nil + if len(pending) == 0 { + buffering = false + mu.Unlock() + break + } + mu.Unlock() + + for _, event := range pending { + mu.Lock() + c := closed + mu.Unlock() + if c { + return + } + // Use <= (not <) to filter events whose timestamp matches the last + // historical row — those rows were already delivered in the backfill + // response. If two distinct events share a timestamp and only one + // appeared in the backfill, the duplicate is lost; this matches the + // TS SDK's dedup behavior and is acceptable because received_timestamp + // has sub-millisecond precision in practice. + if lastTimestamp != "" && event.Timestamp <= lastTimestamp { + continue + } + if sub.Next != nil { + sub.Next(event) + } + } + } + }() + + // Cleanup on context cancel. + go func() { + <-ctx.Done() + mu.Lock() + closed = true + buffer = nil + mu.Unlock() + }() + + return lq +} + +// Close shuts down the live query and the underlying stream. +func (lq *LiveQueryHandle) Close() { + lq.closeOnce.Do(func() { + lq.cancel() + lq.stream.Close() + }) +} diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go new file mode 100644 index 00000000..84bc4b2f --- /dev/null +++ b/clients/go/namespaces_test.go @@ -0,0 +1,193 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func nsClient(handler http.Handler) *Client { + srv := httptest.NewServer(handler) + return NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) +} + +func TestSysNamespace_Health(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/health" { + t.Errorf("want /v1/health, got %s", r.URL.Path) + } + w.WriteHeader(200) + })) + err := c.Sys.Health(context.Background()) + if err != nil { + t.Fatal(err) + } +} + +func TestSchemaNamespace_List(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/schema" { + t.Errorf("want /v1/schema, got %s", r.URL.Path) + } + json.NewEncoder(w).Encode([]TableSchema{ + {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, + }) + })) + schemas, err := c.Schema.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, ok := schemas["clicks"]; !ok { + t.Fatal("want clicks in schemas") + } +} + +func TestSchemaNamespace_Refresh(t *testing.T) { + var gotMethod string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotMethod = r.Method + w.WriteHeader(200) + })) + err := c.Schema.Refresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} + +func TestPolicyNamespace_GetSetValidate(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == "GET": + json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case r.Method == "PUT": + w.WriteHeader(200) + case r.Method == "POST": + json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + } + })) + + pol, err := c.Policy.Get(context.Background()) + if err != nil { + t.Fatal(err) + } + if pol.Tables == nil { + t.Fatal("want tables map") + } + + err = c.Policy.Set(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + + v, err := c.Policy.Validate(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + if !v.Valid { + t.Fatal("want valid=true") + } +} + +func TestDLQNamespace_List(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + })) + stats, err := c.DLQ.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } +} + +func TestDLQNamespace_Table(t *testing.T) { + var gotParam string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotParam = r.URL.Query().Get("table") + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + })) + _, err := c.DLQ.Table(context.Background(), "clicks") + if err != nil { + t.Fatal(err) + } + if gotParam != "clicks" { + t.Fatalf("want table=clicks, got %s", gotParam) + } +} + +func TestPipesNamespace_CRUD(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.Method { + case "GET": + if r.URL.Path == "/v1/admin/pipes" { + json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) + } else { + json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) + } + case "PUT": + w.WriteHeader(200) + case "DELETE": + w.WriteHeader(200) + } + })) + + pipes, err := c.Pipes.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(pipes) != 1 || pipes[0].Name != "p1" { + t.Fatalf("want [p1], got %v", pipes) + } + + p, err := c.Pipes.Get(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } + if p.Name != "p1" { + t.Fatalf("want p1, got %s", p.Name) + } + + err = c.Pipes.Set(context.Background(), "p1", PipeDef{SQL: "SELECT 1"}) + if err != nil { + t.Fatal(err) + } + + err = c.Pipes.Delete(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } +} + +func TestPipeRef_Fetch(t *testing.T) { + var gotPath, gotMethod string + var gotBody map[string]any + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + gotMethod = r.Method + json.NewDecoder(r.Body).Decode(&gotBody) + json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) + })) + rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } + if gotPath != "/v1/pipes/top_pages" { + t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) + } + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} diff --git a/clients/go/pipes.go b/clients/go/pipes.go new file mode 100644 index 00000000..f87f4d64 --- /dev/null +++ b/clients/go/pipes.go @@ -0,0 +1,99 @@ +package wavehouse + +import ( + "context" + "net/url" +) + +// PipesNamespace provides admin-only named-pipe management. +type PipesNamespace struct { + ctx httpContext +} + +// List returns all registered pipes. Admin-only. +func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { + var pipes []Pipe + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/pipes", + }, &pipes); err != nil { + return nil, err + } + return pipes, nil +} + +// Get returns a single pipe definition by name. Admin-only. +func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { + var pipe Pipe + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/pipes/" + url.PathEscape(name), + }, &pipe); err != nil { + return nil, err + } + return &pipe, nil +} + +// Set creates or updates a pipe. Admin-only. +func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "PUT", + path: "/v1/admin/pipes/" + url.PathEscape(name), + body: def, + }, nil) +} + +// Delete removes a pipe by name. Admin-only. +func (p *PipesNamespace) Delete(ctx context.Context, name string) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "DELETE", + path: "/v1/admin/pipes/" + url.PathEscape(name), + }, nil) +} + +// PipeDef is the definition body for creating/updating a pipe (Pipe minus name). +type PipeDef struct { + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// PipeRef is a reference to a named query pipe. Use Fetch to execute it. +type PipeRef struct { + ctx httpContext + name string + params map[string]any + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch executes the pipe and returns the result rows decoded into []T. +func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { + body := p.params + if body == nil { + body = map[string]any{} + } + var rows []Row + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/pipes/" + url.PathEscape(p.name), + body: body, + }, &rows); err != nil { + return nil, err + } + return rows, nil +} + +// FetchUntyped executes the pipe and returns rows as []map[string]any. +func (p *PipeRef) FetchUntyped(ctx context.Context) ([]map[string]any, error) { + return Fetch[map[string]any](ctx, p) +} + +// Stream opens a live event stream from the pipe's underlying query. +// +// This streams by table name, using the pipe's own name as the table — it +// only works when the pipe name is also a valid table name. This matches +// the TS SDK's PipeRef.stream(), which has the same limitation. +func (p *PipeRef) Stream(opts *StreamOptions) *StreamController { + return p.createStream(p.name, opts) +} diff --git a/clients/go/policy.go b/clients/go/policy.go new file mode 100644 index 00000000..c7eaa2af --- /dev/null +++ b/clients/go/policy.go @@ -0,0 +1,42 @@ +package wavehouse + +import "context" + +// PolicyNamespace provides admin-only access-control policy management. +type PolicyNamespace struct { + ctx httpContext +} + +// Get returns the current access-control policy. Admin-only. +func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { + var pol Policy + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/policy", + }, &pol); err != nil { + return nil, err + } + return &pol, nil +} + +// Set replaces the entire access-control policy. Admin-only. +func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "PUT", + path: "/v1/admin/policy", + body: pol, + }, nil) +} + +// Validate checks a policy without applying it (dry run). Admin-only. +func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*ValidationResult, error) { + var result ValidationResult + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/admin/policy/validate", + body: pol, + }, &result); err != nil { + return nil, err + } + return &result, nil +} diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go new file mode 100644 index 00000000..9dda452d --- /dev/null +++ b/clients/go/query_builder.go @@ -0,0 +1,310 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/url" +) + +// DefaultLimit is applied when no explicit limit is set — deliberately tighter +// than the backend's DefaultMaxRows (10000) safety cap. +const DefaultLimit = 1000 + +// queryState is the immutable core of a QueryBuilder. +type queryState struct { + table string + columns []string + selectAll bool + aggregations []Aggregation + filters []QueryFilter + groupBy []string + orderBy []OrderClause + limit *int + timeRange *TimeRange + cacheTTL *int // ponytail: client-side only, not sent to server (#280) +} + +// QueryBuilder builds structured queries. Immutable — every chain method +// returns a new builder. Use Fetch or FetchUntyped to execute. +type QueryBuilder struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController + state queryState +} + +func (q *QueryBuilder) clone(mutate func(*queryState)) *QueryBuilder { + s := q.state + // Deep-copy slices so mutations don't alias. + s.columns = append([]string(nil), s.columns...) + s.aggregations = append([]Aggregation(nil), s.aggregations...) + s.filters = append([]QueryFilter(nil), s.filters...) + s.groupBy = append([]string(nil), s.groupBy...) + s.orderBy = append([]OrderClause(nil), s.orderBy...) + mutate(&s) + return &QueryBuilder{ctx: q.ctx, createStream: q.createStream, state: s} +} + +// Select appends columns to the projection. +func (q *QueryBuilder) Select(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.columns = append(s.columns, columns...) + }) +} + +// SelectAll requests every column the caller's role may read. +func (q *QueryBuilder) SelectAll() *QueryBuilder { + return q.clone(func(s *queryState) { + s.selectAll = true + }) +} + +// Where adds a filter condition. +func (q *QueryBuilder) Where(column string, op FilterOp, value any) *QueryBuilder { + wireOp, ok := opMap[op] + if !ok { + wireOp = string(op) + } + return q.clone(func(s *queryState) { + s.filters = append(s.filters, QueryFilter{Column: column, Op: wireOp, Value: value}) + }) +} + +// Count adds a COUNT aggregation. +func (q *QueryBuilder) Count(column, alias string) *QueryBuilder { + if column == "" { + column = "*" + } + if alias == "" { + alias = "count" + } + return q.addAgg("count", column, alias) +} + +// Sum adds a SUM aggregation. +func (q *QueryBuilder) Sum(column, alias string) *QueryBuilder { + if alias == "" { + alias = "sum_" + column + } + return q.addAgg("sum", column, alias) +} + +// Avg adds an AVG aggregation. +func (q *QueryBuilder) Avg(column, alias string) *QueryBuilder { + if alias == "" { + alias = "avg_" + column + } + return q.addAgg("avg", column, alias) +} + +// Min adds a MIN aggregation. +func (q *QueryBuilder) Min(column, alias string) *QueryBuilder { + if alias == "" { + alias = "min_" + column + } + return q.addAgg("min", column, alias) +} + +// Max adds a MAX aggregation. +func (q *QueryBuilder) Max(column, alias string) *QueryBuilder { + if alias == "" { + alias = "max_" + column + } + return q.addAgg("max", column, alias) +} + +// CountDistinct adds a COUNT DISTINCT aggregation. +func (q *QueryBuilder) CountDistinct(column, alias string) *QueryBuilder { + if alias == "" { + alias = "count_distinct_" + column + } + return q.addAgg("countDistinct", column, alias) +} + +// Aggregate adds a custom aggregation function. +func (q *QueryBuilder) Aggregate(fn, column, alias string) *QueryBuilder { + return q.addAgg(fn, column, alias) +} + +// GroupBy appends columns to the GROUP BY clause. +func (q *QueryBuilder) GroupBy(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.groupBy = append(s.groupBy, columns...) + }) +} + +// OrderBy appends an ORDER BY clause. dir defaults to "asc". +func (q *QueryBuilder) OrderBy(column, dir string) *QueryBuilder { + if dir == "" { + dir = "asc" + } + return q.clone(func(s *queryState) { + s.orderBy = append(s.orderBy, OrderClause{Column: column, Dir: dir}) + }) +} + +// Limit sets the maximum number of rows to return. +func (q *QueryBuilder) Limit(n int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.limit = &n + }) +} + +// TimeRange filters by a time window. since and until accept RFC3339 timestamps +// or relative durations ("1h", "30m", "7d", "2w"). +func (q *QueryBuilder) TimeRange(column, since, until string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.timeRange = &TimeRange{Column: column, Since: since, Until: until} + }) +} + +// CacheTTL records a desired result-cache TTL. Currently client-side only — +// the server derives TTLs adaptively (#280). +func (q *QueryBuilder) CacheTTL(seconds int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.cacheTTL = &seconds + }) +} + +// FetchTyped executes the query and decodes rows into []T. +func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], error) { + limit := DefaultLimit + if q.state.limit != nil { + limit = *q.state.limit + } + ast := q.buildAST(limit) + + var rows []Row + if err := doRequest(q.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/query", + params: url.Values{"table": {q.state.table}}, + body: ast, + }, &rows); err != nil { + return nil, err + } + + hasMore := limit > 0 && len(rows) >= limit + page := &Page[Row]{Data: rows, HasMore: hasMore} + + // Attach Next whenever we have an order column to build a cursor from. + // This doesn't check that the order column is present in the row + // projection — a Select() that omits it means fetchNextTyped can't find + // a cursor value and will quietly return an empty page (matches the TS + // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). + if hasMore && len(q.state.orderBy) > 0 { + page.Next = func(ctx context.Context) (*Page[Row], error) { + return fetchNextTyped[Row](ctx, q, rows, limit) + } + } + + return page, nil +} + +// FetchUntyped executes the query and returns rows as []map[string]any. +func (q *QueryBuilder) FetchUntyped(ctx context.Context) (*Page[map[string]any], error) { + return FetchTyped[map[string]any](ctx, q) +} + +// Stream opens a live SSE event stream for this query's table. +// Filters and column projections are applied client-side. +func (q *QueryBuilder) Stream(opts *StreamOptions) *StreamController { + raw := q.createStream(q.state.table, opts) + if len(q.state.filters) == 0 && len(q.state.columns) == 0 { + return raw + } + return newFilteredStreamController(raw, q.state.filters, q.state.columns) +} + +// LiveQuery starts a live query: fetches historical data, then streams live +// updates. The subscriber's Initial is called once, then Next fires for each +// live event. Returns a LiveQuery handle with a Close method. +func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *LiveQueryHandle { + stream := q.Stream(opts) + fetchFn := func(ctx context.Context) ([]map[string]any, error) { + page, err := q.FetchUntyped(ctx) + if err != nil { + return nil, err + } + return page.Data, nil + } + return newLiveQuery(stream, fetchFn, sub, q.state.filters) +} + +func (q *QueryBuilder) addAgg(fn, column, alias string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.aggregations = append(s.aggregations, Aggregation{Fn: fn, Column: column, Alias: alias}) + }) +} + +func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { + ast := &StructuredQuery{} + hasColumns := len(q.state.columns) > 0 + hasAggs := len(q.state.aggregations) > 0 + + // Projection: explicit select_all, then explicit columns, else — for a bare + // query with no projection and no aggregations — default to select_all so + // from(t).fetch() returns rows. + if q.state.selectAll { + ast.SelectAll = true + } else if hasColumns { + ast.Columns = q.state.columns + } else if !hasAggs { + ast.SelectAll = true + } + + if hasAggs { + ast.Aggregations = q.state.aggregations + } + if len(q.state.filters) > 0 { + ast.Filters = q.state.filters + } + if len(q.state.groupBy) > 0 { + ast.GroupBy = q.state.groupBy + } + if len(q.state.orderBy) > 0 { + ast.OrderBy = q.state.orderBy + } + ast.Limit = &effectiveLimit + if q.state.timeRange != nil { + ast.TimeRange = q.state.timeRange + } + return ast +} + +func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row, limit int) (*Page[Row], error) { + if len(q.state.orderBy) == 0 { + return &Page[Row]{}, nil + } + cursor := q.state.orderBy[0] + + // Extract the last row's value for the cursor column. + lastRow := any(prevRows[len(prevRows)-1]) + m, ok := lastRow.(map[string]any) + if !ok { + // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + raw, _ := json.Marshal(lastRow) + m = make(map[string]any) + _ = json.Unmarshal(raw, &m) + } + lastValue, exists := m[cursor.Column] + if !exists { + // Cursor column wasn't in the projection (e.g. Select() omitted it) — + // no cursor value to page from, so end pagination quietly rather than + // erroring. Matches the TS SDK's _fetchNext(). + return &Page[Row]{}, nil + } + + cursorOp := "gt" + if cursor.Dir == "desc" { + cursorOp = "lt" + } + + next := q.clone(func(s *queryState) { + s.filters = append(s.filters, QueryFilter{ + Column: cursor.Column, + Op: cursorOp, + Value: lastValue, + }) + }) + return FetchTyped[Row](ctx, next) +} diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go new file mode 100644 index 00000000..b2f560bb --- /dev/null +++ b/clients/go/query_builder_test.go @@ -0,0 +1,259 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func queryTestCtx(handler http.Handler) (*Client, *httptest.Server) { + srv := httptest.NewServer(handler) + c := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) + return c, srv +} + +func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { + t.Helper() + var body []byte + wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw := make([]byte, 32*1024) + n, _ := r.Body.Read(raw) + body = raw[:n] + handler.ServeHTTP(w, r) + }) + c, _ := queryTestCtx(wrapper) + return c, func() map[string]any { + var m map[string]any + json.Unmarshal(body, &m) + return m + } +} + +var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) +}) + +func TestQueryBuilder_Immutability(t *testing.T) { + c, _ := queryTestCtx(emptyRows) + b1 := c.From("clicks").Select("page") + b2 := b1.Where("score", OpGt, 10) + if b1 == b2 { + t.Fatal("builder should be immutable — chain methods return new instances") + } +} + +func TestQueryBuilder_SelectColumns(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page", "button").FetchUntyped(context.Background()) + + body := getBody() + cols, ok := body["columns"].([]any) + if !ok || len(cols) != 2 { + t.Fatalf("want [page, button], got %v", body["columns"]) + } +} + +func TestQueryBuilder_SelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").SelectAll().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("want select_all=true, got %v", body) + } +} + +func TestQueryBuilder_BareQueryDefaultsToSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("bare query should default to select_all, got %v", body) + } +} + +func TestQueryBuilder_AggregationOnlyNoSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().Count("*", "n").FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != nil { + t.Fatalf("aggregation-only query should not set select_all, got %v", body) + } +} + +func TestQueryBuilder_Where(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Where("score", OpGt, 10).FetchUntyped(context.Background()) + + body := getBody() + filters, ok := body["filters"].([]any) + if !ok || len(filters) != 1 { + t.Fatalf("want 1 filter, got %v", body["filters"]) + } + f := filters[0].(map[string]any) + if f["column"] != "score" || f["op"] != "gt" { + t.Fatalf("want score/gt filter, got %v", f) + } +} + +func TestQueryBuilder_AllOperators(t *testing.T) { + ops := []struct { + sdk FilterOp + wire string + }{ + {OpEq, "eq"}, + {OpNeq, "neq"}, + {OpGt, "gt"}, + {OpGte, "gte"}, + {OpLt, "lt"}, + {OpLte, "lte"}, + {OpIn, "in"}, + {OpLike, "like"}, + {OpNotLike, "not_like"}, + } + for _, tt := range ops { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) + body := getBody() + filters := body["filters"].([]any) + f := filters[0].(map[string]any) + if f["op"] != tt.wire { + t.Errorf("%s: want wire op %s, got %s", tt.sdk, tt.wire, f["op"]) + } + } +} + +func TestQueryBuilder_Aggregations(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select(). + Count("*", "total"). + Sum("score", ""). + Avg("score", ""). + Min("score", ""). + Max("score", ""). + CountDistinct("page", ""). + Aggregate("uniqExact", "user_id", "unique_users"). + FetchUntyped(context.Background()) + + body := getBody() + aggs, ok := body["aggregations"].([]any) + if !ok || len(aggs) != 7 { + t.Fatalf("want 7 aggregations, got %v", body["aggregations"]) + } +} + +func TestQueryBuilder_GroupBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").GroupBy("page").FetchUntyped(context.Background()) + + body := getBody() + gb, ok := body["group_by"].([]any) + if !ok || len(gb) != 1 || gb[0] != "page" { + t.Fatalf("want [page], got %v", body["group_by"]) + } +} + +func TestQueryBuilder_OrderBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").OrderBy("page", "desc").FetchUntyped(context.Background()) + + body := getBody() + ob := body["order_by"].([]any) + o := ob[0].(map[string]any) + if o["column"] != "page" || o["dir"] != "desc" { + t.Fatalf("want page/desc, got %v", o) + } +} + +func TestQueryBuilder_Limit(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Limit(50).FetchUntyped(context.Background()) + + body := getBody() + if body["limit"] != float64(50) { + t.Fatalf("want 50, got %v", body["limit"]) + } +} + +func TestQueryBuilder_TimeRange(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page"). + TimeRange("received_timestamp", "1h", ""). + FetchUntyped(context.Background()) + + body := getBody() + tr := body["time_range"].(map[string]any) + if tr["column"] != "received_timestamp" || tr["since"] != "1h" { + t.Fatalf("want received_timestamp/1h, got %v", tr) + } +} + +func TestQueryBuilder_Pagination_HasMore(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next == nil { + t.Fatal("want next function") + } +} + +func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) + + page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next != nil { + t.Fatal("want nil next — no order column for cursor") + } +} + +func TestQueryBuilder_ComplexQuery(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks"). + Select("page"). + Where("score", OpGt, 10). + Count("*", "total"). + GroupBy("page"). + OrderBy("total", "desc"). + Limit(50). + TimeRange("received_timestamp", "1h", ""). + CacheTTL(60). + FetchUntyped(context.Background()) + + body := getBody() + if body["columns"].([]any)[0] != "page" { + t.Fatal("missing page column") + } + if body["limit"] != float64(50) { + t.Fatal("wrong limit") + } + if body["group_by"].([]any)[0] != "page" { + t.Fatal("wrong group_by") + } +} diff --git a/clients/go/schema.go b/clients/go/schema.go new file mode 100644 index 00000000..67addc5b --- /dev/null +++ b/clients/go/schema.go @@ -0,0 +1,33 @@ +package wavehouse + +import "context" + +// SchemaNamespace provides admin-only schema introspection. +type SchemaNamespace struct { + ctx httpContext +} + +// List returns all table schemas discovered from ClickHouse. Admin-only. +func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { + // The backend returns []TableSchema; transform to map[string]TableSchema. + var raw []TableSchema + if err := doRequest(s.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/schema", + }, &raw); err != nil { + return nil, err + } + schemas := make(Schemas, len(raw)) + for _, t := range raw { + schemas[t.Name] = t + } + return schemas, nil +} + +// Refresh forces a schema re-discovery from ClickHouse. Admin-only. +func (s *SchemaNamespace) Refresh(ctx context.Context) error { + return doRequest(s.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/schema/refresh", + }, nil) +} diff --git a/clients/go/stream.go b/clients/go/stream.go new file mode 100644 index 00000000..e30d6346 --- /dev/null +++ b/clients/go/stream.go @@ -0,0 +1,561 @@ +package wavehouse + +import ( + "bufio" + "context" + "encoding/json" + "fmt" + "log" + "net/http" + "net/url" + "reflect" + "regexp" + "strings" + "sync" + "time" +) + +// StreamController manages a live SSE event stream. Use Subscribe for +// callback-based consumption or Events for channel-based consumption. +type StreamController struct { + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + cancel context.CancelFunc + done chan struct{} + closed bool +} + +// newStreamController opens an SSE connection for the given table. +func newStreamController(hctx httpContext, table string, opts *StreamOptions) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: StatusConnecting, + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + go sc.run(ctx, hctx, table, opts) + return sc +} + +// Status returns the current connection status. +func (sc *StreamController) Status() StreamStatus { + sc.mu.Lock() + defer sc.mu.Unlock() + return sc.status +} + +// Subscribe registers callbacks for stream events. Returns an unsubscribe +// function. The subscriber's Status callback fires immediately with the +// current status. +func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { + sc.mu.Lock() + sc.subscribers = append(sc.subscribers, sub) + currentStatus := sc.status + sc.mu.Unlock() + + // Benign race: if setStatus fires between the unlock above and the + // callback below, the subscriber may see a stale status here. This is + // harmless because setStatus also invokes the subscriber's callback, + // so the subscriber will receive the up-to-date status immediately + // after. Matches the TS SDK's registration behavior. + if sub.Status != nil { + sub.Status(currentStatus) + } + + return func() { + sc.mu.Lock() + defer sc.mu.Unlock() + for i, s := range sc.subscribers { + if s == sub { + sc.subscribers = append(sc.subscribers[:i], sc.subscribers[i+1:]...) + break + } + } + } +} + +// Events returns a read-only channel that receives stream events. +// The channel is closed when the stream closes. +func (sc *StreamController) Events() <-chan StreamEvent { + return sc.eventCh +} + +// Connected blocks until the stream reaches "live" status or the context +// expires. Returns an error if the stream closes before connecting. +func (sc *StreamController) Connected(ctx context.Context) error { + sc.mu.Lock() + if sc.status == StatusLive { + sc.mu.Unlock() + return nil + } + if sc.status == StatusClosed || sc.closed { + sc.mu.Unlock() + return fmt.Errorf("stream is closed") + } + sc.mu.Unlock() + + // Poll — simple and correct. + // ponytail: condition variable if polling shows up in profiles. + ticker := time.NewTicker(50 * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-sc.done: + return fmt.Errorf("stream closed before connecting") + case <-ticker.C: + sc.mu.Lock() + s := sc.status + sc.mu.Unlock() + if s == StatusLive { + return nil + } + if s == StatusClosed { + return fmt.Errorf("stream closed before connecting") + } + } + } +} + +// Close shuts down the stream and releases resources. Non-blocking so it is +// safe to call from subscriber callbacks (which run on the stream goroutine). +func (sc *StreamController) Close() { + sc.mu.Lock() + if sc.closed { + sc.mu.Unlock() + return + } + sc.closed = true + sc.mu.Unlock() + + sc.cancel() + // Don't block on <-sc.done: callbacks execute on the stream goroutine, + // so waiting here would deadlock if Close is called from a callback. +} + +func (sc *StreamController) setStatus(s StreamStatus) { + sc.mu.Lock() + if s == sc.status { + sc.mu.Unlock() + return + } + sc.status = s + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Status != nil { + sub.Status(s) + } + } +} + +func (sc *StreamController) emitEvent(event StreamEvent) { + sc.mu.Lock() + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Next != nil { + sub.Next(event) + } + } + + // Non-blocking send to the channel. + select { + case sc.eventCh <- event: + default: + log.Printf("[wavehouse] stream event dropped: channel buffer full") + } +} + +func (sc *StreamController) emitError(err error) { + sc.mu.Lock() + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Error != nil { + sub.Error(err) + } + } +} + +// run is the SSE connection loop with reconnect/backoff. +func (sc *StreamController) run(ctx context.Context, hctx httpContext, table string, opts *StreamOptions) { + defer func() { + sc.setStatus(StatusClosed) + close(sc.eventCh) + close(sc.done) + }() + + since := "" + if opts != nil { + since = opts.Since + } + + attempt := 0 + for { + if ctx.Err() != nil { + return + } + + lastID, err := sc.connect(ctx, hctx, table, since) + // Persist the last event ID so the next reconnect resumes from it. + if lastID != "" { + since = lastID + } + if ctx.Err() != nil { + return + } + + if err != nil { + sc.emitError(&Error{ + Status: 0, + Code: "SSE_ERROR", + Message: err.Error(), + Retryable: true, + }) + } + + sc.setStatus(StatusReconnecting) + delay := backoff(attempt) + attempt++ + + select { + case <-ctx.Done(): + return + case <-time.After(delay): + } + } +} + +// connect opens a single SSE connection and reads events until it closes. +// Returns the last seen event ID (empty if none) and any error. +func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, error) { + u, err := url.Parse(hctx.baseURL + "/v1/stream") + if err != nil { + return "", err + } + q := u.Query() + q.Set("table", table) + if since != "" { + q.Set("since", since) + } + + // Auth: Go SDK uses Authorization header (not ?token= like browser EventSource). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + return "", fmt.Errorf("auth: %w", err) + } + if token != "" { + authHeader = "Bearer " + token + } + } + + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) + if err != nil { + return "", err + } + req.Header.Set("Accept", "text/event-stream") + req.Header.Set("Cache-Control", "no-cache") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + resp, err := hctx.httpClient.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + } + + sc.setStatus(StatusLive) + + // Parse SSE frames. + scanner := bufio.NewScanner(resp.Body) + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) // 16 MiB max, matching server ingest cap + var eventID, dataLine string + lastID := since + + for scanner.Scan() { + if ctx.Err() != nil { + return lastID, nil + } + + line := scanner.Text() + + if line == "" { + // Empty line = end of event frame. + if dataLine != "" { + sc.handleSSEData(dataLine, eventID) + // Track last event ID for reconnect gap-fill. + if eventID != "" { + lastID = eventID + } + } + eventID = "" + dataLine = "" + continue + } + + if strings.HasPrefix(line, ":") { + // Comment (keepalive or connected). Skip. + continue + } + + if strings.HasPrefix(line, "id:") { + eventID = strings.TrimSpace(strings.TrimPrefix(line, "id:")) + } else if strings.HasPrefix(line, "data:") { + trimmed := strings.TrimSpace(strings.TrimPrefix(line, "data:")) + if dataLine == "" { + dataLine = trimmed + } else { + dataLine = dataLine + "\n" + trimmed + } + } + } + + return lastID, scanner.Err() +} + +// sseMessage matches the server's SSE event JSON shape. +type sseMessage struct { + TableName string `json:"table_name"` + ReceivedTimestamp string `json:"received_timestamp"` + Data map[string]any `json:"data"` +} + +func (sc *StreamController) handleSSEData(data, eventID string) { + var msg sseMessage + if err := json.Unmarshal([]byte(data), &msg); err != nil { + log.Printf("[wavehouse] SSE received malformed message: %s", data) + return + } + + event := StreamEvent{ + Table: msg.TableName, + Timestamp: msg.ReceivedTimestamp, + Data: msg.Data, + } + sc.emitEvent(event) +} + +// newFilteredStreamController wraps a StreamController with client-side +// filtering and column projection. +func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: inner.Status(), + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + + go func() { + defer func() { + sc.setStatus(StatusClosed) + close(sc.eventCh) + close(sc.done) + }() + + inner.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + if !matchesFilters(event.Data, filters) { + return + } + if len(columns) > 0 { + event.Data = projectColumns(event.Data, columns) + } + sc.emitEvent(event) + }, + Status: func(s StreamStatus) { + sc.setStatus(s) + }, + Error: func(err error) { + sc.emitError(err) + }, + }) + + select { + case <-ctx.Done(): + inner.Close() + case <-inner.done: + } + }() + + return sc +} + +// matchesFilters evaluates all filters against a data row (AND). +func matchesFilters(row map[string]any, filters []QueryFilter) bool { + for _, f := range filters { + val := row[f.Column] + if !evaluateFilter(val, f.Op, f.Value) { + return false + } + } + return true +} + +func evaluateFilter(actual any, op string, expected any) bool { + switch op { + case "eq": + return equalValues(actual, expected) + case "neq": + return !equalValues(actual, expected) + case "gt": + c, ok := compareOrdered(actual, expected) + return ok && c > 0 + case "gte": + c, ok := compareOrdered(actual, expected) + return ok && c >= 0 + case "lt": + c, ok := compareOrdered(actual, expected) + return ok && c < 0 + case "lte": + c, ok := compareOrdered(actual, expected) + return ok && c <= 0 + case "in": + return evaluateIn(actual, expected) + case "like": + aStr, aOK := actual.(string) + eStr, eOK := expected.(string) + if !aOK || !eOK { + return false + } + return matchLike(aStr, eStr) + case "not_like": + aStr, aOK := actual.(string) + eStr, eOK := expected.(string) + if !aOK || !eOK { + return false + } + return !matchLike(aStr, eStr) + default: + return false + } +} + +// equalValues compares two values for equality, normalizing numeric types +// (JSON decodes numbers as float64, but callers may pass int). +func equalValues(a, b any) bool { + if af, aOK := toFloat64(a); aOK { + if bf, bOK := toFloat64(b); bOK { + return af == bf + } + } + // fmt.Sprint is safe for all types (no panic on maps/slices). + return fmt.Sprint(a) == fmt.Sprint(b) +} + +// evaluateIn checks whether actual is contained in the expected slice. +// Handles both []any and typed slices (e.g., []string, []int). +func evaluateIn(actual, expected any) bool { + if arr, ok := expected.([]any); ok { + for _, v := range arr { + if equalValues(actual, v) { + return true + } + } + return false + } + // Handle typed slices via reflection. + rv := reflect.ValueOf(expected) + if rv.Kind() == reflect.Slice { + for i := range rv.Len() { + if equalValues(actual, rv.Index(i).Interface()) { + return true + } + } + } + return false +} + +var likeRegexCache sync.Map // pattern string → *regexp.Regexp + +// matchLike converts a SQL LIKE pattern to a regex and tests it +// (case-insensitive, matching the TS SDK). +func matchLike(actual, pattern string) bool { + if cached, ok := likeRegexCache.Load(pattern); ok { + return cached.(*regexp.Regexp).MatchString(actual) + } + escaped := regexp.QuoteMeta(pattern) + escaped = strings.ReplaceAll(escaped, "%", ".*") + escaped = strings.ReplaceAll(escaped, "_", ".") + re, err := regexp.Compile("(?i)^" + escaped + "$") + if err != nil { + return false + } + likeRegexCache.Store(pattern, re) + return re.MatchString(actual) +} + +// compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, +// or (0, false) when the types cannot be compared. +func compareOrdered(actual, expected any) (int, bool) { + if a, aOK := toFloat64(actual); aOK { + if b, bOK := toFloat64(expected); bOK { + switch { + case a < b: + return -1, true + case a > b: + return 1, true + default: + return 0, true + } + } + } + if aStr, ok := actual.(string); ok { + if bStr, ok := expected.(string); ok { + switch { + case aStr < bStr: + return -1, true + case aStr > bStr: + return 1, true + default: + return 0, true + } + } + } + return 0, false +} + +func toFloat64(v any) (float64, bool) { + switch n := v.(type) { + case float64: + return n, true + case float32: + return float64(n), true + case int: + return float64(n), true + case int64: + return float64(n), true + case json.Number: + f, err := n.Float64() + return f, err == nil + default: + return 0, false + } +} + +func projectColumns(row map[string]any, columns []string) map[string]any { + result := make(map[string]any, len(columns)) + for _, col := range columns { + if v, ok := row[col]; ok { + result[col] = v + } + } + return result +} diff --git a/clients/go/sys.go b/clients/go/sys.go new file mode 100644 index 00000000..4d2af556 --- /dev/null +++ b/clients/go/sys.go @@ -0,0 +1,17 @@ +package wavehouse + +import "context" + +// SysNamespace provides system health checks. +type SysNamespace struct { + ctx httpContext +} + +// Health pings the server's public /v1/health endpoint. Returns nil when the +// server is reachable and past boot, or an error describing the failure. +func (s *SysNamespace) Health(ctx context.Context) error { + return doRequest(s.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/health", + }, nil) +} diff --git a/clients/go/table.go b/clients/go/table.go new file mode 100644 index 00000000..adc8d829 --- /dev/null +++ b/clients/go/table.go @@ -0,0 +1,205 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/url" + "reflect" + "strings" +) + +// TableRef is a reference to a table. Use it for queries, inserts, schema, and +// streams. NOT safe to use concurrently from multiple goroutines for mutations; +// reads (Fetch, Select, etc.) are safe. +type TableRef struct { + ctx httpContext + table string + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch is a SELECT * shortcut with a default limit of 1000. +func (t *TableRef) Fetch(ctx context.Context) (*Page[map[string]any], error) { + return t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx) +} + +// Select starts building a typed query with the given column projection. +func (t *TableRef) Select(columns ...string) *QueryBuilder { + return &QueryBuilder{ + ctx: t.ctx, + createStream: t.createStream, + state: queryState{ + table: t.table, + columns: columns, + }, + } +} + +// SelectAll starts a query that selects every column the caller's role may read. +func (t *TableRef) SelectAll() *QueryBuilder { + return t.Select().SelectAll() +} + +// Insert inserts one or more rows into this table. A single map or struct is +// sent as JSON; any slice — []map[string]any, a generated/user-defined row +// type such as []ClickRow, etc. — is serialized to NDJSON for batch ingest. +func (t *TableRef) Insert(ctx context.Context, data any) (*InsertResult, error) { + if rows, ok := data.([]map[string]any); ok { + return t.insertBatch(ctx, rows) + } + if rv, ok := sliceValue(data); ok { + return t.insertBatchReflect(ctx, rv) + } + return t.insertSingle(ctx, data) +} + +// sliceValue reports whether data is a slice type, returning its +// reflect.Value for iteration. []byte is excluded and treated as an opaque +// single value (matching encoding/json's special-cased handling of byte +// slices) rather than a batch of numbers. +func sliceValue(data any) (reflect.Value, bool) { + if data == nil { + return reflect.Value{}, false + } + if _, isBytes := data.([]byte); isBytes { + return reflect.Value{}, false + } + v := reflect.ValueOf(data) + if v.Kind() != reflect.Slice { + return reflect.Value{}, false + } + return v, true +} + +// InsertNDJSON inserts pre-formatted NDJSON (one record per line). +func (t *TableRef) InsertNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + return t.sendNDJSON(ctx, ndjson) +} + +// Schema returns the table's column definitions from ClickHouse. Admin-only. +func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { + var schema TableSchema + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/schema", + params: url.Values{"table": {t.table}}, + }, &schema); err != nil { + return nil, err + } + return &schema, nil +} + +// Stream opens a live SSE event stream for this table. +func (t *TableRef) Stream(opts *StreamOptions) *StreamController { + return t.createStream(t.table, opts) +} + +func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, error) { + var res struct { + OK *bool `json:"ok"` + Duplicate *bool `json:"duplicate"` + } + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + body: data, + }, &res); err != nil { + return nil, err + } + ok := true + if res.OK != nil { + ok = *res.OK + } + result := &InsertResult{OK: ok} + if res.Duplicate != nil { + result.Duplicate = res.Duplicate + } + return result, nil +} + +func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { + if len(rows) == 0 { + zero := 0 + return &InsertResult{ + OK: true, + Total: &zero, + Succeeded: &zero, + Failed: &zero, + Duplicates: &zero, + }, nil + } + var sb strings.Builder + for i, row := range rows { + if i > 0 { + sb.WriteByte('\n') + } + raw, err := json.Marshal(row) + if err != nil { + return nil, err + } + sb.Write(raw) + } + return t.sendNDJSON(ctx, sb.String()) +} + +// insertBatchReflect is the fallback batch path for any slice type other +// than []map[string]any (the fast path in insertBatch above) — e.g. a +// generated or user-defined row type such as []ClickRow. Each element is +// marshaled to JSON individually and joined as NDJSON, exactly like +// insertBatch, so the server's per-record batch summary (failed, results, +// etc.) is preserved instead of being silently dropped by insertSingle. +func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { + n := rows.Len() + if n == 0 { + zero := 0 + return &InsertResult{ + OK: true, + Total: &zero, + Succeeded: &zero, + Failed: &zero, + Duplicates: &zero, + }, nil + } + var sb strings.Builder + for i := 0; i < n; i++ { + if i > 0 { + sb.WriteByte('\n') + } + raw, err := json.Marshal(rows.Index(i).Interface()) + if err != nil { + return nil, err + } + sb.Write(raw) + } + return t.sendNDJSON(ctx, sb.String()) +} + +func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + var res struct { + Total int `json:"total"` + Succeeded int `json:"succeeded"` + Failed int `json:"failed"` + Duplicates int `json:"duplicates"` + Results []InsertRecordResult `json:"results"` + } + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + rawBody: ndjson, + contentType: "application/x-ndjson", + }, &res); err != nil { + return nil, err + } + result := &InsertResult{ + OK: res.Failed == 0, + Total: &res.Total, + Succeeded: &res.Succeeded, + Failed: &res.Failed, + Duplicates: &res.Duplicates, + } + if len(res.Results) > 0 { + result.Results = res.Results + } + return result, nil +} diff --git a/clients/go/table_test.go b/clients/go/table_test.go new file mode 100644 index 00000000..39bb1efa --- /dev/null +++ b/clients/go/table_test.go @@ -0,0 +1,209 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" +) + +func TestTableRef_InsertSingle(t *testing.T) { + var gotBody map[string]any + var gotPath string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + json.NewDecoder(r.Body).Decode(&gotBody) + json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } + if gotBody["page"] != "/home" { + t.Fatalf("want page=/home, got %v", gotBody) + } +} + +func TestTableRef_InsertBatch(t *testing.T) { + var gotCT string + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ + {"page": "/a"}, + {"page": "/b"}, + }) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } +} + +// TestTableRef_InsertTypedSlice covers the P1 finding: a typed slice (e.g. a +// generated or user-defined row type such as []ClickRow) must take the batch +// NDJSON path — not fall through to insertSingle, which would send the slice +// as a single JSON body and silently ignore any per-record failures the +// server reports. +func TestTableRef_InsertTypedSlice(t *testing.T) { + type ClickRow struct { + Page string `json:"page"` + } + + var gotCT string + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ + {Page: "/a"}, + {Page: "/b"}, + }) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } + if result.OK { + t.Fatal("want ok=false when a batch record fails") + } + if result.Failed == nil || *result.Failed != 1 { + t.Fatalf("want failed=1, got %v", result.Failed) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } +} + +// TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through +// insertSingle rather than being (mis)treated as a slice of per-byte rows. +func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { + var gotPath string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } +} + +func TestTableRef_InsertEmptyBatch(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + t.Fatal("should not make a request for empty batch") + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if result.Total == nil || *result.Total != 0 { + t.Fatal("want total=0") + } +} + +func TestTableRef_InsertNDJSON(t *testing.T) { + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` + result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) + if err != nil { + t.Fatal(err) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } + if gotBody != ndjson { + t.Fatalf("want raw NDJSON, got %s", gotBody) + } +} + +func TestTableRef_Schema(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("table") != "clicks" { + t.Errorf("want table=clicks") + } + json.NewEncoder(w).Encode(TableSchema{ + Name: "clicks", + Columns: []Column{ + {Name: "page", Type: "String"}, + }, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + schema, err := c.From("clicks").Schema(context.Background()) + if err != nil { + t.Fatal(err) + } + if schema.Name != "clicks" { + t.Fatalf("want clicks, got %s", schema.Name) + } + if len(schema.Columns) != 1 || schema.Columns[0].Name != "page" { + t.Fatalf("unexpected columns: %v", schema.Columns) + } +} + +func TestTableRef_InsertDuplicate(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) + if err != nil { + t.Fatal(err) + } + if result.Duplicate == nil || !*result.Duplicate { + t.Fatal("want duplicate=true") + } +} diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json new file mode 100644 index 00000000..4ec8613d --- /dev/null +++ b/clients/go/testdata/wire_cases.json @@ -0,0 +1,547 @@ +[ + { + "name": "bare query defaults to select_all", + "endpoint": "query", + "table": "clicks", + "operations": [], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "select explicit columns", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page", "button"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page", "button"], + "limit": 1000 + } + }, + { + "name": "selectAll sends select_all flag", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "selectAll" } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "where with eq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "eq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with neq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "!=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "neq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with gt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with gte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">=", 10] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gte", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with lt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<", 5] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lt", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with lte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<=", 5] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lte", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with in operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "in", ["/home", "/about"]] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], + "limit": 1000 + } + }, + { + "name": "where with like operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "like", "/home%"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "like", "value": "/home%" }], + "limit": 1000 + } + }, + { + "name": "count aggregation", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "count", "args": ["*", "total"] } + ], + "expected_body": { + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "limit": 1000 + } + }, + { + "name": "sum aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "sum", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], + "limit": 1000 + } + }, + { + "name": "avg aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "avg", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], + "limit": 1000 + } + }, + { + "name": "min aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "min", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], + "limit": 1000 + } + }, + { + "name": "max aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "max", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], + "limit": 1000 + } + }, + { + "name": "countDistinct aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "countDistinct", "args": ["page", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], + "limit": 1000 + } + }, + { + "name": "custom aggregate function", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } + ], + "expected_body": { + "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], + "limit": 1000 + } + }, + { + "name": "groupBy", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "groupBy", "args": ["page"] } + ], + "expected_body": { + "columns": ["page"], + "group_by": ["page"], + "limit": 1000 + } + }, + { + "name": "orderBy ascending (default)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["page", "asc"] } + ], + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "page", "dir": "asc" }], + "limit": 1000 + } + }, + { + "name": "orderBy descending", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["score", "desc"] } + ], + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "score", "dir": "desc" }], + "limit": 1000 + } + }, + { + "name": "explicit limit", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [50] } + ], + "expected_body": { + "columns": ["page"], + "limit": 50 + } + }, + { + "name": "timeRange with since only", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_body": { + "columns": ["page"], + "time_range": { "column": "received_timestamp", "since": "1h" }, + "limit": 1000 + } + }, + { + "name": "timeRange with since and until", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } + ], + "expected_body": { + "columns": ["page"], + "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, + "limit": 1000 + } + }, + { + "name": "multiple where clauses (AND)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [ + { "column": "score", "op": "gt", "value": 10 }, + { "column": "page", "op": "eq", "value": "/home" } + ], + "limit": 1000 + } + }, + { + "name": "complex query combining everything", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "count", "args": ["*", "total"] }, + { "method": "groupBy", "args": ["page"] }, + { "method": "orderBy", "args": ["total", "desc"] }, + { "method": "limit", "args": [50] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "group_by": ["page"], + "order_by": [{ "column": "total", "dir": "desc" }], + "limit": 50, + "time_range": { "column": "received_timestamp", "since": "1h" } + } + }, + { + "name": "insert single row path", + "endpoint": "ingest", + "table": "clicks", + "operations": [ + { "method": "insert", "args": [{ "page": "/home", "button": "cta" }] } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { "page": "/home", "button": "cta" } + }, + { + "name": "insert batch as NDJSON", + "endpoint": "ingest_batch", + "table": "clicks", + "operations": [ + { + "method": "insert", + "args": [[{ "page": "/a" }, { "page": "/b" }]] + } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/x-ndjson", + "expected_raw_body": "{\"page\":\"/a\"}\n{\"page\":\"/b\"}" + }, + { + "name": "pipe execution", + "endpoint": "pipe", + "pipe_name": "top_pages", + "pipe_params": { "limit": 10 }, + "expected_path": "/v1/pipes/top_pages", + "expected_method": "POST", + "expected_body": { "limit": 10 } + }, + { + "name": "pipe execution with no params sends empty object", + "endpoint": "pipe", + "pipe_name": "simple", + "pipe_params": null, + "expected_path": "/v1/pipes/simple", + "expected_method": "POST", + "expected_body": {} + }, + { + "name": "raw SQL", + "endpoint": "sql", + "sql": "SELECT count() FROM clicks", + "expected_path": "/v1/admin/query", + "expected_method": "POST", + "expected_body": { "sql": "SELECT count() FROM clicks" } + }, + { + "name": "health check", + "endpoint": "health", + "expected_path": "/v1/health", + "expected_method": "GET" + }, + { + "name": "schema list", + "endpoint": "schema_list", + "expected_path": "/v1/schema", + "expected_method": "GET" + }, + { + "name": "schema refresh", + "endpoint": "schema_refresh", + "expected_path": "/v1/schema/refresh", + "expected_method": "POST" + }, + { + "name": "policy get", + "endpoint": "policy_get", + "expected_path": "/v1/admin/policy", + "expected_method": "GET" + }, + { + "name": "table with special characters URL-encodes correctly", + "endpoint": "query", + "table": "my table", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [10] } + ], + "expected_path": "/v1/query?table=my+table", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "limit": 10 + } + }, + { + "name": "DLQ list", + "endpoint": "dlq_list", + "expected_path": "/v1/dlq/stats", + "expected_method": "GET" + }, + { + "name": "DLQ table filter", + "endpoint": "dlq_table", + "table": "events", + "expected_path": "/v1/dlq/stats?table=events", + "expected_method": "GET" + }, + { + "name": "policy set", + "endpoint": "policy_set", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/admin/policy", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "policy validate", + "endpoint": "policy_validate", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/admin/policy/validate", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "pipes list", + "endpoint": "pipes_list", + "expected_path": "/v1/admin/pipes", + "expected_method": "GET" + }, + { + "name": "pipes get", + "endpoint": "pipes_get", + "pipe_name": "my_pipe", + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "GET" + }, + { + "name": "pipes set", + "endpoint": "pipes_set", + "pipe_name": "my_pipe", + "pipe_def": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + }, + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + } + }, + { + "name": "pipes delete", + "endpoint": "pipes_delete", + "pipe_name": "my_pipe", + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "DELETE" + } +] diff --git a/clients/go/types.go b/clients/go/types.go new file mode 100644 index 00000000..b59333fd --- /dev/null +++ b/clients/go/types.go @@ -0,0 +1,245 @@ +package wavehouse + +import "context" + +// ── Structured query AST (matches backend wire format) ──────────────────── + +// StructuredQuery is the wire format for POST /v1/query. +type StructuredQuery struct { + // Columns to project. A literal "*" is a column named "*", not a wildcard. + // Omitting columns (with no aggregations and no select_all) selects nothing. + Columns []string `json:"columns,omitempty"` + // SelectAll requests every column the caller's role may read. + // Mutually exclusive with a non-empty Columns list. + SelectAll bool `json:"select_all,omitempty"` + // Aggregations (count, sum, avg, etc.). + Aggregations []Aggregation `json:"aggregations,omitempty"` + // Filters (WHERE conditions, ANDed). + Filters []QueryFilter `json:"filters,omitempty"` + // GroupBy columns. + GroupBy []string `json:"group_by,omitempty"` + // OrderBy clauses. + OrderBy []OrderClause `json:"order_by,omitempty"` + // Limit caps the result set. + Limit *int `json:"limit,omitempty"` + // TimeRange filters by a time window. + TimeRange *TimeRange `json:"time_range,omitempty"` +} + +// Aggregation describes a single aggregation (e.g. count, sum). +type Aggregation struct { + Fn string `json:"fn"` + Column string `json:"column"` + Alias string `json:"alias"` +} + +// QueryFilter describes a single WHERE condition. +type QueryFilter struct { + Column string `json:"column"` + Op string `json:"op"` + Value any `json:"value"` +} + +// OrderClause describes a single ORDER BY clause. +type OrderClause struct { + Column string `json:"column"` + Dir string `json:"dir"` // "asc" or "desc" +} + +// TimeRange filters by a time window on a column. +type TimeRange struct { + Column string `json:"column"` + Since string `json:"since"` + Until string `json:"until,omitempty"` +} + +// FilterOp is an SDK-facing filter operator. +type FilterOp string + +const ( + OpEq FilterOp = "=" + OpNeq FilterOp = "!=" + OpGt FilterOp = ">" + OpGte FilterOp = ">=" + OpLt FilterOp = "<" + OpLte FilterOp = "<=" + OpIn FilterOp = "in" + OpLike FilterOp = "like" + OpNotLike FilterOp = "not_like" +) + +// opMap translates SDK operators to backend wire tokens. +var opMap = map[FilterOp]string{ + OpEq: "eq", + OpNeq: "neq", + OpGt: "gt", + OpGte: "gte", + OpLt: "lt", + OpLte: "lte", + OpIn: "in", + OpLike: "like", + OpNotLike: "not_like", +} + +// ── Schema types ────────────────────────────────────────────────────────── + +// Column describes a single column in a table schema. +type Column struct { + Name string `json:"name"` + Type string `json:"type"` + IsNullable bool `json:"is_nullable"` + HasDefault bool `json:"has_default"` +} + +// TableSchema describes a table's schema. +type TableSchema struct { + Name string `json:"name"` + Columns []Column `json:"columns"` +} + +// Schemas maps table names to their schemas. +type Schemas map[string]TableSchema + +// ── Insert result ───────────────────────────────────────────────────────── + +// InsertRecordResult is a per-record outcome from a batch insert. +type InsertRecordResult struct { + Index int `json:"index"` + OK *bool `json:"ok,omitempty"` + Duplicate *bool `json:"duplicate,omitempty"` + Error string `json:"error,omitempty"` +} + +// InsertResult is the outcome of an insert operation. +type InsertResult struct { + OK bool `json:"ok"` + Duplicate *bool `json:"duplicate,omitempty"` + Total *int `json:"total,omitempty"` + Succeeded *int `json:"succeeded,omitempty"` + Failed *int `json:"failed,omitempty"` + Duplicates *int `json:"duplicates,omitempty"` + Results []InsertRecordResult `json:"results,omitempty"` +} + +// ── DLQ types ───────────────────────────────────────────────────────────── + +// DLQStats describes dead-letter-queue statistics. +type DLQStats struct { + Tables map[string]int `json:"tables"` + Total int `json:"total"` +} + +// ── Pipe types ──────────────────────────────────────────────────────────── + +// Pipe describes a named query pipe definition. +type Pipe struct { + Name string `json:"name"` + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// ParamDef describes a pipe parameter. +type ParamDef struct { + Name string `json:"name"` + Type string `json:"type"` + Required bool `json:"required,omitempty"` + Default any `json:"default,omitempty"` +} + +// ── Policy types ────────────────────────────────────────────────────────── + +// Policy describes the server's access-control policy. +type Policy struct { + DefaultRole string `json:"default_role,omitempty"` + // AdminRole is the role granted full access and the allowlist bypass. + // Empty means the server's default ("admin") applies. + AdminRole string `json:"admin_role,omitempty"` + Tables map[string]TablePolicy `json:"tables"` +} + +// TablePolicy describes per-table access control. +type TablePolicy struct { + Select map[string]RolePermissions `json:"select,omitempty"` + Insert map[string]RolePermissions `json:"insert,omitempty"` +} + +// RolePermissions describes a role's access to a table. +type RolePermissions struct { + AllowColumns []string `json:"allow_columns,omitempty"` + DenyColumns []string `json:"deny_columns,omitempty"` + Filter map[string]PolicyFilter `json:"filter,omitempty"` + Check map[string]PolicyFilter `json:"check,omitempty"` + AllowedAggregations []string `json:"allowed_aggregations,omitempty"` + DeniedAggregations []string `json:"denied_aggregations,omitempty"` + MaxRows *int `json:"max_rows,omitempty"` + MaxExecutionTime any `json:"max_execution_time,omitempty"` + MaxRowsToRead *int `json:"max_rows_to_read,omitempty"` + MaxMemoryUsage any `json:"max_memory_usage,omitempty"` +} + +// PolicyFilter describes a policy filter predicate. Fields are pointers so an +// intentional empty-string comparison (e.g. Eq pointing at "") round-trips +// distinctly from an absent operator, matching the server's semantics. +type PolicyFilter struct { + Eq *string `json:"_eq"` + Neq *string `json:"_neq"` + Gt *string `json:"_gt"` + Lt *string `json:"_lt"` + In *string `json:"_in"` +} + +// ValidationResult is the response from policy validation. +type ValidationResult struct { + Valid bool `json:"valid"` +} + +// ── Streaming types ─────────────────────────────────────────────────────── + +// StreamStatus represents the connection state of a stream. +type StreamStatus string + +const ( + StatusConnecting StreamStatus = "connecting" + StatusLive StreamStatus = "live" + StatusReconnecting StreamStatus = "reconnecting" + StatusClosed StreamStatus = "closed" +) + +// StreamEvent is a single event from an SSE stream. +type StreamEvent struct { + Table string `json:"table"` + Timestamp string `json:"timestamp"` + Data map[string]any `json:"data"` +} + +// StreamSubscriber receives events from a stream. +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} + +// StreamOptions configures a stream. +type StreamOptions struct { + // Since is an RFC3339 timestamp for gap-fill replay. + Since string +} + +// ── Fetch/page types ────────────────────────────────────────────────────── + +// Page wraps a result set with pagination metadata. +type Page[T any] struct { + // Data is the result rows. + Data []T + // HasMore is true if more rows may be available. + HasMore bool + // Next fetches the next page. Nil when no cursor is available. + Next func(ctx context.Context) (*Page[T], error) +} diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go new file mode 100644 index 00000000..6bcb8e5f --- /dev/null +++ b/clients/go/wavehouse.go @@ -0,0 +1,142 @@ +// Package wavehouse is the official Go SDK for WaveHouse — a schema-aware +// real-time API gateway for ClickHouse. Zero third-party runtime dependencies. +// +// Create a client with [NewClient], then use [Client.From] for table +// operations, [Client.Pipe] for named queries, or the admin namespaces +// ([Client.Schema], [Client.Policy], etc.) for management. +// +// client := wavehouse.NewClient(wavehouse.Config{ +// BaseURL: "http://localhost:8080", +// }) +// rows, err := client.From("clicks").SelectAll().Fetch(ctx) +package wavehouse + +import ( + "context" + "net/http" +) + +// Config configures a [Client]. +type Config struct { + // BaseURL of the WaveHouse server (e.g. "http://localhost:8080"). + BaseURL string + + // Auth provides a bearer token for authenticated requests. Called before + // each request; return "" to skip the Authorization header. Nil means + // unauthenticated access (the server falls back to default_role). + Auth func(ctx context.Context) (string, error) + + // Options tunes transport behavior. + Options *ClientOptions + + // HTTPClient overrides the default http.Client. Useful for custom TLS, + // proxies, or test transports. + HTTPClient *http.Client +} + +// ClientOptions tunes transport behavior. +type ClientOptions struct { + // MaxRetries is the maximum number of retry attempts for retryable errors. + // Total attempts = MaxRetries + 1. Default: 2. + MaxRetries int +} + +// StaticToken returns an Auth function that always returns the same token. +// Convenience for cases where the token doesn't rotate. +func StaticToken(token string) func(context.Context) (string, error) { + return func(context.Context) (string, error) { return token, nil } +} + +// Client is the WaveHouse SDK entry point. +type Client struct { + ctx httpContext + + // Schema provides admin-only schema introspection. + Schema *SchemaNamespace + // Policy provides admin-only access-control policy management. + Policy *PolicyNamespace + // DLQ provides admin-only dead-letter-queue statistics. + DLQ *DLQNamespace + // Sys provides system health checks. + Sys *SysNamespace + // Pipes provides admin-only named-pipe management. + Pipes *PipesNamespace +} + +// NewClient creates a new WaveHouse client. +func NewClient(cfg Config) *Client { + maxRetries := 2 + if cfg.Options != nil && cfg.Options.MaxRetries >= 0 { + maxRetries = cfg.Options.MaxRetries + } + + hc := cfg.HTTPClient + if hc == nil { + hc = http.DefaultClient + } + + c := &Client{ + ctx: httpContext{ + baseURL: trimTrailingSlashes(cfg.BaseURL), + auth: cfg.Auth, + maxRetries: maxRetries, + httpClient: hc, + }, + } + + c.Schema = &SchemaNamespace{ctx: c.ctx} + c.Policy = &PolicyNamespace{ctx: c.ctx} + c.DLQ = &DLQNamespace{ctx: c.ctx, createStream: c.createStream} + c.Sys = &SysNamespace{ctx: c.ctx} + c.Pipes = &PipesNamespace{ctx: c.ctx} + + return c +} + +// From returns a reference to a table for queries, inserts, and streams. +func (c *Client) From(table string) *TableRef { + return &TableRef{ + ctx: c.ctx, + table: table, + createStream: c.createStream, + } +} + +// Pipe returns a reference to a named query pipe. Pass params for the pipe's +// template parameters. +func (c *Client) Pipe(name string, params map[string]any) *PipeRef { + return &PipeRef{ + ctx: c.ctx, + name: name, + params: params, + createStream: c.createStream, + } +} + +// SQL executes a raw SQL query against ClickHouse. Requires the admin role. +// The server proxies the SQL verbatim to ClickHouse's HTTP interface. Results +// are decoded into []T; use [map[string]any] for dynamic schemas. +func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { + var rows []Row + err := doRequest(c.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/admin/query", + body: map[string]string{"sql": query}, + }, &rows) + if err != nil { + return nil, err + } + return rows, nil +} + +// createStream opens an SSE stream for the given table. +func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { + return newStreamController(c.ctx, table, opts) +} + +func trimTrailingSlashes(s string) string { + for len(s) > 0 && s[len(s)-1] == '/' { + s = s[:len(s)-1] + } + return s +} diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs new file mode 100644 index 00000000..8eafc796 --- /dev/null +++ b/tests/conformance/conformance_ts.mjs @@ -0,0 +1,279 @@ +#!/usr/bin/env node +/** + * Cross-language wire-format conformance test for the TypeScript SDK. + * + * Reads wire_cases.json (owned by the Go module, at clients/go/testdata/) + * and verifies the TS SDK produces identical HTTP requests (method, path, + * content-type, body) to the shared fixture. + * + * Run: node tests/conformance/conformance_ts.mjs + * Exit 0 = all pass, exit 1 = failures. + */ + +import { readFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +// Import the built SDK. +const { createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js")); + +const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); + +let lastCapture = { method: "", path: "", contentType: "", body: "" }; + +function resetCapture() { + lastCapture = { method: "", path: "", contentType: "", body: "" }; +} + +// Start echo server. +const server = createServer((req, res) => { + const chunks = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + lastCapture = { + method: req.method ?? "", + path: req.url ?? "", + contentType: req.headers["content-type"] ?? "", + body: Buffer.concat(chunks).toString("utf-8"), + }; + res.setHeader("Content-Type", "application/json"); + if (req.url?.startsWith("/v1/dlq")) { + res.end(JSON.stringify({ tables: {}, total: 0 })); + } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { + res.end(JSON.stringify({})); + } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { + res.end(JSON.stringify({ valid: true })); + } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { + res.end(JSON.stringify({ tables: {} })); + } else if (req.url?.startsWith("/v1/admin/pipes/") && req.method === "GET") { + res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); + } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { + res.end(JSON.stringify([])); + } else if (req.url?.startsWith("/v1/ingest")) { + if (lastCapture.contentType === "application/x-ndjson") { + res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); + } else { + res.end(JSON.stringify({ ok: true })); + } + } else if (req.url === "/v1/health") { + res.end(""); + } else { + res.end(JSON.stringify([])); + } + }); +}); + +await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); +const { port } = server.address(); +const baseURL = `http://127.0.0.1:${port}`; + +function applyQueryOps(wh, table, operations) { + let q = wh.from(table).select(); + for (const op of operations) { + switch (op.method) { + case "select": + q = wh.from(table).select(...op.args); + break; + case "selectAll": + q = q.selectAll(); + break; + case "where": + q = q.where(op.args[0], op.args[1], op.args[2]); + break; + case "count": + q = q.count(op.args[0] || "*", op.args[1] || "count"); + break; + case "sum": + q = q.sum(op.args[0], op.args[1] || undefined); + break; + case "avg": + q = q.avg(op.args[0], op.args[1] || undefined); + break; + case "min": + q = q.min(op.args[0], op.args[1] || undefined); + break; + case "max": + q = q.max(op.args[0], op.args[1] || undefined); + break; + case "countDistinct": + q = q.countDistinct(op.args[0], op.args[1] || undefined); + break; + case "aggregate": + q = q.aggregate(op.args[0], op.args[1], op.args[2]); + break; + case "groupBy": + q = q.groupBy(...op.args); + break; + case "orderBy": + q = q.orderBy(op.args[0], op.args[1] || "asc"); + break; + case "limit": + q = q.limit(op.args[0]); + break; + case "timeRange": + q = q.timeRange(op.args[0], op.args[1], op.args[2] || undefined); + break; + case "cacheTTL": + q = q.cacheTTL(op.args[0]); + break; + } + } + return q; +} + +function normalizePath(p) { + return p.replace(/\+/g, "%20"); +} + +function deepEqual(a, b) { + return JSON.stringify(sortKeys(a)) === JSON.stringify(sortKeys(b)); +} + +function sortKeys(v) { + if (v === null || v === undefined) return v; + if (Array.isArray(v)) return v.map(sortKeys); + if (typeof v === "object") { + const sorted = {}; + for (const k of Object.keys(v).sort()) { + sorted[k] = sortKeys(v[k]); + } + return sorted; + } + return v; +} + +let passed = 0; +let failed = 0; +const failures = []; + +for (const tc of cases) { + resetCapture(); + const wh = createClient({ baseURL, options: { maxRetries: 0 } }); + + try { + switch (tc.endpoint) { + case "query": { + const q = applyQueryOps(wh, tc.table, tc.operations ?? []); + await q.fetch(); + break; + } + case "ingest": + if (tc.operations?.[0]?.method === "insert") { + await wh.from(tc.table).insert(tc.operations[0].args[0]); + } + break; + case "ingest_batch": + if (tc.operations?.[0]?.method === "insert") { + await wh.from(tc.table).insert(tc.operations[0].args[0]); + } + break; + case "pipe": + await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); + break; + case "sql": + await wh.sql(tc.sql); + break; + case "health": + await wh.sys.health(); + break; + case "schema_list": + await wh.schema.list(); + break; + case "schema_refresh": + await wh.schema.refresh(); + break; + case "policy_get": + await wh.policy.get(); + break; + case "policy_set": + await wh.policy.set(tc.policy_body); + break; + case "policy_validate": + await wh.policy.validate(tc.policy_body); + break; + case "dlq_list": + await wh.dlq.list(); + break; + case "dlq_table": + await wh.dlq.table(tc.table); + break; + case "pipes_list": + await wh.pipes.list(); + break; + case "pipes_get": + await wh.pipes.get(tc.pipe_name); + break; + case "pipes_set": + await wh.pipes.set(tc.pipe_name, tc.pipe_def); + break; + case "pipes_delete": + await wh.pipes.delete(tc.pipe_name); + break; + default: + passed++; + continue; + } + + const errs = []; + + if (tc.expected_method && lastCapture.method !== tc.expected_method) { + errs.push(`method: want ${tc.expected_method}, got ${lastCapture.method}`); + } + + if (tc.expected_path && normalizePath(lastCapture.path) !== normalizePath(tc.expected_path)) { + errs.push(`path: want ${tc.expected_path}, got ${lastCapture.path}`); + } + + if (tc.expected_content_type && lastCapture.contentType !== tc.expected_content_type) { + errs.push(`content-type: want ${tc.expected_content_type}, got ${lastCapture.contentType}`); + } + + if (tc.expected_raw_body !== undefined) { + if (lastCapture.body !== tc.expected_raw_body) { + errs.push(`raw body:\n want: ${tc.expected_raw_body}\n got: ${lastCapture.body}`); + } + } else if (tc.expected_body !== undefined && tc.expected_body !== null) { + let captured; + try { + captured = JSON.parse(lastCapture.body); + } catch { + errs.push(`body not valid JSON: ${lastCapture.body}`); + } + if (captured !== undefined && !deepEqual(captured, tc.expected_body)) { + errs.push( + `body mismatch:\n want: ${JSON.stringify(tc.expected_body)}\n got: ${JSON.stringify(captured)}`, + ); + } + } + + if (errs.length > 0) { + failed++; + failures.push({ name: tc.name, errors: errs }); + } else { + passed++; + } + } catch (err) { + failed++; + failures.push({ name: tc.name, errors: [`exception: ${err.message}`] }); + } +} + +server.close(); + +console.log(`\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${cases.length} total\n`); + +for (const f of failures) { + console.log(` ✗ ${f.name}`); + for (const e of f.errors) { + console.log(` ${e}`); + } +} + +if (failed > 0) { + process.exit(1); +} else { + console.log(" ✓ All cases passed\n"); +} From 3089c719c72fb5045d7bef77c62bd72b0353bfb1 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:33 -0400 Subject: [PATCH 02/59] docs(sdk): add Go SDK documentation pages Six Starlight pages covering installation, queries, streaming, pipes, admin operations, and API reference. Sidebar nav group added. Cross-link from SDK index page. Architecture page updated with Go SDK. Root README updated with Go SDK install. --- README.md | 1 + docs/src/config/sidebar.ts | 24 +- docs/src/content/docs/architecture.md | 1 + docs/src/content/docs/sdk/go/admin.md | 118 +++++++ docs/src/content/docs/sdk/go/index.md | 248 ++++++++++++++ docs/src/content/docs/sdk/go/pipes.md | 100 ++++++ docs/src/content/docs/sdk/go/queries.md | 390 ++++++++++++++++++++++ docs/src/content/docs/sdk/go/reference.md | 233 +++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 254 ++++++++++++++ docs/src/content/docs/sdk/index.mdx | 21 ++ 10 files changed, 1386 insertions(+), 4 deletions(-) create mode 100644 docs/src/content/docs/sdk/go/admin.md create mode 100644 docs/src/content/docs/sdk/go/index.md create mode 100644 docs/src/content/docs/sdk/go/pipes.md create mode 100644 docs/src/content/docs/sdk/go/queries.md create mode 100644 docs/src/content/docs/sdk/go/reference.md create mode 100644 docs/src/content/docs/sdk/go/streaming.md diff --git a/README.md b/README.md index 966bc683..d9e658c4 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. - **Client** — `@wavehouse/sdk`: zero-dependency TypeScript client with query builder, live queries, streaming, and schema codegen. +- **Go SDK** — `go get github.com/Wave-RF/WaveHouse/clients/go`: full API-tree parity with the TS SDK — ingest, query, streaming, and policy management from Go. ## 📊 How it compares diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index b3a61a20..f114c692 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,10 +25,15 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Topic-first SDK pages: when a second SDK language lands, these - // shared usage pages grow code tabs and each - // language gets its own setup/caveats page — the topic URLs never - // churn (decision in PR #313). + // Topic-first SDK pages: the multi-language plan on record (PR #313) + // was for a second language to grow on these + // shared pages instead of a parallel tree. The Go SDK launched as + // its own docs/src/content/docs/sdk/go/* tree instead — its API + // shape (context.Context, (T, error), generics on package-level + // funcs) diverges enough from the TS builder that shared prose read + // worse than dedicated pages. Revisit folding these into tabs if a + // third language lands and the duplication becomes a maintenance + // cost. label: "TypeScript SDK", items: [ { label: "Overview", slug: "sdk" }, @@ -39,6 +44,17 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ { label: "Reference & CLI", slug: "sdk/reference" }, ], }, + { + label: "Go SDK", + items: [ + { label: "Overview", slug: "sdk/go" }, + { label: "Queries", slug: "sdk/go/queries" }, + { label: "Streaming & Live Queries", slug: "sdk/go/streaming" }, + { label: "Pipes", slug: "sdk/go/pipes" }, + { label: "Admin & System", slug: "sdk/go/admin" }, + { label: "Reference & CLI", slug: "sdk/go/reference" }, + ], + }, ], }, { diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4c6a8958..8b8b6ff2 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -276,4 +276,5 @@ Client GET /v1/stream | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | +| Client SDKs | TypeScript, Go | Typed clients with the same feature set (ingest, query, pipes, streaming, admin) | | Containers | Docker (distroless) | Minimal production images | diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md new file mode 100644 index 00000000..a0941433 --- /dev/null +++ b/docs/src/content/docs/sdk/go/admin.md @@ -0,0 +1,118 @@ +--- +title: "Go SDK Admin & System" +description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." +--- + +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. +Everything here except `client.Sys.Health` requires the admin role +(`policy.admin_role`) — see [Access Control](/access-control) for how roles +resolve. Compare with the TypeScript SDK's [Admin & System](/sdk/admin) +page. + +## Schema — `client.Schema` + +Introspect ClickHouse table schemas. + +```go +// List all table schemas. +schemas, err := wh.Schema.List(ctx) +// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name + +// Force refresh from ClickHouse. +err = wh.Schema.Refresh(ctx) +``` + +Individual table schema is also available via `wh.From("clicks").Schema(ctx)`. + +> `wh.Schema.List`, `wh.Schema.Refresh`, and `wh.From(t).Schema` hit +> `/v1/schema*`, which are **admin-only** endpoints. Against any non-dev +> policy (anything but `default_role: admin`), construct the client with an +> admin-role token or these calls return a `*wavehouse.Error` with +> `Status: 403`. + +--- + +## Policy — `client.Policy` + +Manage Hasura-style access control policies. Requires the admin role +(`policy.admin_role`). + +```go +// Get current policy. +policy, err := wh.Policy.Get(ctx) + +// Update policy. +tenantFilter := "{{ jwt.app_metadata.tenant_id }}" +err = wh.Policy.Set(ctx, &wavehouse.Policy{ + DefaultRole: "viewer", + Tables: map[string]wavehouse.TablePolicy{ + "clicks": { + Select: map[string]wavehouse.RolePermissions{ + "viewer": { + AllowColumns: []string{"page", "button", "received_timestamp"}, + Filter: map[string]wavehouse.PolicyFilter{ + "tenant_id": {Eq: &tenantFilter}, + }, + }, + "admin": {AllowColumns: []string{"*"}}, + }, + }, + }, +}) + +// Validate without applying (dry run). +result, err := wh.Policy.Validate(ctx, policyDraft) +// result.Valid == true, or err wraps the validation failure details +``` + +`PolicyFilter`'s fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, not +`string` — an intentional empty-string comparison round-trips distinctly +from an absent operator. Take the address of a local variable (as above) or +write a small helper if you find yourself doing this often: + +```go +func strPtr(s string) *string { return &s } +``` + +--- + +## DLQ — `client.DLQ` + +Dead Letter Queue operations. Requires the admin role (`policy.admin_role`). + +```go +// Get DLQ statistics. +stats, err := wh.DLQ.List(ctx) +// stats.Tables: map[string]int{"clicks": 3, "users": 0} +// stats.Total: 3 + +// Stats for a specific table. +stats, err = wh.DLQ.Table(ctx, "clicks") +``` + +`wh.DLQ.Stream(opts)` exists in the API but is **not yet functional**: +there is no server-side DLQ stream today (the SSE bridge only carries +`ingest.>` subjects), so it connects and receives no events — live DLQ +streaming is tracked in +[#197](https://github.com/Wave-RF/WaveHouse/issues/197). + +--- + +## System — `client.Sys` + +Content-free server-online check. + +```go +// Health hits the public, content-free /v1/health route — 200 → nil error, +// any other status (including 503) → a non-nil *wavehouse.Error. +// Use it to check a server is reachable before sending data. +if err := wh.Sys.Health(ctx); err != nil { + // server is unreachable or not yet past boot + log.Println(err) +} +``` + +> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — +> it runs a ClickHouse query per call and is a load-balancer / reverse-proxy +> concern, not the client's. Probe `/readyz` directly from your +> orchestrator if you need it. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md new file mode 100644 index 00000000..db515ef2 --- /dev/null +++ b/docs/src/content/docs/sdk/go/index.md @@ -0,0 +1,248 @@ +--- +title: "Go SDK" +description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." +--- + +`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime +dependency Go client for WaveHouse (stdlib only). + +:::tip[Looking for the TypeScript SDK?] +This page and the rest of `/sdk/go/*` cover the Go client. The +JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at +[SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything +you learn about WaveHouse's query builder, streaming, or admin endpoints on +either page mostly carries over. +::: + +## Installation + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +Requires Go 1.26.5 or later (the minimum pinned in the module's `go.mod`). + +## Import + +```go +import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +``` + +The package name is `wavehouse`; aliasing the import isn't required, but +keeps call sites short (`wavehouse.NewClient(...)`, `wavehouse.OpEq`, ...) — +every example on these pages assumes it. + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + ctx := context.Background() + + // Create a client. Auth is optional — omit it for public/unauthenticated + // access (the server falls back to policy.default_role). + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), + }) + + // Health check. + if err := wh.Sys.Health(ctx); err != nil { + log.Fatal(err) + } + + // Insert a row. + if _, err := wh.From("clicks").Insert(ctx, map[string]any{ + "page": "/home", "button": "signup", + }); err != nil { + log.Fatal(err) + } + + // Query with the fluent builder. + page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } + + // Stream. + stream := wh.From("clicks").Stream(nil) + defer stream.Close() + unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, + Status: func(s wavehouse.StreamStatus) { fmt.Println("Stream:", s) }, + }) + defer unsub() +} +``` + +## Creating a Client + +```go +import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" + +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "https://wavehouse.example.com", + Auth: func(ctx context.Context) (string, error) { + return myAuthProvider.GetToken(ctx) + }, + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, + }, +}) +``` + +### `Config` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `BaseURL` | `string` | — | WaveHouse server URL (required) | +| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | +| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | +| `HTTPClient` | `*http.Client` | `http.DefaultClient` | Override for custom TLS, proxies, or test transports | + +### `ClientOptions` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | + +:::caution[`Options` opts you out of the default, not just in] +The default of 2 retries only applies when `Config.Options` is `nil`. If you +set `Options` to configure anything else in the future, an unset +`MaxRetries` field is Go's int zero value — `0` — which is a **valid, +explicit** "no retries" setting, not "use the default." Today `MaxRetries` +is the struct's only field, so this mostly matters if you pass +`&wavehouse.ClientOptions{}` and expect retry-by-default: you won't get it. +::: + +For a static token that never rotates, use `wavehouse.StaticToken(token)` +instead of writing the closure yourself: + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) +``` + +:::note[How the token is transmitted] +Unlike a browser's `EventSource`, Go's `net/http` client can set arbitrary +headers on any request — so the Go SDK sends `Authorization: Bearer ` +on **every** request, including SSE streams. There's no `?token=` query +parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser +concern only; see its [equivalent note](/sdk#creating-a-client)). +::: + +## Typed Rows (Generics) + +Pass a row type as a type parameter to get results decoded straight into +your struct, instead of `map[string]any`: + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + wh.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +Generate row structs from a running server with the +[codegen CLI](/sdk/go/reference#codegen-cli). + +`FetchTyped` is a package-level generic function, not a method — Go doesn't +support generic methods, so this (and `Fetch[Row]` for pipes, and +`SQL[Row]` for raw SQL) are top-level functions that take the client or +builder as an argument. Untyped equivalents (`.FetchUntyped(ctx)`, decoding +into `map[string]any`) are ordinary methods, since they need no type +parameter. + +## Error Handling + +Every SDK operation returns `(T, error)` — the idiomatic Go shape, and the +direct equivalent of the TypeScript SDK's +[`Result`](/sdk#result-type) discriminated union. Errors are always +`*wavehouse.Error`; unwrap with `errors.As`: + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } + return err +} +``` + +```go +type Error struct { + Status int // HTTP status (0 for network/abort errors) + Code string // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" + Message string // Human-readable error message + Details map[string]any // Parsed response body, if available + Retryable bool // Whether the SDK would retry this error +} +``` + +`wavehouse.IsRetryable(err)` is a shortcut for `errors.As` + `.Retryable`. +The full error-code table lives in +[Reference → Error Handling](/sdk/go/reference#error-handling). + +## Differences from the TypeScript SDK + +The two SDKs share a wire format and mirror each other's feature set closely +(a shared `testdata/wire_cases.json` conformance fixture in the repo asserts +both produce identical HTTP requests for equivalent builder calls), but the +languages pull the API shape in different directions: + +- **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in + an `{ok, data, error}` object, and there's no `error: null` sentinel to + check — a non-nil `error` is the only signal. +- **`context.Context` instead of `AbortSignal`.** Every non-streaming call + takes a `ctx context.Context` as its first argument; cancel it (timeout or + `cancel()`) instead of building an `AbortController`. See + [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams are closed explicitly, not via `ctx`.** `TableRef.Stream` / + `QueryBuilder.Stream` don't take a `context.Context` — the returned + `*StreamController` manages its own background goroutine and connection, + torn down by calling `.Close()` (deferred `stream.Close()` is the usual + pattern). See [Streaming](/sdk/go/streaming). +- **Generics live on package-level functions, not methods** (`FetchTyped[Row]`, + `Fetch[Row]`, `SQL[Row]`), because Go doesn't support type parameters on + methods. +- **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call + `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` + explicitly; there's no bare `await builder` shortcut. +- **`Insert` accepts typed row slices, not just maps.** Passing + `[]ClickRow{...}` (any slice type, detected via reflection) batches as + NDJSON exactly like `[]map[string]any` — see + [Queries → Insert](/sdk/go/queries#insertctx-data). + +## Explore the Go SDK + +- [Queries](/sdk/go/queries) — Tables, the chainable query builder, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/go/streaming) — Real-time SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/go/pipes) — Execute and manage named query pipes. +- [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. +- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, the full API tree, and the codegen CLI. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md new file mode 100644 index 00000000..44786601 --- /dev/null +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -0,0 +1,100 @@ +--- +title: "Go SDK Pipes" +description: "Execute and manage named query pipes with the WaveHouse Go SDK." +--- + +Named pipes are server-defined, parameterized queries — the +[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for +any allowed role and manages their definitions under the admin role. +Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. + +## Named Pipes — `client.Pipe(name, params)` + +Execute a pre-defined named query pipe. Returns a `*PipeRef`; unlike the +TypeScript SDK's `PipeRef` (which is `PromiseLike`), you always call +`.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]` explicitly. + +```go +rows, err := wavehouse.Fetch[map[string]any](ctx, + wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), +) +``` + +### `wavehouse.Fetch[Row](ctx, pipeRef)` + +Execute and decode results into `[]Row`. Package-level generic function +(Go has no generic methods) — the same pattern as `FetchTyped` for queries +and `SQL` for raw SQL. + +```go +type TopPage struct { + Page string `json:"page"` + Views int `json:"views"` +} + +rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{"limit": 50})) +``` + +### `.FetchUntyped(ctx)` + +Execute and decode results into `[]map[string]any`. The ordinary +(non-generic) method form of `Fetch`. + +```go +rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) +``` + +Pass `nil` for `params` when the pipe takes none, or the pipe requires only +parameters with server-side defaults. + +### `.Stream(opts)` + +Open a live stream from the pipe's underlying query. See +[Streaming](/sdk/go/streaming). + +This streams by table name, using the pipe's own name as the table — it +only works when the pipe name is also a valid table name. This matches the +TypeScript SDK's `PipeRef.stream()`, which has the same limitation. + +```go +stream := wh.Pipe("top_pages", nil).Stream(nil) +``` + +--- + +## Pipes Admin — `client.Pipes` + +Manage named query pipes. Requires the admin role (`policy.admin_role`). + +```go +// List all pipes. +pipes, err := wh.Pipes.List(ctx) + +// Get a single pipe definition. +pipe, err := wh.Pipes.Get(ctx, "top_pages") + +// Create or update. +err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + Parameters: []wavehouse.ParamDef{ + {Name: "limit", Type: "number", Required: false, Default: 100}, + }, + Description: "Top pages by view count", + AllowedRoles: []string{"viewer", "admin"}, +}) + +// Delete. +err = wh.Pipes.Delete(ctx, "old_pipe") +``` + +`PipeDef` is `Pipe` minus the `Name` field — the name is already in the +`Set`/`Get`/`Delete` call's path argument: + +```go +type PipeDef struct { + SQL string + Parameters []ParamDef + Description string + AllowedRoles []string +} +``` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md new file mode 100644 index 00000000..2b56fd5b --- /dev/null +++ b/docs/src/content/docs/sdk/go/queries.md @@ -0,0 +1,390 @@ +--- +title: "Go SDK Queries" +description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." +--- + +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: +table references, the chainable query builder, cursor pagination, and the +admin-only raw-SQL escape hatch. Every call takes a `context.Context` as its +first argument and returns `(T, error)` — see +[Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's +[Queries](/sdk/queries) page, which covers the same surface with a +`Result`-returning, `PromiseLike` builder. + +## Tables — `client.From(table)` + +`From` returns a `*TableRef` — a reference to a table. It performs no +request by itself, so it's safe to store in a variable or pass around. + +```go +clicks := wh.From("clicks") +``` + +### `.Fetch(ctx)` + +Shortcut for "select every column", with a default limit of 1000 +(`wavehouse.DefaultLimit`). Internally it's +`t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)` — unlike the +TypeScript SDK's `.fetch(opts?)`, there's no options struct to override the +limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself +(see [Query Builder](#query-builder)) if you need a different limit. + +When an access-control policy restricts your role's columns, the server +returns only the columns your role is allowed to read — `.Fetch()` is never +a way around `deny_columns`/`allow_columns` (see +[Access control](/access-control#column-permissions)). + +```go +page, err := clicks.Fetch(ctx) +if err != nil { + log.Fatal(err) +} +for _, row := range page.Data { + fmt.Println(row["page"]) +} +``` + +To paginate, use the query builder with an explicit `.OrderBy()` instead — +see [Pagination](#pagination). + +### `.Insert(ctx, data)` + +Insert one row or many. What you pass determines the wire format: + +- A single **map or struct** (anything that isn't a slice, and isn't + `[]byte`) is sent as JSON: `POST /v1/ingest?table={table}`. +- **Any slice** — `[]map[string]any`, a generated/user-defined row type like + `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via + reflection for non-`[]map[string]any` slices) and sent as a single + `application/x-ndjson` request, so a bad record doesn't fail or hide the + rest of the batch. Per-record outcomes come back in the result. + +```go +// Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) +res, err := clicks.Insert(ctx, map[string]any{"page": "/home", "button": "cta"}) + +// Many rows (map slice) → one NDJSON request, per-record summary +res, err = clicks.Insert(ctx, []map[string]any{ + {"page": "/home", "button": "cta"}, + {"page": "/about", "button": "nav"}, +}) +// res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results + +// Many rows (typed slice) — same NDJSON path, via reflection +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` +} +res, err = clicks.Insert(ctx, []ClickRow{ + {Page: "/home", Button: "cta"}, + {Page: "/about", Button: "nav"}, +}) +``` + +For a batch insert, `res.OK` is `true` only when every record succeeded +(`*res.Failed == 0`). Inspect `res.Failed` and `res.Results` (each +`InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for +partial failures — the returned `error` is reserved for whole-request +failures (network, `404` unknown table, `403` forbidden, `503` +backpressure). An empty slice is a no-op and sends no request. + +> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw +> JSON array or a single object directly (the `Content-Type` is only a +> hint), so non-SDK clients can send whichever shape is convenient. See the +> [API reference](/api#post-v1ingesttabletable--ingest-data). + +### `.InsertNDJSON(ctx, ndjson)` + +Insert pre-formatted NDJSON you already have, as a plain `string` — a file +you've read, or a string you built yourself — without first parsing it into +Go values. Returns the same per-record summary as a slice `Insert`. + +```go +// From a literal string. +res, err := clicks.InsertNDJSON(ctx, `{"page":"/a"}`+"\n"+`{"page":"/b"}`) + +// From a file on disk. +raw, err := os.ReadFile("events.ndjson") +if err != nil { + log.Fatal(err) +} +res, err = clicks.InsertNDJSON(ctx, string(raw)) +``` + +### `.Schema(ctx)` + +Fetch the table's column definitions from ClickHouse. Admin-only. + +```go +schema, err := clicks.Schema(ctx) +// schema.Name == "clicks" +// schema.Columns: []Column{{Name: "page", Type: "String", IsNullable: false, HasDefault: false}, ...} +``` + +### `.Select(...columns)` + +Start a query builder chain. See [Query Builder](#query-builder). + +```go +page, err := clicks.Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) +``` + +### `.SelectAll()` + +Start a query that selects **every column your role is allowed to read** — +the explicit form of what `.Fetch()` does. Mutually exclusive with +`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); the +server expands it to your allowed columns (never a raw `SELECT *`) and never +bypasses `deny_columns`/`allow_columns`. See +[Access control → Column permissions](/access-control#column-permissions). + +```go +page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) +``` + +### `.Stream(opts)` + +Open a real-time event subscription. See [Streaming](/sdk/go/streaming). + +```go +stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) +``` + +--- + +## Query Builder + +Returned by `tableRef.Select()`. Immutable — every chain method returns a +new `*QueryBuilder`, so intermediate values can be reused safely. Unlike the +TypeScript SDK's `PromiseLike` builder, a Go `*QueryBuilder` doesn't +auto-execute — call `.FetchUntyped(ctx)` or the package-level +`wavehouse.FetchTyped[Row](ctx, builder)` explicitly: + +```go +page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) +``` + +### Chain Methods + +All methods return a new `*QueryBuilder` — the original is unchanged. + +#### `.Select(...columns)` + +Append columns to the SELECT clause. A literal `"*"` is the column *named* +`*`, not a wildcard — use `.SelectAll()` for all columns. + +```go +q := clicks.Select("page").Select("button") // SELECT page, button +``` + +#### `.SelectAll()` + +Select every column your role may read (the all-columns wildcard, expanded +server-side to your allowed columns). Mutually exclusive with `.Select(...)` +and with aggregations (`.Count()`, `.Sum()`, etc.). + +```go +q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") +``` + +#### `.Where(column, op, value)` + +Add a filter condition, using the `FilterOp` constants: + +```go +clicks.Select("page"). + Where("score", wavehouse.OpGt, 10). + Where("page", wavehouse.OpLike, "/home%") +``` + +| `FilterOp` constant | Backend wire token | Description | +|----------------------|---------------------|--------------| +| `wavehouse.OpEq` | `eq` | Equal | +| `wavehouse.OpNeq` | `neq` | Not equal | +| `wavehouse.OpGt` | `gt` | Greater than | +| `wavehouse.OpGte` | `gte` | Greater than or equal | +| `wavehouse.OpLt` | `lt` | Less than | +| `wavehouse.OpLte` | `lte` | Less than or equal | +| `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | +| `wavehouse.OpLike` | `like` | SQL LIKE pattern | +| `wavehouse.OpNotLike` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | + +#### Aggregations + +```go +clicks.Select("page"). + Count("*", "total"). // COUNT(*) + Sum("score", "total_score"). // SUM(score) + Avg("score", "avg_score"). // AVG(score) + Min("score", "min_score"). // MIN(score) + Max("score", "max_score"). // MAX(score) + CountDistinct("page", "unique_pages"). + Aggregate("uniqExact", "user_id", "unique_users") // custom fn +``` + +Each aggregation method signature: `(column, alias string) *QueryBuilder`. +`Count` defaults to `column="*"` when `column` is `""`, and `alias="count"` +when `alias` is `""`; the other aggregations default `alias` to +`"_"` when left empty. + +#### `.GroupBy(...columns)` + +```go +clicks.Select("page").Count("", "").GroupBy("page") +``` + +#### `.OrderBy(column, dir)` + +```go +clicks.Select("page").Count("", "total").OrderBy("total", "desc") +``` + +`dir` defaults to `"asc"` when passed as `""`. + +#### `.Limit(n)` + +```go +clicks.Select().Limit(100) +``` + +If no limit is specified, `wavehouse.DefaultLimit` (1000) is applied +automatically to prevent unbounded result sets. The server also enforces the +configured maximum (`query.default_max_rows`, default 10,000 rows). + +#### `.TimeRange(column, since, until)` + +Filter by a time window. `since` and `until` accept RFC3339 timestamps or +relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"` — day and week suffixes +expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` to leave it +open-ended. + +```go +clicks.Select("page").TimeRange("received_timestamp", "1h", "") +clicks.Select("page").TimeRange( + "received_timestamp", "2026-01-01T00:00:00Z", "2026-02-01T00:00:00Z", +) +``` + +#### `.CacheTTL(seconds)` + +Records a desired result-cache TTL on the builder. **Currently client-side +state only** — the value is never sent to the server, which derives each +result's cache TTL adaptively from query execution time. Wiring it through +the wire format is tracked in +[#280](https://github.com/Wave-RF/WaveHouse/issues/280). + +```go +clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 +``` + +### `wavehouse.FetchTyped[Row](ctx, q)` + +Execute the query and decode rows into `[]Row`. Package-level generic +function (Go has no generic methods) — takes the builder as its argument. + +```go +type PageCount struct { + Page string `json:"page"` + Count int `json:"total"` +} + +page, err := wavehouse.FetchTyped[PageCount](ctx, + clicks.Select("page").Count("*", "total").GroupBy("page"), +) +// page.Data is []PageCount +``` + +### `.FetchUntyped(ctx)` + +Execute the query and decode rows into `[]map[string]any`. The ordinary +(non-generic) method form of `FetchTyped`. + +```go +page, err := clicks.Select("page").Limit(50).FetchUntyped(ctx) + +if page.HasMore && page.Next != nil { + page2, err := page.Next(ctx) // cursor-based pagination +} +``` + +### `.Stream(opts)` + +Open a live stream from the builder's table, applying `.Where()`/`.Select()` +filters and column projection client-side. See +[Streaming](/sdk/go/streaming). + +### Pagination + +`Page[T]`: + +```go +type Page[T any] struct { + Data []T + HasMore bool + Next func(ctx context.Context) (*Page[T], error) // nil when no cursor is available +} +``` + +When `Limit` is set and the result contains at least that many rows, +`HasMore` is `true`. Cursor-based pagination's `Next` walks the **first** +`.OrderBy()` column — it adds a filter on that column using the last row's +value — so `Next` is only attached when the query has an explicit +`.OrderBy()`. With no order column the result still reports `HasMore` +honestly, but `Next` is `nil` (there is no deterministic cursor to build) — +add an `.OrderBy()` to paginate. If the order column was left out of an +explicit `.Select(...)` projection, `Next` quietly returns an empty page +instead of erroring (there is no cursor value to read). + +```go +page, err := clicks.Select(). + OrderBy("received_timestamp", "desc"). + Limit(100). + FetchUntyped(ctx) +if err != nil { + log.Fatal(err) +} + +allRows := append([]map[string]any(nil), page.Data...) +for page.HasMore && page.Next != nil { + page, err = page.Next(ctx) + if err != nil { + log.Fatal(err) + } + allRows = append(allRows, page.Data...) +} +``` + +--- + +## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` + +Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT +must resolve to the policy admin role (`admin_role`, `"admin"` by default). +A request with no token, or an invalid/expired one, falls back to the +`default_role` and is rejected. Package-level generic function — use +`map[string]any` for a dynamic/unknown schema. + +```go +rows, err := wavehouse.SQL[map[string]any](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") + +// Or decode into a struct that matches the projected columns/aliases: +type PageTotal struct { + Page string `json:"page"` + Total int `json:"total"` +} +rows, err := wavehouse.SQL[PageTotal](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") +``` + +:::note[No parameter binding through the SDK] +Positional `?` substitution is not supported, and the SDK has no way to +forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + +`param_id=42` query-string combo) — the proxy doesn't forward arbitrary +query-string params and `SQL[Row]` doesn't expose a hook to add them. Inline +literals into the SQL, or — for safe binding from user-supplied input — use +the structured query builder (`wh.From(table)...`). +::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md new file mode 100644 index 00000000..7e6e90cb --- /dev/null +++ b/docs/src/content/docs/sdk/go/reference.md @@ -0,0 +1,233 @@ +--- +title: "Go SDK Reference & CLI" +description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." +--- + +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: +cancellation, the error model behind every SDK call's `(T, error)` return, +the complete API tree at a glance, and the `wavehouse-codegen` tool that +ships with the module. Compare with the TypeScript SDK's +[Reference & CLI](/sdk/reference) page. + +## Context Cancellation + +Every non-streaming operation takes a `context.Context` as its first +argument — Go's equivalent of the TypeScript SDK's `AbortSignal` support. +Cancel it with a timeout or an explicit `cancel()`: + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() + +page, err := wh.From("clicks").Fetch(ctx) +var whErr *wavehouse.Error +if errors.As(err, &whErr) && whErr.Code == "ABORTED" { + fmt.Println("Request timed out") +} +``` + +Context cancellation returns immediately (no retry) with +`&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. + +Streams work differently: `.Stream(opts)` doesn't take a `context.Context` +at all — the returned `*StreamController` owns its own internal context and +background goroutine, torn down explicitly via `.Close()`. See +[Streaming](/sdk/go/streaming#streamoptions). + +--- + +## Error Handling + +The SDK never panics on API or network failures — every operation returns +`(T, error)`, and errors are always `*wavehouse.Error` (unwrap with +`errors.As`). This is the direct Go equivalent of the TypeScript SDK's "the +SDK never throws" guarantee. + +| Status | Code | Retryable | Description | +|--------|------|-----------|--------------| +| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | +| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 403 | `HTTP_403` | No | Insufficient permissions | +| 404 | `HTTP_404` | No | Table or pipe not found | +| 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `ABORTED` | No | Request canceled via `context.Context` | + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } + return err +} +``` + +`wavehouse.IsRetryable(err)` is a shortcut for the `errors.As` + `.Retryable` +check above. + +Retries apply uniformly to every HTTP method the SDK issues (not just GET) — +matching the TypeScript SDK's `http.ts` behavior. For `/v1/ingest`, +at-least-once delivery on retry is a documented contract (see the API +docs' ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data) +note); dedup is the prescribed server-side safety net when duplicate +suppression matters. `/v1/admin/query` (raw SQL) is gated by `admin_role`, +so repeated execution on retry is an accepted risk for admin-only usage. + +--- + +## Full API Tree + +```text +NewClient(Config) → *Client +├── .From(table) → *TableRef +│ ├── .Fetch(ctx) → (*Page[map[string]any], error) +│ ├── .Select(...cols) → *QueryBuilder +│ │ ├── .Select() .SelectAll() .Where() .Count() .Sum() .Avg() .Min() .Max() +│ │ │ .CountDistinct() .Aggregate() .GroupBy() .OrderBy() +│ │ │ .Limit() .TimeRange() .CacheTTL() +│ │ ├── FetchTyped[Row](ctx, q) → (*Page[Row], error) // package-level generic func +│ │ ├── .FetchUntyped(ctx) → (*Page[map[string]any], error) +│ │ ├── .Stream(opts) → *StreamController +│ │ └── .LiveQuery(sub, opts) → *LiveQueryHandle +│ ├── .SelectAll() → *QueryBuilder +│ ├── .Insert(ctx, data) → (*InsertResult, error) +│ ├── .InsertNDJSON(ctx, ndjson) → (*InsertResult, error) +│ ├── .Schema(ctx) → (*TableSchema, error) +│ └── .Stream(opts) → *StreamController +├── .Pipe(name, params) → *PipeRef +│ ├── Fetch[Row](ctx, p) → ([]Row, error) // package-level generic func +│ ├── .FetchUntyped(ctx) → ([]map[string]any, error) +│ └── .Stream(opts) → *StreamController +├── .Pipes (admin) → *PipesNamespace +│ ├── .List(ctx) → ([]Pipe, error) +│ ├── .Get(ctx, name) → (*Pipe, error) +│ ├── .Set(ctx, name, PipeDef) → error +│ └── .Delete(ctx, name) → error +├── SQL[Row](ctx, client, query) → ([]Row, error) // package-level generic func, admin-only +├── .Schema (admin) → *SchemaNamespace +│ ├── .List(ctx) → (Schemas, error) +│ └── .Refresh(ctx) → error +├── .Policy (admin) → *PolicyNamespace +│ ├── .Get(ctx) → (*Policy, error) +│ ├── .Set(ctx, *Policy) → error +│ └── .Validate(ctx, *Policy) → (*ValidationResult, error) +├── .DLQ (admin) → *DLQNamespace +│ ├── .List(ctx) → (*DLQStats, error) +│ ├── .Table(ctx, name) → (*DLQStats, error) +│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 +└── .Sys → *SysNamespace + └── .Health(ctx) → error + +*StreamController +├── .Subscribe(*StreamSubscriber) → func() // unsubscribe +├── .Events() → <-chan StreamEvent // idiomatic Go alternative to an async iterator +├── .Close() +├── .Status() → StreamStatus +└── .Connected(ctx) → error // Go-only addition, blocks until live +``` + +## Codegen CLI + +Generate Go structs from a running WaveHouse instance. The module ships a +`wavehouse-codegen` command under `cmd/`: + +```bash +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ + --url http://localhost:8080 \ + --auth \ + --out ./db_types.go \ + --package myapp +``` + +Or, working inside this repo (`clients/go/`): + +```bash +go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go +``` + +Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev +server, pass an admin-role token with `--auth ` or the request is +denied with `403`. + +**Options:** + +| Flag | Description | Default | +|------|-------------|---------| +| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | +| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | +| `--auth`, `-a` | Bearer token (if auth required) | — | +| `--package`, `-p` | Go package name for the generated file | `main` | +| `--help`, `-h` | Show usage and exit | — | + +The output is run through `go/format` before being written — if a table or +column name would produce invalid Go source (rare, but possible with exotic +names), codegen fails loudly instead of writing broken code. + +**Example output:** + +```go +// Code generated by wavehouse-codegen. DO NOT EDIT. + +package myapp + +// ClicksRow represents a row in the "clicks" table. +type ClicksRow struct { + EventID string `json:"event_id"` + Page string `json:"page"` + UserID string `json:"user_id"` + DurationMS int `json:"duration_ms"` + ReceivedTimestamp string `json:"received_timestamp"` +} +``` + +Table and column names are converted to `PascalCase` for Go field/type names +(a leading digit gets an `X` prefix — e.g. a table named `2fa_events` +becomes `X2faEventsRow` — to stay a valid Go identifier). A column with +`has_default: true` in the schema gets `,omitempty` appended to its JSON +tag. + +**ClickHouse → Go type mapping:** + +| ClickHouse Type | Go Type | +|------------------|---------| +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` | `bool` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `Float32` | `float32` | +| `Float64` | `float64` | +| `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (big numbers are strings in JSON) | +| `Nullable(T)` | `*T` | +| `LowCardinality(T)` | same as `T` | +| `Array(T)` | `[]T` | +| `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| anything unrecognized | `any` | + +This differs from the TypeScript SDK's mapping in one notable way: Go's +codegen preserves ClickHouse's integer **widths** (`UInt32` → `uint32`, not +a generic `number`), since Go — unlike TypeScript — has native fixed-width +integer types. + +## Testing + +The Go SDK ships with unit tests colocated in `clients/go/` (its own Go +module — `clients/go/go.mod` — separate from the root `WaveHouse` module), +plus a wire-format **conformance suite** +(`clients/go/conformance_test.go` + `clients/go/testdata/wire_cases.json`) +that replays a shared fixture of builder calls and asserts the Go SDK +produces the exact same HTTP method, path, content type, and body as the +TypeScript SDK for each one — keeping the two clients honest about the wire +format they both speak. + +```bash +cd clients/go +go test ./... +``` + +Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's +`make test-e2e` harness — see the TypeScript SDK's +[E2E Testing](/sdk/reference#e2e-testing) section for that suite's +architecture, which the Go client doesn't currently participate in. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md new file mode 100644 index 00000000..55b31243 --- /dev/null +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -0,0 +1,254 @@ +--- +title: "Go SDK Streaming & Live Queries" +description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." +--- + +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE +event streams from tables, builders, and pipes, plus live queries that +backfill history before going live. Builders and table refs come from +[Queries](/sdk/go/queries). Compare with the TypeScript SDK's +[Streaming & Live Queries](/sdk/streaming) page — the two implement the same +protocol and mostly the same client-side filtering, but connection lifecycle +differs: Go streams are goroutine-backed and closed explicitly, not tied to +a `context.Context` or a browser's `EventSource`. + +## Streaming + +Streams use SSE (Server-Sent Events), parsed by hand over `net/http` (no +third-party SSE library — the SDK has zero runtime dependencies). + +### `*StreamController` + +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and +`*DLQNamespace` (the DLQ variant is not yet functional server-side — +[#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` +returns immediately; the connection opens in a background goroutine. + +```go +stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() +``` + +### `.Subscribe(sub) → func()` + +Callback-based consumption. Returns an unsubscribe function. The +subscriber's `Status` callback fires immediately with the current status. + +```go +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { + // e: {Table: "clicks", Timestamp: "2026-...", Data: map[string]any{"page": "/", ...}} + fmt.Println("New event:", e.Data) + }, + Status: func(s wavehouse.StreamStatus) { + // s: StatusConnecting | StatusLive | StatusReconnecting | StatusClosed + updateIndicator(s) + }, + Error: func(err error) { + fmt.Println("Stream error:", err) + }, +}) + +// Cleanup — removes this subscriber; the connection stays open for any +// others and must still be closed with stream.Close() when you're done +// with the stream itself. +defer unsub() +``` + +### Channel-based consumption — `.Events()` + +The idiomatic Go alternative to the TypeScript SDK's async iterator: a +read-only channel, closed automatically when the stream shuts down. + +```go +stream := wh.From("clicks").Stream(nil) +defer stream.Close() + +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) + if shouldStop { + break + } +} +``` + +:::caution[`break` does not close the stream] +Unlike the TypeScript SDK's async iterator — where breaking out of a +`for await` loop auto-closes the underlying connection — breaking a Go +`for range stream.Events()` loop only stops consuming from the channel; the +background goroutine and its HTTP connection keep running. Always pair a +stream with `defer stream.Close()` (or an explicit `stream.Close()` on every +exit path) regardless of which consumption style you use. +::: + +The channel is buffered (256 events); a slow consumer that never drains it +causes the SDK to **drop** new events for that channel rather than block the +stream's read loop (`.Subscribe` callbacks still fire per event +regardless of channel backpressure). + +### `.Close()` + +Explicitly close the stream and release its resources. Non-blocking — safe +to call from inside a subscriber callback (which runs on the stream's own +goroutine); it signals the goroutine to stop without waiting for it to +finish. + +```go +stream.Close() +``` + +### `.Status()` + +Returns the current `StreamStatus`. A method (not a field), since Go has no +JS-style reactive property access. + +```go +status := stream.Status() +``` + +### `.Connected(ctx)` + +**Go-only addition** — not present in the TypeScript SDK. Blocks until the +stream reaches `StatusLive` or `ctx` is canceled; returns an error if the +stream closes before connecting. Useful when you need to know a stream is +live before doing something else (e.g. before starting a producer in a +test). + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() +if err := stream.Connected(ctx); err != nil { + log.Fatal(err) +} +``` + +### `StreamOptions` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `Since` | `string` | RFC3339 timestamp for gap-fill replay | + +There's no `Signal`/context field here — a stream isn't canceled by passing +a `context.Context` into `.Stream()`; call `.Close()` instead (see above). + +### `StreamEvent` + +```go +type StreamEvent struct { + Table string // table name (e.g. "clicks") + Timestamp string // received_timestamp (RFC3339Nano) + Data map[string]any // row data +} +``` + +### Transport Behavior + +| Transport | Reconnect | Protocol | +| --------- | --------- | -------- | +| SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | + +Auth is sent as an `Authorization: Bearer` header on every stream +(re)connection — see +[the note in the Getting Started guide](/sdk/go#creating-a-client). The +TypeScript SDK's "more than 5 concurrent connections" warning is a +browser-specific `EventSource` limit and doesn't apply here. + +### Client-Side Stream Filtering + +When a `*QueryBuilder` with `.Where()` filters or `.Select()` columns calls +`.Stream()`, the returned stream applies those filters client-side: + +```go +stream := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Stream(nil) + +// Only events where page == "/home" are emitted, with only page + button fields +``` + +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, +`not_like` — the same `FilterOp` set `.Where()` takes everywhere. `like` / +`not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any +single character), case-insensitively. `in` accepts any Go slice type on +the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. + +--- + +## Live Queries + +Live queries combine a historical backfill (`.FetchUntyped`) with a +real-time stream, providing a seamless initial-load + live-updates +experience. Only available on `*QueryBuilder` (there's no `TableRef.LiveQuery` +shortcut, matching the TypeScript SDK). + +```go +lq := wh.From("clicks"). + SelectAll(). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Called once with the historical backfill. + setRows(rows) + }, + Next: func(e wavehouse.StreamEvent) { + // Called for each live event after backfill. + addRow(e.Data) + }, + Error: func(err error) { + log.Println(err) + }, + }, nil) + +// Cleanup +defer lq.Close() +``` + +### `StreamSubscriber` + +```go +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} +``` + +:::note[`Initial` is always untyped] +Unlike the TypeScript SDK's `initial: (result: Result) => void`, the Go +SDK's `LiveQuery` doesn't accept a type parameter — `Initial` always +receives `[]map[string]any` plus a plain `error`, even if you'd otherwise +use `wavehouse.FetchTyped[Row]` for the same query outside a live query. +Decode into your own type inside the callback if you need one. +::: + +### How it works + +1. Subscribes to the stream **immediately** and buffers incoming events. +2. Runs the `.FetchUntyped(ctx)` query for historical data, calls + `sub.Initial(rows, err)` with the result. +3. Deduplicates buffered events by comparing timestamps against the latest + historical row's `received_timestamp`. +4. Flushes remaining buffered events (re-checking for anything that arrived + mid-flush) and switches to live mode. + +This "stream-first" approach ensures no events are lost between the fetch +and stream start. + +### `.Close()` + +Shuts down the live query and its underlying stream. Safe to call more than +once (idempotent via `sync.Once`). + +```go +lq.Close() +``` diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 01c5fb84..aa4ca864 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -7,6 +7,15 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — Zero-dependency TypeScript client for WaveHouse. +:::tip[Writing Go instead?] +WaveHouse also ships an official Go SDK +(`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, +`context.Context`-first, generics for typed rows. See the +[Go SDK docs](/sdk/go). The two clients speak the same wire format, so +everything below about tables, the query builder, streaming, and admin +endpoints carries over conceptually — only the language idioms differ. +::: + ## Installation @@ -411,3 +420,15 @@ The full error-code table lives in href="/sdk/reference" /> + +## Go SDK + +Prefer Go? The same server, the same wire format, an idiomatic Go client: + + + + From 61b436f0de0ae0f2f2e60618df61afa6fccb500d Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:40 -0400 Subject: [PATCH 03/59] ci(sdk): wire Go SDK into CI and update AGENTS.md - Add test-go-sdk to CI unit job - AGENTS.md: Go SDK file structure + feature parity table - lint-go-sdk already wired via verify-parallel in Makefile --- .github/workflows/ci.yml | 4 ++-- AGENTS.md | 30 +++++++++++++++++++----------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 591820e8..8646bf53 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -204,8 +204,8 @@ jobs: uses: ./.github/actions/setup-env with: go-cache-suffix: "-unit" - - name: Run Go unit tests + SDK vitest tests - run: make test-unit test-ts COV_DEFER=1 + - name: Run Go unit tests + SDK vitest + Go SDK tests + run: make test-unit test-ts test-go-sdk COV_DEFER=1 - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/AGENTS.md b/AGENTS.md index 1e3ec198..4ff5f47c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync). +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -331,22 +331,22 @@ Diagrams render inside the Starlight content column (~46–58rem wide) as build- ## SDK Sync -The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) is the canonical client and ships from this repo. When backend changes alter the public API surface, the SDK needs corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. +The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. | Backend change | SDK considerations | | -------------- | ------------------ | -| New user-facing API endpoint | Add a typed client method (in `clients/ts/src/client.ts` or the relevant subsystem file: `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`, etc.); update the matching SDK doc page under `docs/src/content/docs/sdk/` (`queries`, `streaming`, `pipes`, `admin`, or `reference` by topic — plus the API tree in `reference.md`) | -| Change to JWT auth / role extraction | Update auth handling in `clients/ts/src/http.ts` and types in `clients/ts/src/client.ts` | -| Change to `EventMessage` / ingest event format | Update payload types in `clients/ts/src/` (some are codegen-regenerated — re-run the SDK codegen CLI) | -| New / changed structured query AST | Update `clients/ts/src/query-builder.ts` types + builder methods | -| Change to live-query aggregation classification | Update live-query helpers in `clients/ts/src/stream/` | -| Named pipes API change | Update `clients/ts/src/pipes.ts` | -| Policy / access-control change | Update `clients/ts/src/policy.ts` | -| ClickHouse schema-driven type changes | Re-run the SDK codegen CLI; commit regenerated types | +| New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update doc pages under `docs/src/content/docs/sdk/` for both `ts/` and `go/`. Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | +| Change to JWT auth / role extraction | TS: `clients/ts/src/http.ts` + `client.ts`. Go: `clients/go/http.go` + `wavehouse.go`. | +| Change to `EventMessage` / ingest event format | Update payload types in both SDKs (some are codegen-regenerated — re-run both codegen CLIs). | +| New / changed structured query AST | TS: `clients/ts/src/query-builder.ts`. Go: `clients/go/query_builder.go` + `types.go`. | +| Change to live-query aggregation classification | TS: `clients/ts/src/stream/`. Go: `clients/go/live_query.go`. | +| Named pipes API change | TS: `clients/ts/src/pipes.ts`. Go: `clients/go/pipes.go`. | +| Policy / access-control change | TS: `clients/ts/src/policy.ts`. Go: `clients/go/policy.go`. | +| ClickHouse schema-driven type changes | Re-run both SDK codegen CLIs; commit regenerated types. | Internal-only backend changes (middleware refactors, observability internals, dedup implementation, sweeper logic, NATS plumbing) generally don't need SDK updates. Use judgement — table above is the source of truth; nothing automated nudges you. -**The decision test**: would a `@wavehouse/sdk` user's *code* need to change to take advantage of (or be compatible with) this change? If yes, SDK update needed. If no (purely internal optimization), no. +**The decision test**: would a user's *code* need to change to take advantage of (or be compatible with) this change? If yes, both SDKs need updates. If no (purely internal optimization), no. ## Common Tasks @@ -387,6 +387,14 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) +clients/ts/ → TypeScript SDK (@wavehouse/sdk) +clients/go/ → Go SDK (wavehouse-go) + wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) + query_builder.go, table.go → Structured query builder + per-table typed client + stream.go, live_query.go → SSE streaming + live queries + pipes.go, policy.go, schema.go, dlq.go, sys.go → Subsystem clients (pipes, policy, schema, DLQ, health) + cmd/wavehouse-codegen/main.go → Codegen CLI + testdata/wire_cases.json → Wire-format conformance fixtures internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/policy/pipes endpoints) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) internal/cache/ → Caching (interface + L1/L2/tiered implementations) From 28eb1cf3a7845349738361d2c8f58ed359a5e950 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 16:06:33 -0400 Subject: [PATCH 04/59] chore: trim verbose comments across SDK, Makefile, and docs --- Makefile | 20 +++----------------- README.md | 3 +-- clients/go/http.go | 10 ++-------- clients/go/live_query.go | 21 +++++---------------- clients/go/stream.go | 7 ++----- docs/src/config/sidebar.ts | 12 +++--------- 6 files changed, 16 insertions(+), 57 deletions(-) diff --git a/Makefile b/Makefile index 3174e5c6..0c614912 100644 --- a/Makefile +++ b/Makefile @@ -428,13 +428,7 @@ endif tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) -# verify-go-sdk: static checks for clients/go/ — a nested Go module (its own -# go.mod), so it's invisible to `go list ./...` from the root and every leaf -# above (GO_DIRS, lint-go, vulncheck, tidy) silently skips it. Scoped and run -# explicitly here instead. `go vet` needs its own module context (cd -# clients/go), but gofumpt is a pure syntax formatter with no module -# resolution of its own, so the repo-pinned $(GOFUMPT) binary can format it -# directly by path from the root — no second tool pin needed. +# verify-go-sdk: nested module at clients/go/ — invisible to root go list. .PHONY: verify-go-sdk verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) @@ -732,21 +726,13 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # test-go-sdk: unit tests for clients/go/ — a nested Go module (its own # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and -# needs its own leaf; a root `go test ./...` wouldn't reach it either. Zero -# third-party runtime or test deps (stdlib only, no go.sum), so no -# go-mod-download prereq. Not yet wired into the Go/TS coverage gate — see -# verify-go-sdk above for the same "nested module, own leaf" reasoning. +# test-go-sdk: nested module — needs its own target. .PHONY: test-go-sdk test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" @cd clients/go && go test ./... -# test-go-sdk-e2e: runs the Go SDK's E2E tests against a live WaveHouse -# instance. Requires a running server (e.g. `make dev` in the main repo). -# Env vars: -# WAVEHOUSE_URL base URL of the server (default: http://localhost:8080) -# WAVEHOUSE_AUTH bearer token for auth (optional; omit for default_role) -# The tests skip gracefully when the server is unreachable. +# test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. .PHONY: test-go-sdk-e2e test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" diff --git a/README.md b/README.md index d9e658c4..dc63707d 100644 --- a/README.md +++ b/README.md @@ -60,8 +60,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client** — `@wavehouse/sdk`: zero-dependency TypeScript client with query builder, live queries, streaming, and schema codegen. -- **Go SDK** — `go get github.com/Wave-RF/WaveHouse/clients/go`: full API-tree parity with the TS SDK — ingest, query, streaming, and policy management from Go. +- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): zero-dependency clients with query builder, live queries, streaming, and schema codegen. ## 📊 How it compares diff --git a/clients/go/http.go b/clients/go/http.go index 52aae6b4..9646a0a2 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -68,14 +68,8 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a var lastErr error maxAttempts := hctx.maxRetries + 1 - // Retries below are not restricted by HTTP method — this matches the TS - // SDK's http.ts, which retries POST the same as GET on network errors, - // 503/Retry-After, and other retryable 5xx. For /v1/ingest, at-least-once - // delivery on retry is a documented contract (see docs/api.md's - // "At-least-once on retry" note); dedup is the prescribed server-side - // safety net when duplicate suppression matters. The only other mutation - // path, /v1/admin/query, is gated by admin_role, so repeated execution on - // retry is assumed to be an accepted risk for admin-only raw SQL. + // Retries all methods including POST. For /v1/ingest, at-least-once delivery + // is the documented contract; dedup is the server-side safety net. for attempt := range maxAttempts { var bodyReader io.Reader if bodyBytes != nil { diff --git a/clients/go/live_query.go b/clients/go/live_query.go index fad4e80a..97cc6c58 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -89,16 +89,9 @@ func newLiveQuery( } } - // Step 5: Flush buffered events newer than the fetch. - // - // buffering stays true for the whole flush: events that arrive - // concurrently (after Subscribe's Next handler releases mu but - // before we're done here) must keep landing in buffer rather than - // being dispatched directly by the live path, or two goroutines - // could call sub.Next at once. We only flip buffering to false - // once a lock-protected check finds the buffer empty, which - // guarantees no event is ever handed to sub.Next by both paths - // and that delivery stays in arrival order. + // Step 5: Flush buffered events. buffering stays true until the + // buffer is provably empty under the lock — prevents concurrent + // sub.Next calls and preserves delivery order. for { mu.Lock() if closed { @@ -121,12 +114,8 @@ func newLiveQuery( if c { return } - // Use <= (not <) to filter events whose timestamp matches the last - // historical row — those rows were already delivered in the backfill - // response. If two distinct events share a timestamp and only one - // appeared in the backfill, the duplicate is lost; this matches the - // TS SDK's dedup behavior and is acceptable because received_timestamp - // has sub-millisecond precision in practice. + // <= dedupes events already delivered in the backfill. + // Sub-millisecond received_timestamp precision makes collisions rare. if lastTimestamp != "" && event.Timestamp <= lastTimestamp { continue } diff --git a/clients/go/stream.go b/clients/go/stream.go index e30d6346..770550d7 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -56,11 +56,8 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { currentStatus := sc.status sc.mu.Unlock() - // Benign race: if setStatus fires between the unlock above and the - // callback below, the subscriber may see a stale status here. This is - // harmless because setStatus also invokes the subscriber's callback, - // so the subscriber will receive the up-to-date status immediately - // after. Matches the TS SDK's registration behavior. + // Benign race: setStatus also calls the subscriber, so a stale + // status here is immediately followed by the correct one. if sub.Status != nil { sub.Status(currentStatus) } diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index f114c692..b4ec5da3 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,15 +25,9 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Topic-first SDK pages: the multi-language plan on record (PR #313) - // was for a second language to grow on these - // shared pages instead of a parallel tree. The Go SDK launched as - // its own docs/src/content/docs/sdk/go/* tree instead — its API - // shape (context.Context, (T, error), generics on package-level - // funcs) diverges enough from the TS builder that shared prose read - // worse than dedicated pages. Revisit folding these into tabs if a - // third language lands and the duplication becomes a maintenance - // cost. + // Separate trees per SDK — API shapes diverge enough that shared + // prose reads worse than dedicated pages. Revisit with tabs if a + // third language lands. label: "TypeScript SDK", items: [ { label: "Overview", slug: "sdk" }, From b66413a5867ee2bd4bff23bb2016bcaa47f1d0dc Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 16:16:35 -0400 Subject: [PATCH 05/59] fix(sdk): resolve 18 golangci-lint findings in Go SDK - context.Context as first param in doRequest (revive) - Checked all json Encode/Decode/Unmarshal returns (errcheck) - Wrapped defer Body.Close with error discard (errcheck) - if-else chain to switch in buildAST (gocritic) - Tagged switch on r.Method in test (staticcheck) - Renamed built-in shadow cap to capt (revive) - Removed wasted msg assignment (wastedassign) - WriteFile 0o644 to 0o600 (gosec) - nolint:gosec for cancel called in Close (gosec) --- clients/go/client_test.go | 6 ++-- clients/go/cmd/wavehouse-codegen/main.go | 4 +-- clients/go/conformance_test.go | 44 ++++++++++++------------ clients/go/dlq.go | 4 +-- clients/go/errors.go | 2 +- clients/go/http.go | 6 ++-- clients/go/http_test.go | 28 +++++++-------- clients/go/live_query.go | 2 +- clients/go/namespaces_test.go | 26 +++++++------- clients/go/pipes.go | 10 +++--- clients/go/policy.go | 6 ++-- clients/go/query_builder.go | 9 ++--- clients/go/query_builder_test.go | 8 ++--- clients/go/schema.go | 4 +-- clients/go/stream.go | 2 +- clients/go/sys.go | 2 +- clients/go/table.go | 6 ++-- clients/go/table_test.go | 16 ++++----- clients/go/wavehouse.go | 2 +- 19 files changed, 94 insertions(+), 93 deletions(-) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 4f57507f..2b5239e1 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -60,7 +60,7 @@ func TestClient_From(t *testing.T) { if r.URL.Query().Get("table") != "events" { t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) } - json.NewEncoder(w).Encode([]map[string]any{}) + _ = json.NewEncoder(w).Encode([]map[string]any{}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) _, _ = c.From("events").Fetch(context.Background()) @@ -72,11 +72,11 @@ func TestClient_SQL(t *testing.T) { t.Errorf("want /v1/admin/query, got %s", r.URL.Path) } var body map[string]string - json.NewDecoder(r.Body).Decode(&body) + _ = json.NewDecoder(r.Body).Decode(&body) if body["sql"] != "SELECT 1" { t.Errorf("want sql=SELECT 1, got %s", body["sql"]) } - json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 1df138b9..213710e8 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -89,7 +89,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc if err != nil { return nil, err } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != 200 { return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) } @@ -329,7 +329,7 @@ func main() { os.Exit(1) } - if err := os.WriteFile(args.out, formatted, 0o644); err != nil { + if err := os.WriteFile(args.out, formatted, 0o600); err != nil { fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) os.Exit(1) } diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 5f5edf74..8809457a 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -64,31 +64,31 @@ func TestConformance_WireFormat(t *testing.T) { for _, tc := range cases { t.Run(tc.Name, func(t *testing.T) { - var cap captured + var capt captured srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - cap.method = r.Method - cap.path = r.URL.RequestURI() - cap.contentType = r.Header.Get("Content-Type") + capt.method = r.Method + capt.path = r.URL.RequestURI() + capt.contentType = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) - cap.body = string(raw) + capt.body = string(raw) // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") switch { case strings.HasPrefix(r.URL.Path, "/v1/dlq"): - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": - json.NewEncoder(w).Encode([]TableSchema{}) + _ = json.NewEncoder(w).Encode([]TableSchema{}) case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": - json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": - json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": - json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) + _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": - json.NewEncoder(w).Encode([]Pipe{}) + _ = json.NewEncoder(w).Encode([]Pipe{}) default: - json.NewEncoder(w).Encode([]map[string]any{}) + _ = json.NewEncoder(w).Encode([]map[string]any{}) } })) defer srv.Close() @@ -186,29 +186,29 @@ func TestConformance_WireFormat(t *testing.T) { } // Verify method. - if tc.ExpectedMethod != "" && cap.method != tc.ExpectedMethod { - t.Errorf("method: want %s, got %s", tc.ExpectedMethod, cap.method) + if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { + t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) } // Verify path. if tc.ExpectedPath != "" { // Normalize: the SDK may use different encoding (+ vs %20). wantPath := normalizePath(tc.ExpectedPath) - gotPath := normalizePath(cap.path) + gotPath := normalizePath(capt.path) if wantPath != gotPath { - t.Errorf("path: want %s, got %s", tc.ExpectedPath, cap.path) + t.Errorf("path: want %s, got %s", tc.ExpectedPath, capt.path) } } // Verify content type. - if tc.ExpectedContentType != "" && cap.contentType != tc.ExpectedContentType { - t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, cap.contentType) + if tc.ExpectedContentType != "" && capt.contentType != tc.ExpectedContentType { + t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, capt.contentType) } // Verify raw body (for NDJSON). if tc.ExpectedRawBody != nil { - if cap.body != *tc.ExpectedRawBody { - t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, cap.body) + if capt.body != *tc.ExpectedRawBody { + t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, capt.body) } return } @@ -219,8 +219,8 @@ func TestConformance_WireFormat(t *testing.T) { if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { t.Fatalf("parse expected_body: %v", err) } - if err := json.Unmarshal([]byte(cap.body), &got); err != nil { - t.Fatalf("parse captured body: %v (body: %s)", err, cap.body) + if err := json.Unmarshal([]byte(capt.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, capt.body) } if !deepEqualJSON(want, got) { wantJSON, _ := json.MarshalIndent(want, "", " ") diff --git a/clients/go/dlq.go b/clients/go/dlq.go index 01248b2c..b523ab33 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -14,7 +14,7 @@ type DLQNamespace struct { // List returns DLQ statistics (message counts per table). Admin-only. func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { var stats DLQStats - if err := doRequest(d.ctx, ctx, requestOptions{ + if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", }, &stats); err != nil { @@ -26,7 +26,7 @@ func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { // Table returns DLQ stats filtered by table name. Admin-only. func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { var stats DLQStats - if err := doRequest(d.ctx, ctx, requestOptions{ + if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", params: url.Values{"table": {name}}, diff --git a/clients/go/errors.go b/clients/go/errors.go index 1a161a1b..f1fb48ac 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -47,7 +47,7 @@ func parseErrorResponse(res *http.Response) *Error { _ = json.Unmarshal(raw, &body) } - msg := "" + var msg string if s, ok := body["error"].(string); ok { msg = s } else if s, ok := body["message"].(string); ok { diff --git a/clients/go/http.go b/clients/go/http.go index 9646a0a2..c7dc1c11 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -34,7 +34,7 @@ type requestOptions struct { // doRequest is the internal fetch wrapper with auth, retry, and backoff. // It decodes the response body into dst (unless dst is nil). -func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst any) error { +func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst any) error { reqURL := buildURL(hctx.baseURL, opts.path, opts.params) ct := opts.contentType if ct == "" { @@ -107,7 +107,7 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a } if res.StatusCode >= 200 && res.StatusCode < 300 { - defer res.Body.Close() + defer func() { _ = res.Body.Close() }() if dst == nil { _, _ = io.Copy(io.Discard, res.Body) return nil @@ -131,7 +131,7 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a } apiErr := parseErrorResponse(res) - res.Body.Close() + _ = res.Body.Close() // 503 with Retry-After: wait the specified duration. if res.StatusCode == http.StatusServiceUnavailable { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 7ce4c964..5a095c62 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -22,11 +22,11 @@ func testCtx(handler http.Handler) httpContext { func TestDoRequest_SuccessfulGET(t *testing.T) { hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) })) var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/health", }, &result) @@ -43,11 +43,11 @@ func TestDoRequest_POSTWithBody(t *testing.T) { var gotCT string hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") - json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewDecoder(r.Body).Decode(&gotBody) w.WriteHeader(200) })) - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/ingest", body: map[string]string{"page": "/home"}, @@ -71,10 +71,10 @@ func TestDoRequest_RawBody(t *testing.T) { raw := make([]byte, 1024) n, _ := r.Body.Read(raw) gotBody = string(raw[:n]) - json.NewEncoder(w).Encode(map[string]int{"total": 1}) + _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) })) - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/ingest", rawBody: `{"page":"/a"}`, @@ -99,7 +99,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { })) hctx.auth = StaticToken("my-token") - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/v1/schema", }, nil) @@ -117,11 +117,11 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { count.Add(1) w.Header().Set("Content-Type", "application/json") w.WriteHeader(404) - json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) })) hctx.maxRetries = 2 - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/v1/schema", }, nil) @@ -141,15 +141,15 @@ func TestDoRequest_5xxRetried(t *testing.T) { if n < 3 { w.Header().Set("Content-Type", "application/json") w.WriteHeader(500) - json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) return } - json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + _ = json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) })) hctx.maxRetries = 2 var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/health", }, &result) @@ -169,7 +169,7 @@ func TestDoRequest_AbortedOnCancel(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() // cancel immediately - err := doRequest(hctx, ctx, requestOptions{ + err := doRequest(ctx, hctx, requestOptions{ method: "GET", path: "/health", }, nil) @@ -185,7 +185,7 @@ func TestDoRequest_EmptyResponse(t *testing.T) { })) var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", }, &result) diff --git a/clients/go/live_query.go b/clients/go/live_query.go index 97cc6c58..d6c665fe 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -21,7 +21,7 @@ func newLiveQuery( sub *StreamSubscriber, filters []QueryFilter, ) *LiveQueryHandle { - ctx, cancel := context.WithCancel(context.Background()) + ctx, cancel := context.WithCancel(context.Background()) //nolint:gosec // cancel is called in Close() lq := &LiveQueryHandle{ stream: stream, cancel: cancel, diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 84bc4b2f..18ff1b39 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -35,7 +35,7 @@ func TestSchemaNamespace_List(t *testing.T) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } - json.NewEncoder(w).Encode([]TableSchema{ + _ = json.NewEncoder(w).Encode([]TableSchema{ {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, }) })) @@ -65,13 +65,13 @@ func TestSchemaNamespace_Refresh(t *testing.T) { func TestPolicyNamespace_GetSetValidate(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - switch { - case r.Method == "GET": - json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) - case r.Method == "PUT": + switch r.Method { + case "GET": + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case "PUT": w.WriteHeader(200) - case r.Method == "POST": - json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + case "POST": + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) } })) @@ -99,7 +99,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace_List(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) if err != nil { @@ -114,7 +114,7 @@ func TestDLQNamespace_Table(t *testing.T) { var gotParam string c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotParam = r.URL.Query().Get("table") - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) })) _, err := c.DLQ.Table(context.Background(), "clicks") if err != nil { @@ -130,9 +130,9 @@ func TestPipesNamespace_CRUD(t *testing.T) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { - json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) + _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) } else { - json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) + _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) } case "PUT": w.WriteHeader(200) @@ -174,8 +174,8 @@ func TestPipeRef_Fetch(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path gotMethod = r.Method - json.NewDecoder(r.Body).Decode(&gotBody) - json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) + _ = json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) })) rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) if err != nil { diff --git a/clients/go/pipes.go b/clients/go/pipes.go index f87f4d64..767078a6 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -13,7 +13,7 @@ type PipesNamespace struct { // List returns all registered pipes. Admin-only. func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { var pipes []Pipe - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/pipes", }, &pipes); err != nil { @@ -25,7 +25,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { // Get returns a single pipe definition by name. Admin-only. func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { var pipe Pipe - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/pipes/" + url.PathEscape(name), }, &pipe); err != nil { @@ -36,7 +36,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { // Set creates or updates a pipe. Admin-only. func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/pipes/" + url.PathEscape(name), body: def, @@ -45,7 +45,7 @@ func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) erro // Delete removes a pipe by name. Admin-only. func (p *PipesNamespace) Delete(ctx context.Context, name string) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", path: "/v1/admin/pipes/" + url.PathEscape(name), }, nil) @@ -74,7 +74,7 @@ func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { body = map[string]any{} } var rows []Row - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", path: "/v1/pipes/" + url.PathEscape(p.name), body: body, diff --git a/clients/go/policy.go b/clients/go/policy.go index c7eaa2af..c8a2306b 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -10,7 +10,7 @@ type PolicyNamespace struct { // Get returns the current access-control policy. Admin-only. func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { var pol Policy - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/policy", }, &pol); err != nil { @@ -21,7 +21,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { // Set replaces the entire access-control policy. Admin-only. func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/policy", body: pol, @@ -31,7 +31,7 @@ func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { // Validate checks a policy without applying it (dry run). Admin-only. func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*ValidationResult, error) { var result ValidationResult - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", path: "/v1/admin/policy/validate", body: pol, diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 9dda452d..6e9ab68b 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -174,7 +174,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro ast := q.buildAST(limit) var rows []Row - if err := doRequest(q.ctx, ctx, requestOptions{ + if err := doRequest(ctx, q.ctx, requestOptions{ method: "POST", path: "/v1/query", params: url.Values{"table": {q.state.table}}, @@ -244,11 +244,12 @@ func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { // Projection: explicit select_all, then explicit columns, else — for a bare // query with no projection and no aggregations — default to select_all so // from(t).fetch() returns rows. - if q.state.selectAll { + switch { + case q.state.selectAll: ast.SelectAll = true - } else if hasColumns { + case hasColumns: ast.Columns = q.state.columns - } else if !hasAggs { + case !hasAggs: ast.SelectAll = true } diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index b2f560bb..84261a05 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -30,14 +30,14 @@ func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[s c, _ := queryTestCtx(wrapper) return c, func() map[string]any { var m map[string]any - json.Unmarshal(body, &m) + _ = json.Unmarshal(body, &m) return m } } var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) }) func TestQueryBuilder_Immutability(t *testing.T) { @@ -199,7 +199,7 @@ func TestQueryBuilder_TimeRange(t *testing.T) { func TestQueryBuilder_Pagination_HasMore(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) @@ -217,7 +217,7 @@ func TestQueryBuilder_Pagination_HasMore(t *testing.T) { func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) diff --git a/clients/go/schema.go b/clients/go/schema.go index 67addc5b..3c7f4307 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -11,7 +11,7 @@ type SchemaNamespace struct { func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // The backend returns []TableSchema; transform to map[string]TableSchema. var raw []TableSchema - if err := doRequest(s.ctx, ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/schema", }, &raw); err != nil { @@ -26,7 +26,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // Refresh forces a schema re-discovery from ClickHouse. Admin-only. func (s *SchemaNamespace) Refresh(ctx context.Context) error { - return doRequest(s.ctx, ctx, requestOptions{ + return doRequest(ctx, s.ctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", }, nil) diff --git a/clients/go/stream.go b/clients/go/stream.go index 770550d7..84075005 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -272,7 +272,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if err != nil { return "", err } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) diff --git a/clients/go/sys.go b/clients/go/sys.go index 4d2af556..e52c252c 100644 --- a/clients/go/sys.go +++ b/clients/go/sys.go @@ -10,7 +10,7 @@ type SysNamespace struct { // Health pings the server's public /v1/health endpoint. Returns nil when the // server is reachable and past boot, or an error describing the failure. func (s *SysNamespace) Health(ctx context.Context) error { - return doRequest(s.ctx, ctx, requestOptions{ + return doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/health", }, nil) diff --git a/clients/go/table.go b/clients/go/table.go index adc8d829..3f32d030 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -78,7 +78,7 @@ func (t *TableRef) InsertNDJSON(ctx context.Context, ndjson string) (*InsertResu // Schema returns the table's column definitions from ClickHouse. Admin-only. func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { var schema TableSchema - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "GET", path: "/v1/schema", params: url.Values{"table": {t.table}}, @@ -98,7 +98,7 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e OK *bool `json:"ok"` Duplicate *bool `json:"duplicate"` } - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "POST", path: "/v1/ingest", params: url.Values{"table": {t.table}}, @@ -182,7 +182,7 @@ func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult Duplicates int `json:"duplicates"` Results []InsertRecordResult `json:"results"` } - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "POST", path: "/v1/ingest", params: url.Values{"table": {t.table}}, diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 39bb1efa..707e13ed 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -14,8 +14,8 @@ func TestTableRef_InsertSingle(t *testing.T) { var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path - json.NewDecoder(r.Body).Decode(&gotBody) - json.NewEncoder(w).Encode(map[string]any{"ok": true}) + _ = json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) @@ -40,7 +40,7 @@ func TestTableRef_InsertBatch(t *testing.T) { gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) @@ -79,7 +79,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) @@ -114,7 +114,7 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path - json.NewEncoder(w).Encode(map[string]any{"ok": true}) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) @@ -151,7 +151,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) @@ -174,7 +174,7 @@ func TestTableRef_Schema(t *testing.T) { if r.URL.Query().Get("table") != "clicks" { t.Errorf("want table=clicks") } - json.NewEncoder(w).Encode(TableSchema{ + _ = json.NewEncoder(w).Encode(TableSchema{ Name: "clicks", Columns: []Column{ {Name: "page", Type: "String"}, @@ -196,7 +196,7 @@ func TestTableRef_Schema(t *testing.T) { func TestTableRef_InsertDuplicate(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) + _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 6bcb8e5f..591e4b75 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -118,7 +118,7 @@ func (c *Client) Pipe(name string, params map[string]any) *PipeRef { // are decoded into []T; use [map[string]any] for dynamic schemas. func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { var rows []Row - err := doRequest(c.ctx, ctx, requestOptions{ + err := doRequest(ctx, c.ctx, requestOptions{ method: "POST", path: "/v1/admin/query", body: map[string]string{"sql": query}, From 38b23970b851fd68ee931919ac1d23ecc955ae1e Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 17:07:41 -0400 Subject: [PATCH 06/59] refactor(sdk): shrink Go SDK diff by 114 lines, zero functionality loss Source: extract helpers (errAborted, aggDefault, emptyInsertResult, marshalNDJSON, dlq.stats, sortedKeys), collapse like/not_like, inline trimTrailingSlashes, one-line IsRetryable, map-based numeric type lookup. Tests: table-driven parseErrorResponse, loop namespace nil checks, merge DLQ List+Table subcases, skipIfUnauthorized helper. Docs: dedupe Quick Start + Error Handling in index.md (link to README and reference.md), drop codegen type table from README (link to docs). --- clients/go/README.md | 14 +-- clients/go/client_test.go | 18 +--- clients/go/cmd/wavehouse-codegen/main.go | 52 ++++------ clients/go/dlq.go | 15 ++- clients/go/e2e_test.go | 22 +++-- clients/go/errors.go | 5 +- clients/go/errors_test.go | 118 ++++++++++++----------- clients/go/http.go | 15 ++- clients/go/live_query.go | 12 +-- clients/go/namespaces_test.go | 52 +++++----- clients/go/query_builder.go | 32 +++--- clients/go/stream.go | 11 +-- clients/go/table.go | 64 ++++++------ clients/go/wavehouse.go | 6 +- docs/src/content/docs/sdk/go/index.md | 78 +++------------ 15 files changed, 200 insertions(+), 314 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index e3af7bd4..6a313cc4 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -200,19 +200,7 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --package myapp ``` -The CLI reads `/v1/schema` (admin-only) and maps ClickHouse types to Go types: - -| ClickHouse | Go | -|---|---| -| `String`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | -| `UInt8/16/32/64` | `uint8/16/32/64` | -| `Int8/16/32/64` | `int8/16/32/64` | -| `Float32/64` | `float32/64` | -| `Bool` | `bool` | -| `Nullable(T)` | `*T` | -| `Array(T)` | `[]T` | -| `Map(K,V)` | `map[K]V` | -| `UInt128/256`, `Int128/256`, `Decimal*` | `string` | +See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/#codegen-cli). ## Error Handling diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 2b5239e1..020124b1 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -37,20 +37,10 @@ func TestNewClient_CustomMaxRetries(t *testing.T) { func TestNewClient_HasNamespaces(t *testing.T) { c := NewClient(Config{BaseURL: "http://localhost:8080"}) - if c.Schema == nil { - t.Fatal("Schema namespace nil") - } - if c.Policy == nil { - t.Fatal("Policy namespace nil") - } - if c.DLQ == nil { - t.Fatal("DLQ namespace nil") - } - if c.Sys == nil { - t.Fatal("Sys namespace nil") - } - if c.Pipes == nil { - t.Fatal("Pipes namespace nil") + for name, ns := range map[string]any{"Sys": c.Sys, "Schema": c.Schema, "Policy": c.Policy, "Pipes": c.Pipes, "DLQ": c.DLQ} { + if ns == nil { + t.Fatalf("%s namespace is nil", name) + } } } diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 213710e8..5dd9752d 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -175,30 +175,15 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric — map widths honestly. + // Numeric — map lookup. + if mapped, ok := map[string]string{ + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", + "Float32": "float32", "Float64": "float64", "BFloat16": "float32", + }[chType]; ok { + return mapped + } switch { - case chType == "UInt8": - return "uint8" - case chType == "UInt16": - return "uint16" - case chType == "UInt32": - return "uint32" - case chType == "UInt64": - return "uint64" - case chType == "Int8": - return "int8" - case chType == "Int16": - return "int16" - case chType == "Int32": - return "int32" - case chType == "Int64": - return "int64" - case chType == "Float32": - return "float32" - case chType == "Float64": - return "float64" - case chType == "BFloat16": - return "float32" case strings.HasPrefix(chType, "Decimal"), strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), @@ -268,15 +253,20 @@ func pascalCase(s string) string { return result } +func sortedKeys(m map[string]tableSchema) []string { + names := make([]string, 0, len(m)) + for name := range m { + names = append(names, name) + } + slices.Sort(names) + return names +} + func generate(schemas map[string]tableSchema, pkg string) string { var sb strings.Builder fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) - names := make([]string, 0, len(schemas)) - for name := range schemas { - names = append(names, name) - } - slices.Sort(names) + names := sortedKeys(schemas) for _, name := range names { schema := schemas[name] @@ -311,11 +301,7 @@ func main() { os.Exit(1) } - names := make([]string, 0, len(schemas)) - for name := range schemas { - names = append(names, name) - } - slices.Sort(names) + names := sortedKeys(schemas) fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) output := generate(schemas, args.pkg) diff --git a/clients/go/dlq.go b/clients/go/dlq.go index b523ab33..e833a41a 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -13,23 +13,20 @@ type DLQNamespace struct { // List returns DLQ statistics (message counts per table). Admin-only. func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { - var stats DLQStats - if err := doRequest(ctx, d.ctx, requestOptions{ - method: "GET", - path: "/v1/dlq/stats", - }, &stats); err != nil { - return nil, err - } - return &stats, nil + return d.stats(ctx, nil) } // Table returns DLQ stats filtered by table name. Admin-only. func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { + return d.stats(ctx, url.Values{"table": {name}}) +} + +func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, error) { var stats DLQStats if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", - params: url.Values{"table": {name}}, + params: params, }, &stats); err != nil { return nil, err } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index ea8aa7bc..130e6db9 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -257,10 +257,7 @@ func TestE2E_SQLQuery(t *testing.T) { rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") if err != nil { - // SQL requires admin role — skip gracefully if forbidden. - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: SQL query requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "SQL query") t.Fatalf("SQL query failed: %v", err) } if len(rows) != 1 { @@ -289,9 +286,7 @@ func TestE2E_PolicyGetSet(t *testing.T) { pol, err := c.Policy.Get(ctx) if err != nil { - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: Policy.Get requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "Policy.Get") t.Fatalf("Policy.Get failed: %v", err) } @@ -322,9 +317,7 @@ func TestE2E_PipesCRUD(t *testing.T) { Description: "E2E test pipe — safe to delete", } if err := c.Pipes.Set(ctx, pipeName, def); err != nil { - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: Pipes.Set requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "Pipes.Set") t.Fatalf("Pipes.Set (create) failed: %v", err) } @@ -424,6 +417,15 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { return row } +// skipIfUnauthorized skips the test when err indicates a 401 or 403, +// meaning the operation requires admin auth the current token lacks. +func skipIfUnauthorized(t *testing.T, err error, op string) { + t.Helper() + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("%s requires admin auth, skipping", op) + } +} + // isHTTPStatus checks whether err is a wavehouse.Error with the given status. func isHTTPStatus(err error, status int) bool { if err == nil { diff --git a/clients/go/errors.go b/clients/go/errors.go index f1fb48ac..f2ea1b24 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -33,10 +33,7 @@ func (e *Error) Error() string { // IsRetryable reports whether err wraps a retryable [*Error]. func IsRetryable(err error) bool { var e *Error - if errors.As(err, &e) { - return e.Retryable - } - return false + return errors.As(err, &e) && e.Retryable } // parseErrorResponse creates an Error from an HTTP response. diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go index 0c9dd05f..c5af80df 100644 --- a/clients/go/errors_test.go +++ b/clients/go/errors_test.go @@ -8,63 +8,69 @@ import ( "testing" ) -func TestParseErrorResponse_JSONError(t *testing.T) { - res := &http.Response{ - StatusCode: 404, - Body: io.NopCloser(strings.NewReader(`{"error":"unknown table: foo"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Status != 404 { - t.Fatalf("want status 404, got %d", e.Status) - } - if e.Code != "HTTP_404" { - t.Fatalf("want code HTTP_404, got %s", e.Code) - } - if e.Message != "unknown table: foo" { - t.Fatalf("want message 'unknown table: foo', got %s", e.Message) - } - if e.Retryable { - t.Fatal("4xx should not be retryable") - } -} - -func TestParseErrorResponse_MessageField(t *testing.T) { - res := &http.Response{ - StatusCode: 400, - Body: io.NopCloser(strings.NewReader(`{"message":"bad request"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "bad request" { - t.Fatalf("want 'bad request', got %s", e.Message) - } -} - -func TestParseErrorResponse_FallsBackToStatusText(t *testing.T) { - res := &http.Response{ - StatusCode: 500, - Body: io.NopCloser(strings.NewReader(`{"code":123}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "Internal Server Error" { - t.Fatalf("want status text fallback, got %s", e.Message) - } -} - -func TestParseErrorResponse_NonJSONBody(t *testing.T) { - res := &http.Response{ - StatusCode: 502, - Body: io.NopCloser(strings.NewReader("plain text")), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "Bad Gateway" { - t.Fatalf("want 'Bad Gateway', got %s", e.Message) +func TestParseErrorResponse(t *testing.T) { + tests := []struct { + name string + status int + body string + wantMsg string + wantCode string + wantRetry bool + nilDetails bool + }{ + { + name: "JSONError", + status: 404, + body: `{"error":"unknown table: foo"}`, + wantMsg: "unknown table: foo", + wantCode: "HTTP_404", + }, + { + name: "MessageField", + status: 400, + body: `{"message":"bad request"}`, + wantMsg: "bad request", + }, + { + name: "FallsBackToStatusText", + status: 500, + body: `{"code":123}`, + wantMsg: "Internal Server Error", + wantRetry: true, + }, + { + name: "NonJSONBody", + status: 502, + body: "plain text", + wantMsg: "Bad Gateway", + wantRetry: true, + nilDetails: true, + }, } - if e.Details != nil { - t.Fatal("details should be nil for non-JSON body") + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(tt.body)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Status != tt.status { + t.Fatalf("want status %d, got %d", tt.status, e.Status) + } + if tt.wantCode != "" && e.Code != tt.wantCode { + t.Fatalf("want code %s, got %s", tt.wantCode, e.Code) + } + if e.Message != tt.wantMsg { + t.Fatalf("want message %q, got %q", tt.wantMsg, e.Message) + } + if e.Retryable != tt.wantRetry { + t.Fatalf("want retryable=%v, got %v", tt.wantRetry, e.Retryable) + } + if tt.nilDetails && e.Details != nil { + t.Fatal("details should be nil") + } + }) } } diff --git a/clients/go/http.go b/clients/go/http.go index c7dc1c11..33c3785c 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -14,6 +14,8 @@ import ( "time" ) +var errAborted = &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + // httpContext carries per-client state needed by every request. type httpContext struct { baseURL string @@ -90,17 +92,12 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a if err != nil { // Context cancellation — return immediately, no retry. if ctx.Err() != nil { - return &Error{ - Status: 0, - Code: "ABORTED", - Message: "Request aborted", - Retryable: false, - } + return errAborted } lastErr = networkError(err) if attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } } continue @@ -145,7 +142,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a } } if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } lastErr = apiErr continue @@ -155,7 +152,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a // Retryable server errors (5xx). if apiErr.Retryable && attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } lastErr = apiErr continue diff --git a/clients/go/live_query.go b/clients/go/live_query.go index d6c665fe..1af2b770 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -48,16 +48,8 @@ func newLiveQuery( sub.Next(event) } }, - Status: func(s StreamStatus) { - if sub.Status != nil { - sub.Status(s) - } - }, - Error: func(err error) { - if sub.Error != nil { - sub.Error(err) - } - }, + Status: sub.Status, + Error: sub.Error, }) // Step 2–5: Fetch historical and flush. diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 18ff1b39..2801c1ab 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -97,32 +97,34 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { } } -func TestDLQNamespace_List(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) - })) - stats, err := c.DLQ.List(context.Background()) - if err != nil { - t.Fatal(err) - } - if stats.Total != 3 { - t.Fatalf("want total=3, got %d", stats.Total) - } -} +func TestDLQNamespace(t *testing.T) { + t.Run("List", func(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + })) + stats, err := c.DLQ.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } + }) -func TestDLQNamespace_Table(t *testing.T) { - var gotParam string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotParam = r.URL.Query().Get("table") - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) - })) - _, err := c.DLQ.Table(context.Background(), "clicks") - if err != nil { - t.Fatal(err) - } - if gotParam != "clicks" { - t.Fatalf("want table=clicks, got %s", gotParam) - } + t.Run("Table", func(t *testing.T) { + var gotParam string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotParam = r.URL.Query().Get("table") + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + })) + _, err := c.DLQ.Table(context.Background(), "clicks") + if err != nil { + t.Fatal(err) + } + if gotParam != "clicks" { + t.Fatalf("want table=clicks, got %s", gotParam) + } + }) } func TestPipesNamespace_CRUD(t *testing.T) { diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 6e9ab68b..65679ad1 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -82,42 +82,27 @@ func (q *QueryBuilder) Count(column, alias string) *QueryBuilder { // Sum adds a SUM aggregation. func (q *QueryBuilder) Sum(column, alias string) *QueryBuilder { - if alias == "" { - alias = "sum_" + column - } - return q.addAgg("sum", column, alias) + return q.aggDefault("sum", "sum_", column, alias) } // Avg adds an AVG aggregation. func (q *QueryBuilder) Avg(column, alias string) *QueryBuilder { - if alias == "" { - alias = "avg_" + column - } - return q.addAgg("avg", column, alias) + return q.aggDefault("avg", "avg_", column, alias) } // Min adds a MIN aggregation. func (q *QueryBuilder) Min(column, alias string) *QueryBuilder { - if alias == "" { - alias = "min_" + column - } - return q.addAgg("min", column, alias) + return q.aggDefault("min", "min_", column, alias) } // Max adds a MAX aggregation. func (q *QueryBuilder) Max(column, alias string) *QueryBuilder { - if alias == "" { - alias = "max_" + column - } - return q.addAgg("max", column, alias) + return q.aggDefault("max", "max_", column, alias) } // CountDistinct adds a COUNT DISTINCT aggregation. func (q *QueryBuilder) CountDistinct(column, alias string) *QueryBuilder { - if alias == "" { - alias = "count_distinct_" + column - } - return q.addAgg("countDistinct", column, alias) + return q.aggDefault("countDistinct", "count_distinct_", column, alias) } // Aggregate adds a custom aggregation function. @@ -230,6 +215,13 @@ func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *Li return newLiveQuery(stream, fetchFn, sub, q.state.filters) } +func (q *QueryBuilder) aggDefault(fn, prefix, column, alias string) *QueryBuilder { + if alias == "" { + alias = prefix + column + } + return q.addAgg(fn, column, alias) +} + func (q *QueryBuilder) addAgg(fn, column, alias string) *QueryBuilder { return q.clone(func(s *queryState) { s.aggregations = append(s.aggregations, Aggregation{Fn: fn, Column: column, Alias: alias}) diff --git a/clients/go/stream.go b/clients/go/stream.go index 84075005..2fabf1e2 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -426,20 +426,13 @@ func evaluateFilter(actual any, op string, expected any) bool { return ok && c <= 0 case "in": return evaluateIn(actual, expected) - case "like": + case "like", "not_like": aStr, aOK := actual.(string) eStr, eOK := expected.(string) if !aOK || !eOK { return false } - return matchLike(aStr, eStr) - case "not_like": - aStr, aOK := actual.(string) - eStr, eOK := expected.(string) - if !aOK || !eOK { - return false - } - return !matchLike(aStr, eStr) + return (op == "like") == matchLike(aStr, eStr) default: return false } diff --git a/clients/go/table.go b/clients/go/table.go index 3f32d030..d466d5b4 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "fmt" "net/url" "reflect" "strings" @@ -117,29 +118,35 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e return result, nil } -func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { - if len(rows) == 0 { - zero := 0 - return &InsertResult{ - OK: true, - Total: &zero, - Succeeded: &zero, - Failed: &zero, - Duplicates: &zero, - }, nil - } +func emptyInsertResult() *InsertResult { + z := 0 + return &InsertResult{OK: true, Total: &z, Succeeded: &z, Failed: &z, Duplicates: &z} +} + +func marshalNDJSON(n int, elem func(int) any) (string, error) { var sb strings.Builder - for i, row := range rows { + for i := range n { if i > 0 { sb.WriteByte('\n') } - raw, err := json.Marshal(row) + raw, err := json.Marshal(elem(i)) if err != nil { - return nil, err + return "", fmt.Errorf("wavehouse: marshal row %d: %w", i, err) } sb.Write(raw) } - return t.sendNDJSON(ctx, sb.String()) + return sb.String(), nil +} + +func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { + if len(rows) == 0 { + return emptyInsertResult(), nil + } + ndjson, err := marshalNDJSON(len(rows), func(i int) any { return rows[i] }) + if err != nil { + return nil, err + } + return t.sendNDJSON(ctx, ndjson) } // insertBatchReflect is the fallback batch path for any slice type other @@ -149,29 +156,14 @@ func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*Ins // insertBatch, so the server's per-record batch summary (failed, results, // etc.) is preserved instead of being silently dropped by insertSingle. func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { - n := rows.Len() - if n == 0 { - zero := 0 - return &InsertResult{ - OK: true, - Total: &zero, - Succeeded: &zero, - Failed: &zero, - Duplicates: &zero, - }, nil + if rows.Len() == 0 { + return emptyInsertResult(), nil } - var sb strings.Builder - for i := 0; i < n; i++ { - if i > 0 { - sb.WriteByte('\n') - } - raw, err := json.Marshal(rows.Index(i).Interface()) - if err != nil { - return nil, err - } - sb.Write(raw) + ndjson, err := marshalNDJSON(rows.Len(), func(i int) any { return rows.Index(i).Interface() }) + if err != nil { + return nil, err } - return t.sendNDJSON(ctx, sb.String()) + return t.sendNDJSON(ctx, ndjson) } func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 591e4b75..3e645eaf 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -14,6 +14,7 @@ package wavehouse import ( "context" "net/http" + "strings" ) // Config configures a [Client]. @@ -135,8 +136,5 @@ func (c *Client) createStream(table string, opts *StreamOptions) *StreamControll } func trimTrailingSlashes(s string) string { - for len(s) > 0 && s[len(s)-1] == '/' { - s = s[:len(s)-1] - } - return s + return strings.TrimRight(s, "/") } diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index db515ef2..d95a83d5 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -35,62 +35,31 @@ every example on these pages assumes it. ## Quick Start ```go -package main - import ( "context" "fmt" - "log" wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ) -func main() { - ctx := context.Background() - - // Create a client. Auth is optional — omit it for public/unauthenticated - // access (the server falls back to policy.default_role). - wh := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), - }) - - // Health check. - if err := wh.Sys.Health(ctx); err != nil { - log.Fatal(err) - } - - // Insert a row. - if _, err := wh.From("clicks").Insert(ctx, map[string]any{ - "page": "/home", "button": "signup", - }); err != nil { - log.Fatal(err) - } - - // Query with the fluent builder. - page, err := wh.From("clicks"). - Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Limit(10). - FetchUntyped(ctx) - if err != nil { - log.Fatal(err) - } - for _, row := range page.Data { - fmt.Println(row["page"], row["button"]) - } +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) - // Stream. - stream := wh.From("clicks").Stream(nil) - defer stream.Close() - unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ - Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, - Status: func(s wavehouse.StreamStatus) { fmt.Println("Stream:", s) }, - }) - defer unsub() +page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(context.Background()) +if err != nil { /* handle */ } +for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) } ``` +See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md) for more quick-start examples. + ## Creating a Client ```go @@ -179,10 +148,7 @@ parameter. ## Error Handling -Every SDK operation returns `(T, error)` — the idiomatic Go shape, and the -direct equivalent of the TypeScript SDK's -[`Result`](/sdk#result-type) discriminated union. Errors are always -`*wavehouse.Error`; unwrap with `errors.As`: +All SDK operations return `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`: ```go page, err := wh.From("clicks").Fetch(ctx) @@ -195,19 +161,7 @@ if err != nil { } ``` -```go -type Error struct { - Status int // HTTP status (0 for network/abort errors) - Code string // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" - Message string // Human-readable error message - Details map[string]any // Parsed response body, if available - Retryable bool // Whether the SDK would retry this error -} -``` - -`wavehouse.IsRetryable(err)` is a shortcut for `errors.As` + `.Retryable`. -The full error-code table lives in -[Reference → Error Handling](/sdk/go/reference#error-handling). +See [Reference → Error Handling](/sdk/go/reference/#error-handling) for retry behavior and error codes. ## Differences from the TypeScript SDK From 5a480bf9edb0c7f57d9eadd057c6f5f13fc675d8 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 12:59:51 -0400 Subject: [PATCH 07/59] =?UTF-8?q?fix(sdk):=20address=20PR=20#434=20review?= =?UTF-8?q?=20feedback=20=E2=80=94=20CI=20blockers=20+=2051=20review=20fin?= =?UTF-8?q?dings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI blockers: - Bump google.golang.org/grpc 1.82.1, golang.org/x/text 0.39.0, klauspost/compress 1.18.7 (GO-2026-6061/-5970/-5841) — vulncheck green - Drop forbidden trailing slash in sdk/go/index.md anchor link SDK behavior (CodeRabbit/Copilot review): - Retry: 429 now retryable, Retry-After honored for 429 and clamped to 30s, ±20% backoff jitter, dead 503 clause removed - NewClient uses a fresh http.Client instead of mutable http.DefaultClient - LiveQueryHandle: close state applied synchronously in Close(); dedup bound is the max backfilled timestamp compared as parsed time.Time - Stream reconnect backoff resets after a connection reaches live - Pagination replaces the cursor filter instead of stacking one per page - PolicyFilter operators marshal with omitempty (absent, never null) - Errors wrapped with operation context at every SDK boundary - codegen: unknown args rejected, 30s HTTP timeout, wrapped errors, field/type collision detection, WAVEHOUSE_AUTH env var for the token, Int64/UInt64 map to string (ClickHouse quotes 64-bit ints in JSON output) Tests: - httptest servers closed via t.Cleanup; handler captures synchronized; captureQueryBody uses io.ReadAll; multi-scenario tables use t.Run subtests - e2e: probe timeout, deterministic table pick returning its schema, polling instead of fixed sleeps, errors.As, unsupported-type skip - conformance: SDK errors logged, arg guards, normalizePath compares decoded query values (Go + TS), TS harness counts unhandled endpoints as skipped, stubs aligned to real server shapes, deterministic exit, cacheTTL fixture Docs/Makefile: policyDraft defined in admin example, (T, error) claim scoped to request-response ops, operator-key path documented for admin SQL, test-go-sdk-e2e documented, test-all includes test-go-sdk, --allow-parallel-runners on SDK lint, HTTPS caution for bearer tokens. --- CHANGELOG.md | 2 + Makefile | 5 +- clients/go/README.md | 2 +- clients/go/client_test.go | 15 +++ clients/go/cmd/wavehouse-codegen/main.go | 58 +++++++-- clients/go/conformance_test.go | 90 ++++++++++---- clients/go/e2e_test.go | 134 ++++++++++----------- clients/go/errors.go | 2 +- clients/go/errors_test.go | 30 ++--- clients/go/example_test.go | 10 +- clients/go/http.go | 14 ++- clients/go/http_test.go | 26 +++-- clients/go/live_query.go | 136 +++++++++++++--------- clients/go/namespaces_test.go | 36 ++++-- clients/go/pipes.go | 21 ++-- clients/go/policy.go | 16 ++- clients/go/query_builder.go | 10 +- clients/go/query_builder_test.go | 52 +++++---- clients/go/stream.go | 27 +++-- clients/go/sys.go | 12 +- clients/go/table.go | 6 +- clients/go/testdata/wire_cases.json | 15 +++ clients/go/types.go | 17 +-- clients/go/wavehouse.go | 9 +- docs/src/content/docs/sdk/go/admin.md | 5 +- docs/src/content/docs/sdk/go/index.md | 13 ++- docs/src/content/docs/sdk/go/queries.md | 10 +- docs/src/content/docs/sdk/go/reference.md | 46 +++++--- docs/src/content/docs/sdk/index.mdx | 4 +- go.mod | 20 ++-- go.sum | 40 +++---- tests/conformance/conformance_ts.mjs | 43 ++++++- 32 files changed, 599 insertions(+), 327 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e0ba485d..627487f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Go toolchain requirement bumped to 1.26.5** (`go.mod`): the `go` directive moves from `1.26.4` to `1.26.5` so local builds (via `GOTOOLCHAIN=auto`) and CI's `setup-go` (which reads `go.mod` via `go-version-file`) install a Go whose standard library clears the `govulncheck` findings GO-2026-5856 (`crypto/tls`) and GO-2026-4970 (`os`), both fixed in 1.26.5 — those findings were failing `make verify`'s `vulncheck` leaf (and with it the pre-commit hook) on every tree. Patch-level toolchain bump only — no source changes — and the released binaries pick up the patched stdlib too. +- **Dependency bumps clearing new govulncheck findings** (`go.mod`, `go.sum`): `google.golang.org/grpc` 1.81.1 → 1.82.1 (GO-2026-6061), `golang.org/x/text` 0.37.0 → 0.39.0 (GO-2026-5970), `github.com/klauspost/compress` 1.18.6 → 1.18.7 (GO-2026-5841) — all three were failing `make verify`'s `vulncheck` leaf after the advisories published. No API changes. + ### Security - **Policy `_in` is now enforced on both the row-`filter` and insert-`check` paths, closing a fail-open row-security gap** (`internal/policy/policy.go`, `internal/api/ingest.go`, `docs/src/content/docs/access-control.mdx`, plus tests in `internal/policy/policy_test.go`, `internal/api/ingest_test.go`): closes #224. The `Filter` schema accepted `_in` but the engine never read it: on the row-`filter`/SELECT path `resolveFilters` had no `_in` branch, so a row-security filter like `tenant_id: { _in: … }` produced **no `WHERE` predicate** and the role saw every row instead of its tenant subset (a fail-open, same family as #223); on the `check`/INSERT path only `_eq` was honored, silently dropping any other operator. `_in` now takes a single claim that resolves to a JSON **array** (the multi-tenant case — a token's `tenant_ids` list) and emits `col IN (?, …)` with one bound param per element; a scalar claim is a one-element set, and an empty/absent claim matches **no rows** (fail-closed) rather than widening to all of them. On the insert path an `_in` check requires the column be present and one of the set — there is no single value to auto-inject as `_eq` does, so an omitted column is rejected (`403 check failed`). The comparison operators are enforced on `filter` (`_eq`/`_neq`/`_gt`/`_lt`/`_in` all produce predicates now, so nothing is rejected there) and, on `check`, `_neq`/`_gt`/`_lt` become a loud config-load rejection (no insert-time semantics; `check` honors `_eq` + `_in`). The `_in` value stays a single templated string in the wire schema (Go `Filter.In`, SDK `PolicyFilter._in`), matching the established "set = array" shape of the caller-query `in` operator. diff --git a/Makefile b/Makefile index 0c614912..fdb9a7ab 100644 --- a/Makefile +++ b/Makefile @@ -364,7 +364,7 @@ lint-go: $(GOLANGCI_LINT) go-mod-download .PHONY: lint-go-sdk lint-go-sdk: $(GOLANGCI_LINT) - $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./...,) + $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./... --allow-parallel-runners,) .PHONY: lint-ts lint-ts: pnpm-install @@ -477,7 +477,7 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (10): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, +# Leaves (see verify-parallel prerequisites): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, # verify-go-sdk (go vet + gofumpt on the nested clients/go module) on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) for JS/TS + Markdown + prose; @@ -744,6 +744,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 + @$(MAKE) test-go-sdk @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 diff --git a/clients/go/README.md b/clients/go/README.md index 6a313cc4..c71e5644 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/# ## Error Handling -All SDK operations return `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`): +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`). Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`) deliver errors through callbacks instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 020124b1..22a45067 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -52,6 +52,7 @@ func TestClient_From(t *testing.T) { } _ = json.NewEncoder(w).Encode([]map[string]any{}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) _, _ = c.From("events").Fetch(context.Background()) } @@ -68,6 +69,7 @@ func TestClient_SQL(t *testing.T) { } _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") if err != nil { @@ -88,3 +90,16 @@ func TestStaticToken(t *testing.T) { t.Fatalf("want abc, got %s", token) } } + +func TestPolicyFilter_MarshalOperators(t *testing.T) { + empty := "" + raw, err := json.Marshal(PolicyFilter{Eq: &empty}) + if err != nil { + t.Fatal(err) + } + // Intentional empty-string comparison survives; unset operators are + // omitted entirely, never sent as null. + if string(raw) != `{"_eq":""}` { + t.Fatalf(`want {"_eq":""}, got %s`, raw) + } +} diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 5dd9752d..9775b525 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -3,7 +3,7 @@ // // Usage: // -// wavehouse-codegen --url http://localhost:8080 --out ./db.go --auth +// WAVEHOUSE_AUTH= wavehouse-codegen --url http://localhost:8080 --out ./db.go package main import ( @@ -15,6 +15,7 @@ import ( "os" "slices" "strings" + "time" "unicode" ) @@ -56,11 +57,19 @@ Options: --url, -u WaveHouse base URL (default: http://localhost:8080) --out, -o Output .go file path (default: ./wavehouse_types.go) --auth, -a Bearer token for authenticated /v1/schema endpoint + (prefer the WAVEHOUSE_AUTH env var — argv leaks into + shell history and process listings) --package, -p Go package name (default: main) --help, -h Show this help`) os.Exit(0) + default: + fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", os.Args[i]) + os.Exit(2) } } + if args.auth == "" { + args.auth = os.Getenv("WAVEHOUSE_AUTH") + } return args } @@ -80,14 +89,15 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc url := strings.TrimRight(baseURL, "/") + "/v1/schema" req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { - return nil, err + return nil, fmt.Errorf("build schema request for %s: %w", url, err) } if auth != "" { req.Header.Set("Authorization", "Bearer "+auth) } - resp, err := http.DefaultClient.Do(req) + client := &http.Client{Timeout: 30 * time.Second} + resp, err := client.Do(req) if err != nil { - return nil, err + return nil, fmt.Errorf("fetch schema from %s: %w", url, err) } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != 200 { @@ -97,7 +107,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc // Server returns either []tableSchema or map[string]tableSchema. var raw json.RawMessage if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { - return nil, err + return nil, fmt.Errorf("read schema response: %w", err) } // Try array first. var arr []tableSchema @@ -110,7 +120,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc } var m map[string]tableSchema if err := json.Unmarshal(raw, &m); err != nil { - return nil, err + return nil, fmt.Errorf("decode schema JSON: %w", err) } return m, nil } @@ -177,19 +187,25 @@ func chTypeToGo(chType string) string { } // Numeric — map lookup. if mapped, ok := map[string]string{ - "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", - "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Float32": "float32", "Float64": "float64", "BFloat16": "float32", }[chType]; ok { return mapped } switch { case strings.HasPrefix(chType, "Decimal"), + strings.HasPrefix(chType, "UInt64"), strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), + strings.HasPrefix(chType, "Int64"), strings.HasPrefix(chType, "Int128"), strings.HasPrefix(chType, "Int256"): - return "string" // big numbers are strings in JSON + // 64-bit and bigger integers (and decimals) are strings on the wire: + // ClickHouse's JSON output quotes them by default + // (output_format_json_quote_64bit_integers=1) and the server forwards + // the ClickHouse `data` array verbatim. + return "string" } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { @@ -262,19 +278,33 @@ func sortedKeys(m map[string]tableSchema) []string { return names } -func generate(schemas map[string]tableSchema, pkg string) string { +func generate(schemas map[string]tableSchema, pkg string) (string, error) { var sb strings.Builder fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) names := sortedKeys(schemas) + // pascalCase is not injective ("user_id" and "userId" both yield + // "UserId"), and format.Source only parses — it doesn't type-check — so + // a duplicate identifier would be written as a non-compiling file with a + // success message. Fail loudly instead. + seenTypes := make(map[string]string, len(names)) for _, name := range names { schema := schemas[name] typeName := pascalCase(name) + "Row" + if prev, dup := seenTypes[typeName]; dup { + return "", fmt.Errorf("tables %q and %q both map to type %q; rename one or generate separately", prev, name, typeName) + } + seenTypes[typeName] = name fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) + seenFields := make(map[string]string, len(schema.Columns)) for _, col := range schema.Columns { goType := chTypeToGo(col.Type) fieldName := pascalCase(col.Name) + if prev, dup := seenFields[fieldName]; dup { + return "", fmt.Errorf("table %q: columns %q and %q both map to field %q", name, prev, col.Name, fieldName) + } + seenFields[fieldName] = col.Name jsonTag := col.Name if col.HasDefault { jsonTag += ",omitempty" @@ -284,7 +314,7 @@ func generate(schemas map[string]tableSchema, pkg string) string { sb.WriteString("}\n\n") } - return sb.String() + return sb.String(), nil } func main() { @@ -304,7 +334,11 @@ func main() { names := sortedKeys(schemas) fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) - output := generate(schemas, args.pkg) + output, err := generate(schemas, args.pkg) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } // gofmt the output. A failure here means the generated source is not // valid Go (e.g. a table/column name produced an invalid identifier); diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 8809457a..e41c7b09 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -7,11 +7,22 @@ import ( "io" "net/http" "net/http/httptest" + "net/url" "reflect" "strings" "testing" ) +// logCallErr surfaces SDK-call errors that the conformance harness otherwise +// ignores — the assertions only inspect the captured request, but when a call +// fails before sending, the failure message should name the real cause. +func logCallErr(t *testing.T, err error) { + t.Helper() + if err != nil { + t.Logf("SDK call returned error (request may still be valid): %v", err) + } +} + // wireCasesJSON embeds the shared wire-format conformance fixture so the // test binary is self-contained: it works from a module archive or a // standalone checkout without depending on paths outside the Go module. @@ -103,18 +114,25 @@ func TestConformance_WireFormat(t *testing.T) { // Execute the case. switch tc.Endpoint { case "query": - q := c.From(tc.Table).Select() - q = applyOps(t, q, tc.Table, c, tc.Operations) - _, _ = q.FetchUntyped(ctx) + q := applyOps(t, tc.Table, c, tc.Operations) + _, err := q.FetchUntyped(ctx) + logCallErr(t, err) case "ingest": if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } data := tc.Operations[0].Args[0] - _, _ = c.From(tc.Table).Insert(ctx, data) + _, err := c.From(tc.Table).Insert(ctx, data) + logCallErr(t, err) } case "ingest_batch": if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } rawArr, ok := tc.Operations[0].Args[0].([]any) if !ok { t.Fatalf("batch insert args[0] is not an array") @@ -123,63 +141,73 @@ func TestConformance_WireFormat(t *testing.T) { for i, r := range rawArr { rows[i] = toStringMap(r) } - _, _ = c.From(tc.Table).Insert(ctx, rows) + _, err := c.From(tc.Table).Insert(ctx, rows) + logCallErr(t, err) } case "pipe": p := c.Pipe(tc.PipeName, tc.PipeParams) - _, _ = p.FetchUntyped(ctx) + _, err := p.FetchUntyped(ctx) + logCallErr(t, err) case "sql": - _, _ = SQL[map[string]any](ctx, c, tc.SQL) + _, err := SQL[map[string]any](ctx, c, tc.SQL) + logCallErr(t, err) case "health": - _ = c.Sys.Health(ctx) + logCallErr(t, c.Sys.Health(ctx)) case "schema_list": - _, _ = c.Schema.List(ctx) + _, err := c.Schema.List(ctx) + logCallErr(t, err) case "schema_refresh": - _ = c.Schema.Refresh(ctx) + logCallErr(t, c.Schema.Refresh(ctx)) case "policy_get": - _, _ = c.Policy.Get(ctx) + _, err := c.Policy.Get(ctx) + logCallErr(t, err) case "policy_set": var pol Policy if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { t.Fatalf("parse policy_body: %v", err) } - _ = c.Policy.Set(ctx, &pol) + logCallErr(t, c.Policy.Set(ctx, &pol)) case "policy_validate": var pol Policy if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { t.Fatalf("parse policy_body: %v", err) } - _, _ = c.Policy.Validate(ctx, &pol) + _, err := c.Policy.Validate(ctx, &pol) + logCallErr(t, err) case "dlq_list": - _, _ = c.DLQ.List(ctx) + _, err := c.DLQ.List(ctx) + logCallErr(t, err) case "dlq_table": - _, _ = c.DLQ.Table(ctx, tc.Table) + _, err := c.DLQ.Table(ctx, tc.Table) + logCallErr(t, err) case "pipes_list": - _, _ = c.Pipes.List(ctx) + _, err := c.Pipes.List(ctx) + logCallErr(t, err) case "pipes_get": - _, _ = c.Pipes.Get(ctx, tc.PipeName) + _, err := c.Pipes.Get(ctx, tc.PipeName) + logCallErr(t, err) case "pipes_set": var def PipeDef if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { t.Fatalf("parse pipe_def: %v", err) } - _ = c.Pipes.Set(ctx, tc.PipeName, def) + logCallErr(t, c.Pipes.Set(ctx, tc.PipeName, def)) case "pipes_delete": - _ = c.Pipes.Delete(ctx, tc.PipeName) + logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) default: t.Skipf("unhandled endpoint: %s", tc.Endpoint) @@ -235,7 +263,7 @@ func TestConformance_WireFormat(t *testing.T) { // applyOps replays the operation chain from the fixture onto a QueryBuilder. // Fixtures always put select first (mirroring real usage), so rebuilding on // select is safe and keeps this simple. -func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wireOp) *QueryBuilder { +func applyOps(t *testing.T, table string, c *Client, ops []wireOp) *QueryBuilder { t.Helper() q := c.From(table).Select() @@ -249,8 +277,15 @@ func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wire if len(op.Args) != 3 { t.Fatalf("where needs 3 args, got %d", len(op.Args)) } - col := op.Args[0].(string) - opStr := FilterOp(op.Args[1].(string)) + col, ok := op.Args[0].(string) + if !ok { + t.Fatalf("where: column arg is %T, want string", op.Args[0]) + } + rawOp, ok := op.Args[1].(string) + if !ok { + t.Fatalf("where: operator arg is %T, want string", op.Args[1]) + } + opStr := FilterOp(rawOp) val := op.Args[2] q = q.Where(col, opStr, val) case "count": @@ -371,7 +406,14 @@ func normalizeJSON(v any) any { } } +// normalizePath compares request URIs by meaning: same path, same decoded +// query values regardless of + vs %20 spelling or parameter order. A raw +// string replace would also rewrite literal + characters and stop asserting +// the encoding at all. func normalizePath(p string) string { - // Normalize URL encoding differences (+ vs %20 for spaces). - return strings.ReplaceAll(p, "+", "%20") + u, err := url.ParseRequestURI(p) + if err != nil { + return p + } + return u.Path + "?" + u.Query().Encode() } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 130e6db9..69b724e3 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -4,9 +4,11 @@ package wavehouse import ( "context" + "errors" "fmt" "net/http" "os" + "slices" "strings" "testing" "time" @@ -33,14 +35,15 @@ func e2eClient(t *testing.T) *Client { cfg.Auth = StaticToken(tok) } - // Probe the server before committing to the test. - probe, err := http.NewRequestWithContext( - context.Background(), "GET", base+"/v1/health", nil, - ) + // Probe the server before committing to the test. Bounded so a host that + // accepts the connection but never responds still yields the graceful skip. + probeCtx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + probe, err := http.NewRequestWithContext(probeCtx, "GET", base+"/v1/health", nil) if err != nil { t.Skipf("e2e: bad WAVEHOUSE_URL %q: %v", base, err) } - resp, err := http.DefaultClient.Do(probe) + resp, err := (&http.Client{Timeout: 3 * time.Second}).Do(probe) if err != nil { t.Skipf("e2e: server unreachable at %s: %v", base, err) } @@ -58,20 +61,47 @@ func marker(t *testing.T) string { return fmt.Sprintf("%s_%d", safe, time.Now().UnixNano()) } -// firstTable discovers a usable table from the schema list. Many E2E tests -// need a real table to insert/query — this avoids hardcoding a name. -func firstTable(t *testing.T, c *Client) string { +// firstTable discovers a usable table from the schema list and returns its +// schema alongside the name. Many E2E tests need a real table to insert/query +// — this avoids hardcoding a name, and returning the schema avoids a second +// Schema.List whose result set might no longer contain the chosen table. +// Sorted so every run picks the same table (map iteration order is random). +func firstTable(t *testing.T, c *Client) (string, TableSchema) { t.Helper() - ctx := context.Background() - schemas, err := c.Schema.List(ctx) + schemas, err := c.Schema.List(context.Background()) if err != nil { t.Skipf("e2e: cannot list schemas (auth?): %v", err) } + names := make([]string, 0, len(schemas)) for name := range schemas { - return name + names = append(names, name) + } + if len(names) == 0 { + t.Skip("e2e: no tables found — server has an empty schema") + } + slices.Sort(names) + return names[0], schemas[names[0]] +} + +// waitForRows polls the marker query until at least want rows are visible or +// the deadline expires. Ingestion is asynchronous — a fixed sleep fails on a +// loaded runner without any real defect. +func waitForRows(t *testing.T, c *Client, table, markerCol, mk string, want int) []map[string]any { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(max(want, 1)). + FetchUntyped(context.Background()) + if err != nil { + t.Fatalf("query for marker %q: %v", mk, err) + } + if len(page.Data) >= want || time.Now().After(deadline) { + return page.Data + } + time.Sleep(200 * time.Millisecond) } - t.Skip("e2e: no tables found — server has an empty schema") - return "" } // --------------------------------------------------------------------------- @@ -109,21 +139,9 @@ func TestE2E_SchemaList(t *testing.T) { func TestE2E_InsertAndQuery(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) + table, ts := firstTable(t, c) mk := marker(t) - // Discover columns so we can build a valid row. We need at least one - // string-ish column to inject our marker. Fall back to skipping if the - // table's schema doesn't have one we can use. - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts, ok := schemas[table] - if !ok { - t.Skipf("table %q vanished between discovery and use", table) - } - row := buildMarkerRow(t, ts, mk) markerCol := markerColumn(t, ts) @@ -135,21 +153,11 @@ func TestE2E_InsertAndQuery(t *testing.T) { t.Fatalf("Insert into %s: OK=false", table) } - // Allow a moment for async ingestion to settle. - time.Sleep(500 * time.Millisecond) - - // Query it back. - page, err := c.From(table).Select(markerCol). - Where(markerCol, OpEq, mk). - Limit(1). - FetchUntyped(ctx) - if err != nil { - t.Fatalf("Query failed: %v", err) - } - if len(page.Data) == 0 { + rows := waitForRows(t, c, table, markerCol, mk, 1) + if len(rows) == 0 { t.Fatal("Query returned zero rows — expected the inserted marker row") } - got, _ := page.Data[0][markerCol].(string) + got, _ := rows[0][markerCol].(string) if got != mk { t.Errorf("marker mismatch: want %q, got %q", mk, got) } @@ -158,13 +166,7 @@ func TestE2E_InsertAndQuery(t *testing.T) { func TestE2E_BatchInsert(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) - - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts := schemas[table] + table, ts := firstTable(t, c) mk := marker(t) markerCol := markerColumn(t, ts) @@ -183,30 +185,16 @@ func TestE2E_BatchInsert(t *testing.T) { t.Fatalf("Batch insert: OK=false") } - time.Sleep(500 * time.Millisecond) - - page, err := c.From(table).Select(markerCol). - Where(markerCol, OpEq, mk). - Limit(10). - FetchUntyped(ctx) - if err != nil { - t.Fatalf("Query after batch insert failed: %v", err) - } - if len(page.Data) < 3 { - t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(page.Data)) + got := waitForRows(t, c, table, markerCol, mk, 3) + if len(got) < 3 { + t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(got)) } } func TestE2E_QueryBuilder(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) - - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts := schemas[table] + table, ts := firstTable(t, c) // Pick two columns for a minimal projection. var cols []string @@ -216,6 +204,9 @@ func TestE2E_QueryBuilder(t *testing.T) { break } } + if len(cols) == 0 { + t.Skipf("e2e: table %q has no columns", table) + } page, err := c.From(table). Select(cols...). @@ -235,7 +226,7 @@ func TestE2E_QueryBuilder(t *testing.T) { func TestE2E_TypedFetch(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) + table, _ := firstTable(t, c) q := c.From(table).SelectAll().Limit(3) page, err := FetchTyped[map[string]any](ctx, q) @@ -408,7 +399,10 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { case strings.Contains(ct, "bool"): row[col.Name] = false default: - row[col.Name] = "" + // No safe synthetic value for this type (Array, Map, Tuple, UUID, + // ...) — an empty string would make the insert fail with a type + // error that looks like an SDK defect. + t.Skipf("e2e: table %q requires column %q of unsupported type %q", ts.Name, col.Name, col.Type) } } if !markerSet { @@ -426,12 +420,10 @@ func skipIfUnauthorized(t *testing.T, err error, op string) { } } -// isHTTPStatus checks whether err is a wavehouse.Error with the given status. +// isHTTPStatus checks whether err wraps a wavehouse.Error with the given status. func isHTTPStatus(err error, status int) bool { - if err == nil { - return false - } - if e, ok := err.(*Error); ok { + var e *Error + if errors.As(err, &e) { return e.Status == status } return false diff --git a/clients/go/errors.go b/clients/go/errors.go index f2ea1b24..8ec3b2e8 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -53,7 +53,7 @@ func parseErrorResponse(res *http.Response) *Error { msg = http.StatusText(res.StatusCode) } - retryable := res.StatusCode == http.StatusServiceUnavailable || res.StatusCode >= 500 + retryable := res.StatusCode >= 500 || res.StatusCode == http.StatusTooManyRequests return &Error{ Status: res.StatusCode, Code: fmt.Sprintf("HTTP_%d", res.StatusCode), diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go index c5af80df..160cb5c4 100644 --- a/clients/go/errors_test.go +++ b/clients/go/errors_test.go @@ -76,24 +76,28 @@ func TestParseErrorResponse(t *testing.T) { func TestParseErrorResponse_5xxRetryable(t *testing.T) { tests := []struct { + name string status int retryable bool }{ - {400, false}, - {403, false}, - {500, true}, - {503, true}, + {"BadRequest", 400, false}, + {"Forbidden", 403, false}, + {"TooManyRequests", 429, true}, + {"InternalServerError", 500, true}, + {"ServiceUnavailable", 503, true}, } for _, tt := range tests { - res := &http.Response{ - StatusCode: tt.status, - Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Retryable != tt.retryable { - t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) - } + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Retryable != tt.retryable { + t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) + } + }) } } diff --git a/clients/go/example_test.go b/clients/go/example_test.go index 03d8b518..b8e566c7 100644 --- a/clients/go/example_test.go +++ b/clients/go/example_test.go @@ -17,8 +17,9 @@ func ExampleNewClient() { }) // Health check — returns nil when the server is reachable. - err := client.Sys.Health(context.Background()) - _ = err + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } } func ExampleNewClient_withAuth() { @@ -40,12 +41,15 @@ func ExampleClient_From() { }) // Query with the builder. - page, _ := client.From("clicks"). + page, err := client.From("clicks"). Select("page", "button"). Where("page", wavehouse.OpEq, "/home"). OrderBy("page", "asc"). Limit(10). FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } for _, row := range page.Data { fmt.Println(row["page"]) diff --git a/clients/go/http.go b/clients/go/http.go index 33c3785c..f56a1235 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -8,6 +8,7 @@ import ( "fmt" "io" "math" + "math/rand/v2" "net/http" "net/url" "strconv" @@ -16,6 +17,10 @@ import ( var errAborted = &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} +// maxRetryAfter caps server-supplied Retry-After delays so a hostile or +// misconfigured server can't park the calling goroutine for hours. +const maxRetryAfter = 30 * time.Second + // httpContext carries per-client state needed by every request. type httpContext struct { baseURL string @@ -130,10 +135,10 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a apiErr := parseErrorResponse(res) _ = res.Body.Close() - // 503 with Retry-After: wait the specified duration. - if res.StatusCode == http.StatusServiceUnavailable { + // 503/429 with Retry-After: wait the specified duration (capped). + if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { - delay := 30 * time.Second + delay := maxRetryAfter if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { delay = time.Duration(secs) * time.Second } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { @@ -141,6 +146,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a delay = d } } + delay = min(delay, maxRetryAfter) if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { return errAborted } @@ -177,6 +183,8 @@ func backoff(attempt int) time.Duration { if ms > 30000 { ms = 30000 } + // ±20% jitter so clients failing at the same moment don't retry in lockstep. + ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic return time.Duration(ms) * time.Millisecond } diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 5a095c62..ede6543c 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -200,19 +200,25 @@ func TestDoRequest_EmptyResponse(t *testing.T) { func TestBackoff(t *testing.T) { tests := []struct { + name string attempt int - want time.Duration + base time.Duration }{ - {0, 1 * time.Second}, - {1, 2 * time.Second}, - {2, 4 * time.Second}, - {3, 8 * time.Second}, - {10, 30 * time.Second}, // capped at 30s + {"Attempt0", 0, 1 * time.Second}, + {"Attempt1", 1, 2 * time.Second}, + {"Attempt2", 2, 4 * time.Second}, + {"Attempt3", 3, 8 * time.Second}, + {"CappedAt30s", 10, 30 * time.Second}, } for _, tt := range tests { - got := backoff(tt.attempt) - if got != tt.want { - t.Errorf("backoff(%d) = %v, want %v", tt.attempt, got, tt.want) - } + t.Run(tt.name, func(t *testing.T) { + // backoff applies ±20% jitter around the exponential base. + lo := time.Duration(float64(tt.base) * 0.8) + hi := time.Duration(float64(tt.base) * 1.2) + got := backoff(tt.attempt) + if got < lo || got > hi { + t.Errorf("backoff(%d) = %v, want within [%v, %v]", tt.attempt, got, lo, hi) + } + }) } } diff --git a/clients/go/live_query.go b/clients/go/live_query.go index 1af2b770..d47ce618 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "sync" + "time" ) // LiveQueryHandle controls a live query that combines historical backfill @@ -10,7 +11,13 @@ import ( type LiveQueryHandle struct { stream *StreamController cancel context.CancelFunc + unsub func() closeOnce sync.Once + + mu sync.Mutex + buffer []StreamEvent + buffering bool + closed bool } // newLiveQuery starts a live query: opens the stream immediately, fetches @@ -19,43 +26,50 @@ func newLiveQuery( stream *StreamController, fetchFn func(ctx context.Context) ([]map[string]any, error), sub *StreamSubscriber, - filters []QueryFilter, ) *LiveQueryHandle { ctx, cancel := context.WithCancel(context.Background()) //nolint:gosec // cancel is called in Close() lq := &LiveQueryHandle{ - stream: stream, - cancel: cancel, + stream: stream, + cancel: cancel, + buffering: true, } - var ( - mu sync.Mutex - buffer []StreamEvent - buffering = true - closed = false - ) - - // Step 1: Subscribe to live events and buffer them. - stream.Subscribe(&StreamSubscriber{ + // Step 1: Subscribe to live events and buffer them. User callbacks are + // invoked outside lq.mu so a subscriber may call Close() without + // deadlocking. + lq.unsub = stream.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { - mu.Lock() - defer mu.Unlock() - if closed { + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() + return + } + if lq.buffering { + lq.buffer = append(lq.buffer, event) + lq.mu.Unlock() return } - if buffering { - buffer = append(buffer, event) - } else if sub.Next != nil { + lq.mu.Unlock() + if sub.Next != nil { sub.Next(event) } }, - Status: sub.Status, - Error: sub.Error, + Status: func(s StreamStatus) { + if !lq.isClosed() && sub.Status != nil { + sub.Status(s) + } + }, + Error: func(err error) { + if !lq.isClosed() && sub.Error != nil { + sub.Error(err) + } + }, }) // Step 2–5: Fetch historical and flush. go func() { rows, err := fetchFn(ctx) - if ctx.Err() != nil { + if ctx.Err() != nil || lq.isClosed() { return } @@ -65,19 +79,23 @@ func newLiveQuery( } if err != nil { - mu.Lock() - buffering = false - buffer = nil - mu.Unlock() + lq.mu.Lock() + lq.buffering = false + lq.buffer = nil + lq.mu.Unlock() return } - // Step 4: Deduplicate buffered events. - var lastTimestamp string - if len(rows) > 0 { - lastRow := rows[len(rows)-1] - if ts, ok := lastRow["received_timestamp"].(string); ok { - lastTimestamp = ts + // Step 4: Dedup bound — the maximum backfilled timestamp, compared as + // parsed times. OrderBy(..., "desc") makes the *last* row the oldest, + // and RFC3339 strings with varying fractional digits don't sort + // lexically, so neither "last row" nor raw string compare is safe. + var lastTS time.Time + for _, row := range rows { + if s, ok := row["received_timestamp"].(string); ok { + if ts, perr := time.Parse(time.RFC3339Nano, s); perr == nil && ts.After(lastTS) { + lastTS = ts + } } } @@ -85,31 +103,31 @@ func newLiveQuery( // buffer is provably empty under the lock — prevents concurrent // sub.Next calls and preserves delivery order. for { - mu.Lock() - if closed { - mu.Unlock() + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() return } - pending := buffer - buffer = nil + pending := lq.buffer + lq.buffer = nil if len(pending) == 0 { - buffering = false - mu.Unlock() + lq.buffering = false + lq.mu.Unlock() break } - mu.Unlock() + lq.mu.Unlock() for _, event := range pending { - mu.Lock() - c := closed - mu.Unlock() - if c { + if lq.isClosed() { return } - // <= dedupes events already delivered in the backfill. - // Sub-millisecond received_timestamp precision makes collisions rare. - if lastTimestamp != "" && event.Timestamp <= lastTimestamp { - continue + // Skip events already delivered in the backfill. + // Sub-millisecond received_timestamp precision makes + // boundary collisions rare. + if !lastTS.IsZero() { + if ts, perr := time.Parse(time.RFC3339Nano, event.Timestamp); perr == nil && !ts.After(lastTS) { + continue + } } if sub.Next != nil { sub.Next(event) @@ -118,21 +136,25 @@ func newLiveQuery( } }() - // Cleanup on context cancel. - go func() { - <-ctx.Done() - mu.Lock() - closed = true - buffer = nil - mu.Unlock() - }() - return lq } -// Close shuts down the live query and the underlying stream. +func (lq *LiveQueryHandle) isClosed() bool { + lq.mu.Lock() + defer lq.mu.Unlock() + return lq.closed +} + +// Close shuts down the live query and the underlying stream. The close state +// is applied synchronously: no new subscriber callbacks start after Close +// returns (a callback already in flight may still complete). func (lq *LiveQueryHandle) Close() { lq.closeOnce.Do(func() { + lq.mu.Lock() + lq.closed = true + lq.buffer = nil + lq.mu.Unlock() + lq.unsub() lq.cancel() lq.stream.Close() }) diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 2801c1ab..76acc176 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -5,11 +5,14 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "sync" "testing" ) -func nsClient(handler http.Handler) *Client { +func nsClient(t *testing.T, handler http.Handler) *Client { + t.Helper() srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) return NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), @@ -18,7 +21,7 @@ func nsClient(handler http.Handler) *Client { } func TestSysNamespace_Health(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/health" { t.Errorf("want /v1/health, got %s", r.URL.Path) } @@ -31,7 +34,7 @@ func TestSysNamespace_Health(t *testing.T) { } func TestSchemaNamespace_List(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } @@ -49,22 +52,27 @@ func TestSchemaNamespace_List(t *testing.T) { } func TestSchemaNamespace_Refresh(t *testing.T) { + var mu sync.Mutex var gotMethod string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotMethod = r.Method + mu.Unlock() w.WriteHeader(200) })) err := c.Schema.Refresh(context.Background()) if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotMethod != "POST" { t.Fatalf("want POST, got %s", gotMethod) } } func TestPolicyNamespace_GetSetValidate(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) @@ -99,7 +107,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace(t *testing.T) { t.Run("List", func(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) @@ -112,15 +120,20 @@ func TestDLQNamespace(t *testing.T) { }) t.Run("Table", func(t *testing.T) { + var mu sync.Mutex var gotParam string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotParam = r.URL.Query().Get("table") + mu.Unlock() _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) })) _, err := c.DLQ.Table(context.Background(), "clicks") if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotParam != "clicks" { t.Fatalf("want table=clicks, got %s", gotParam) } @@ -128,7 +141,7 @@ func TestDLQNamespace(t *testing.T) { } func TestPipesNamespace_CRUD(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { @@ -171,12 +184,15 @@ func TestPipesNamespace_CRUD(t *testing.T) { } func TestPipeRef_Fetch(t *testing.T) { + var mu sync.Mutex var gotPath, gotMethod string var gotBody map[string]any - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotPath = r.URL.Path gotMethod = r.Method _ = json.NewDecoder(r.Body).Decode(&gotBody) + mu.Unlock() _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) })) rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) @@ -186,6 +202,8 @@ func TestPipeRef_Fetch(t *testing.T) { if len(rows) != 1 { t.Fatalf("want 1 row, got %d", len(rows)) } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/pipes/top_pages" { t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) } diff --git a/clients/go/pipes.go b/clients/go/pipes.go index 767078a6..71b25720 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "fmt" "net/url" ) @@ -17,7 +18,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { method: "GET", path: "/v1/admin/pipes", }, &pipes); err != nil { - return nil, err + return nil, fmt.Errorf("list pipes: %w", err) } return pipes, nil } @@ -29,26 +30,32 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { method: "GET", path: "/v1/admin/pipes/" + url.PathEscape(name), }, &pipe); err != nil { - return nil, err + return nil, fmt.Errorf("get pipe %q: %w", name, err) } return &pipe, nil } // Set creates or updates a pipe. Admin-only. func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/pipes/" + url.PathEscape(name), body: def, - }, nil) + }, nil); err != nil { + return fmt.Errorf("set pipe %q: %w", name, err) + } + return nil } // Delete removes a pipe by name. Admin-only. func (p *PipesNamespace) Delete(ctx context.Context, name string) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", path: "/v1/admin/pipes/" + url.PathEscape(name), - }, nil) + }, nil); err != nil { + return fmt.Errorf("delete pipe %q: %w", name, err) + } + return nil } // PipeDef is the definition body for creating/updating a pipe (Pipe minus name). @@ -79,7 +86,7 @@ func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { path: "/v1/pipes/" + url.PathEscape(p.name), body: body, }, &rows); err != nil { - return nil, err + return nil, fmt.Errorf("execute pipe %q: %w", p.name, err) } return rows, nil } diff --git a/clients/go/policy.go b/clients/go/policy.go index c8a2306b..9fbc5bb6 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // PolicyNamespace provides admin-only access-control policy management. type PolicyNamespace struct { @@ -14,18 +17,21 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { method: "GET", path: "/v1/admin/policy", }, &pol); err != nil { - return nil, err + return nil, fmt.Errorf("get policy: %w", err) } return &pol, nil } // Set replaces the entire access-control policy. Admin-only. func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/policy", body: pol, - }, nil) + }, nil); err != nil { + return fmt.Errorf("set policy: %w", err) + } + return nil } // Validate checks a policy without applying it (dry run). Admin-only. @@ -36,7 +42,7 @@ func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*Validatio path: "/v1/admin/policy/validate", body: pol, }, &result); err != nil { - return nil, err + return nil, fmt.Errorf("validate policy: %w", err) } return &result, nil } diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 65679ad1..2855d367 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -212,7 +212,7 @@ func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *Li } return page.Data, nil } - return newLiveQuery(stream, fetchFn, sub, q.state.filters) + return newLiveQuery(stream, fetchFn, sub) } func (q *QueryBuilder) aggDefault(fn, prefix, column, alias string) *QueryBuilder { @@ -293,6 +293,14 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro } next := q.clone(func(s *queryState) { + // Replace an existing cursor filter instead of appending — otherwise + // page N carries N stacked filters on the cursor column. + for i := range s.filters { + if s.filters[i].Column == cursor.Column && s.filters[i].Op == cursorOp { + s.filters[i].Value = lastValue + return + } + } s.filters = append(s.filters, QueryFilter{ Column: cursor.Column, Op: cursorOp, diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 84261a05..71d2808e 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -3,32 +3,44 @@ package wavehouse import ( "context" "encoding/json" + "io" "net/http" "net/http/httptest" + "sync" "testing" ) -func queryTestCtx(handler http.Handler) (*Client, *httptest.Server) { +func queryTestCtx(t *testing.T, handler http.Handler) *Client { + t.Helper() srv := httptest.NewServer(handler) - c := NewClient(Config{ + t.Cleanup(srv.Close) + return NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}, }) - return c, srv } func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { t.Helper() + // body is written on the server goroutine and read on the test goroutine; + // the mutex is what makes that visible under -race. + var mu sync.Mutex var body []byte wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw := make([]byte, 32*1024) - n, _ := r.Body.Read(raw) - body = raw[:n] + raw, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("read request body: %v", err) + } + mu.Lock() + body = raw + mu.Unlock() handler.ServeHTTP(w, r) }) - c, _ := queryTestCtx(wrapper) + c := queryTestCtx(t, wrapper) return c, func() map[string]any { + mu.Lock() + defer mu.Unlock() var m map[string]any _ = json.Unmarshal(body, &m) return m @@ -41,7 +53,7 @@ var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { }) func TestQueryBuilder_Immutability(t *testing.T) { - c, _ := queryTestCtx(emptyRows) + c := queryTestCtx(t, emptyRows) b1 := c.From("clicks").Select("page") b2 := b1.Where("score", OpGt, 10) if b1 == b2 { @@ -121,14 +133,16 @@ func TestQueryBuilder_AllOperators(t *testing.T) { {OpNotLike, "not_like"}, } for _, tt := range ops { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) - body := getBody() - filters := body["filters"].([]any) - f := filters[0].(map[string]any) - if f["op"] != tt.wire { - t.Errorf("%s: want wire op %s, got %s", tt.sdk, tt.wire, f["op"]) - } + t.Run(tt.wire, func(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) + body := getBody() + filters := body["filters"].([]any) + f := filters[0].(map[string]any) + if f["op"] != tt.wire { + t.Errorf("want wire op %s, got %s", tt.wire, f["op"]) + } + }) } } @@ -198,10 +212,9 @@ func TestQueryBuilder_TimeRange(t *testing.T) { } func TestQueryBuilder_Pagination_HasMore(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) if err != nil { @@ -216,10 +229,9 @@ func TestQueryBuilder_Pagination_HasMore(t *testing.T) { } func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) if err != nil { diff --git a/clients/go/stream.go b/clients/go/stream.go index 2fabf1e2..af14cd7f 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -201,7 +201,7 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } - lastID, err := sc.connect(ctx, hctx, table, since) + lastID, live, err := sc.connect(ctx, hctx, table, since) // Persist the last event ID so the next reconnect resumes from it. if lastID != "" { since = lastID @@ -210,6 +210,12 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } + // A connection that reached "live" resets the backoff so a long-lived + // stream doesn't inherit a maxed-out delay on its first drop. + if live { + attempt = 0 + } + if err != nil { sc.emitError(&Error{ Status: 0, @@ -232,11 +238,12 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } // connect opens a single SSE connection and reads events until it closes. -// Returns the last seen event ID (empty if none) and any error. -func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, error) { +// Returns the last seen event ID (empty if none), whether the connection +// reached the live state, and any error. +func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { u, err := url.Parse(hctx.baseURL + "/v1/stream") if err != nil { - return "", err + return "", false, err } q := u.Query() q.Set("table", table) @@ -249,7 +256,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if hctx.auth != nil { token, err := hctx.auth(ctx) if err != nil { - return "", fmt.Errorf("auth: %w", err) + return "", false, fmt.Errorf("auth: %w", err) } if token != "" { authHeader = "Bearer " + token @@ -260,7 +267,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) if err != nil { - return "", err + return "", false, err } req.Header.Set("Accept", "text/event-stream") req.Header.Set("Cache-Control", "no-cache") @@ -270,12 +277,12 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table resp, err := hctx.httpClient.Do(req) if err != nil { - return "", err + return "", false, err } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + return "", false, fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) } sc.setStatus(StatusLive) @@ -288,7 +295,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table for scanner.Scan() { if ctx.Err() != nil { - return lastID, nil + return lastID, true, nil } line := scanner.Text() @@ -324,7 +331,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table } } - return lastID, scanner.Err() + return lastID, true, scanner.Err() } // sseMessage matches the server's SSE event JSON shape. diff --git a/clients/go/sys.go b/clients/go/sys.go index e52c252c..ed875cdb 100644 --- a/clients/go/sys.go +++ b/clients/go/sys.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // SysNamespace provides system health checks. type SysNamespace struct { @@ -10,8 +13,11 @@ type SysNamespace struct { // Health pings the server's public /v1/health endpoint. Returns nil when the // server is reachable and past boot, or an error describing the failure. func (s *SysNamespace) Health(ctx context.Context) error { - return doRequest(ctx, s.ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/health", - }, nil) + }, nil); err != nil { + return fmt.Errorf("health check: %w", err) + } + return nil } diff --git a/clients/go/table.go b/clients/go/table.go index d466d5b4..02bf4a0a 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -84,7 +84,7 @@ func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { path: "/v1/schema", params: url.Values{"table": {t.table}}, }, &schema); err != nil { - return nil, err + return nil, fmt.Errorf("get schema for table %q: %w", t.table, err) } return &schema, nil } @@ -105,7 +105,7 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e params: url.Values{"table": {t.table}}, body: data, }, &res); err != nil { - return nil, err + return nil, fmt.Errorf("insert into %q: %w", t.table, err) } ok := true if res.OK != nil { @@ -181,7 +181,7 @@ func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult rawBody: ndjson, contentType: "application/x-ndjson", }, &res); err != nil { - return nil, err + return nil, fmt.Errorf("ingest into %q: %w", t.table, err) } result := &InsertResult{ OK: res.Failed == 0, diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index 4ec8613d..2c27b46a 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -543,5 +543,20 @@ "pipe_name": "my_pipe", "expected_path": "/v1/admin/pipes/my_pipe", "expected_method": "DELETE" + }, + { + "name": "cacheTTL is client-side only and not sent on the wire", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "cacheTTL", "args": [60] }, + { "method": "limit", "args": [5] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 5 + } } ] diff --git a/clients/go/types.go b/clients/go/types.go index b59333fd..0d17699f 100644 --- a/clients/go/types.go +++ b/clients/go/types.go @@ -179,15 +179,16 @@ type RolePermissions struct { MaxMemoryUsage any `json:"max_memory_usage,omitempty"` } -// PolicyFilter describes a policy filter predicate. Fields are pointers so an -// intentional empty-string comparison (e.g. Eq pointing at "") round-trips -// distinctly from an absent operator, matching the server's semantics. +// PolicyFilter describes a policy filter predicate. Fields are pointers with +// omitempty so an intentional empty-string comparison (e.g. Eq pointing at "") +// is sent as "", while an unset operator is omitted entirely — never null — +// matching the server's absent-operator semantics. type PolicyFilter struct { - Eq *string `json:"_eq"` - Neq *string `json:"_neq"` - Gt *string `json:"_gt"` - Lt *string `json:"_lt"` - In *string `json:"_in"` + Eq *string `json:"_eq,omitempty"` + Neq *string `json:"_neq,omitempty"` + Gt *string `json:"_gt,omitempty"` + Lt *string `json:"_lt,omitempty"` + In *string `json:"_in,omitempty"` } // ValidationResult is the response from policy validation. diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 3e645eaf..a4d338e8 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -8,11 +8,12 @@ // client := wavehouse.NewClient(wavehouse.Config{ // BaseURL: "http://localhost:8080", // }) -// rows, err := client.From("clicks").SelectAll().Fetch(ctx) +// rows, err := client.From("clicks").SelectAll().FetchUntyped(ctx) package wavehouse import ( "context" + "fmt" "net/http" "strings" ) @@ -73,7 +74,9 @@ func NewClient(cfg Config) *Client { hc := cfg.HTTPClient if hc == nil { - hc = http.DefaultClient + // Not http.DefaultClient: it's mutable global state another package + // could reconfigure (timeout, transport, redirects) after we're built. + hc = &http.Client{} } c := &Client{ @@ -125,7 +128,7 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { body: map[string]string{"sql": query}, }, &rows) if err != nil { - return nil, err + return nil, fmt.Errorf("sql query: %w", err) } return rows, nil } diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index a0941433..957713fd 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -43,7 +43,7 @@ policy, err := wh.Policy.Get(ctx) // Update policy. tenantFilter := "{{ jwt.app_metadata.tenant_id }}" -err = wh.Policy.Set(ctx, &wavehouse.Policy{ +policyDraft := &wavehouse.Policy{ DefaultRole: "viewer", Tables: map[string]wavehouse.TablePolicy{ "clicks": { @@ -58,7 +58,8 @@ err = wh.Policy.Set(ctx, &wavehouse.Policy{ }, }, }, -}) +} +err = wh.Policy.Set(ctx, policyDraft) // Validate without applying (dry run). result, err := wh.Policy.Validate(ctx, policyDraft) diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index d95a83d5..2202f95d 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -83,7 +83,7 @@ wh := wavehouse.NewClient(wavehouse.Config{ | `BaseURL` | `string` | — | WaveHouse server URL (required) | | `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | | `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | `http.DefaultClient` | Override for custom TLS, proxies, or test transports | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports | ### `ClientOptions` @@ -118,6 +118,13 @@ parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser concern only; see its [equivalent note](/sdk#creating-a-client)). ::: +:::caution[Use HTTPS for authenticated non-local servers] +The SDK doesn't forbid `http://` base URLs — local development and +private-network deployments rely on them — but a bearer token sent over +plaintext HTTP is readable by anything on the path. Point authenticated +clients at `https://` endpoints outside a trusted network. +::: + ## Typed Rows (Generics) Pass a row type as a type parameter to get results decoded straight into @@ -148,7 +155,7 @@ parameter. ## Error Handling -All SDK operations return `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close` — deliver errors through callbacks instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) @@ -161,7 +168,7 @@ if err != nil { } ``` -See [Reference → Error Handling](/sdk/go/reference/#error-handling) for retry behavior and error codes. +See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry behavior and error codes. ## Differences from the TypeScript SDK diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 2b56fd5b..59c588bc 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -361,10 +361,12 @@ for page.HasMore && page.Next != nil { ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT -must resolve to the policy admin role (`admin_role`, `"admin"` by default). -A request with no token, or an invalid/expired one, falls back to the -`default_role` and is rejected. Package-level generic function — use +Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, +the token must resolve to the policy admin role (`admin_role`, `"admin"` by +default) — a JWT request with no token, or an invalid/expired one, falls +back to the `default_role` and is rejected. Alternatively, a configured +operator key (`Authorization: Operator ` or `X-Operator-Key`) +authorizes `/v1/admin/*` without a JWT. Package-level generic function — use `map[string]any` for a dynamic/unknown schema. ```go diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 7e6e90cb..75cf8cad 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -4,7 +4,7 @@ description: "Error codes, context cancellation, the full API tree, and the code --- Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: -cancellation, the error model behind every SDK call's `(T, error)` return, +cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. @@ -38,10 +38,13 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling -The SDK never panics on API or network failures — every operation returns -`(T, error)`, and errors are always `*wavehouse.Error` (unwrap with -`errors.As`). This is the direct Go equivalent of the TypeScript SDK's "the -SDK never throws" guarantee. +The SDK never panics on API or network failures — every request-response +operation (queries, ingest, pipes, admin) returns `(T, error)`, and errors +are always `*wavehouse.Error` (unwrap with `errors.As`). Streaming lifecycle +methods (`Stream`, `Subscribe`, `Close`) don't return `(T, error)`; stream +errors are delivered via the subscriber's `Error` callback. This is the +direct Go equivalent of the TypeScript SDK's "the SDK never throws" +guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -135,9 +138,9 @@ Generate Go structs from a running WaveHouse instance. The module ships a `wavehouse-codegen` command under `cmd/`: ```bash +export WAVEHOUSE_AUTH= # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --url http://localhost:8080 \ - --auth \ --out ./db_types.go \ --package myapp ``` @@ -149,8 +152,9 @@ go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev -server, pass an admin-role token with `--auth ` or the request is -denied with `403`. +server, provide an admin-role token or the request is denied with `403`. +Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with +`--auth ` ends up in shell history and process listings. **Options:** @@ -158,7 +162,7 @@ denied with `403`. |------|-------------|---------| | `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | | `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token (if auth required) | — | +| `--auth`, `-a` | Bearer token (if auth required); prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | | `--package`, `-p` | Go package name for the generated file | `main` | | `--help`, `-h` | Show usage and exit | — | @@ -195,11 +199,11 @@ tag. |------------------|---------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | | `Bool` | `bool` | -| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | -| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | +| `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | | `Float32` | `float32` | | `Float64` | `float64` | -| `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (big numbers are strings in JSON) | +| `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | | `Array(T)` | `[]T` | @@ -207,9 +211,10 @@ tag. | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** (`UInt32` → `uint32`, not -a generic `number`), since Go — unlike TypeScript — has native fixed-width -integer types. +codegen preserves ClickHouse's integer **widths** up to 32 bits (`UInt32` → +`uint32`, not a generic `number`), since Go — unlike TypeScript — has +native fixed-width integer types. 64-bit integers stay `string` because +that is what actually arrives on the wire. ## Testing @@ -227,6 +232,17 @@ cd clients/go go test ./... ``` +E2E tests (build tag `e2e`) run against a live WaveHouse instance and have +their own Make target, separate from the repo's `make test-e2e`: + +```bash +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH= make test-go-sdk-e2e +``` + +`WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is +optional (admin-only cases skip without it). When the server is unreachable +the suite skips instead of failing. + Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's `make test-e2e` harness — see the TypeScript SDK's [E2E Testing](/sdk/reference#e2e-testing) section for that suite's diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index aa4ca864..659fe9fe 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -13,7 +13,9 @@ WaveHouse also ships an official Go SDK `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin -endpoints carries over conceptually — only the language idioms differ. +endpoints carries over conceptually, but API and lifecycle details differ — +Go uses context-first calls and package-level generics, and streams must be +closed explicitly. ::: ## Installation diff --git a/go.mod b/go.mod index 0ea6370e..db6d4e11 100644 --- a/go.mod +++ b/go.mod @@ -47,7 +47,7 @@ require ( go.opentelemetry.io/otel/trace v1.44.0 go.opentelemetry.io/proto/otlp v1.10.0 golang.org/x/sync v0.21.0 - google.golang.org/grpc v1.81.1 + google.golang.org/grpc v1.82.1 gopkg.in/yaml.v3 v3.0.1 ) @@ -131,7 +131,7 @@ require ( github.com/jedib0t/go-pretty/v6 v6.7.10 // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/joho/godotenv v1.5.1 // indirect - github.com/klauspost/compress v1.18.6 // indirect + github.com/klauspost/compress v1.18.7 // indirect github.com/knadh/profiler v0.2.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/kr/text v0.2.0 // indirect @@ -196,17 +196,17 @@ require ( go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect golang.org/x/arch v0.26.0 // indirect - golang.org/x/crypto v0.52.0 // indirect + golang.org/x/crypto v0.53.0 // indirect golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa // indirect golang.org/x/image v0.38.0 // indirect - golang.org/x/mod v0.35.0 // indirect - golang.org/x/net v0.55.0 // indirect - golang.org/x/sys v0.45.0 // indirect - golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e // indirect - golang.org/x/term v0.43.0 // indirect - golang.org/x/text v0.37.0 // indirect + golang.org/x/mod v0.37.0 // indirect + golang.org/x/net v0.56.0 // indirect + golang.org/x/sys v0.46.0 // indirect + golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect + golang.org/x/term v0.44.0 // indirect + golang.org/x/text v0.39.0 // indirect golang.org/x/time v0.15.0 // indirect - golang.org/x/tools v0.44.0 // indirect + golang.org/x/tools v0.47.0 // indirect golang.org/x/vuln v1.3.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect diff --git a/go.sum b/go.sum index d34ebedc..4990f509 100644 --- a/go.sum +++ b/go.sum @@ -221,8 +221,8 @@ github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwA github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= -github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao= -github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw= +github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/knadh/profiler v0.2.0 h1:jaY0xlQs8iaWxKdvGHOftaZnX7d8l7yrCGQPSecwnng= github.com/knadh/profiler v0.2.0/go.mod h1:LqNkAu++MfFkbEDA63AmRaIf6UkGrLXyZ5VQQdekZiI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= @@ -439,23 +439,23 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= -golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= +golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto= +golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio= golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa h1:Zt3DZoOFFYkKhDT3v7Lm9FDMEV06GpzjG2jrqW+QTE0= golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa/go.mod h1:K79w1Vqn7PoiZn+TkNpx3BUWUQksGO3JcVX6qIjytmA= golang.org/x/image v0.38.0 h1:5l+q+Y9JDC7mBOMjo4/aPhMDcxEptsX+Tt3GgRQRPuE= golang.org/x/image v0.38.0/go.mod h1:/3f6vaXC+6CEanU4KJxbcUZyEePbyKbaLoDOe4ehFYY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ= +golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= -golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= +golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -475,27 +475,27 @@ golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= -golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e h1:OXgN37M6hqjaAvb7CJK9vJ+7Z/6lvIm5bXho5poo/Wk= -golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE= +golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= +golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 h1:nwGZBCt+FnXUrGsj5vjzAsEmkcaFvd82BbOjECiFYZc= +golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57/go.mod h1:3AWMyWHS+caVoiEXpiq6+tzKA40J4vQT3MYr80ZtQpc= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= -golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= +golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc= +golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= -golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus= +golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= @@ -512,8 +512,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1: google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= -google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ= -google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I= +google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE= +google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 8eafc796..6d8630df 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -18,7 +18,14 @@ import { fileURLToPath } from "node:url"; const __dirname = dirname(fileURLToPath(import.meta.url)); // Import the built SDK. -const { createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js")); +let createClient; +try { + ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); +} catch (err) { + console.error("Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`)."); + console.error(err.message); + process.exit(1); +} const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); @@ -43,7 +50,7 @@ const server = createServer((req, res) => { if (req.url?.startsWith("/v1/dlq")) { res.end(JSON.stringify({ tables: {}, total: 0 })); } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { - res.end(JSON.stringify({})); + res.end(JSON.stringify([])); } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { res.end(JSON.stringify({ valid: true })); } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { @@ -53,13 +60,15 @@ const server = createServer((req, res) => { } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { res.end(JSON.stringify([])); } else if (req.url?.startsWith("/v1/ingest")) { + // Same shapes the real server returns (internal/api/ingest.go). if (lastCapture.contentType === "application/x-ndjson") { res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); } else { res.end(JSON.stringify({ ok: true })); } } else if (req.url === "/v1/health") { - res.end(""); + // Real server shape (internal/api/health.go). + res.end(JSON.stringify({ status: "ok" })); } else { res.end(JSON.stringify([])); } @@ -124,8 +133,17 @@ function applyQueryOps(wh, table, operations) { return q; } +// Compare request URIs by meaning: same path, same decoded query values, +// regardless of + vs %20 spelling or parameter order (mirrors the Go harness). function normalizePath(p) { - return p.replace(/\+/g, "%20"); + let u; + try { + u = new URL(p, "http://conformance.invalid"); + } catch { + return p; + } + u.searchParams.sort(); + return `${u.pathname}?${u.searchParams.toString()}`; } function deepEqual(a, b) { @@ -147,6 +165,8 @@ function sortKeys(v) { let passed = 0; let failed = 0; +let skipped = 0; +const skippedNames = []; const failures = []; for (const tc of cases) { @@ -213,7 +233,10 @@ for (const tc of cases) { await wh.pipes.delete(tc.pipe_name); break; default: - passed++; + // Not a pass — the Go harness skips these too. Fixture cases with a + // new endpoint value must be wired up here before they count. + skipped++; + skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); continue; } @@ -261,9 +284,16 @@ for (const tc of cases) { } } +server.closeAllConnections?.(); server.close(); -console.log(`\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${cases.length} total\n`); +console.log( + `\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${skipped} skipped, ${cases.length} total\n`, +); + +for (const name of skippedNames) { + console.log(` - skipped: ${name}`); +} for (const f of failures) { console.log(` ✗ ${f.name}`); @@ -276,4 +306,5 @@ if (failed > 0) { process.exit(1); } else { console.log(" ✓ All cases passed\n"); + process.exit(0); } From fc6346e312ea6919680b346d9fbec33a4bde628a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:20:46 -0400 Subject: [PATCH 08/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=201=20=E2=80=94=20stream=20close=20race,=20tests,=20do?= =?UTF-8?q?c=20sync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Streaming correctness: - Filtered stream no longer panics with send-on-closed-channel when Close() races in-flight inner deliveries: channel close and channel send now serialize under the controller mutex, and the wrapper unsubscribes from the inner stream before closing - Events() channel is fed only once Events() has been called, so Subscribe-only consumers no longer overflow a channel nobody reads; the buffer-full drop is logged once, not per event - Malformed-SSE log omits the payload (can carry tenant/PII fields) - Unparseable Retry-After falls back to backoff(attempt), not the 30s max; parsing extracted to retryAfterDelay for testability Tests (SDK coverage 33% → 80%+; streaming subsystem was 0%): - stream_test.go: SSE lifecycle over httptest, filtered close-under-load regression test, Events()/Connected, full filter-engine tables - live_query_test.go: backfill buffering, desc-order dedup bound, fetch error, no-callbacks-after-Close - http_test.go: retryAfterDelay table + live 429 Retry-After flow - Remaining unclosed httptest servers in table_test/http_test now use t.Cleanup Build plumbing: - test-go-sdk runs with -race; test/lint/fix aggregates now include the nested clients/go module; mangled test-go-sdk comment restored - New test-conformance-ts target runs the TS conformance runner (was wired to nothing); CI unit job runs it Docs: - development.md synced: suites/targets tables, CI unit job, project structure with clients/, Go SDK in the dev-loop section - reference.md: real codegen output (EventId not EventID; no bare int), initialism note, missing type-mapping rows (SimpleAggregateFunction, Time/Time64, Boolean, BFloat16), two-runner conformance wording, 401 row corrected (missing token → 403) here and in sdk/reference.md - Go SDK added alongside TS on the landing page, getting-started, and why-wavehouse comparison tables --- .github/workflows/ci.yml | 2 +- Makefile | 19 +- clients/go/http.go | 26 +- clients/go/http_test.go | 79 +++++- clients/go/live_query_test.go | 173 +++++++++++++ clients/go/stream.go | 59 ++++- clients/go/stream_test.go | 301 ++++++++++++++++++++++ clients/go/table_test.go | 8 + docs/src/content/docs/development.md | 23 +- docs/src/content/docs/getting-started.md | 3 +- docs/src/content/docs/index.mdx | 11 +- docs/src/content/docs/sdk/go/index.md | 5 +- docs/src/content/docs/sdk/go/reference.md | 38 ++- docs/src/content/docs/sdk/reference.md | 2 +- docs/src/content/docs/why-wavehouse.md | 4 +- 15 files changed, 686 insertions(+), 67 deletions(-) create mode 100644 clients/go/live_query_test.go create mode 100644 clients/go/stream_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8646bf53..1de4d6ab 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -205,7 +205,7 @@ jobs: with: go-cache-suffix: "-unit" - name: Run Go unit tests + SDK vitest + Go SDK tests - run: make test-unit test-ts test-go-sdk COV_DEFER=1 + run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/Makefile b/Makefile index fdb9a7ab..6e3d1c28 100644 --- a/Makefile +++ b/Makefile @@ -356,7 +356,7 @@ fmt-ts: pnpm-install $(call run,Biome (format),$(PNPM) -s -w run format,run make fix to apply formatting) .PHONY: lint -lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. +lint: lint-go lint-go-sdk lint-ts lint-md lint-prose ## Lint across Go (root + clients/go golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. .PHONY: lint-go lint-go: $(GOLANGCI_LINT) go-mod-download @@ -451,6 +451,8 @@ fix-go: $(GOLANGCI_LINT) @$(GOFUMPT) -w $(GO_DIRS) @$(GOIMPORTS) -w $(GO_DIRS) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" + @cd clients/go && go mod tidy && $(GOFUMPT) -w . && $(GOIMPORTS) -w . && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install @@ -681,7 +683,7 @@ test-unit: go-mod-download ## Run Go unit tests + render coverage + gate thresho # Hidden alias: `make test` matches `go test ./...` muscle memory; test-unit # is the explicit form. .PHONY: test -test: test-unit +test: test-unit test-go-sdk .PHONY: test-integration test-integration: go-mod-download ## Run Go integration tests + render coverage + gate threshold (requires Docker) @@ -726,11 +728,20 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # test-go-sdk: unit tests for clients/go/ — a nested Go module (its own # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and -# test-go-sdk: nested module — needs its own target. +# needs its own target. -race because the SDK's streaming subsystem is the +# most concurrent code in the repo. .PHONY: test-go-sdk test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" - @cd clients/go && go test ./... + @cd clients/go && go test -race ./... + +# test-conformance-ts: the TS half of the cross-SDK wire-format conformance +# suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). +# Both replay clients/go/testdata/wire_cases.json. +.PHONY: test-conformance-ts +test-conformance-ts: build-ts ## Run TS SDK wire-format conformance against the shared fixture + @printf "$(CYAN)==> Running TS wire-format conformance...$(RESET)\n" + @node tests/conformance/conformance_ts.mjs # test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. .PHONY: test-go-sdk-e2e diff --git a/clients/go/http.go b/clients/go/http.go index f56a1235..0fc59d9e 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -138,16 +138,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a // 503/429 with Retry-After: wait the specified duration (capped). if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { - delay := maxRetryAfter - if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { - delay = time.Duration(secs) * time.Second - } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { - if d := time.Until(parsed); d > 0 { - delay = d - } - } - delay = min(delay, maxRetryAfter) - if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { + if sleepErr := sleepWithContext(ctx, retryAfterDelay(ra, attempt)); sleepErr != nil { return errAborted } lastErr = apiErr @@ -178,6 +169,21 @@ func buildURL(base, path string, params url.Values) string { return u } +// retryAfterDelay resolves a Retry-After header (delta-seconds or HTTP-date) +// into a wait, clamped to maxRetryAfter. An unparseable header falls back to +// the ordinary backoff for this attempt, not the maximum. +func retryAfterDelay(ra string, attempt int) time.Duration { + delay := backoff(attempt) + if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { + delay = time.Duration(secs) * time.Second + } else if parsed, err := http.ParseTime(ra); err == nil { + if d := time.Until(parsed); d > 0 { + delay = d + } + } + return min(delay, maxRetryAfter) +} + func backoff(attempt int) time.Duration { ms := 1000 * math.Pow(2, float64(attempt)) if ms > 30000 { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index ede6543c..db1693b3 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -10,8 +10,10 @@ import ( "time" ) -func testCtx(handler http.Handler) httpContext { +func testCtx(t *testing.T, handler http.Handler) httpContext { + t.Helper() srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) return httpContext{ baseURL: srv.URL, maxRetries: 0, @@ -20,7 +22,7 @@ func testCtx(handler http.Handler) httpContext { } func TestDoRequest_SuccessfulGET(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) })) @@ -41,7 +43,7 @@ func TestDoRequest_SuccessfulGET(t *testing.T) { func TestDoRequest_POSTWithBody(t *testing.T) { var gotBody map[string]string var gotCT string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") _ = json.NewDecoder(r.Body).Decode(&gotBody) w.WriteHeader(200) @@ -66,7 +68,7 @@ func TestDoRequest_POSTWithBody(t *testing.T) { func TestDoRequest_RawBody(t *testing.T) { var gotBody string var gotCT string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") raw := make([]byte, 1024) n, _ := r.Body.Read(raw) @@ -93,7 +95,7 @@ func TestDoRequest_RawBody(t *testing.T) { func TestDoRequest_AuthInjection(t *testing.T) { var gotAuth string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotAuth = r.Header.Get("Authorization") w.WriteHeader(200) })) @@ -113,7 +115,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { func TestDoRequest_4xxNotRetried(t *testing.T) { var count atomic.Int32 - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { count.Add(1) w.Header().Set("Content-Type", "application/json") w.WriteHeader(404) @@ -136,7 +138,7 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { func TestDoRequest_5xxRetried(t *testing.T) { var count atomic.Int32 - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { n := count.Add(1) if n < 3 { w.Header().Set("Content-Type", "application/json") @@ -162,7 +164,7 @@ func TestDoRequest_5xxRetried(t *testing.T) { } func TestDoRequest_AbortedOnCancel(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(5 * time.Second) })) @@ -180,7 +182,7 @@ func TestDoRequest_AbortedOnCancel(t *testing.T) { } func TestDoRequest_EmptyResponse(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(200) })) @@ -222,3 +224,62 @@ func TestBackoff(t *testing.T) { }) } } + +func TestRetryAfterDelay(t *testing.T) { + tests := []struct { + name string + ra string + want time.Duration + }{ + {"DeltaSeconds", "5", 5 * time.Second}, + {"ClampedToMax", "3600", maxRetryAfter}, + {"HTTPDateFuture", time.Now().Add(10 * time.Second).UTC().Format(http.TimeFormat), 0}, // range-checked below + {"Garbage", "not-a-delay", 0}, // range-checked below + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := retryAfterDelay(tt.ra, 0) + switch tt.name { + case "HTTPDateFuture": + if got <= 0 || got > 10*time.Second { + t.Fatalf("want ~10s, got %v", got) + } + case "Garbage": + // Falls back to backoff(0): 1s ±20% jitter. + if got < 800*time.Millisecond || got > 1200*time.Millisecond { + t.Fatalf("want backoff(0) fallback, got %v", got) + } + default: + if got != tt.want { + t.Fatalf("want %v, got %v", tt.want, got) + } + } + }) + } +} + +func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { + var calls atomic.Int64 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) == 1 { + w.Header().Set("Retry-After", "1") + w.WriteHeader(http.StatusTooManyRequests) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + hctx.maxRetries = 1 + + start := time.Now() + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{method: "GET", path: "/x"}, &result) + if err != nil { + t.Fatalf("want success after 429 retry, got %v", err) + } + if got := calls.Load(); got != 2 { + t.Fatalf("want 2 attempts, got %d", got) + } + if elapsed := time.Since(start); elapsed < 900*time.Millisecond { + t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) + } +} diff --git a/clients/go/live_query_test.go b/clients/go/live_query_test.go new file mode 100644 index 00000000..f703a68e --- /dev/null +++ b/clients/go/live_query_test.go @@ -0,0 +1,173 @@ +package wavehouse + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" +) + +// bareStream builds a StreamController that never dials anything — events are +// injected with emitEvent, exactly how the run loop feeds real ones. +func bareStream() *StreamController { + return &StreamController{ + status: StatusLive, + eventCh: make(chan StreamEvent, 16), + done: make(chan struct{}), + cancel: func() {}, + } +} + +func liveEvent(ts string) StreamEvent { + return StreamEvent{Table: "clicks", Timestamp: ts, Data: map[string]any{"page": "/home"}} +} + +func awaitInitial(t *testing.T, ch <-chan []map[string]any) []map[string]any { + t.Helper() + select { + case rows := <-ch: + return rows + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired") + return nil + } +} + +func TestLiveQuery_InitialThenLiveWithDedup(t *testing.T) { + sc := bareStream() + fetched := []map[string]any{ + {"page": "/a", "received_timestamp": "2026-01-01T00:00:05Z"}, + // Descending order: the max timestamp is NOT the last row. + {"page": "/b", "received_timestamp": "2026-01-01T00:00:03Z"}, + } + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return fetched, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + if err != nil { + t.Errorf("Initial err: %v", err) + } + initialCh <- rows + }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Buffered while the backfill is in flight; deduped against the *max* + // backfilled timestamp (5Z despite descending order) on flush. + sc.emitEvent(liveEvent("2026-01-01T00:00:04Z")) // ≤ max backfill → skipped + sc.emitEvent(liveEvent("2026-01-01T00:00:06Z")) // newer → delivered + close(gate) + + rows := awaitInitial(t, initialCh) + if len(rows) != 2 { + t.Fatalf("want 2 backfill rows, got %d", len(rows)) + } + + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:06Z" { + t.Fatalf("want the newer event only, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("live event never delivered") + } + select { + case e := <-nextCh: + t.Fatalf("stale event delivered despite dedup: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_BuffersDuringBackfill(t *testing.T) { + sc := bareStream() + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return []map[string]any{{"received_timestamp": "2026-01-01T00:00:01Z"}}, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Events arriving mid-backfill are buffered, then flushed post-Initial. + sc.emitEvent(liveEvent("2026-01-01T00:00:02Z")) + sc.emitEvent(liveEvent("2026-01-01T00:00:00Z")) // older than backfill → dropped in flush + close(gate) + + awaitInitial(t, initialCh) + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("want buffered 02Z event, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("buffered event never flushed") + } + select { + case e := <-nextCh: + t.Fatalf("pre-backfill event should have been deduped: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_FetchErrorReportedOnce(t *testing.T) { + sc := bareStream() + errCh := make(chan error, 1) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, errors.New("boom") }, + &StreamSubscriber{ + Initial: func(_ []map[string]any, err error) { errCh <- err }, + }) + defer lq.Close() + + select { + case err := <-errCh: + if err == nil || err.Error() != "boom" { + t.Fatalf("want boom, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired on fetch error") + } +} + +func TestLiveQuery_NoCallbacksAfterClose(t *testing.T) { + sc := bareStream() + initialCh := make(chan []map[string]any, 1) + var delivered atomic.Int64 + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, nil }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(StreamEvent) { delivered.Add(1) }, + Status: func(StreamStatus) { delivered.Add(1) }, + Error: func(error) { delivered.Add(1) }, + }) + awaitInitial(t, initialCh) + + lq.Close() + before := delivered.Load() + sc.emitEvent(liveEvent("2026-01-01T00:00:09Z")) + sc.emitError(errors.New("late")) + sc.setStatus(StatusReconnecting) + time.Sleep(50 * time.Millisecond) + if got := delivered.Load(); got != before { + t.Fatalf("callbacks fired after Close: before=%d after=%d", before, got) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index af14cd7f..4cb9582f 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -18,13 +18,15 @@ import ( // StreamController manages a live SSE event stream. Use Subscribe for // callback-based consumption or Events for channel-based consumption. type StreamController struct { - mu sync.Mutex - status StreamStatus - subscribers []*StreamSubscriber - eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption - cancel context.CancelFunc - done chan struct{} - closed bool + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + chanRequested bool // set by Events(); until then emitEvent skips the channel + dropLogOnce sync.Once + cancel context.CancelFunc + done chan struct{} + closed bool } // newStreamController opens an SSE connection for the given table. @@ -75,8 +77,13 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { } // Events returns a read-only channel that receives stream events. -// The channel is closed when the stream closes. +// The channel is closed when the stream closes. Events are fed to the +// channel only from the first Events() call onward — a Subscribe-only +// consumer never fills (and overflows) a channel it isn't reading. func (sc *StreamController) Events() <-chan StreamEvent { + sc.mu.Lock() + sc.chanRequested = true + sc.mu.Unlock() return sc.eventCh } @@ -162,11 +169,20 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } - // Non-blocking send to the channel. + // Non-blocking send to the channel. Guarded by mu so the send and + // closeEventCh serialize — a late event can never hit a closed channel — + // and skipped entirely until Events() opts in. + sc.mu.Lock() + defer sc.mu.Unlock() + if sc.closed || !sc.chanRequested { + return + } select { case sc.eventCh <- event: default: - log.Printf("[wavehouse] stream event dropped: channel buffer full") + sc.dropLogOnce.Do(func() { + log.Printf("[wavehouse] stream event dropped: Events() channel buffer full (further drops not logged)") + }) } } @@ -182,11 +198,20 @@ func (sc *StreamController) emitError(err error) { } } +// closeEventCh marks the controller closed and closes the events channel. +// Must serialize with emitEvent's send via mu. +func (sc *StreamController) closeEventCh() { + sc.mu.Lock() + sc.closed = true + close(sc.eventCh) + sc.mu.Unlock() +} + // run is the SSE connection loop with reconnect/backoff. func (sc *StreamController) run(ctx context.Context, hctx httpContext, table string, opts *StreamOptions) { defer func() { sc.setStatus(StatusClosed) - close(sc.eventCh) + sc.closeEventCh() close(sc.done) }() @@ -344,7 +369,9 @@ type sseMessage struct { func (sc *StreamController) handleSSEData(data, eventID string) { var msg sseMessage if err := json.Unmarshal([]byte(data), &msg); err != nil { - log.Printf("[wavehouse] SSE received malformed message: %s", data) + // Deliberately omits the payload: event data can carry tenant/PII + // fields and this goes to the process-global logger. + log.Printf("[wavehouse] SSE received malformed message (%d bytes): %v", len(data), err) return } @@ -370,11 +397,14 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, go func() { defer func() { sc.setStatus(StatusClosed) - close(sc.eventCh) + // closeEventCh serializes with any in-flight emitEvent (which + // runs on the inner controller's goroutine), so the channel is + // never closed under a pending send. + sc.closeEventCh() close(sc.done) }() - inner.Subscribe(&StreamSubscriber{ + unsub := inner.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { if !matchesFilters(event.Data, filters) { return @@ -394,6 +424,7 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, select { case <-ctx.Done(): + unsub() inner.Close() case <-inner.done: } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go new file mode 100644 index 00000000..4da86d51 --- /dev/null +++ b/clients/go/stream_test.go @@ -0,0 +1,301 @@ +package wavehouse + +import ( + "context" + "fmt" + "io" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +// sseServer serves the given SSE frames on any request, then holds the +// connection open until the client disconnects. +func sseServer(t *testing.T, frames []string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl, ok := w.(http.Flusher) + if !ok { + t.Error("response writer is not a flusher") + return + } + w.WriteHeader(200) + fl.Flush() + for _, f := range frames { + _, _ = io.WriteString(w, f) + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + return srv +} + +func sseFrame(ts, page string) string { + return "id: " + ts + "\n" + + `data: {"table_name":"clicks","received_timestamp":"` + ts + `","data":{"page":"` + page + `"}}` + + "\n\n" +} + +func streamClient(t *testing.T, srv *httptest.Server) *Client { + t.Helper() + return NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) +} + +func TestStream_SubscribeReceivesEvents(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/about"), + }) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{ + Next: func(e StreamEvent) { got <- e }, + }) + + e1 := recvEvent(t, got) + if e1.Table != "clicks" || e1.Data["page"] != "/home" { + t.Fatalf("unexpected first event: %+v", e1) + } + e2 := recvEvent(t, got) + if e2.Data["page"] != "/about" || e2.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("unexpected second event: %+v", e2) + } +} + +func recvEvent(t *testing.T, ch <-chan StreamEvent) StreamEvent { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for stream event") + return StreamEvent{} + } +} + +func TestStream_EventsChannel(t *testing.T) { + srv := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + + ch := stream.Events() + e := recvEvent(t, ch) + if e.Data["page"] != "/home" { + t.Fatalf("unexpected event: %+v", e) + } + + stream.Close() + select { + case _, open := <-ch: + if open { + // A buffered event may arrive before close; drain once more. + if _, open2 := <-ch; open2 { + t.Fatal("events channel not closed after Close") + } + } + case <-time.After(5 * time.Second): + t.Fatal("events channel never closed after Close") + } +} + +func TestStream_Connected(t *testing.T) { + srv := sseServer(t, nil) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err != nil { + t.Fatalf("Connected: %v", err) + } + if s := stream.Status(); s != StatusLive { + t.Fatalf("want live, got %s", s) + } +} + +func TestStream_FilteredDeliversMatchesAndProjects(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/miss"), + sseFrame("2026-01-01T00:00:03Z", "/home"), + }) + stream := streamClient(t, srv).From("clicks"). + Select("page"). + Where("page", OpEq, "/home"). + Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + + for range 2 { + e := recvEvent(t, got) + if e.Data["page"] != "/home" { + t.Fatalf("filter leaked event: %+v", e) + } + if len(e.Data) != 1 { + t.Fatalf("projection kept extra columns: %+v", e.Data) + } + } + select { + case e := <-got: + t.Fatalf("unexpected third event: %+v", e) + case <-time.After(100 * time.Millisecond): + } +} + +// TestStream_FilteredCloseUnderLoad exercises the wrapper-close path while the +// inner stream is still delivering — the send-on-closed-channel regression. +func TestStream_FilteredCloseUnderLoad(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + for i := 0; ; i++ { + select { + case <-r.Context().Done(): + return + default: + } + _, err := io.WriteString(w, sseFrame(fmt.Sprintf("2026-01-01T00:00:%02dZ", i%60), "/home")) + if err != nil { + return + } + fl.Flush() + } + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks"). + Select(). + Where("page", OpEq, "/home"). + Stream(nil) + + var n atomic.Int64 + stream.Subscribe(&StreamSubscriber{Next: func(StreamEvent) { n.Add(1) }}) + // Also exercise the Events() channel feed path during close. + _ = stream.Events() + + deadline := time.Now().Add(5 * time.Second) + for n.Load() < 10 && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + if n.Load() == 0 { + t.Fatal("no events delivered before close") + } + stream.Close() // must not panic or race with in-flight emits + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("filtered stream goroutine never exited") + } +} + +func TestStream_HandleMalformedSSEData(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 1)} + sc.handleSSEData("not json", "id1") // must not panic or emit + select { + case e := <-sc.eventCh: + t.Fatalf("malformed data emitted event: %+v", e) + default: + } +} + +// --------------------------------------------------------------------------- +// Client-side filter engine +// --------------------------------------------------------------------------- + +func TestEvaluateFilter(t *testing.T) { + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"EqNumericCrossType", float64(10), "eq", 10, true}, + {"EqString", "a", "eq", "a", true}, + {"EqMismatch", "a", "eq", "b", false}, + {"Neq", "a", "neq", "b", true}, + {"GtTrue", float64(11), "gt", 10, true}, + {"GtFalse", float64(10), "gt", 10, false}, + {"Gte", float64(10), "gte", 10, true}, + {"Lt", float64(9), "lt", 10, true}, + {"LteString", "a", "lte", "b", true}, + {"GtIncomparable", "a", "gt", 10, false}, + {"InAnySlice", "b", "in", []any{"a", "b"}, true}, + {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, + {"InMiss", "c", "in", []any{"a", "b"}, false}, + {"InNotASlice", "a", "in", "a", false}, + {"Like", "hello world", "like", "hello%", true}, + {"LikeCaseInsensitive", "HELLO", "like", "hello", true}, + {"LikeUnderscore", "cat", "like", "c_t", true}, + {"LikeAnchored", "xhello", "like", "hello%", false}, + {"NotLike", "abc", "not_like", "x%", true}, + {"LikeNonString", 5, "like", "5", false}, + {"UnknownOp", "a", "regex", "a", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := evaluateFilter(tt.actual, tt.op, tt.expected); got != tt.want { + t.Errorf("evaluateFilter(%v, %q, %v) = %v, want %v", tt.actual, tt.op, tt.expected, got, tt.want) + } + }) + } +} + +func TestMatchesFilters_AllMustMatch(t *testing.T) { + row := map[string]any{"page": "/home", "score": float64(10)} + both := []QueryFilter{ + {Column: "page", Op: "eq", Value: "/home"}, + {Column: "score", Op: "gt", Value: 5}, + } + if !matchesFilters(row, both) { + t.Fatal("want match when every filter passes") + } + oneFails := append(append([]QueryFilter(nil), both...), QueryFilter{Column: "score", Op: "gt", Value: 99}) + if matchesFilters(row, oneFails) { + t.Fatal("want no match when any filter fails") + } + if !matchesFilters(row, nil) { + t.Fatal("want match with no filters") + } +} + +func TestCompareOrdered(t *testing.T) { + if c, ok := compareOrdered(float64(1), 2); !ok || c != -1 { + t.Fatalf("numeric compare: got (%d, %v)", c, ok) + } + if c, ok := compareOrdered("b", "a"); !ok || c != 1 { + t.Fatalf("string compare: got (%d, %v)", c, ok) + } + if _, ok := compareOrdered(map[string]any{}, 1); ok { + t.Fatal("incomparable types must return ok=false") + } +} + +func TestToFloat64(t *testing.T) { + for _, v := range []any{float64(1), float32(1), int(1), int64(1)} { + if f, ok := toFloat64(v); !ok || f != 1 { + t.Fatalf("toFloat64(%T) = (%v, %v)", v, f, ok) + } + } + if _, ok := toFloat64("1"); ok { + t.Fatal("strings must not convert") + } +} + +func TestProjectColumns(t *testing.T) { + row := map[string]any{"a": 1, "b": 2, "c": 3} + got := projectColumns(row, []string{"a", "c", "missing"}) + if len(got) != 2 || got["a"] != 1 || got["c"] != 3 { + t.Fatalf("unexpected projection: %+v", got) + } +} diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 707e13ed..5116d53f 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -17,6 +17,7 @@ func TestTableRef_InsertSingle(t *testing.T) { _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) if err != nil { @@ -44,6 +45,7 @@ func TestTableRef_InsertBatch(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ {"page": "/a"}, @@ -83,6 +85,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ {Page: "/a"}, @@ -116,6 +119,7 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) if err != nil { @@ -133,6 +137,7 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("should not make a request for empty batch") })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) if err != nil { @@ -155,6 +160,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) @@ -181,6 +187,7 @@ func TestTableRef_Schema(t *testing.T) { }, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) schema, err := c.From("clicks").Schema(context.Background()) if err != nil { @@ -198,6 +205,7 @@ func TestTableRef_InsertDuplicate(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) if err != nil { diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 476ae773..14128047 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -188,7 +188,7 @@ They block the terminal and stream logs; simply press `Ctrl+C` to instantly tear ### Using the SDK against `make dev` -There's no bundled playground — point the published `@wavehouse/sdk` client at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: +There's no bundled playground — point the published `@wavehouse/sdk` client (or the Go SDK, `github.com/Wave-RF/WaveHouse/clients/go`) at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev @@ -196,7 +196,7 @@ WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev See the [SDK guide](/sdk) for the client API and examples. -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. +Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see the [Go SDK docs](/sdk/go). ### Validating tokens @@ -337,7 +337,10 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | -| SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`) | +| Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | +| SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | @@ -429,8 +432,13 @@ WaveHouse/ │ ├── policy/ # Access control policies (evaluation + NATS KV store) │ ├── query/ # Structured query AST + SQL builder │ └── testutil/ # Shared test helpers and mocks +├── clients/ # Official SDKs +│ ├── ts/ # TypeScript SDK (@wavehouse/sdk) +│ └── go/ # Go SDK — a NESTED Go module (own go.mod, invisible +│ # to root `go list`; hence the *-go-sdk make targets) ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) +│ ├── conformance/ # TS half of the cross-SDK wire-format conformance suite │ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) │ ├── fixtures/ # ClickHouse DDL + config/policy fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) @@ -476,7 +484,7 @@ Run `make help` to see all targets. Key ones: | **Static checks** | | | `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`) + TS (Biome) | +| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | | `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | @@ -486,8 +494,11 @@ Run `make help` to see all targets. Key ones: | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/` | | **Test** | | -| `make test` | Alias for `test-unit` | +| `make test` | Alias for `test-unit` + `test-go-sdk` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | +| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` | +| `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | +| `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | @@ -581,7 +592,7 @@ If the title doesn't match, a sticky comment posts on the PR explaining the form The `main branch protection` ruleset requires one status check to pass before any PR can merge: -- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). +- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts test-go-sdk test-conformance-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 5ae0e978..d8cf0800 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -76,7 +76,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ `POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/admin/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] -The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming; the [Go SDK](/sdk/go) offers the same builder with generics for typed rows. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). ::: ## 5. Subscribe to real-time updates @@ -106,6 +106,7 @@ The handful of things that most often trip up a first session — each is expect - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. - **[API Reference](/api)** — every endpoint, request/response shape, and error code. - **[TypeScript SDK](/sdk)** — zero-dependency client with query builder, live queries, and codegen. +- **[Go SDK](/sdk/go)** — the same surface for Go: context-first, generics for typed rows, codegen CLI. - **[Configuration](/configuration)** — full YAML + environment variable reference. - **[Deployment](/deployment)** — Docker images, releases, health checks. - **[Development](/development)** — building from source, running tests, hot-reload workflow. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index b10e97b1..1492292e 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -95,8 +95,8 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - - `@wavehouse/sdk` — zero-dependency client with type-safe query builder, live queries, real-time streaming, and codegen from your schemas. + + `@wavehouse/sdk` and `clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. @@ -104,7 +104,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket. -The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming: +The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming. Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set: @@ -192,6 +192,11 @@ WaveHouse is fail-closed, so the standalone stack ships a permissive trial polic description="Query builder, live queries, streaming, and schema codegen." href="/sdk" /> +
diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 2202f95d..280fab00 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -173,8 +173,9 @@ See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry b ## Differences from the TypeScript SDK The two SDKs share a wire format and mirror each other's feature set closely -(a shared `testdata/wire_cases.json` conformance fixture in the repo asserts -both produce identical HTTP requests for equivalent builder calls), but the +(a shared `wire_cases.json` conformance fixture is replayed by a test runner +per SDK — both run in CI — asserting each produces the expected HTTP request +for equivalent builder calls), but the languages pull the API shape in different directions: - **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 75cf8cad..54e2d2be 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -49,7 +49,7 @@ guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | @@ -179,14 +179,21 @@ package myapp // ClicksRow represents a row in the "clicks" table. type ClicksRow struct { - EventID string `json:"event_id"` - Page string `json:"page"` - UserID string `json:"user_id"` - DurationMS int `json:"duration_ms"` - ReceivedTimestamp string `json:"received_timestamp"` + Page string `json:"page"` + Button string `json:"button"` + Score float64 `json:"score"` + ReceivedTimestamp string `json:"received_timestamp,omitempty"` } ``` +(That's the exact output for the `clicks` table from the +[development quick-start](/development#quick-start) — `received_timestamp` +gets `,omitempty` because it has a `DEFAULT` clause.) + +Note the generator does **not** special-case initialisms: `event_id` becomes +`EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply +gets its first letter upper-cased. + Table and column names are converted to `PascalCase` for Go field/type names (a leading digit gets an `X` prefix — e.g. a table named `2fa_events` becomes `X2faEventsRow` — to stay a valid Go identifier). A column with @@ -197,17 +204,18 @@ tag. | ClickHouse Type | Go Type | |------------------|---------| -| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | -| `Bool` | `bool` | +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` / `Boolean` | `bool` | | `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | | `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | -| `Float32` | `float32` | +| `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | | `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | | `Array(T)` | `[]T` | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's @@ -220,11 +228,13 @@ that is what actually arrives on the wire. The Go SDK ships with unit tests colocated in `clients/go/` (its own Go module — `clients/go/go.mod` — separate from the root `WaveHouse` module), -plus a wire-format **conformance suite** -(`clients/go/conformance_test.go` + `clients/go/testdata/wire_cases.json`) -that replays a shared fixture of builder calls and asserts the Go SDK -produces the exact same HTTP method, path, content type, and body as the -TypeScript SDK for each one — keeping the two clients honest about the wire +plus the Go half of the cross-language wire-format **conformance suite**: +`clients/go/conformance_test.go` replays the shared fixture +(`clients/go/testdata/wire_cases.json`) and asserts the Go SDK produces the +expected HTTP method, path, content type, and body for each case. The +TypeScript half — `tests/conformance/conformance_ts.mjs`, run with +`make test-conformance-ts` (it builds the TS SDK first) — replays the same +fixture, and CI runs both, keeping the two clients honest about the wire format they both speak. ```bash diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index f631d1a7..2c9bd2d5 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -30,7 +30,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 9cb5b63d..5cce04ba 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -154,7 +154,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, zero-dep, codegen) | +| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `clients/go` (Go) — zero-dep, codegen | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. @@ -194,7 +194,7 @@ Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and | Thundering-herd coalescing | ✗ | Custom | ✓ | ✓ Ristretto + singleflight | | Row/column policies with JWT claims | ✗ | Custom | Tokens only | ✓ Hasura-style | | Named parameterized pipes | ✗ | Custom | ✓ | ✓ stored in NATS KV | -| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ `@wavehouse/sdk` | +| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ TypeScript + Go SDKs | | Cost model | Infra only | Infra + eng time | Per-vCPU SaaS | Infra only | ## Part IV — End-to-end data journey From e783f5b8af1fd6773886591f578915288eca6047 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:37:44 -0400 Subject: [PATCH 09/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=202=20=E2=80=94=20terminal=20SSE=20errors,=20codegen?= =?UTF-8?q?=20pointers,=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SDK behavior: - Non-retryable SSE connect errors (401/403/404) are now terminal: connect surfaces the parsed API error, run emits it and closes the stream instead of reconnecting forever; Connected() unblocks with "stream closed" - codegen: defaulted columns generate pointer fields (*T + omitempty) — the Go spelling of the TS codegen's `field?: T` — so an explicit zero value is sent instead of silently dropped in favor of the server default - Typed-row pagination cursor decodes with json.Number, keeping int64 cursor values past 2^53 exact Tests: - Pagination: page.Next walked across three pages asserting exactly one replaced cursor filter with the right op/value, desc → lt, quiet end when the projection omits the order column, int64 precision regression test - Terminal 403 stream test: error surfaced, StatusClosed, Connected fails - e2e: buildMarkerRow returns the column it used (markerColumn could pick a defaulted column the row never wrote); http_test raw-body via io.ReadAll - conformance_ts exits non-zero when nothing ran or any case was skipped Docs/Makefile: - Error-model claim scoped: HTTP-exchange errors are *wavehouse.Error; pre-request failures (auth provider, marshal) are plain wrapped errors — examples gain the else branch (go/index, go/reference, README) - codegen README example uses WAVEHOUSE_AUTH; reference.md documents the pointer rule and sample output - queries.md: real aggregation signatures/alias defaults, pagination example gains the OrderBy it needs - development.md: coverage sentence scoped to instrumented suites, "four suites" count dropped, Releasing the SDKs covers the Go module - access-control.mdx + pipes.mdx list the Go SDK method equivalents - make ci runs test-conformance-ts (parity with the CI unit job) --- Makefile | 2 +- clients/go/README.md | 4 +- clients/go/cmd/wavehouse-codegen/main.go | 7 + clients/go/e2e_test.go | 32 ++--- clients/go/http_test.go | 6 +- clients/go/query_builder.go | 7 +- clients/go/query_builder_test.go | 159 ++++++++++++++++++++++ clients/go/stream.go | 12 +- clients/go/stream_test.go | 41 ++++++ docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/development.md | 12 +- docs/src/content/docs/pipes.mdx | 2 +- docs/src/content/docs/sdk/go/index.md | 4 +- docs/src/content/docs/sdk/go/queries.md | 14 +- docs/src/content/docs/sdk/go/reference.md | 28 ++-- docs/src/content/docs/sdk/index.mdx | 2 +- tests/conformance/conformance_ts.mjs | 4 +- 17 files changed, 282 insertions(+), 56 deletions(-) diff --git a/Makefile b/Makefile index 6e3d1c28..243ddcfa 100644 --- a/Makefile +++ b/Makefile @@ -786,7 +786,7 @@ cov: ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-r # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk test-conformance-ts .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage diff --git a/clients/go/README.md b/clients/go/README.md index c71e5644..bd991359 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -193,9 +193,9 @@ rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM click Generate Go structs from a running WaveHouse instance: ```bash +export WAVEHOUSE_AUTH= # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --url http://localhost:8080 \ - --auth \ --out ./db_types.go \ --package myapp ``` @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/# ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`). Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`) deliver errors through callbacks instead: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 9775b525..793e1938 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -307,7 +307,14 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { seenFields[fieldName] = col.Name jsonTag := col.Name if col.HasDefault { + // Pointer + omitempty is the Go spelling of the TS codegen's + // `field?: T`: nil omits the field (server default applies), + // while a pointer to the zero value still sends an explicit + // 0/false/"" instead of silently dropping it. jsonTag += ",omitempty" + if !strings.HasPrefix(goType, "*") { + goType = "*" + goType + } } fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 69b724e3..23f59221 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -142,8 +142,7 @@ func TestE2E_InsertAndQuery(t *testing.T) { table, ts := firstTable(t, c) mk := marker(t) - row := buildMarkerRow(t, ts, mk) - markerCol := markerColumn(t, ts) + row, markerCol := buildMarkerRow(t, ts, mk) res, err := c.From(table).Insert(ctx, row) if err != nil { @@ -169,12 +168,12 @@ func TestE2E_BatchInsert(t *testing.T) { table, ts := firstTable(t, c) mk := marker(t) - markerCol := markerColumn(t, ts) // Build 3 rows, each with the same marker so we can count them. rows := make([]map[string]any, 3) + markerCol := "" for i := range rows { - rows[i] = buildMarkerRow(t, ts, mk) + rows[i], markerCol = buildMarkerRow(t, ts, mk) } res, err := c.From(table).Insert(ctx, rows) @@ -358,26 +357,14 @@ func TestE2E_PipesCRUD(t *testing.T) { // Helpers // --------------------------------------------------------------------------- -// markerColumn finds the first String/LowCardinality(String) column in the -// schema that we can use to inject a test marker value. -func markerColumn(t *testing.T, ts TableSchema) string { - t.Helper() - for _, col := range ts.Columns { - ct := strings.ToLower(col.Type) - if ct == "string" || strings.Contains(ct, "string") { - return col.Name - } - } - t.Skipf("e2e: table %q has no string column for marker injection", ts.Name) - return "" -} - // buildMarkerRow constructs a minimal valid row for the table, injecting the -// marker into the first string column and using sensible defaults for other -// required columns. -func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { +// marker into the first non-default string column and using sensible values +// for other required columns. It returns the row and the marker column, so +// callers query back the exact column the marker went into. +func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, string) { t.Helper() row := make(map[string]any) + markerCol := "" markerSet := false for _, col := range ts.Columns { if col.HasDefault { @@ -387,6 +374,7 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { switch { case !markerSet && strings.Contains(ct, "string"): row[col.Name] = mk + markerCol = col.Name markerSet = true case strings.Contains(ct, "string"): row[col.Name] = "e2e" @@ -408,7 +396,7 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { if !markerSet { t.Skipf("e2e: table %q has no non-default string column for marker", ts.Name) } - return row + return row, markerCol } // skipIfUnauthorized skips the test when err indicates a 401 or 403, diff --git a/clients/go/http_test.go b/clients/go/http_test.go index db1693b3..aef9aa0f 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "io" "net/http" "net/http/httptest" "sync/atomic" @@ -70,9 +71,8 @@ func TestDoRequest_RawBody(t *testing.T) { var gotCT string hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") - raw := make([]byte, 1024) - n, _ := r.Body.Read(raw) - gotBody = string(raw[:n]) + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) })) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 2855d367..c1c540e6 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -1,6 +1,7 @@ package wavehouse import ( + "bytes" "context" "encoding/json" "net/url" @@ -275,9 +276,13 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro m, ok := lastRow.(map[string]any) if !ok { // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + // UseNumber keeps int64 cursor values exact; plain float64 decoding + // corrupts IDs past 2^53 and pagination would repeat or skip a row. raw, _ := json.Marshal(lastRow) m = make(map[string]any) - _ = json.Unmarshal(raw, &m) + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() + _ = dec.Decode(&m) } lastValue, exists := m[cursor.Column] if !exists { diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 71d2808e..4c4b7656 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -1,8 +1,10 @@ package wavehouse import ( + "bytes" "context" "encoding/json" + "fmt" "io" "net/http" "net/http/httptest" @@ -269,3 +271,160 @@ func TestQueryBuilder_ComplexQuery(t *testing.T) { t.Fatal("wrong group_by") } } + +// pagingServer returns limit-sized pages of rows and captures each request +// body, so tests can walk page.Next and inspect the cursor filters sent. +func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, func() []map[string]any) { + t.Helper() + var mu sync.Mutex + var bodies []map[string]any + call := 0 + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + var body map[string]any + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() // keep int64 cursor values exact on the capture side too + _ = dec.Decode(&body) + mu.Lock() + bodies = append(bodies, body) + idx := call + call++ + mu.Unlock() + page := []map[string]any{} + if idx < len(pages) { + page = pages[idx] + } + _ = json.NewEncoder(w).Encode(page) + })) + return c, func() []map[string]any { + mu.Lock() + defer mu.Unlock() + return append([]map[string]any(nil), bodies...) + } +} + +func filtersOf(t *testing.T, body map[string]any) []map[string]any { + t.Helper() + raw, ok := body["filters"].([]any) + if !ok { + return nil + } + out := make([]map[string]any, len(raw)) + for i, f := range raw { + out[i] = f.(map[string]any) + } + return out +} + +func TestQueryBuilder_Pagination_NextWalksPages(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "a"}, {"id": "b"}}, + {{"id": "c"}, {"id": "d"}}, + {{"id": "e"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + page2, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if page2.Data[0]["id"] != "c" || !page2.HasMore || page2.Next == nil { + t.Fatalf("unexpected page 2: %+v", page2) + } + page3, err := page2.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(page3.Data) != 1 || page3.HasMore { + t.Fatalf("unexpected page 3: %+v", page3) + } + + bodies := getBodies() + if len(bodies) != 3 { + t.Fatalf("want 3 requests, got %d", len(bodies)) + } + if f := filtersOf(t, bodies[0]); len(f) != 0 { + t.Fatalf("page 1 must have no cursor filter, got %v", f) + } + // Page 2 and 3: exactly ONE cursor filter (replaced, not stacked), with + // the ascending op and the previous page's last cursor value. + for i, want := range []string{"b", "d"} { + f := filtersOf(t, bodies[i+1]) + if len(f) != 1 { + t.Fatalf("page %d: want exactly 1 cursor filter, got %v", i+2, f) + } + if f[0]["column"] != "id" || f[0]["op"] != "gt" || f[0]["value"] != want { + t.Fatalf("page %d: unexpected cursor filter %v", i+2, f[0]) + } + } +} + +func TestQueryBuilder_Pagination_DescUsesLt(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "z"}, {"id": "y"}}, + {{"id": "x"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "desc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 || f[0]["op"] != "lt" || f[0]["value"] != "y" { + t.Fatalf("desc cursor filter wrong: %v", f) + } +} + +func TestQueryBuilder_Pagination_CursorColumnMissingEndsQuietly(t *testing.T) { + c, _ := pagingServer(t, [][]map[string]any{ + {{"other": "1"}, {"other": "2"}}, // projection omits the order column + }) + + page, err := c.From("clicks").Select("other").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + next, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(next.Data) != 0 || next.HasMore || next.Next != nil { + t.Fatalf("want quiet empty page, got %+v", next) + } +} + +func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { + type idRow struct { + ID int64 `json:"id"` + } + const bigID = int64(9007199254740993) // 2^53 + 1: float64 round-trip corrupts it + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": bigID}}, + {}, + }) + + q := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2) + page, err := FetchTyped[idRow](context.Background(), q) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // json.Number survives the round-trip; float64 would have sent ...992. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740993" { + t.Fatalf("cursor value lost precision: %s", got) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index 4cb9582f..418b11ec 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -4,6 +4,7 @@ import ( "bufio" "context" "encoding/json" + "errors" "fmt" "log" "net/http" @@ -242,6 +243,15 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } if err != nil { + // A non-retryable API error (401/403/404, ...) is terminal: + // reconnecting can't fix a bad token or a missing table, and the + // TS SDK's EventSource likewise ends up closed on a non-200. + // Emit it and exit — the deferred cleanup sets StatusClosed. + var apiErr *Error + if errors.As(err, &apiErr) && !apiErr.Retryable { + sc.emitError(apiErr) + return + } sc.emitError(&Error{ Status: 0, Code: "SSE_ERROR", @@ -307,7 +317,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return "", false, fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + return "", false, parseErrorResponse(resp) } sc.setStatus(StatusLive) diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 4da86d51..18216aba 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "errors" "fmt" "io" "net/http" @@ -299,3 +300,43 @@ func TestProjectColumns(t *testing.T) { t.Fatalf("unexpected projection: %+v", got) } } + +// TestStream_NonRetryableConnectErrorIsTerminal: a 403 must close the stream +// (no infinite reconnect) and surface the API error to Error subscribers. +func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden) + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) || apiErr.Status != http.StatusForbidden || apiErr.Retryable { + t.Fatalf("want non-retryable HTTP_403, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("stream never closed after non-retryable connect error") + } + if s := stream.Status(); s != StatusClosed { + t.Fatalf("want closed, got %s", s) + } + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err == nil { + t.Fatal("Connected must fail on a terminally-closed stream") + } +} diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index dda1e0b5..69c21cb7 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -399,7 +399,7 @@ curl -X PUT http://localhost:8080/v1/admin/policy \ -d @policy.json ``` -The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`. +The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`, and the [Go SDK](/sdk/go/admin) as `wh.Policy.Get(ctx)`, `wh.Policy.Set(ctx, policy)`, and `wh.Policy.Validate(ctx, policy)`. ## Bootstrapping and the policy lifecycle diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 14128047..a7bbf169 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -313,7 +313,7 @@ make test-ts # E2E SDK suite against bin/wavehouse-cov make test-e2e -# All four suites sequentially + merged coverage +# All suites sequentially + merged coverage make test-all # Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, @@ -324,7 +324,7 @@ make ci make cov ``` -Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. **Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). @@ -503,7 +503,7 @@ Run `make help` to see all targets. Key ones: | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | -| `make test-all` | All four suites sequentially + merged coverage gate | +| `make test-all` | All suites sequentially + merged coverage gate | | `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | | **Analysis** (informational, not in CI) | | | `make size` | Binary size analysis → `tmp/analysis/` (text + SVG + interactive HTML) | @@ -550,9 +550,11 @@ PRs are grouped per config to reduce noise. The npm config is pointed at the wor **No auto-merge.** Dependabot PRs go through the same merge gate as any other PR — an approval from the `@Wave-RF/wavehouse-admins` team (the ruleset's `required_reviewers` rule) plus the required checks. (The former `dependabot-automerge.yml`, which auto-approved and merged patch/minor bumps hands-off, was removed — every bump now gets a human admin review.) -## Releasing the SDK +## Releasing the SDKs -The TypeScript SDK (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing — no `NPM_TOKEN`. It is independent of the server's Go/Docker release (`release.yml`): the `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint, so the two never collide. There are two channels: +**Go SDK** (`github.com/Wave-RF/WaveHouse/clients/go`): no tagged releases yet — `go get` resolves a pseudo-version from `main`. A tagged release requires a `clients/go/vX.Y.Z` tag (Go's nested-module tag form); the server's `v*` and npm's `sdk-v*` tag globs deliberately don't cover it, and no release workflow exists for it yet. + +**TypeScript SDK** (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing — no `NPM_TOKEN`. It is independent of the server's Go/Docker release (`release.yml`): the `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint, so the two never collide. There are two channels: - **Dev snapshots.** Every push to `main` publishes `0.0.0-dev.` under the `dev` dist-tag — but only when the built `dist/` actually changed (the version is a hash of the build output, so an unchanged build resolves to an already-published version and is skipped). Install the bleeding edge with `npm install @wavehouse/sdk@dev`. - **Tagged releases.** Pushing a `sdk-vX.Y.Z` tag publishes that version and creates a GitHub Release. A stable version goes to the `latest` dist-tag; a prerelease (`sdk-v0.2.0-rc.1`) is published under `alpha`/`beta`/`rc`/`next` — derived from the suffix — and marked as a GitHub pre-release. The tag **must** match `clients/ts/package.json`'s `version`, or the job fails fast. diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index dc9f262c..9aa39486 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -148,7 +148,7 @@ curl -X PUT http://localhost:8080/v1/admin/pipes/top_pages \ }' ``` -A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`. +A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`; the [Go SDK](/sdk/go/pipes) as `wh.Pipes.List(ctx)`, `wh.Pipes.Get(ctx, name)`, `wh.Pipes.Set(ctx, name, def)`, and `wh.Pipes.Delete(ctx, name)`. ## Executing a pipe diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 280fab00..49d35c72 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -155,7 +155,7 @@ parameter. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close` — deliver errors through callbacks instead; see [Streaming](/sdk/go/streaming).) +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) @@ -163,6 +163,8 @@ if err != nil { var whErr *wavehouse.Error if errors.As(err, &whErr) { fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... } return err } diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 59c588bc..c7e342fb 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -225,10 +225,12 @@ clicks.Select("page"). Aggregate("uniqExact", "user_id", "unique_users") // custom fn ``` -Each aggregation method signature: `(column, alias string) *QueryBuilder`. -`Count` defaults to `column="*"` when `column` is `""`, and `alias="count"` -when `alias` is `""`; the other aggregations default `alias` to -`"_"` when left empty. +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias +string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias +defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ +`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; +`CountDistinct` → `count_distinct_`. `Aggregate` has **no** alias +default — pass one explicitly or the query is sent with `"alias": ""`. #### `.GroupBy(...columns)` @@ -303,10 +305,10 @@ Execute the query and decode rows into `[]map[string]any`. The ordinary (non-generic) method form of `FetchTyped`. ```go -page, err := clicks.Select("page").Limit(50).FetchUntyped(ctx) +page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) if page.HasMore && page.Next != nil { - page2, err := page.Next(ctx) // cursor-based pagination + page2, err := page.Next(ctx) // cursor-based pagination — needs OrderBy } ``` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 54e2d2be..9e040031 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -39,12 +39,15 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)`, and errors -are always `*wavehouse.Error` (unwrap with `errors.As`). Streaming lifecycle -methods (`Stream`, `Subscribe`, `Close`) don't return `(T, error)`; stream -errors are delivered via the subscriber's `Error` callback. This is the -direct Go equivalent of the TypeScript SDK's "the SDK never throws" -guarantee. +operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors +originating from the HTTP exchange are `*wavehouse.Error` (unwrap with +`errors.As`); client-side failures before a request goes out (an `Auth` +provider error, a request-body marshal failure) are plain wrapped errors, +so handle the `errors.As == false` case too. Streaming lifecycle methods +(`Stream`, `Subscribe`, `Close`) don't return `(T, error)` — stream errors +are delivered via the subscriber's `Error` callback — and `Connected(ctx)` +returns plain errors. This is the direct Go equivalent of the TypeScript +SDK's "the SDK never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -63,6 +66,8 @@ if err != nil { var whErr *wavehouse.Error if errors.As(err, &whErr) { fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... } return err } @@ -182,13 +187,13 @@ type ClicksRow struct { Page string `json:"page"` Button string `json:"button"` Score float64 `json:"score"` - ReceivedTimestamp string `json:"received_timestamp,omitempty"` + ReceivedTimestamp *string `json:"received_timestamp,omitempty"` } ``` (That's the exact output for the `clicks` table from the [development quick-start](/development#quick-start) — `received_timestamp` -gets `,omitempty` because it has a `DEFAULT` clause.) +becomes `*string` + `,omitempty` because it has a `DEFAULT` clause.) Note the generator does **not** special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply @@ -197,8 +202,11 @@ gets its first letter upper-cased. Table and column names are converted to `PascalCase` for Go field/type names (a leading digit gets an `X` prefix — e.g. a table named `2fa_events` becomes `X2faEventsRow` — to stay a valid Go identifier). A column with -`has_default: true` in the schema gets `,omitempty` appended to its JSON -tag. +`has_default: true` in the schema becomes a **pointer field** with +`,omitempty` — the Go spelling of the TS codegen's `field?: T`: leave it +`nil` to omit the field (the server default applies), or point it at a +value to send it — including an explicit `0`/`false`/`""`, which a plain +value field with `omitempty` would silently drop. **ClickHouse → Go type mapping:** diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 659fe9fe..5724e623 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -69,7 +69,7 @@ closed explicitly. either way. A bare CDN URL tracks the latest published release; use a range (`@0`, `@0.1`) to float within a major or minor, or the `@dev` tag for unreleased builds from `main` (see - [Releasing the SDK](/development#releasing-the-sdk)). + [Releasing the SDKs](/development#releasing-the-sdks)). diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 6d8630df..d430bcfe 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -302,7 +302,9 @@ for (const f of failures) { } } -if (failed > 0) { +if (failed > 0 || skipped > 0 || passed === 0) { + if (passed === 0) console.log(" ✗ nothing ran — every case skipped or the fixture is empty\n"); + if (skipped > 0) console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); process.exit(1); } else { console.log(" ✓ All cases passed\n"); From 6e166da11be3d386301b3c0f0c5a3b443694fbeb Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:54:57 -0400 Subject: [PATCH 10/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=203=20=E2=80=94=20cursor=20ceiling=20honesty,=20Array(?= =?UTF-8?q?UInt8),=20codegen=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Untyped-path cursor precision: acknowledged as a documented ceiling rather than claimed fixed — FetchUntyped rows are float64-decoded before pagination sees them (same 2^53 ceiling as the TS SDK's JS numbers), so the code comment now says exactly that, queries.md documents the caveat next to the pagination example, and a regression test pins the behavior; FetchTyped and codegen structs remain exact - codegen: Array(UInt8) no longer generates []uint8 ([]byte, which encoding/json base64-encodes and the server rejects) — widened to []uint16; new main_test.go covers chTypeToGo (incl. this case), pascalCase's digit guard, findTopLevelComma, pointer-default output, and both collision failures (codegen package was 0% covered) - Docs: streaming.md documents terminal non-retryable stream errors, SSE_ERROR row added to both SDK reference tables, make test/fmt/ci descriptions synced, internal/stream added to the project tree, SQL example no longer redeclares rows :=, CONTRIBUTING gains the SDK-sync bullet (+ configuration.mdx path, also in SUPPORT.md), AGENTS.md drops the nonexistent "wavehouse-go" name for the real module path, landing/ why-wavehouse name the Go module importably, 404 page links the Go SDK --- AGENTS.md | 6 +- CONTRIBUTING.md | 3 +- SUPPORT.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 10 +- clients/go/cmd/wavehouse-codegen/main_test.go | 139 ++++++++++++++++++ clients/go/query_builder.go | 9 +- clients/go/query_builder_test.go | 28 ++++ docs/src/content/docs/404.md | 1 + docs/src/content/docs/development.md | 11 +- docs/src/content/docs/index.mdx | 2 +- docs/src/content/docs/sdk/go/index.md | 7 +- docs/src/content/docs/sdk/go/queries.md | 9 +- docs/src/content/docs/sdk/go/reference.md | 1 + docs/src/content/docs/sdk/go/streaming.md | 6 + docs/src/content/docs/sdk/reference.md | 1 + docs/src/content/docs/why-wavehouse.md | 2 +- 16 files changed, 217 insertions(+), 20 deletions(-) create mode 100644 clients/go/cmd/wavehouse-codegen/main_test.go diff --git a/AGENTS.md b/AGENTS.md index 4ff5f47c..658706f8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`, package `wavehouse`, in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -331,7 +331,7 @@ Diagrams render inside the Starlight content column (~46–58rem wide) as build- ## SDK Sync -The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. +The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`, in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. | Backend change | SDK considerations | | -------------- | ------------------ | @@ -388,7 +388,7 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) clients/ts/ → TypeScript SDK (@wavehouse/sdk) -clients/go/ → Go SDK (wavehouse-go) +clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) query_builder.go, table.go → Structured query builder + per-table typed client stream.go, live_query.go → SSE streaming + live queries diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 82372627..1b459bd7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,9 +43,10 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` - - Configuration options → update `docs/src/content/docs/configuration.md` + - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), their doc trees (`docs/src/content/docs/sdk/` and `.../sdk/go/`), and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). diff --git a/SUPPORT.md b/SUPPORT.md index 30f185cc..eb331fe3 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -23,7 +23,7 @@ WaveHouse is in **alpha**. We're a small team building publicly while shipping t - Bug reports against the latest tagged release or `main` HEAD. - Reproducible regressions vs. the previous tag. -- Documentation gaps or wrong examples (especially `getting-started.md`, `api.md`, `configuration.md`). +- Documentation gaps or wrong examples (especially `getting-started.md`, `api.md`, `configuration.mdx`). - Configuration questions where the docs disagree with reality. ## Out of scope during alpha diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 793e1938..0ecfb885 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -209,8 +209,14 @@ func chTypeToGo(chType string) string { } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { - inner := chType[6 : len(chType)-1] - return "[]" + chTypeToGo(inner) + inner := chTypeToGo(chType[6 : len(chType)-1]) + // []uint8 is []byte, which encoding/json base64-encodes as a string — + // the server requires a real JSON array for Array(...) columns, so + // widen the element type instead. + if inner == "uint8" { + inner = "uint16" + } + return "[]" + inner } // Map. if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go new file mode 100644 index 00000000..2fe23a0e --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -0,0 +1,139 @@ +package main + +import ( + "strings" + "testing" +) + +func TestChTypeToGo(t *testing.T) { + tests := []struct { + ch string + want string + }{ + {"String", "string"}, + {"FixedString(16)", "string"}, + {"UUID", "string"}, + {"DateTime64(3, 'UTC')", "string"}, + {"Date", "string"}, + {"Time64(3)", "string"}, + {"Enum8('a' = 1)", "string"}, + {"IPv4", "string"}, + {"Bool", "bool"}, + {"Boolean", "bool"}, + {"UInt8", "uint8"}, + {"UInt16", "uint16"}, + {"UInt32", "uint32"}, + {"Int8", "int8"}, + {"Int32", "int32"}, + {"Float32", "float32"}, + {"BFloat16", "float32"}, + {"Float64", "float64"}, + // 64-bit and wider integers are quoted in ClickHouse JSON output. + {"UInt64", "string"}, + {"Int64", "string"}, + {"UInt128", "string"}, + {"Int256", "string"}, + {"Decimal(18, 4)", "string"}, + {"Nullable(Int32)", "*int32"}, + {"Nullable(Int64)", "*string"}, + {"LowCardinality(String)", "string"}, + {"LowCardinality(Nullable(String))", "*string"}, + {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, + {"SimpleAggregateFunction(any)", "any"}, + {"Array(String)", "[]string"}, + {"Array(Nullable(Int32))", "[]*int32"}, + // []uint8 is []byte → base64 on marshal; must widen. + {"Array(UInt8)", "[]uint16"}, + {"Map(String, UInt32)", "map[string]uint32"}, + {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, + {"Tuple(String, UInt8)", "any"}, + {"SomethingNew", "any"}, + } + for _, tt := range tests { + if got := chTypeToGo(tt.ch); got != tt.want { + t.Errorf("chTypeToGo(%q) = %q, want %q", tt.ch, got, tt.want) + } + } +} + +func TestPascalCase(t *testing.T) { + tests := []struct { + in string + want string + }{ + {"clicks", "Clicks"}, + {"user_id", "UserId"}, + {"received_timestamp", "ReceivedTimestamp"}, + {"multi-part.name here", "MultiPartNameHere"}, + {"2fa_events", "X2faEvents"}, // leading digit gets the X prefix + {"", ""}, + } + for _, tt := range tests { + if got := pascalCase(tt.in); got != tt.want { + t.Errorf("pascalCase(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} + +func TestFindTopLevelComma(t *testing.T) { + tests := []struct { + in string + want int + }{ + {"String, UInt32", 6}, + {"Map(String, String), UInt8", 19}, + {"NoComma", -1}, + } + for _, tt := range tests { + if got := findTopLevelComma(tt.in); got != tt.want { + t.Errorf("findTopLevelComma(%q) = %d, want %d", tt.in, got, tt.want) + } + } +} + +func TestGenerate_Basic(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "clicks": {Name: "clicks", Columns: []column{ + {Name: "page", Type: "String"}, + {Name: "score", Type: "Float64"}, + {Name: "received_timestamp", Type: "DateTime64(3, 'UTC')", HasDefault: true}, + }}, + }, "myapp") + if err != nil { + t.Fatal(err) + } + for _, want := range []string{ + "package myapp", + "type ClicksRow struct {", + "Page string `json:\"page\"`", + "Score float64 `json:\"score\"`", + // Defaulted column: pointer + omitempty so an explicit zero still sends. + "ReceivedTimestamp *string `json:\"received_timestamp,omitempty\"`", + } { + if !strings.Contains(out, want) { + t.Errorf("generated output missing %q:\n%s", want, out) + } + } +} + +func TestGenerate_FieldCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{ + {Name: "user_id", Type: "String"}, + {Name: "userId", Type: "String"}, + }}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "UserId") { + t.Fatalf("want field-collision error naming UserId, got %v", err) + } +} + +func TestGenerate_TypeCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "2fa": {Name: "2fa", Columns: []column{{Name: "a", Type: "String"}}}, + "x2fa": {Name: "x2fa", Columns: []column{{Name: "a", Type: "String"}}}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "X2faRow") { + t.Fatalf("want type-collision error naming X2faRow, got %v", err) + } +} diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index c1c540e6..07c16783 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -276,8 +276,13 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro m, ok := lastRow.(map[string]any) if !ok { // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. - // UseNumber keeps int64 cursor values exact; plain float64 decoding - // corrupts IDs past 2^53 and pagination would repeat or skip a row. + // UseNumber keeps typed int64 cursor values exact past 2^53. The + // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this + // protection: its rows were already decoded to float64 by + // encoding/json, so precision above 2^53 is gone before we get here — + // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or + // codegen structs, whose 64-bit int columns are strings) when paging + // on >2^53 integer cursors. raw, _ := json.Marshal(lastRow) m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 4c4b7656..02431746 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -428,3 +428,31 @@ func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { t.Fatalf("cursor value lost precision: %s", got) } } + +// TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling documents the known +// ceiling on the untyped path: rows decode to float64, so an integer cursor +// past 2^53 loses precision before pagination sees it (same as the TS SDK's +// JS-number ceiling). Use FetchTyped or codegen structs past 2^53. +func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": int64(9007199254740993)}}, + {}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // float64 rounds 2^53+1 down to 2^53 — the documented untyped ceiling. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740992" { + t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) + } +} diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md index 36e006ca..70e2069b 100644 --- a/docs/src/content/docs/404.md +++ b/docs/src/content/docs/404.md @@ -44,6 +44,7 @@ head: ArchitectureHow the pieces fit together API referenceEndpoints, payloads, and error semantics TypeScript SDKTyped client for browser and Node + Go SDKTyped client for Go services

Followed a link that should have worked? File an issue — broken links are bugs.

diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index a7bbf169..ea642152 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -297,7 +297,7 @@ All tests run with Go's **race detector** (`-race`) enabled by default. WaveHous ```bash # Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` -# Unit tests (compact output) — alias for `test-unit` +# Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test # Run specific test(s) @@ -316,8 +316,8 @@ make test-e2e # All suites sequentially + merged coverage make test-all -# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, -# then test-integration + test-e2e + cov +# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts + +# test-conformance-ts, then test-integration + test-e2e + cov make ci # Merge available covdata + gate against total threshold @@ -431,6 +431,7 @@ WaveHouse/ │ ├── pipes/ # Named query pipes (NATS KV + .sql bootstrap) │ ├── policy/ # Access control policies (evaluation + NATS KV store) │ ├── query/ # Structured query AST + SQL builder +│ ├── stream/ # SSE fan-out (event Hub, subscriber queues, keepalive) │ └── testutil/ # Shared test helpers and mocks ├── clients/ # Official SDKs │ ├── ts/ # TypeScript SDK (@wavehouse/sdk) @@ -482,11 +483,11 @@ Run `make help` to see all targets. Key ones: | `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | | `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | | **Static checks** | | -| `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | +| `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: Go incl. `clients/go` (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 1492292e..9308a38e 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -96,7 +96,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - `@wavehouse/sdk` and `clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. + `@wavehouse/sdk` (TypeScript) and `github.com/Wave-RF/WaveHouse/clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 49d35c72..0cc2f4c3 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -198,9 +198,10 @@ languages pull the API shape in different directions: - **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; there's no bare `await builder` shortcut. -- **`Insert` accepts typed row slices, not just maps.** Passing - `[]ClickRow{...}` (any slice type, detected via reflection) batches as - NDJSON exactly like `[]map[string]any` — see +- **Any slice batches, not just `[]map[string]any`.** Go detects slice-ness + via reflection, so `[]ClickRow{...}` takes the same NDJSON batch path as + `[]map[string]any` (the TS SDK's `insert` likewise accepts arrays of typed + rows — this bullet is about the Go mechanics, not a TS gap) — see [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index c7e342fb..6ef9380f 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -340,6 +340,13 @@ add an `.OrderBy()` to paginate. If the order column was left out of an explicit `.Select(...)` projection, `Next` quietly returns an empty page instead of erroring (there is no cursor value to read). +One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): +rows decode into `map[string]any`, where JSON numbers become `float64`, so an +integer cursor column loses exactness past 2^53 and pagination can repeat or +skip a row at that scale — the same ceiling the TypeScript SDK has with JS +numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and +codegen structs are unaffected (their 64-bit integer columns are `string`). + ```go page, err := clicks.Select(). OrderBy("received_timestamp", "desc"). @@ -380,7 +387,7 @@ type PageTotal struct { Page string `json:"page"` Total int `json:"total"` } -rows, err := wavehouse.SQL[PageTotal](ctx, wh, +typed, err := wavehouse.SQL[PageTotal](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") ``` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 9e040031..271143b6 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -59,6 +59,7 @@ SDK's "the SDK never throws" guarantee. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | +| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | ```go page, err := wh.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 55b31243..90107898 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -149,6 +149,12 @@ type StreamEvent struct { | --------- | --------- | -------- | | SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | +Reconnect covers transport failures and retryable (5xx) responses. A +non-retryable response (401/403/404) is terminal: the error is delivered to +the subscriber's `Error` callback, status goes to `StatusClosed`, and the +stream does not reconnect — fix the cause (refresh the token, correct the +table) and open a new stream. + Auth is sent as an `Authorization: Bearer` header on every stream (re)connection — see [the note in the Getting Started guide](/sdk/go#creating-a-client). The diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 2c9bd2d5..5d54ca4b 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -37,6 +37,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries with `Retry-After`) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `AbortSignal` | +| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the stream's error callback; the stream reconnects automatically | --- diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 5cce04ba..8302d3a4 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -154,7 +154,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `clients/go` (Go) — zero-dep, codegen | +| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `github.com/Wave-RF/WaveHouse/clients/go` — zero-dep, codegen | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. From 4b199b617400f5f47c592b7b007e684e191a27b5 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 14:51:03 -0400 Subject: [PATCH 11/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=204=20=E2=80=94=2064-bit=20codegen=20mapping,=20LIKE?= =?UTF-8?q?=20compile,=20doc=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - codegen: Int64/UInt64 map to int64/uint64 and 128/256-bit ints to json.Number (with a conditional encoding/json import) — generated structs target /v1/query and /v1/pipes/*, where the server scans ClickHouse values into Go types and re-marshals them as UNQUOTED numbers; the round-1 string mapping only held for /v1/admin/query, which forwards ClickHouse's own quoted JSON (use map[string]any with SQL[Row] there). Decode round-trip test pins the wire shape; docs type table and parity paragraph updated, pagination caveat now notes 64-bit codegen columns decode exactly - Filtered streams compile LIKE patterns once at construction — the process-global likeRegexCache sync.Map (unbounded, keyed on caller input) is gone, and per-event matching is a plain regex call - Docs: /sdk/go Quick Start is compilable (package main + func main, like the README); client concurrency-safety documented; Events() first-call feeding note; SELECT * expansion claim scoped to column-restricted roles (both SDK pages); Array(UInt8) exception in the type table; codegen go run uses @latest so it works outside the repo; stale TableRef "NOT safe for mutations" comment corrected (it holds no mutable state) - AGENTS.md: configuration.mdx path, both SDK readmes in the prose list - CHANGELOG: Go SDK entry expanded to house style (module path, nested- module caveat, new targets, conformance wiring) --- AGENTS.md | 8 +-- CHANGELOG.md | 2 +- clients/go/README.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 44 ++++++++---- clients/go/cmd/wavehouse-codegen/main_test.go | 54 ++++++++++++-- clients/go/stream.go | 71 ++++++++++++------- clients/go/stream_test.go | 7 +- clients/go/table.go | 6 +- docs/src/content/docs/sdk/go/index.md | 40 +++++++---- docs/src/content/docs/sdk/go/queries.md | 17 +++-- docs/src/content/docs/sdk/go/reference.md | 23 +++--- docs/src/content/docs/sdk/go/streaming.md | 7 ++ docs/src/content/docs/sdk/queries.md | 2 +- 13 files changed, 193 insertions(+), 90 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 658706f8..262d3512 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -293,7 +293,7 @@ Then run the reviewers relevant to the PR's diff (the same set from `scripts/pre Documentation *prose* — accuracy against the code, runnable examples, clarity, completeness — **and code↔docs sync** (code that changed but whose docs didn't) are reviewed by the **`docs-reviewer`** subagent, not the code-focused `pre-push-reviewer`. The canonical rubric is `.github/prompts/docs-review.md`. It complements the deterministic prose tools — misspell, markdownlint, starlight-links-validator — reviewing only what they can't, and it never edits docs or posts PR comments. -**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readme `clients/ts/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. +**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readmes `clients/ts/README.md` / `clients/go/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. **It is a hard pre-push gate**, run in parallel with the other pre-push reviewers (see §Pre-push self-review). Invoked with the **default (branch) scope** it emits a `VERDICT:` line; on `ship_it` the `review-marker.sh` SubagentStop hook writes `tmp/docs-reviewer-passed-`, which the push gate requires — unconditionally, on every PR-branch push (even code-only ones). Run it via **`/docs-review`**; with **no arg** that's the gating review (branch scope), while an explicit **path/glob** or **`all`** is **advisory** (no `VERDICT:`, no marker) for ad-hoc audits. The whole dev team runs Claude Code and this command is tracked in-repo, so everyone runs it themselves; there is intentionally **no PR/cloud path** for docs review. @@ -304,7 +304,7 @@ Every code change should update the corresponding docs in the same PR. A code ch | Change | Files to update | | ------ | --------------- | | Add/modify API endpoint | `docs/src/content/docs/api.md`, `README.md` (if user-facing) | -| Add/modify config option | `docs/src/content/docs/configuration.md`, `config.yaml`, `deployments/compose/*` env blocks, `docs/src/content/docs/deployment.md` | +| Add/modify config option | `docs/src/content/docs/configuration.mdx`, `config.yaml`, `deployments/compose/*` env blocks, `docs/src/content/docs/deployment.md` | | Change architecture / add a package | `docs/src/content/docs/architecture.md`, `AGENTS.md` | | Change ingest / event format | `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md` (CH schema) | | Change deployment / Docker | `docs/src/content/docs/deployment.md`, compose files | @@ -313,7 +313,7 @@ Every code change should update the corresponding docs in the same PR. A code ch Source-of-truth pairs that must agree: -- Config struct tags in `internal/config/config.go` ↔ `docs/src/content/docs/configuration.md`, `config.yaml`, compose env blocks +- Config struct tags in `internal/config/config.go` ↔ `docs/src/content/docs/configuration.mdx`, `config.yaml`, compose env blocks - `EventMessage` JSON tags ↔ `docs/src/content/docs/api.md` event format, SSE examples, ClickHouse INSERT columns - Route registrations in `router.go` ↔ `docs/src/content/docs/api.md` endpoint list - Handler error responses ↔ `docs/src/content/docs/api.md` error tables @@ -363,7 +363,7 @@ Internal-only backend changes (middleware refactors, observability internals, de 1. Add the field to the appropriate struct in `internal/config/config.go` with `yaml`, `env`, and `env-default` tags. 2. Use the new config value in `cmd/wavehouse/main.go` or the relevant internal package. -3. Document in `docs/src/content/docs/configuration.md`. +3. Document in `docs/src/content/docs/configuration.mdx`. ### Adding a new internal package diff --git a/CHANGELOG.md b/CHANGELOG.md index 627487f8..d478b4a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Go SDK (`clients/go/`) — official Go client with full API-tree parity against the TypeScript SDK, zero third-party runtime dependencies, cross-language wire-format conformance tests +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. diff --git a/clients/go/README.md b/clients/go/README.md index bd991359..cf2c1348 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -194,7 +194,7 @@ Generate Go structs from a running WaveHouse instance: ```bash export WAVEHOUSE_AUTH= # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ --package myapp diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 0ecfb885..f46998c1 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -185,27 +185,32 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric — map lookup. + // Numeric — map lookup. Generated structs target the structured-query and + // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse + // values into Go types and re-marshals them — so 64-bit integers arrive + // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. + // (Only /v1/admin/query forwards ClickHouse's own JSON, which quotes + // 64-bit ints; use map[string]any with SQL[Row] there.) if mapped, ok := map[string]string{ - "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", - "Int8": "int8", "Int16": "int16", "Int32": "int32", + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", "Float32": "float32", "Float64": "float64", "BFloat16": "float32", }[chType]; ok { return mapped } switch { - case strings.HasPrefix(chType, "Decimal"), - strings.HasPrefix(chType, "UInt64"), - strings.HasPrefix(chType, "UInt128"), + case strings.HasPrefix(chType, "Decimal"): + // Decimals are marshaled as quoted strings on the structured path + // (shopspring decimal.MarshalJSON quotes by default). + return "string" + case strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), - strings.HasPrefix(chType, "Int64"), strings.HasPrefix(chType, "Int128"), strings.HasPrefix(chType, "Int256"): - // 64-bit and bigger integers (and decimals) are strings on the wire: - // ClickHouse's JSON output quotes them by default - // (output_format_json_quote_64bit_integers=1) and the server forwards - // the ClickHouse `data` array verbatim. - return "string" + // 128/256-bit ints scan into *big.Int server-side and marshal as + // unquoted JSON numbers of arbitrary width — json.Number preserves + // them exactly where int64/uint64 would overflow. + return "json.Number" } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { @@ -286,10 +291,23 @@ func sortedKeys(m map[string]tableSchema) []string { func generate(schemas map[string]tableSchema, pkg string) (string, error) { var sb strings.Builder - fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) names := sortedKeys(schemas) + // json.Number fields (128/256-bit integer columns) need the import. + needsJSON := false + for _, name := range names { + for _, col := range schemas[name].Columns { + if strings.Contains(chTypeToGo(col.Type), "json.Number") { + needsJSON = true + } + } + } + fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) + if needsJSON { + sb.WriteString("import \"encoding/json\"\n\n") + } + // pascalCase is not injective ("user_id" and "userId" both yield // "UserId"), and format.Source only parses — it doesn't type-check — so // a duplicate identifier would be written as a non-compiling file with a diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 2fe23a0e..47b2ce3e 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -1,6 +1,8 @@ package main import ( + "encoding/json" + "go/format" "strings" "testing" ) @@ -28,14 +30,14 @@ func TestChTypeToGo(t *testing.T) { {"Float32", "float32"}, {"BFloat16", "float32"}, {"Float64", "float64"}, - // 64-bit and wider integers are quoted in ClickHouse JSON output. - {"UInt64", "string"}, - {"Int64", "string"}, - {"UInt128", "string"}, - {"Int256", "string"}, + // /v1/query re-marshals server-side: 64-bit ints arrive unquoted. + {"UInt64", "uint64"}, + {"Int64", "int64"}, + {"UInt128", "json.Number"}, + {"Int256", "json.Number"}, {"Decimal(18, 4)", "string"}, {"Nullable(Int32)", "*int32"}, - {"Nullable(Int64)", "*string"}, + {"Nullable(Int64)", "*int64"}, {"LowCardinality(String)", "string"}, {"LowCardinality(Nullable(String))", "*string"}, {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, @@ -137,3 +139,43 @@ func TestGenerate_TypeCollisionFails(t *testing.T) { t.Fatalf("want type-collision error naming X2faRow, got %v", err) } } + +// TestGeneratedShapeDecodesStructuredQueryPayload asserts the mapping choices +// actually decode what /v1/query emits: the server scans ClickHouse values +// into Go types and re-marshals, so 64-bit ints are unquoted numbers, +// 128/256-bit ints are unquoted arbitrary-width numbers, and Decimals are +// quoted strings. +func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { + type row struct { + ID uint64 `json:"id"` + Delta int64 `json:"delta"` + Big json.Number `json:"big"` + Price string `json:"price"` + } + payload := `[{"id":18446744073709551615,"delta":-9007199254740993,"big":170141183460469231731687303715884105727,"price":"12.3400"}]` + var rows []row + if err := json.Unmarshal([]byte(payload), &rows); err != nil { + t.Fatalf("generated shape failed to decode /v1/query payload: %v", err) + } + if rows[0].ID != 18446744073709551615 || rows[0].Delta != -9007199254740993 { + t.Fatalf("64-bit values corrupted: %+v", rows[0]) + } + if rows[0].Big.String() != "170141183460469231731687303715884105727" { + t.Fatalf("128-bit value corrupted: %s", rows[0].Big) + } +} + +func TestGenerate_JSONNumberImport(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{{Name: "big", Type: "UInt128"}}}, + }, "main") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, `import "encoding/json"`) { + t.Fatalf("json.Number field without encoding/json import:\n%s", out) + } + if _, err := format.Source([]byte(out)); err != nil { + t.Fatalf("generated output is not valid Go: %v", err) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index 418b11ec..6da259be 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -396,6 +396,7 @@ func (sc *StreamController) handleSSEData(data, eventID string) { // newFilteredStreamController wraps a StreamController with client-side // filtering and column projection. func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { + compiled := compileFilters(filters) ctx, cancel := context.WithCancel(context.Background()) sc := &StreamController{ status: inner.Status(), @@ -416,7 +417,7 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, unsub := inner.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { - if !matchesFilters(event.Data, filters) { + if !matchesFilters(event.Data, compiled) { return } if len(columns) > 0 { @@ -443,18 +444,54 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, return sc } +// compiledFilter pairs a filter with its precompiled LIKE regex (nil for +// every other operator, or when the pattern isn't a string / doesn't compile). +type compiledFilter struct { + QueryFilter + re *regexp.Regexp +} + +// compileFilters precompiles LIKE/NOT LIKE patterns once per stream. A +// controller's filters never change after construction, so this replaces a +// per-event compile (and avoids any process-global pattern cache). +func compileFilters(filters []QueryFilter) []compiledFilter { + out := make([]compiledFilter, len(filters)) + for i, f := range filters { + out[i] = compiledFilter{QueryFilter: f} + if f.Op == "like" || f.Op == "not_like" { + if pattern, ok := f.Value.(string); ok { + out[i].re = compileLike(pattern) + } + } + } + return out +} + +// compileLike converts a SQL LIKE pattern to a case-insensitive anchored +// regex (matching the TS SDK). Returns nil if the pattern doesn't compile. +func compileLike(pattern string) *regexp.Regexp { + escaped := regexp.QuoteMeta(pattern) + escaped = strings.ReplaceAll(escaped, "%", ".*") + escaped = strings.ReplaceAll(escaped, "_", ".") + re, err := regexp.Compile("(?i)^" + escaped + "$") + if err != nil { + return nil + } + return re +} + // matchesFilters evaluates all filters against a data row (AND). -func matchesFilters(row map[string]any, filters []QueryFilter) bool { +func matchesFilters(row map[string]any, filters []compiledFilter) bool { for _, f := range filters { val := row[f.Column] - if !evaluateFilter(val, f.Op, f.Value) { + if !evaluateFilter(val, f.Op, f.Value, f.re) { return false } } return true } -func evaluateFilter(actual any, op string, expected any) bool { +func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool { switch op { case "eq": return equalValues(actual, expected) @@ -475,12 +512,11 @@ func evaluateFilter(actual any, op string, expected any) bool { case "in": return evaluateIn(actual, expected) case "like", "not_like": - aStr, aOK := actual.(string) - eStr, eOK := expected.(string) - if !aOK || !eOK { + aStr, ok := actual.(string) + if !ok || re == nil { return false } - return (op == "like") == matchLike(aStr, eStr) + return (op == "like") == re.MatchString(aStr) default: return false } @@ -521,25 +557,6 @@ func evaluateIn(actual, expected any) bool { return false } -var likeRegexCache sync.Map // pattern string → *regexp.Regexp - -// matchLike converts a SQL LIKE pattern to a regex and tests it -// (case-insensitive, matching the TS SDK). -func matchLike(actual, pattern string) bool { - if cached, ok := likeRegexCache.Load(pattern); ok { - return cached.(*regexp.Regexp).MatchString(actual) - } - escaped := regexp.QuoteMeta(pattern) - escaped = strings.ReplaceAll(escaped, "%", ".*") - escaped = strings.ReplaceAll(escaped, "_", ".") - re, err := regexp.Compile("(?i)^" + escaped + "$") - if err != nil { - return false - } - likeRegexCache.Store(pattern, re) - return re.MatchString(actual) -} - // compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, // or (0, false) when the types cannot be compared. func compareOrdered(actual, expected any) (int, bool) { diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 18216aba..3721df23 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -245,7 +245,8 @@ func TestEvaluateFilter(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if got := evaluateFilter(tt.actual, tt.op, tt.expected); got != tt.want { + cf := compileFilters([]QueryFilter{{Column: "c", Op: tt.op, Value: tt.expected}})[0] + if got := evaluateFilter(tt.actual, tt.op, tt.expected, cf.re); got != tt.want { t.Errorf("evaluateFilter(%v, %q, %v) = %v, want %v", tt.actual, tt.op, tt.expected, got, tt.want) } }) @@ -258,11 +259,11 @@ func TestMatchesFilters_AllMustMatch(t *testing.T) { {Column: "page", Op: "eq", Value: "/home"}, {Column: "score", Op: "gt", Value: 5}, } - if !matchesFilters(row, both) { + if !matchesFilters(row, compileFilters(both)) { t.Fatal("want match when every filter passes") } oneFails := append(append([]QueryFilter(nil), both...), QueryFilter{Column: "score", Op: "gt", Value: 99}) - if matchesFilters(row, oneFails) { + if matchesFilters(row, compileFilters(oneFails)) { t.Fatal("want no match when any filter fails") } if !matchesFilters(row, nil) { diff --git a/clients/go/table.go b/clients/go/table.go index 02bf4a0a..204a3b26 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -9,9 +9,9 @@ import ( "strings" ) -// TableRef is a reference to a table. Use it for queries, inserts, schema, and -// streams. NOT safe to use concurrently from multiple goroutines for mutations; -// reads (Fetch, Select, etc.) are safe. +// TableRef is a reference to a table. Use it for queries, inserts, schema, +// and streams. Safe for concurrent use: it holds no mutable state, and every +// builder method returns a fresh value. type TableRef struct { ctx httpContext table string diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 0cc2f4c3..24a640c5 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -35,26 +35,33 @@ every example on these pages assumes it. ## Quick Start ```go +package main + import ( "context" "fmt" + "log" wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ) -wh := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), -}) - -page, err := wh.From("clicks"). - Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Limit(10). - FetchUntyped(context.Background()) -if err != nil { /* handle */ } -for _, row := range page.Data { - fmt.Println(row["page"], row["button"]) +func main() { + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), + }) + + page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } } ``` @@ -63,8 +70,6 @@ See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/READM ## Creating a Client ```go -import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" - wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "https://wavehouse.example.com", Auth: func(ctx context.Context) (string, error) { @@ -91,6 +96,11 @@ wh := wavehouse.NewClient(wavehouse.Config{ |-------|------|---------|-------------| | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | +A `*Client` is safe for concurrent use by multiple goroutines — client state +is immutable after `NewClient`, and every builder chain copies. Supply a +concurrency-safe `Auth` func (it's called from any goroutine that issues a +request). + :::caution[`Options` opts you out of the default, not just in] The default of 2 retries only applies when `Config.Options` is `nil`. If you set `Options` to configure anything else in the future, an unset diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 6ef9380f..e790cdbb 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -136,9 +136,10 @@ page, err := clicks.Select("page", "button"). Start a query that selects **every column your role is allowed to read** — the explicit form of what `.Fetch()` does. Mutually exclusive with -`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); the -server expands it to your allowed columns (never a raw `SELECT *`) and never -bypasses `deny_columns`/`allow_columns`. See +`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); for a +column-restricted role the server expands it to exactly that role's allowed +columns rather than a bare `SELECT *` (unrestricted/admin roles do get +`SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). ```go @@ -182,9 +183,10 @@ q := clicks.Select("page").Select("button") // SELECT page, button #### `.SelectAll()` -Select every column your role may read (the all-columns wildcard, expanded -server-side to your allowed columns). Mutually exclusive with `.Select(...)` -and with aggregations (`.Count()`, `.Sum()`, etc.). +Select every column your role may read (the all-columns wildcard; a +column-restricted role's projection is expanded server-side to its allowed +columns). Mutually exclusive with `.Select(...)` and with aggregations +(`.Count()`, `.Sum()`, etc.). ```go q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") @@ -345,7 +347,8 @@ rows decode into `map[string]any`, where JSON numbers become `float64`, so an integer cursor column loses exactness past 2^53 and pagination can repeat or skip a row at that scale — the same ceiling the TypeScript SDK has with JS numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and -codegen structs are unaffected (their 64-bit integer columns are `string`). +codegen structs are unaffected (their 64-bit integer columns are `int64`/ +`uint64`, decoded exactly). ```go page, err := clicks.Select(). diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 271143b6..dd58c6bf 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -145,7 +145,7 @@ Generate Go structs from a running WaveHouse instance. The module ships a ```bash export WAVEHOUSE_AUTH= # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ --package myapp @@ -215,23 +215,28 @@ value field with `omitempty` would silently drop. |------------------|---------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | | `Bool` / `Boolean` | `bool` | -| `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | -| `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | | `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | -| `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` (arbitrary-width unquoted numbers on the structured-query path) | +| `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` | +| `Array(T)` | `[]T` (`Array(UInt8)` → `[]uint16`: `[]byte` would JSON-encode as base64, not an array) | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** up to 32 bits (`UInt32` → -`uint32`, not a generic `number`), since Go — unlike TypeScript — has -native fixed-width integer types. 64-bit integers stay `string` because -that is what actually arrives on the wire. +codegen preserves ClickHouse's integer **widths** (`UInt64` → `uint64`, not +a generic `number`), since Go — unlike TypeScript — has native fixed-width +integer types; 64-bit columns decode exactly where TS hits the JS-number +2^53 ceiling. Generated structs target the structured-query and pipe paths +(`/v1/query`, `/v1/pipes/*`), where the server re-marshals values as plain +JSON numbers. The raw-SQL path (`/v1/admin/query`) instead forwards +ClickHouse's own JSON, which **quotes** 64-bit-and-wider integers — use +`map[string]any` with `SQL[Row]` there rather than generated structs. ## Testing diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 90107898..093ff1ae 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -143,6 +143,13 @@ type StreamEvent struct { } ``` +:::note[`Events()` starts feeding on first call] +The channel only receives events emitted **after** the first `Events()` +call — a stream you set up but don't consume yet buffers nothing for the +channel. Call `Events()` immediately after `.Stream()` (or use +`.Subscribe`) if you can't start ranging right away. +::: + ### Transport Behavior | Transport | Reconnect | Protocol | diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 7eead8ba..50e1e8e3 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -86,7 +86,7 @@ const { data } = await clicks.select('page', 'button').where('page', '=', '/home ### `.selectAll()` -Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); the server expands it to your allowed columns (never a raw `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). +Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). ```ts const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); From 978451c24c9b4ad49b4a381159745be20992aeda Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:10:19 -0400 Subject: [PATCH 12/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=205=20=E2=80=94=20Array(UInt8)=20round-trip,=20SSE=20r?= =?UTF-8?q?esume=20test,=20docs=20accuracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - codegen: Array(UInt8) maps to json.RawMessage — the wire is asymmetric (ingest requires a JSON array; /v1/query responses base64-encode the column because the server scans it into []byte and doesn't normalize it), so neither []uint8 nor the round-4 []uint16 could decode a query response; RawMessage round-trips both directions. Server-side normalization tracked in #436 - New reconnect-resume test pins the gap-fill contract: initial request carries StreamOptions.Since, the reconnect carries ?since= - docs: raw-SQL typed example decodes count() (UInt64, quoted on the admin path) via a `,string` tag instead of a plain int that fails; Insert's []byte rule stated the right way around; select_all "never a raw SELECT *" claim qualified for unrestricted/admin roles in api.md, access-control.mdx, architecture.md (matching the SDK pages); SDK health ping named in both languages in api.md, reverse-proxy.mdx, deployment.md; landing-page Go pointer no longer introduces TS-only tabs with a colon - make test-all runs test-conformance-ts, so "all suites" stays true --- Makefile | 1 + clients/go/cmd/wavehouse-codegen/main.go | 12 ++-- clients/go/cmd/wavehouse-codegen/main_test.go | 5 +- clients/go/stream_test.go | 58 +++++++++++++++++++ docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/api.md | 6 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/deployment.md | 2 +- docs/src/content/docs/index.mdx | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/go/queries.md | 14 +++-- docs/src/content/docs/sdk/go/reference.md | 2 +- 12 files changed, 88 insertions(+), 20 deletions(-) diff --git a/Makefile b/Makefile index 243ddcfa..f468fff6 100644 --- a/Makefile +++ b/Makefile @@ -756,6 +756,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 @$(MAKE) test-go-sdk + @$(MAKE) test-conformance-ts @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index f46998c1..8861860e 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -215,11 +215,13 @@ func chTypeToGo(chType string) string { // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { inner := chTypeToGo(chType[6 : len(chType)-1]) - // []uint8 is []byte, which encoding/json base64-encodes as a string — - // the server requires a real JSON array for Array(...) columns, so - // widen the element type instead. + // Array(UInt8) is asymmetric on the wire: ingest requires a real JSON + // array, but /v1/query responses currently base64-encode it (the + // server scans into []byte and encoding/json base64s that — #436). + // json.RawMessage is the + // only shape that round-trips both directions without a decode error. if inner == "uint8" { - inner = "uint16" + return "json.RawMessage" } return "[]" + inner } @@ -298,7 +300,7 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { needsJSON := false for _, name := range names { for _, col := range schemas[name].Columns { - if strings.Contains(chTypeToGo(col.Type), "json.Number") { + if strings.Contains(chTypeToGo(col.Type), "json.") { needsJSON = true } } diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 47b2ce3e..f6207ddc 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -44,8 +44,9 @@ func TestChTypeToGo(t *testing.T) { {"SimpleAggregateFunction(any)", "any"}, {"Array(String)", "[]string"}, {"Array(Nullable(Int32))", "[]*int32"}, - // []uint8 is []byte → base64 on marshal; must widen. - {"Array(UInt8)", "[]uint16"}, + // []uint8 is []byte → base64 on marshal; RawMessage round-trips both + // the ingest array form and the (currently base64) query response. + {"Array(UInt8)", "json.RawMessage"}, {"Map(String, UInt32)", "map[string]uint32"}, {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, {"Tuple(String, UInt8)", "any"}, diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 3721df23..21be46f2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/http/httptest" + "sync" "sync/atomic" "testing" "time" @@ -341,3 +342,60 @@ func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { t.Fatal("Connected must fail on a terminally-closed stream") } } + +// TestStream_ReconnectResumesFromLastEventID: the gap-fill contract. The +// initial request carries StreamOptions.Since; after the connection drops, +// the reconnect carries ?since=. +func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { + var mu sync.Mutex + var sinceParams []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + sinceParams = append(sinceParams, r.URL.Query().Get("since")) + n := len(sinceParams) + mu.Unlock() + + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + if n == 1 { + _, _ = io.WriteString(w, sseFrame("2026-01-01T00:00:01Z", "/home")) + fl.Flush() + return // server closes → client must reconnect with since= + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(&StreamOptions{Since: "seed-id"}) + defer stream.Close() + + got := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + recvEvent(t, got) + + // Reconnect happens after ~backoff(0) (≈1s with jitter). + deadline := time.Now().Add(10 * time.Second) + for { + mu.Lock() + n := len(sinceParams) + mu.Unlock() + if n >= 2 { + break + } + if time.Now().After(deadline) { + t.Fatal("stream never reconnected") + } + time.Sleep(20 * time.Millisecond) + } + + mu.Lock() + defer mu.Unlock() + if sinceParams[0] != "seed-id" { + t.Fatalf("initial request: want since=seed-id, got %q", sinceParams[0]) + } + if sinceParams[1] != "2026-01-01T00:00:01Z" { + t.Fatalf("reconnect: want since=, got %q", sinceParams[1]) + } +} diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 69c21cb7..8a25cd1e 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -174,7 +174,7 @@ The rules, in order: 2. **An empty (or `["*"]`) `allow_columns` means "all columns"** — every column not in `deny_columns` is permitted. Use this with `deny_columns` for a blocklist posture: see everything *except* a few sensitive columns. 3. **A non-empty `allow_columns` is an allowlist** — only the named columns (and never the denied ones) are permitted. -On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`, which expands to exactly the columns the role may read — never a raw `SELECT *` that could include a denied column; if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. +On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`; for a column-restricted role it expands to exactly the columns the role may read rather than a bare `SELECT *` that could include a denied column (an unrestricted/admin role does get `SELECT *`); if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. ## Row-level security diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 3302b5af..91aa3e72 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -150,7 +150,7 @@ Status code: `503 Service Unavailable` Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. -This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. --- @@ -416,7 +416,7 @@ curl -X POST http://localhost:8080/v1/admin/query \ Executes a type-safe structured query against a table. The query AST is validated against the schema and converted to parameterized SQL. Permissions from the access control policy are enforced (column filtering, row-level security, aggregation restrictions). :::note[The column allowlist is a hard cap on every clause] -Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (for a column-restricted role, expanded to exactly the columns the role may read rather than a bare `SELECT *`; unrestricted/admin roles do get `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). ::: **Request:** @@ -444,7 +444,7 @@ Every column the query references — in `columns`, an aggregation argument, `fi | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `columns` | string \| string[] | No | Columns to SELECT — an array, or a single string for one column. A literal `"*"` is the column *named* `*`, **not** a wildcard. Omit (or send `[]` / `""`) to select nothing; use `select_all` for a full-row read. Mutually exclusive with `select_all`. | -| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard, expanded server-side to the allow/deny set). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | +| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard; a column-restricted role's projection is expanded server-side to its allow/deny set, an unrestricted/admin role gets `SELECT *`). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | | `aggregations` | object[] | No | Aggregation functions (`fn`, `column`, `alias`). | | `filters` | object[] | No | WHERE conditions (`column`, `op`, `value`). Ops: eq, neq, gt, gte, lt, lte, in, like. | | `group_by` | string[] | No | GROUP BY columns. | diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 8b8b6ff2..35eefd5f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -149,7 +149,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `query/` — Structured Query Engine - **ast.go** — `StructuredQuery` AST types: columns, aggregations, filters, group by, order by, limit, time range. -- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. +- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which for a column-restricted role expands to the role's allowed columns rather than emitting a raw `SELECT *` (unrestricted/admin roles get `SELECT *`); an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. ### `chsql/` — ClickHouse SQL Helpers diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 93e30086..bb72452a 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -264,7 +264,7 @@ API servers in standalone mode expose liveness and readiness endpoints under the Configure your load balancer or orchestrator to use these endpoints. -**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDK's content-free liveness ping, which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. +**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDKs' content-free liveness ping — `wh.sys.health()` / `wh.Sys.Health(ctx)` — which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. ### Boot-time degraded mode diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 9308a38e..aaffef84 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -104,7 +104,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket. -The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming. Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set: +The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming (examples below are TypeScript). Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set — see the [Go quick start](/sdk/go#quick-start). diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 34900c0e..e462e4ea 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -156,7 +156,7 @@ WaveHouse serves Kubernetes-convention probes on `:8080` (full behavior in [Depl - **`/v1/health`** — the SDK's content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. :::caution[Recommended: keep the bare probe paths off the public vhost] -Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDK's `wh.sys.health()` calls. +Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`). ::: ## Timeouts and slow links diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index e790cdbb..9b88fd77 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -51,8 +51,11 @@ see [Pagination](#pagination). Insert one row or many. What you pass determines the wire format: -- A single **map or struct** (anything that isn't a slice, and isn't - `[]byte`) is sent as JSON: `POST /v1/ingest?table={table}`. +- A single **map or struct** (anything that isn't a slice — plus `[]byte`, + which is treated as one opaque value rather than a batch of numbers, and + would reach the server as a base64 string it rejects; pass raw NDJSON + through `.InsertNDJSON(ctx, string(raw))` instead) is sent as JSON: + `POST /v1/ingest?table={table}`. - **Any slice** — `[]map[string]any`, a generated/user-defined row type like `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via reflection for non-`[]map[string]any` slices) and sent as a single @@ -385,10 +388,13 @@ authorizes `/v1/admin/*` without a JWT. Package-level generic function — use rows, err := wavehouse.SQL[map[string]any](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") -// Or decode into a struct that matches the projected columns/aliases: +// Or decode into a struct that matches the projected columns/aliases. +// NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit +// integers (count() is UInt64) — decode them with the `,string` tag, or +// use map[string]any. See Reference → Codegen CLI for the full story. type PageTotal struct { Page string `json:"page"` - Total int `json:"total"` + Total uint64 `json:"total,string"` } typed, err := wavehouse.SQL[PageTotal](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index dd58c6bf..0074b7f0 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -223,7 +223,7 @@ value field with `omitempty` would silently drop. | `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` (`Array(UInt8)` → `[]uint16`: `[]byte` would JSON-encode as base64, not an array) | +| `Array(T)` | `[]T` — except `Array(UInt8)` → `json.RawMessage`: the wire is asymmetric (ingest takes a JSON array, but query responses currently base64-encode the column), and `RawMessage` is the one shape that decodes both; server-side normalization tracked in [#436](https://github.com/Wave-RF/WaveHouse/issues/436) | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | From 3a2640144ad7cf4be8dd898955bb13eafb78b846 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:26:04 -0400 Subject: [PATCH 13/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=206=20=E2=80=94=20e2e=20container=20gate,=20module=20f?= =?UTF-8?q?loor,=20test/docs=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - e2e buildMarkerRow: container types (Array/Map/Tuple/Nested) are gated before the substring cases — "array(string)" contains "string", so the skip branch was unreachable and a scalar marker went into array columns - TestStream_HandleMalformedSSEData sets chanRequested so its no-emit assertion is real instead of vacuously passing on the opt-in guard - clients/go/go.mod floor lowered 1.26.5 → 1.24: the SDK needs nothing past Go 1.22 (range-over-int, math/rand/v2) and a patch-pinned floor exists for the server's stdlib CVEs, not for a zero-dep library consumers go get - development.md/CONTRIBUTING.md: gotestsum/ARGS/V=1/race claims scoped to the instrumented suites; nested-module gofumpt path documented; SDK test locations added - queries.md: operator-key note explains Config.Auth always sends Bearer — use a custom HTTPClient transport for X-Operator-Key --- CONTRIBUTING.md | 4 ++-- clients/go/e2e_test.go | 12 +++++++++--- clients/go/go.mod | 5 ++++- clients/go/stream_test.go | 4 +++- docs/src/content/docs/development.md | 9 +++++---- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 7 +++++-- 7 files changed, 29 insertions(+), 14 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b459bd7..d831fcff 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,7 +39,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t The pre-push hook (installed by `make tools`) blocks a push until the tree has been validated locally: a code change needs `make ci`, a docs/prose-only change needs only `make verify` (the same split CI makes). `make lint` / `make test` / `make build` are fast inner-loop subsets. -2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests go in `tests/` with the `//go:build integration` tag. +2. Write tests for new functionality. Unit tests go alongside the code in `internal/`; SDK tests live in `clients/ts/src/` and `clients/go/`. Integration tests go in `tests/` with the `//go:build integration` tag. 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` @@ -90,7 +90,7 @@ test(cache): add tiered cache stampede test ## Code Style -- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks it (CI runs the same target); `make fix` applies it. +- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks the root module; the nested `clients/go` module is checked by `make verify` (its `verify-go-sdk` leaf, which the pre-commit hook and CI run). `make fix` applies gofumpt to both. - **Linting**: All lint checks in `.golangci.yml` must pass (see `make lint`). - **Naming**: Follow [Go naming conventions](https://go.dev/doc/effective_go#names). - **Interfaces**: Define interfaces where they are consumed, not where they are implemented. diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 23f59221..3bce0341 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -372,6 +372,12 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, st } ct := strings.ToLower(col.Type) switch { + case strings.Contains(ct, "array(") || strings.Contains(ct, "map(") || + strings.Contains(ct, "tuple(") || strings.Contains(ct, "nested("): + // Container types must be gated BEFORE the substring cases below — + // "array(string)" contains "string" and would otherwise get a + // scalar marker injected into an array column. + t.Skipf("e2e: table %q requires container column %q of type %q", ts.Name, col.Name, col.Type) case !markerSet && strings.Contains(ct, "string"): row[col.Name] = mk markerCol = col.Name @@ -387,9 +393,9 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, st case strings.Contains(ct, "bool"): row[col.Name] = false default: - // No safe synthetic value for this type (Array, Map, Tuple, UUID, - // ...) — an empty string would make the insert fail with a type - // error that looks like an SDK defect. + // No safe synthetic value for this type (UUID, IPv6, ...) — an + // empty string would make the insert fail with a type error that + // looks like an SDK defect. t.Skipf("e2e: table %q requires column %q of unsupported type %q", ts.Name, col.Name, col.Type) } } diff --git a/clients/go/go.mod b/clients/go/go.mod index 84e065de..b78eaa50 100644 --- a/clients/go/go.mod +++ b/clients/go/go.mod @@ -1,3 +1,6 @@ module github.com/Wave-RF/WaveHouse/clients/go -go 1.26.5 +// Library floor, deliberately lower than the server's pinned toolchain: +// the newest things this module uses are range-over-int and math/rand/v2 +// (Go 1.22). Keep it a supported-releases floor, not a patch pin. +go 1.24 diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 21be46f2..bcf782dd 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -201,7 +201,9 @@ func TestStream_FilteredCloseUnderLoad(t *testing.T) { } func TestStream_HandleMalformedSSEData(t *testing.T) { - sc := &StreamController{eventCh: make(chan StreamEvent, 1)} + // chanRequested must be true or emitEvent skips the channel entirely and + // the no-emit assertion below would pass vacuously. + sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} sc.handleSSEData("not json", "id1") // must not panic or emit select { case e := <-sc.eventCh: diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index ea642152..d3b2c7fc 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,9 +288,9 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -All test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. +The coverage-instrumented suite targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary, and accept `ARGS`/`V=1`. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. -All tests run with Go's **race detector** (`-race`) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. +Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. ### Quick Reference @@ -300,7 +300,8 @@ All tests run with Go's **race detector** (`-race`) enabled by default. WaveHous # Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test -# Run specific test(s) +# Run specific root-module test(s) — ARGS reaches test-unit only; the +# test-go-sdk half of `make test` runs its full suite regardless make test ARGS="-run TestValidate" # Go integration tests (requires Docker) @@ -518,7 +519,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -All test targets accept `ARGS="..."` for pass-through `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` switches to verbose `gotestsum` output. +The gotestsum-driven targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) accept `ARGS="..."` for pass-through `go test` flags and `V=1` for verbose output; `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 24a640c5..c292bc5c 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -20,7 +20,7 @@ either page mostly carries over. go get github.com/Wave-RF/WaveHouse/clients/go ``` -Requires Go 1.26.5 or later (the minimum pinned in the module's `go.mod`). +Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a supported-releases floor rather than the server's patch-pinned toolchain). ## Import diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 9b88fd77..e1572c9f 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -381,8 +381,11 @@ the token must resolve to the policy admin role (`admin_role`, `"admin"` by default) — a JWT request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. Alternatively, a configured operator key (`Authorization: Operator ` or `X-Operator-Key`) -authorizes `/v1/admin/*` without a JWT. Package-level generic function — use -`map[string]any` for a dynamic/unknown schema. +authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always +sends its token as `Bearer `, so to use an operator key from this +SDK supply a `Config.HTTPClient` whose `Transport` sets the +`X-Operator-Key` header on each request. Package-level generic function — +use `map[string]any` for a dynamic/unknown schema. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, From 10cb253800496e0963f625b12f7a41316e59a011 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:42:36 -0400 Subject: [PATCH 14/59] test(sdk): guard conformance request capture with a mutex The handler-goroutine write / test-goroutine read pattern was mutex-guarded everywhere else in the package after review; conformance_test.go was the one holdout. No happens-before edge exists via the TCP socket, so this was a latent race -race hadn't tripped yet. --- clients/go/conformance_test.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index e41c7b09..c30bb6ba 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -10,6 +10,7 @@ import ( "net/url" "reflect" "strings" + "sync" "testing" ) @@ -75,13 +76,18 @@ func TestConformance_WireFormat(t *testing.T) { for _, tc := range cases { t.Run(tc.Name, func(t *testing.T) { + // capt is written on the server goroutine and read on the test + // goroutine; the mutex is what makes that visible under -race. + var mu sync.Mutex var capt captured srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() capt.method = r.Method capt.path = r.URL.RequestURI() capt.contentType = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) capt.body = string(raw) + mu.Unlock() // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") @@ -214,6 +220,8 @@ func TestConformance_WireFormat(t *testing.T) { } // Verify method. + mu.Lock() + defer mu.Unlock() if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) } From cade9d7de6830b7422cf9e373a8d4b098a93a807 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:47:02 -0400 Subject: [PATCH 15/59] =?UTF-8?q?docs(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=207=20=E2=80=94=20channel=20error=20visibility,=20agg?= =?UTF-8?q?=20allowlist,=20titles?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - streaming.md: Events() carries events only — Error/Status arrive solely via Subscribe, so a channel-only consumer sees the range end silently on a terminal 401/403/404; documented with the pairing recommendation - Both SDK query pages document the server's aggregation-function allowlist (isValidAggFn) instead of implying any custom fn works - TS reference gains the SSE_CONNECT_ERROR row (auth provider threw / invalid baseURL — the more reachable of the two stream codes) - "the SDK's" → "the SDKs'" in reverse-proxy.mdx and architecture.md, matching the sweep already applied to api.md/deployment.md - Five TS topic pages retitled "TypeScript SDK …" now that two SDK doc trees sit side by side in search and tabs - query_builder.go comment no longer claims codegen 64-bit columns are strings (they're int64/uint64; 128/256-bit are json.Number) --- clients/go/query_builder.go | 4 ++-- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/admin.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 9 ++++++++- docs/src/content/docs/sdk/go/streaming.md | 8 ++++++++ docs/src/content/docs/sdk/pipes.md | 2 +- docs/src/content/docs/sdk/queries.md | 11 +++++++++-- docs/src/content/docs/sdk/reference.md | 3 ++- docs/src/content/docs/sdk/streaming.md | 2 +- 10 files changed, 34 insertions(+), 11 deletions(-) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 07c16783..a8453e16 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -281,8 +281,8 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro // protection: its rows were already decoded to float64 by // encoding/json, so precision above 2^53 is gone before we get here — // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or - // codegen structs, whose 64-bit int columns are strings) when paging - // on >2^53 integer cursors. + // codegen structs — their 64-bit int columns are int64/uint64, and + // 128/256-bit are json.Number) when paging on >2^53 integer cursors. raw, _ := json.Marshal(lastRow) m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 35eefd5f..617e26b9 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -79,7 +79,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Each connection registers one `Subscriber` (the `stream/` package) with both the event `Hub` (under its `(topic, role)`) and the shared keepalive wheel, then drains both from a single byte-pump — so idle streams keep emitting `:` keepalive comments (surviving reverse-proxy idle timeouts) while live events arrive already projected and serialized. Per-event projection/serialization happens **once per role** in the `Hub`, not once per subscriber ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)). Gap-fill replay from NATS JetStream (`DeliverByStartTime`) stays per-connection (low-volume, one-time on connect). - **schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh. - **dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream. -- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. +- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDKs' public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. ### `stream/` — SSE keepalive & fan-out diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index e462e4ea..d2e647fc 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -153,7 +153,7 @@ WaveHouse serves Kubernetes-convention probes on `:8080` (full behavior in [Depl - **`/livez`** — liveness; sticky-200 after first successful boot. Does not touch ClickHouse. - **`/readyz`** — readiness; issues a ClickHouse `Ping` on **every** call. Point your load balancer's (internal) health check here so it routes around an instance whose ClickHouse is unreachable. - **`/healthz`** — permanent alias of `/livez`. -- **`/v1/health`** — the SDK's content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. +- **`/v1/health`** — the SDKs' content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. :::caution[Recommended: keep the bare probe paths off the public vhost] Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`). diff --git a/docs/src/content/docs/sdk/admin.md b/docs/src/content/docs/sdk/admin.md index 2a4971b5..453e7163 100644 --- a/docs/src/content/docs/sdk/admin.md +++ b/docs/src/content/docs/sdk/admin.md @@ -1,5 +1,5 @@ --- -title: "SDK Admin & System" +title: "TypeScript SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index e1572c9f..a7ff744c 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -227,9 +227,16 @@ clicks.Select("page"). Min("score", "min_score"). // MIN(score) Max("score", "max_score"). // MAX(score) CountDistinct("page", "unique_pages"). - Aggregate("uniqExact", "user_id", "unique_users") // custom fn + Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn ``` +Custom function names pass through `.Aggregate(fn, column, alias)` but are +validated server-side against a fixed allowlist (matched case-insensitively): +`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, +`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, +`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected +with `400 unsupported aggregation function`. + `Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 093ff1ae..f491795b 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -143,6 +143,14 @@ type StreamEvent struct { } ``` +:::note[`Events()` carries events only] +`Error` and `Status` are delivered exclusively through `.Subscribe(...)` — +the channel is typed `chan StreamEvent` and simply ends (closes) when the +stream closes, including on a terminal 401/403/404. Pair `Events()` with a +`Subscribe(&StreamSubscriber{Error: ..., Status: ...})` if you need to know +*why* a stream ended. +::: + :::note[`Events()` starts feeding on first call] The channel only receives events emitted **after** the first `Events()` call — a stream you set up but don't consume yet buffers nothing for the diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md index 3e1cca2f..86c0d763 100644 --- a/docs/src/content/docs/sdk/pipes.md +++ b/docs/src/content/docs/sdk/pipes.md @@ -1,5 +1,5 @@ --- -title: "SDK Pipes" +title: "TypeScript SDK Pipes" description: "Execute and manage named query pipes with @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 50e1e8e3..a75ee3e5 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -1,5 +1,5 @@ --- -title: "SDK Queries" +title: "TypeScript SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in @wavehouse/sdk." --- @@ -162,9 +162,16 @@ clicks.select('page') .min('score', 'min_score') // MIN(score) .max('score', 'max_score') // MAX(score) .countDistinct('page', 'unique_pages') - .aggregate('uniqExact', 'user_id', 'unique_users') // custom fn + .aggregate('uniqExact', 'user_id', 'unique_users') // allowlisted fn ``` +Custom function names pass through `.aggregate(fn, column, alias)` but are +validated server-side against a fixed allowlist (matched case-insensitively): +`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, +`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, +`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected +with `400 unsupported aggregation function`. + Each aggregation method signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 5d54ca4b..cef0e82d 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -1,5 +1,5 @@ --- -title: "SDK Reference & CLI" +title: "TypeScript SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- @@ -38,6 +38,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `AbortSignal` | | 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the stream's error callback; the stream reconnects automatically | +| 0 | `SSE_CONNECT_ERROR` | Yes | Stream could not be opened (auth provider threw, invalid `baseURL`) | --- diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 2a9868a5..d0297878 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -1,5 +1,5 @@ --- -title: "SDK Streaming & Live Queries" +title: "TypeScript SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." --- From 791c198b71fab92f2b71f12ab2dd40a582e05264 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:03:09 -0400 Subject: [PATCH 16/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=208=20=E2=80=94=20dead=20param,=20error=20wrapping,=20?= =?UTF-8?q?doc=20accuracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code (pre-push-reviewer): - Drop unused limit param from fetchNextTyped; FetchTyped recomputes it from the cloned builder's state. - Wrap errors in Schema.List/Refresh and DLQ.stats with operation context, matching every other namespace. Docs (docs-reviewer): - development.md: make verify row no longer claims tidy/vulncheck cover clients/go (nested module — tracked in #437). - sdk/go/index.md: caution on timeouts — context deadlines, not http.Client.Timeout (which kills SSE streams). - sdk/go/queries.md: OpNotLike does emit not_like on the wire. - api.md: pluralize three leftover singular-SDK references. - clients/go/README.md: drop trailing slashes (trailingSlash: "never"). --- clients/go/README.md | 4 ++-- clients/go/dlq.go | 3 ++- clients/go/query_builder.go | 4 ++-- clients/go/schema.go | 14 ++++++++++---- docs/src/content/docs/api.md | 6 +++--- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/index.md | 11 ++++++++++- docs/src/content/docs/sdk/go/queries.md | 2 +- 8 files changed, 31 insertions(+), 15 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index cf2c1348..bd6109d3 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -4,7 +4,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a s **Zero third-party runtime dependencies** — stdlib only. -**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go/)** +**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** ## Install @@ -200,7 +200,7 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --package myapp ``` -See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/#codegen-cli). +See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#codegen-cli). ## Error Handling diff --git a/clients/go/dlq.go b/clients/go/dlq.go index e833a41a..0009e4e3 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "fmt" "net/url" ) @@ -28,7 +29,7 @@ func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, path: "/v1/dlq/stats", params: params, }, &stats); err != nil { - return nil, err + return nil, fmt.Errorf("get dlq stats: %w", err) } return &stats, nil } diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index a8453e16..06092bbc 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -179,7 +179,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). if hasMore && len(q.state.orderBy) > 0 { page.Next = func(ctx context.Context) (*Page[Row], error) { - return fetchNextTyped[Row](ctx, q, rows, limit) + return fetchNextTyped[Row](ctx, q, rows) } } @@ -265,7 +265,7 @@ func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { return ast } -func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row, limit int) (*Page[Row], error) { +func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row) (*Page[Row], error) { if len(q.state.orderBy) == 0 { return &Page[Row]{}, nil } diff --git a/clients/go/schema.go b/clients/go/schema.go index 3c7f4307..3f5b4a88 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // SchemaNamespace provides admin-only schema introspection. type SchemaNamespace struct { @@ -15,7 +18,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { method: "GET", path: "/v1/schema", }, &raw); err != nil { - return nil, err + return nil, fmt.Errorf("list schemas: %w", err) } schemas := make(Schemas, len(raw)) for _, t := range raw { @@ -26,8 +29,11 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // Refresh forces a schema re-discovery from ClickHouse. Admin-only. func (s *SchemaNamespace) Refresh(ctx context.Context) error { - return doRequest(ctx, s.ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", - }, nil) + }, nil); err != nil { + return fmt.Errorf("refresh schema: %w", err) + } + return nil } diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 91aa3e72..16254300 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -150,7 +150,7 @@ Status code: `503 Service Unavailable` Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. -This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDKs rely on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. --- @@ -254,7 +254,7 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ #### Batch Ingest -A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (The SDK's `insert([...])` array helper uses the NDJSON form automatically; both forms return the same response.) +A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (Both SDKs' array/slice insert helpers use the NDJSON form automatically; both forms return the same response.) - **JSON array** — the most convenient form from most HTTP clients. A structural JSON syntax error fails the whole request (`400`), but a wrong-typed element (a non-object) is reported per-record like any other rejection. An explicit empty array (`[]`) is a valid, record-less batch (`200`, `total: 0`). - **NDJSON** — the streaming-friendly form for very large uploads. Blank lines are skipped, and a single malformed *line* is reported and skipped (the newline reframes the next record). @@ -316,7 +316,7 @@ A `200` is returned whenever the body was read and the records were processed | 503 | `{"error":"service unavailable"}` | NATS JetStream full (backpressure) mid-batch; includes `Retry-After: 30` | :::caution[At-least-once on retry] -A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDKs retry both on `503`). ::: **curl example (JSON array):** diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index d3b2c7fc..cd9c6d46 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -488,7 +488,7 @@ Run `make help` to see all targets. Key ones: | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go incl. `clients/go` (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index c292bc5c..8e0b2329 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -88,7 +88,16 @@ wh := wavehouse.NewClient(wavehouse.Config{ | `BaseURL` | `string` | — | WaveHouse server URL (required) | | `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | | `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports (see caution below) | + +:::caution[Timeouts: use contexts, not `http.Client.Timeout`] +The default client sets no `Timeout` — a `context.Context` deadline is the +only bound on a request, so pass one for anything that mustn't hang on a +stalled server. If you supply your own `HTTPClient`, leave `Timeout` unset: +it covers body reads too, so it would kill every long-lived SSE stream at +the timeout and force a reconnect loop. Use `Transport`-level dial / +TLS / response-header timeouts instead. +::: ### `ClientOptions` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index a7ff744c..894f05a1 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -215,7 +215,7 @@ clicks.Select("page"). | `wavehouse.OpLte` | `lte` | Less than or equal | | `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | | `wavehouse.OpLike` | `like` | SQL LIKE pattern | -| `wavehouse.OpNotLike` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | #### Aggregations From 478b215d54cb49bc4c9ead39ea0eeb540e631e6a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:18:45 -0400 Subject: [PATCH 17/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=209=20=E2=80=94=20SSE=20error=20delivery,=20test=20rac?= =?UTF-8?q?es,=20parity=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code (pre-push-reviewer): - handleSSEData delivers malformed-frame errors via the subscriber Error callback instead of unbounded process-global logging; drop its unused eventID param. - table_test.go: mutex-guard all five handler-goroutine captures, matching the package-wide convention from 10cb253. - conformance_test.go: unhandled fixture endpoint is now t.Fatalf, matching the TS runner's skipped-cases-break-parity stance. - Convert ponytail: comment prefixes to TODO:/plain notes (repo uses TODO). - Move test-only errIs helper into http_test.go. Docs (docs-reviewer): - Both streaming pages: live-query dedup caution — projection must include received_timestamp or overlap-window events deliver twice; Go page also corrects the bound to max-across-rows (desc order puts oldest last). --- clients/go/conformance_test.go | 4 +++- clients/go/http.go | 10 -------- clients/go/http_test.go | 10 ++++++++ clients/go/query_builder.go | 6 ++--- clients/go/stream.go | 16 +++++++------ clients/go/stream_test.go | 8 ++++++- clients/go/table_test.go | 28 +++++++++++++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 12 ++++++++-- docs/src/content/docs/sdk/streaming.md | 4 ++++ 9 files changed, 74 insertions(+), 24 deletions(-) diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index c30bb6ba..26fbac07 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -216,7 +216,9 @@ func TestConformance_WireFormat(t *testing.T) { logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) default: - t.Skipf("unhandled endpoint: %s", tc.Endpoint) + // Hard failure, matching the TS runner: skipped cases break + // cross-SDK parity. + t.Fatalf("unhandled endpoint %q — wire it up in the dispatch switch", tc.Endpoint) } // Verify method. diff --git a/clients/go/http.go b/clients/go/http.go index 0fc59d9e..74dcbdad 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "encoding/json" - "errors" "fmt" "io" "math" @@ -207,12 +206,3 @@ func sleepWithContext(ctx context.Context, d time.Duration) error { return ctx.Err() } } - -// errIs checks if err wraps a *Error with the given code. -func errIs(err error, code string) bool { - var e *Error - if errors.As(err, &e) { - return e.Code == code - } - return false -} diff --git a/clients/go/http_test.go b/clients/go/http_test.go index aef9aa0f..df9045ff 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "errors" "io" "net/http" "net/http/httptest" @@ -11,6 +12,15 @@ import ( "time" ) +// errIs checks if err wraps a *Error with the given code. +func errIs(err error, code string) bool { + var e *Error + if errors.As(err, &e) { + return e.Code == code + } + return false +} + func testCtx(t *testing.T, handler http.Handler) httpContext { t.Helper() srv := httptest.NewServer(handler) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 06092bbc..6991371f 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -22,7 +22,7 @@ type queryState struct { orderBy []OrderClause limit *int timeRange *TimeRange - cacheTTL *int // ponytail: client-side only, not sent to server (#280) + cacheTTL *int // client-side only, not sent to server (#280) } // QueryBuilder builds structured queries. Immutable — every chain method @@ -179,7 +179,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). if hasMore && len(q.state.orderBy) > 0 { page.Next = func(ctx context.Context) (*Page[Row], error) { - return fetchNextTyped[Row](ctx, q, rows) + return fetchNextTyped(ctx, q, rows) } } @@ -275,7 +275,7 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro lastRow := any(prevRows[len(prevRows)-1]) m, ok := lastRow.(map[string]any) if !ok { - // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + // TODO: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. // UseNumber keeps typed int64 cursor values exact past 2^53. The // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this // protection: its rows were already decoded to float64 by diff --git a/clients/go/stream.go b/clients/go/stream.go index 6da259be..d1c4c206 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -22,7 +22,7 @@ type StreamController struct { mu sync.Mutex status StreamStatus subscribers []*StreamSubscriber - eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + eventCh chan StreamEvent // single buffered channel for Go-native consumption chanRequested bool // set by Events(); until then emitEvent skips the channel dropLogOnce sync.Once cancel context.CancelFunc @@ -103,7 +103,7 @@ func (sc *StreamController) Connected(ctx context.Context) error { sc.mu.Unlock() // Poll — simple and correct. - // ponytail: condition variable if polling shows up in profiles. + // TODO: switch to a condition variable if polling shows up in profiles. ticker := time.NewTicker(50 * time.Millisecond) defer ticker.Stop() for { @@ -338,7 +338,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if line == "" { // Empty line = end of event frame. if dataLine != "" { - sc.handleSSEData(dataLine, eventID) + sc.handleSSEData(dataLine) // Track last event ID for reconnect gap-fill. if eventID != "" { lastID = eventID @@ -376,12 +376,14 @@ type sseMessage struct { Data map[string]any `json:"data"` } -func (sc *StreamController) handleSSEData(data, eventID string) { +func (sc *StreamController) handleSSEData(data string) { var msg sseMessage if err := json.Unmarshal([]byte(data), &msg); err != nil { - // Deliberately omits the payload: event data can carry tenant/PII - // fields and this goes to the process-global logger. - log.Printf("[wavehouse] SSE received malformed message (%d bytes): %v", len(data), err) + // Delivered via the subscriber Error callback rather than the + // process-global logger, so consumers control visibility and a + // malformed-frame flood can't spam host-application logs. Payload + // deliberately omitted: event data can carry tenant/PII fields. + sc.emitError(fmt.Errorf("malformed SSE message (%d bytes): %w", len(data), err)) return } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index bcf782dd..c201fa2f 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/http/httptest" + "strings" "sync" "sync/atomic" "testing" @@ -204,12 +205,17 @@ func TestStream_HandleMalformedSSEData(t *testing.T) { // chanRequested must be true or emitEvent skips the channel entirely and // the no-emit assertion below would pass vacuously. sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} - sc.handleSSEData("not json", "id1") // must not panic or emit + var gotErr error + sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} + sc.handleSSEData("not json") // must not panic or emit select { case e := <-sc.eventCh: t.Fatalf("malformed data emitted event: %+v", e) default: } + if gotErr == nil || !strings.Contains(gotErr.Error(), "malformed SSE message") { + t.Fatalf("want malformed-SSE error via subscriber, got %v", gotErr) + } } // --------------------------------------------------------------------------- diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 5116d53f..e935657f 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -6,13 +6,19 @@ import ( "io" "net/http" "net/http/httptest" + "sync" "testing" ) func TestTableRef_InsertSingle(t *testing.T) { + // mu guards handler captures throughout this file: the handler runs on the + // server goroutine and no happens-before edge exists via the TCP socket. + var mu sync.Mutex var gotBody map[string]any var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotPath = r.URL.Path _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) @@ -26,6 +32,8 @@ func TestTableRef_InsertSingle(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } @@ -35,9 +43,12 @@ func TestTableRef_InsertSingle(t *testing.T) { } func TestTableRef_InsertBatch(t *testing.T) { + var mu sync.Mutex var gotCT string var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) @@ -57,6 +68,8 @@ func TestTableRef_InsertBatch(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotCT != "application/x-ndjson" { t.Fatalf("want ndjson content type, got %s", gotCT) } @@ -75,9 +88,12 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { Page string `json:"page"` } + var mu sync.Mutex var gotCT string var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) @@ -94,6 +110,8 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotCT != "application/x-ndjson" { t.Fatalf("want ndjson content type, got %s", gotCT) } @@ -114,8 +132,11 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { // TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through // insertSingle rather than being (mis)treated as a slice of per-byte rows. func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { + var mu sync.Mutex var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) @@ -128,6 +149,8 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } @@ -152,8 +175,11 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { } func TestTableRef_InsertNDJSON(t *testing.T) { + var mu sync.Mutex var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() raw, _ := io.ReadAll(r.Body) gotBody = string(raw) _ = json.NewEncoder(w).Encode(map[string]any{ @@ -170,6 +196,8 @@ func TestTableRef_InsertNDJSON(t *testing.T) { if result.Total == nil || *result.Total != 2 { t.Fatalf("want total=2, got %v", result.Total) } + mu.Lock() + defer mu.Unlock() if gotBody != ndjson { t.Fatalf("want raw NDJSON, got %s", gotBody) } diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index f491795b..011c3c6e 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -257,14 +257,22 @@ Decode into your own type inside the callback if you need one. 1. Subscribes to the stream **immediately** and buffers incoming events. 2. Runs the `.FetchUntyped(ctx)` query for historical data, calls `sub.Initial(rows, err)` with the result. -3. Deduplicates buffered events by comparing timestamps against the latest - historical row's `received_timestamp`. +3. Deduplicates buffered events against the **newest** `received_timestamp` + in the backfill — the maximum across all rows, not the last row's (an + `OrderBy(..., "desc")` puts the *oldest* row last). 4. Flushes remaining buffered events (re-checking for anything that arrived mid-flush) and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. +:::caution[Dedup needs `received_timestamp` in the projection] +The dedup bound comes from the backfill rows' `received_timestamp` values. +`.SelectAll()` (or no projection) includes it; a `.Select(...)` projection +that omits it disables dedup, and events in the fetch/stream overlap window +are delivered twice — once in `Initial`, again via `Next`. +::: + ### `.Close()` Shuts down the live query and its underlying stream. Safe to call more than diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index d0297878..7bc4b7fb 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -158,3 +158,7 @@ interface StreamSubscriber { 4. Flushes remaining buffered events and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. + +:::caution[Dedup needs `received_timestamp` in the projection] +The dedup boundary comes from the fetched rows' `received_timestamp` values. A `.select(...)` projection that omits that column disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `initial()`, again via `next()`. +::: From a841f1a8cb464411e0859601bf856a576f87aa23 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:34:37 -0400 Subject: [PATCH 18/59] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=2010=20=E2=80=94=20terminal=20StatusClosed,=20doc=20pr?= =?UTF-8?q?ecision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - stream.go: StatusClosed is terminal in setStatus; a filtered wrapper's stale inner Status callback can land after Close and must not resurrect the status. - conformance_ts.mjs: comment no longer claims the Go harness skips unhandled endpoints (it hard-fails since round 9). - sdk/go/queries.md: scope the ctx/(T, error) claim to request-response operations, matching the sibling pages. - sdk/go/streaming.md: note the one-time drop log line; document that Auth provider errors during (re)connect retry forever (MaxRetries bounds request retries only). PR #434 body refreshed separately (stale vulncheck note, commit list, test counts). --- clients/go/stream.go | 5 ++++- docs/src/content/docs/sdk/go/queries.md | 5 +++-- docs/src/content/docs/sdk/go/streaming.md | 11 ++++++++--- tests/conformance/conformance_ts.mjs | 5 +++-- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/clients/go/stream.go b/clients/go/stream.go index d1c4c206..3bab2ac2 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -144,7 +144,10 @@ func (sc *StreamController) Close() { func (sc *StreamController) setStatus(s StreamStatus) { sc.mu.Lock() - if s == sc.status { + // StatusClosed is terminal: a filtered wrapper's inner controller can + // have copied its subscriber slice before unsub, so a stale Status + // callback may land after Close — it must not resurrect the status. + if s == sc.status || sc.status == StatusClosed { sc.mu.Unlock() return } diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 894f05a1..d63c6715 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -5,8 +5,9 @@ description: "Tables, the chainable query builder, pagination, and raw SQL in th Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the -admin-only raw-SQL escape hatch. Every call takes a `context.Context` as its -first argument and returns `(T, error)` — see +admin-only raw-SQL escape hatch. Every request-response operation takes a +`context.Context` as its first argument and returns `(T, error)`; the +chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 011c3c6e..30c46a3c 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -85,8 +85,9 @@ exit path) regardless of which consumption style you use. The channel is buffered (256 events); a slow consumer that never drains it causes the SDK to **drop** new events for that channel rather than block the -stream's read loop (`.Subscribe` callbacks still fire per event -regardless of channel backpressure). +stream's read loop — the first drop logs one line via the standard `log` +package, further drops are silent (`.Subscribe` callbacks still fire per +event regardless of channel backpressure). ### `.Close()` @@ -168,7 +169,11 @@ Reconnect covers transport failures and retryable (5xx) responses. A non-retryable response (401/403/404) is terminal: the error is delivered to the subscriber's `Error` callback, status goes to `StatusClosed`, and the stream does not reconnect — fix the cause (refresh the token, correct the -table) and open a new stream. +table) and open a new stream. An `Auth` provider error during a (re)connect +is treated as retryable (`SSE_ERROR`) and the stream keeps reconnecting — +`ClientOptions.MaxRetries` bounds request retries only, not stream +reconnects — so call `.Close()` if your token provider is failing +permanently. Auth is sent as an `Authorization: Bearer` header on every stream (re)connection — see diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index d430bcfe..4996b594 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -233,8 +233,9 @@ for (const tc of cases) { await wh.pipes.delete(tc.pipe_name); break; default: - // Not a pass — the Go harness skips these too. Fixture cases with a - // new endpoint value must be wired up here before they count. + // Not a pass — the Go harness hard-fails on these; we count and exit + // non-zero below. Fixture cases with a new endpoint value must be + // wired up here before they count. skipped++; skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); continue; From 6a4e189ed60fbb784760870ed3d9263648469e71 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 20:45:52 -0400 Subject: [PATCH 19/59] =?UTF-8?q?fix(sdk):=20address=20verified=20Codex=20?= =?UTF-8?q?review=20findings=20=E2=80=94=20Events()=20buffering,=20filter?= =?UTF-8?q?=20kinds,=20make=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit External Codex review of PR #434 produced 8 findings; each was verified against source and git history before acting. Fixed here (4): - stream.go: Events() channel buffers from construction again, restoring TS parity (controller.ts buffers unconditionally). The chanRequested opt-in gate (round 1) silently discarded events emitted before the first Events() call; the dropLogOnce once-only drop log it was added alongside is kept. New test pins the buffering behavior. - stream.go: toFloat64 handles all int/uint kinds; a codegen-typed narrow/unsigned operand in an ordered .Where() filter silently dropped matching live events. Test table extended. - stream.go: scanner-cap comment cited the wrong bound — the binding per-event ceiling is the embedded NATS 1 MiB MaxPayload, not the 16 MiB HTTP ingest cap. - types.go: MaxRowsToRead *int -> *int64, matching the server's wire type (internal/policy: int64, can exceed 2^31). - Makefile: fix-go ran gofumpt/goimports via `go tool` inside the nested module, which has no tool directives — `make fix` failed with "go: no such tool" (regression from round 1). Formatters now run from the root module against clients/go; tidy + golangci-lint keep their nested cd. Remaining findings triaged to issues (drafted separately): failed-backfill buffer flush (Go+TS), LIKE case semantics (Go+TS+server), Nullable+DEFAULT explicit-null collapse (codegen). --- Makefile | 2 +- clients/go/stream.go | 58 +++++++++++++++-------- clients/go/stream_test.go | 31 ++++++++++-- clients/go/types.go | 2 +- docs/src/content/docs/sdk/go/streaming.md | 12 +++-- 5 files changed, 74 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index f468fff6..2985b92b 100644 --- a/Makefile +++ b/Makefile @@ -452,7 +452,7 @@ fix-go: $(GOLANGCI_LINT) @$(GOIMPORTS) -w $(GO_DIRS) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" - @cd clients/go && go mod tidy && $(GOFUMPT) -w . && $(GOIMPORTS) -w . && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @$(GOFUMPT) -w clients/go && $(GOIMPORTS) -w clients/go && cd clients/go && go mod tidy && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install diff --git a/clients/go/stream.go b/clients/go/stream.go index 3bab2ac2..b6402489 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -19,15 +19,14 @@ import ( // StreamController manages a live SSE event stream. Use Subscribe for // callback-based consumption or Events for channel-based consumption. type StreamController struct { - mu sync.Mutex - status StreamStatus - subscribers []*StreamSubscriber - eventCh chan StreamEvent // single buffered channel for Go-native consumption - chanRequested bool // set by Events(); until then emitEvent skips the channel - dropLogOnce sync.Once - cancel context.CancelFunc - done chan struct{} - closed bool + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // single buffered channel for Go-native consumption + dropLogOnce sync.Once + cancel context.CancelFunc + done chan struct{} + closed bool } // newStreamController opens an SSE connection for the given table. @@ -78,13 +77,12 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { } // Events returns a read-only channel that receives stream events. -// The channel is closed when the stream closes. Events are fed to the -// channel only from the first Events() call onward — a Subscribe-only -// consumer never fills (and overflows) a channel it isn't reading. +// The channel is closed when the stream closes. Events buffer into it from +// stream construction (matching the TS SDK), so events that arrive before +// the first Events() call are not lost. A Subscribe-only consumer that never +// calls Events() at most fills the 256-slot buffer and trips the one-time +// drop log. func (sc *StreamController) Events() <-chan StreamEvent { - sc.mu.Lock() - sc.chanRequested = true - sc.mu.Unlock() return sc.eventCh } @@ -173,12 +171,13 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } - // Non-blocking send to the channel. Guarded by mu so the send and - // closeEventCh serialize — a late event can never hit a closed channel — - // and skipped entirely until Events() opts in. + // Non-blocking send to the channel, which buffers from construction (TS + // parity) so events emitted before the first Events() call survive. + // Guarded by mu so the send and closeEventCh serialize — a late event can + // never hit a closed channel. sc.mu.Lock() defer sc.mu.Unlock() - if sc.closed || !sc.chanRequested { + if sc.closed { return } select { @@ -327,7 +326,10 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table // Parse SSE frames. scanner := bufio.NewScanner(resp.Body) - scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) // 16 MiB max, matching server ingest cap + // 16 MiB max line: generous headroom over the ~1 MiB NATS MaxPayload + // ceiling on a single event envelope (oversized records are rejected at + // ingest publish and never reach the stream). + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) var eventID, dataLine string lastID := since @@ -600,8 +602,24 @@ func toFloat64(v any) (float64, bool) { return float64(n), true case int: return float64(n), true + case int8: + return float64(n), true + case int16: + return float64(n), true + case int32: + return float64(n), true case int64: return float64(n), true + case uint: + return float64(n), true + case uint8: + return float64(n), true + case uint16: + return float64(n), true + case uint32: + return float64(n), true + case uint64: + return float64(n), true case json.Number: f, err := n.Float64() return f, err == nil diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index c201fa2f..1d9a9f69 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "encoding/json" "errors" "fmt" "io" @@ -202,9 +203,7 @@ func TestStream_FilteredCloseUnderLoad(t *testing.T) { } func TestStream_HandleMalformedSSEData(t *testing.T) { - // chanRequested must be true or emitEvent skips the channel entirely and - // the no-emit assertion below would pass vacuously. - sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} + sc := &StreamController{eventCh: make(chan StreamEvent, 1)} var gotErr error sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} sc.handleSSEData("not json") // must not panic or emit @@ -218,6 +217,23 @@ func TestStream_HandleMalformedSSEData(t *testing.T) { } } +// TestStream_EventsBufferBeforeFirstEventsCall pins TS parity: the channel +// buffers from construction, so events emitted before the first Events() call +// are still delivered once the consumer starts reading. +func TestStream_EventsBufferBeforeFirstEventsCall(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 256)} + sc.emitEvent(StreamEvent{Table: "clicks", Data: map[string]any{"page": "/"}}) + + select { + case e := <-sc.Events(): + if e.Table != "clicks" { + t.Fatalf("want event for clicks, got %+v", e) + } + default: + t.Fatal("event emitted before Events() was not buffered") + } +} + // --------------------------------------------------------------------------- // Client-side filter engine // --------------------------------------------------------------------------- @@ -240,6 +256,8 @@ func TestEvaluateFilter(t *testing.T) { {"Lt", float64(9), "lt", 10, true}, {"LteString", "a", "lte", "b", true}, {"GtIncomparable", "a", "gt", 10, false}, + // Narrow/unsigned codegen-struct fields must compare, not silently drop. + {"GtUnsignedOperand", float64(10), "gt", uint32(5), true}, {"InAnySlice", "b", "in", []any{"a", "b"}, true}, {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, {"InMiss", "c", "in", []any{"a", "b"}, false}, @@ -293,7 +311,12 @@ func TestCompareOrdered(t *testing.T) { } func TestToFloat64(t *testing.T) { - for _, v := range []any{float64(1), float32(1), int(1), int64(1)} { + for _, v := range []any{ + float64(1), float32(1), + int(1), int8(1), int16(1), int32(1), int64(1), + uint(1), uint8(1), uint16(1), uint32(1), uint64(1), + json.Number("1"), + } { if f, ok := toFloat64(v); !ok || f != 1 { t.Fatalf("toFloat64(%T) = (%v, %v)", v, f, ok) } diff --git a/clients/go/types.go b/clients/go/types.go index 0d17699f..1257e446 100644 --- a/clients/go/types.go +++ b/clients/go/types.go @@ -175,7 +175,7 @@ type RolePermissions struct { DeniedAggregations []string `json:"denied_aggregations,omitempty"` MaxRows *int `json:"max_rows,omitempty"` MaxExecutionTime any `json:"max_execution_time,omitempty"` - MaxRowsToRead *int `json:"max_rows_to_read,omitempty"` + MaxRowsToRead *int64 `json:"max_rows_to_read,omitempty"` MaxMemoryUsage any `json:"max_memory_usage,omitempty"` } diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 30c46a3c..70f44a37 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -152,11 +152,13 @@ stream closes, including on a terminal 401/403/404. Pair `Events()` with a *why* a stream ended. ::: -:::note[`Events()` starts feeding on first call] -The channel only receives events emitted **after** the first `Events()` -call — a stream you set up but don't consume yet buffers nothing for the -channel. Call `Events()` immediately after `.Stream()` (or use -`.Subscribe`) if you can't start ranging right away. +:::note[The channel buffers from stream construction] +Events buffer into the channel (up to 256) from the moment `.Stream()` +constructs the stream, matching the TypeScript SDK — events arriving before +your first `Events()` call are **not** lost, so you don't have to call +`Events()` immediately. A consumer that never drains the channel still +drops everything past the 256th buffered event (with the one-time log line +described above). ::: ### Transport Behavior From c88d1b1072599720f2a8227ca317da05fa02ebc4 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 21:12:24 -0400 Subject: [PATCH 20/59] fix(sdk): apply verified cavecrew/ponytail wave findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correctness (cavecrew, all verified): - http.go: cap backoff after jitter so the documented 30s max holds (was cap-then-jitter, allowing 36s). - stream.go: filtered wrapper unsubscribes from the inner controller in the inner-closed branch too, not just on wrapper Close. - codegen: a flag missing its value now errors (exit 2) instead of silently using the default; flagValue helper replaces four copies. - conformance: ingest/ingest_batch hard-fail on fixture cases without an insert operation (both runners); toStringMap fails on non-object rows. - wire_cases.json: all 23 query cases now assert expected_path and expected_method (previously body-only). Complexity (ponytail, safe cuts only): - stream.go: snapshotSubs helper replaces two copy-pasted subscriber snapshots (setStatus keeps its inline copy — its snapshot must share the critical section with the status write); evaluateIn drops the redundant []any special case (reflect covers it); toFloat64 collapses ten integer cases to reflect CanInt/CanUint. - wavehouse.go: inline single-caller trimTrailingSlashes. - codegen: drop dead IsNullable field; avoid rune-slice alloc in pascalCase digit check. - tests: nsClient (byte-identical duplicate of queryTestCtx) deleted, table_test boilerplate collapsed onto queryTestCtx (9 + 8 sites). - Makefile: drop test-go-sdk from ci-parallel (already reached via test). Rejected after verification: 1000<= len(os.Args) { + fmt.Fprintf(os.Stderr, "Error: missing value for %s (use --help)\n", flag) + os.Exit(2) + } + return os.Args[*i] +} + func parseArgs() cliArgs { args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} for i := 1; i < len(os.Args); i++ { switch os.Args[i] { case "--url", "-u": - i++ - if i < len(os.Args) { - args.url = os.Args[i] - } + args.url = flagValue(&i) case "--out", "-o": - i++ - if i < len(os.Args) { - args.out = os.Args[i] - } + args.out = flagValue(&i) case "--auth", "-a": - i++ - if i < len(os.Args) { - args.auth = os.Args[i] - } + args.auth = flagValue(&i) case "--package", "-p": - i++ - if i < len(os.Args) { - args.pkg = os.Args[i] - } + args.pkg = flagValue(&i) case "--help", "-h": fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema @@ -76,7 +76,6 @@ Options: type column struct { Name string `json:"name"` Type string `json:"type"` - IsNullable bool `json:"is_nullable"` HasDefault bool `json:"has_default"` } @@ -276,7 +275,7 @@ func pascalCase(s string) string { // Go identifiers can't start with a digit (e.g. a table named // "2fa_events" would otherwise produce the invalid identifier // "2faEvents"). Prefix with "X" to keep it a valid, exported name. - if unicode.IsDigit([]rune(result)[0]) { + if unicode.IsDigit(rune(result[0])) { // digits are ASCII; no rune-slice needed result = "X" + result } return result diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 26fbac07..e90f8412 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -125,31 +125,33 @@ func TestConformance_WireFormat(t *testing.T) { logCallErr(t, err) case "ingest": - if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - data := tc.Operations[0].Args[0] - _, err := c.From(tc.Table).Insert(ctx, data) - logCallErr(t, err) + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest case %q has no insert operation", tc.Name) } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + data := tc.Operations[0].Args[0] + _, err := c.From(tc.Table).Insert(ctx, data) + logCallErr(t, err) case "ingest_batch": - if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - rawArr, ok := tc.Operations[0].Args[0].([]any) - if !ok { - t.Fatalf("batch insert args[0] is not an array") - } - rows := make([]map[string]any, len(rawArr)) - for i, r := range rawArr { - rows[i] = toStringMap(r) - } - _, err := c.From(tc.Table).Insert(ctx, rows) - logCallErr(t, err) + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest_batch case %q has no insert operation", tc.Name) + } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + rawArr, ok := tc.Operations[0].Args[0].([]any) + if !ok { + t.Fatalf("batch insert args[0] is not an array") } + rows := make([]map[string]any, len(rawArr)) + for i, r := range rawArr { + rows[i] = toStringMap(t, r) + } + _, batchErr := c.From(tc.Table).Insert(ctx, rows) + logCallErr(t, batchErr) case "pipe": p := c.Pipe(tc.PipeName, tc.PipeParams) @@ -377,12 +379,13 @@ func toStringSlice(args []any) []string { return out } -func toStringMap(v any) map[string]any { +func toStringMap(t *testing.T, v any) map[string]any { + t.Helper() m, ok := v.(map[string]any) - if ok { - return m + if !ok { + t.Fatalf("fixture row is not an object: %T", v) } - return nil + return m } // deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal diff --git a/clients/go/http.go b/clients/go/http.go index 74dcbdad..d8a8b37b 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -185,12 +185,10 @@ func retryAfterDelay(ra string, attempt int) time.Duration { func backoff(attempt int) time.Duration { ms := 1000 * math.Pow(2, float64(attempt)) - if ms > 30000 { - ms = 30000 - } - // ±20% jitter so clients failing at the same moment don't retry in lockstep. + // ±20% jitter so clients failing at the same moment don't retry in + // lockstep; capped after jitter so the documented 30s max holds. ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic - return time.Duration(ms) * time.Millisecond + return time.Duration(min(ms, 30000)) * time.Millisecond } func sleepWithContext(ctx context.Context, d time.Duration) error { diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 76acc176..e7340b6f 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -4,24 +4,12 @@ import ( "context" "encoding/json" "net/http" - "net/http/httptest" "sync" "testing" ) -func nsClient(t *testing.T, handler http.Handler) *Client { - t.Helper() - srv := httptest.NewServer(handler) - t.Cleanup(srv.Close) - return NewClient(Config{ - BaseURL: srv.URL, - HTTPClient: srv.Client(), - Options: &ClientOptions{MaxRetries: 0}, - }) -} - func TestSysNamespace_Health(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/health" { t.Errorf("want /v1/health, got %s", r.URL.Path) } @@ -34,7 +22,7 @@ func TestSysNamespace_Health(t *testing.T) { } func TestSchemaNamespace_List(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } @@ -54,7 +42,7 @@ func TestSchemaNamespace_List(t *testing.T) { func TestSchemaNamespace_Refresh(t *testing.T) { var mu sync.Mutex var gotMethod string - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotMethod = r.Method mu.Unlock() @@ -72,7 +60,7 @@ func TestSchemaNamespace_Refresh(t *testing.T) { } func TestPolicyNamespace_GetSetValidate(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) @@ -107,7 +95,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace(t *testing.T) { t.Run("List", func(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) @@ -122,7 +110,7 @@ func TestDLQNamespace(t *testing.T) { t.Run("Table", func(t *testing.T) { var mu sync.Mutex var gotParam string - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotParam = r.URL.Query().Get("table") mu.Unlock() @@ -141,7 +129,7 @@ func TestDLQNamespace(t *testing.T) { } func TestPipesNamespace_CRUD(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { @@ -187,7 +175,7 @@ func TestPipeRef_Fetch(t *testing.T) { var mu sync.Mutex var gotPath, gotMethod string var gotBody map[string]any - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotPath = r.URL.Path gotMethod = r.Method diff --git a/clients/go/stream.go b/clients/go/stream.go index b6402489..8307e2fb 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -160,12 +160,17 @@ func (sc *StreamController) setStatus(s StreamStatus) { } } -func (sc *StreamController) emitEvent(event StreamEvent) { +// snapshotSubs copies the subscriber list under mu so callbacks run unlocked. +// setStatus keeps its own inline copy: there the snapshot must share the +// critical section with the status write to keep callback order consistent. +func (sc *StreamController) snapshotSubs() []*StreamSubscriber { sc.mu.Lock() - subs := append([]*StreamSubscriber(nil), sc.subscribers...) - sc.mu.Unlock() + defer sc.mu.Unlock() + return append([]*StreamSubscriber(nil), sc.subscribers...) +} - for _, sub := range subs { +func (sc *StreamController) emitEvent(event StreamEvent) { + for _, sub := range sc.snapshotSubs() { if sub.Next != nil { sub.Next(event) } @@ -190,11 +195,7 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } func (sc *StreamController) emitError(err error) { - sc.mu.Lock() - subs := append([]*StreamSubscriber(nil), sc.subscribers...) - sc.mu.Unlock() - - for _, sub := range subs { + for _, sub := range sc.snapshotSubs() { if sub.Error != nil { sub.Error(err) } @@ -445,6 +446,9 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, unsub() inner.Close() case <-inner.done: + // Inner closed on its own — still unsubscribe so the closed + // controller doesn't retain a reference to this wrapper. + unsub() } }() @@ -542,23 +546,15 @@ func equalValues(a, b any) bool { } // evaluateIn checks whether actual is contained in the expected slice. -// Handles both []any and typed slices (e.g., []string, []int). +// Reflection handles []any and typed slices (e.g., []string, []int) alike. func evaluateIn(actual, expected any) bool { - if arr, ok := expected.([]any); ok { - for _, v := range arr { - if equalValues(actual, v) { - return true - } - } + rv := reflect.ValueOf(expected) + if rv.Kind() != reflect.Slice { return false } - // Handle typed slices via reflection. - rv := reflect.ValueOf(expected) - if rv.Kind() == reflect.Slice { - for i := range rv.Len() { - if equalValues(actual, rv.Index(i).Interface()) { - return true - } + for i := range rv.Len() { + if equalValues(actual, rv.Index(i).Interface()) { + return true } } return false @@ -600,32 +596,19 @@ func toFloat64(v any) (float64, bool) { return n, true case float32: return float64(n), true - case int: - return float64(n), true - case int8: - return float64(n), true - case int16: - return float64(n), true - case int32: - return float64(n), true - case int64: - return float64(n), true - case uint: - return float64(n), true - case uint8: - return float64(n), true - case uint16: - return float64(n), true - case uint32: - return float64(n), true - case uint64: - return float64(n), true case json.Number: f, err := n.Float64() return f, err == nil - default: - return 0, false } + // All int/uint widths in two cases (codegen structs use the narrow ones). + rv := reflect.ValueOf(v) + switch { + case rv.CanInt(): + return float64(rv.Int()), true + case rv.CanUint(): + return float64(rv.Uint()), true + } + return 0, false } func projectColumns(row map[string]any, columns []string) map[string]any { diff --git a/clients/go/table_test.go b/clients/go/table_test.go index e935657f..2dfd0cff 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -5,7 +5,6 @@ import ( "encoding/json" "io" "net/http" - "net/http/httptest" "sync" "testing" ) @@ -16,15 +15,13 @@ func TestTableRef_InsertSingle(t *testing.T) { var mu sync.Mutex var gotBody map[string]any var gotPath string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) if err != nil { t.Fatal(err) @@ -46,7 +43,7 @@ func TestTableRef_InsertBatch(t *testing.T) { var mu sync.Mutex var gotCT string var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotCT = r.Header.Get("Content-Type") @@ -56,8 +53,6 @@ func TestTableRef_InsertBatch(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ {"page": "/a"}, {"page": "/b"}, @@ -91,7 +86,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { var mu sync.Mutex var gotCT string var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotCT = r.Header.Get("Content-Type") @@ -101,8 +96,6 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ {Page: "/a"}, {Page: "/b"}, @@ -134,14 +127,12 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var mu sync.Mutex var gotPath string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) if err != nil { t.Fatal(err) @@ -157,11 +148,9 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { } func TestTableRef_InsertEmptyBatch(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("should not make a request for empty batch") })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) if err != nil { t.Fatal(err) @@ -177,7 +166,7 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { func TestTableRef_InsertNDJSON(t *testing.T) { var mu sync.Mutex var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() raw, _ := io.ReadAll(r.Body) @@ -186,8 +175,6 @@ func TestTableRef_InsertNDJSON(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) if err != nil { @@ -204,7 +191,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { } func TestTableRef_Schema(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Query().Get("table") != "clicks" { t.Errorf("want table=clicks") } @@ -215,8 +202,6 @@ func TestTableRef_Schema(t *testing.T) { }, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) schema, err := c.From("clicks").Schema(context.Background()) if err != nil { t.Fatal(err) @@ -230,11 +215,9 @@ func TestTableRef_Schema(t *testing.T) { } func TestTableRef_InsertDuplicate(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) if err != nil { t.Fatal(err) diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index 2c27b46a..e721afb6 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -47,6 +47,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "eq", "value": "/home" }], @@ -61,6 +63,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "!=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "neq", "value": "/home" }], @@ -75,6 +79,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">", 10] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], @@ -89,6 +95,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">=", 10] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gte", "value": 10 }], @@ -103,6 +111,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<", 5] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lt", "value": 5 }], @@ -117,6 +127,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<=", 5] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lte", "value": 5 }], @@ -131,6 +143,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "in", ["/home", "/about"]] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], @@ -145,6 +159,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "like", "/home%"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "like", "value": "/home%" }], @@ -158,6 +174,8 @@ "operations": [ { "method": "count", "args": ["*", "total"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], "limit": 1000 @@ -170,6 +188,8 @@ "operations": [ { "method": "sum", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], "limit": 1000 @@ -182,6 +202,8 @@ "operations": [ { "method": "avg", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], "limit": 1000 @@ -194,6 +216,8 @@ "operations": [ { "method": "min", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], "limit": 1000 @@ -206,6 +230,8 @@ "operations": [ { "method": "max", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], "limit": 1000 @@ -218,6 +244,8 @@ "operations": [ { "method": "countDistinct", "args": ["page", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], "limit": 1000 @@ -230,6 +258,8 @@ "operations": [ { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], "limit": 1000 @@ -243,6 +273,8 @@ { "method": "select", "args": ["page"] }, { "method": "groupBy", "args": ["page"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "group_by": ["page"], @@ -257,6 +289,8 @@ { "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["page", "asc"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "order_by": [{ "column": "page", "dir": "asc" }], @@ -271,6 +305,8 @@ { "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["score", "desc"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "order_by": [{ "column": "score", "dir": "desc" }], @@ -285,6 +321,8 @@ { "method": "select", "args": ["page"] }, { "method": "limit", "args": [50] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "limit": 50 @@ -298,6 +336,8 @@ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "time_range": { "column": "received_timestamp", "since": "1h" }, @@ -312,6 +352,8 @@ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, @@ -327,6 +369,8 @@ { "method": "where", "args": ["score", ">", 10] }, { "method": "where", "args": ["page", "=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [ @@ -349,6 +393,8 @@ { "method": "limit", "args": [50] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index a4d338e8..5acfc46b 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -81,7 +81,7 @@ func NewClient(cfg Config) *Client { c := &Client{ ctx: httpContext{ - baseURL: trimTrailingSlashes(cfg.BaseURL), + baseURL: strings.TrimRight(cfg.BaseURL, "/"), auth: cfg.Auth, maxRetries: maxRetries, httpClient: hc, @@ -137,7 +137,3 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { return newStreamController(c.ctx, table, opts) } - -func trimTrailingSlashes(s string) string { - return strings.TrimRight(s, "/") -} diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 4996b594..0a0f3ef8 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -181,14 +181,12 @@ for (const tc of cases) { break; } case "ingest": - if (tc.operations?.[0]?.method === "insert") { - await wh.from(tc.table).insert(tc.operations[0].args[0]); - } - break; case "ingest_batch": - if (tc.operations?.[0]?.method === "insert") { - await wh.from(tc.table).insert(tc.operations[0].args[0]); + if (tc.operations?.[0]?.method !== "insert") { + // Hard failure, matching the Go harness. + throw new Error(`${tc.name}: ingest case has no insert operation`); } + await wh.from(tc.table).insert(tc.operations[0].args[0]); break; case "pipe": await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); From c2c1021d3d5551564a6da7e3f3e4ac462f3e84ab Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 12:47:09 -0400 Subject: [PATCH 21/59] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2011=20=E2=80=94=20stale=20changelog=20entry,=20ARGS/V=20acc?= =?UTF-8?q?uracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops the CHANGELOG dependency-bump entry: after merging origin/main, go.mod and go.sum are byte-identical to main and no longer appear in this PR's delta, and the bumps it claimed credit for landed on main via Dependabot (#438). The matching bullet in the PR description was removed too. Corrects the test-target flag documentation in development.md, which claimed all test targets accept ARGS. Per the Makefile: gotestsum drives test-unit and test-integration only; ARGS reaches test-unit, test-integration, and test-ts; V=1 reaches test-unit, test-integration, and test-e2e (the orchestrator reads it directly). Adds the Go SDK unit test and wire-conformance case entries to "Adding New Tests", which the SDK sync rule requires for every new endpoint. Narrows the "every operation returns (T, error)" claim in the Go SDK index, reference, and README: the void operations (Pipes.Set/Delete, Policy.Set, Schema.Refresh, Sys.Health) return a bare error, as reference.md's own API tree already showed. Lists the FilterOp constants rather than raw symbols in the streaming filter docs, matching how every example calls .Where(), and fixes "the SDK readme" to "readmes" now that docs-prose.sh resolves both. --- CHANGELOG.md | 2 -- clients/go/README.md | 2 +- docs/src/content/docs/claude-code.md | 2 +- docs/src/content/docs/development.md | 10 ++++++---- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/reference.md | 4 +++- docs/src/content/docs/sdk/go/streaming.md | 6 ++++-- 7 files changed, 16 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ade0f381..5915fe88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,8 +34,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Go toolchain requirement bumped to 1.26.5** (`go.mod`): the `go` directive moves from `1.26.4` to `1.26.5` so local builds (via `GOTOOLCHAIN=auto`) and CI's `setup-go` (which reads `go.mod` via `go-version-file`) install a Go whose standard library clears the `govulncheck` findings GO-2026-5856 (`crypto/tls`) and GO-2026-4970 (`os`), both fixed in 1.26.5 — those findings were failing `make verify`'s `vulncheck` leaf (and with it the pre-commit hook) on every tree. Patch-level toolchain bump only — no source changes — and the released binaries pick up the patched stdlib too. -- **Dependency bumps clearing new govulncheck findings** (`go.mod`, `go.sum`): `google.golang.org/grpc` 1.81.1 → 1.82.1 (GO-2026-6061), `golang.org/x/text` 0.37.0 → 0.39.0 (GO-2026-5970), `github.com/klauspost/compress` 1.18.6 → 1.18.7 (GO-2026-5841) — all three were failing `make verify`'s `vulncheck` leaf after the advisories published. No API changes. - ### Security - **Policy `_in` is now enforced on both the row-`filter` and insert-`check` paths, closing a fail-open row-security gap** (`internal/policy/policy.go`, `internal/api/ingest.go`, `docs/src/content/docs/access-control.mdx`, plus tests in `internal/policy/policy_test.go`, `internal/api/ingest_test.go`): closes #224. The `Filter` schema accepted `_in` but the engine never read it: on the row-`filter`/SELECT path `resolveFilters` had no `_in` branch, so a row-security filter like `tenant_id: { _in: … }` produced **no `WHERE` predicate** and the role saw every row instead of its tenant subset (a fail-open, same family as #223); on the `check`/INSERT path only `_eq` was honored, silently dropping any other operator. `_in` now takes a single claim that resolves to a JSON **array** (the multi-tenant case — a token's `tenant_ids` list) and emits `col IN (?, …)` with one bound param per element; a scalar claim is a one-element set, and an empty/absent claim matches **no rows** (fail-closed) rather than widening to all of them. On the insert path an `_in` check requires the column be present and one of the set — there is no single value to auto-inject as `_eq` does, so an omitted column is rejected (`403 check failed`). The comparison operators are enforced on `filter` (`_eq`/`_neq`/`_gt`/`_lt`/`_in` all produce predicates now, so nothing is rejected there) and, on `check`, `_neq`/`_gt`/`_lt` become a loud config-load rejection (no insert-time semantics; `check` honors `_eq` + `_in`). The `_in` value stays a single templated string in the wire schema (Go `Filter.In`, SDK `PolicyFilter._in`), matching the established "set = array" shape of the caller-query `in` operator. diff --git a/clients/go/README.md b/clients/go/README.md index bd6109d3..5db4eaac 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#c ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/claude-code.md b/docs/src/content/docs/claude-code.md index c418517f..c57abc4c 100644 --- a/docs/src/content/docs/claude-code.md +++ b/docs/src/content/docs/claude-code.md @@ -80,7 +80,7 @@ To add a command: drop a `.md` file in `.claude/commands/`. Filename becomes the | Subagent | When to use | | -------- | ----------- | | `pre-push-reviewer` | **Mandatory before pushing to a PR branch** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel with the other reviewers in `scripts/pre-push-reviewers.sh` — all must reach `ship_it`. Also used for auditing someone else's PR after `wt switch pr:`. Runs the canonical `.github/prompts/pr-review.md` workflow against the local branch in fresh context. Fetches PR comments + CI status + linked-issue acceptance criteria when on a PR branch. Returns `[MUST]`/`[SHOULD]`/`[MAY]` findings + a parseable `VERDICT: ship_it\|iterate\|block` line that drives the `tmp/pre-push-reviewer-passed-` marker. | -| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readme). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | +| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readmes). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | Invoke via the `Agent` tool with `subagent_type: pre-push-reviewer`, or via `/agents`. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index cd9c6d46..d4718565 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,7 +288,7 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -The coverage-instrumented suite targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary, and accept `ARGS`/`V=1`. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. +The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. @@ -327,9 +327,9 @@ make cov Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). +**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. -**Extra flags**: All test targets accept `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`, `-timeout`). +**Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). @@ -355,6 +355,8 @@ Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). - **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. +- **Go SDK unit test** → add to `clients/go/*_test.go` (nested module — outside `test-unit`'s scope). Run with `make test-go-sdk`. +- **Wire-format parity case** → when you add or change an endpoint, add an entry to `clients/go/testdata/wire_cases.json` plus its dispatch in both runners (`clients/go/conformance_test.go` and `tests/conformance/conformance_ts.mjs`). Required by the SDK sync rule in `AGENTS.md` / `CONTRIBUTING.md`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). ### E2E Tests via SDK @@ -519,7 +521,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -The gotestsum-driven targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) accept `ARGS="..."` for pass-through `go test` flags and `V=1` for verbose output; `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. +`test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags; `test-unit`, `test-integration`, and `test-e2e` accept `V=1` for verbose output. `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 8e0b2329..4ece842f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -174,7 +174,7 @@ parameter. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 0074b7f0..8eb16881 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -39,7 +39,9 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors +operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare +`error` for operations with no result body (`Pipes.Set`/`Delete`, +`Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` (unwrap with `errors.As`); client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 70f44a37..01484290 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -197,8 +197,10 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, -`not_like` — the same `FilterOp` set `.Where()` takes everywhere. `like` / +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, +`OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes +everywhere (the SDK maps them to wire tokens such as `eq`/`neq` +internally). `like` / `not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any single character), case-insensitively. `in` accepts any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. From 0ee17344e9d0f2ade266a34094214481e37ceb1a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 13:01:33 -0400 Subject: [PATCH 22/59] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2012=20=E2=80=94=20E2E=20lifecycle,=20lint=20install,=20429?= =?UTF-8?q?=20sync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The E2E section of development.md still described the pre-orchestrator flow. tests/e2e/sdk/setup.ts starts nothing: it probes the orchestrator-supplied CLICKHOUSE_URL/WAVEHOUSE_URL and throws if either is unreachable, and the orchestrator has no :8080 handling at all — it launches wavehouse-cov on a random free port, so a running `make dev` is neither needed nor reused. golangci-lint is auto-installed pinned to v2.11.4 in .bin/_/ by the Makefile, not looked up on PATH. The manual install list steered contributors toward an unpinned version the build never uses, whose findings diverge from CI's. Also narrows the V=1 comment, which claimed every test target honors it. Documents 429 as retryable in the Go SDK reference and streaming pages, which clients/go/errors.go has classified as retryable since the review rounds, and notes the 30s Retry-After clamp on both retryable rows. Finishes the round-11 FilterOp constant switch, whose follow-on sentences still used wire tokens. The TS live query derives its dedup boundary from the last fetched row rather than the newest, so a desc-ordered fetch re-delivers overlap-window events; the Go client takes the max instead. Prose now describes actual behavior and links #449. TS not retrying 429 while Go does is tracked in #450. Extends the Go SDK changelog entry's file list to all 33 touched files and records the TS SDK doc corrections under Fixed, per the exhaustive-list convention the surrounding entries follow. --- CHANGELOG.md | 4 +++- docs/src/content/docs/development.md | 15 +++++---------- docs/src/content/docs/sdk/go/reference.md | 3 ++- docs/src/content/docs/sdk/go/streaming.md | 12 ++++++------ docs/src/content/docs/sdk/streaming.md | 2 +- 5 files changed, 17 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5915fe88..6eb553f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. @@ -51,6 +51,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table was missing `SSE_ERROR` and `SSE_CONNECT_ERROR` entirely, and described `401` as "missing or invalid JWT" when a *missing* token actually resolves to `default_role` and is denied with `403` (`internal/auth/auth.go`) — only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. + - **Go module cache stored once instead of once per compile flavor** (`.github/actions/setup-env/action.yml`, `.github/workflows/README.md`, `.github/workflows/publish-dev.yml`, `.github/workflows/release.yml`, `Makefile`): closes [#443](https://github.com/Wave-RF/WaveHouse/issues/443). `setup-env` cached `~/go/pkg/mod` together with `~/.cache/go-build` under a key partitioned by `go-cache-suffix`, but the module cache is a pure function of `go.mod` + `go.sum` and byte-identical for every flavor — so that tree was stored five times over (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`) — five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation (the tree is ~1.6 GB on disk; 0.48 GB as a stored archive on a cold save, drifting up as superseded versions accumulate). Two live generations is the steady state (a bump mints a new set while the previous is still warm), so the repo sat near GitHub's hard 10 GB cache cap; the 24-module go-deps bump ([#438](https://github.com/Wave-RF/WaveHouse/pull/438)) tipped it to 10.53 GB and GitHub began LRU-evicting warm entries mid-run. The one cache is now two: `gomod-v1--` on `~/go/pkg/mod`, **unsuffixed** and shared by every `ci.yml` Go job that goes through `setup-env`, and `gobuild-v3--go-` on `~/.cache/go-build` only, still per flavor. Measured after the split: **1.05 GB per generation** (0.48 GB module + 0.57 GB across the five build entries), down from 5.18 GB — a 4.9x reduction. All sizes are stored-archive bytes / 2^30, the unit the README's usage check prints. The `v3` bump is load-bearing — saves fire only on an exact-key miss, so without it the old `v2` entry (still carrying the module cache) would exact-hit forever and the smaller content would never be saved — and `gobuild-v3` drops the bare-prefix restore-key, which existed solely to borrow another flavor's copy of the module cache. Separately, `publish-dev.yml` and `release.yml` now pass `cache: false` to `actions/setup-go` (matching `goreleaser-validate.yml`), which was holding a sixth ~1 GB entry — the module tree `gomod-v1` already keeps once, plus that job's own 8-target cross-compile objects — re-saved on every cache miss. (That entry is keyed on the root `go.mod`: setup-go hashed `go.sum` through v6.2.0 and `go.mod` from v6.3.0, [actions/setup-go#705](https://github.com/actions/setup-go/pull/705).) `publish-dev.yml` re-caches only the half that pays for itself, under `gobuild-v3--go-release-` (~0.5 GB — the bundled entry minus `gomod-v1`'s share): across its last 20 runs GoReleaser takes 36–246 s with the cross-compile objects warm and 401–446 s cold (measured on `setup-go`'s bundled cache, which carried the same `~/.cache/go-build` tree), so dropping the cross-compile objects outright would have cost roughly 2.5–7 minutes on every push to main (mean delta ≈4.8 min). The `-release` suffix keeps those 8-target objects from being restored by CI's native-only flavors and vice versa. Because those timings were taken with setup-go's bundled entry (which also held `~/go/pkg/mod`), `publish-dev` additionally *restores* `gomod-v1` from `main`'s scope via `actions/cache/restore` — read-only, so it costs no budget and cannot write a partial tree to the key every `ci.yml` Go job shares. Without that restore the job would re-download ~112 MB of modules per push and land above the warm range quoted above. Both Go keys now hash `go.mod` alongside `go.sum`, for a different reason each: the GOTOOLCHAIN=auto toolchain lives in `~/go/pkg/mod` and `go.sum` records no entry for it, so a `go`-directive bump would otherwise exact-hit a toolchain-less archive and — saves firing only on an exact-key miss — re-download it every run; and the compiler's build ID keys every build object, so the same bump invalidates `~/.cache/go-build` too, where the failure mode is a permanent cold recompile rather than a re-download. Two guards come with the shared entry: `setup-env` now fails a `go: true` job that passes no `go-cache-suffix` (an empty one yields a restore-key prefix-matching every other flavor), and `make cov` gains the `go-mod-download` prerequisite its siblings already had — CI's coverage job shares the unsuffixed `gomod-v1` and races to save it, but ran only `go run ./scripts/cov report`, so winning that race would have stored a partial `~/go/pkg/mod` that then exact-hit for every other job until the next rotation. The workflows README gains a sizing policy — the 10 GB cap, the two-generations rule, how to check the current footprint, and the rule that lockfile-derived content is keyed once and shared — plus the narrowing-rotation exception to the key-versioning policy. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index d4718565..962d6ae5 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -295,7 +295,8 @@ Go tests run with the **race detector** (`-race`) enabled by default (including ### Quick Reference ```bash -# Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` +# V=1 gives verbose output on test-unit / test-integration / test-e2e, +# e.g. `V=1 make test-unit` # Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test @@ -366,7 +367,7 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty **Architecture**: - `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. -- `tests/e2e/sdk/setup.ts` — Smart `globalSetup` that probes ports before starting Docker services, so tests work seamlessly whether you started services manually or let the setup do it. +- `tests/e2e/sdk/setup.ts` — `globalSetup` that probes the orchestrator-supplied `CLICKHOUSE_URL`/`WAVEHOUSE_URL`, creates the per-suite tables, refreshes the schema, and bootstraps a baseline policy. Lifecycle is owned by the orchestrator: if either URL isn't reachable it fails fast rather than starting anything itself. - `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. **Running E2E tests**: @@ -378,7 +379,7 @@ make test-e2e `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. -**If you already have `make dev` running**, the setup detects the healthy WaveHouse on `:8080` and skips starting it via Docker — only ClickHouse is started if needed. +`make test-e2e` is self-contained — it always starts its own ClickHouse testcontainer and `wavehouse-cov` on a random free port, so it neither needs nor reuses a running `make dev`. **Test files** (`tests/e2e/sdk/*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`. @@ -388,13 +389,7 @@ make test-e2e make lint ``` -`golangci-lint` is installed separately (not in `go.mod` — its massive dependency tree causes conflicts). If not found, `make lint` prints install instructions. - -Install options: - -- **macOS**: `brew install golangci-lint` -- **Binary**: See [golangci-lint.run/welcome/install/](https://golangci-lint.run/welcome/install/) -- **Go install**: `go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest` +`golangci-lint` is pinned in the `Makefile` (v2.11.4) and auto-installed to `.bin/_/` on first `make lint` (or `make tools`) — no manual install needed. It's kept out of `go.mod` because its dependency tree conflicts with the main module. Install it globally and you'll get an unpinned version the build never uses, with findings that diverge from CI. The configuration is in `.golangci.yml` (v2 format with `default: none` for explicit control) — that file is the authoritative list of enabled linters. Highlights: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 8eb16881..7471371d 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -57,8 +57,9 @@ SDK's "the SDK never throws" guarantee. | 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | +| 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | | 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | -| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | | 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 01484290..d64c9305 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -167,7 +167,7 @@ described above). | --------- | --------- | -------- | | SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | -Reconnect covers transport failures and retryable (5xx) responses. A +Reconnect covers transport failures and retryable (5xx/429) responses. A non-retryable response (401/403/404) is terminal: the error is delivered to the subscriber's `Error` callback, status goes to `StatusClosed`, and the stream does not reconnect — fix the cause (refresh the token, correct the @@ -199,11 +199,11 @@ stream := wh.From("clicks"). Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes -everywhere (the SDK maps them to wire tokens such as `eq`/`neq` -internally). `like` / -`not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any -single character), case-insensitively. `in` accepts any Go slice type on -the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. +everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). +`OpLike` / `OpNotLike` match SQL LIKE semantics (`%` → any run of +characters, `_` → any single character), case-insensitively. `OpIn` accepts +any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, +...), not just `[]any`. --- diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 7bc4b7fb..9b569128 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -154,7 +154,7 @@ interface StreamSubscriber { 1. Opens the stream **immediately** and buffers incoming events. 2. Runs the `.fetch()` query for historical data, calls `subscriber.initial()` with the result. -3. Deduplicates buffered events by comparing timestamps against the latest historical timestamp. +3. Deduplicates buffered events against the **last** fetched row's `received_timestamp` (a string comparison). With the `.orderBy('received_timestamp', 'desc')` used above, the last row is the *oldest* one, so the bound is looser than intended and events in the overlap window can still be re-delivered — tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). The Go SDK takes the maximum instead. 4. Flushes remaining buffered events and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. From 51d8b4514ff2405ad8f2b0bef207395058a8070d Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 13:13:22 -0400 Subject: [PATCH 23/59] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2013=20=E2=80=94=20429=20sweep,=20gotestsum=20format,=20LIKE?= =?UTF-8?q?=20case=20split?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-12 429 sweep updated the two site pages but missed the Go SDK's own README, which is the module's front page on GitHub and pkg.go.dev: it still claimed only 5xx/network are retried and only 503 honors Retry-After. clients/go/errors.go classifies 429 retryable and clients/go/http.go honors Retry-After on both, clamped to 30s. The MaxRetries row on the Go SDK index page had the same incomplete enumeration. development.md named testdox as the compact gotestsum format; the Makefile uses pkgname-and-test-fails, switching to standard-verbose under V=1 (gotestdox is only a transitive dependency). The sentence was rewritten last round to add per-target scope, so the stale format name passed through. Documents a real semantic split in live queries: client-side like/not_like compiles to a case-insensitive regex (clients/go/stream.go compileLike, and the TS query builder it deliberately matches) while the server compiles the same operator to ClickHouse LIKE, which is case-sensitive. A live query can therefore disagree with itself — the backfill excludes rows the live stream includes. Both streaming pages now carry a caution beside the existing dedup one. Pre-existing since the TS SDK; the alignment decision is tracked in #451. --- clients/go/README.md | 2 +- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/streaming.md | 9 +++++++++ docs/src/content/docs/sdk/streaming.md | 4 ++++ 5 files changed, 16 insertions(+), 3 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index 5db4eaac..a44069e0 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -216,7 +216,7 @@ if err != nil { } ``` -The HTTP layer retries 5xx and network errors with exponential backoff (default 2 retries). 503 with `Retry-After` is honored. Context cancellation returns immediately with code `ABORTED`. +The HTTP layer retries 5xx, 429, and network errors with exponential backoff (default 2 retries). `Retry-After` on a 503 or 429 is honored, capped at 30s. Context cancellation returns immediately with code `ABORTED`. ## License diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 962d6ae5..bee9476d 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -328,7 +328,7 @@ make cov Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. +**Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. **Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 4ece842f..a41000a7 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -103,7 +103,7 @@ TLS / response-header timeouts instead. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures) | A `*Client` is safe for concurrent use by multiple goroutines — client state is immutable after `NewClient`, and every builder chain copies. Supply a diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index d64c9305..e5e33de8 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -282,6 +282,15 @@ that omits it disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `Initial`, again via `Next`. ::: +:::caution[`OpLike` matching differs between backfill and live] +Client-side `OpLike` / `OpNotLike` matching is case-**insensitive**, but the +backfill runs server-side where the same operator compiles to ClickHouse +`LIKE`, which is case-**sensitive**. A live query filtering on `OpLike` can +therefore disagree with itself: the backfill excludes rows the live stream +includes. Tracked in +[#451](https://github.com/Wave-RF/WaveHouse/issues/451). +::: + ### `.Close()` Shuts down the live query and its underlying stream. Safe to call more than diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 9b569128..5aa5b73d 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -162,3 +162,7 @@ This "stream-first" approach ensures no events are lost between the fetch and st :::caution[Dedup needs `received_timestamp` in the projection] The dedup boundary comes from the fetched rows' `received_timestamp` values. A `.select(...)` projection that omits that column disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `initial()`, again via `next()`. ::: + +:::caution[`like` matching differs between backfill and live] +Client-side `like` / `not_like` matching is case-**insensitive**, but the backfill runs server-side where the same operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). +::: From 4ee4f1f4c9682abdddf09e46c7bd5a0cb8c20d21 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 16:56:30 -0400 Subject: [PATCH 24/59] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2014=20=E2=80=94=20not=5Flike=20is=20rejected,=20not=20case-?= =?UTF-8?q?split?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LIKE caution added last round was right about like and wrong about not_like. internal/query/builder.go's filterToSQL has cases for eq/neq/gt/gte/lt/lte/like/in only, so not_like falls through to default and /v1/query rejects it with 400 — a live query filtering on it fails its Initial callback rather than quietly disagreeing with itself. Both streaming pages now scope the case-sensitivity claim to like and describe not_like separately, matching what the two queries pages and api.md already said. Also corrects "the Makefile uses go run" to go tool (Makefile:144-150 defines every pinned tool as `go tool `; the same page already said so thirty lines earlier), and gives the TS selectAll() description the same restricted-vs-unrestricted split this branch added everywhere else — an unrestricted role gets a bare SELECT *, not an expansion. --- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/streaming.md | 16 ++++++++++------ docs/src/content/docs/sdk/queries.md | 2 +- docs/src/content/docs/sdk/streaming.md | 4 +++- 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index bee9476d..b62d7635 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,7 +288,7 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. +The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go tool` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index e5e33de8..c105f28c 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -283,12 +283,16 @@ are delivered twice — once in `Initial`, again via `Next`. ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` / `OpNotLike` matching is case-**insensitive**, but the -backfill runs server-side where the same operator compiles to ClickHouse -`LIKE`, which is case-**sensitive**. A live query filtering on `OpLike` can -therefore disagree with itself: the backfill excludes rows the live stream -includes. Tracked in -[#451](https://github.com/Wave-RF/WaveHouse/issues/451). +Client-side `OpLike` matching is case-**insensitive**, but the backfill runs +server-side where the operator compiles to ClickHouse `LIKE`, which is +case-**sensitive**. A live query filtering on `OpLike` can therefore +disagree with itself: the backfill excludes rows the live stream includes. +Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). + +`OpNotLike` never reaches the backfill at all — `/v1/query` rejects the +operator with a `400`, so a live query filtering on it fails its `Initial` +callback. See the operator table in +[Queries](/sdk/go/queries#wherecolumn-op-value). ::: ### `.Close()` diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index a75ee3e5..92c98062 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -126,7 +126,7 @@ const q = clicks.select('page').select('button'); // SELECT page, button #### `.selectAll()` -Select every column your role may read (the all-columns wildcard, expanded server-side to your allowed columns). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). +Select every column your role may read. For a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). ```ts const q = clicks.selectAll().where('country', '=', 'US'); diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 5aa5b73d..18ba2ef0 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -164,5 +164,7 @@ The dedup boundary comes from the fetched rows' `received_timestamp` values. A ` ::: :::caution[`like` matching differs between backfill and live] -Client-side `like` / `not_like` matching is case-**insensitive**, but the backfill runs server-side where the same operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). +Client-side `like` matching is case-**insensitive**, but the backfill runs server-side where the operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). + +`not_like` never reaches the backfill at all — `/v1/query` rejects the operator with a `400`, so a live query filtering on it fails its `initial()` callback. See the operator table in [Queries](/sdk/queries#wherecolumn-op-value). ::: From 25e36cccaf73b3b6cc51363d2c6f8583a11280d5 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:35:21 -0400 Subject: [PATCH 25/59] docs: restore branch edits lost in the merge, extend path-prefix docs to Go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolving the sdk/reference.md conflict by taking main's page wholesale was too blunt. Main's SSE rows and their explanatory paragraph are correct and stay — the branch's "the stream reconnects automatically" was wrong for the TS SDK, which never re-dials a stream itself. But the same resolution also reverted two unrelated branch edits on that page: - The 401 row went back to "Missing or invalid JWT". internal/auth/auth.go returns no auth error for an empty token, so a missing token resolves to default_role and is denied 403; only a present-but-invalid or expired one yields 401. api.md and the Go reference page still said so, so the page contradicted both. - The title lost its "TypeScript SDK" prefix, leaving the only unqualified "SDK ..." title in a two-SDK tree — it reads as the shared reference in breadcrumbs and search results when it is TS-only. Main's new path-prefix guidance is TypeScript-only, which understates it now that the Go SDK is co-canonical and preserves a prefix on both transports (wavehouse.go trims the trailing slash; http.go and stream.go concatenate). reverse-proxy.mdx now shows both clients and scopes the "upgrade your SDK" caution to TypeScript, since the Go client never had #428. The Go README gains the sentence its TypeScript counterpart got in the merge. --- clients/go/README.md | 2 ++ docs/src/content/docs/reverse-proxy.mdx | 14 ++++++++++++-- docs/src/content/docs/sdk/reference.md | 4 ++-- 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index a44069e0..c1412c6d 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -80,6 +80,8 @@ client = wavehouse.NewClient(wavehouse.Config{ }) ``` +`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`) when WaveHouse is served under one. A trailing `/` is trimmed and every request path is appended to it, on both REST and SSE — see [Config](https://wavehouse.dev/sdk/go#config). + ## Typed Queries (Generics) ```go diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 672d0374..c2ce9b60 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -80,10 +80,20 @@ handle_path /api/warehouse/* { The ingress controller forwards the full path unless you ask it to rewrite. Pair a capture-group path (`/api/warehouse(/|$)(.*)` with `pathType: ImplementationSpecific`) with the `nginx.ingress.kubernetes.io/rewrite-target: /$2` annotation, or the prefix arrives at WaveHouse unstripped. ::: -Point the SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/warehouse' })` sends both REST calls and SSE streams under the prefix ([SDK → Serving under a path prefix](/sdk#serving-under-a-path-prefix)). +Point either SDK at the prefixed URL and it does the rest — both send REST calls and SSE streams under the prefix: + +```ts +// TypeScript — see /sdk#serving-under-a-path-prefix +createClient({ baseURL: 'https://app.example.com/api/warehouse' }); +``` + +```go +// Go — see /sdk/go#config +wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/warehouse"}) +``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: an SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix on every released version. ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 39b42d52..db6fc36b 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -1,5 +1,5 @@ --- -title: "SDK Reference & CLI" +title: "TypeScript SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- @@ -30,7 +30,7 @@ The SDK **never throws** for anything the server returns — all API errors come | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | From 1158ff31072f77b55c603096dd17711a20164a81 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:37:42 -0400 Subject: [PATCH 26/59] test(sdk): pin BaseURL path-prefix support; qualify the 401 rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge documented path-prefix support as a promise on the Go SDK's Config row and README, but nothing tested it — every test server in clients/go was root-hosted. The TS side shipped url.test.ts and friends pinning exactly this after #428. Adds a guard per transport, since the SSE URL is built in stream.go independently of buildURL: a mux serving only the prefixed path, so a dropped prefix 404s (REST) or never arrives (SSE). Both verified failing against a deliberately broken buildURL and stream URL before being kept. Qualifies the 401 row on both reference pages: "denied with 403" was too absolute. A missing token is evaluated as default_role, which may well succeed — the point is that it never yields 401. Also trims the changelog's claim that this branch added SSE_ERROR/SSE_CONNECT_ERROR to the TS error table; post-merge those rows come from #448 in the same Unreleased section. --- CHANGELOG.md | 2 +- clients/go/http_test.go | 27 +++++++++++++++++ clients/go/stream_test.go | 36 +++++++++++++++++++++++ docs/src/content/docs/sdk/go/reference.md | 2 +- docs/src/content/docs/sdk/reference.md | 2 +- 5 files changed, 66 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54bef1b9..848f2325 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table was missing `SSE_ERROR` and `SSE_CONNECT_ERROR` entirely, and described `401` as "missing or invalid JWT" when a *missing* token actually resolves to `default_role` and is denied with `403` (`internal/auth/auth.go`) — only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. +- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. - **Go module cache stored once instead of once per compile flavor** (`.github/actions/setup-env/action.yml`, `.github/workflows/README.md`, `.github/workflows/publish-dev.yml`, `.github/workflows/release.yml`, `Makefile`): closes [#443](https://github.com/Wave-RF/WaveHouse/issues/443). `setup-env` cached `~/go/pkg/mod` together with `~/.cache/go-build` under a key partitioned by `go-cache-suffix`, but the module cache is a pure function of `go.mod` + `go.sum` and byte-identical for every flavor — so that tree was stored five times over (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`) — five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation (the tree is ~1.6 GB on disk; 0.48 GB as a stored archive on a cold save, drifting up as superseded versions accumulate). Two live generations is the steady state (a bump mints a new set while the previous is still warm), so the repo sat near GitHub's hard 10 GB cache cap; the 24-module go-deps bump ([#438](https://github.com/Wave-RF/WaveHouse/pull/438)) tipped it to 10.53 GB and GitHub began LRU-evicting warm entries mid-run. The one cache is now two: `gomod-v1--` on `~/go/pkg/mod`, **unsuffixed** and shared by every `ci.yml` Go job that goes through `setup-env`, and `gobuild-v3--go-` on `~/.cache/go-build` only, still per flavor. Measured after the split: **1.05 GB per generation** (0.48 GB module + 0.57 GB across the five build entries), down from 5.18 GB — a 4.9x reduction. All sizes are stored-archive bytes / 2^30, the unit the README's usage check prints. The `v3` bump is load-bearing — saves fire only on an exact-key miss, so without it the old `v2` entry (still carrying the module cache) would exact-hit forever and the smaller content would never be saved — and `gobuild-v3` drops the bare-prefix restore-key, which existed solely to borrow another flavor's copy of the module cache. Separately, `publish-dev.yml` and `release.yml` now pass `cache: false` to `actions/setup-go` (matching `goreleaser-validate.yml`), which was holding a sixth ~1 GB entry — the module tree `gomod-v1` already keeps once, plus that job's own 8-target cross-compile objects — re-saved on every cache miss. (That entry is keyed on the root `go.mod`: setup-go hashed `go.sum` through v6.2.0 and `go.mod` from v6.3.0, [actions/setup-go#705](https://github.com/actions/setup-go/pull/705).) `publish-dev.yml` re-caches only the half that pays for itself, under `gobuild-v3--go-release-` (~0.5 GB — the bundled entry minus `gomod-v1`'s share): across its last 20 runs GoReleaser takes 36–246 s with the cross-compile objects warm and 401–446 s cold (measured on `setup-go`'s bundled cache, which carried the same `~/.cache/go-build` tree), so dropping the cross-compile objects outright would have cost roughly 2.5–7 minutes on every push to main (mean delta ≈4.8 min). The `-release` suffix keeps those 8-target objects from being restored by CI's native-only flavors and vice versa. Because those timings were taken with setup-go's bundled entry (which also held `~/go/pkg/mod`), `publish-dev` additionally *restores* `gomod-v1` from `main`'s scope via `actions/cache/restore` — read-only, so it costs no budget and cannot write a partial tree to the key every `ci.yml` Go job shares. Without that restore the job would re-download ~112 MB of modules per push and land above the warm range quoted above. Both Go keys now hash `go.mod` alongside `go.sum`, for a different reason each: the GOTOOLCHAIN=auto toolchain lives in `~/go/pkg/mod` and `go.sum` records no entry for it, so a `go`-directive bump would otherwise exact-hit a toolchain-less archive and — saves firing only on an exact-key miss — re-download it every run; and the compiler's build ID keys every build object, so the same bump invalidates `~/.cache/go-build` too, where the failure mode is a permanent cold recompile rather than a re-download. Two guards come with the shared entry: `setup-env` now fails a `go: true` job that passes no `go-cache-suffix` (an empty one yields a restore-key prefix-matching every other flavor), and `make cov` gains the `go-mod-download` prerequisite its siblings already had — CI's coverage job shares the unsuffixed `gomod-v1` and races to save it, but ran only `go run ./scripts/cov report`, so winning that race would have stored a partial `~/go/pkg/mod` that then exact-hit for every other job until the next rotation. The workflows README gains a sizing policy — the 10 GB cap, the two-generations rule, how to check the current footprint, and the rule that lockfile-derived content is keyed once and shared — plus the narrowing-rotation exception to the key-versioning policy. - **A path prefix in the SDK's `baseURL` now survives instead of being silently discarded** (`clients/ts/src/url.ts` (new), `clients/ts/src/http.ts`, `clients/ts/src/stream/sse.ts`, `clients/ts/src/cli/codegen.ts`, `clients/ts/src/url.test.ts` (new), `clients/ts/src/stream/sse.test.ts` (new), `clients/ts/src/{http,client}.test.ts`, `docs/src/content/docs/sdk/index.mdx`, `docs/src/content/docs/reverse-proxy.mdx`): closes #428. Pointing the SDK at a WaveHouse served under a prefix — `createClient({ baseURL: 'https://app.example.com/api/warehouse' })`, the shape you get behind a BFF, an app-server route, or a path-routed ingress — dropped the prefix from every request. Both transports resolved *absolute* request paths against the base (`new URL('/v1/query', base)` in `http.ts`, `new URL('/v1/stream', baseURL)` in `stream/sse.ts`), and per the URL spec an absolute path replaces the base's path entirely, so calls went to the origin root. The failure mode was the bad kind: no error, a client that looks correctly configured, and every request quietly going somewhere else — with no workaround from outside the SDK, since `baseURL` was the only path input and it couldn't survive. Request paths are now joined **onto** the base by a single shared `resolveURL` helper that both transports and the codegen CLI call (previously three separate constructions, one of which — codegen's string concat — already handled prefixes, so they disagreed). The helper normalizes the base to a directory before resolving, so a bare last segment or a stray query/fragment on `baseURL` can't eat the prefix either, and a root-hosted base (`http://localhost:8080`, the overwhelmingly common case) resolves exactly as before. Tests pin a prefixed base end-to-end across both transports. The proxy in front must still strip the prefix before forwarding — WaveHouse has no configurable base path by design — which the reverse-proxy guide now covers with nginx/Caddy snippets. diff --git a/clients/go/http_test.go b/clients/go/http_test.go index df9045ff..acf857d1 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -293,3 +293,30 @@ func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) } } + +// A BaseURL carrying a path prefix must survive on both transports — the bug +// #428 fixed in the TS client, which Go avoids by concatenating rather than +// resolving. Guards against a future switch to url.JoinPath/ResolveReference. +func TestBaseURLPathPrefixIsPreserved(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/query", func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + for _, base := range []string{srv.URL + "/api/warehouse", srv.URL + "/api/warehouse/"} { + client := NewClient(Config{BaseURL: base, Options: &ClientOptions{}, HTTPClient: srv.Client()}) + + var result map[string]string + if err := doRequest(context.Background(), client.ctx, requestOptions{ + method: "POST", + path: "/v1/query", + }, &result); err != nil { + t.Fatalf("base %q: %v", base, err) + } + if result["status"] != "ok" { + t.Fatalf("base %q: want ok, got %v", base, result) + } + } +} diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 1d9a9f69..efe10fd2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -430,3 +430,39 @@ func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { t.Fatalf("reconnect: want since=, got %q", sinceParams[1]) } } + +// The SSE transport builds its URL separately from buildURL (stream.go), so a +// BaseURL path prefix needs its own guard. See TestBaseURLPathPrefixIsPreserved. +func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { + gotPath := make(chan string, 1) + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/stream", func(w http.ResponseWriter, r *http.Request) { + select { + case gotPath <- r.URL.Path: + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(200) + w.(http.Flusher).Flush() + <-r.Context().Done() + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL + "/api/warehouse", + Options: &ClientOptions{}, + HTTPClient: srv.Client(), + }) + sc := client.From("clicks").Stream(nil) + t.Cleanup(sc.Close) + + select { + case p := <-gotPath: + if p != "/api/warehouse/v1/stream" { + t.Fatalf("want prefixed stream path, got %q", p) + } + case <-time.After(3 * time.Second): + t.Fatal("stream never reached the prefixed path") + } +} diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 7471371d..35e0ba2f 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -54,7 +54,7 @@ SDK's "the SDK never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index db6fc36b..92b63460 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -30,7 +30,7 @@ The SDK **never throws** for anything the server returns — all API errors come | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | From 0a46251e30e3dbc9daf63c2ac7eadbe9880d20ac Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:53:03 -0400 Subject: [PATCH 27/59] fix(sdk): clamp Retry-After before overflow; unalias insert-result pointers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triage of the 24 open review threads surfaced one real defect. retryAfterDelay computed time.Duration(secs) * time.Second before any range check, which wraps negative past ~9.2e9 seconds — and because the wrap goes negative, the min(delay, maxRetryAfter) clamp below selected it rather than capping. The retry timer then fired immediately, so a server answering 429/503 with a large Retry-After got hammered with zero-delay retries: the exact inverse of the header's purpose, and a silent breach of the documented 30s cap. Verified: "10000000000" yielded -2346317h47m53s, MaxInt64 yielded -1s. The HTTP-date branch is unaffected — time.Sub saturates rather than wrapping. emptyInsertResult aliased one &z across Total/Succeeded/Failed/Duplicates, so a caller writing through any one of those exported *int fields mutated all four. Separate vars now. fetchNextTyped discarded its json.Marshal error, truncating a result set to look like normal end-of-pagination. Propagated. The Decode error stays deliberate — a Row marshaling to a non-object ends pagination the same way an absent cursor column does — now commented as such, with the parity question tracked in #452. Test fixes for guards that could not fail: - InsertByteSliceNotBatch asserted only the path and OK, both of which the batch path also satisfies; it now pins Content-Type and body, and was confirmed to fail with the []byte carve-out removed. - backoff's CappedAt30s accepted 24-36s where the cap is applied after jitter and returns exactly 30s, so moving the cap before the jitter still passed. - retryAfterDelay's HTTPDateFuture window subsumed the ~1s parse-failure fallback; tightened, and two overflow cases added. - TestNewClient_HasNamespaces boxed typed pointers into map[string]any, where a nil typed pointer is never == nil — it could not fail. Compares concrete fields now. - TestClient_From discarded Fetch's error, so an early return left its handler assertions unreached. Docs: a pagination snippet that did not compile, three shell examples that were bash syntax errors (unquoted ), the cursor tie-breaker caveat (#452), and the raw-SQL claim that a tokenless request is rejected — untrue when default_role is the admin role, which AGENTS.md permits as dev-only. Fixed on both SDKs' pages. --- clients/go/README.md | 2 +- clients/go/client_test.go | 27 ++++++++++++++++++----- clients/go/http.go | 6 +++++ clients/go/http_test.go | 19 ++++++++++++++-- clients/go/query_builder.go | 12 +++++++++- clients/go/table.go | 6 +++-- clients/go/table_test.go | 17 +++++++++++++- docs/src/content/docs/sdk/go/queries.md | 18 +++++++++++++-- docs/src/content/docs/sdk/go/reference.md | 4 ++-- docs/src/content/docs/sdk/queries.md | 2 +- 10 files changed, 96 insertions(+), 17 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index c1412c6d..112aae2b 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -195,7 +195,7 @@ rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM click Generate Go structs from a running WaveHouse instance: ```bash -export WAVEHOUSE_AUTH= # avoids leaking the token via argv +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 22a45067..aca92d0d 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -37,10 +37,23 @@ func TestNewClient_CustomMaxRetries(t *testing.T) { func TestNewClient_HasNamespaces(t *testing.T) { c := NewClient(Config{BaseURL: "http://localhost:8080"}) - for name, ns := range map[string]any{"Sys": c.Sys, "Schema": c.Schema, "Policy": c.Policy, "Pipes": c.Pipes, "DLQ": c.DLQ} { - if ns == nil { - t.Fatalf("%s namespace is nil", name) - } + // Compared as concrete typed pointers, not boxed into map[string]any: a nil + // typed pointer in an interface is never == nil, so the map form passed + // even if NewClient stopped assigning a namespace entirely. + if c.Sys == nil { + t.Error("Sys namespace is nil") + } + if c.Schema == nil { + t.Error("Schema namespace is nil") + } + if c.Policy == nil { + t.Error("Policy namespace is nil") + } + if c.Pipes == nil { + t.Error("Pipes namespace is nil") + } + if c.DLQ == nil { + t.Error("DLQ namespace is nil") } } @@ -54,7 +67,11 @@ func TestClient_From(t *testing.T) { })) t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) - _, _ = c.From("events").Fetch(context.Background()) + // Checked, not discarded: if Fetch returns before issuing the request, the + // handler never runs and the table= assertion above proves nothing. + if _, err := c.From("events").Fetch(context.Background()); err != nil { + t.Fatal(err) + } } func TestClient_SQL(t *testing.T) { diff --git a/clients/go/http.go b/clients/go/http.go index d8a8b37b..9d149e90 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -174,6 +174,12 @@ func buildURL(base, path string, params url.Values) string { func retryAfterDelay(ra string, attempt int) time.Duration { delay := backoff(attempt) if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { + // Compare before converting: time.Duration(secs) * time.Second wraps + // negative past ~9.2e9 seconds, and min() below would then pick the + // negative value, firing the retry timer instantly. + if secs > int(maxRetryAfter/time.Second) { + return maxRetryAfter + } delay = time.Duration(secs) * time.Second } else if parsed, err := http.ParseTime(ra); err == nil { if d := time.Until(parsed); d > 0 { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index acf857d1..57109f3b 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -224,10 +224,19 @@ func TestBackoff(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { + got := backoff(tt.attempt) + if tt.name == "CappedAt30s" { + // The cap is applied *after* jitter, so this is exact — a ±20% + // window here would also accept capping before the jitter, + // which lets the documented 30s max drift to 36s. + if got != 30*time.Second { + t.Errorf("backoff(%d) = %v, want exactly 30s", tt.attempt, got) + } + return + } // backoff applies ±20% jitter around the exponential base. lo := time.Duration(float64(tt.base) * 0.8) hi := time.Duration(float64(tt.base) * 1.2) - got := backoff(tt.attempt) if got < lo || got > hi { t.Errorf("backoff(%d) = %v, want within [%v, %v]", tt.attempt, got, lo, hi) } @@ -243,6 +252,10 @@ func TestRetryAfterDelay(t *testing.T) { }{ {"DeltaSeconds", "5", 5 * time.Second}, {"ClampedToMax", "3600", maxRetryAfter}, + // time.Duration(secs) * time.Second wraps negative past ~9.2e9s; an + // unguarded min() then picks the negative and retries instantly. + {"OverflowClamped", "10000000000", maxRetryAfter}, + {"MaxIntClamped", "9223372036854775807", maxRetryAfter}, {"HTTPDateFuture", time.Now().Add(10 * time.Second).UTC().Format(http.TimeFormat), 0}, // range-checked below {"Garbage", "not-a-delay", 0}, // range-checked below } @@ -251,7 +264,9 @@ func TestRetryAfterDelay(t *testing.T) { got := retryAfterDelay(tt.ra, 0) switch tt.name { case "HTTPDateFuture": - if got <= 0 || got > 10*time.Second { + // Lower bound well clear of the backoff(0) fallback (~1s), so a + // broken HTTP-date branch can't pass by falling through to it. + if got < 5*time.Second || got > 10*time.Second { t.Fatalf("want ~10s, got %v", got) } case "Garbage": diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 6991371f..8d0e0212 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "encoding/json" + "fmt" "net/url" ) @@ -283,10 +284,19 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or // codegen structs — their 64-bit int columns are int64/uint64, and // 128/256-bit are json.Number) when paging on >2^53 integer cursors. - raw, _ := json.Marshal(lastRow) + raw, err := json.Marshal(lastRow) + if err != nil { + // Row itself is unmarshalable (e.g. a func field absent from the + // response). Silently truncating the result set would look like + // normal end-of-pagination, so surface it. + return nil, fmt.Errorf("wavehouse: marshal cursor row: %w", err) + } m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) dec.UseNumber() + // Decode error is deliberate: a Row that marshals to a non-object + // (FetchTyped[[]any], a scalar row type) leaves m empty and ends + // pagination below, same as an absent cursor column. Tracked in #452. _ = dec.Decode(&m) } lastValue, exists := m[cursor.Column] diff --git a/clients/go/table.go b/clients/go/table.go index 204a3b26..e66f7f05 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -119,8 +119,10 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e } func emptyInsertResult() *InsertResult { - z := 0 - return &InsertResult{OK: true, Total: &z, Succeeded: &z, Failed: &z, Duplicates: &z} + // Separate vars, not one aliased &z: the fields are exported *int, so a + // caller writing through one would otherwise mutate all four. + total, succeeded, failed, duplicates := 0, 0, 0, 0 + return &InsertResult{OK: true, Total: &total, Succeeded: &succeeded, Failed: &failed, Duplicates: &duplicates} } func marshalNDJSON(n int, elem func(int) any) (string, error) { diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 2dfd0cff..652daf88 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -126,11 +126,14 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { // insertSingle rather than being (mis)treated as a slice of per-byte rows. func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var mu sync.Mutex - var gotPath string + var gotPath, gotCT, gotBody string c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path + gotCT = r.Header.Get("Content-Type") + b, _ := io.ReadAll(r.Body) + gotBody = string(b) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) @@ -145,6 +148,18 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } + // The batch path posts to the same URL and also yields ok=true, so the + // wire format is the only thing that distinguishes them: one opaque JSON + // value vs. NDJSON of 16 per-byte rows. + if gotCT != "application/json" { + t.Fatalf("want application/json (single insert), got %q", gotCT) + } + // encoding/json base64s a []byte — documented in queries.md as a value the + // server rejects (use InsertNDJSON for raw bytes). Pinned here because it + // proves the batch path wasn't taken. + if gotBody != `"eyJwYWdlIjoiL2hvbWUifQ=="` { + t.Fatalf("want single base64 value, got %q", gotBody) + } } func TestTableRef_InsertEmptyBatch(t *testing.T) { diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index d63c6715..8806e4cf 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -319,9 +319,15 @@ Execute the query and decode rows into `[]map[string]any`. The ordinary ```go page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) +if err != nil { + return err +} if page.HasMore && page.Next != nil { - page2, err := page.Next(ctx) // cursor-based pagination — needs OrderBy + page, err = page.Next(ctx) // cursor-based pagination — needs OrderBy + if err != nil { + return err + } } ``` @@ -353,6 +359,13 @@ add an `.OrderBy()` to paginate. If the order column was left out of an explicit `.Select(...)` projection, `Next` quietly returns an empty page instead of erroring (there is no cursor value to read). +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses +only that first `.OrderBy()` column, with no tie-breaker — so rows sharing the +boundary value with the last row of a page are skipped. Paginate on a column +that is unique per row (or made unique by a monotonic timestamp), or accept +that ties at a page edge can be dropped. The TypeScript SDK's `next()` has the +same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): rows decode into `map[string]any`, where JSON numbers become `float64`, so an integer cursor column loses exactness past 2^53 and pagination can repeat or @@ -387,7 +400,8 @@ for page.HasMore && page.Next != nil { Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, the token must resolve to the policy admin role (`admin_role`, `"admin"` by default) — a JWT request with no token, or an invalid/expired one, falls -back to the `default_role` and is rejected. Alternatively, a configured +back to the `default_role`, and is rejected unless the deployment sets +`default_role` to the admin role (permitted, but dev-only). Alternatively, a configured operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always sends its token as `Bearer `, so to use an operator key from this diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 35e0ba2f..6a8590d5 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -147,7 +147,7 @@ Generate Go structs from a running WaveHouse instance. The module ships a `wavehouse-codegen` command under `cmd/`: ```bash -export WAVEHOUSE_AUTH= # avoids leaking the token via argv +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ @@ -263,7 +263,7 @@ E2E tests (build tag `e2e`) run against a live WaveHouse instance and have their own Make target, separate from the repo's `make test-e2e`: ```bash -WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH= make test-go-sdk-e2e +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e ``` `WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 77aecbe1..5dd51b6c 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -259,7 +259,7 @@ while (result.hasMore && result.next) { ## Raw SQL — `wh.sql(query, opts?)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. +Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role`, and is rejected unless the deployment sets `default_role` to the admin role (permitted, but dev-only). ```ts const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); From 9da52dd99604814a823c625f26901c95f2600e78 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:04:06 -0400 Subject: [PATCH 28/59] test(sdk): pin the fetchNextTyped marshal-error branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The guard added in 0a46251 had no test — the same gap that commit existed to close. A Row that unmarshals cleanly but fails to marshal back (an exported func field, absent from the response) previously produced an empty page, indistinguishable from real exhaustion. Verified failing with the guard swallowing the error instead of returning it. --- clients/go/query_builder_test.go | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 02431746..7ccb9027 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -456,3 +456,26 @@ func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) } } + +// The cursor round-trip re-marshals the last row to read its cursor value. A +// Row that unmarshals cleanly but can't be marshaled back (an exported func +// field, absent from the response) must surface an error rather than an empty +// page, which is indistinguishable from real exhaustion. +func TestQueryBuilder_Pagination_UnmarshalableRowErrors(t *testing.T) { + type row struct { + ID string `json:"id"` + Cb func() `json:"cb"` + } + c, _ := pagingServer(t, [][]map[string]any{{{"id": "a"}, {"id": "b"}}}) + page, err := FetchTyped[row](context.Background(), + c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2)) + if err != nil { + t.Fatal(err) + } + if page.Next == nil { + t.Fatal("want a Next cursor") + } + if _, err := page.Next(context.Background()); err == nil { + t.Fatal("want a marshal error, got a silently empty page") + } +} From 4cc229a36bacfb0ee232b0f82856c8342e20f8a7 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:08:13 -0400 Subject: [PATCH 29/59] docs: finish the TS side of the pagination parity sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Go queries page documents two pagination footguns and names the TS SDK as sharing both, but the TS page carried neither — so a data-loss caveat lived only on the page TS readers don't open. Ports both: the strict single-column cursor that drops rows tying the page boundary (#452, verified against clients/ts/src/query-builder.ts), and the Number.MAX_SAFE_INTEGER ceiling on integer cursors. Also fills the not_like row's Backend column on the TS operator table — the SDK does send a wire token (query-builder.ts maps not_like to "not_like"), and a reader debugging the resulting 400 needs to know which token the server rejected. The Go table already said so. Two smaller corrections: reverse-proxy.mdx claimed the Go SDK "preserves the prefix on every released version", asserting a property of a set that development.md says is empty (no tagged Go releases yet); and the key-targets table under-reported make lint / verify / fix, all three of which fan out to markdown, prose, shell, workflow, and astro checks beyond what was listed. --- docs/src/content/docs/development.md | 6 +++--- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/queries.md | 6 +++++- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index b62d7635..88e1f262 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -483,10 +483,10 @@ Run `make help` to see all targets. Key ones: | **Static checks** | | | `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | +| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) + Markdown (markdownlint) + docs prose (misspell) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | -| `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | +| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) + Markdown/prose, shell (shellcheck), workflows (actionlint), and `astro check` (parallel-safe: `make -j verify`) | +| `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`), TS (Biome `--write`), Markdown (markdownlint) + docs prose (misspell) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | | `make build-release` | Stripped release-style build → `bin/wavehouse-release` | diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index c2ce9b60..94015a00 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -93,7 +93,7 @@ wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wareh ``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix on every released version. +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK has supported prefixed base URLs since its first release. ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 5dd51b6c..0f6a8efd 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -152,7 +152,7 @@ clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') | `'<='` | `lte` | Less than or equal | | `'in'` | `in` | Value in array | | `'like'` | `like` | SQL LIKE pattern | -| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | #### Aggregations @@ -255,6 +255,10 @@ while (result.hasMore && result.next) { } ``` +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. The Go SDK's `Next` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +Rows decode with JSON numbers as JS `number`s, so an integer cursor column past `Number.MAX_SAFE_INTEGER` (2^53) loses exactness and pagination can repeat or skip a row at that scale. The Go SDK's `FetchTyped` with an `int64` field avoids this; there is no JS equivalent short of a string or `bigint` column. + --- ## Raw SQL — `wh.sql(query, opts?)` From 6d29538357f0fe324d6adc576f40f8c12e4fc3fd Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:19:52 -0400 Subject: [PATCH 30/59] docs: flag the TS codegen Decimal mismatch; fix the Go release framing The TypeScript codegen type table promised `number` for Decimal* columns, but /v1/query returns them as quoted strings: transformRow in internal/api/clickhouse_exec.go converts only UUID and time.Time, so a shopspring decimal.Decimal reaches json.Marshal and marshals quoted. A TS user with a Decimal price column gets a number-typed field holding "12.34" and silently wrong arithmetic, with no type error. The Go page already documented the real shape, which is what made the discrepancy visible. Footnoted both that row and Array(UInt8) (base64 on the same path, #436); the codegen fix itself is tracked in #453. Also corrects the path-prefix sentence I got wrong in both directions: it now says the prefix is preserved in every version `go get` can resolve, rather than asserting a release history that doesn't exist yet. --- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/reference.md | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 94015a00..6b1f58a7 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -93,7 +93,7 @@ wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wareh ``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK has supported prefixed base URLs since its first release. +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix in every version `go get` can resolve — it has behaved this way since its first commit (there are no tagged Go releases yet). ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 92b63460..d7a6fdae 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -136,10 +136,11 @@ export interface ClicksRow { | ClickHouse Type | TypeScript Type | |----------------|-----------------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | -| `UInt*`, `Int*`, `Float*`, `Decimal*` | `number` | +| `UInt*`, `Int*`, `Float*` | `number` | +| `Decimal*` | `number` *(generated)* — but `/v1/query` returns Decimals as **quoted strings**, so treat the field as `string` until codegen is fixed ([#453](https://github.com/Wave-RF/WaveHouse/issues/453)) | | `Bool` | `boolean` | | `Nullable(T)` | `T \| null` | -| `Array(T)` | `T[]` | +| `Array(T)` | `T[]` — except `Array(UInt8)`, which `/v1/query` base64-encodes, so the generated `number[]` is a `string` at runtime ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | | `Map(K, V)` | `Record` | | `LowCardinality(T)` | same as `T` | From 7b85e0687844d032aa439276ef582636a974b5db Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 12 Aug 2026 15:14:46 -0400 Subject: [PATCH 31/59] chore(docs): revising Go SDK documentation --- clients/go/README.md | 8 +- docs/src/content/docs/sdk/go/admin.md | 39 ++--- docs/src/content/docs/sdk/go/index.md | 111 ++++---------- docs/src/content/docs/sdk/go/pipes.md | 29 +--- docs/src/content/docs/sdk/go/queries.md | 173 +++++----------------- docs/src/content/docs/sdk/go/reference.md | 120 ++++----------- docs/src/content/docs/sdk/go/streaming.md | 141 +++++------------- 7 files changed, 143 insertions(+), 478 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index 112aae2b..64e19cac 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -2,7 +2,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. -**Zero third-party runtime dependencies** — stdlib only. +**Zero third-party runtime dependencies** (stdlib only). **[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** @@ -80,7 +80,7 @@ client = wavehouse.NewClient(wavehouse.Config{ }) ``` -`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`) when WaveHouse is served under one. A trailing `/` is trimmed and every request path is appended to it, on both REST and SSE — see [Config](https://wavehouse.dev/sdk/go#config). +`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`). A trailing `/` is trimmed, and request paths are appended, for both REST and SSE alike ([Config](https://wavehouse.dev/sdk/go#config)). ## Typed Queries (Generics) @@ -206,7 +206,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#c ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: +Request-response ops return `(T, error)`, or bare `error` for body-less calls (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP errors are `*wavehouse.Error` (unwrap with `errors.As`); failures before the request goes out (`Auth` provider, body marshal) are plain wrapped errors, so handle `errors.As == false` too. Streaming lifecycle (`Stream`, `Subscribe`, `Close`, `Connected`) reports via callbacks or plain errors: ```go page, err := client.From("clicks").Fetch(ctx) @@ -218,7 +218,7 @@ if err != nil { } ``` -The HTTP layer retries 5xx, 429, and network errors with exponential backoff (default 2 retries). `Retry-After` on a 503 or 429 is honored, capped at 30s. Context cancellation returns immediately with code `ABORTED`. +The HTTP layer retries 5xx, 429, and network errors with exponential backoff (2 retries by default). `Retry-After` on 503/429 is honored, capped at 30s. Context cancellation returns `ABORTED` immediately. ## License diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index 957713fd..65695dbf 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -3,15 +3,11 @@ title: "Go SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." --- -Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. -Everything here except `client.Sys.Health` requires the admin role -(`policy.admin_role`) — see [Access Control](/access-control) for how roles -resolve. Compare with the TypeScript SDK's [Admin & System](/sdk/admin) -page. +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. ## Schema — `client.Schema` -Introspect ClickHouse table schemas. +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-only** `/v1/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. ```go // List all table schemas. @@ -22,20 +18,13 @@ schemas, err := wh.Schema.List(ctx) err = wh.Schema.Refresh(ctx) ``` -Individual table schema is also available via `wh.From("clicks").Schema(ctx)`. - -> `wh.Schema.List`, `wh.Schema.Refresh`, and `wh.From(t).Schema` hit -> `/v1/schema*`, which are **admin-only** endpoints. Against any non-dev -> policy (anything but `default_role: admin`), construct the client with an -> admin-role token or these calls return a `*wavehouse.Error` with -> `Status: 403`. +Individual table schema: `wh.From("clicks").Schema(ctx)`. --- ## Policy — `client.Policy` -Manage Hasura-style access control policies. Requires the admin role -(`policy.admin_role`). +Manage Hasura-style access control policies (admin role required). ```go // Get current policy. @@ -66,10 +55,7 @@ result, err := wh.Policy.Validate(ctx, policyDraft) // result.Valid == true, or err wraps the validation failure details ``` -`PolicyFilter`'s fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, not -`string` — an intentional empty-string comparison round-trips distinctly -from an absent operator. Take the address of a local variable (as above) or -write a small helper if you find yourself doing this often: +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string` to distinguish empty strings from absent operators. Use a helper: ```go func strPtr(s string) *string { return &s } @@ -79,7 +65,7 @@ func strPtr(s string) *string { return &s } ## DLQ — `client.DLQ` -Dead Letter Queue operations. Requires the admin role (`policy.admin_role`). +Dead Letter Queue operations (admin role required). ```go // Get DLQ statistics. @@ -91,17 +77,13 @@ stats, err := wh.DLQ.List(ctx) stats, err = wh.DLQ.Table(ctx, "clicks") ``` -`wh.DLQ.Stream(opts)` exists in the API but is **not yet functional**: -there is no server-side DLQ stream today (the SSE bridge only carries -`ingest.>` subjects), so it connects and receives no events — live DLQ -streaming is tracked in -[#197](https://github.com/Wave-RF/WaveHouse/issues/197). +`wh.DLQ.Stream(opts)` is **not yet functional**: no server-side DLQ stream exists (the SSE bridge carries only `ingest.>` subjects), so it connects and receives nothing. Tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). --- ## System — `client.Sys` -Content-free server-online check. +Server-online check. ```go // Health hits the public, content-free /v1/health route — 200 → nil error, @@ -113,7 +95,4 @@ if err := wh.Sys.Health(ctx); err != nil { } ``` -> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — -> it runs a ClickHouse query per call and is a load-balancer / reverse-proxy -> concern, not the client's. Probe `/readyz` directly from your -> orchestrator if you need it. +> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern. Probe it directly from your orchestrator. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index c70e6fd9..f2a4fc7f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -20,7 +20,7 @@ either page mostly carries over. go get github.com/Wave-RF/WaveHouse/clients/go ``` -Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a supported-releases floor rather than the server's patch-pinned toolchain). +Requires Go 1.24+ (the `go.mod` floor, matching supported releases rather than server's patch-pinned toolchain). ## Import @@ -28,9 +28,7 @@ Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a suppor import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ``` -The package name is `wavehouse`; aliasing the import isn't required, but -keeps call sites short (`wavehouse.NewClient(...)`, `wavehouse.OpEq`, ...) — -every example on these pages assumes it. +Aliasing `wavehouse` is optional but keeps call sites short; all examples here assume it. ## Quick Start @@ -65,7 +63,7 @@ func main() { } ``` -See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md) for more quick-start examples. +Find more examples in the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md). ## Creating a Client @@ -85,42 +83,28 @@ wh := wavehouse.NewClient(wavehouse.Config{ | Field | Type | Default | Description | |-------|------|---------|-------------| -| `BaseURL` | `string` | — | WaveHouse server URL, optionally including a path prefix (required). A trailing `/` is trimmed; every request path is appended to it on both transports, so a WaveHouse served under `https://app.example.com/wavehouse` works as-is. | -| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | -| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports (see caution below) | +| `BaseURL` | `string` | — | Required WaveHouse server URL, optionally with a path prefix. A trailing `/` is trimmed and every request path is appended on both transports, so a server under `https://app.example.com/wavehouse` works as-is. | +| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider called before each request. `nil` means unauthenticated access. | +| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below). | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports. | :::caution[Timeouts: use contexts, not `http.Client.Timeout`] -The default client sets no `Timeout` — a `context.Context` deadline is the -only bound on a request, so pass one for anything that mustn't hang on a -stalled server. If you supply your own `HTTPClient`, leave `Timeout` unset: -it covers body reads too, so it would kill every long-lived SSE stream at -the timeout and force a reconnect loop. Use `Transport`-level dial / -TLS / response-header timeouts instead. +The default client has no `Timeout`; use a `context.Context` deadline to prevent hangs. If supplying your own `HTTPClient`, leave `Timeout` unset, as it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. ::: ### `ClientOptions` | Field | Type | Default | Description | |-------|------|---------|-------------| -| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures) | +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | -A `*Client` is safe for concurrent use by multiple goroutines — client state -is immutable after `NewClient`, and every builder chain copies. Supply a -concurrency-safe `Auth` func (it's called from any goroutine that issues a -request). +`*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. :::caution[`Options` opts you out of the default, not just in] -The default of 2 retries only applies when `Config.Options` is `nil`. If you -set `Options` to configure anything else in the future, an unset -`MaxRetries` field is Go's int zero value — `0` — which is a **valid, -explicit** "no retries" setting, not "use the default." Today `MaxRetries` -is the struct's only field, so this mostly matters if you pass -`&wavehouse.ClientOptions{}` and expect retry-by-default: you won't get it. +The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. ::: -For a static token that never rotates, use `wavehouse.StaticToken(token)` -instead of writing the closure yourself: +For static tokens, use `wavehouse.StaticToken(token)`: ```go wh := wavehouse.NewClient(wavehouse.Config{ @@ -130,24 +114,16 @@ wh := wavehouse.NewClient(wavehouse.Config{ ``` :::note[How the token is transmitted] -Unlike a browser's `EventSource`, Go's `net/http` client can set arbitrary -headers on any request — so the Go SDK sends `Authorization: Bearer ` -on **every** request, including SSE streams. There's no `?token=` query -parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser -concern only; see its [equivalent note](/sdk#creating-a-client)). +Unlike a browser's `EventSource`, Go's `net/http` client sets arbitrary headers on any request, so the Go SDK sends `Authorization: Bearer ` on every request, including SSE streams. No `?token=` query fallback (a TypeScript-in-the-browser concern; see its [equivalent note](/sdk#creating-a-client)). ::: :::caution[Use HTTPS for authenticated non-local servers] -The SDK doesn't forbid `http://` base URLs — local development and -private-network deployments rely on them — but a bearer token sent over -plaintext HTTP is readable by anything on the path. Point authenticated -clients at `https://` endpoints outside a trusted network. +While the SDK allows `http://` for local development or private networks, bearer tokens over plaintext HTTP are insecure. Use `https://` for endpoints outside trusted networks. ::: ## Typed Rows (Generics) -Pass a row type as a type parameter to get results decoded straight into -your struct, instead of `map[string]any`: +Pass a row type parameter to decode results into your struct instead of `map[string]any`: ```go type ClickRow struct { @@ -162,19 +138,13 @@ page, err := wavehouse.FetchTyped[ClickRow](ctx, // page.Data is []ClickRow ``` -Generate row structs from a running server with the -[codegen CLI](/sdk/go/reference#codegen-cli). +Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. -`FetchTyped` is a package-level generic function, not a method — Go doesn't -support generic methods, so this (and `Fetch[Row]` for pipes, and -`SQL[Row]` for raw SQL) are top-level functions that take the client or -builder as an argument. Untyped equivalents (`.FetchUntyped(ctx)`, decoding -into `map[string]any`) are ordinary methods, since they need no type -parameter. +`FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods. Untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) +Request-response operations (queries, ingest, pipes, admin) return `(T, error)` or just `error` if no body exists (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (e.g., `Auth` provider, marshal errors) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) use callbacks or plain errors; see [Streaming](/sdk/go/streaming). ```go page, err := wh.From("clicks").Fetch(ctx) @@ -193,40 +163,19 @@ See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry b ## Differences from the TypeScript SDK -The two SDKs share a wire format and mirror each other's feature set closely -(a shared `wire_cases.json` conformance fixture is replayed by a test runner -per SDK — both run in CI — asserting each produces the expected HTTP request -for equivalent builder calls), but the -languages pull the API shape in different directions: - -- **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in - an `{ok, data, error}` object, and there's no `error: null` sentinel to - check — a non-nil `error` is the only signal. -- **`context.Context` instead of `AbortSignal`.** Every non-streaming call - takes a `ctx context.Context` as its first argument; cancel it (timeout or - `cancel()`) instead of building an `AbortController`. See - [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). -- **Streams are closed explicitly, not via `ctx`.** `TableRef.Stream` / - `QueryBuilder.Stream` don't take a `context.Context` — the returned - `*StreamController` manages its own background goroutine and connection, - torn down by calling `.Close()` (deferred `stream.Close()` is the usual - pattern). See [Streaming](/sdk/go/streaming). -- **Generics live on package-level functions, not methods** (`FetchTyped[Row]`, - `Fetch[Row]`, `SQL[Row]`), because Go doesn't support type parameters on - methods. -- **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call - `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` - explicitly; there's no bare `await builder` shortcut. -- **Any slice batches, not just `[]map[string]any`.** Go detects slice-ness - via reflection, so `[]ClickRow{...}` takes the same NDJSON batch path as - `[]map[string]any` (the TS SDK's `insert` likewise accepts arrays of typed - rows — this bullet is about the Go mechanics, not a TS gap) — see - [Queries → Insert](/sdk/go/queries#insertctx-data). +Both SDKs share a wire format and feature set, verified by a shared `wire_cases.json` fixture in CI to ensure equivalent HTTP requests for builder calls. However, API shapes differ: + +- **No `Result` union.** Go returns `(T, error)`. A non-nil `error` is the only failure signal; no `{ok, data, error}` objects or `error: null` sentinels are used. +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` as the first argument. Use timeout or `cancel()` instead of `AbortController`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). +- **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. +- **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. +- **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK -- [Queries](/sdk/go/queries) — Tables, the chainable query builder, pagination, and raw SQL. -- [Streaming & Live Queries](/sdk/go/streaming) — Real-time SSE streams, client-side filtering, and backfill-then-live queries. -- [Pipes](/sdk/go/pipes) — Execute and manage named query pipes. +- [Queries](/sdk/go/queries) — Tables, chainable query builder, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/go/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/go/pipes) — Manage named query pipes. - [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, the full API tree, and the codegen CLI. +- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, API tree, and codegen CLI. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 44786601..69e426a1 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -3,16 +3,11 @@ title: "Go SDK Pipes" description: "Execute and manage named query pipes with the WaveHouse Go SDK." --- -Named pipes are server-defined, parameterized queries — the -[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for -any allowed role and manages their definitions under the admin role. -Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. +Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. ## Named Pipes — `client.Pipe(name, params)` -Execute a pre-defined named query pipe. Returns a `*PipeRef`; unlike the -TypeScript SDK's `PipeRef` (which is `PromiseLike`), you always call -`.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]` explicitly. +Execute a pre-defined named query pipe. Returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you must explicitly call `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. ```go rows, err := wavehouse.Fetch[map[string]any](ctx, @@ -22,9 +17,7 @@ rows, err := wavehouse.Fetch[map[string]any](ctx, ### `wavehouse.Fetch[Row](ctx, pipeRef)` -Execute and decode results into `[]Row`. Package-level generic function -(Go has no generic methods) — the same pattern as `FetchTyped` for queries -and `SQL` for raw SQL. +Execute and decode results into `[]Row`. Package-level generic function (Go has no generic methods) — same pattern as `FetchTyped` for queries and `SQL` for raw SQL. ```go type TopPage struct { @@ -37,24 +30,17 @@ rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{" ### `.FetchUntyped(ctx)` -Execute and decode results into `[]map[string]any`. The ordinary -(non-generic) method form of `Fetch`. +Execute and decode results into `[]map[string]any`. The non-generic method form of `Fetch`. ```go rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) ``` -Pass `nil` for `params` when the pipe takes none, or the pipe requires only -parameters with server-side defaults. +Pass `nil` for `params` if the pipe takes none or only requires server-side defaults. ### `.Stream(opts)` -Open a live stream from the pipe's underlying query. See -[Streaming](/sdk/go/streaming). - -This streams by table name, using the pipe's own name as the table — it -only works when the pipe name is also a valid table name. This matches the -TypeScript SDK's `PipeRef.stream()`, which has the same limitation. +Open a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). Streams by table name using the pipe's own name, so it works only when that name is a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. ```go stream := wh.Pipe("top_pages", nil).Stream(nil) @@ -87,8 +73,7 @@ err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ err = wh.Pipes.Delete(ctx, "old_pipe") ``` -`PipeDef` is `Pipe` minus the `Name` field — the name is already in the -`Set`/`Get`/`Delete` call's path argument: +`PipeDef` is `Pipe` minus `Name` — the name is the method's path argument: ```go type PipeDef struct { diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 8806e4cf..337cc042 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -14,8 +14,7 @@ chainable builder methods and `.Stream(opts)` are the exceptions — see ## Tables — `client.From(table)` -`From` returns a `*TableRef` — a reference to a table. It performs no -request by itself, so it's safe to store in a variable or pass around. +`From` returns a `*TableRef`. It performs no request, making it safe to store or pass around. ```go clicks := wh.From("clicks") @@ -23,17 +22,9 @@ clicks := wh.From("clicks") ### `.Fetch(ctx)` -Shortcut for "select every column", with a default limit of 1000 -(`wavehouse.DefaultLimit`). Internally it's -`t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)` — unlike the -TypeScript SDK's `.fetch(opts?)`, there's no options struct to override the -limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself -(see [Query Builder](#query-builder)) if you need a different limit. +Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`). Internally it is `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. Unlike the TypeScript SDK's `.fetch(opts?)`, there is no options struct to override the limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself ([Query Builder](#query-builder)). -When an access-control policy restricts your role's columns, the server -returns only the columns your role is allowed to read — `.Fetch()` is never -a way around `deny_columns`/`allow_columns` (see -[Access control](/access-control#column-permissions)). +Access-control policies restrict returned columns; `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). ```go page, err := clicks.Fetch(ctx) @@ -45,23 +36,14 @@ for _, row := range page.Data { } ``` -To paginate, use the query builder with an explicit `.OrderBy()` instead — -see [Pagination](#pagination). +For pagination, use the query builder with `.OrderBy()` (see [Pagination](#pagination)). ### `.Insert(ctx, data)` -Insert one row or many. What you pass determines the wire format: +Inserts one or many rows based on the input type: -- A single **map or struct** (anything that isn't a slice — plus `[]byte`, - which is treated as one opaque value rather than a batch of numbers, and - would reach the server as a base64 string it rejects; pass raw NDJSON - through `.InsertNDJSON(ctx, string(raw))` instead) is sent as JSON: - `POST /v1/ingest?table={table}`. -- **Any slice** — `[]map[string]any`, a generated/user-defined row type like - `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via - reflection for non-`[]map[string]any` slices) and sent as a single - `application/x-ndjson` request, so a bad record doesn't fail or hide the - rest of the batch. Per-record outcomes come back in the result. +- **Map or struct** (excluding slices and `[]byte`): Sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. +- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): Serialized to NDJSON via reflection and sent as one `application/x-ndjson` request. Per-record outcomes are returned in the result. ```go // Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) @@ -85,23 +67,13 @@ res, err = clicks.Insert(ctx, []ClickRow{ }) ``` -For a batch insert, `res.OK` is `true` only when every record succeeded -(`*res.Failed == 0`). Inspect `res.Failed` and `res.Results` (each -`InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for -partial failures — the returned `error` is reserved for whole-request -failures (network, `404` unknown table, `403` forbidden, `503` -backpressure). An empty slice is a no-op and sends no request. +For batches, `res.OK` is `true` only if all records succeeded (`*res.Failed == 0`). Check `res.Failed` and `res.Results` (each `InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for partial failures. The returned `error` indicates whole-request failures (network, `404`, `403`, `503`). Empty slices are no-ops. -> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw -> JSON array or a single object directly (the `Content-Type` is only a -> hint), so non-SDK clients can send whichever shape is convenient. See the -> [API reference](/api#post-v1ingesttabletable--ingest-data). +> The server is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object (`Content-Type` is only a hint). See [API reference](/api#post-v1ingesttabletable--ingest-data). ### `.InsertNDJSON(ctx, ndjson)` -Insert pre-formatted NDJSON you already have, as a plain `string` — a file -you've read, or a string you built yourself — without first parsing it into -Go values. Returns the same per-record summary as a slice `Insert`. +Inserts pre-formatted NDJSON as a `string` without parsing it into Go values. Returns the same summary as slice `Insert`. ```go // From a literal string. @@ -117,7 +89,7 @@ res, err = clicks.InsertNDJSON(ctx, string(raw)) ### `.Schema(ctx)` -Fetch the table's column definitions from ClickHouse. Admin-only. +Fetch table column definitions from ClickHouse. Admin-only. ```go schema, err := clicks.Schema(ctx) @@ -138,13 +110,7 @@ page, err := clicks.Select("page", "button"). ### `.SelectAll()` -Start a query that selects **every column your role is allowed to read** — -the explicit form of what `.Fetch()` does. Mutually exclusive with -`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); for a -column-restricted role the server expands it to exactly that role's allowed -columns rather than a bare `SELECT *` (unrestricted/admin roles do get -`SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See -[Access control → Column permissions](/access-control#column-permissions). +Selects every column your role is allowed to read. This is the explicit version of `.Fetch()`. It is mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles, the server expands this to allowed columns rather than a bare `SELECT *`; it never bypasses `deny_columns`/`allow_columns` (see [Access control → Column permissions](/access-control#column-permissions)). ```go page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) @@ -158,15 +124,9 @@ Open a real-time event subscription. See [Streaming](/sdk/go/streaming). stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) ``` ---- - ## Query Builder -Returned by `tableRef.Select()`. Immutable — every chain method returns a -new `*QueryBuilder`, so intermediate values can be reused safely. Unlike the -TypeScript SDK's `PromiseLike` builder, a Go `*QueryBuilder` doesn't -auto-execute — call `.FetchUntyped(ctx)` or the package-level -`wavehouse.FetchTyped[Row](ctx, builder)` explicitly: +Returned by `tableRef.Select()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: ```go page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) @@ -174,12 +134,11 @@ page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) ### Chain Methods -All methods return a new `*QueryBuilder` — the original is unchanged. +All methods return a new `*QueryBuilder`; the original remains unchanged. #### `.Select(...columns)` -Append columns to the SELECT clause. A literal `"*"` is the column *named* -`*`, not a wildcard — use `.SelectAll()` for all columns. +Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*`—use `.SelectAll()` for all columns. ```go q := clicks.Select("page").Select("button") // SELECT page, button @@ -187,10 +146,7 @@ q := clicks.Select("page").Select("button") // SELECT page, button #### `.SelectAll()` -Select every column your role may read (the all-columns wildcard; a -column-restricted role's projection is expanded server-side to its allowed -columns). Mutually exclusive with `.Select(...)` and with aggregations -(`.Count()`, `.Sum()`, etc.). +Selects every readable column (expanded server-side based on role). Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`, etc.). ```go q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") @@ -198,7 +154,7 @@ q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") #### `.Where(column, op, value)` -Add a filter condition, using the `FilterOp` constants: +Add a filter using `FilterOp` constants: ```go clicks.Select("page"). @@ -214,9 +170,9 @@ clicks.Select("page"). | `wavehouse.OpGte` | `gte` | Greater than or equal | | `wavehouse.OpLt` | `lt` | Less than | | `wavehouse.OpLte` | `lte` | Less than or equal | -| `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | +| `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | | `wavehouse.OpLike` | `like` | SQL LIKE pattern | -| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | #### Aggregations @@ -231,19 +187,9 @@ clicks.Select("page"). Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn ``` -Custom function names pass through `.Aggregate(fn, column, alias)` but are -validated server-side against a fixed allowlist (matched case-insensitively): -`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, -`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, -`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected -with `400 unsupported aggregation function`. +Custom functions via `.Aggregate(fn, column, alias)` are validated server-side (case-insensitive). Allowlist: `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Others return `400 unsupported aggregation function`. -`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias -string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias -defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ -`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; -`CountDistinct` → `count_distinct_`. `Aggregate` has **no** alias -default — pass one explicitly or the query is sent with `"alias": ""`. +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `Aggregate` takes `(fn, column, alias)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; `CountDistinct` uses `count_distinct_`. `Aggregate` has no default; pass one or it is sent as `""`. #### `.GroupBy(...columns)` @@ -257,7 +203,7 @@ clicks.Select("page").Count("", "").GroupBy("page") clicks.Select("page").Count("", "total").OrderBy("total", "desc") ``` -`dir` defaults to `"asc"` when passed as `""`. +`dir` defaults to `"asc"` if `""`. #### `.Limit(n)` @@ -265,16 +211,11 @@ clicks.Select("page").Count("", "total").OrderBy("total", "desc") clicks.Select().Limit(100) ``` -If no limit is specified, `wavehouse.DefaultLimit` (1000) is applied -automatically to prevent unbounded result sets. The server also enforces the -configured maximum (`query.default_max_rows`, default 10,000 rows). +If unspecified, `wavehouse.DefaultLimit` (1000) is applied. The server also enforces a maximum (`query.default_max_rows`, default 10,000). #### `.TimeRange(column, since, until)` -Filter by a time window. `since` and `until` accept RFC3339 timestamps or -relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"` — day and week suffixes -expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` to leave it -open-ended. +Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for open-ended ranges. ```go clicks.Select("page").TimeRange("received_timestamp", "1h", "") @@ -285,11 +226,7 @@ clicks.Select("page").TimeRange( #### `.CacheTTL(seconds)` -Records a desired result-cache TTL on the builder. **Currently client-side -state only** — the value is never sent to the server, which derives each -result's cache TTL adaptively from query execution time. Wiring it through -the wire format is tracked in -[#280](https://github.com/Wave-RF/WaveHouse/issues/280). +Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time. See [#280](https://github.com/Wave-RF/WaveHouse/issues/280). ```go clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 @@ -297,8 +234,7 @@ clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side ### `wavehouse.FetchTyped[Row](ctx, q)` -Execute the query and decode rows into `[]Row`. Package-level generic -function (Go has no generic methods) — takes the builder as its argument. +Executes the query and decodes rows into `[]Row`. ```go type PageCount struct { @@ -314,8 +250,7 @@ page, err := wavehouse.FetchTyped[PageCount](ctx, ### `.FetchUntyped(ctx)` -Execute the query and decode rows into `[]map[string]any`. The ordinary -(non-generic) method form of `FetchTyped`. +Executes the query and decodes rows into `[]map[string]any`. ```go page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) @@ -333,9 +268,7 @@ if page.HasMore && page.Next != nil { ### `.Stream(opts)` -Open a live stream from the builder's table, applying `.Where()`/`.Select()` -filters and column projection client-side. See -[Streaming](/sdk/go/streaming). +Opens a live stream from the builder's table with client-side filtering and projection. See [Streaming](/sdk/go/streaming). ### Pagination @@ -349,30 +282,11 @@ type Page[T any] struct { } ``` -When `Limit` is set and the result contains at least that many rows, -`HasMore` is `true`. Cursor-based pagination's `Next` walks the **first** -`.OrderBy()` column — it adds a filter on that column using the last row's -value — so `Next` is only attached when the query has an explicit -`.OrderBy()`. With no order column the result still reports `HasMore` -honestly, but `Next` is `nil` (there is no deterministic cursor to build) — -add an `.OrderBy()` to paginate. If the order column was left out of an -explicit `.Select(...)` projection, `Next` quietly returns an empty page -instead of erroring (there is no cursor value to read). - -The cursor filter is strict (`gt`/`lt` against the last row's value) and uses -only that first `.OrderBy()` column, with no tie-breaker — so rows sharing the -boundary value with the last row of a page are skipped. Paginate on a column -that is unique per row (or made unique by a monotonic timestamp), or accept -that ties at a page edge can be dropped. The TypeScript SDK's `next()` has the -same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). - -One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): -rows decode into `map[string]any`, where JSON numbers become `float64`, so an -integer cursor column loses exactness past 2^53 and pagination can repeat or -skip a row at that scale — the same ceiling the TypeScript SDK has with JS -numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and -codegen structs are unaffected (their 64-bit integer columns are `int64`/ -`uint64`, decoded exactly). +If `Limit` is set and results meet that limit, `HasMore` is `true`. `Next` walks the **first** `.OrderBy()` column using a filter on the last row's value; thus, `Next` requires an explicit `.OrderBy()`. Without one, `Next` is `nil`. If the order column is omitted from `.Select(...)`, `Next` returns an empty page. + +The cursor filter is strict (`gt`/`lt` on the first `.OrderBy()` column, no tie-breaker), so rows sharing a boundary value with the last row are skipped. Paginate on a per-row-unique column, or accept dropped ties; the TypeScript SDK's `next()` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row. `FetchTyped` with an `int64` field, or codegen structs, keep it exact. ```go page, err := clicks.Select(). @@ -393,21 +307,9 @@ for page.HasMore && page.Next != nil { } ``` ---- - ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, -the token must resolve to the policy admin role (`admin_role`, `"admin"` by -default) — a JWT request with no token, or an invalid/expired one, falls -back to the `default_role`, and is rejected unless the deployment sets -`default_role` to the admin role (permitted, but dev-only). Alternatively, a configured -operator key (`Authorization: Operator ` or `X-Operator-Key`) -authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always -sends its token as `Bearer `, so to use an operator key from this -SDK supply a `Config.HTTPClient` whose `Transport` sets the -`X-Operator-Key` header on each request. Package-level generic function — -use `map[string]any` for a dynamic/unknown schema. +Execute a raw SQL query via `/v1/admin/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, @@ -426,10 +328,5 @@ typed, err := wavehouse.SQL[PageTotal](ctx, wh, ``` :::note[No parameter binding through the SDK] -Positional `?` substitution is not supported, and the SDK has no way to -forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + -`param_id=42` query-string combo) — the proxy doesn't forward arbitrary -query-string params and `SQL[Row]` doesn't expose a hook to add them. Inline -literals into the SQL, or — for safe binding from user-supplied input — use -the structured query builder (`wh.From(table)...`). +Positional `?` substitution is unsupported. The SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` lacks a hook to add them. Use inline literals or the structured query builder (`wh.From(table)...`) for safe binding of user input. ::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 6a8590d5..8a076a41 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -11,9 +11,7 @@ ships with the module. Compare with the TypeScript SDK's ## Context Cancellation -Every non-streaming operation takes a `context.Context` as its first -argument — Go's equivalent of the TypeScript SDK's `AbortSignal` support. -Cancel it with a timeout or an explicit `cancel()`: +Non-streaming operations take a `context.Context` as their first argument (similar to TypeScript's `AbortSignal`). Cancel it using a timeout or explicit `cancel()`: ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -26,35 +24,20 @@ if errors.As(err, &whErr) && whErr.Code == "ABORTED" { } ``` -Context cancellation returns immediately (no retry) with -`&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. +Cancellation returns immediately (no retry) with `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. -Streams work differently: `.Stream(opts)` doesn't take a `context.Context` -at all — the returned `*StreamController` owns its own internal context and -background goroutine, torn down explicitly via `.Close()`. See -[Streaming](/sdk/go/streaming#streamoptions). - ---- +`.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming](/sdk/go/streaming#streamoptions). ## Error Handling -The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare -`error` for operations with no result body (`Pipes.Set`/`Delete`, -`Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors -originating from the HTTP exchange are `*wavehouse.Error` (unwrap with -`errors.As`); client-side failures before a request goes out (an `Auth` -provider error, a request-body marshal failure) are plain wrapped errors, -so handle the `errors.As == false` case too. Streaming lifecycle methods -(`Stream`, `Subscribe`, `Close`) don't return `(T, error)` — stream errors -are delivered via the subscriber's `Error` callback — and `Connected(ctx)` -returns plain errors. This is the direct Go equivalent of the TypeScript -SDK's "the SDK never throws" guarantee. +The SDK never panics on API or network failures. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less operations (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. + +HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-side failures (e.g., `Auth` provider, marshal failures) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`) do not return `(T, error)`; stream errors use the subscriber's `Error` callback. `Connected(ctx)` returns plain errors. This mirrors the TypeScript SDK's "never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | +| 401 | `HTTP_401` | No | Invalid or expired JWT (missing tokens use `default_role`, resulting in success or 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | @@ -62,7 +45,7 @@ SDK's "the SDK never throws" guarantee. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | -| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | +| 0 | `SSE_ERROR` | Yes | Stream connection failure; delivered to subscriber's `Error` callback; auto-reconnects | ```go page, err := wh.From("clicks").Fetch(ctx) @@ -77,18 +60,9 @@ if err != nil { } ``` -`wavehouse.IsRetryable(err)` is a shortcut for the `errors.As` + `.Retryable` -check above. +`wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. -Retries apply uniformly to every HTTP method the SDK issues (not just GET) — -matching the TypeScript SDK's `http.ts` behavior. For `/v1/ingest`, -at-least-once delivery on retry is a documented contract (see the API -docs' ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data) -note); dedup is the prescribed server-side safety net when duplicate -suppression matters. `/v1/admin/query` (raw SQL) is gated by `admin_role`, -so repeated execution on retry is an accepted risk for admin-only usage. - ---- +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/admin/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. ## Full API Tree @@ -143,8 +117,7 @@ NewClient(Config) → *Client ## Codegen CLI -Generate Go structs from a running WaveHouse instance. The module ships a -`wavehouse-codegen` command under `cmd/`: +Generate Go structs from a running WaveHouse instance using the `wavehouse-codegen` command in `cmd/`: ```bash export WAVEHOUSE_AUTH='' # avoids leaking the token via argv @@ -154,16 +127,13 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --package myapp ``` -Or, working inside this repo (`clients/go/`): +Or, inside `clients/go/`: ```bash go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` -Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev -server, provide an admin-role token or the request is denied with `403`. -Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with -`--auth ` ends up in shell history and process listings. +Codegen reads the admin-only `/v1/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. **Options:** @@ -171,13 +141,11 @@ Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with |------|-------------|---------| | `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | | `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token (if auth required); prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | +| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | | `--package`, `-p` | Go package name for the generated file | `main` | | `--help`, `-h` | Show usage and exit | — | -The output is run through `go/format` before being written — if a table or -column name would produce invalid Go source (rare, but possible with exotic -names), codegen fails loudly instead of writing broken code. +Output is processed via `go/format`. If a table or column name produces invalid Go source, codegen fails loudly. **Example output:** @@ -195,22 +163,9 @@ type ClicksRow struct { } ``` -(That's the exact output for the `clicks` table from the -[development quick-start](/development#quick-start) — `received_timestamp` -becomes `*string` + `,omitempty` because it has a `DEFAULT` clause.) - -Note the generator does **not** special-case initialisms: `event_id` becomes -`EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply -gets its first letter upper-cased. +(Example for the [development quick-start](/development#quick-start) `clicks` table; `received_timestamp` is `*string` + `,omitempty` due to its `DEFAULT` clause.) -Table and column names are converted to `PascalCase` for Go field/type names -(a leading digit gets an `X` prefix — e.g. a table named `2fa_events` -becomes `X2faEventsRow` — to stay a valid Go identifier). A column with -`has_default: true` in the schema becomes a **pointer field** with -`,omitempty` — the Go spelling of the TS codegen's `field?: T`: leave it -`nil` to omit the field (the server default applies), or point it at a -value to send it — including an explicit `0`/`false`/`""`, which a plain -value field with `omitempty` would silently drop. +The generator does not special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part just gets its first letter upper-cased. Table and column names are converted to `PascalCase`; leading digits get an `X` prefix (e.g., `2fa_events` $\rightarrow$ `X2faEventsRow`). Columns with `has_default: true` become pointer fields with `,omitempty`: `nil` uses the server default, a pointed-at value is sent — including an explicit `0`/`false`/`""`. **ClickHouse → Go type mapping:** @@ -222,55 +177,30 @@ value field with `omitempty` would silently drop. | `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | | `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | -| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` (arbitrary-width unquoted numbers on the structured-query path) | -| `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` | +| `Decimal*` | `string` | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` — except `Array(UInt8)` → `json.RawMessage`: the wire is asymmetric (ingest takes a JSON array, but query responses currently base64-encode the column), and `RawMessage` is the one shape that decodes both; server-side normalization tracked in [#436](https://github.com/Wave-RF/WaveHouse/issues/436) | -| `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| `Array(T)` | `[]T` (except `Array(UInt8)` $\rightarrow$ `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | -This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** (`UInt64` → `uint64`, not -a generic `number`), since Go — unlike TypeScript — has native fixed-width -integer types; 64-bit columns decode exactly where TS hits the JS-number -2^53 ceiling. Generated structs target the structured-query and pipe paths -(`/v1/query`, `/v1/pipes/*`), where the server re-marshals values as plain -JSON numbers. The raw-SQL path (`/v1/admin/query`) instead forwards -ClickHouse's own JSON, which **quotes** 64-bit-and-wider integers — use -`map[string]any` with `SQL[Row]` there rather than generated structs. +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/admin/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. ## Testing -The Go SDK ships with unit tests colocated in `clients/go/` (its own Go -module — `clients/go/go.mod` — separate from the root `WaveHouse` module), -plus the Go half of the cross-language wire-format **conformance suite**: -`clients/go/conformance_test.go` replays the shared fixture -(`clients/go/testdata/wire_cases.json`) and asserts the Go SDK produces the -expected HTTP method, path, content type, and body for each case. The -TypeScript half — `tests/conformance/conformance_ts.mjs`, run with -`make test-conformance-ts` (it builds the TS SDK first) — replays the same -fixture, and CI runs both, keeping the two clients honest about the wire -format they both speak. +Unit tests are colocated in `clients/go/` (module `clients/go/go.mod`), separate from the root `WaveHouse` module. The cross-language wire-format **conformance suite** uses `clients/go/conformance_test.go` to replay the shared fixture (`clients/go/testdata/wire_cases.json`), asserting correct HTTP methods, paths, content types, and bodies. The TypeScript half—`tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` (builds TS SDK first)—uses the same fixture; CI runs both to ensure wire format consistency. ```bash cd clients/go go test ./... ``` -E2E tests (build tag `e2e`) run against a live WaveHouse instance and have -their own Make target, separate from the repo's `make test-e2e`: +E2E tests (build tag `e2e`) run against a live WaveHouse instance via a dedicated Make target: ```bash WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e ``` -`WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is -optional (admin-only cases skip without it). When the server is unreachable -the suite skips instead of failing. - -Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's -`make test-e2e` harness — see the TypeScript SDK's -[E2E Testing](/sdk/reference#e2e-testing) section for that suite's -architecture, which the Go client doesn't currently participate in. +`WAVEHOUSE_URL` defaults to `http://localhost:8080`; optional `WAVEHOUSE_AUTH` is for admin cases. The suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index c105f28c..c16033b6 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -14,15 +14,11 @@ a `context.Context` or a browser's `EventSource`. ## Streaming -Streams use SSE (Server-Sent Events), parsed by hand over `net/http` (no -third-party SSE library — the SDK has zero runtime dependencies). +Streams use SSE (Server-Sent Events) parsed via `net/http` with zero runtime dependencies. ### `*StreamController` -Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and -`*DLQNamespace` (the DLQ variant is not yet functional server-side — -[#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` -returns immediately; the connection opens in a background goroutine. +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` returns immediately; the connection opens in a background goroutine. ```go stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ @@ -33,8 +29,7 @@ defer stream.Close() ### `.Subscribe(sub) → func()` -Callback-based consumption. Returns an unsubscribe function. The -subscriber's `Status` callback fires immediately with the current status. +Callback-based consumption. Returns an unsubscribe function. The `Status` callback fires immediately with the current status. ```go unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ @@ -57,10 +52,11 @@ unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ defer unsub() ``` +Cleanup via `unsub()` removes the subscriber; the connection remains open for others and must be closed with `stream.Close()`. + ### Channel-based consumption — `.Events()` -The idiomatic Go alternative to the TypeScript SDK's async iterator: a -read-only channel, closed automatically when the stream shuts down. +A read-only channel, closed automatically when the stream shuts down. ```go stream := wh.From("clicks").Stream(nil) @@ -75,26 +71,14 @@ for event := range stream.Events() { ``` :::caution[`break` does not close the stream] -Unlike the TypeScript SDK's async iterator — where breaking out of a -`for await` loop auto-closes the underlying connection — breaking a Go -`for range stream.Events()` loop only stops consuming from the channel; the -background goroutine and its HTTP connection keep running. Always pair a -stream with `defer stream.Close()` (or an explicit `stream.Close()` on every -exit path) regardless of which consumption style you use. +Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. ::: -The channel is buffered (256 events); a slow consumer that never drains it -causes the SDK to **drop** new events for that channel rather than block the -stream's read loop — the first drop logs one line via the standard `log` -package, further drops are silent (`.Subscribe` callbacks still fire per -event regardless of channel backpressure). +The channel is buffered (256 events). A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop. The first drop logs via `log`; later drops are silent (`.Subscribe` callbacks fire regardless). ### `.Close()` -Explicitly close the stream and release its resources. Non-blocking — safe -to call from inside a subscriber callback (which runs on the stream's own -goroutine); it signals the goroutine to stop without waiting for it to -finish. +Explicitly closes the stream and releases resources. Non-blocking and safe to call from inside a subscriber callback. ```go stream.Close() @@ -102,8 +86,7 @@ stream.Close() ### `.Status()` -Returns the current `StreamStatus`. A method (not a field), since Go has no -JS-style reactive property access. +Returns the current `StreamStatus`. ```go status := stream.Status() @@ -111,11 +94,7 @@ status := stream.Status() ### `.Connected(ctx)` -**Go-only addition** — not present in the TypeScript SDK. Blocks until the -stream reaches `StatusLive` or `ctx` is canceled; returns an error if the -stream closes before connecting. Useful when you need to know a stream is -live before doing something else (e.g. before starting a producer in a -test). +Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (e.g., in tests). ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -131,8 +110,7 @@ if err := stream.Connected(ctx); err != nil { | ----- | ---- | ----------- | | `Since` | `string` | RFC3339 timestamp for gap-fill replay | -There's no `Signal`/context field here — a stream isn't canceled by passing -a `context.Context` into `.Stream()`; call `.Close()` instead (see above). +There's no `Signal`/context field: a stream isn't canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. ### `StreamEvent` @@ -145,48 +123,26 @@ type StreamEvent struct { ``` :::note[`Events()` carries events only] -`Error` and `Status` are delivered exclusively through `.Subscribe(...)` — -the channel is typed `chan StreamEvent` and simply ends (closes) when the -stream closes, including on a terminal 401/403/404. Pair `Events()` with a -`Subscribe(&StreamSubscriber{Error: ..., Status: ...})` if you need to know -*why* a stream ended. +`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. ::: :::note[The channel buffers from stream construction] -Events buffer into the channel (up to 256) from the moment `.Stream()` -constructs the stream, matching the TypeScript SDK — events arriving before -your first `Events()` call are **not** lost, so you don't have to call -`Events()` immediately. A consumer that never drains the channel still -drops everything past the 256th buffered event (with the one-time log line -described above). +Events buffer (up to 256) starting at `.Stream()`; events arriving before the first `Events()` call are not lost. ::: ### Transport Behavior | Transport | Reconnect | Protocol | | --------- | --------- | -------- | -| SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | - -Reconnect covers transport failures and retryable (5xx/429) responses. A -non-retryable response (401/403/404) is terminal: the error is delivered to -the subscriber's `Error` callback, status goes to `StatusClosed`, and the -stream does not reconnect — fix the cause (refresh the token, correct the -table) and open a new stream. An `Auth` provider error during a (re)connect -is treated as retryable (`SSE_ERROR`) and the stream keeps reconnecting — -`ClientOptions.MaxRetries` bounds request retries only, not stream -reconnects — so call `.Close()` if your token provider is failing -permanently. - -Auth is sent as an `Authorization: Bearer` header on every stream -(re)connection — see -[the note in the Getting Started guide](/sdk/go#creating-a-client). The -TypeScript SDK's "more than 5 concurrent connections" warning is a -browser-specific `EventSource` limit and doesn't apply here. +| SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | + +Reconnect covers transport failures and retryable (5xx/429) responses. Non-retryable ones (401/403/404) are terminal: the `Error` callback fires, status goes `StatusClosed`, no reconnect. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. + +Auth goes as an `Authorization: Bearer` header on every connection ([note in Getting Started](/sdk/go#creating-a-client)). Browser `EventSource` limits don't apply. ### Client-Side Stream Filtering -When a `*QueryBuilder` with `.Where()` filters or `.Select()` columns calls -`.Stream()`, the returned stream applies those filters client-side: +When a `*QueryBuilder` with `.Where()` or `.Select()` calls `.Stream()`, filters are applied client-side: ```go stream := wh.From("clicks"). @@ -197,22 +153,11 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, -`OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes -everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). -`OpLike` / `OpNotLike` match SQL LIKE semantics (`%` → any run of -characters, `_` → any single character), case-insensitively. `OpIn` accepts -any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, -...), not just `[]any`. - ---- +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). ## Live Queries -Live queries combine a historical backfill (`.FetchUntyped`) with a -real-time stream, providing a seamless initial-load + live-updates -experience. Only available on `*QueryBuilder` (there's no `TableRef.LiveQuery` -shortcut, matching the TypeScript SDK). +Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. ```go lq := wh.From("clicks"). @@ -254,51 +199,31 @@ type StreamSubscriber struct { ``` :::note[`Initial` is always untyped] -Unlike the TypeScript SDK's `initial: (result: Result) => void`, the Go -SDK's `LiveQuery` doesn't accept a type parameter — `Initial` always -receives `[]map[string]any` plus a plain `error`, even if you'd otherwise -use `wavehouse.FetchTyped[Row]` for the same query outside a live query. -Decode into your own type inside the callback if you need one. +Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even if you'd use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. ::: ### How it works -1. Subscribes to the stream **immediately** and buffers incoming events. -2. Runs the `.FetchUntyped(ctx)` query for historical data, calls - `sub.Initial(rows, err)` with the result. -3. Deduplicates buffered events against the **newest** `received_timestamp` - in the backfill — the maximum across all rows, not the last row's (an - `OrderBy(..., "desc")` puts the *oldest* row last). -4. Flushes remaining buffered events (re-checking for anything that arrived - mid-flush) and switches to live mode. +1. Subscribes to the stream immediately and buffers events. +2. Runs `.FetchUntyped(ctx)` for historical data, then calls `sub.Initial(rows, err)`. +3. Deduplicates buffered events against the maximum `received_timestamp` in the backfill (not necessarily the last row). +4. Flushes remaining buffered events and switches to live mode. -This "stream-first" approach ensures no events are lost between the fetch -and stream start. +This "stream-first" approach prevents event loss between fetch and stream start. :::caution[Dedup needs `received_timestamp` in the projection] -The dedup bound comes from the backfill rows' `received_timestamp` values. -`.SelectAll()` (or no projection) includes it; a `.Select(...)` projection -that omits it disables dedup, and events in the fetch/stream overlap window -are delivered twice — once in `Initial`, again via `Next`. +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, causing events in the overlap window to be delivered twice (via `Initial` and `Next`). ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` matching is case-**insensitive**, but the backfill runs -server-side where the operator compiles to ClickHouse `LIKE`, which is -case-**sensitive**. A live query filtering on `OpLike` can therefore -disagree with itself: the backfill excludes rows the live stream includes. -Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). - -`OpNotLike` never reaches the backfill at all — `/v1/query` rejects the -operator with a `400`, so a live query filtering on it fails its `Initial` -callback. See the operator table in -[Queries](/sdk/go/queries#wherecolumn-op-value). +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive. Consequently, a live query filtering on `OpLike` may exclude rows in the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). + +`OpNotLike` is rejected by `/v1/query` with a `400`, causing `Initial` callbacks to fail. See [Queries](/sdk/go/queries#wherecolumn-op-value). ::: ### `.Close()` -Shuts down the live query and its underlying stream. Safe to call more than -once (idempotent via `sync.Once`). +Shuts down the live query and its underlying stream. Safe to call more than once (idempotent via `sync.Once`). ```go lq.Close() From 59b27bbff7fd028fa062f67f439accb6cfc0e1cb Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:01:25 -0400 Subject: [PATCH 32/59] fix(sdk): bring the Go SDK current with main's API and SSE changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three catch-up changes, all client-side. The server is untouched. Routes: main merged every admin-gated endpoint under /v1/ops (#479) with no aliases, so thirteen call sites were 404ing against a current server — schema list/refresh, DLQ stats, raw SQL, policy get/put/validate, and pipes CRUD, plus the codegen CLI's schema fetch. Rewrote them along with the tests, the shared wire_cases.json fixture, and the Go SDK docs. The fixture is replayed by both conformance runners, so the stale paths broke the TypeScript half too; `make test-conformance-ts` is back to 45/45. ClientOptions.Headers: the TypeScript SDK gained options.headers in #456 and Go had no equivalent. Headers now apply to every request the client makes, REST and SSE alike — which is also how an operator sends the server's non-JWT X-Operator-Key. The SDK's own headers are set afterwards and win a collision; net/http canonicalizes names, so matching is case-insensitive; the map is copied at construction so later mutation can't reach into requests. SSE robustness, mirroring main's fetch-based rewrite (#470). The Go SDK already authenticated by header, so that part was never stale, but three gaps were: - A credentialed stream followed redirects. net/http drops Authorization on a cross-host hop while forwarding custom headers verbatim, so a redirect either downgraded the stream to default_role in silence or handed configured secrets to wherever it pointed. Now refused with a terminal SSE_REDIRECT. Uncredentialed streams still follow. - A 200 with any content type was treated as an event stream, so an auth gateway's login page left the stream sitting in StatusLive delivering nothing. Now a terminal SSE_BAD_CONTENT_TYPE. - Every failure collapsed into one retryable SSE_ERROR, and malformed frames came back as a bare fmt.Errorf, so errors.As and IsRetryable didn't work on them. Replaced with the taxonomy the TypeScript SDK uses — SSE_AUTH_ERROR, SSE_NETWORK_ERROR, SSE_CONNECT_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, SSE_PARSE_ERROR, SSE_READ_ERROR — each with its own retryable flag, all delivered as *Error. Also documents what main changed underneath the Go SDK without changing its code: DateTime values arrive canonicalized to RFC 3339 UTC (#402), SSE applies policy row-filters per subscriber and fails closed (#381, #457), /v1/stream is ungated so WaveHouse never 401s a stream, and /v1/ops/dlq/stats is absent (404) when the DLQ is disabled rather than returning empty stats. Tests: terminal-failure table (bad content type, missing content type, credentialed redirect, non-HTTP scheme), redirect-followed-when- uncredentialed, typed retryable parse errors, and header precedence and copying on both transports. --- clients/go/client_test.go | 4 +- clients/go/cmd/wavehouse-codegen/main.go | 12 +- clients/go/conformance_test.go | 12 +- clients/go/dlq.go | 7 +- clients/go/http.go | 12 ++ clients/go/http_test.go | 101 +++++++++++- clients/go/namespaces_test.go | 6 +- clients/go/pipes.go | 8 +- clients/go/policy.go | 6 +- clients/go/schema.go | 4 +- clients/go/stream.go | 142 ++++++++++++++--- clients/go/stream_test.go | 186 ++++++++++++++++++++++ clients/go/table.go | 2 +- clients/go/testdata/wire_cases.json | 24 +-- clients/go/wavehouse.go | 22 ++- docs/src/content/docs/sdk/go/admin.md | 4 +- docs/src/content/docs/sdk/go/index.md | 24 ++- docs/src/content/docs/sdk/go/pipes.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 2 +- docs/src/content/docs/sdk/go/reference.md | 15 +- docs/src/content/docs/sdk/go/streaming.md | 16 +- tests/conformance/conformance_ts.mjs | 12 +- 22 files changed, 545 insertions(+), 78 deletions(-) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index aca92d0d..2ba55833 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -76,8 +76,8 @@ func TestClient_From(t *testing.T) { func TestClient_SQL(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/admin/query" { - t.Errorf("want /v1/admin/query, got %s", r.URL.Path) + if r.URL.Path != "/v1/ops/query" { + t.Errorf("want /v1/ops/query, got %s", r.URL.Path) } var body map[string]string _ = json.NewDecoder(r.Body).Decode(&body) diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 9876ee07..9fa68258 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -1,4 +1,4 @@ -// Command wavehouse-codegen reads a WaveHouse server's /v1/schema endpoint +// Command wavehouse-codegen reads a WaveHouse server's /v1/ops/schema endpoint // and generates Go struct definitions for use with the wavehouse SDK. // // Usage: @@ -56,7 +56,7 @@ func parseArgs() cliArgs { Options: --url, -u WaveHouse base URL (default: http://localhost:8080) --out, -o Output .go file path (default: ./wavehouse_types.go) - --auth, -a Bearer token for authenticated /v1/schema endpoint + --auth, -a Bearer token for authenticated /v1/ops/schema endpoint (prefer the WAVEHOUSE_AUTH env var — argv leaks into shell history and process listings) --package, -p Go package name (default: main) @@ -85,7 +85,7 @@ type tableSchema struct { } func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { - url := strings.TrimRight(baseURL, "/") + "/v1/schema" + url := strings.TrimRight(baseURL, "/") + "/v1/ops/schema" req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return nil, fmt.Errorf("build schema request for %s: %w", url, err) @@ -124,7 +124,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc return m, nil } -// chTypeToGo maps a ClickHouse type string (as reported by /v1/schema) to a +// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to a // Go type name suitable for a JSON struct field. // // We deliberately don't import clickhouse-go's type catalog @@ -135,7 +135,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc // types the *driver* scans query results into over the native protocol // (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for // Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types -// that round-trip cleanly through the JSON the /v1/schema and query +// that round-trip cleanly through the JSON the /v1/ops/schema and query // endpoints actually speak. ClickHouse's JSON output renders DateTime as // "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default // time.Time JSON unmarshaling; big integers and decimals are similarly @@ -188,7 +188,7 @@ func chTypeToGo(chType string) string { // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse // values into Go types and re-marshals them — so 64-bit integers arrive // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. - // (Only /v1/admin/query forwards ClickHouse's own JSON, which quotes + // (Only /v1/ops/query forwards ClickHouse's own JSON, which quotes // 64-bit ints; use map[string]any with SQL[Row] there.) if mapped, ok := map[string]string{ "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index e90f8412..c8af2899 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -92,17 +92,17 @@ func TestConformance_WireFormat(t *testing.T) { // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") switch { - case strings.HasPrefix(r.URL.Path, "/v1/dlq"): + case strings.HasPrefix(r.URL.Path, "/v1/ops/dlq"): _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) - case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/schema") && r.Method == "GET": _ = json.NewEncoder(w).Encode([]TableSchema{}) - case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": + case r.URL.Path == "/v1/ops/policy/validate" && r.Method == "POST": _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) - case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/policy") && r.Method == "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) - case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/pipes/") && r.Method == "GET": _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) - case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": + case r.URL.Path == "/v1/ops/pipes" && r.Method == "GET": _ = json.NewEncoder(w).Encode([]Pipe{}) default: _ = json.NewEncoder(w).Encode([]map[string]any{}) diff --git a/clients/go/dlq.go b/clients/go/dlq.go index 0009e4e3..65712765 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -7,6 +7,11 @@ import ( ) // DLQNamespace provides admin-only dead-letter-queue statistics. +// +// The server registers /v1/ops/dlq/stats only when the DLQ is enabled, so on a +// deployment with dlq.enabled: false these calls return an [*Error] with +// Status 404 — "the DLQ is switched off", not "the DLQ is empty". Check +// Status before reading a zero DLQStats as a healthy result. type DLQNamespace struct { ctx httpContext createStream func(table string, opts *StreamOptions) *StreamController @@ -26,7 +31,7 @@ func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, var stats DLQStats if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", - path: "/v1/dlq/stats", + path: "/v1/ops/dlq/stats", params: params, }, &stats); err != nil { return nil, fmt.Errorf("get dlq stats: %w", err) diff --git a/clients/go/http.go b/clients/go/http.go index 9d149e90..eaab0e94 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -26,6 +26,17 @@ type httpContext struct { auth func(ctx context.Context) (string, error) maxRetries int httpClient *http.Client + headers map[string]string +} + +// applyConfiguredHeaders writes the client's configured headers onto a request. +// Call it *before* the SDK sets its own headers: Set replaces, so whatever the +// SDK writes afterwards wins a collision. http.Header canonicalizes names, so +// "x-tenant" and "X-Tenant" are the same entry. +func applyConfiguredHeaders(h http.Header, configured map[string]string) { + for k, v := range configured { + h.Set(k, v) + } } // requestOptions describes a single HTTP request. @@ -86,6 +97,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a if err != nil { return fmt.Errorf("wavehouse: build request: %w", err) } + applyConfiguredHeaders(req.Header, hctx.headers) req.Header.Set("Content-Type", ct) req.Header.Set("Accept", "application/json") if authHeader != "" { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 57109f3b..52212c16 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -113,7 +113,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, nil) if err != nil { t.Fatal(err) @@ -135,7 +135,7 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, nil) if !errIs(err, "HTTP_404") { @@ -199,7 +199,7 @@ func TestDoRequest_EmptyResponse(t *testing.T) { var result map[string]string err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", - path: "/v1/schema/refresh", + path: "/v1/ops/schema/refresh", }, &result) if err != nil { t.Fatal(err) @@ -335,3 +335,98 @@ func TestBaseURLPathPrefixIsPreserved(t *testing.T) { } } } + +// TestConfiguredHeadersOnRESTRequests: ClientOptions.Headers apply to every +// REST call, are matched case-insensitively, and always lose to the SDK's own +// headers rather than appending alongside them. +func TestConfiguredHeadersOnRESTRequests(t *testing.T) { + tests := []struct { + name string + configured map[string]string + auth func(context.Context) (string, error) + header string + want string + }{ + { + name: "custom header is forwarded", + configured: map[string]string{"X-Operator-Key": "op-secret"}, + header: "X-Operator-Key", + want: "op-secret", + }, + { + name: "name matching is case-insensitive", + configured: map[string]string{"x-tenant-id": "acme"}, + header: "X-Tenant-Id", + want: "acme", + }, + { + name: "SDK Accept outranks a configured one", + configured: map[string]string{"Accept": "text/plain"}, + header: "Accept", + want: "application/json", + }, + { + name: "SDK Authorization outranks a configured one", + configured: map[string]string{"Authorization": "Bearer configured"}, + auth: StaticToken("real-token"), + header: "Authorization", + want: "Bearer real-token", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + client := NewClient(Config{ + BaseURL: srv.URL, + Auth: tc.auth, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: tc.configured}, + }) + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Values(tc.header); len(v) != 1 { + t.Fatalf("want exactly one %s header, got %v", tc.header, v) + } + if v := got.Get(tc.header); v != tc.want { + t.Fatalf("want %s: %q, got %q", tc.header, tc.want, v) + } + }) + } +} + +// TestConfiguredHeadersAreCopied: mutating the caller's map after NewClient +// must not change what later requests send. +func TestConfiguredHeadersAreCopied(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + headers := map[string]string{"X-Tenant-Id": "acme"} + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: headers}, + }) + headers["X-Tenant-Id"] = "attacker" + delete(headers, "X-Tenant-Id") + + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Get("X-Tenant-Id"); v != "acme" { + t.Fatalf("want the value captured at construction, got %q", v) + } +} diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index e7340b6f..27453e6b 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -23,8 +23,8 @@ func TestSysNamespace_Health(t *testing.T) { func TestSchemaNamespace_List(t *testing.T) { c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/schema" { - t.Errorf("want /v1/schema, got %s", r.URL.Path) + if r.URL.Path != "/v1/ops/schema" { + t.Errorf("want /v1/ops/schema, got %s", r.URL.Path) } _ = json.NewEncoder(w).Encode([]TableSchema{ {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, @@ -132,7 +132,7 @@ func TestPipesNamespace_CRUD(t *testing.T) { c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": - if r.URL.Path == "/v1/admin/pipes" { + if r.URL.Path == "/v1/ops/pipes" { _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) } else { _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) diff --git a/clients/go/pipes.go b/clients/go/pipes.go index 71b25720..5501fa86 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -16,7 +16,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { var pipes []Pipe if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/pipes", + path: "/v1/ops/pipes", }, &pipes); err != nil { return nil, fmt.Errorf("list pipes: %w", err) } @@ -28,7 +28,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { var pipe Pipe if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), }, &pipe); err != nil { return nil, fmt.Errorf("get pipe %q: %w", name, err) } @@ -39,7 +39,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), body: def, }, nil); err != nil { return fmt.Errorf("set pipe %q: %w", name, err) @@ -51,7 +51,7 @@ func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) erro func (p *PipesNamespace) Delete(ctx context.Context, name string) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), }, nil); err != nil { return fmt.Errorf("delete pipe %q: %w", name, err) } diff --git a/clients/go/policy.go b/clients/go/policy.go index 9fbc5bb6..7e356bee 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -15,7 +15,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { var pol Policy if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/policy", + path: "/v1/ops/policy", }, &pol); err != nil { return nil, fmt.Errorf("get policy: %w", err) } @@ -26,7 +26,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", - path: "/v1/admin/policy", + path: "/v1/ops/policy", body: pol, }, nil); err != nil { return fmt.Errorf("set policy: %w", err) @@ -39,7 +39,7 @@ func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*Validatio var result ValidationResult if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", - path: "/v1/admin/policy/validate", + path: "/v1/ops/policy/validate", body: pol, }, &result); err != nil { return nil, fmt.Errorf("validate policy: %w", err) diff --git a/clients/go/schema.go b/clients/go/schema.go index 3f5b4a88..c79dcb11 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -16,7 +16,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { var raw []TableSchema if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, &raw); err != nil { return nil, fmt.Errorf("list schemas: %w", err) } @@ -31,7 +31,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { func (s *SchemaNamespace) Refresh(ctx context.Context) error { if err := doRequest(ctx, s.ctx, requestOptions{ method: "POST", - path: "/v1/schema/refresh", + path: "/v1/ops/schema/refresh", }, nil); err != nil { return fmt.Errorf("refresh schema: %w", err) } diff --git a/clients/go/stream.go b/clients/go/stream.go index 8307e2fb..48ce32e8 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" "log" + "mime" "net/http" "net/url" "reflect" @@ -246,21 +247,28 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } if err != nil { - // A non-retryable API error (401/403/404, ...) is terminal: - // reconnecting can't fix a bad token or a missing table, and the - // TS SDK's EventSource likewise ends up closed on a non-200. - // Emit it and exit — the deferred cleanup sets StatusClosed. + // connect classifies its own failures (SSE_AUTH_ERROR, + // SSE_NETWORK_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, + // SSE_READ_ERROR, HTTP_nnn), so pass the typed error straight + // through and let Retryable decide whether to reconnect. A + // non-retryable error is terminal: reconnecting can't fix a bad + // token, a missing table, or a proxy answering with HTML. var apiErr *Error - if errors.As(err, &apiErr) && !apiErr.Retryable { + if errors.As(err, &apiErr) { sc.emitError(apiErr) - return + if !apiErr.Retryable { + return + } + } else { + // Unclassified — retry, but keep the generic code so callers + // can still match on it. + sc.emitError(&Error{ + Status: 0, + Code: "SSE_ERROR", + Message: err.Error(), + Retryable: true, + }) } - sc.emitError(&Error{ - Status: 0, - Code: "SSE_ERROR", - Message: err.Error(), - Retryable: true, - }) } sc.setStatus(StatusReconnecting) @@ -281,7 +289,22 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { u, err := url.Parse(hctx.baseURL + "/v1/stream") if err != nil { - return "", false, err + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("invalid baseURL: %v", err), + Retryable: false, + } + } + // A non-HTTP scheme can never carry SSE. Terminal, not retryable: retrying + // a ws:// or file:// baseURL just spins. + if u.Scheme != "http" && u.Scheme != "https" { + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("baseURL scheme %q is not http or https", u.Scheme), + Retryable: false, + } } q := u.Query() q.Set("table", table) @@ -294,7 +317,14 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if hctx.auth != nil { token, err := hctx.auth(ctx) if err != nil { - return "", false, fmt.Errorf("auth: %w", err) + // Retryable: a token endpoint having a bad minute shouldn't tear + // down a healthy long-lived stream. + return "", false, &Error{ + Status: 0, + Code: "SSE_AUTH_ERROR", + Message: err.Error(), + Retryable: true, + } } if token != "" { authHeader = "Bearer " + token @@ -307,22 +337,75 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if err != nil { return "", false, err } + applyConfiguredHeaders(req.Header, hctx.headers) req.Header.Set("Accept", "text/event-stream") req.Header.Set("Cache-Control", "no-cache") if authHeader != "" { req.Header.Set("Authorization", authHeader) } - resp, err := hctx.httpClient.Do(req) + client := hctx.httpClient + credentialed := authHeader != "" || len(hctx.headers) > 0 + if credentialed { + // Refuse to follow a redirect while carrying a credential. net/http + // drops Authorization on a cross-host hop but forwards custom headers + // verbatim, so following one would either downgrade the stream to + // default_role without saying so, or hand configured secrets to + // wherever the redirect points. Copy the client so a caller-supplied + // one keeps its own CheckRedirect for every other request. + c := *client + c.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + client = &c + } + + resp, err := client.Do(req) if err != nil { - return "", false, err + if ctx.Err() != nil { + return "", false, errAborted + } + return "", false, &Error{ + Status: 0, + Code: "SSE_NETWORK_ERROR", + Message: err.Error(), + Retryable: true, + } } defer func() { _ = resp.Body.Close() }() + if credentialed && resp.StatusCode >= 300 && resp.StatusCode < 400 { + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_REDIRECT", + Message: fmt.Sprintf( + "stream endpoint redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", + resp.Header.Get("Location")), + Retryable: false, + } + } + if resp.StatusCode != http.StatusOK { return "", false, parseErrorResponse(resp) } + // A 200 that isn't an event stream means something between the caller and + // WaveHouse answered — a captive portal or an auth gateway's login page. + // Without this check the stream sits in StatusLive and silently delivers + // nothing. + if ct := resp.Header.Get("Content-Type"); !isEventStream(ct) { + shown := ct + if shown == "" { + shown = "(none)" + } + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_BAD_CONTENT_TYPE", + Message: fmt.Sprintf("expected Content-Type text/event-stream, got %s", shown), + Retryable: false, + } + } + sc.setStatus(StatusLive) // Parse SSE frames. @@ -372,7 +455,25 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table } } - return lastID, true, scanner.Err() + if scanErr := scanner.Err(); scanErr != nil { + return lastID, true, &Error{ + Status: 0, + Code: "SSE_READ_ERROR", + Message: scanErr.Error(), + Retryable: true, + } + } + return lastID, true, nil +} + +// isEventStream reports whether a Content-Type header names text/event-stream, +// ignoring any parameters (charset, boundary) and case. +func isEventStream(contentType string) bool { + mediaType, _, err := mime.ParseMediaType(contentType) + if err != nil { + return false + } + return mediaType == "text/event-stream" } // sseMessage matches the server's SSE event JSON shape. @@ -389,7 +490,12 @@ func (sc *StreamController) handleSSEData(data string) { // process-global logger, so consumers control visibility and a // malformed-frame flood can't spam host-application logs. Payload // deliberately omitted: event data can carry tenant/PII fields. - sc.emitError(fmt.Errorf("malformed SSE message (%d bytes): %w", len(data), err)) + sc.emitError(&Error{ + Status: 0, + Code: "SSE_PARSE_ERROR", + Message: fmt.Sprintf("malformed SSE message (%d bytes): %v", len(data), err), + Retryable: true, + }) return } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index efe10fd2..c65bb863 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -466,3 +466,189 @@ func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { t.Fatal("stream never reached the prefixed path") } } + +// TestStream_TerminalConnectFailures: every way a connection can fail in a way +// reconnecting cannot fix. Each case must surface a specific, non-retryable +// code and close the stream — the generic retryable SSE_ERROR would spin here. +func TestStream_TerminalConnectFailures(t *testing.T) { + tests := []struct { + name string + handler http.HandlerFunc + baseURL string // overrides the test server URL when non-empty + auth func(context.Context) (string, error) + wantCode string + }{ + { + name: "200 that is not an event stream", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(http.StatusOK) + _, _ = io.WriteString(w, "Please sign in") + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "200 with no Content-Type at all", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + w.WriteHeader(http.StatusOK) + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "credentialed request is redirected", + handler: func(w http.ResponseWriter, _ *http.Request) { + http.Redirect(w, &http.Request{}, "https://elsewhere.example/v1/stream", http.StatusFound) + }, + auth: StaticToken("secret-token"), + wantCode: "SSE_REDIRECT", + }, + { + name: "baseURL scheme cannot carry SSE", + handler: func(http.ResponseWriter, *http.Request) {}, + baseURL: "ws://example.invalid", + wantCode: "SSE_CONNECT_ERROR", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(tc.handler) + t.Cleanup(srv.Close) + + base := srv.URL + if tc.baseURL != "" { + base = tc.baseURL + } + client := NewClient(Config{BaseURL: base, Auth: tc.auth, HTTPClient: srv.Client()}) + + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != tc.wantCode { + t.Fatalf("want code %s, got %s (%v)", tc.wantCode, apiErr.Code, err) + } + if apiErr.Retryable { + t.Fatalf("%s must not be retryable", apiErr.Code) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatalf("stream never closed after terminal %s", tc.wantCode) + } + }) + } +} + +// TestStream_RedirectFollowedWhenUncredentialed: the refusal is scoped to +// requests carrying a credential. Without one there is nothing to leak or +// silently downgrade, so the redirect is followed as usual. +func TestStream_RedirectFollowedWhenUncredentialed(t *testing.T) { + target := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + + front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/v1/stream?table=clicks", http.StatusFound) + })) + t.Cleanup(front.Close) + + stream := streamClient(t, front).From("clicks").Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Table != "clicks" { + t.Fatalf("want table clicks, got %s", e.Table) + } + case <-time.After(5 * time.Second): + t.Fatal("redirect was not followed for an uncredentialed stream") + } +} + +// TestStream_MalformedFrameIsTypedAndRetryable: a bad frame must arrive as an +// *Error so errors.As and IsRetryable work on it — a bare fmt.Errorf would +// leave callers string-matching. +func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { + srv := sseServer(t, []string{"id: 1\ndata: {not json\n\n"}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != "SSE_PARSE_ERROR" { + t.Fatalf("want SSE_PARSE_ERROR, got %s", apiErr.Code) + } + if !IsRetryable(err) { + t.Fatal("a malformed frame must stay retryable") + } + if strings.Contains(apiErr.Message, "not json") { + t.Fatal("payload must not be echoed into the error message") + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } +} + +// TestStream_ConfiguredHeadersReachTheStream: ClientOptions.Headers apply to +// SSE, not just REST — and the SDK's own headers still win a collision. +func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { + seen := make(chan http.Header, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case seen <- r.Header.Clone(): + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + if fl, ok := w.(http.Flusher); ok { + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: map[string]string{ + "X-Operator-Key": "op-secret", + "accept": "application/json", // must lose to the SDK's own Accept + }}, + }) + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + select { + case h := <-seen: + if got := h.Get("X-Operator-Key"); got != "op-secret" { + t.Fatalf("want configured header on the stream request, got %q", got) + } + if got := h.Get("Accept"); got != "text/event-stream" { + t.Fatalf("SDK Accept must win, got %q", got) + } + case <-time.After(5 * time.Second): + t.Fatal("stream request never arrived") + } +} diff --git a/clients/go/table.go b/clients/go/table.go index e66f7f05..73c2ea3f 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -81,7 +81,7 @@ func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { var schema TableSchema if err := doRequest(ctx, t.ctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", params: url.Values{"table": {t.table}}, }, &schema); err != nil { return nil, fmt.Errorf("get schema for table %q: %w", t.table, err) diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index e721afb6..d715b3bc 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -454,7 +454,7 @@ "name": "raw SQL", "endpoint": "sql", "sql": "SELECT count() FROM clicks", - "expected_path": "/v1/admin/query", + "expected_path": "/v1/ops/query", "expected_method": "POST", "expected_body": { "sql": "SELECT count() FROM clicks" } }, @@ -467,19 +467,19 @@ { "name": "schema list", "endpoint": "schema_list", - "expected_path": "/v1/schema", + "expected_path": "/v1/ops/schema", "expected_method": "GET" }, { "name": "schema refresh", "endpoint": "schema_refresh", - "expected_path": "/v1/schema/refresh", + "expected_path": "/v1/ops/schema/refresh", "expected_method": "POST" }, { "name": "policy get", "endpoint": "policy_get", - "expected_path": "/v1/admin/policy", + "expected_path": "/v1/ops/policy", "expected_method": "GET" }, { @@ -500,14 +500,14 @@ { "name": "DLQ list", "endpoint": "dlq_list", - "expected_path": "/v1/dlq/stats", + "expected_path": "/v1/ops/dlq/stats", "expected_method": "GET" }, { "name": "DLQ table filter", "endpoint": "dlq_table", "table": "events", - "expected_path": "/v1/dlq/stats?table=events", + "expected_path": "/v1/ops/dlq/stats?table=events", "expected_method": "GET" }, { @@ -519,7 +519,7 @@ "events": {} } }, - "expected_path": "/v1/admin/policy", + "expected_path": "/v1/ops/policy", "expected_method": "PUT", "expected_content_type": "application/json", "expected_body": { @@ -538,7 +538,7 @@ "events": {} } }, - "expected_path": "/v1/admin/policy/validate", + "expected_path": "/v1/ops/policy/validate", "expected_method": "POST", "expected_content_type": "application/json", "expected_body": { @@ -551,14 +551,14 @@ { "name": "pipes list", "endpoint": "pipes_list", - "expected_path": "/v1/admin/pipes", + "expected_path": "/v1/ops/pipes", "expected_method": "GET" }, { "name": "pipes get", "endpoint": "pipes_get", "pipe_name": "my_pipe", - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "GET" }, { @@ -572,7 +572,7 @@ ], "description": "Top pages by view count" }, - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "PUT", "expected_content_type": "application/json", "expected_body": { @@ -587,7 +587,7 @@ "name": "pipes delete", "endpoint": "pipes_delete", "pipe_name": "my_pipe", - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "DELETE" }, { diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 5acfc46b..3698aedd 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -41,6 +41,16 @@ type ClientOptions struct { // MaxRetries is the maximum number of retry attempts for retryable errors. // Total attempts = MaxRetries + 1. Default: 2. MaxRetries int + + // Headers are sent on every request the client makes — REST calls and SSE + // streams alike. Use them for a gateway credential, a tenant selector, or + // tracing metadata that has no first-class option. + // + // The SDK's own headers win: Authorization, Accept, Content-Type, and the + // stream's Cache-Control are set after these and overwrite any entry that + // collides. Names are matched case-insensitively (canonicalized by + // net/http), and each entry replaces rather than appends. + Headers map[string]string } // StaticToken returns an Auth function that always returns the same token. @@ -72,6 +82,15 @@ func NewClient(cfg Config) *Client { maxRetries = cfg.Options.MaxRetries } + // Copy so a later mutation of the caller's map can't reach into requests. + var headers map[string]string + if cfg.Options != nil && len(cfg.Options.Headers) > 0 { + headers = make(map[string]string, len(cfg.Options.Headers)) + for k, v := range cfg.Options.Headers { + headers[k] = v + } + } + hc := cfg.HTTPClient if hc == nil { // Not http.DefaultClient: it's mutable global state another package @@ -85,6 +104,7 @@ func NewClient(cfg Config) *Client { auth: cfg.Auth, maxRetries: maxRetries, httpClient: hc, + headers: headers, }, } @@ -124,7 +144,7 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { var rows []Row err := doRequest(ctx, c.ctx, requestOptions{ method: "POST", - path: "/v1/admin/query", + path: "/v1/ops/query", body: map[string]string{"sql": query}, }, &rows) if err != nil { diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index 65695dbf..7e06eeed 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -5,9 +5,11 @@ description: "Schema introspection, access-control policy, DLQ stats, and health Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. +Every namespace on this page is admin-gated: the server mounts them under `/v1/ops/*` behind one gate, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). + ## Schema — `client.Schema` -Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-only** `/v1/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-gated** `/v1/ops/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. ```go // List all table schemas. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index f2a4fc7f..219c1105 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -97,6 +97,7 @@ The default client has no `Timeout`; use a `context.Context` deadline to prevent | Field | Type | Default | Description | |-------|------|---------|-------------| | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | +| `Headers` | `map[string]string` | `nil` | Sent on every request the client makes — REST calls and SSE streams alike. | `*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. @@ -104,6 +105,22 @@ The default client has no `Timeout`; use a `context.Context` deadline to prevent The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. ::: +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata that has no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, // Options opts out of the default — set it explicitly. + Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, + }, +}) +``` + +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any entry that collides. Names are matched case-insensitively (`net/http` canonicalizes them), and each entry replaces rather than appends. The map is copied at `NewClient`, so mutating it afterwards changes nothing. + +For the two remaining TypeScript knobs there is no Go field, because `Config.HTTPClient` already covers them: `options.fetch` maps to supplying your own `*http.Client`, and `options.fetchOptions` maps to a custom `http.RoundTripper` on that client's `Transport`. + For static tokens, use `wavehouse.StaticToken(token)`: ```go @@ -114,7 +131,11 @@ wh := wavehouse.NewClient(wavehouse.Config{ ``` :::note[How the token is transmitted] -Unlike a browser's `EventSource`, Go's `net/http` client sets arbitrary headers on any request, so the Go SDK sends `Authorization: Bearer ` on every request, including SSE streams. No `?token=` query fallback (a TypeScript-in-the-browser concern; see its [equivalent note](/sdk#creating-a-client)). +The Go SDK sends `Authorization: Bearer ` on every request, including SSE streams, and never uses a `?token=` query fallback. Both SDKs work this way: the TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is now the shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +::: + +:::caution[A credentialed stream will not follow a redirect] +When the stream request carries a credential — an `Auth` token or a `ClientOptions.Headers` entry — the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. ::: :::caution[Use HTTPS for authenticated non-local servers] @@ -170,6 +191,7 @@ Both SDKs share a wire format and feature set, verified by a shared `wire_cases. - **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). - **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. - **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. +- **No third-party dependencies.** The Go SDK is stdlib-only, including its SSE frame parser. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). - **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 69e426a1..7ec96f91 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -50,7 +50,7 @@ stream := wh.Pipe("top_pages", nil).Stream(nil) ## Pipes Admin — `client.Pipes` -Manage named query pipes. Requires the admin role (`policy.admin_role`). +Manage named query pipes. These sit behind the admin gate on `/v1/ops/*`, which a caller clears one of two ways: a JWT resolving to the policy admin role (`policy.admin_role`), or the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). ```go // List all pipes. diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 337cc042..8a7d223d 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -309,7 +309,7 @@ for page.HasMore && page.Next != nil { ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query via `/v1/admin/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. +Execute a raw SQL query via `/v1/ops/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/ops/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 8a076a41..e5e733ec 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -45,7 +45,14 @@ HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-sid | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | -| 0 | `SSE_ERROR` | Yes | Stream connection failure; delivered to subscriber's `Error` callback; auto-reconnects | +| 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | +| 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | +| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | +| *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | +| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | +| 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | +| 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | +| 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | ```go page, err := wh.From("clicks").Fetch(ctx) @@ -62,7 +69,7 @@ if err != nil { `wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. -Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/admin/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. ## Full API Tree @@ -133,7 +140,7 @@ Or, inside `clients/go/`: go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` -Codegen reads the admin-only `/v1/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. +Codegen reads the admin-only `/v1/ops/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. **Options:** @@ -186,7 +193,7 @@ The generator does not special-case initialisms: `event_id` becomes `EventId`, n | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | -Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/admin/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/ops/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. ## Testing diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index c16033b6..95950a0b 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -122,6 +122,8 @@ type StreamEvent struct { } ``` +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value — the ingest handler rewrites them before publishing, so a live frame and a later query can't disagree on the spelling of an instant. Two consequences worth knowing: a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open — a value the server can't parse, or one whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). + :::note[`Events()` carries events only] `Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. ::: @@ -136,9 +138,19 @@ Events buffer (up to 256) starting at `.Stream()`; events arriving before the fi | --------- | --------- | -------- | | SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | -Reconnect covers transport failures and retryable (5xx/429) responses. Non-retryable ones (401/403/404) are terminal: the `Error` callback fires, status goes `StatusClosed`, no reconnect. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. +Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). + +Note that `/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`. A `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. + +Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). The TypeScript SDK streams over `fetch` and authenticates the same way, so this is shared behavior rather than a Go-only property — what Go avoids is the browser's per-domain connection ceiling, not a different auth mechanism. + +Delivery across a reconnect is **at-least-once**: the server replays from the last event ID *inclusively*, so the first frame after a gap-fill is usually one you already saw. Replay reaches back only as far as the server's `mq.gap_window_minutes` (15 minutes by default); a longer outage resumes live with a hole. + +### Server-Side Policy Filtering + +Before anything reaches the client, the server applies the caller's policy to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber — on live frames and on `Since` gap-fill replay alike. The claims are captured from the JWT at connect time. -Auth goes as an `Authorization: Bearer` header on every connection ([note in Getting Started](/sdk/go#creating-a-client)). Browser `EventSource` limits don't apply. +Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control](/access-control#row-level-security). ### Client-Side Stream Filtering diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 0a0f3ef8..4f937cda 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -47,17 +47,17 @@ const server = createServer((req, res) => { body: Buffer.concat(chunks).toString("utf-8"), }; res.setHeader("Content-Type", "application/json"); - if (req.url?.startsWith("/v1/dlq")) { + if (req.url?.startsWith("/v1/ops/dlq")) { res.end(JSON.stringify({ tables: {}, total: 0 })); - } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/schema") && req.method === "GET") { res.end(JSON.stringify([])); - } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { + } else if (req.url === "/v1/ops/policy/validate" && req.method === "POST") { res.end(JSON.stringify({ valid: true })); - } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/policy") && req.method === "GET") { res.end(JSON.stringify({ tables: {} })); - } else if (req.url?.startsWith("/v1/admin/pipes/") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/pipes/") && req.method === "GET") { res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); - } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { + } else if (req.url === "/v1/ops/pipes" && req.method === "GET") { res.end(JSON.stringify([])); } else if (req.url?.startsWith("/v1/ingest")) { // Same shapes the real server returns (internal/api/ingest.go). From 5d26c34eaea6b0d17c3e6324cc2685d6ef49d3c9 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:01:38 -0400 Subject: [PATCH 33/59] docs(changelog): note the Go SDK's Headers option in the unreleased entry --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f6bd31a7..e45eac98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. From 90d59ef0a84461ce3f14a1027a4dd6c7b98e80f8 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:20:55 -0400 Subject: [PATCH 34/59] style(docs): unwrap hard-wrapped prose in the Go SDK pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The six Go SDK pages were authored before #489 landed WH001 (no-hard-wrapped-prose), so they wrapped prose at ~76 columns while the rest of the docs tree had been reflowed. `make fix` output, whitespace only — verified no content changed in any of the four files. --- docs/src/content/docs/sdk/go/index.md | 9 ++------- docs/src/content/docs/sdk/go/queries.md | 9 +-------- docs/src/content/docs/sdk/go/reference.md | 6 +----- docs/src/content/docs/sdk/go/streaming.md | 9 +-------- 4 files changed, 5 insertions(+), 28 deletions(-) diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 219c1105..65afc19f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -3,15 +3,10 @@ title: "Go SDK" description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." --- -`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime -dependency Go client for WaveHouse (stdlib only). +`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime dependency Go client for WaveHouse (stdlib only). :::tip[Looking for the TypeScript SDK?] -This page and the rest of `/sdk/go/*` cover the Go client. The -JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at -[SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything -you learn about WaveHouse's query builder, streaming, or admin endpoints on -either page mostly carries over. +This page and the rest of `/sdk/go/*` cover the Go client. The JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at [SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything you learn about WaveHouse's query builder, streaming, or admin endpoints on either page mostly carries over. ::: ## Installation diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 8a7d223d..c1fc8e2a 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -3,14 +3,7 @@ title: "Go SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." --- -Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: -table references, the chainable query builder, cursor pagination, and the -admin-only raw-SQL escape hatch. Every request-response operation takes a -`context.Context` as its first argument and returns `(T, error)`; the -chainable builder methods and `.Stream(opts)` are the exceptions — see -[Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's -[Queries](/sdk/queries) page, which covers the same surface with a -`Result`-returning, `PromiseLike` builder. +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` as its first argument and returns `(T, error)`; the chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. ## Tables — `client.From(table)` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index e5e733ec..b7123fce 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -3,11 +3,7 @@ title: "Go SDK Reference & CLI" description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." --- -Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: -cancellation, the error model behind every request-response call's `(T, error)` return, -the complete API tree at a glance, and the `wavehouse-codegen` tool that -ships with the module. Compare with the TypeScript SDK's -[Reference & CLI](/sdk/reference) page. +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. ## Context Cancellation diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 95950a0b..69487e16 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -3,14 +3,7 @@ title: "Go SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." --- -Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE -event streams from tables, builders, and pipes, plus live queries that -backfill history before going live. Builders and table refs come from -[Queries](/sdk/go/queries). Compare with the TypeScript SDK's -[Streaming & Live Queries](/sdk/streaming) page — the two implement the same -protocol and mostly the same client-side filtering, but connection lifecycle -differs: Go streams are goroutine-backed and closed explicitly, not tied to -a `context.Context` or a browser's `EventSource`. +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/go/queries). Compare with the TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) page — the two implement the same protocol and mostly the same client-side filtering, but connection lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. ## Streaming From c460c02c23fddc0a7f94ec86d176c968a088b1d9 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:15 -0400 Subject: [PATCH 35/59] build(lint): put the shared conformance runner under Biome MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit biome.json's files.includes covered clients/ts, tests/e2e/sdk, docs, and scripts, but not tests/conformance — so the 311-line conformance_ts.mjs this branch adds was invisible to make lint-ts / fmt-ts / fix-ts. Including it surfaced one formatting fix, applied here. --- biome.json | 1 + tests/conformance/conformance_ts.mjs | 11 ++++++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/biome.json b/biome.json index e29be7f3..700f3729 100644 --- a/biome.json +++ b/biome.json @@ -10,6 +10,7 @@ "clients/ts/src/**", "clients/ts/*.{ts,js,mjs,cjs,json}", "tests/e2e/sdk/**/*.{ts,js,mjs,cjs,json}", + "tests/conformance/**/*.{ts,js,mjs,cjs,json}", "docs/**/*.{ts,js,mjs,cjs,json}", "scripts/**/*.{ts,js,mjs,cjs}" ] diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 4f937cda..b145e994 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -22,12 +22,16 @@ let createClient; try { ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); } catch (err) { - console.error("Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`)."); + console.error( + "Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`).", + ); console.error(err.message); process.exit(1); } -const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); +const cases = JSON.parse( + readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8"), +); let lastCapture = { method: "", path: "", contentType: "", body: "" }; @@ -303,7 +307,8 @@ for (const f of failures) { if (failed > 0 || skipped > 0 || passed === 0) { if (passed === 0) console.log(" ✗ nothing ran — every case skipped or the fixture is empty\n"); - if (skipped > 0) console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); + if (skipped > 0) + console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); process.exit(1); } else { console.log(" ✓ All cases passed\n"); From 941b09758bcaf178d7bdc7cd3b225fc925ac7345 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:15 -0400 Subject: [PATCH 36/59] style(docs): unwrap the Go SDK aside in sdk/index.mdx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WH001's autofix deliberately never runs over .mdx, so this aside — added on this branch before the rule landed — had to be joined by hand. Text unchanged. --- docs/src/content/docs/sdk/index.mdx | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 8d62bdfb..652f9ba5 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -8,14 +8,7 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — TypeScript client for WaveHouse. One runtime dependency: `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. :::tip[Writing Go instead?] -WaveHouse also ships an official Go SDK -(`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, -`context.Context`-first, generics for typed rows. See the -[Go SDK docs](/sdk/go). The two clients speak the same wire format, so -everything below about tables, the query builder, streaming, and admin -endpoints carries over conceptually, but API and lifecycle details differ — -Go uses context-first calls and package-level generics, and streams must be -closed explicitly. +WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually, but API and lifecycle details differ — Go uses context-first calls and package-level generics, and streams must be closed explicitly. ::: ## Installation From c538ee37e98fea919c6f04913728c7b810a66874 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:28 -0400 Subject: [PATCH 37/59] build(deps): track the nested clients/go module in Dependabot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gomod entry was `directory: /`, which covers the root module only — Dependabot does not descend into nested modules, so clients/go/go.mod was untracked. Converted to `directories:` (the form the github-actions entry already uses for the same reason) so both share one schedule, group, and commit prefix rather than forking the policy into a second block. clients/go is stdlib-only today, so this catches the first dependency it takes on rather than closing a gap that already exists — the same argument the github-actions comment block makes about composite actions. --- .github/dependabot.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index e1c5d695..38d5c98c 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,8 +1,17 @@ version: 2 updates: # Go modules + # + # TWO directories, not one — the same trap as github-actions below: + # `directory: /` covers the root module only, and Dependabot does not + # descend into nested modules. clients/go is its own module, so its + # go.mod needs its own entry here. That module is stdlib-only today + # (no `require` block, no go.sum), so this catches the first dependency + # it takes on rather than closing a gap that already exists. - package-ecosystem: gomod - directory: / + directories: + - / + - /clients/go schedule: interval: weekly day: monday From 167e16db68f44c9db2dd5b98343c43fb25716bd3 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:25:06 -0400 Subject: [PATCH 38/59] build(labels): map the Go SDK paths to area/sdk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit area/sdk globbed clients/ts/** and tests/e2e/sdk/** only, so every file in the new Go SDK and the shared conformance runner went unlabeled. Adds clients/go/** and tests/conformance/**. Also broadens the dependencies label from root-anchored go.mod/go.sum to **/go.mod and **/go.sum, so the nested clients/go module's manifests get labeled the way clients/ts/package.json already does via **/package.json. Left area/infra root-anchored — the TypeScript SDK's manifest doesn't carry that label either. Verified by replaying the config's globs over the branch's diff: clients/go sources land on area/sdk + go, the codegen CLI on area/sdk, and clients/go/go.mod on area/sdk + dependencies. Note this takes effect from the next PR, not this one: housekeeping.yml resolves labeler.yml from the default branch, not the PR head. --- .github/labeler.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/labeler.yml b/.github/labeler.yml index 4df42c01..d164e157 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -43,7 +43,9 @@ - changed-files: - any-glob-to-any-file: - "clients/ts/**" + - "clients/go/**" - "tests/e2e/sdk/**" + - "tests/conformance/**" "area/docs": - changed-files: @@ -83,8 +85,8 @@ dependencies: - changed-files: - any-glob-to-any-file: - - "go.mod" - - "go.sum" + - "**/go.mod" + - "**/go.sum" - "**/package.json" - "**/pnpm-lock.yaml" - "**/package-lock.json" From 73edcbe3e74b5d9c444b26e6e379857f589deb52 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:33:09 -0400 Subject: [PATCH 39/59] test(cov): give the Go SDK its own coverage floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test-go-sdk ran a bare `go test -race ./...` — no profile, no threshold — so the Go SDK would have shipped as the only component in the repo with no coverage gate while its TypeScript peer has three. It now collects covdata the same way the root-module Go suites do and renders through scripts/cov, with a `go-sdk` suite gated at 75% against a measured 82.7% (clients/go 88.6%, cmd/wavehouse-codegen 55.8%). That ratio sits inside the band the existing floors already use — unit is 80 against ~91% actual — leaving headroom without being slack enough to rot. Gated standalone rather than merged into the Go total: the total feeds threshold.total and the README badge, and folding a shipped client library into the server's project-wide number would move that badge for unrelated reasons. The same separation the TypeScript SDK gets via ts-*. Being a nested module, clients/go is invisible to the root -coverpkg anyway, so it cannot leak into the total and needs no exclude.paths entry — verified: zero clients/go rows in the merged profile, Go total unmoved at 81.2%. One wrinkle worth recording: `go tool cover -html` resolves a profile's package paths through the module in the working directory, so rendering from the repo root fails to find the nested packages. renderHTML now runs with cmd.Dir set to clients/go and absolute paths. CI needed no functional change — the unit job already uploads the whole tmp/coverage tree, so the fragment arrives and the coverage job gates it. --- .claude/commands/cover.md | 5 +- .github/workflows/ci.yml | 6 ++ .testcoverage.yml | 16 +++++- AGENTS.md | 2 +- Makefile | 27 ++++++++- docs/src/content/docs/development.md | 6 +- scripts/cov/main.go | 83 +++++++++++++++++++++++++++- 7 files changed, 132 insertions(+), 13 deletions(-) diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 021def65..93b5a33e 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -1,6 +1,6 @@ --- description: Render coverage HTML for a suite and surface drops below threshold -argument-hint: [unit|integration|e2e|sdk|merge|all] (default: merge whatever exists) +argument-hint: [unit|integration|e2e|go-sdk|sdk|merge|all] (default: merge whatever exists) --- Generate the coverage report and surface anything below threshold from `.testcoverage.yml`. @@ -13,10 +13,11 @@ Behavior: - **unit**: `make test-unit` (gates per-suite + writes `tmp/coverage/unit/`) - **integration**: `make test-integration` (requires Docker) - **e2e**: `make test-e2e` (requires Docker; orchestrator + cover binary) +- **go-sdk**: `make test-go-sdk` (nested module `clients/go`; gates against `suites.go-sdk`, rendered separately and never merged into the Go total) - **ts-unit**: `make test-ts` (SDK unit tests + coverage + gate against `suites.ts-unit`) - **ts-e2e**: emitted as a side effect of `make test-e2e` (the orchestrator always passes `--coverage` to the e2e vitest run; informational only, no standalone gate) - **ts-total**: `make cov` (runs `cov report` — one consolidated Go + TS summary with per-suite HTML links + all gates; fails if *no* suite has data) -- **all**: `make test-all` (all four suites sequentially + `make cov`) +- **all**: `make test-all` (every suite sequentially + `make cov`) After the run completes: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 12fc070c..deebc508 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,6 +206,12 @@ jobs: go-cache-suffix: "-unit" - name: Run Go unit tests + SDK vitest + Go SDK tests run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 + # This one fragment carries THREE suites' data: unit covdata, ts-unit + # (vitest/istanbul) and go-sdk covdata from the nested clients/go + # module — every target above ran under COV_DEFER, so the `coverage` + # job renders and gates all three. No per-suite paths here on purpose: + # uploading the whole tmp/coverage tree means a new suite collected by + # this job needs no workflow change. - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/.testcoverage.yml b/.testcoverage.yml index 669d65be..ee02af8d 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -12,7 +12,9 @@ # together, so `threshold.total` below applies to the *project-wide* # coverage number — unit alone covers only `./internal/...` packages # exercised by `*_test.go` files (below total threshold); merged -# coverage adds integration- and e2e-only paths and clears it. +# coverage adds integration- and e2e-only paths and clears it. The +# nested-module suites (go-sdk) are gated separately and never merged +# in — see suites.go-sdk. profile: tmp/coverage/total/coverage.txt local-prefix: github.com/Wave-RF/WaveHouse @@ -30,6 +32,18 @@ suites: unit: 80 integration: 20 e2e: 60 + # Go SDK (clients/go) — a NESTED module, so its coverage is rendered and + # gated on its own and is deliberately NOT part of the merged Go total + # above. Nothing from clients/go can leak into that total: the root + # module can't see a nested one (`go list ./...` at the repo root never + # yields clients/go), so the unit/integration/e2e `-coverpkg=./...` never + # reaches these files — which is also why there is no `^clients/go/` + # entry under exclude.paths; there is nothing to exclude. + # Measured 82.7% when this floor was set (SDK package 88.6%, + # cmd/wavehouse-codegen 55.8%). 75 leaves the same order of headroom + # unit's 80 leaves under its real ~91%. Raise it as the codegen + # command's tests fill in. + go-sdk: 75 # TypeScript SDK suites — see scripts/cov for the merge / render logic. # vitest gates via --coverage.thresholds.statements; the merged ts-total # is gated by `cov ts-merge` against the value below. Tune ts-total diff --git a/AGENTS.md b/AGENTS.md index 10a2dcb6..c06b32c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,7 +125,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Policy helpers**: Use `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. - **Pipes helpers**: Use `pipes.NewMemoryStore(queries...)` for in-memory pipes testing without NATS. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. -- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, sdk 50%. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. +- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts SDK 50%. The Go SDK (`clients/go`) is a nested module — invisible to the root module's `-coverpkg=./...`, so it is gated on its own `suites.go-sdk` floor and never merged into the project-wide total. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. - **Every new function should have corresponding test cases.** Run `make lint` and `make test` before considering work complete. - **E2E tests via SDK**: The TypeScript SDK is the primary E2E test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) and simultaneously validate backend behavior and SDK correctness. Use `make test-e2e` to run. Add new E2E scenarios as `tests/e2e/sdk/*.test.ts` files using helpers from `tests/e2e/sdk/helpers.ts`. - **Per-suite table isolation**: Each e2e test file owns its own ClickHouse tables — `clicks_` / `events_` / `users_`, generated from `tests/e2e/sdk/tables.ts` and created by `setup.ts`. A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`. This makes cross-file *data* contamination structurally impossible. Files still run **sequentially** (`vitest.config.ts` `maxWorkers: 1`): running them in parallel is blocked by shared *global policy* state (several files read-modify-write the single policy document; `streaming.test.ts` flips the global `default_role`), so policy-mutating tests snapshot the full policy and restore it. Dropping `maxWorkers: 1` is a deferred follow-up tracked in #214 (per-table policy storage; see `docs/src/content/docs/ingest-pipeline.md` § Deferred). diff --git a/Makefile b/Makefile index 95ff1879..a4696334 100644 --- a/Makefile +++ b/Makefile @@ -199,6 +199,9 @@ ACTIONLINT := $(LOCAL_BIN)/actionlint-$(ACTIONLINT_VERSION) COV_UNIT := tmp/coverage/unit COV_INT := tmp/coverage/integration COV_E2E := tmp/coverage/e2e +# go-sdk is the nested module at clients/go — same layout, own gate, but +# deliberately outside COV_TOTAL (see test-go-sdk below). +COV_GOSDK := tmp/coverage/go-sdk COV_TOTAL := tmp/coverage/total # --- Coverage Thresholds ------------------------------------------------------ @@ -805,10 +808,28 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and # needs its own target. -race because the SDK's streaming subsystem is the # most concurrent code in the repo. +# +# Coverage is collected exactly like the root-module Go suites (covdata into +# tmp/coverage//data via -test.gocoverdir), so `cov render go-sdk` +# renders + gates it with no new machinery and CI's coverage fragment — +# `path: tmp/coverage` on the unit job, which already runs this target — +# carries it to the `coverage` job unchanged. -coverpkg=./... resolves +# inside clients/go, so the denominator is the SDK package + the codegen +# command, nothing from the server. +# +# The go-sdk suite is NOT part of the merged Go total: a nested module is +# invisible to the root module (`go list ./...` at the repo root never +# yields clients/go), so the other suites' -coverpkg=./... cannot reach +# these files — they can't leak into tmp/coverage/total, and no +# exclude.paths entry is needed to keep them out. Same separation the TS +# SDK gets via ts-*. Gate: suites.go-sdk in .testcoverage.yml. .PHONY: test-go-sdk -test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests +test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render coverage + gate threshold @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" - @cd clients/go && go test -race ./... + @rm -rf $(COV_GOSDK)/data && mkdir -p $(COV_GOSDK)/data + @cd clients/go && GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" go test -cover -coverpkg=./... -race ./... \ + -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" + @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi # test-conformance-ts: the TS half of the cross-SDK wire-format conformance # suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). @@ -830,7 +851,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 - @$(MAKE) test-go-sdk + @$(MAKE) test-go-sdk COV_DEFER=1 @$(MAKE) test-conformance-ts @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index a1373581..cc07e18f 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -324,7 +324,7 @@ make ci make cov ``` -Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-go-sdk`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. `test-go-sdk` is gated but **not** merged: `clients/go` is a nested Go module, invisible to the root module's `-coverpkg=./...`, so its statements can never reach `tmp/coverage/total` — it carries its own `suites.go-sdk` floor instead, the same way the TS SDK carries `ts-*`. The remaining SDK/conformance targets (`test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. **Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. @@ -338,7 +338,7 @@ Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test- | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | | SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | -| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`, always includes coverage + gate) | | Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | | SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | @@ -534,7 +534,7 @@ Run `make help` to see all targets. Key ones: | **Test** | | | `make test` | Alias for `test-unit` + `test-go-sdk` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | -| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` | +| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` + render coverage + gate `suites.go-sdk` (own gate; never merged into the Go total) | | `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | diff --git a/scripts/cov/main.go b/scripts/cov/main.go index f4250bfa..2a3948bb 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -59,6 +59,30 @@ const ( // see the ts-total path below. var goSuites = []string{"unit", "integration", "e2e"} +// Go suites that produce covdata in the same layout as goSuites and are +// rendered + gated identically, but are deliberately NOT merged into the +// Go total: they come from a NESTED module (clients/go has its own +// go.mod). The root module cannot see a nested one — `go list ./...` at +// the repo root never yields clients/go — so the goSuites' -coverpkg=./... +// can't reach these files in the first place; they can only ever be in +// tmp/coverage/total if we put them there, which we don't. Folding a +// shipped client library into the server's project-wide number (and into +// the README badge `cov badge` derives from it) would move that number for +// reasons that have nothing to do with the server, so the SDK gets its own +// floor instead — the same separation the TS SDK gets via ts-*. +var standaloneGoSuites = []string{"go-sdk"} + +// suiteModuleDir maps a suite to the module directory its covdata was +// produced in, for suites that aren't the root module. `go tool cover +// -html` reads the source of every package named in the profile and +// resolves it through the module in the process's working directory, so a +// nested module's profile has to be rendered from inside that module — +// from the repo root the tool fails with "no required module provides +// package github.com/Wave-RF/WaveHouse/clients/go/...". `go tool covdata +// textfmt` has no such constraint (it only reads the covdata files), so +// only the HTML step needs the chdir. +var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} + // TypeScript SDK suites (vitest). ts-unit comes from clients/ts; ts-e2e // from tests/e2e/sdk run with --coverage. Both produce Istanbul-format // coverage-final.json that `cov ts-merge` combines into ts-total. @@ -212,7 +236,7 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int if err = sh("go", "tool", "covdata", "textfmt", "-i="+dataDir, "-o", profile); err != nil { return nil, 0, 0, "", err } - if err = sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut); err != nil { + if err = renderHTML(suite, profile, htmlOut); err != nil { return nil, 0, 0, "", err } rows, total, covered, err = parseCoverage(profile, c, c.excludesFor(suite)) @@ -222,6 +246,27 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int return rows, total, covered, htmlOut, nil } +// renderHTML turns a textfmt profile into the clickable HTML report. For a +// suite whose covdata came from a nested module (see suiteModuleDir) the +// tool runs with that module as its working directory — otherwise it can't +// resolve the profile's package paths to source and bails — so the profile +// and output paths are made absolute first. +func renderHTML(suite, profile, htmlOut string) error { + dir, nested := suiteModuleDir[suite] + if !nested { + return sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut) + } + absProfile, err := filepath.Abs(profile) + if err != nil { + return err + } + absHTML, err := filepath.Abs(htmlOut) + if err != nil { + return err + } + return shIn(dir, "go", "tool", "cover", "-html="+absProfile, "-o", absHTML) +} + func renderSuite(c *config, suite string) error { rows, total, covered, htmlOut, err := goSuiteCoverage(c, suite) if err != nil { @@ -247,7 +292,7 @@ func renderSuite(c *config, suite string) error { // "one side legitimately absent" (skip, fine) from "nothing ran at all" // (fail, because the caller expected a gate). func hasAnyCoverage() bool { - for _, s := range goSuites { + for _, s := range slices.Concat(goSuites, standaloneGoSuites) { if hasCovdata(filepath.Join(root, s, "data")) { return true } @@ -298,6 +343,13 @@ func merge(c *config) error { for _, s := range goSuites { fmt.Printf(" %s%-13s%s %s\n", cyan, s+":", reset, suitePct(c, s)) } + // Nested-module Go suites: gated on their own, never merged above. + for _, s := range standaloneGoSuites { + if pct := suitePct(c, s); pct != "n/a" { + fmt.Printf(" %s%-13s%s %s %s(separate gate; not in merge above)%s\n", + cyan, s+":", reset, pct, yellow, reset) + } + } // Surface TS SDK coverage alongside the Go total — informational only, // not part of the Go merged number above. `make cov` is the gate. for _, s := range append(tsSuites, "ts-total") { @@ -551,6 +603,25 @@ func report(c *config) error { }) } + // --- Nested-module Go suites (own gate, below the Go total) --- + // Rendered and gated exactly like the suites above, but listed after + // the total they are deliberately not part of — see standaloneGoSuites. + for i, s := range standaloneGoSuites { + if !hasCovdata(filepath.Join(root, s, "data")) { + rows = append(rows, reportRow{name: s, pct: "n/a", rule: i == 0}) + continue + } + _, total, covered, html, err := goSuiteCoverage(c, s) + if err != nil { + return err + } + th := thresholdFor(c, s) + rows = append(rows, reportRow{ + name: s, pct: formatPctBare(covered, total), gated: true, thresh: th, + pass: meetsThreshold(covered, total, th), html: html, rule: i == 0, + }) + } + // --- TS suites + merged ts-total --- merged, err := mergeTSArtifacts("html", "json-summary") if err != nil { @@ -917,9 +988,15 @@ func meetsThreshold(covered, total, threshold int) bool { // sh runs an external command with stdio wired through. Every call site // passes "go" as the program and a fixed series of "tool", "", // flag, … args; the only variable bits are paths we computed ourselves. +func sh(name string, args ...string) error { return shIn("", name, args...) } + +// shIn is sh with an explicit working directory ("" = inherit ours) — for +// the one tool that cares which module it runs in, `go tool cover -html` +// on a nested module's profile. See renderHTML. // #nosec G204,G702 — name and args are not user input. -func sh(name string, args ...string) error { +func shIn(dir, name string, args ...string) error { cmd := exec.CommandContext(context.Background(), name, args...) + cmd.Dir = dir cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr return cmd.Run() From 3ed16983e418ffdd1aed37d3f98ed10b0d53d48b Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:48:03 -0400 Subject: [PATCH 40/59] fix(sdk): compare stream filter timestamps as instants, not as text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client-side filter behind .Stream() compared every string operand lexically. That was defensible before #402; it isn't now. The server canonicalizes every top-level DateTime/DateTime64 value to RFC 3339 UTC before publishing, so a payload reads 2026-06-21T04:00:00Z while a caller's filter constant may name the same instant as 2026-06-21T06:00:00+02:00 — and lexically the payload sorts BELOW the constant, so OpGte withheld a row that was chronologically equal, and OpEq called two spellings of one instant different. Both sides now parse as instants, mirroring what internal/policy's row filter does for a DateTime column. Deliberately narrow: only RFC 3339 with an explicit offset or Z counts. A zone-less spelling names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — reading it as UTC would move the instant, so those fall through rather than being silently reinterpreted. A ',' fraction is ISO 8601 but not ClickHouse, and is refused for the same reason the ingest grammar refuses it. The operand length is pre-gated at 64 bytes like the server's, so a megabyte 'timestamp' isn't scanned once per filter per event. Ordering an instant against a non-instant now fails closed instead of falling back to text, which could admit rows the query path excludes — the same direction the server errs in. The usual trigger is a zone-less constant, which now yields no rows rather than wrong ones. Also: a column missing from the payload no longer matches the literal string "" through the fmt.Sprint equality fallback. Not fixed here, and worth its own change: the TypeScript SDK's matchesFilters compares the same way, and its compareOrdered comment still asserts that lexicographic order 'is correct for ISO-8601 timestamps' — true only while both sides share an offset spelling, which #402 stopped guaranteeing. Its unknown-operator branch also returns true where Go and the server fail closed. --- CHANGELOG.md | 2 + clients/go/stream.go | 73 +++++++++++++- clients/go/stream_test.go | 116 ++++++++++++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 14 +++ 4 files changed, 204 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e45eac98..849b018f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- **Go SDK client-side stream filters compare timestamps as instants, not as text** (`clients/go/stream.go`, `clients/go/stream_test.go`, `docs/src/content/docs/sdk/go/streaming.md`): the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing (#402), so a payload reads `2026-06-21T04:00:00Z` while a caller's filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Compared as text those disagree in both directions — lexically the payload sorts *below* the constant, so `OpGte` withheld a row that was chronologically equal. Both sides now parse as instants, mirroring the server's row-filter rule for DateTime columns. Only unambiguous spellings count (RFC 3339 with an explicit offset or `Z`): a zone-less constant names an instant only relative to the column's declared timezone, which a stream subscriber doesn't have, so reading it as UTC would move the instant. Ordering an instant against a non-instant now fails closed rather than falling back to text comparison. Also fixes a missing column matching the literal string `""` through the equality fallback. The TypeScript SDK's `matchesFilters` has the same text-comparison behavior and needs the same change for parity. + - **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. ## [0.1.0] - 2026-08-19 diff --git a/clients/go/stream.go b/clients/go/stream.go index 48ce32e8..a9388896 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -640,17 +640,68 @@ func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool } // equalValues compares two values for equality, normalizing numeric types -// (JSON decodes numbers as float64, but callers may pass int). +// (JSON decodes numbers as float64, but callers may pass int) and comparing +// timestamps as instants rather than as text — see asInstant. func equalValues(a, b any) bool { + // nil only equals nil: without this the fmt.Sprint fallback would match a + // missing column against the literal string "". + if a == nil || b == nil { + return a == nil && b == nil + } if af, aOK := toFloat64(a); aOK { if bf, bOK := toFloat64(b); bOK { return af == bf } } + if at, aOK := asInstant(a); aOK { + if bt, bOK := asInstant(b); bOK { + return at.Equal(bt) + } + } // fmt.Sprint is safe for all types (no panic on maps/slices). return fmt.Sprint(a) == fmt.Sprint(b) } +// maxTimeOperandChars mirrors the server's row-filter pre-gate: the longest +// spelling the ingest grammar accepts (RFC 3339 with nanoseconds and a numeric +// offset) is 35 bytes, so 64 is generous slack while keeping a megabyte +// "timestamp" from being scanned once per filter per event. +const maxTimeOperandChars = 64 + +// asInstant reports the instant a value denotes, but only for spellings that +// name one unambiguously — RFC 3339 with an explicit offset or `Z`. +// +// This exists because the server canonicalizes every top-level DateTime value +// to RFC 3339 UTC before publishing (#402), so a payload reads `...T04:00:00Z` +// while a caller's filter constant may name the same instant as +// `...T06:00:00+02:00`. Comparing those as text is wrong in both directions: +// lexically the payload sorts *below* the constant, so `gte` misses a row that +// is chronologically equal. The server compares DateTime columns as instants +// for exactly this reason; this is the client-side twin of that rule. +// +// Deliberately narrow. A zone-less spelling ("2026-06-21 04:00:00") names an +// instant only relative to the column's declared timezone, which the server +// reads from the schema and a stream subscriber does not have. Guessing UTC +// would move the instant, so those fail to parse here and fall through to text +// comparison rather than being silently reinterpreted. +func asInstant(v any) (time.Time, bool) { + s, ok := v.(string) + if !ok || len(s) > maxTimeOperandChars { + return time.Time{}, false + } + // ClickHouse has no ',' decimal separator, but Go's RFC3339Nano accepts one + // per ISO 8601. Reject it so the client can't admit a spelling the server + // would refuse. + if strings.ContainsRune(s, ',') { + return time.Time{}, false + } + t, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + return time.Time{}, false + } + return t, true +} + // evaluateIn checks whether actual is contained in the expected slice. // Reflection handles []any and typed slices (e.g., []string, []int) alike. func evaluateIn(actual, expected any) bool { @@ -681,6 +732,26 @@ func compareOrdered(actual, expected any) (int, bool) { } } } + // Timestamps compare chronologically, not lexically. If either side names + // an instant the other must too: ordering a canonicalized payload against a + // spelling that isn't a provable instant is meaningless, and text + // comparison there would admit rows the query path excludes. Fail closed, + // as the server's row filter does for a DateTime column. + aTime, aIsTime := asInstant(actual) + bTime, bIsTime := asInstant(expected) + if aIsTime || bIsTime { + if !aIsTime || !bIsTime { + return 0, false + } + switch { + case aTime.Before(bTime): + return -1, true + case aTime.After(bTime): + return 1, true + default: + return 0, true + } + } if aStr, ok := actual.(string); ok { if bStr, ok := expected.(string); ok { switch { diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index c65bb863..533c14f2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -652,3 +652,119 @@ func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { t.Fatal("stream request never arrived") } } + +// TestEvaluateFilter_TimestampsCompareAsInstants: the server canonicalizes +// every top-level DateTime value to RFC 3339 UTC before publishing (#402), so +// a payload and a caller's filter constant routinely spell the same instant +// differently. Comparing those as text disagrees with the server's row filter, +// which compares DateTime columns chronologically. +func TestEvaluateFilter_TimestampsCompareAsInstants(t *testing.T) { + // The canonicalized payload value, and the same instant in +02:00 — which + // sorts ABOVE it lexically ("06" > "04") while being chronologically equal. + const canonical = "2026-06-21T04:00:00Z" + const sameInstantOffset = "2026-06-21T06:00:00+02:00" + const oneSecondLater = "2026-06-21T06:00:01+02:00" + + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"equal across offsets", canonical, "eq", sameInstantOffset, true}, + {"neq is false across offsets", canonical, "neq", sameInstantOffset, false}, + {"gte holds at the same instant", canonical, "gte", sameInstantOffset, true}, + {"lte holds at the same instant", canonical, "lte", sameInstantOffset, true}, + {"gt is false at the same instant", canonical, "gt", sameInstantOffset, false}, + {"lt sees a later offset instant", canonical, "lt", oneSecondLater, true}, + {"gt is false against a later instant", canonical, "gt", oneSecondLater, false}, + {"in matches across offsets", canonical, "in", []any{"2020-01-01T00:00:00Z", sameInstantOffset}, true}, + + // Same-offset spellings must keep working exactly as before. + {"gt within UTC", "2026-06-21T04:00:01Z", "gt", canonical, true}, + {"lt within UTC", "2026-06-21T03:59:59Z", "lt", canonical, true}, + {"eq identical text", canonical, "eq", canonical, true}, + + // Sub-second precision survives the round trip. + {"fractional seconds order correctly", "2026-06-21T04:00:00.500Z", "gt", canonical, true}, + + // A zone-less constant names an instant only relative to the column's + // declared timezone, which a stream subscriber does not have. It must + // not be silently read as UTC — ordering fails closed. + {"zone-less constant fails closed on gt", canonical, "gt", "2026-06-21 03:00:00", false}, + {"zone-less constant fails closed on lt", canonical, "lt", "2026-06-21 05:00:00", false}, + + // A ',' fraction is ISO 8601 but not ClickHouse, so it is not an instant. + {"comma fraction is not an instant", canonical, "eq", "2026-06-21T04:00:00,000Z", false}, + + // Non-timestamp strings keep lexicographic ordering. + {"plain strings still order lexically", "banana", "gt", "apple", true}, + {"plain strings still compare equal", "apple", "eq", "apple", true}, + + // Numbers are untouched by any of this. + {"numbers still order numerically", 100.0, "gt", 9.0, true}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := evaluateFilter(tc.actual, tc.op, tc.expected, nil); got != tc.want { + t.Fatalf("evaluateFilter(%v, %q, %v) = %v, want %v", + tc.actual, tc.op, tc.expected, got, tc.want) + } + }) + } +} + +// TestEqualValues_NilOnlyEqualsNil: a column missing from the payload must not +// match the literal string "" through the fmt.Sprint fallback. +func TestEqualValues_NilOnlyEqualsNil(t *testing.T) { + tests := []struct { + name string + a, b any + want bool + }{ + {"nil equals nil", nil, nil, true}, + {"nil does not equal the string ", nil, "", false}, + {"the string does not equal nil", "", nil, false}, + {"nil does not equal empty string", nil, "", false}, + {"nil does not equal zero", nil, 0, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := equalValues(tc.a, tc.b); got != tc.want { + t.Fatalf("equalValues(%v, %v) = %v, want %v", tc.a, tc.b, got, tc.want) + } + }) + } +} + +// TestStream_FilterMatchesCanonicalizedPayload: the end-to-end shape of the +// same bug — a caller filters on a non-UTC spelling and the server delivers the +// canonicalized one. +func TestStream_FilterMatchesCanonicalizedPayload(t *testing.T) { + frame := `event: message +id: 2026-06-21T04:00:00Z +data: {"table_name":"clicks","received_timestamp":"2026-06-21T04:00:00Z","data":{"page":"/home","event_ts":"2026-06-21T04:00:00Z"}} + +` + srv := sseServer(t, []string{frame}) + + stream := streamClient(t, srv).From("clicks"). + SelectAll(). + Where("event_ts", OpGte, "2026-06-21T06:00:00+02:00"). + Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Data["page"] != "/home" { + t.Fatalf("unexpected row: %v", e.Data) + } + case <-time.After(5 * time.Second): + t.Fatal("a row chronologically equal to the filter constant was withheld") + } +} diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 69487e16..e136c814 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -160,6 +160,20 @@ stream := wh.From("clicks"). Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). +#### How values are compared + +The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: + +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload reads `2026-06-21T04:00:00Z` while your filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a row that is chronologically equal. Both sides are parsed as instants instead. +- **Only unambiguous spellings count as instants.** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — guessing UTC would move the instant. Such a constant is not treated as a timestamp. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could otherwise admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. +- **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. + +:::caution[Integer precision above 2^53] +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond `Number.MAX_SAFE_INTEGER` (2^53) has already lost exactness before any filter runs — the server compares such columns in their exact storage domain, so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +::: + ## Live Queries Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. From 3831741639ecc4a5ef587d0086ef2a26c1c88015 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 09:51:14 -0400 Subject: [PATCH 41/59] fix(sdk): address unresolved CodeRabbit review threads - http_test.go: hand captured request headers back over a channel instead of a shared variable, so -race sees the happens-before edge between the httptest handler goroutine and the test. Both call sites now share one headerCaptureServer helper. - codegen: the 64-bit-integer note said "map[string]any with SQL[Row]", which is self-contradictory. /v1/ops/query needs SQL[map[string]any]. - AGENTS.md: spell out all three TypeScript coverage floors (ts-unit 40, ts-e2e 40, ts-total 50) rather than a single "ts SDK 50%". - .claude/commands/cover.md: the argument hint listed an undefined "sdk" and omitted the ts-* suites the behavior section documents. - docs/sdk/go/queries.md: QueryBuilder also comes from TableRef.SelectAll(). --- .claude/commands/cover.md | 2 +- AGENTS.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 10 +++---- clients/go/http_test.go | 36 +++++++++++++----------- docs/src/content/docs/sdk/go/queries.md | 2 +- 5 files changed, 26 insertions(+), 26 deletions(-) diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 93b5a33e..7154b8de 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -1,6 +1,6 @@ --- description: Render coverage HTML for a suite and surface drops below threshold -argument-hint: [unit|integration|e2e|go-sdk|sdk|merge|all] (default: merge whatever exists) +argument-hint: [unit|integration|e2e|go-sdk|ts-unit|ts-e2e|ts-total|merge|all] (default: merge whatever exists) --- Generate the coverage report and surface anything below threshold from `.testcoverage.yml`. diff --git a/AGENTS.md b/AGENTS.md index c06b32c5..abbb182c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,7 +125,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Policy helpers**: Use `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. - **Pipes helpers**: Use `pipes.NewMemoryStore(queries...)` for in-memory pipes testing without NATS. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. -- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts SDK 50%. The Go SDK (`clients/go`) is a nested module — invisible to the root module's `-coverpkg=./...`, so it is gated on its own `suites.go-sdk` floor and never merged into the project-wide total. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. +- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts-unit 40%, ts-e2e 40%, ts-total 50%. The Go SDK (`clients/go`) is a nested module — invisible to the root module's `-coverpkg=./...`, so it is gated on its own `suites.go-sdk` floor and never merged into the project-wide total. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. - **Every new function should have corresponding test cases.** Run `make lint` and `make test` before considering work complete. - **E2E tests via SDK**: The TypeScript SDK is the primary E2E test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) and simultaneously validate backend behavior and SDK correctness. Use `make test-e2e` to run. Add new E2E scenarios as `tests/e2e/sdk/*.test.ts` files using helpers from `tests/e2e/sdk/helpers.ts`. - **Per-suite table isolation**: Each e2e test file owns its own ClickHouse tables — `clicks_` / `events_` / `users_`, generated from `tests/e2e/sdk/tables.ts` and created by `setup.ts`. A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`. This makes cross-file *data* contamination structurally impossible. Files still run **sequentially** (`vitest.config.ts` `maxWorkers: 1`): running them in parallel is blocked by shared *global policy* state (several files read-modify-write the single policy document; `streaming.test.ts` flips the global `default_role`), so policy-mutating tests snapshot the full policy and restore it. Dropping `maxWorkers: 1` is a deferred follow-up tracked in #214 (per-table policy storage; see `docs/src/content/docs/ingest-pipeline.md` § Deferred). diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 9fa68258..fa10cfaa 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -184,12 +184,10 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric — map lookup. Generated structs target the structured-query and - // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse - // values into Go types and re-marshals them — so 64-bit integers arrive - // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. - // (Only /v1/ops/query forwards ClickHouse's own JSON, which quotes - // 64-bit ints; use map[string]any with SQL[Row] there.) + // Numeric. Generated structs target /v1/query and /v1/pipes/*, where the + // server re-marshals ClickHouse values, so 64-bit integers arrive as + // unquoted JSON numbers. /v1/ops/query forwards ClickHouse's own JSON, + // which quotes them — use SQL[map[string]any] there. if mapped, ok := map[string]string{ "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 52212c16..c8b0d4c5 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -336,6 +336,21 @@ func TestBaseURLPathPrefixIsPreserved(t *testing.T) { } } +// headerCaptureServer answers any request with `[]` and hands that request's +// headers back over the channel — a channel, not a shared variable, so -race +// sees the edge between the server goroutine and the test. +func headerCaptureServer(t *testing.T) (*httptest.Server, <-chan http.Header) { + t.Helper() + captured := make(chan http.Header, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + captured <- r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + t.Cleanup(srv.Close) + return srv, captured +} + // TestConfiguredHeadersOnRESTRequests: ClientOptions.Headers apply to every // REST call, are matched case-insensitively, and always lose to the SDK's own // headers rather than appending alongside them. @@ -376,14 +391,7 @@ func TestConfiguredHeadersOnRESTRequests(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - var got http.Header - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - got = r.Header.Clone() - w.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(w, `[]`) - })) - defer srv.Close() - + srv, headers := headerCaptureServer(t) client := NewClient(Config{ BaseURL: srv.URL, Auth: tc.auth, @@ -393,6 +401,7 @@ func TestConfiguredHeadersOnRESTRequests(t *testing.T) { if _, err := client.Schema.List(context.Background()); err != nil { t.Fatalf("schema list: %v", err) } + got := <-headers if v := got.Values(tc.header); len(v) != 1 { t.Fatalf("want exactly one %s header, got %v", tc.header, v) } @@ -406,14 +415,7 @@ func TestConfiguredHeadersOnRESTRequests(t *testing.T) { // TestConfiguredHeadersAreCopied: mutating the caller's map after NewClient // must not change what later requests send. func TestConfiguredHeadersAreCopied(t *testing.T) { - var got http.Header - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - got = r.Header.Clone() - w.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(w, `[]`) - })) - defer srv.Close() - + srv, captured := headerCaptureServer(t) headers := map[string]string{"X-Tenant-Id": "acme"} client := NewClient(Config{ BaseURL: srv.URL, @@ -426,7 +428,7 @@ func TestConfiguredHeadersAreCopied(t *testing.T) { if _, err := client.Schema.List(context.Background()); err != nil { t.Fatalf("schema list: %v", err) } - if v := got.Get("X-Tenant-Id"); v != "acme" { + if v := (<-captured).Get("X-Tenant-Id"); v != "acme" { t.Fatalf("want the value captured at construction, got %q", v) } } diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index c1fc8e2a..6004ba17 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -119,7 +119,7 @@ stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) ## Query Builder -Returned by `tableRef.Select()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: +Returned by `tableRef.Select(...)` or `tableRef.SelectAll()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: ```go page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) From ed2eacecd0ad17fa7573e973432fbcc18d89399f Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 09:57:52 -0400 Subject: [PATCH 42/59] docs(cov): compress the nested-module coverage comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The go-sdk plumbing explained the same fact — clients/go is a nested module, so the root module's -coverpkg can't reach it and its coverage is gated on its own instead of merged into the Go total — in four places at length. Keep it once, next to suites.go-sdk in .testcoverage.yml, and leave one-line pointers at the other sites. Comments only: no Makefile recipe, Go identifier, YAML key or threshold changed. --- .github/dependabot.yml | 10 ++-------- .github/workflows/ci.yml | 8 ++------ .testcoverage.yml | 20 ++++++-------------- AGENTS.md | 8 +------- Makefile | 36 ++++++++++------------------------- scripts/cov/main.go | 41 ++++++++++------------------------------ 6 files changed, 31 insertions(+), 92 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 38d5c98c..ba89ab59 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,13 +1,7 @@ version: 2 updates: - # Go modules - # - # TWO directories, not one — the same trap as github-actions below: - # `directory: /` covers the root module only, and Dependabot does not - # descend into nested modules. clients/go is its own module, so its - # go.mod needs its own entry here. That module is stdlib-only today - # (no `require` block, no go.sum), so this catches the first dependency - # it takes on rather than closing a gap that already exists. + # Go modules. TWO directories: Dependabot does not descend into nested + # modules, so clients/go needs its own entry (stdlib-only today). - package-ecosystem: gomod directories: - / diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index deebc508..c6c4d15f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,12 +206,8 @@ jobs: go-cache-suffix: "-unit" - name: Run Go unit tests + SDK vitest + Go SDK tests run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 - # This one fragment carries THREE suites' data: unit covdata, ts-unit - # (vitest/istanbul) and go-sdk covdata from the nested clients/go - # module — every target above ran under COV_DEFER, so the `coverage` - # job renders and gates all three. No per-suite paths here on purpose: - # uploading the whole tmp/coverage tree means a new suite collected by - # this job needs no workflow change. + # One fragment, three suites (unit, ts-unit, go-sdk) — all deferred to + # the `coverage` job. Whole-tree path so a new suite needs no edit here. - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/.testcoverage.yml b/.testcoverage.yml index ee02af8d..0d0595ed 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -12,9 +12,8 @@ # together, so `threshold.total` below applies to the *project-wide* # coverage number — unit alone covers only `./internal/...` packages # exercised by `*_test.go` files (below total threshold); merged -# coverage adds integration- and e2e-only paths and clears it. The -# nested-module suites (go-sdk) are gated separately and never merged -# in — see suites.go-sdk. +# coverage adds integration- and e2e-only paths and clears it. +# Nested-module suites (go-sdk) are gated separately — see suites.go-sdk. profile: tmp/coverage/total/coverage.txt local-prefix: github.com/Wave-RF/WaveHouse @@ -32,17 +31,10 @@ suites: unit: 80 integration: 20 e2e: 60 - # Go SDK (clients/go) — a NESTED module, so its coverage is rendered and - # gated on its own and is deliberately NOT part of the merged Go total - # above. Nothing from clients/go can leak into that total: the root - # module can't see a nested one (`go list ./...` at the repo root never - # yields clients/go), so the unit/integration/e2e `-coverpkg=./...` never - # reaches these files — which is also why there is no `^clients/go/` - # entry under exclude.paths; there is nothing to exclude. - # Measured 82.7% when this floor was set (SDK package 88.6%, - # cmd/wavehouse-codegen 55.8%). 75 leaves the same order of headroom - # unit's 80 leaves under its real ~91%. Raise it as the codegen - # command's tests fill in. + # Go SDK (clients/go) — a NESTED module, so the root module's + # `-coverpkg=./...` can never reach it: rendered and gated on its own, + # never merged into the total above, nothing to add under exclude.paths. + # 75 vs 82.7% measured; the headroom is the codegen CLI. Raise as it fills. go-sdk: 75 # TypeScript SDK suites — see scripts/cov for the merge / render logic. # vitest gates via --coverage.thresholds.statements; the merged ts-total diff --git a/AGENTS.md b/AGENTS.md index abbb182c..cf72a163 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -417,13 +417,7 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) clients/ts/ → TypeScript SDK (@wavehouse/sdk) -clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) - wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) - query_builder.go, table.go → Structured query builder + per-table typed client - stream.go, live_query.go → SSE streaming + live queries - pipes.go, policy.go, schema.go, dlq.go, sys.go → Subsystem clients (pipes, policy, schema, DLQ, health) - cmd/wavehouse-codegen/main.go → Codegen CLI - testdata/wire_cases.json → Wire-format conformance fixtures +clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) — nested module; file layout mirrors clients/ts/, plus cmd/wavehouse-codegen/ and testdata/wire_cases.json internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/policy/pipes endpoints) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) internal/cache/ → Caching (interface + L1/L2/tiered implementations) diff --git a/Makefile b/Makefile index a4696334..fbc1fa92 100644 --- a/Makefile +++ b/Makefile @@ -199,8 +199,7 @@ ACTIONLINT := $(LOCAL_BIN)/actionlint-$(ACTIONLINT_VERSION) COV_UNIT := tmp/coverage/unit COV_INT := tmp/coverage/integration COV_E2E := tmp/coverage/e2e -# go-sdk is the nested module at clients/go — same layout, own gate, but -# deliberately outside COV_TOTAL (see test-go-sdk below). +# Nested module at clients/go — same layout, own gate, outside COV_TOTAL. COV_GOSDK := tmp/coverage/go-sdk COV_TOTAL := tmp/coverage/total @@ -541,8 +540,7 @@ fix-prose: $(MISSPELL) # markdownlint along behind it). # # Leaves (16): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, -# lint-go-sdk (golangci) + verify-go-sdk (go vet + gofumpt), both on the -# nested clients/go module, on the Go +# lint-go-sdk + verify-go-sdk (both on the nested clients/go module) on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) + test-md-rules (node --test over the WH001/WH002 fixtures) # for JS/TS + Markdown + prose; lint-sh (shellcheck), lint-gha (actionlint), @@ -804,25 +802,12 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi -# test-go-sdk: unit tests for clients/go/ — a nested Go module (its own -# go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and -# needs its own target. -race because the SDK's streaming subsystem is the -# most concurrent code in the repo. -# -# Coverage is collected exactly like the root-module Go suites (covdata into -# tmp/coverage//data via -test.gocoverdir), so `cov render go-sdk` -# renders + gates it with no new machinery and CI's coverage fragment — -# `path: tmp/coverage` on the unit job, which already runs this target — -# carries it to the `coverage` job unchanged. -coverpkg=./... resolves -# inside clients/go, so the denominator is the SDK package + the codegen -# command, nothing from the server. -# -# The go-sdk suite is NOT part of the merged Go total: a nested module is -# invisible to the root module (`go list ./...` at the repo root never -# yields clients/go), so the other suites' -coverpkg=./... cannot reach -# these files — they can't leak into tmp/coverage/total, and no -# exclude.paths entry is needed to keep them out. Same separation the TS -# SDK gets via ts-*. Gate: suites.go-sdk in .testcoverage.yml. +# test-go-sdk: unit tests for the nested clients/go module — outside +# test-unit's scope, so it needs its own target (-race: the SDK's streaming +# subsystem is the most concurrent code in the repo). Covdata lands in the +# same layout as the root-module suites, so `cov render go-sdk` gates it with +# no new machinery, but it is never merged into the Go total — see the go-sdk +# comment in .testcoverage.yml. .PHONY: test-go-sdk test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render coverage + gate threshold @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" @@ -831,9 +816,8 @@ test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render cov -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi -# test-conformance-ts: the TS half of the cross-SDK wire-format conformance -# suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). -# Both replay clients/go/testdata/wire_cases.json. +# test-conformance-ts: TS half of the cross-SDK wire-format conformance suite +# (Go half: clients/go/conformance_test.go); both replay the same fixture. .PHONY: test-conformance-ts test-conformance-ts: build-ts ## Run TS SDK wire-format conformance against the shared fixture @printf "$(CYAN)==> Running TS wire-format conformance...$(RESET)\n" diff --git a/scripts/cov/main.go b/scripts/cov/main.go index 2a3948bb..f543676a 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -59,28 +59,14 @@ const ( // see the ts-total path below. var goSuites = []string{"unit", "integration", "e2e"} -// Go suites that produce covdata in the same layout as goSuites and are -// rendered + gated identically, but are deliberately NOT merged into the -// Go total: they come from a NESTED module (clients/go has its own -// go.mod). The root module cannot see a nested one — `go list ./...` at -// the repo root never yields clients/go — so the goSuites' -coverpkg=./... -// can't reach these files in the first place; they can only ever be in -// tmp/coverage/total if we put them there, which we don't. Folding a -// shipped client library into the server's project-wide number (and into -// the README badge `cov badge` derives from it) would move that number for -// reasons that have nothing to do with the server, so the SDK gets its own -// floor instead — the same separation the TS SDK gets via ts-*. +// Go suites rendered and gated like goSuites but never merged into the Go +// total: they come from the nested clients/go module — see the go-sdk +// comment in .testcoverage.yml. var standaloneGoSuites = []string{"go-sdk"} -// suiteModuleDir maps a suite to the module directory its covdata was -// produced in, for suites that aren't the root module. `go tool cover -// -html` reads the source of every package named in the profile and -// resolves it through the module in the process's working directory, so a -// nested module's profile has to be rendered from inside that module — -// from the repo root the tool fails with "no required module provides -// package github.com/Wave-RF/WaveHouse/clients/go/...". `go tool covdata -// textfmt` has no such constraint (it only reads the covdata files), so -// only the HTML step needs the chdir. +// suiteModuleDir maps a non-root-module suite to the module directory its +// covdata came from. `go tool cover -html` resolves the profile's package +// paths through the module in its working directory, so it must run there. var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} // TypeScript SDK suites (vitest). ts-unit comes from clients/ts; ts-e2e @@ -246,11 +232,8 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int return rows, total, covered, htmlOut, nil } -// renderHTML turns a textfmt profile into the clickable HTML report. For a -// suite whose covdata came from a nested module (see suiteModuleDir) the -// tool runs with that module as its working directory — otherwise it can't -// resolve the profile's package paths to source and bails — so the profile -// and output paths are made absolute first. +// renderHTML turns a textfmt profile into the clickable HTML report, running +// from the suite's own module (hence absolute paths) when it is a nested one. func renderHTML(suite, profile, htmlOut string) error { dir, nested := suiteModuleDir[suite] if !nested { @@ -603,9 +586,7 @@ func report(c *config) error { }) } - // --- Nested-module Go suites (own gate, below the Go total) --- - // Rendered and gated exactly like the suites above, but listed after - // the total they are deliberately not part of — see standaloneGoSuites. + // --- Nested-module Go suites: own gate, not in the total above --- for i, s := range standaloneGoSuites { if !hasCovdata(filepath.Join(root, s, "data")) { rows = append(rows, reportRow{name: s, pct: "n/a", rule: i == 0}) @@ -990,9 +971,7 @@ func meetsThreshold(covered, total, threshold int) bool { // flag, … args; the only variable bits are paths we computed ourselves. func sh(name string, args ...string) error { return shIn("", name, args...) } -// shIn is sh with an explicit working directory ("" = inherit ours) — for -// the one tool that cares which module it runs in, `go tool cover -html` -// on a nested module's profile. See renderHTML. +// shIn is sh with an explicit working directory ("" = inherit ours). // #nosec G204,G702 — name and args are not user input. func shIn(dir, name string, args ...string) error { cmd := exec.CommandContext(context.Background(), name, args...) From 20bb752c7b0f9a58ac57e5f8349be5534d69f251 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 10:04:09 -0400 Subject: [PATCH 43/59] refactor(sdk/go): cut comment bloat and mechanical verbosity Tighten the Go SDK's prose down to what a future editor actually needs: one-line godoc on every exported symbol, and an inline comment kept only where it encodes a non-obvious invariant (header precedence, redirect refusal while credentialed, fail-closed instant comparison, terminal StatusClosed, mu-serialized channel close). Removed: rationale essays, cross-SDK comparisons, historical narration, and paragraphs restating the adjacent line. Code changes are behavior-preserving: - stream.go: sseError() helper replaces nine repeated *Error literals; newController() replaces the duplicated controller literal; order() and time.Time.Compare collapse three identical sign switches. - query_builder.go: buildAST assigns omitempty-elided fields directly. - table.go: insertSingle folds its result assembly into one return. - wavehouse.go: maps.Clone for the configured-header copy. - errors.go/stream.go: drop redundant locals. The exported API is unchanged: all 209 declarations compare byte-identical under `go doc -all` with whitespace and doc prose normalized. --- clients/go/cmd/wavehouse-codegen/main.go | 91 ++----- clients/go/dlq.go | 9 +- clients/go/errors.go | 19 +- clients/go/http.go | 26 +- clients/go/live_query.go | 25 +- clients/go/pipes.go | 8 +- clients/go/query_builder.go | 73 ++---- clients/go/schema.go | 1 - clients/go/stream.go | 315 ++++++++--------------- clients/go/sys.go | 4 +- clients/go/table.go | 32 +-- clients/go/types.go | 71 ++--- clients/go/wavehouse.go | 47 ++-- 13 files changed, 242 insertions(+), 479 deletions(-) diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index fa10cfaa..536488ac 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -26,8 +26,8 @@ type cliArgs struct { pkg string } -// flagValue consumes and returns the value following os.Args[*i], erroring -// out instead of silently falling back to the default when it's missing. +// flagValue consumes and returns the value following os.Args[*i], exiting +// rather than silently falling back to the default when it is missing. func flagValue(i *int) string { flag := os.Args[*i] *i++ @@ -103,12 +103,11 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) } - // Server returns either []tableSchema or map[string]tableSchema. + // The server returns either []tableSchema or map[string]tableSchema. var raw json.RawMessage if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { return nil, fmt.Errorf("read schema response: %w", err) } - // Try array first. var arr []tableSchema if err := json.Unmarshal(raw, &arr); err == nil { m := make(map[string]tableSchema, len(arr)) @@ -124,27 +123,10 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc return m, nil } -// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to a -// Go type name suitable for a JSON struct field. -// -// We deliberately don't import clickhouse-go's type catalog -// (github.com/ClickHouse/clickhouse-go/v2/lib/column) for this. It's public -// and does expose a real ClickHouse-type-string parser — -// column.Type(chType).Column(name, sc).ScanType() — but it answers a -// different question than the one we're asking. That catalog maps to the Go -// types the *driver* scans query results into over the native protocol -// (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for -// Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types -// that round-trip cleanly through the JSON the /v1/ops/schema and query -// endpoints actually speak. ClickHouse's JSON output renders DateTime as -// "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default -// time.Time JSON unmarshaling; big integers and decimals are similarly -// rendered as JSON strings, not driver-native types. Adopting the driver's -// ScanType() as-is would produce generated structs that don't unmarshal the -// server's actual JSON, and would drag uuid/decimal/orb/net imports into -// generated output that today has zero non-stdlib dependencies. So we keep -// the hand-rolled JSON-oriented mapping below, informed by (but not bound -// to) the type set clickhouse-go's lib/column recognizes. +// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to +// a Go type name suitable for a JSON struct field. The mapping targets what +// round-trips through the server's JSON, not clickhouse-go's native scan +// types, and keeps generated output free of non-stdlib imports. func chTypeToGo(chType string) string { // Unwrap Nullable → pointer. if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { @@ -155,10 +137,8 @@ func chTypeToGo(chType string) string { if strings.HasPrefix(chType, "LowCardinality(") && strings.HasSuffix(chType, ")") { return chTypeToGo(chType[15 : len(chType)-1]) } - // Unwrap SimpleAggregateFunction(func, InnerType) — readable columns in - // AggregatingMergeTree/SummingMergeTree rollup tables. The value on the - // wire is just InnerType; the aggregate function name only describes how - // merges combine rows. + // Unwrap SimpleAggregateFunction(func, InnerType): the wire value is just + // InnerType, the function name only describes how merges combine rows. if strings.HasPrefix(chType, "SimpleAggregateFunction(") && strings.HasSuffix(chType, ")") { inner := chType[len("SimpleAggregateFunction(") : len(chType)-1] if comma := findTopLevelComma(inner); comma != -1 { @@ -173,9 +153,7 @@ func chTypeToGo(chType string) string { chType == "UUID", strings.HasPrefix(chType, "DateTime"), strings.HasPrefix(chType, "Date"), - // Time/Time64 are ClickHouse's newer time-of-day types (distinct - // from DateTime); same JSON-string-not-RFC3339 story applies. - strings.HasPrefix(chType, "Time"), + strings.HasPrefix(chType, "Time"), // Time/Time64 time-of-day types strings.HasPrefix(chType, "Enum8("), strings.HasPrefix(chType, "Enum16("), chType == "IPv4", @@ -184,10 +162,10 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric. Generated structs target /v1/query and /v1/pipes/*, where the - // server re-marshals ClickHouse values, so 64-bit integers arrive as - // unquoted JSON numbers. /v1/ops/query forwards ClickHouse's own JSON, - // which quotes them — use SQL[map[string]any] there. + // Generated structs target /v1/query and /v1/pipes/*, where the server + // re-marshals values so 64-bit integers arrive as unquoted JSON numbers. + // /v1/ops/query forwards ClickHouse's own JSON, which quotes them — use + // SQL[map[string]any] there. if mapped, ok := map[string]string{ "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", @@ -197,32 +175,24 @@ func chTypeToGo(chType string) string { } switch { case strings.HasPrefix(chType, "Decimal"): - // Decimals are marshaled as quoted strings on the structured path - // (shopspring decimal.MarshalJSON quotes by default). - return "string" + return "string" // marshaled as a quoted string on the structured path case strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), strings.HasPrefix(chType, "Int128"), strings.HasPrefix(chType, "Int256"): - // 128/256-bit ints scan into *big.Int server-side and marshal as - // unquoted JSON numbers of arbitrary width — json.Number preserves - // them exactly where int64/uint64 would overflow. + // Arbitrary-width unquoted JSON numbers; int64/uint64 would overflow. return "json.Number" } - // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { inner := chTypeToGo(chType[6 : len(chType)-1]) - // Array(UInt8) is asymmetric on the wire: ingest requires a real JSON - // array, but /v1/query responses currently base64-encode it (the - // server scans into []byte and encoding/json base64s that — #436). - // json.RawMessage is the - // only shape that round-trips both directions without a decode error. + // Array(UInt8) is asymmetric on the wire — ingest takes a JSON array, + // /v1/query returns base64 (#436) — and only json.RawMessage + // round-trips both without a decode error. if inner == "uint8" { return "json.RawMessage" } return "[]" + inner } - // Map. if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { inner := chType[4 : len(chType)-1] comma := findTopLevelComma(inner) @@ -270,9 +240,8 @@ func pascalCase(s string) string { if result == "" { return result } - // Go identifiers can't start with a digit (e.g. a table named - // "2fa_events" would otherwise produce the invalid identifier - // "2faEvents"). Prefix with "X" to keep it a valid, exported name. + // Go identifiers can't start with a digit, so "2fa_events" needs a prefix + // to stay a valid exported name. if unicode.IsDigit(rune(result[0])) { // digits are ASCII; no rune-slice needed result = "X" + result } @@ -307,10 +276,8 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { sb.WriteString("import \"encoding/json\"\n\n") } - // pascalCase is not injective ("user_id" and "userId" both yield - // "UserId"), and format.Source only parses — it doesn't type-check — so - // a duplicate identifier would be written as a non-compiling file with a - // success message. Fail loudly instead. + // pascalCase is not injective and format.Source only parses, so a + // collision would otherwise be written out as a non-compiling file. seenTypes := make(map[string]string, len(names)) for _, name := range names { schema := schemas[name] @@ -330,10 +297,9 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { seenFields[fieldName] = col.Name jsonTag := col.Name if col.HasDefault { - // Pointer + omitempty is the Go spelling of the TS codegen's - // `field?: T`: nil omits the field (server default applies), - // while a pointer to the zero value still sends an explicit - // 0/false/"" instead of silently dropping it. + // Pointer + omitempty means nil omits the field so the server + // default applies, while a pointer to the zero value still + // sends an explicit 0/false/"". jsonTag += ",omitempty" if !strings.HasPrefix(goType, "*") { goType = "*" + goType @@ -370,9 +336,8 @@ func main() { os.Exit(1) } - // gofmt the output. A failure here means the generated source is not - // valid Go (e.g. a table/column name produced an invalid identifier); - // don't write unusable output and claim success. + // A format failure means the generated source is not valid Go; don't write + // unusable output and claim success. formatted, err := format.Source([]byte(output)) if err != nil { fmt.Fprintf(os.Stderr, "Error: generated code is not valid Go: %v\n", err) diff --git a/clients/go/dlq.go b/clients/go/dlq.go index 65712765..327000ed 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -6,12 +6,9 @@ import ( "net/url" ) -// DLQNamespace provides admin-only dead-letter-queue statistics. -// -// The server registers /v1/ops/dlq/stats only when the DLQ is enabled, so on a -// deployment with dlq.enabled: false these calls return an [*Error] with -// Status 404 — "the DLQ is switched off", not "the DLQ is empty". Check -// Status before reading a zero DLQStats as a healthy result. +// DLQNamespace provides admin-only dead-letter-queue statistics. A deployment +// with the DLQ disabled answers these calls with an [*Error] of Status 404 +// rather than an empty DLQStats, so check Status before reading a zero total. type DLQNamespace struct { ctx httpContext createStream func(table string, opts *StreamOptions) *StreamController diff --git a/clients/go/errors.go b/clients/go/errors.go index 8ec3b2e8..dfa0e7f6 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -11,16 +11,11 @@ import ( // Error is the structured error returned by all SDK operations. Use // [errors.As] to extract it from wrapped errors. type Error struct { - // Status is the HTTP status code (0 for network/abort errors). - Status int `json:"status"` - // Code is a machine-readable error code (e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED"). - Code string `json:"code"` - // Message is a human-readable description. - Message string `json:"message"` - // Details contains the full parsed error body, if available. - Details map[string]any `json:"details,omitempty"` - // Retryable indicates whether the request can be retried. - Retryable bool `json:"retryable"` + Status int `json:"status"` // 0 for network/abort errors + Code string `json:"code"` // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" + Message string `json:"message"` + Details map[string]any `json:"details,omitempty"` // full parsed error body, when present + Retryable bool `json:"retryable"` } func (e *Error) Error() string { @@ -44,13 +39,11 @@ func parseErrorResponse(res *http.Response) *Error { _ = json.Unmarshal(raw, &body) } - var msg string + msg := http.StatusText(res.StatusCode) if s, ok := body["error"].(string); ok { msg = s } else if s, ok := body["message"].(string); ok { msg = s - } else { - msg = http.StatusText(res.StatusCode) } retryable := res.StatusCode >= 500 || res.StatusCode == http.StatusTooManyRequests diff --git a/clients/go/http.go b/clients/go/http.go index eaab0e94..eec76588 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -30,9 +30,8 @@ type httpContext struct { } // applyConfiguredHeaders writes the client's configured headers onto a request. -// Call it *before* the SDK sets its own headers: Set replaces, so whatever the -// SDK writes afterwards wins a collision. http.Header canonicalizes names, so -// "x-tenant" and "X-Tenant" are the same entry. +// Call it before the SDK sets its own headers: Set replaces, so the SDK's +// later writes win any collision. func applyConfiguredHeaders(h http.Header, configured map[string]string) { for k, v := range configured { h.Set(k, v) @@ -49,8 +48,8 @@ type requestOptions struct { params url.Values } -// doRequest is the internal fetch wrapper with auth, retry, and backoff. -// It decodes the response body into dst (unless dst is nil). +// doRequest issues a request with auth, retry and backoff, decoding the +// response body into dst unless dst is nil. func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst any) error { reqURL := buildURL(hctx.baseURL, opts.path, opts.params) ct := opts.contentType @@ -85,8 +84,8 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a var lastErr error maxAttempts := hctx.maxRetries + 1 - // Retries all methods including POST. For /v1/ingest, at-least-once delivery - // is the documented contract; dedup is the server-side safety net. + // Retries every method including POST: /v1/ingest documents at-least-once + // delivery, with server-side dedup as the safety net. for attempt := range maxAttempts { var bodyReader io.Reader if bodyBytes != nil { @@ -106,7 +105,6 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a res, err := hctx.httpClient.Do(req) if err != nil { - // Context cancellation — return immediately, no retry. if ctx.Err() != nil { return errAborted } @@ -146,7 +144,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a apiErr := parseErrorResponse(res) _ = res.Body.Close() - // 503/429 with Retry-After: wait the specified duration (capped). + // 503/429 with Retry-After: wait the header's duration, capped. if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, retryAfterDelay(ra, attempt)); sleepErr != nil { @@ -157,7 +155,6 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a } } - // Retryable server errors (5xx). if apiErr.Retryable && attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { return errAborted @@ -186,9 +183,8 @@ func buildURL(base, path string, params url.Values) string { func retryAfterDelay(ra string, attempt int) time.Duration { delay := backoff(attempt) if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { - // Compare before converting: time.Duration(secs) * time.Second wraps - // negative past ~9.2e9 seconds, and min() below would then pick the - // negative value, firing the retry timer instantly. + // Compare before converting: secs*time.Second wraps negative past + // ~9.2e9 seconds and min() below would then fire the timer instantly. if secs > int(maxRetryAfter/time.Second) { return maxRetryAfter } @@ -203,8 +199,8 @@ func retryAfterDelay(ra string, attempt int) time.Duration { func backoff(attempt int) time.Duration { ms := 1000 * math.Pow(2, float64(attempt)) - // ±20% jitter so clients failing at the same moment don't retry in - // lockstep; capped after jitter so the documented 30s max holds. + // ±20% jitter so simultaneous failures don't retry in lockstep; capped + // after jitter so the documented 30s maximum holds. ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic return time.Duration(min(ms, 30000)) * time.Millisecond } diff --git a/clients/go/live_query.go b/clients/go/live_query.go index d47ce618..715ca63d 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -20,8 +20,8 @@ type LiveQueryHandle struct { closed bool } -// newLiveQuery starts a live query: opens the stream immediately, fetches -// historical data, deduplicates buffered events, then goes live. +// newLiveQuery buffers stream events while the historical fetch runs, then +// replays the buffer minus anything the backfill already delivered. func newLiveQuery( stream *StreamController, fetchFn func(ctx context.Context) ([]map[string]any, error), @@ -34,9 +34,8 @@ func newLiveQuery( buffering: true, } - // Step 1: Subscribe to live events and buffer them. User callbacks are - // invoked outside lq.mu so a subscriber may call Close() without - // deadlocking. + // User callbacks are invoked outside lq.mu so a subscriber may call Close + // without deadlocking. lq.unsub = stream.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { lq.mu.Lock() @@ -66,14 +65,12 @@ func newLiveQuery( }, }) - // Step 2–5: Fetch historical and flush. go func() { rows, err := fetchFn(ctx) if ctx.Err() != nil || lq.isClosed() { return } - // Step 3: Deliver initial snapshot. if sub.Initial != nil { sub.Initial(rows, err) } @@ -86,10 +83,9 @@ func newLiveQuery( return } - // Step 4: Dedup bound — the maximum backfilled timestamp, compared as - // parsed times. OrderBy(..., "desc") makes the *last* row the oldest, - // and RFC3339 strings with varying fractional digits don't sort - // lexically, so neither "last row" nor raw string compare is safe. + // Dedup bound: the maximum backfilled timestamp as a parsed time. The + // last row can be the oldest, and RFC3339 strings with varying + // fractional digits do not sort lexically. var lastTS time.Time for _, row := range rows { if s, ok := row["received_timestamp"].(string); ok { @@ -99,9 +95,8 @@ func newLiveQuery( } } - // Step 5: Flush buffered events. buffering stays true until the - // buffer is provably empty under the lock — prevents concurrent - // sub.Next calls and preserves delivery order. + // buffering stays true until the buffer is provably empty under the + // lock, which keeps delivery ordered and single-threaded. for { lq.mu.Lock() if lq.closed { @@ -122,8 +117,6 @@ func newLiveQuery( return } // Skip events already delivered in the backfill. - // Sub-millisecond received_timestamp precision makes - // boundary collisions rare. if !lastTS.IsZero() { if ts, perr := time.Parse(time.RFC3339Nano, event.Timestamp); perr == nil && !ts.After(lastTS) { continue diff --git a/clients/go/pipes.go b/clients/go/pipes.go index 5501fa86..a038ec73 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -96,11 +96,9 @@ func (p *PipeRef) FetchUntyped(ctx context.Context) ([]map[string]any, error) { return Fetch[map[string]any](ctx, p) } -// Stream opens a live event stream from the pipe's underlying query. -// -// This streams by table name, using the pipe's own name as the table — it -// only works when the pipe name is also a valid table name. This matches -// the TS SDK's PipeRef.stream(), which has the same limitation. +// Stream opens a live event stream from the pipe's underlying query. It +// subscribes by table name using the pipe's own name, so it only works when +// the pipe name is also a valid table name. func (p *PipeRef) Stream(opts *StreamOptions) *StreamController { return p.createStream(p.name, opts) } diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 8d0e0212..dc9e83bc 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -8,8 +8,8 @@ import ( "net/url" ) -// DefaultLimit is applied when no explicit limit is set — deliberately tighter -// than the backend's DefaultMaxRows (10000) safety cap. +// DefaultLimit is applied when no explicit limit is set. It is deliberately +// tighter than the backend's DefaultMaxRows (10000) safety cap. const DefaultLimit = 1000 // queryState is the immutable core of a QueryBuilder. @@ -26,8 +26,8 @@ type queryState struct { cacheTTL *int // client-side only, not sent to server (#280) } -// QueryBuilder builds structured queries. Immutable — every chain method -// returns a new builder. Use Fetch or FetchUntyped to execute. +// QueryBuilder builds structured queries. It is immutable: every chain method +// returns a new builder. Use [FetchTyped] or FetchUntyped to execute. type QueryBuilder struct { ctx httpContext createStream func(table string, opts *StreamOptions) *StreamController @@ -173,11 +173,9 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro hasMore := limit > 0 && len(rows) >= limit page := &Page[Row]{Data: rows, HasMore: hasMore} - // Attach Next whenever we have an order column to build a cursor from. - // This doesn't check that the order column is present in the row - // projection — a Select() that omits it means fetchNextTyped can't find - // a cursor value and will quietly return an empty page (matches the TS - // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). + // Attached whenever there is an order column to build a cursor from; a + // projection that omits that column yields an empty next page rather than + // an error. if hasMore && len(q.state.orderBy) > 0 { page.Next = func(ctx context.Context) (*Page[Row], error) { return fetchNextTyped(ctx, q, rows) @@ -202,9 +200,9 @@ func (q *QueryBuilder) Stream(opts *StreamOptions) *StreamController { return newFilteredStreamController(raw, q.state.filters, q.state.columns) } -// LiveQuery starts a live query: fetches historical data, then streams live -// updates. The subscriber's Initial is called once, then Next fires for each -// live event. Returns a LiveQuery handle with a Close method. +// LiveQuery fetches historical data, then streams live updates: the +// subscriber's Initial fires once, then Next fires per live event. Close the +// returned handle to stop it. func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *LiveQueryHandle { stream := q.Stream(opts) fetchFn := func(ctx context.Context) ([]map[string]any, error) { @@ -247,22 +245,14 @@ func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { ast.SelectAll = true } - if hasAggs { - ast.Aggregations = q.state.aggregations - } - if len(q.state.filters) > 0 { - ast.Filters = q.state.filters - } - if len(q.state.groupBy) > 0 { - ast.GroupBy = q.state.groupBy - } - if len(q.state.orderBy) > 0 { - ast.OrderBy = q.state.orderBy - } + // Empty slices and a nil TimeRange are omitempty-elided, so these can be + // assigned unconditionally. + ast.Aggregations = q.state.aggregations + ast.Filters = q.state.filters + ast.GroupBy = q.state.groupBy + ast.OrderBy = q.state.orderBy ast.Limit = &effectiveLimit - if q.state.timeRange != nil { - ast.TimeRange = q.state.timeRange - } + ast.TimeRange = q.state.timeRange return ast } @@ -272,38 +262,27 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro } cursor := q.state.orderBy[0] - // Extract the last row's value for the cursor column. lastRow := any(prevRows[len(prevRows)-1]) m, ok := lastRow.(map[string]any) if !ok { - // TODO: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. - // UseNumber keeps typed int64 cursor values exact past 2^53. The - // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this - // protection: its rows were already decoded to float64 by - // encoding/json, so precision above 2^53 is gone before we get here — - // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or - // codegen structs — their 64-bit int columns are int64/uint64, and - // 128/256-bit are json.Number) when paging on >2^53 integer cursors. + // UseNumber keeps typed int64 cursor values exact past 2^53; rows that + // arrived through the untyped path were already float64 by then. raw, err := json.Marshal(lastRow) if err != nil { - // Row itself is unmarshalable (e.g. a func field absent from the - // response). Silently truncating the result set would look like - // normal end-of-pagination, so surface it. + // Surfaced rather than swallowed: an empty page here would be + // indistinguishable from normal end-of-pagination. return nil, fmt.Errorf("wavehouse: marshal cursor row: %w", err) } m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) dec.UseNumber() - // Decode error is deliberate: a Row that marshals to a non-object - // (FetchTyped[[]any], a scalar row type) leaves m empty and ends - // pagination below, same as an absent cursor column. Tracked in #452. + // Ignored deliberately: a Row that marshals to a non-object leaves m + // empty and ends pagination below, as an absent cursor column does (#452). _ = dec.Decode(&m) } lastValue, exists := m[cursor.Column] if !exists { - // Cursor column wasn't in the projection (e.g. Select() omitted it) — - // no cursor value to page from, so end pagination quietly rather than - // erroring. Matches the TS SDK's _fetchNext(). + // No cursor value to page from; end pagination quietly. return &Page[Row]{}, nil } @@ -313,8 +292,8 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro } next := q.clone(func(s *queryState) { - // Replace an existing cursor filter instead of appending — otherwise - // page N carries N stacked filters on the cursor column. + // Replace an existing cursor filter rather than appending, or page N + // would carry N stacked filters on the cursor column. for i := range s.filters { if s.filters[i].Column == cursor.Column && s.filters[i].Op == cursorOp { s.filters[i].Value = lastValue diff --git a/clients/go/schema.go b/clients/go/schema.go index c79dcb11..6ac38fe4 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -12,7 +12,6 @@ type SchemaNamespace struct { // List returns all table schemas discovered from ClickHouse. Admin-only. func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { - // The backend returns []TableSchema; transform to map[string]TableSchema. var raw []TableSchema if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", diff --git a/clients/go/stream.go b/clients/go/stream.go index a9388896..3a652be1 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -30,15 +30,21 @@ type StreamController struct { closed bool } -// newStreamController opens an SSE connection for the given table. -func newStreamController(hctx httpContext, table string, opts *StreamOptions) *StreamController { - ctx, cancel := context.WithCancel(context.Background()) - sc := &StreamController{ - status: StatusConnecting, +// newController builds a controller whose event channel buffers from +// construction, so events predating the first Events call are not lost. +func newController(status StreamStatus, cancel context.CancelFunc) *StreamController { + return &StreamController{ + status: status, eventCh: make(chan StreamEvent, 256), cancel: cancel, done: make(chan struct{}), } +} + +// newStreamController opens an SSE connection for the given table. +func newStreamController(hctx httpContext, table string, opts *StreamOptions) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := newController(StatusConnecting, cancel) go sc.run(ctx, hctx, table, opts) return sc } @@ -50,17 +56,16 @@ func (sc *StreamController) Status() StreamStatus { return sc.status } -// Subscribe registers callbacks for stream events. Returns an unsubscribe -// function. The subscriber's Status callback fires immediately with the -// current status. +// Subscribe registers callbacks and returns an unsubscribe function. The +// subscriber's Status callback fires immediately with the current status. func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { sc.mu.Lock() sc.subscribers = append(sc.subscribers, sub) currentStatus := sc.status sc.mu.Unlock() - // Benign race: setStatus also calls the subscriber, so a stale - // status here is immediately followed by the correct one. + // Benign race: setStatus also calls the subscriber, so a stale status here + // is immediately followed by the correct one. if sub.Status != nil { sub.Status(currentStatus) } @@ -77,12 +82,9 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { } } -// Events returns a read-only channel that receives stream events. -// The channel is closed when the stream closes. Events buffer into it from -// stream construction (matching the TS SDK), so events that arrive before -// the first Events() call are not lost. A Subscribe-only consumer that never -// calls Events() at most fills the 256-slot buffer and trips the one-time -// drop log. +// Events returns a read-only channel of stream events, closed when the stream +// closes. A consumer that never reads it fills the buffer and trips a +// one-time drop log. func (sc *StreamController) Events() <-chan StreamEvent { return sc.eventCh } @@ -101,8 +103,6 @@ func (sc *StreamController) Connected(ctx context.Context) error { } sc.mu.Unlock() - // Poll — simple and correct. - // TODO: switch to a condition variable if polling shows up in profiles. ticker := time.NewTicker(50 * time.Millisecond) defer ticker.Stop() for { @@ -136,16 +136,15 @@ func (sc *StreamController) Close() { sc.closed = true sc.mu.Unlock() + // Deliberately does not wait on sc.done: callbacks run on the stream + // goroutine, so waiting would deadlock a Close made from a callback. sc.cancel() - // Don't block on <-sc.done: callbacks execute on the stream goroutine, - // so waiting here would deadlock if Close is called from a callback. } func (sc *StreamController) setStatus(s StreamStatus) { sc.mu.Lock() - // StatusClosed is terminal: a filtered wrapper's inner controller can - // have copied its subscriber slice before unsub, so a stale Status - // callback may land after Close — it must not resurrect the status. + // StatusClosed is terminal: a stale Status callback can land after Close + // and must not resurrect the status. if s == sc.status || sc.status == StatusClosed { sc.mu.Unlock() return @@ -162,8 +161,8 @@ func (sc *StreamController) setStatus(s StreamStatus) { } // snapshotSubs copies the subscriber list under mu so callbacks run unlocked. -// setStatus keeps its own inline copy: there the snapshot must share the -// critical section with the status write to keep callback order consistent. +// setStatus keeps its own inline copy so the snapshot shares a critical +// section with the status write. func (sc *StreamController) snapshotSubs() []*StreamSubscriber { sc.mu.Lock() defer sc.mu.Unlock() @@ -177,9 +176,7 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } - // Non-blocking send to the channel, which buffers from construction (TS - // parity) so events emitted before the first Events() call survive. - // Guarded by mu so the send and closeEventCh serialize — a late event can + // Guarded by mu so the send and closeEventCh serialize: a late event can // never hit a closed channel. sc.mu.Lock() defer sc.mu.Unlock() @@ -203,8 +200,8 @@ func (sc *StreamController) emitError(err error) { } } -// closeEventCh marks the controller closed and closes the events channel. -// Must serialize with emitEvent's send via mu. +// closeEventCh marks the controller closed and closes the events channel. It +// must serialize with emitEvent's send via mu. func (sc *StreamController) closeEventCh() { sc.mu.Lock() sc.closed = true @@ -240,19 +237,15 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } - // A connection that reached "live" resets the backoff so a long-lived - // stream doesn't inherit a maxed-out delay on its first drop. + // Reaching "live" resets the backoff so a long-lived stream doesn't + // inherit a maxed-out delay on its first drop. if live { attempt = 0 } if err != nil { - // connect classifies its own failures (SSE_AUTH_ERROR, - // SSE_NETWORK_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, - // SSE_READ_ERROR, HTTP_nnn), so pass the typed error straight - // through and let Retryable decide whether to reconnect. A - // non-retryable error is terminal: reconnecting can't fix a bad - // token, a missing table, or a proxy answering with HTML. + // connect classifies its own failures, so Retryable decides + // whether to reconnect: a non-retryable error is terminal. var apiErr *Error if errors.As(err, &apiErr) { sc.emitError(apiErr) @@ -260,51 +253,31 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } } else { - // Unclassified — retry, but keep the generic code so callers - // can still match on it. - sc.emitError(&Error{ - Status: 0, - Code: "SSE_ERROR", - Message: err.Error(), - Retryable: true, - }) + sc.emitError(sseError(0, "SSE_ERROR", err.Error(), true)) } } sc.setStatus(StatusReconnecting) - delay := backoff(attempt) - attempt++ - select { case <-ctx.Done(): return - case <-time.After(delay): + case <-time.After(backoff(attempt)): } + attempt++ } } -// connect opens a single SSE connection and reads events until it closes. -// Returns the last seen event ID (empty if none), whether the connection -// reached the live state, and any error. +// connect reads one SSE connection until it closes, returning the last seen +// event ID, whether the connection ever reached the live state, and any error. func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { u, err := url.Parse(hctx.baseURL + "/v1/stream") if err != nil { - return "", false, &Error{ - Status: 0, - Code: "SSE_CONNECT_ERROR", - Message: fmt.Sprintf("invalid baseURL: %v", err), - Retryable: false, - } + return "", false, sseError(0, "SSE_CONNECT_ERROR", fmt.Sprintf("invalid baseURL: %v", err), false) } - // A non-HTTP scheme can never carry SSE. Terminal, not retryable: retrying - // a ws:// or file:// baseURL just spins. + // A non-HTTP scheme can never carry SSE, so retrying one just spins. if u.Scheme != "http" && u.Scheme != "https" { - return "", false, &Error{ - Status: 0, - Code: "SSE_CONNECT_ERROR", - Message: fmt.Sprintf("baseURL scheme %q is not http or https", u.Scheme), - Retryable: false, - } + return "", false, sseError(0, "SSE_CONNECT_ERROR", + fmt.Sprintf("baseURL scheme %q is not http or https", u.Scheme), false) } q := u.Query() q.Set("table", table) @@ -312,19 +285,15 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table q.Set("since", since) } - // Auth: Go SDK uses Authorization header (not ?token= like browser EventSource). + // Auth travels in the Authorization header, not ?token= as browser + // EventSource requires. var authHeader string if hctx.auth != nil { token, err := hctx.auth(ctx) if err != nil { // Retryable: a token endpoint having a bad minute shouldn't tear // down a healthy long-lived stream. - return "", false, &Error{ - Status: 0, - Code: "SSE_AUTH_ERROR", - Message: err.Error(), - Retryable: true, - } + return "", false, sseError(0, "SSE_AUTH_ERROR", err.Error(), true) } if token != "" { authHeader = "Bearer " + token @@ -347,12 +316,9 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table client := hctx.httpClient credentialed := authHeader != "" || len(hctx.headers) > 0 if credentialed { - // Refuse to follow a redirect while carrying a credential. net/http - // drops Authorization on a cross-host hop but forwards custom headers - // verbatim, so following one would either downgrade the stream to - // default_role without saying so, or hand configured secrets to - // wherever the redirect points. Copy the client so a caller-supplied - // one keeps its own CheckRedirect for every other request. + // Never follow a redirect while carrying a credential: net/http drops + // Authorization across hosts but forwards custom headers verbatim. + // Copied so a caller-supplied client keeps its own CheckRedirect. c := *client c.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse @@ -365,54 +331,36 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if ctx.Err() != nil { return "", false, errAborted } - return "", false, &Error{ - Status: 0, - Code: "SSE_NETWORK_ERROR", - Message: err.Error(), - Retryable: true, - } + return "", false, sseError(0, "SSE_NETWORK_ERROR", err.Error(), true) } defer func() { _ = resp.Body.Close() }() if credentialed && resp.StatusCode >= 300 && resp.StatusCode < 400 { - return "", false, &Error{ - Status: resp.StatusCode, - Code: "SSE_REDIRECT", - Message: fmt.Sprintf( - "stream endpoint redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", - resp.Header.Get("Location")), - Retryable: false, - } + return "", false, sseError(resp.StatusCode, "SSE_REDIRECT", fmt.Sprintf( + "stream endpoint redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", + resp.Header.Get("Location")), false) } if resp.StatusCode != http.StatusOK { return "", false, parseErrorResponse(resp) } - // A 200 that isn't an event stream means something between the caller and - // WaveHouse answered — a captive portal or an auth gateway's login page. - // Without this check the stream sits in StatusLive and silently delivers - // nothing. + // A 200 that isn't an event stream means an intermediary answered; without + // this the stream would sit in StatusLive delivering nothing. if ct := resp.Header.Get("Content-Type"); !isEventStream(ct) { shown := ct if shown == "" { shown = "(none)" } - return "", false, &Error{ - Status: resp.StatusCode, - Code: "SSE_BAD_CONTENT_TYPE", - Message: fmt.Sprintf("expected Content-Type text/event-stream, got %s", shown), - Retryable: false, - } + return "", false, sseError(resp.StatusCode, "SSE_BAD_CONTENT_TYPE", + fmt.Sprintf("expected Content-Type text/event-stream, got %s", shown), false) } sc.setStatus(StatusLive) - // Parse SSE frames. scanner := bufio.NewScanner(resp.Body) - // 16 MiB max line: generous headroom over the ~1 MiB NATS MaxPayload - // ceiling on a single event envelope (oversized records are rejected at - // ingest publish and never reach the stream). + // 16 MiB max line: headroom over the ~1 MiB NATS MaxPayload ceiling on a + // single event envelope. scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) var eventID, dataLine string lastID := since @@ -425,7 +373,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table line := scanner.Text() if line == "" { - // Empty line = end of event frame. + // End of an event frame. if dataLine != "" { sc.handleSSEData(dataLine) // Track last event ID for reconnect gap-fill. @@ -438,8 +386,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table continue } - if strings.HasPrefix(line, ":") { - // Comment (keepalive or connected). Skip. + if strings.HasPrefix(line, ":") { // comment: keepalive or connected continue } @@ -456,16 +403,16 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table } if scanErr := scanner.Err(); scanErr != nil { - return lastID, true, &Error{ - Status: 0, - Code: "SSE_READ_ERROR", - Message: scanErr.Error(), - Retryable: true, - } + return lastID, true, sseError(0, "SSE_READ_ERROR", scanErr.Error(), true) } return lastID, true, nil } +// sseError builds a stream [*Error] with the SDK's SSE_* taxonomy. +func sseError(status int, code, msg string, retryable bool) *Error { + return &Error{Status: status, Code: code, Message: msg, Retryable: retryable} +} + // isEventStream reports whether a Content-Type header names text/event-stream, // ignoring any parameters (charset, boundary) and case. func isEventStream(contentType string) bool { @@ -486,25 +433,14 @@ type sseMessage struct { func (sc *StreamController) handleSSEData(data string) { var msg sseMessage if err := json.Unmarshal([]byte(data), &msg); err != nil { - // Delivered via the subscriber Error callback rather than the - // process-global logger, so consumers control visibility and a - // malformed-frame flood can't spam host-application logs. Payload - // deliberately omitted: event data can carry tenant/PII fields. - sc.emitError(&Error{ - Status: 0, - Code: "SSE_PARSE_ERROR", - Message: fmt.Sprintf("malformed SSE message (%d bytes): %v", len(data), err), - Retryable: true, - }) + // Reported through the subscriber rather than the global logger, and + // without the payload, which can carry tenant/PII fields. + sc.emitError(sseError(0, "SSE_PARSE_ERROR", + fmt.Sprintf("malformed SSE message (%d bytes): %v", len(data), err), true)) return } - event := StreamEvent{ - Table: msg.TableName, - Timestamp: msg.ReceivedTimestamp, - Data: msg.Data, - } - sc.emitEvent(event) + sc.emitEvent(StreamEvent{Table: msg.TableName, Timestamp: msg.ReceivedTimestamp, Data: msg.Data}) } // newFilteredStreamController wraps a StreamController with client-side @@ -512,19 +448,13 @@ func (sc *StreamController) handleSSEData(data string) { func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { compiled := compileFilters(filters) ctx, cancel := context.WithCancel(context.Background()) - sc := &StreamController{ - status: inner.Status(), - eventCh: make(chan StreamEvent, 256), - cancel: cancel, - done: make(chan struct{}), - } + sc := newController(inner.Status(), cancel) go func() { defer func() { sc.setStatus(StatusClosed) - // closeEventCh serializes with any in-flight emitEvent (which - // runs on the inner controller's goroutine), so the channel is - // never closed under a pending send. + // closeEventCh serializes with any in-flight emitEvent on the + // inner goroutine, so the channel never closes under a send. sc.closeEventCh() close(sc.done) }() @@ -552,8 +482,8 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, unsub() inner.Close() case <-inner.done: - // Inner closed on its own — still unsubscribe so the closed - // controller doesn't retain a reference to this wrapper. + // Unsubscribe anyway so the closed inner controller doesn't + // retain a reference to this wrapper. unsub() } }() @@ -568,9 +498,8 @@ type compiledFilter struct { re *regexp.Regexp } -// compileFilters precompiles LIKE/NOT LIKE patterns once per stream. A -// controller's filters never change after construction, so this replaces a -// per-event compile (and avoids any process-global pattern cache). +// compileFilters precompiles LIKE/NOT LIKE patterns once per stream, which a +// controller's immutable filter list makes safe. func compileFilters(filters []QueryFilter) []compiledFilter { out := make([]compiledFilter, len(filters)) for i, f := range filters { @@ -585,7 +514,7 @@ func compileFilters(filters []QueryFilter) []compiledFilter { } // compileLike converts a SQL LIKE pattern to a case-insensitive anchored -// regex (matching the TS SDK). Returns nil if the pattern doesn't compile. +// regex, returning nil if the pattern doesn't compile. func compileLike(pattern string) *regexp.Regexp { escaped := regexp.QuoteMeta(pattern) escaped = strings.ReplaceAll(escaped, "%", ".*") @@ -600,8 +529,7 @@ func compileLike(pattern string) *regexp.Regexp { // matchesFilters evaluates all filters against a data row (AND). func matchesFilters(row map[string]any, filters []compiledFilter) bool { for _, f := range filters { - val := row[f.Column] - if !evaluateFilter(val, f.Op, f.Value, f.re) { + if !evaluateFilter(row[f.Column], f.Op, f.Value, f.re) { return false } } @@ -639,11 +567,10 @@ func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool } } -// equalValues compares two values for equality, normalizing numeric types -// (JSON decodes numbers as float64, but callers may pass int) and comparing -// timestamps as instants rather than as text — see asInstant. +// equalValues compares two values for equality, normalizing numeric types and +// comparing timestamps as instants rather than as text (see asInstant). func equalValues(a, b any) bool { - // nil only equals nil: without this the fmt.Sprint fallback would match a + // nil only equals nil: otherwise the fmt.Sprint fallback would match a // missing column against the literal string "". if a == nil || b == nil { return a == nil && b == nil @@ -663,35 +590,20 @@ func equalValues(a, b any) bool { } // maxTimeOperandChars mirrors the server's row-filter pre-gate: the longest -// spelling the ingest grammar accepts (RFC 3339 with nanoseconds and a numeric -// offset) is 35 bytes, so 64 is generous slack while keeping a megabyte -// "timestamp" from being scanned once per filter per event. +// accepted spelling is 35 bytes, so this bounds per-event parse work. const maxTimeOperandChars = 64 -// asInstant reports the instant a value denotes, but only for spellings that -// name one unambiguously — RFC 3339 with an explicit offset or `Z`. -// -// This exists because the server canonicalizes every top-level DateTime value -// to RFC 3339 UTC before publishing (#402), so a payload reads `...T04:00:00Z` -// while a caller's filter constant may name the same instant as -// `...T06:00:00+02:00`. Comparing those as text is wrong in both directions: -// lexically the payload sorts *below* the constant, so `gte` misses a row that -// is chronologically equal. The server compares DateTime columns as instants -// for exactly this reason; this is the client-side twin of that rule. -// -// Deliberately narrow. A zone-less spelling ("2026-06-21 04:00:00") names an -// instant only relative to the column's declared timezone, which the server -// reads from the schema and a stream subscriber does not have. Guessing UTC -// would move the instant, so those fail to parse here and fall through to text -// comparison rather than being silently reinterpreted. +// asInstant reports the instant a value denotes, and only for RFC 3339 +// spellings carrying an explicit offset or `Z`. Zone-less spellings are +// rejected on purpose: they name an instant only relative to the column's +// timezone, which a subscriber does not know. func asInstant(v any) (time.Time, bool) { s, ok := v.(string) if !ok || len(s) > maxTimeOperandChars { return time.Time{}, false } - // ClickHouse has no ',' decimal separator, but Go's RFC3339Nano accepts one - // per ISO 8601. Reject it so the client can't admit a spelling the server - // would refuse. + // Go's RFC3339Nano accepts a ',' decimal separator per ISO 8601 but + // ClickHouse does not, so reject a spelling the server would refuse. if strings.ContainsRune(s, ',') { return time.Time{}, false } @@ -702,8 +614,8 @@ func asInstant(v any) (time.Time, bool) { return t, true } -// evaluateIn checks whether actual is contained in the expected slice. -// Reflection handles []any and typed slices (e.g., []string, []int) alike. +// evaluateIn reports whether actual is contained in the expected slice. +// Reflection handles []any and typed slices alike. func evaluateIn(actual, expected any) bool { rv := reflect.ValueOf(expected) if rv.Kind() != reflect.Slice { @@ -717,51 +629,40 @@ func evaluateIn(actual, expected any) bool { return false } +// order reports the sign of a-b, mirroring Go's comparison operators (so an +// incomparable float pair such as NaN reports equal rather than ordered). +func order[T float64 | string](a, b T) (int, bool) { + switch { + case a < b: + return -1, true + case a > b: + return 1, true + default: + return 0, true + } +} + // compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, // or (0, false) when the types cannot be compared. func compareOrdered(actual, expected any) (int, bool) { if a, aOK := toFloat64(actual); aOK { if b, bOK := toFloat64(expected); bOK { - switch { - case a < b: - return -1, true - case a > b: - return 1, true - default: - return 0, true - } + return order(a, b) } } - // Timestamps compare chronologically, not lexically. If either side names - // an instant the other must too: ordering a canonicalized payload against a - // spelling that isn't a provable instant is meaningless, and text - // comparison there would admit rows the query path excludes. Fail closed, - // as the server's row filter does for a DateTime column. + // Timestamps compare chronologically and fail closed when only one side is + // a provable instant, matching the server's row filter. aTime, aIsTime := asInstant(actual) bTime, bIsTime := asInstant(expected) if aIsTime || bIsTime { if !aIsTime || !bIsTime { return 0, false } - switch { - case aTime.Before(bTime): - return -1, true - case aTime.After(bTime): - return 1, true - default: - return 0, true - } + return aTime.Compare(bTime), true } if aStr, ok := actual.(string); ok { if bStr, ok := expected.(string); ok { - switch { - case aStr < bStr: - return -1, true - case aStr > bStr: - return 1, true - default: - return 0, true - } + return order(aStr, bStr) } } return 0, false diff --git a/clients/go/sys.go b/clients/go/sys.go index ed875cdb..64e2ad5c 100644 --- a/clients/go/sys.go +++ b/clients/go/sys.go @@ -10,8 +10,8 @@ type SysNamespace struct { ctx httpContext } -// Health pings the server's public /v1/health endpoint. Returns nil when the -// server is reachable and past boot, or an error describing the failure. +// Health pings the server's public /v1/health endpoint, returning nil when the +// server is reachable and past boot. func (s *SysNamespace) Health(ctx context.Context) error { if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", diff --git a/clients/go/table.go b/clients/go/table.go index 73c2ea3f..edd8e4f4 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -9,9 +9,8 @@ import ( "strings" ) -// TableRef is a reference to a table. Use it for queries, inserts, schema, -// and streams. Safe for concurrent use: it holds no mutable state, and every -// builder method returns a fresh value. +// TableRef is a reference to a table, used for queries, inserts, schema and +// streams. Safe for concurrent use: it holds no mutable state. type TableRef struct { ctx httpContext table string @@ -53,10 +52,9 @@ func (t *TableRef) Insert(ctx context.Context, data any) (*InsertResult, error) return t.insertSingle(ctx, data) } -// sliceValue reports whether data is a slice type, returning its -// reflect.Value for iteration. []byte is excluded and treated as an opaque -// single value (matching encoding/json's special-cased handling of byte -// slices) rather than a batch of numbers. +// sliceValue reports whether data is a slice, returning its reflect.Value for +// iteration. []byte is excluded so it stays an opaque single value, as +// encoding/json treats it, rather than a batch of numbers. func sliceValue(data any) (reflect.Value, bool) { if data == nil { return reflect.Value{}, false @@ -107,15 +105,8 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e }, &res); err != nil { return nil, fmt.Errorf("insert into %q: %w", t.table, err) } - ok := true - if res.OK != nil { - ok = *res.OK - } - result := &InsertResult{OK: ok} - if res.Duplicate != nil { - result.Duplicate = res.Duplicate - } - return result, nil + // An absent "ok" field means success. + return &InsertResult{OK: res.OK == nil || *res.OK, Duplicate: res.Duplicate}, nil } func emptyInsertResult() *InsertResult { @@ -151,12 +142,9 @@ func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*Ins return t.sendNDJSON(ctx, ndjson) } -// insertBatchReflect is the fallback batch path for any slice type other -// than []map[string]any (the fast path in insertBatch above) — e.g. a -// generated or user-defined row type such as []ClickRow. Each element is -// marshaled to JSON individually and joined as NDJSON, exactly like -// insertBatch, so the server's per-record batch summary (failed, results, -// etc.) is preserved instead of being silently dropped by insertSingle. +// insertBatchReflect is the batch path for slice types other than +// []map[string]any. It builds the same NDJSON body as insertBatch so the +// server's per-record summary survives instead of being dropped. func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { if rows.Len() == 0 { return emptyInsertResult(), nil diff --git a/clients/go/types.go b/clients/go/types.go index 1257e446..492eb5e7 100644 --- a/clients/go/types.go +++ b/clients/go/types.go @@ -2,28 +2,19 @@ package wavehouse import "context" -// ── Structured query AST (matches backend wire format) ──────────────────── - // StructuredQuery is the wire format for POST /v1/query. type StructuredQuery struct { - // Columns to project. A literal "*" is a column named "*", not a wildcard. - // Omitting columns (with no aggregations and no select_all) selects nothing. - Columns []string `json:"columns,omitempty"` - // SelectAll requests every column the caller's role may read. - // Mutually exclusive with a non-empty Columns list. - SelectAll bool `json:"select_all,omitempty"` - // Aggregations (count, sum, avg, etc.). + // Columns names an explicit projection. A literal "*" is a column named + // "*", not a wildcard, and an empty list with no aggregations and no + // SelectAll selects nothing. Mutually exclusive with SelectAll. + Columns []string `json:"columns,omitempty"` + SelectAll bool `json:"select_all,omitempty"` Aggregations []Aggregation `json:"aggregations,omitempty"` - // Filters (WHERE conditions, ANDed). - Filters []QueryFilter `json:"filters,omitempty"` - // GroupBy columns. - GroupBy []string `json:"group_by,omitempty"` - // OrderBy clauses. - OrderBy []OrderClause `json:"order_by,omitempty"` - // Limit caps the result set. - Limit *int `json:"limit,omitempty"` - // TimeRange filters by a time window. - TimeRange *TimeRange `json:"time_range,omitempty"` + Filters []QueryFilter `json:"filters,omitempty"` + GroupBy []string `json:"group_by,omitempty"` + OrderBy []OrderClause `json:"order_by,omitempty"` + Limit *int `json:"limit,omitempty"` + TimeRange *TimeRange `json:"time_range,omitempty"` } // Aggregation describes a single aggregation (e.g. count, sum). @@ -81,8 +72,6 @@ var opMap = map[FilterOp]string{ OpNotLike: "not_like", } -// ── Schema types ────────────────────────────────────────────────────────── - // Column describes a single column in a table schema. type Column struct { Name string `json:"name"` @@ -100,8 +89,6 @@ type TableSchema struct { // Schemas maps table names to their schemas. type Schemas map[string]TableSchema -// ── Insert result ───────────────────────────────────────────────────────── - // InsertRecordResult is a per-record outcome from a batch insert. type InsertRecordResult struct { Index int `json:"index"` @@ -121,16 +108,12 @@ type InsertResult struct { Results []InsertRecordResult `json:"results,omitempty"` } -// ── DLQ types ───────────────────────────────────────────────────────────── - // DLQStats describes dead-letter-queue statistics. type DLQStats struct { Tables map[string]int `json:"tables"` Total int `json:"total"` } -// ── Pipe types ──────────────────────────────────────────────────────────── - // Pipe describes a named query pipe definition. type Pipe struct { Name string `json:"name"` @@ -148,13 +131,11 @@ type ParamDef struct { Default any `json:"default,omitempty"` } -// ── Policy types ────────────────────────────────────────────────────────── - // Policy describes the server's access-control policy. type Policy struct { DefaultRole string `json:"default_role,omitempty"` - // AdminRole is the role granted full access and the allowlist bypass. - // Empty means the server's default ("admin") applies. + // AdminRole grants full access and the allowlist bypass; empty means the + // server's default ("admin"). AdminRole string `json:"admin_role,omitempty"` Tables map[string]TablePolicy `json:"tables"` } @@ -180,9 +161,8 @@ type RolePermissions struct { } // PolicyFilter describes a policy filter predicate. Fields are pointers with -// omitempty so an intentional empty-string comparison (e.g. Eq pointing at "") -// is sent as "", while an unset operator is omitted entirely — never null — -// matching the server's absent-operator semantics. +// omitempty so an intentional empty-string comparison is sent as "" while an +// unset operator is omitted entirely, never null. type PolicyFilter struct { Eq *string `json:"_eq,omitempty"` Neq *string `json:"_neq,omitempty"` @@ -196,8 +176,6 @@ type ValidationResult struct { Valid bool `json:"valid"` } -// ── Streaming types ─────────────────────────────────────────────────────── - // StreamStatus represents the connection state of a stream. type StreamStatus string @@ -215,16 +193,13 @@ type StreamEvent struct { Data map[string]any `json:"data"` } -// StreamSubscriber receives events from a stream. +// StreamSubscriber receives events from a stream. Every callback is optional. type StreamSubscriber struct { - // Initial is called once with historical backfill data (live queries only). + // Initial fires once with the historical backfill (live queries only). Initial func(rows []map[string]any, err error) - // Next is called for each live event. - Next func(event StreamEvent) - // Status is called when the connection status changes. - Status func(status StreamStatus) - // Error is called on stream errors. - Error func(err error) + Next func(event StreamEvent) + Status func(status StreamStatus) + Error func(err error) } // StreamOptions configures a stream. @@ -233,14 +208,10 @@ type StreamOptions struct { Since string } -// ── Fetch/page types ────────────────────────────────────────────────────── - // Page wraps a result set with pagination metadata. type Page[T any] struct { - // Data is the result rows. - Data []T - // HasMore is true if more rows may be available. + Data []T HasMore bool - // Next fetches the next page. Nil when no cursor is available. + // Next fetches the next page, and is nil when no cursor is available. Next func(ctx context.Context) (*Page[T], error) } diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 3698aedd..74c78938 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -14,6 +14,7 @@ package wavehouse import ( "context" "fmt" + "maps" "net/http" "strings" ) @@ -23,16 +24,14 @@ type Config struct { // BaseURL of the WaveHouse server (e.g. "http://localhost:8080"). BaseURL string - // Auth provides a bearer token for authenticated requests. Called before - // each request; return "" to skip the Authorization header. Nil means - // unauthenticated access (the server falls back to default_role). + // Auth returns a bearer token, called once per request. Return "" to skip + // the Authorization header; nil means unauthenticated (server default_role). Auth func(ctx context.Context) (string, error) // Options tunes transport behavior. Options *ClientOptions - // HTTPClient overrides the default http.Client. Useful for custom TLS, - // proxies, or test transports. + // HTTPClient overrides the default http.Client. HTTPClient *http.Client } @@ -42,19 +41,13 @@ type ClientOptions struct { // Total attempts = MaxRetries + 1. Default: 2. MaxRetries int - // Headers are sent on every request the client makes — REST calls and SSE - // streams alike. Use them for a gateway credential, a tenant selector, or - // tracing metadata that has no first-class option. - // - // The SDK's own headers win: Authorization, Accept, Content-Type, and the - // stream's Cache-Control are set after these and overwrite any entry that - // collides. Names are matched case-insensitively (canonicalized by - // net/http), and each entry replaces rather than appends. + // Headers are sent on every REST call and SSE stream. They are applied + // before the SDK's own headers, so Authorization, Accept, Content-Type and + // Cache-Control win any collision; names are canonicalized by net/http. Headers map[string]string } // StaticToken returns an Auth function that always returns the same token. -// Convenience for cases where the token doesn't rotate. func StaticToken(token string) func(context.Context) (string, error) { return func(context.Context) (string, error) { return token, nil } } @@ -63,16 +56,11 @@ func StaticToken(token string) func(context.Context) (string, error) { type Client struct { ctx httpContext - // Schema provides admin-only schema introspection. Schema *SchemaNamespace - // Policy provides admin-only access-control policy management. Policy *PolicyNamespace - // DLQ provides admin-only dead-letter-queue statistics. - DLQ *DLQNamespace - // Sys provides system health checks. - Sys *SysNamespace - // Pipes provides admin-only named-pipe management. - Pipes *PipesNamespace + DLQ *DLQNamespace + Sys *SysNamespace + Pipes *PipesNamespace } // NewClient creates a new WaveHouse client. @@ -85,16 +73,13 @@ func NewClient(cfg Config) *Client { // Copy so a later mutation of the caller's map can't reach into requests. var headers map[string]string if cfg.Options != nil && len(cfg.Options.Headers) > 0 { - headers = make(map[string]string, len(cfg.Options.Headers)) - for k, v := range cfg.Options.Headers { - headers[k] = v - } + headers = maps.Clone(cfg.Options.Headers) } hc := cfg.HTTPClient if hc == nil { - // Not http.DefaultClient: it's mutable global state another package - // could reconfigure (timeout, transport, redirects) after we're built. + // Not http.DefaultClient: another package could reconfigure that + // mutable global (timeout, transport, redirects) after we are built. hc = &http.Client{} } @@ -137,9 +122,8 @@ func (c *Client) Pipe(name string, params map[string]any) *PipeRef { } } -// SQL executes a raw SQL query against ClickHouse. Requires the admin role. -// The server proxies the SQL verbatim to ClickHouse's HTTP interface. Results -// are decoded into []T; use [map[string]any] for dynamic schemas. +// SQL executes a raw SQL query against ClickHouse, decoding results into []Row. +// Requires the admin role; the server proxies the SQL verbatim. func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { var rows []Row err := doRequest(ctx, c.ctx, requestOptions{ @@ -153,7 +137,6 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { return rows, nil } -// createStream opens an SSE stream for the given table. func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { return newStreamController(c.ctx, table, opts) } From 527dfea7c028081ac21d0808caef25940b5df20f Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 10:04:16 -0400 Subject: [PATCH 44/59] refactor(sdk): compact the conformance fixture and both wire runners MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit wire_cases.json is re-emitted densely — one JSON object per case, short scalar fields sharing a line — so a case still reads at a glance in a third of the space. The parsed value is unchanged: same 45 cases, same key order per case, and `json.load` of the old and new files compares equal (canonical sort_keys dumps are identical too). No expected_path, method, query, content-type or body value was touched, and no case was removed: the only pair asserting an identical wire tuple ("bare query defaults to select_all" / "selectAll sends select_all flag") reaches it from different builder calls, and dropping either would leave selectAll unexercised in both SDKs. Both runners lose narration and fold their repeated blocks, in lockstep: - The six (column, alias) aggregation ops become one table each — aggOps in Go, AGGREGATIONS in the .mjs — instead of six near-identical cases. - Go's endpoint dispatch is one line per endpoint via errOf (drop the value, keep the error) and jsonArg (decode a fixture field into the SDK request type); the TS dispatch becomes an endpoint -> call table, with the ingest-case validation shared by both ingest endpoints. - The method and content-type assertions share a helper/loop; the mock server's canned responses become early returns in the .mjs. - Go drops normalizeJSON: expected and captured bodies both decode into `any` through encoding/json, so every number is a float64 on both sides and reflect.DeepEqual is already exact. The parity guards are unchanged — an endpoint missing from the dispatch still hard-fails the Go test and exits the TS runner non-zero. 1356 -> 942 lines across the three files. Verified: gofmt and gofumpt clean, go vet clean, `go test -race -run Conformance ./...` passes all 45 subtests, `biome check` clean, and `make test-conformance-ts` reports 45 passed / 0 failed / 0 skipped. Mutating a path, method, content-type, raw body and JSON body in the fixture makes exactly those five cases fail in both runners. --- clients/go/conformance_test.go | 347 ++++++----------- clients/go/testdata/wire_cases.json | 541 +++++++-------------------- tests/conformance/conformance_ts.mjs | 204 ++++------ 3 files changed, 339 insertions(+), 753 deletions(-) diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index c8af2899..0d1324b0 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -14,19 +14,10 @@ import ( "testing" ) -// logCallErr surfaces SDK-call errors that the conformance harness otherwise -// ignores — the assertions only inspect the captured request, but when a call -// fails before sending, the failure message should name the real cause. -func logCallErr(t *testing.T, err error) { - t.Helper() - if err != nil { - t.Logf("SDK call returned error (request may still be valid): %v", err) - } -} - -// wireCasesJSON embeds the shared wire-format conformance fixture so the -// test binary is self-contained: it works from a module archive or a -// standalone checkout without depending on paths outside the Go module. +// wireCasesJSON embeds the shared wire-format fixture — the same file the TS +// runner replays (tests/conformance/conformance_ts.mjs) — so the test binary +// is self-contained: it works from a module archive or a standalone checkout +// without depending on paths outside the Go module. // //go:embed testdata/wire_cases.json var wireCasesJSON []byte @@ -54,15 +45,6 @@ type wireOp struct { Args []any `json:"args"` } -func loadWireCases(t *testing.T) []wireCase { - t.Helper() - var cases []wireCase - if err := json.Unmarshal(wireCasesJSON, &cases); err != nil { - t.Fatalf("parse wire_cases.json: %v", err) - } - return cases -} - // captured holds the HTTP request details from a single SDK call. type captured struct { method string @@ -71,8 +53,34 @@ type captured struct { body string } +// logErr surfaces SDK-call errors the assertions otherwise ignore: they only +// inspect the captured request, but a call that fails before sending should +// name the real cause. +func logErr(t *testing.T, err error) { + t.Helper() + if err != nil { + t.Logf("SDK call returned error (request may still be valid): %v", err) + } +} + +// errOf drops the value of a two-result SDK call, keeping only the error. +func errOf[T any](_ T, err error) error { return err } + +// jsonArg decodes a fixture field into the SDK request type it stands for. +func jsonArg[T any](t *testing.T, field string, raw json.RawMessage) T { + t.Helper() + var v T + if err := json.Unmarshal(raw, &v); err != nil { + t.Fatalf("parse %s: %v", field, err) + } + return v +} + func TestConformance_WireFormat(t *testing.T) { - cases := loadWireCases(t) + var cases []wireCase + if err := json.Unmarshal(wireCasesJSON, &cases); err != nil { + t.Fatalf("parse wire_cases.json: %v", err) + } for _, tc := range cases { t.Run(tc.Name, func(t *testing.T) { @@ -117,161 +125,100 @@ func TestConformance_WireFormat(t *testing.T) { }) ctx := context.Background() - // Execute the case. switch tc.Endpoint { case "query": - q := applyOps(t, tc.Table, c, tc.Operations) - _, err := q.FetchUntyped(ctx) - logCallErr(t, err) - - case "ingest": - if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { - t.Fatalf("ingest case %q has no insert operation", tc.Name) - } - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - data := tc.Operations[0].Args[0] - _, err := c.From(tc.Table).Insert(ctx, data) - logCallErr(t, err) - - case "ingest_batch": - if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { - t.Fatalf("ingest_batch case %q has no insert operation", tc.Name) - } - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - rawArr, ok := tc.Operations[0].Args[0].([]any) - if !ok { - t.Fatalf("batch insert args[0] is not an array") - } - rows := make([]map[string]any, len(rawArr)) - for i, r := range rawArr { - rows[i] = toStringMap(t, r) - } - _, batchErr := c.From(tc.Table).Insert(ctx, rows) - logCallErr(t, batchErr) - + logErr(t, errOf(applyOps(t, tc.Table, c, tc.Operations).FetchUntyped(ctx))) + case "ingest", "ingest_batch": + logErr(t, errOf(c.From(tc.Table).Insert(ctx, insertArg(t, tc)))) case "pipe": - p := c.Pipe(tc.PipeName, tc.PipeParams) - _, err := p.FetchUntyped(ctx) - logCallErr(t, err) - + logErr(t, errOf(c.Pipe(tc.PipeName, tc.PipeParams).FetchUntyped(ctx))) case "sql": - _, err := SQL[map[string]any](ctx, c, tc.SQL) - logCallErr(t, err) - + logErr(t, errOf(SQL[map[string]any](ctx, c, tc.SQL))) case "health": - logCallErr(t, c.Sys.Health(ctx)) - + logErr(t, c.Sys.Health(ctx)) case "schema_list": - _, err := c.Schema.List(ctx) - logCallErr(t, err) - + logErr(t, errOf(c.Schema.List(ctx))) case "schema_refresh": - logCallErr(t, c.Schema.Refresh(ctx)) - + logErr(t, c.Schema.Refresh(ctx)) case "policy_get": - _, err := c.Policy.Get(ctx) - logCallErr(t, err) - + logErr(t, errOf(c.Policy.Get(ctx))) case "policy_set": - var pol Policy - if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { - t.Fatalf("parse policy_body: %v", err) - } - logCallErr(t, c.Policy.Set(ctx, &pol)) - + logErr(t, c.Policy.Set(ctx, jsonArg[*Policy](t, "policy_body", tc.PolicyBody))) case "policy_validate": - var pol Policy - if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { - t.Fatalf("parse policy_body: %v", err) - } - _, err := c.Policy.Validate(ctx, &pol) - logCallErr(t, err) - + logErr(t, errOf(c.Policy.Validate(ctx, jsonArg[*Policy](t, "policy_body", tc.PolicyBody)))) case "dlq_list": - _, err := c.DLQ.List(ctx) - logCallErr(t, err) - + logErr(t, errOf(c.DLQ.List(ctx))) case "dlq_table": - _, err := c.DLQ.Table(ctx, tc.Table) - logCallErr(t, err) - + logErr(t, errOf(c.DLQ.Table(ctx, tc.Table))) case "pipes_list": - _, err := c.Pipes.List(ctx) - logCallErr(t, err) - + logErr(t, errOf(c.Pipes.List(ctx))) case "pipes_get": - _, err := c.Pipes.Get(ctx, tc.PipeName) - logCallErr(t, err) - + logErr(t, errOf(c.Pipes.Get(ctx, tc.PipeName))) case "pipes_set": - var def PipeDef - if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { - t.Fatalf("parse pipe_def: %v", err) - } - logCallErr(t, c.Pipes.Set(ctx, tc.PipeName, def)) - + logErr(t, c.Pipes.Set(ctx, tc.PipeName, jsonArg[PipeDef](t, "pipe_def", tc.PipeDefBody))) case "pipes_delete": - logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) - + logErr(t, c.Pipes.Delete(ctx, tc.PipeName)) default: // Hard failure, matching the TS runner: skipped cases break // cross-SDK parity. t.Fatalf("unhandled endpoint %q — wire it up in the dispatch switch", tc.Endpoint) } - // Verify method. mu.Lock() defer mu.Unlock() - if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { - t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) - } - // Verify path. - if tc.ExpectedPath != "" { - // Normalize: the SDK may use different encoding (+ vs %20). - wantPath := normalizePath(tc.ExpectedPath) - gotPath := normalizePath(capt.path) - if wantPath != gotPath { - t.Errorf("path: want %s, got %s", tc.ExpectedPath, capt.path) + wantEq := func(what, want, got string) { + t.Helper() + if want != "" && want != got { + t.Errorf("%s: want %s, got %s", what, want, got) } } - - // Verify content type. - if tc.ExpectedContentType != "" && capt.contentType != tc.ExpectedContentType { - t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, capt.contentType) + wantEq("method", tc.ExpectedMethod, capt.method) + wantEq("content-type", tc.ExpectedContentType, capt.contentType) + // Paths compare by meaning (see normalizePath) but report as written. + if tc.ExpectedPath != "" && normalizePath(tc.ExpectedPath) != normalizePath(capt.path) { + t.Errorf("path: want %s, got %s", tc.ExpectedPath, capt.path) } - // Verify raw body (for NDJSON). + // Raw body: the NDJSON cases, where the byte layout is the point. if tc.ExpectedRawBody != nil { if capt.body != *tc.ExpectedRawBody { t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, capt.body) } return } - - // Verify JSON body. - if tc.ExpectedBody != nil && string(tc.ExpectedBody) != "null" { - var want, got any - if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { - t.Fatalf("parse expected_body: %v", err) - } - if err := json.Unmarshal([]byte(capt.body), &got); err != nil { - t.Fatalf("parse captured body: %v (body: %s)", err, capt.body) - } - if !deepEqualJSON(want, got) { - wantJSON, _ := json.MarshalIndent(want, "", " ") - gotJSON, _ := json.MarshalIndent(got, "", " ") - t.Errorf("body mismatch:\n want: %s\n got: %s", wantJSON, gotJSON) - } + if tc.ExpectedBody == nil || string(tc.ExpectedBody) == "null" { + return + } + // Both sides decode into any, so every number is a float64 on both + // sides and map key order is irrelevant to DeepEqual. + var want, got any + if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { + t.Fatalf("parse expected_body: %v", err) + } + if err := json.Unmarshal([]byte(capt.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, capt.body) + } + if !reflect.DeepEqual(want, got) { + wantJSON, _ := json.MarshalIndent(want, "", " ") + gotJSON, _ := json.MarshalIndent(got, "", " ") + t.Errorf("body mismatch:\n want: %s\n got: %s", wantJSON, gotJSON) } }) } } +// aggOps are the fixture ops that map one-to-one onto a (column, alias) +// aggregation method. Empty args fall through to each method's own defaults. +var aggOps = map[string]func(*QueryBuilder, string, string) *QueryBuilder{ + "count": (*QueryBuilder).Count, + "sum": (*QueryBuilder).Sum, + "avg": (*QueryBuilder).Avg, + "min": (*QueryBuilder).Min, + "max": (*QueryBuilder).Max, + "countDistinct": (*QueryBuilder).CountDistinct, +} + // applyOps replays the operation chain from the fixture onto a QueryBuilder. // Fixtures always put select first (mirroring real usage), so rebuilding on // select is safe and keeps this simple. @@ -280,6 +227,10 @@ func applyOps(t *testing.T, table string, c *Client, ops []wireOp) *QueryBuilder q := c.From(table).Select() for _, op := range ops { + if agg, ok := aggOps[op.Method]; ok { + q = agg(q, stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "")) + continue + } switch op.Method { case "select": q = c.From(table).Select(toStringSlice(op.Args)...) @@ -289,63 +240,53 @@ func applyOps(t *testing.T, table string, c *Client, ops []wireOp) *QueryBuilder if len(op.Args) != 3 { t.Fatalf("where needs 3 args, got %d", len(op.Args)) } - col, ok := op.Args[0].(string) - if !ok { - t.Fatalf("where: column arg is %T, want string", op.Args[0]) + col, colOK := op.Args[0].(string) + rawOp, opOK := op.Args[1].(string) + if !colOK || !opOK { + t.Fatalf("where: want (string, string, any) args, got (%T, %T)", op.Args[0], op.Args[1]) } - rawOp, ok := op.Args[1].(string) - if !ok { - t.Fatalf("where: operator arg is %T, want string", op.Args[1]) - } - opStr := FilterOp(rawOp) - val := op.Args[2] - q = q.Where(col, opStr, val) - case "count": - col, alias := stringArg(op.Args, 0, "*"), stringArg(op.Args, 1, "count") - q = q.Count(col, alias) - case "sum": - col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") - q = q.Sum(col, alias) - case "avg": - col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") - q = q.Avg(col, alias) - case "min": - col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") - q = q.Min(col, alias) - case "max": - col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") - q = q.Max(col, alias) - case "countDistinct": - col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") - q = q.CountDistinct(col, alias) + q = q.Where(col, FilterOp(rawOp), op.Args[2]) case "aggregate": - fn := stringArg(op.Args, 0, "") - col := stringArg(op.Args, 1, "") - alias := stringArg(op.Args, 2, "") - q = q.Aggregate(fn, col, alias) + q = q.Aggregate(stringArg(op.Args, 0, ""), stringArg(op.Args, 1, ""), stringArg(op.Args, 2, "")) case "groupBy": - cols := toStringSlice(op.Args) - q = q.GroupBy(cols...) + q = q.GroupBy(toStringSlice(op.Args)...) case "orderBy": - col := stringArg(op.Args, 0, "") - dir := stringArg(op.Args, 1, "asc") - q = q.OrderBy(col, dir) + q = q.OrderBy(stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "asc")) case "limit": - n := intArg(op.Args, 0) - q = q.Limit(n) + q = q.Limit(intArg(op.Args, 0)) case "timeRange": - col := stringArg(op.Args, 0, "") - since := stringArg(op.Args, 1, "") - until := stringArg(op.Args, 2, "") - q = q.TimeRange(col, since, until) + q = q.TimeRange(stringArg(op.Args, 0, ""), stringArg(op.Args, 1, ""), stringArg(op.Args, 2, "")) case "cacheTTL": - n := intArg(op.Args, 0) - q = q.CacheTTL(n) + q = q.CacheTTL(intArg(op.Args, 0)) } } return q } +// insertArg returns the payload for an ingest case. Batch rows arrive from +// JSON as []any; hand Insert the []map[string]any a caller would pass so it +// takes the batch (NDJSON) path a typed slice takes. +func insertArg(t *testing.T, tc wireCase) any { + t.Helper() + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" || len(tc.Operations[0].Args) == 0 { + t.Fatalf("ingest case %q needs an insert operation with one arg", tc.Name) + } + arg := tc.Operations[0].Args[0] + batch, ok := arg.([]any) + if !ok { + return arg + } + rows := make([]map[string]any, len(batch)) + for i, r := range batch { + row, ok := r.(map[string]any) + if !ok { + t.Fatalf("fixture row is not an object: %T", r) + } + rows[i] = row + } + return rows +} + func stringArg(args []any, i int, fallback string) string { if i >= len(args) { return fallback @@ -379,46 +320,6 @@ func toStringSlice(args []any) []string { return out } -func toStringMap(t *testing.T, v any) map[string]any { - t.Helper() - m, ok := v.(map[string]any) - if !ok { - t.Fatalf("fixture row is not an object: %T", v) - } - return m -} - -// deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal -// to ints (JSON numbers decode as float64 in Go). -func deepEqualJSON(a, b any) bool { - return reflect.DeepEqual(normalizeJSON(a), normalizeJSON(b)) -} - -func normalizeJSON(v any) any { - switch val := v.(type) { - case map[string]any: - m := make(map[string]any, len(val)) - for k, v := range val { - m[k] = normalizeJSON(v) - } - return m - case []any: - s := make([]any, len(val)) - for i, v := range val { - s[i] = normalizeJSON(v) - } - return s - case float64: - // Normalize integer-valued floats to int for comparison. - if val == float64(int64(val)) { - return int64(val) - } - return val - default: - return val - } -} - // normalizePath compares request URIs by meaning: same path, same decoded // query values regardless of + vs %20 spelling or parameter order. A raw // string replace would also rewrite literal + characters and stop asserting diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index d715b3bc..f61efa53 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -1,389 +1,208 @@ [ { "name": "bare query defaults to select_all", - "endpoint": "query", - "table": "clicks", - "operations": [], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "select_all": true, - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "select_all": true, "limit": 1000 } }, { "name": "select explicit columns", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page", "button"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page", "button"], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "select", "args": ["page", "button"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page", "button"], "limit": 1000 } }, { "name": "selectAll sends select_all flag", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "selectAll" } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "select_all": true, - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "selectAll" }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "select_all": true, "limit": 1000 } }, { "name": "where with eq operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["page", "=", "/home"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "=", "/home"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "page", "op": "eq", "value": "/home" }], - "limit": 1000 + "columns": ["page"], "filters": [{ "column": "page", "op": "eq", "value": "/home" }], "limit": 1000 } }, { "name": "where with neq operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["page", "!=", "/home"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "!=", "/home"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "page", "op": "neq", "value": "/home" }], - "limit": 1000 + "columns": ["page"], "filters": [{ "column": "page", "op": "neq", "value": "/home" }], "limit": 1000 } }, { "name": "where with gt operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["score", ">", 10] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "filters": [{ "column": "score", "op": "gt", "value": 10 }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">", 10] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], "limit": 1000 } }, { "name": "where with gte operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["score", ">=", 10] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">=", 10] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "score", "op": "gte", "value": 10 }], - "limit": 1000 + "columns": ["page"], "filters": [{ "column": "score", "op": "gte", "value": 10 }], "limit": 1000 } }, { "name": "where with lt operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["score", "<", 5] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "filters": [{ "column": "score", "op": "lt", "value": 5 }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<", 5] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lt", "value": 5 }], "limit": 1000 } }, { "name": "where with lte operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["score", "<=", 5] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "filters": [{ "column": "score", "op": "lte", "value": 5 }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<=", 5] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lte", "value": 5 }], "limit": 1000 } }, { "name": "where with in operator", - "endpoint": "query", - "table": "clicks", + "endpoint": "query", "table": "clicks", "operations": [ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "in", ["/home", "/about"]] } ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], - "limit": 1000 + "columns": ["page"], "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], "limit": 1000 } }, { "name": "where with like operator", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "where", "args": ["page", "like", "/home%"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "like", "/home%"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "page", "op": "like", "value": "/home%" }], - "limit": 1000 + "columns": ["page"], "filters": [{ "column": "page", "op": "like", "value": "/home%" }], "limit": 1000 } }, { "name": "count aggregation", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "count", "args": ["*", "total"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "count", "args": ["*", "total"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], "limit": 1000 } }, { "name": "sum aggregation with default alias", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "sum", "args": ["score", ""] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "sum", "args": ["score", ""] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], "limit": 1000 } }, { "name": "avg aggregation with default alias", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "avg", "args": ["score", ""] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "avg", "args": ["score", ""] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], "limit": 1000 } }, { "name": "min aggregation with default alias", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "min", "args": ["score", ""] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "min", "args": ["score", ""] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], "limit": 1000 } }, { "name": "max aggregation with default alias", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "max", "args": ["score", ""] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", "operations": [{ "method": "max", "args": ["score", ""] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], "limit": 1000 } }, { "name": "countDistinct aggregation with default alias", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "countDistinct", "args": ["page", ""] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", "operations": [{ "method": "countDistinct", "args": ["page", ""] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], - "limit": 1000 + "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], "limit": 1000 } }, { "name": "custom aggregate function", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], - "limit": 1000 + "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], "limit": 1000 } }, { "name": "groupBy", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "groupBy", "args": ["page"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "group_by": ["page"], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "groupBy", "args": ["page"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "group_by": ["page"], "limit": 1000 } }, { "name": "orderBy ascending (default)", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "orderBy", "args": ["page", "asc"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "order_by": [{ "column": "page", "dir": "asc" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["page", "asc"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "order_by": [{ "column": "page", "dir": "asc" }], "limit": 1000 } }, { "name": "orderBy descending", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "orderBy", "args": ["score", "desc"] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "order_by": [{ "column": "score", "dir": "desc" }], - "limit": 1000 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["score", "desc"] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "order_by": [{ "column": "score", "dir": "desc" }], "limit": 1000 } }, { "name": "explicit limit", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "limit", "args": [50] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "limit": 50 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "limit", "args": [50] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "columns": ["page"], "limit": 50 } }, { "name": "timeRange with since only", - "endpoint": "query", - "table": "clicks", + "endpoint": "query", "table": "clicks", "operations": [ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "time_range": { "column": "received_timestamp", "since": "1h" }, - "limit": 1000 + "columns": ["page"], "time_range": { "column": "received_timestamp", "since": "1h" }, "limit": 1000 } }, { "name": "timeRange with since and until", - "endpoint": "query", - "table": "clicks", + "endpoint": "query", "table": "clicks", "operations": [ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, - "limit": 1000 + "columns": ["page"], "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, "limit": 1000 } }, { "name": "multiple where clauses (AND)", - "endpoint": "query", - "table": "clicks", + "endpoint": "query", "table": "clicks", "operations": [ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">", 10] }, { "method": "where", "args": ["page", "=", "/home"] } ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { "columns": ["page"], - "filters": [ - { "column": "score", "op": "gt", "value": 10 }, - { "column": "page", "op": "eq", "value": "/home" } - ], + "filters": [{ "column": "score", "op": "gt", "value": 10 }, { "column": "page", "op": "eq", "value": "/home" }], "limit": 1000 } }, { "name": "complex query combining everything", - "endpoint": "query", - "table": "clicks", + "endpoint": "query", "table": "clicks", "operations": [ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">", 10] }, @@ -393,216 +212,126 @@ { "method": "limit", "args": [50] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", "expected_body": { - "columns": ["page"], - "filters": [{ "column": "score", "op": "gt", "value": 10 }], - "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], - "group_by": ["page"], - "order_by": [{ "column": "total", "dir": "desc" }], - "limit": 50, + "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], "group_by": ["page"], + "order_by": [{ "column": "total", "dir": "desc" }], "limit": 50, "time_range": { "column": "received_timestamp", "since": "1h" } } }, { "name": "insert single row path", - "endpoint": "ingest", - "table": "clicks", - "operations": [ - { "method": "insert", "args": [{ "page": "/home", "button": "cta" }] } - ], - "expected_path": "/v1/ingest?table=clicks", - "expected_method": "POST", - "expected_content_type": "application/json", + "endpoint": "ingest", "table": "clicks", + "operations": [{ "method": "insert", "args": [{ "page": "/home", "button": "cta" }] }], + "expected_path": "/v1/ingest?table=clicks", "expected_method": "POST", "expected_content_type": "application/json", "expected_body": { "page": "/home", "button": "cta" } }, { "name": "insert batch as NDJSON", - "endpoint": "ingest_batch", - "table": "clicks", - "operations": [ - { - "method": "insert", - "args": [[{ "page": "/a" }, { "page": "/b" }]] - } - ], - "expected_path": "/v1/ingest?table=clicks", - "expected_method": "POST", - "expected_content_type": "application/x-ndjson", - "expected_raw_body": "{\"page\":\"/a\"}\n{\"page\":\"/b\"}" + "endpoint": "ingest_batch", "table": "clicks", + "operations": [{ "method": "insert", "args": [[{ "page": "/a" }, { "page": "/b" }]] }], + "expected_path": "/v1/ingest?table=clicks", "expected_method": "POST", + "expected_content_type": "application/x-ndjson", "expected_raw_body": "{\"page\":\"/a\"}\n{\"page\":\"/b\"}" }, { "name": "pipe execution", - "endpoint": "pipe", - "pipe_name": "top_pages", - "pipe_params": { "limit": 10 }, - "expected_path": "/v1/pipes/top_pages", - "expected_method": "POST", - "expected_body": { "limit": 10 } + "endpoint": "pipe", "pipe_name": "top_pages", "pipe_params": { "limit": 10 }, + "expected_path": "/v1/pipes/top_pages", "expected_method": "POST", "expected_body": { "limit": 10 } }, { "name": "pipe execution with no params sends empty object", - "endpoint": "pipe", - "pipe_name": "simple", - "pipe_params": null, - "expected_path": "/v1/pipes/simple", - "expected_method": "POST", - "expected_body": {} + "endpoint": "pipe", "pipe_name": "simple", "pipe_params": null, + "expected_path": "/v1/pipes/simple", "expected_method": "POST", "expected_body": {} }, { "name": "raw SQL", - "endpoint": "sql", - "sql": "SELECT count() FROM clicks", - "expected_path": "/v1/ops/query", - "expected_method": "POST", + "endpoint": "sql", "sql": "SELECT count() FROM clicks", + "expected_path": "/v1/ops/query", "expected_method": "POST", "expected_body": { "sql": "SELECT count() FROM clicks" } }, { "name": "health check", "endpoint": "health", - "expected_path": "/v1/health", - "expected_method": "GET" + "expected_path": "/v1/health", "expected_method": "GET" }, { "name": "schema list", "endpoint": "schema_list", - "expected_path": "/v1/ops/schema", - "expected_method": "GET" + "expected_path": "/v1/ops/schema", "expected_method": "GET" }, { "name": "schema refresh", "endpoint": "schema_refresh", - "expected_path": "/v1/ops/schema/refresh", - "expected_method": "POST" + "expected_path": "/v1/ops/schema/refresh", "expected_method": "POST" }, { "name": "policy get", "endpoint": "policy_get", - "expected_path": "/v1/ops/policy", - "expected_method": "GET" + "expected_path": "/v1/ops/policy", "expected_method": "GET" }, { "name": "table with special characters URL-encodes correctly", - "endpoint": "query", - "table": "my table", - "operations": [ - { "method": "select", "args": ["page"] }, - { "method": "limit", "args": [10] } - ], - "expected_path": "/v1/query?table=my+table", - "expected_method": "POST", - "expected_body": { - "columns": ["page"], - "limit": 10 - } + "endpoint": "query", "table": "my table", + "operations": [{ "method": "select", "args": ["page"] }, { "method": "limit", "args": [10] }], + "expected_path": "/v1/query?table=my+table", "expected_method": "POST", + "expected_body": { "columns": ["page"], "limit": 10 } }, { "name": "DLQ list", "endpoint": "dlq_list", - "expected_path": "/v1/ops/dlq/stats", - "expected_method": "GET" + "expected_path": "/v1/ops/dlq/stats", "expected_method": "GET" }, { "name": "DLQ table filter", - "endpoint": "dlq_table", - "table": "events", - "expected_path": "/v1/ops/dlq/stats?table=events", - "expected_method": "GET" + "endpoint": "dlq_table", "table": "events", + "expected_path": "/v1/ops/dlq/stats?table=events", "expected_method": "GET" }, { "name": "policy set", - "endpoint": "policy_set", - "policy_body": { - "default_role": "viewer", - "tables": { - "events": {} - } - }, - "expected_path": "/v1/ops/policy", - "expected_method": "PUT", - "expected_content_type": "application/json", - "expected_body": { - "default_role": "viewer", - "tables": { - "events": {} - } - } + "endpoint": "policy_set", "policy_body": { "default_role": "viewer", "tables": { "events": {} } }, + "expected_path": "/v1/ops/policy", "expected_method": "PUT", "expected_content_type": "application/json", + "expected_body": { "default_role": "viewer", "tables": { "events": {} } } }, { "name": "policy validate", - "endpoint": "policy_validate", - "policy_body": { - "default_role": "viewer", - "tables": { - "events": {} - } - }, - "expected_path": "/v1/ops/policy/validate", - "expected_method": "POST", - "expected_content_type": "application/json", - "expected_body": { - "default_role": "viewer", - "tables": { - "events": {} - } - } + "endpoint": "policy_validate", "policy_body": { "default_role": "viewer", "tables": { "events": {} } }, + "expected_path": "/v1/ops/policy/validate", "expected_method": "POST", "expected_content_type": "application/json", + "expected_body": { "default_role": "viewer", "tables": { "events": {} } } }, { "name": "pipes list", "endpoint": "pipes_list", - "expected_path": "/v1/ops/pipes", - "expected_method": "GET" + "expected_path": "/v1/ops/pipes", "expected_method": "GET" }, { "name": "pipes get", - "endpoint": "pipes_get", - "pipe_name": "my_pipe", - "expected_path": "/v1/ops/pipes/my_pipe", - "expected_method": "GET" + "endpoint": "pipes_get", "pipe_name": "my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "GET" }, { "name": "pipes set", - "endpoint": "pipes_set", - "pipe_name": "my_pipe", + "endpoint": "pipes_set", "pipe_name": "my_pipe", "pipe_def": { "sql": "SELECT page, count() AS views FROM events GROUP BY page", - "parameters": [ - { "name": "limit", "type": "Int32", "default": 100 } - ], - "description": "Top pages by view count" + "parameters": [{ "name": "limit", "type": "Int32", "default": 100 }], "description": "Top pages by view count" }, - "expected_path": "/v1/ops/pipes/my_pipe", - "expected_method": "PUT", - "expected_content_type": "application/json", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "PUT", "expected_content_type": "application/json", "expected_body": { "sql": "SELECT page, count() AS views FROM events GROUP BY page", - "parameters": [ - { "name": "limit", "type": "Int32", "default": 100 } - ], - "description": "Top pages by view count" + "parameters": [{ "name": "limit", "type": "Int32", "default": 100 }], "description": "Top pages by view count" } }, { "name": "pipes delete", - "endpoint": "pipes_delete", - "pipe_name": "my_pipe", - "expected_path": "/v1/ops/pipes/my_pipe", - "expected_method": "DELETE" + "endpoint": "pipes_delete", "pipe_name": "my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "DELETE" }, { "name": "cacheTTL is client-side only and not sent on the wire", - "endpoint": "query", - "table": "clicks", - "operations": [ - { "method": "cacheTTL", "args": [60] }, - { "method": "limit", "args": [5] } - ], - "expected_path": "/v1/query?table=clicks", - "expected_method": "POST", - "expected_body": { - "select_all": true, - "limit": 5 - } + "endpoint": "query", "table": "clicks", + "operations": [{ "method": "cacheTTL", "args": [60] }, { "method": "limit", "args": [5] }], + "expected_path": "/v1/query?table=clicks", "expected_method": "POST", + "expected_body": { "select_all": true, "limit": 5 } } ] diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index b145e994..54c58424 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -2,9 +2,10 @@ /** * Cross-language wire-format conformance test for the TypeScript SDK. * - * Reads wire_cases.json (owned by the Go module, at clients/go/testdata/) - * and verifies the TS SDK produces identical HTTP requests (method, path, - * content-type, body) to the shared fixture. + * Replays the shared fixture (clients/go/testdata/wire_cases.json, owned by + * the Go module and also replayed by clients/go/conformance_test.go) and + * verifies the TS SDK produces the same HTTP request: method, path, + * content-type, body. * * Run: node tests/conformance/conformance_ts.mjs * Exit 0 = all pass, exit 1 = failures. @@ -17,7 +18,6 @@ import { fileURLToPath } from "node:url"; const __dirname = dirname(fileURLToPath(import.meta.url)); -// Import the built SDK. let createClient; try { ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); @@ -35,11 +35,25 @@ const cases = JSON.parse( let lastCapture = { method: "", path: "", contentType: "", body: "" }; -function resetCapture() { - lastCapture = { method: "", path: "", contentType: "", body: "" }; +// Canned responses, matching the real server's shapes (internal/api/*.go) so +// the SDK never errors on decode. Mirrors the Go harness's handler. +function cannedResponse(url, method, contentType) { + if (url.startsWith("/v1/ops/dlq")) return { tables: {}, total: 0 }; + if (url.startsWith("/v1/ops/schema") && method === "GET") return []; + if (url === "/v1/ops/policy/validate" && method === "POST") return { valid: true }; + if (url.startsWith("/v1/ops/policy") && method === "GET") return { tables: {} }; + if (url.startsWith("/v1/ops/pipes/") && method === "GET") + return { name: "test", sql: "SELECT 1" }; + if (url === "/v1/ops/pipes" && method === "GET") return []; + if (url.startsWith("/v1/ingest")) { + return contentType === "application/x-ndjson" + ? { total: 0, succeeded: 0, failed: 0, duplicates: 0 } + : { ok: true }; + } + if (url === "/v1/health") return { status: "ok" }; + return []; } -// Start echo server. const server = createServer((req, res) => { const chunks = []; req.on("data", (c) => chunks.push(c)); @@ -51,31 +65,9 @@ const server = createServer((req, res) => { body: Buffer.concat(chunks).toString("utf-8"), }; res.setHeader("Content-Type", "application/json"); - if (req.url?.startsWith("/v1/ops/dlq")) { - res.end(JSON.stringify({ tables: {}, total: 0 })); - } else if (req.url?.startsWith("/v1/ops/schema") && req.method === "GET") { - res.end(JSON.stringify([])); - } else if (req.url === "/v1/ops/policy/validate" && req.method === "POST") { - res.end(JSON.stringify({ valid: true })); - } else if (req.url?.startsWith("/v1/ops/policy") && req.method === "GET") { - res.end(JSON.stringify({ tables: {} })); - } else if (req.url?.startsWith("/v1/ops/pipes/") && req.method === "GET") { - res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); - } else if (req.url === "/v1/ops/pipes" && req.method === "GET") { - res.end(JSON.stringify([])); - } else if (req.url?.startsWith("/v1/ingest")) { - // Same shapes the real server returns (internal/api/ingest.go). - if (lastCapture.contentType === "application/x-ndjson") { - res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); - } else { - res.end(JSON.stringify({ ok: true })); - } - } else if (req.url === "/v1/health") { - // Real server shape (internal/api/health.go). - res.end(JSON.stringify({ status: "ok" })); - } else { - res.end(JSON.stringify([])); - } + res.end( + JSON.stringify(cannedResponse(lastCapture.path, lastCapture.method, lastCapture.contentType)), + ); }); }); @@ -83,9 +75,19 @@ await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); const { port } = server.address(); const baseURL = `http://127.0.0.1:${port}`; +// Fixture ops that map one-to-one onto a (column, alias) aggregation method. +// An empty arg falls through to the SDK's own default. +const AGGREGATIONS = new Set(["count", "sum", "avg", "min", "max", "countDistinct"]); + +// applyQueryOps replays a fixture operation chain onto a query builder. +// Fixtures always put select first, so rebuilding on select is safe. function applyQueryOps(wh, table, operations) { let q = wh.from(table).select(); for (const op of operations) { + if (AGGREGATIONS.has(op.method)) { + q = q[op.method](op.args[0] || undefined, op.args[1] || undefined); + continue; + } switch (op.method) { case "select": q = wh.from(table).select(...op.args); @@ -96,24 +98,6 @@ function applyQueryOps(wh, table, operations) { case "where": q = q.where(op.args[0], op.args[1], op.args[2]); break; - case "count": - q = q.count(op.args[0] || "*", op.args[1] || "count"); - break; - case "sum": - q = q.sum(op.args[0], op.args[1] || undefined); - break; - case "avg": - q = q.avg(op.args[0], op.args[1] || undefined); - break; - case "min": - q = q.min(op.args[0], op.args[1] || undefined); - break; - case "max": - q = q.max(op.args[0], op.args[1] || undefined); - break; - case "countDistinct": - q = q.countDistinct(op.args[0], op.args[1] || undefined); - break; case "aggregate": q = q.aggregate(op.args[0], op.args[1], op.args[2]); break; @@ -137,6 +121,38 @@ function applyQueryOps(wh, table, operations) { return q; } +// insertArg returns an ingest case's payload. Hard failure on a malformed +// case, matching the Go harness. +function insertArg(tc) { + const op = tc.operations?.[0]; + if (op?.method !== "insert" || !op.args?.length) { + throw new Error(`${tc.name}: ingest case needs an insert operation with one arg`); + } + return op.args[0]; +} + +// One entry per fixture `endpoint` value. A case naming an endpoint that is +// missing here counts as skipped and fails the run — see below. +const ENDPOINTS = { + query: (wh, tc) => applyQueryOps(wh, tc.table, tc.operations ?? []).fetch(), + ingest: (wh, tc) => wh.from(tc.table).insert(insertArg(tc)), + ingest_batch: (wh, tc) => wh.from(tc.table).insert(insertArg(tc)), + pipe: (wh, tc) => wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(), + sql: (wh, tc) => wh.sql(tc.sql), + health: (wh) => wh.sys.health(), + schema_list: (wh) => wh.schema.list(), + schema_refresh: (wh) => wh.schema.refresh(), + policy_get: (wh) => wh.policy.get(), + policy_set: (wh, tc) => wh.policy.set(tc.policy_body), + policy_validate: (wh, tc) => wh.policy.validate(tc.policy_body), + dlq_list: (wh) => wh.dlq.list(), + dlq_table: (wh, tc) => wh.dlq.table(tc.table), + pipes_list: (wh) => wh.pipes.list(), + pipes_get: (wh, tc) => wh.pipes.get(tc.pipe_name), + pipes_set: (wh, tc) => wh.pipes.set(tc.pipe_name, tc.pipe_def), + pipes_delete: (wh, tc) => wh.pipes.delete(tc.pipe_name), +}; + // Compare request URIs by meaning: same path, same decoded query values, // regardless of + vs %20 spelling or parameter order (mirrors the Go harness). function normalizePath(p) { @@ -169,94 +185,33 @@ function sortKeys(v) { let passed = 0; let failed = 0; -let skipped = 0; const skippedNames = []; const failures = []; for (const tc of cases) { - resetCapture(); + lastCapture = { method: "", path: "", contentType: "", body: "" }; const wh = createClient({ baseURL, options: { maxRetries: 0 } }); + const run = ENDPOINTS[tc.endpoint]; + if (!run) { + skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); + continue; + } + try { - switch (tc.endpoint) { - case "query": { - const q = applyQueryOps(wh, tc.table, tc.operations ?? []); - await q.fetch(); - break; - } - case "ingest": - case "ingest_batch": - if (tc.operations?.[0]?.method !== "insert") { - // Hard failure, matching the Go harness. - throw new Error(`${tc.name}: ingest case has no insert operation`); - } - await wh.from(tc.table).insert(tc.operations[0].args[0]); - break; - case "pipe": - await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); - break; - case "sql": - await wh.sql(tc.sql); - break; - case "health": - await wh.sys.health(); - break; - case "schema_list": - await wh.schema.list(); - break; - case "schema_refresh": - await wh.schema.refresh(); - break; - case "policy_get": - await wh.policy.get(); - break; - case "policy_set": - await wh.policy.set(tc.policy_body); - break; - case "policy_validate": - await wh.policy.validate(tc.policy_body); - break; - case "dlq_list": - await wh.dlq.list(); - break; - case "dlq_table": - await wh.dlq.table(tc.table); - break; - case "pipes_list": - await wh.pipes.list(); - break; - case "pipes_get": - await wh.pipes.get(tc.pipe_name); - break; - case "pipes_set": - await wh.pipes.set(tc.pipe_name, tc.pipe_def); - break; - case "pipes_delete": - await wh.pipes.delete(tc.pipe_name); - break; - default: - // Not a pass — the Go harness hard-fails on these; we count and exit - // non-zero below. Fixture cases with a new endpoint value must be - // wired up here before they count. - skipped++; - skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); - continue; - } + await run(wh, tc); const errs = []; - - if (tc.expected_method && lastCapture.method !== tc.expected_method) { - errs.push(`method: want ${tc.expected_method}, got ${lastCapture.method}`); + for (const [what, want, got] of [ + ["method", tc.expected_method, lastCapture.method], + ["content-type", tc.expected_content_type, lastCapture.contentType], + ]) { + if (want && want !== got) errs.push(`${what}: want ${want}, got ${got}`); } - if (tc.expected_path && normalizePath(lastCapture.path) !== normalizePath(tc.expected_path)) { errs.push(`path: want ${tc.expected_path}, got ${lastCapture.path}`); } - if (tc.expected_content_type && lastCapture.contentType !== tc.expected_content_type) { - errs.push(`content-type: want ${tc.expected_content_type}, got ${lastCapture.contentType}`); - } - if (tc.expected_raw_body !== undefined) { if (lastCapture.body !== tc.expected_raw_body) { errs.push(`raw body:\n want: ${tc.expected_raw_body}\n got: ${lastCapture.body}`); @@ -290,6 +245,7 @@ for (const tc of cases) { server.closeAllConnections?.(); server.close(); +const skipped = skippedNames.length; console.log( `\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${skipped} skipped, ${cases.length} total\n`, ); From 059ad16a59ab3273dd9bf017c1dad9de3e9108a0 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 10:05:32 -0400 Subject: [PATCH 45/59] docs(go-sdk): cut duplication and padding from the Go SDK docs The Go module README duplicated all six /sdk/go/* pages; it is now a real module README (what it is, go get, one quickstart, links to the docs site). The docs pages keep every documented symbol and every non-trivial example, with the repeated preambles, restated caveats, and over-narration removed: - clients/go/README.md 225 -> 59 - sdk/go/queries.md 325 -> 284 - sdk/go/streaming.md 249 -> 217 - sdk/go/reference.md 209 -> 185 - sdk/go/index.md 198 -> 178 - sdk/go/admin.md 100 -> 94 - sdk/go/pipes.md 85 -> 79 - sdk/index.mdx +16 -> +9 (Go card folded into the existing grid) No API surface dropped: symbol sets before/after are identical. Verified with make lint-md, make lint-prose, and make build-docs (all internal links valid). --- clients/go/README.md | 198 ++-------------------- docs/src/content/docs/sdk/go/admin.md | 24 +-- docs/src/content/docs/sdk/go/index.md | 66 +++----- docs/src/content/docs/sdk/go/pipes.md | 26 ++- docs/src/content/docs/sdk/go/queries.md | 87 +++------- docs/src/content/docs/sdk/go/reference.md | 56 ++---- docs/src/content/docs/sdk/go/streaming.md | 86 +++------- docs/src/content/docs/sdk/index.mdx | 11 +- 8 files changed, 126 insertions(+), 428 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index 64e19cac..68809b8c 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -1,10 +1,8 @@ # WaveHouse Go SDK -Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. +Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. Zero third-party runtime dependencies, SSE parser included. -**Zero third-party runtime dependencies** (stdlib only). - -**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** +**Full documentation: [wavehouse.dev/sdk/go](https://wavehouse.dev/sdk/go)** ## Install @@ -12,7 +10,9 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a s go get github.com/Wave-RF/WaveHouse/clients/go ``` -## Quick Start +Requires Go 1.24 or newer. + +## Quick start ```go package main @@ -26,33 +26,16 @@ import ( ) func main() { - // Create an unauthenticated client (uses the server's default_role). - client := wavehouse.NewClient(wavehouse.Config{ + wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), // omit for unauthenticated access }) - // Health check. - if err := client.Sys.Health(context.Background()); err != nil { - log.Fatal(err) - } - - ctx := context.Background() - - // Insert a row. - _, err := client.From("clicks").Insert(ctx, map[string]any{ - "page": "/home", "button": "cta", - }) - if err != nil { - log.Fatal(err) - } - - // Query with the fluent builder. - page, err := client.From("clicks"). + page, err := wh.From("clicks"). Select("page", "button"). Where("page", wavehouse.OpEq, "/home"). - OrderBy("page", "asc"). Limit(10). - FetchUntyped(ctx) + FetchUntyped(context.Background()) if err != nil { log.Fatal(err) } @@ -62,163 +45,14 @@ func main() { } ``` -## Authentication - -```go -// Static token. -client := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), -}) - -// Dynamic token (e.g. rotated). -client = wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: func(ctx context.Context) (string, error) { - return fetchFreshToken(ctx) - }, -}) -``` - -`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`). A trailing `/` is trimmed, and request paths are appended, for both REST and SSE alike ([Config](https://wavehouse.dev/sdk/go#config)). - -## Typed Queries (Generics) - -```go -type ClickRow struct { - Page string `json:"page"` - Button string `json:"button"` - DurationMS int `json:"duration_ms"` -} - -page, err := wavehouse.FetchTyped[ClickRow](ctx, - client.From("clicks").Select("page", "button", "duration_ms").Limit(100), -) -// page.Data is []ClickRow -``` - -## Batch Insert (NDJSON) - -```go -// Array of maps — serialized to NDJSON automatically. -result, _ := client.From("clicks").Insert(ctx, []map[string]any{ - {"page": "/a", "button": "cta"}, - {"page": "/b", "button": "nav"}, -}) -// result.OK, result.Total, result.Succeeded, result.Failed - -// Pre-formatted NDJSON string. -result, _ = client.From("clicks").InsertNDJSON(ctx, - `{"page":"/a"}`+"\n"+`{"page":"/b"}`, -) -``` - -## Streaming (SSE) - -```go -stream := client.From("clicks").Stream(&wavehouse.StreamOptions{ - Since: "2026-01-01T00:00:00Z", -}) -defer stream.Close() - -// Channel-based consumption. -for event := range stream.Events() { - fmt.Println(event.Table, event.Data) -} - -// Or callback-based. -unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ - Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, - Status: func(s wavehouse.StreamStatus) { fmt.Println("status:", s) }, -}) -defer unsub() -``` - -## Live Queries - -```go -lq := client.From("clicks"). - SelectAll(). - OrderBy("received_timestamp", "desc"). - Limit(100). - LiveQuery(&wavehouse.StreamSubscriber{ - Initial: func(rows []map[string]any, err error) { - // Historical backfill. - fmt.Println("initial rows:", len(rows)) - }, - Next: func(e wavehouse.StreamEvent) { - // Live events after backfill. - fmt.Println("live:", e.Data) - }, - }, nil) -defer lq.Close() -``` - -## Named Pipes - -```go -// Execute a pipe. -rows, _ := wavehouse.Fetch[map[string]any](ctx, - client.Pipe("top_pages", map[string]any{"limit": 10}), -) - -// Admin: manage pipes. -client.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ - SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", - AllowedRoles: []string{"viewer", "admin"}, -}) -pipes, _ := client.Pipes.List(ctx) -client.Pipes.Delete(ctx, "old_pipe") -``` - -## Admin - -```go -// Schema introspection (admin-only). -schemas, _ := client.Schema.List(ctx) -client.Schema.Refresh(ctx) - -// Policy management (admin-only). -policy, _ := client.Policy.Get(ctx) -client.Policy.Set(ctx, policy) -result, _ := client.Policy.Validate(ctx, policy) - -// DLQ stats (admin-only). -stats, _ := client.DLQ.List(ctx) - -// Raw SQL (admin-only). -rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM clicks") -``` - -## Codegen - -Generate Go structs from a running WaveHouse instance: - -```bash -export WAVEHOUSE_AUTH='' # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ - --url http://localhost:8080 \ - --out ./db_types.go \ - --package myapp -``` - -See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#codegen-cli). - -## Error Handling - -Request-response ops return `(T, error)`, or bare `error` for body-less calls (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP errors are `*wavehouse.Error` (unwrap with `errors.As`); failures before the request goes out (`Auth` provider, body marshal) are plain wrapped errors, so handle `errors.As == false` too. Streaming lifecycle (`Stream`, `Subscribe`, `Close`, `Connected`) reports via callbacks or plain errors: - -```go -page, err := client.From("clicks").Fetch(ctx) -if err != nil { - var whErr *wavehouse.Error - if errors.As(err, &whErr) { - fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) - } -} -``` +## Documentation -The HTTP layer retries 5xx, 429, and network errors with exponential backoff (2 retries by default). `Retry-After` on 503/429 is honored, capped at 30s. Context cancellation returns `ABORTED` immediately. +- [Go SDK](https://wavehouse.dev/sdk/go) — client config, auth, typed rows via generics, error handling. +- [Queries](https://wavehouse.dev/sdk/go/queries) — tables, the query builder, inserts, pagination, raw SQL. +- [Streaming & Live Queries](https://wavehouse.dev/sdk/go/streaming) — SSE streams, client-side filtering, backfill-then-live. +- [Pipes](https://wavehouse.dev/sdk/go/pipes) — execute and manage named query pipes. +- [Admin & System](https://wavehouse.dev/sdk/go/admin) — schema, policy, DLQ stats, health. +- [Reference & CLI](https://wavehouse.dev/sdk/go/reference) — error codes, the full API tree, and the `wavehouse-codegen` struct generator. ## License diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index 7e06eeed..6475a070 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -3,13 +3,11 @@ title: "Go SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." --- -Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. - -Every namespace on this page is admin-gated: the server mounts them under `/v1/ops/*` behind one gate, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. Every namespace here except `client.Sys.Health` sits behind the server's admin gate on `/v1/ops/*`, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). Without one, these calls return a `*wavehouse.Error` with `Status: 403` — unless the deployment sets `default_role` to admin, which is dev-only. See [Access Control](/access-control), and the TypeScript SDK's [Admin & System](/sdk/admin) page. ## Schema — `client.Schema` -Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-gated** `/v1/ops/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` all hit `/v1/ops/schema*`. ```go // List all table schemas. @@ -18,15 +16,15 @@ schemas, err := wh.Schema.List(ctx) // Force refresh from ClickHouse. err = wh.Schema.Refresh(ctx) -``` -Individual table schema: `wh.From("clicks").Schema(ctx)`. +// One table: wh.From("clicks").Schema(ctx) +``` --- ## Policy — `client.Policy` -Manage Hasura-style access control policies (admin role required). +Manage Hasura-style access control policies. ```go // Get current policy. @@ -57,20 +55,16 @@ result, err := wh.Policy.Validate(ctx, policyDraft) // result.Valid == true, or err wraps the validation failure details ``` -`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string` to distinguish empty strings from absent operators. Use a helper: - -```go -func strPtr(s string) *string { return &s } -``` +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator — hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. --- ## DLQ — `client.DLQ` -Dead Letter Queue operations (admin role required). +Dead Letter Queue statistics. ```go -// Get DLQ statistics. +// Totals across tables. stats, err := wh.DLQ.List(ctx) // stats.Tables: map[string]int{"clicks": 3, "users": 0} // stats.Total: 3 @@ -85,7 +79,7 @@ stats, err = wh.DLQ.Table(ctx, "clicks") ## System — `client.Sys` -Server-online check. +The one surface on this page that needs no credentials. ```go // Health hits the public, content-free /v1/health route — 200 → nil error, diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 65afc19f..e1f496e0 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -3,10 +3,10 @@ title: "Go SDK" description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." --- -`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime dependency Go client for WaveHouse (stdlib only). +`github.com/Wave-RF/WaveHouse/clients/go` — a Go client for WaveHouse with zero third-party runtime dependencies, SSE parser included. :::tip[Looking for the TypeScript SDK?] -This page and the rest of `/sdk/go/*` cover the Go client. The JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at [SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything you learn about WaveHouse's query builder, streaming, or admin endpoints on either page mostly carries over. +`/sdk/go/*` covers the Go client; the JavaScript/TypeScript client (`@wavehouse/sdk`) starts at [SDK Overview](/sdk). Both speak the same wire format, so concepts carry over — only the [API shapes differ](#differences-from-the-typescript-sdk). ::: ## Installation @@ -15,15 +15,13 @@ This page and the rest of `/sdk/go/*` cover the Go client. The JavaScript/TypeSc go get github.com/Wave-RF/WaveHouse/clients/go ``` -Requires Go 1.24+ (the `go.mod` floor, matching supported releases rather than server's patch-pinned toolchain). - -## Import +Requires Go 1.24+ (the `go.mod` floor, which tracks supported releases rather than the server's patch-pinned toolchain). ```go import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ``` -Aliasing `wavehouse` is optional but keeps call sites short; all examples here assume it. +The `wavehouse` alias is optional but keeps call sites short; all examples here assume it. ## Quick Start @@ -58,19 +56,16 @@ func main() { } ``` -Find more examples in the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md). - ## Creating a Client +`Auth` is any function returning a token; `wavehouse.StaticToken(token)` wraps a fixed one, as in the Quick Start above. + ```go wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "https://wavehouse.example.com", Auth: func(ctx context.Context) (string, error) { return myAuthProvider.GetToken(ctx) }, - Options: &wavehouse.ClientOptions{ - MaxRetries: 2, - }, }) ``` @@ -84,7 +79,7 @@ wh := wavehouse.NewClient(wavehouse.Config{ | `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports. | :::caution[Timeouts: use contexts, not `http.Client.Timeout`] -The default client has no `Timeout`; use a `context.Context` deadline to prevent hangs. If supplying your own `HTTPClient`, leave `Timeout` unset, as it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. +The default client sets no `Timeout`; use a `context.Context` deadline to prevent hangs. Leave `Timeout` unset on a custom `HTTPClient` too — it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. ::: ### `ClientOptions` @@ -94,13 +89,13 @@ The default client has no `Timeout`; use a `context.Context` deadline to prevent | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | | `Headers` | `map[string]string` | `nil` | Sent on every request the client makes — REST calls and SSE streams alike. | -`*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. +`*Client` is safe for concurrent use — state is immutable after `NewClient` and builder chains copy — provided your `Auth` function is concurrency-safe. :::caution[`Options` opts you out of the default, not just in] -The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. +The 2-retry default applies only when `Config.Options` is `nil`. Passing `&wavehouse.ClientOptions{}` leaves `MaxRetries` at Go's zero value (`0`), which disables retries — set it explicitly. ::: -`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata that has no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): ```go wh := wavehouse.NewClient(wavehouse.Config{ @@ -112,30 +107,17 @@ wh := wavehouse.NewClient(wavehouse.Config{ }) ``` -The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any entry that collides. Names are matched case-insensitively (`net/http` canonicalizes them), and each entry replaces rather than appends. The map is copied at `NewClient`, so mutating it afterwards changes nothing. - -For the two remaining TypeScript knobs there is no Go field, because `Config.HTTPClient` already covers them: `options.fetch` maps to supplying your own `*http.Client`, and `options.fetchOptions` maps to a custom `http.RoundTripper` on that client's `Transport`. - -For static tokens, use `wavehouse.StaticToken(token)`: - -```go -wh := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), -}) -``` +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both — supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. :::note[How the token is transmitted] -The Go SDK sends `Authorization: Bearer ` on every request, including SSE streams, and never uses a `?token=` query fallback. Both SDKs work this way: the TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is now the shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. ::: :::caution[A credentialed stream will not follow a redirect] When the stream request carries a credential — an `Auth` token or a `ClientOptions.Headers` entry — the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. ::: -:::caution[Use HTTPS for authenticated non-local servers] -While the SDK allows `http://` for local development or private networks, bearer tokens over plaintext HTTP are insecure. Use `https://` for endpoints outside trusted networks. -::: +Use `https://` for any authenticated server outside a trusted network. The SDK allows `http://` for local development and private networks, but bearer tokens over plaintext are insecure. ## Typed Rows (Generics) @@ -154,13 +136,11 @@ page, err := wavehouse.FetchTyped[ClickRow](ctx, // page.Data is []ClickRow ``` -Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. - -`FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods. Untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. +Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. `FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods; the untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. ## Error Handling -Request-response operations (queries, ingest, pipes, admin) return `(T, error)` or just `error` if no body exists (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (e.g., `Auth` provider, marshal errors) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) use callbacks or plain errors; see [Streaming](/sdk/go/streaming). +Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, or a bare `error` when there is no body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (`Auth` provider, marshal errors) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) report through callbacks or plain errors; see [Streaming](/sdk/go/streaming). ```go page, err := wh.From("clicks").Fetch(ctx) @@ -179,15 +159,15 @@ See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry b ## Differences from the TypeScript SDK -Both SDKs share a wire format and feature set, verified by a shared `wire_cases.json` fixture in CI to ensure equivalent HTTP requests for builder calls. However, API shapes differ: +Both SDKs share a wire format and feature set, verified in CI against a shared `wire_cases.json` fixture that asserts equivalent HTTP requests for builder calls. The API shapes differ: -- **No `Result` union.** Go returns `(T, error)`. A non-nil `error` is the only failure signal; no `{ok, data, error}` objects or `error: null` sentinels are used. -- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` as the first argument. Use timeout or `cancel()` instead of `AbortController`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). -- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). -- **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. -- **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. -- **No third-party dependencies.** The Go SDK is stdlib-only, including its SSE frame parser. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). -- **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). +- **No `Result` union.** Go returns `(T, error)`; a non-nil `error` is the only failure signal. No `{ok, data, error}` objects, no `error: null` sentinels. +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` first; use a deadline or `cancel()`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` take no context; the returned `*StreamController` owns its goroutine and connection until `.Close()` (usually deferred). See [Streaming](/sdk/go/streaming). +- **Generics on package functions.** Go has no type parameters on methods, so use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. +- **No implicit "await."** `QueryBuilder` is not `PromiseLike`; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly. +- **No third-party dependencies.** Stdlib only, SSE frame parser included. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). +- **Any slice batches.** Reflection lets `[]ClickRow{...}` take the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 7ec96f91..7cbee2c8 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -3,21 +3,15 @@ title: "Go SDK Pipes" description: "Execute and manage named query pipes with the WaveHouse Go SDK." --- -Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. +Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages their definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. ## Named Pipes — `client.Pipe(name, params)` -Execute a pre-defined named query pipe. Returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you must explicitly call `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. - -```go -rows, err := wavehouse.Fetch[map[string]any](ctx, - wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), -) -``` +Returns a `*PipeRef` for a pre-defined named query pipe. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you execute it explicitly with `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. Pass `nil` for `params` if the pipe takes none, or only needs its server-side defaults. ### `wavehouse.Fetch[Row](ctx, pipeRef)` -Execute and decode results into `[]Row`. Package-level generic function (Go has no generic methods) — same pattern as `FetchTyped` for queries and `SQL` for raw SQL. +Executes the pipe and decodes results into `[]Row`. A package-level generic function, since Go has no generic methods — the same pattern as `FetchTyped` for queries and `SQL` for raw SQL. ```go type TopPage struct { @@ -25,22 +19,22 @@ type TopPage struct { Views int `json:"views"` } -rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{"limit": 50})) +rows, err := wavehouse.Fetch[TopPage](ctx, + wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), +) ``` ### `.FetchUntyped(ctx)` -Execute and decode results into `[]map[string]any`. The non-generic method form of `Fetch`. +The non-generic method form: decodes results into `[]map[string]any`. ```go rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) ``` -Pass `nil` for `params` if the pipe takes none or only requires server-side defaults. - ### `.Stream(opts)` -Open a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). Streams by table name using the pipe's own name, so it works only when that name is a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. +Opens a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). It streams by table name using the pipe's own name, so it works only where that name is also a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. ```go stream := wh.Pipe("top_pages", nil).Stream(nil) @@ -50,7 +44,7 @@ stream := wh.Pipe("top_pages", nil).Stream(nil) ## Pipes Admin — `client.Pipes` -Manage named query pipes. These sit behind the admin gate on `/v1/ops/*`, which a caller clears one of two ways: a JWT resolving to the policy admin role (`policy.admin_role`), or the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). +Create, read, and delete pipe definitions. These sit behind the [admin gate](/sdk/go/admin) on `/v1/ops/*`. ```go // List all pipes. @@ -73,7 +67,7 @@ err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ err = wh.Pipes.Delete(ctx, "old_pipe") ``` -`PipeDef` is `Pipe` minus `Name` — the name is the method's path argument: +`PipeDef` is `Pipe` minus `Name`, which the methods take as a path argument: ```go type PipeDef struct { diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 6004ba17..282ba25a 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -3,11 +3,11 @@ title: "Go SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." --- -Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` as its first argument and returns `(T, error)`; the chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` first and returns `(T, error)`, the chainable builder methods and `.Stream(opts)` excepted — see [Error Handling](/sdk/go#error-handling). The TypeScript SDK covers the same surface on its [Queries](/sdk/queries) page, with a `Result`-returning, `PromiseLike` builder. ## Tables — `client.From(table)` -`From` returns a `*TableRef`. It performs no request, making it safe to store or pass around. +`From` returns a `*TableRef`. It performs no request, so it is safe to store or pass around. ```go clicks := wh.From("clicks") @@ -15,9 +15,7 @@ clicks := wh.From("clicks") ### `.Fetch(ctx)` -Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`). Internally it is `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. Unlike the TypeScript SDK's `.fetch(opts?)`, there is no options struct to override the limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself ([Query Builder](#query-builder)). - -Access-control policies restrict returned columns; `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). +Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`) — internally `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. There is no options struct as in the TypeScript SDK's `.fetch(opts?)`: to override the limit or paginate, chain `.SelectAll().Limit(n).OrderBy(...)` yourself ([Query Builder](#query-builder)). Access-control policies restrict the returned columns, and `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). ```go page, err := clicks.Fetch(ctx) @@ -29,14 +27,12 @@ for _, row := range page.Data { } ``` -For pagination, use the query builder with `.OrderBy()` (see [Pagination](#pagination)). - ### `.Insert(ctx, data)` Inserts one or many rows based on the input type: -- **Map or struct** (excluding slices and `[]byte`): Sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. -- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): Serialized to NDJSON via reflection and sent as one `application/x-ndjson` request. Per-record outcomes are returned in the result. +- **Map or struct** (excluding slices and `[]byte`): sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. +- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): serialized to NDJSON via reflection and sent as one `application/x-ndjson` request, with per-record outcomes in the result. ```go // Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) @@ -60,9 +56,7 @@ res, err = clicks.Insert(ctx, []ClickRow{ }) ``` -For batches, `res.OK` is `true` only if all records succeeded (`*res.Failed == 0`). Check `res.Failed` and `res.Results` (each `InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for partial failures. The returned `error` indicates whole-request failures (network, `404`, `403`, `503`). Empty slices are no-ops. - -> The server is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object (`Content-Type` is only a hint). See [API reference](/api#post-v1ingesttabletable--ingest-data). +For batches, `res.OK` is `true` only if every record succeeded (`*res.Failed == 0`); check `res.Failed` and `res.Results` (each an `InsertRecordResult{Index, OK, Duplicate, Error}` with a 1-based `Index`) for partial failures. The returned `error` signals whole-request failures instead (network, `404`, `403`, `503`), and empty slices are no-ops. The server itself is format-agnostic — `POST /v1/ingest` also accepts a raw JSON array or a single object, since `Content-Type` is only a hint ([API reference](/api#post-v1ingesttabletable--ingest-data)). ### `.InsertNDJSON(ctx, ndjson)` @@ -92,18 +86,11 @@ schema, err := clicks.Schema(ctx) ### `.Select(...columns)` -Start a query builder chain. See [Query Builder](#query-builder). - -```go -page, err := clicks.Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Limit(10). - FetchUntyped(ctx) -``` +Starts a query builder chain — see [Query Builder](#query-builder) for the chainable methods and how to execute it. ### `.SelectAll()` -Selects every column your role is allowed to read. This is the explicit version of `.Fetch()`. It is mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles, the server expands this to allowed columns rather than a bare `SELECT *`; it never bypasses `deny_columns`/`allow_columns` (see [Access control → Column permissions](/access-control#column-permissions)). +The explicit version of `.Fetch()`: selects every column your role may read. Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles the server expands it to the allowed columns rather than a bare `SELECT *`, so it never bypasses `deny_columns`/`allow_columns` ([Access control](/access-control#column-permissions)). ```go page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) @@ -111,15 +98,11 @@ page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10) ### `.Stream(opts)` -Open a real-time event subscription. See [Streaming](/sdk/go/streaming). - -```go -stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) -``` +Opens a real-time event subscription on the table, e.g. `clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"})`. See [Streaming](/sdk/go/streaming). ## Query Builder -Returned by `tableRef.Select(...)` or `tableRef.SelectAll()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: +Returned by `tableRef.Select(...)` or `tableRef.SelectAll()`. Immutable — every chain method returns a new `*QueryBuilder`, leaving the original unchanged. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: ```go page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) @@ -127,11 +110,9 @@ page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) ### Chain Methods -All methods return a new `*QueryBuilder`; the original remains unchanged. - #### `.Select(...columns)` -Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*`—use `.SelectAll()` for all columns. +Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*` — use `.SelectAll()` for all columns. ```go q := clicks.Select("page").Select("button") // SELECT page, button @@ -139,7 +120,7 @@ q := clicks.Select("page").Select("button") // SELECT page, button #### `.SelectAll()` -Selects every readable column (expanded server-side based on role). Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`, etc.). +Same expansion rules as [`tableRef.SelectAll()`](#selectall), from an existing builder. ```go q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") @@ -180,35 +161,27 @@ clicks.Select("page"). Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn ``` -Custom functions via `.Aggregate(fn, column, alias)` are validated server-side (case-insensitive). Allowlist: `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Others return `400 unsupported aggregation function`. - -`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `Aggregate` takes `(fn, column, alias)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; `CountDistinct` uses `count_distinct_`. `Aggregate` has no default; pass one or it is sent as `""`. +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `.Aggregate(fn, column, alias)` runs a custom function, validated server-side (case-insensitively) against the allowlist `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp` — anything else returns `400 unsupported aggregation function`. With an empty alias, `Count` defaults to `count` (and `column=""` becomes `*`), `Sum`/`Avg`/`Min`/`Max` to `sum_`/`avg_`/`min_`/`max_`, and `CountDistinct` to `count_distinct_`; `Aggregate` has no default and sends `""`. #### `.GroupBy(...columns)` -```go -clicks.Select("page").Count("", "").GroupBy("page") -``` +Group the result set, as in `clicks.Select("page").Count("", "").GroupBy("page")`. #### `.OrderBy(column, dir)` +`dir` defaults to `"asc"` if `""`. + ```go clicks.Select("page").Count("", "total").OrderBy("total", "desc") ``` -`dir` defaults to `"asc"` if `""`. - #### `.Limit(n)` -```go -clicks.Select().Limit(100) -``` - -If unspecified, `wavehouse.DefaultLimit` (1000) is applied. The server also enforces a maximum (`query.default_max_rows`, default 10,000). +Caps the row count, as in `clicks.Select().Limit(100)`. Defaults to `wavehouse.DefaultLimit` (1000); the server also enforces a maximum (`query.default_max_rows`, default 10,000). #### `.TimeRange(column, since, until)` -Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for open-ended ranges. +Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for an open-ended range. ```go clicks.Select("page").TimeRange("received_timestamp", "1h", "") @@ -219,7 +192,7 @@ clicks.Select("page").TimeRange( #### `.CacheTTL(seconds)` -Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time. See [#280](https://github.com/Wave-RF/WaveHouse/issues/280). +Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time ([#280](https://github.com/Wave-RF/WaveHouse/issues/280)). ```go clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 @@ -247,16 +220,6 @@ Executes the query and decodes rows into `[]map[string]any`. ```go page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) -if err != nil { - return err -} - -if page.HasMore && page.Next != nil { - page, err = page.Next(ctx) // cursor-based pagination — needs OrderBy - if err != nil { - return err - } -} ``` ### `.Stream(opts)` @@ -265,7 +228,7 @@ Opens a live stream from the builder's table with client-side filtering and proj ### Pagination -`Page[T]`: +Both fetch methods return a `Page[T]`: ```go type Page[T any] struct { @@ -275,11 +238,7 @@ type Page[T any] struct { } ``` -If `Limit` is set and results meet that limit, `HasMore` is `true`. `Next` walks the **first** `.OrderBy()` column using a filter on the last row's value; thus, `Next` requires an explicit `.OrderBy()`. Without one, `Next` is `nil`. If the order column is omitted from `.Select(...)`, `Next` returns an empty page. - -The cursor filter is strict (`gt`/`lt` on the first `.OrderBy()` column, no tie-breaker), so rows sharing a boundary value with the last row are skipped. Paginate on a per-row-unique column, or accept dropped ties; the TypeScript SDK's `next()` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). - -On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row. `FetchTyped` with an `int64` field, or codegen structs, keep it exact. +`HasMore` is `true` when a `Limit` is set and the results meet it. `Next` walks the **first** `.OrderBy()` column by filtering on the last row's value, so it requires an explicit `.OrderBy()` — without one `Next` is `nil`, and if the order column is missing from `.Select(...)` it returns an empty page. The cursor filter is strict (`gt`/`lt`, no tie-breaker), so rows sharing a boundary value with the last row are skipped: paginate on a per-row-unique column or accept dropped ties, a limitation the TypeScript SDK's `next()` shares ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row; `FetchTyped` with an `int64` field, or codegen structs, keep it exact. ```go page, err := clicks.Select(). @@ -302,7 +261,7 @@ for page.HasMore && page.Next != nil { ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query via `/v1/ops/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/ops/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. +Executes a raw SQL query via the admin-only `/v1/ops/query`. A JWT must resolve to the admin role (`admin_role`, default `"admin"`); requests without a valid token fall back to `default_role` and are rejected unless that role is admin (dev-only). An operator key authorizes `/v1/ops/*` as well, but since `Config.Auth` always sends `Bearer `, pass it by giving `Config.HTTPClient` a `Transport` that sets the `X-Operator-Key` header ([API authentication](/api#authentication)). Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, @@ -321,5 +280,5 @@ typed, err := wavehouse.SQL[PageTotal](ctx, wh, ``` :::note[No parameter binding through the SDK] -Positional `?` substitution is unsupported. The SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` lacks a hook to add them. Use inline literals or the structured query builder (`wh.From(table)...`) for safe binding of user input. +Positional `?` substitution is unsupported, and the SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` has no hook to add them. Use inline literals, or the structured query builder (`wh.From(table)...`) for safe binding of user input. ::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index b7123fce..4bdb794c 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -3,11 +3,11 @@ title: "Go SDK Reference & CLI" description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." --- -Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference). ## Context Cancellation -Non-streaming operations take a `context.Context` as their first argument (similar to TypeScript's `AbortSignal`). Cancel it using a timeout or explicit `cancel()`: +Non-streaming operations take a `context.Context` as their first argument (the analog of TypeScript's `AbortSignal`). Cancel via a timeout or an explicit `cancel()`; cancellation returns immediately, without retrying, as `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -20,15 +20,11 @@ if errors.As(err, &whErr) && whErr.Code == "ABORTED" { } ``` -Cancellation returns immediately (no retry) with `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. - `.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming](/sdk/go/streaming#streamoptions). ## Error Handling -The SDK never panics on API or network failures. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less operations (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. - -HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-side failures (e.g., `Auth` provider, marshal failures) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`) do not return `(T, error)`; stream errors use the subscriber's `Error` callback. `Connected(ctx)` returns plain errors. This mirrors the TypeScript SDK's "never throws" guarantee. +The SDK never panics on API or network failures, mirroring the TypeScript SDK's "never throws" guarantee. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less ones (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`, or use `wavehouse.IsRetryable(err)` to shortcut the `errors.As` + `.Retryable` check); client-side failures such as an `Auth` provider or marshal error are plain wrapped errors, so handle the `errors.As == false` case — see the [worked example](/sdk/go#error-handling). Streaming methods (`Stream`, `Subscribe`, `Close`) report through the subscriber's `Error` callback instead, and `Connected(ctx)` returns plain errors. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -50,22 +46,7 @@ HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-sid | 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | | 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | -```go -page, err := wh.From("clicks").Fetch(ctx) -if err != nil { - var whErr *wavehouse.Error - if errors.As(err, &whErr) { - fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) - } else { - fmt.Println("client-side failure:", err) // auth provider, marshal, ... - } - return err -} -``` - -`wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. - -Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup to suppress duplicates. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. ## Full API Tree @@ -120,7 +101,7 @@ NewClient(Config) → *Client ## Codegen CLI -Generate Go structs from a running WaveHouse instance using the `wavehouse-codegen` command in `cmd/`: +Generate Go structs from a running WaveHouse instance with the `wavehouse-codegen` command in `cmd/`: ```bash export WAVEHOUSE_AUTH='' # avoids leaking the token via argv @@ -128,15 +109,12 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ --package myapp -``` -Or, inside `clients/go/`: - -```bash +# Or, from inside a checkout of clients/go/: go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` -Codegen reads the admin-only `/v1/ops/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. +Codegen reads the admin-only `/v1/ops/schema` endpoint, so a non-dev server needs an admin token or returns `403`. Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history and process listings. **Options:** @@ -148,9 +126,7 @@ Codegen reads the admin-only `/v1/ops/schema` endpoint; non-dev servers require | `--package`, `-p` | Go package name for the generated file | `main` | | `--help`, `-h` | Show usage and exit | — | -Output is processed via `go/format`. If a table or column name produces invalid Go source, codegen fails loudly. - -**Example output:** +**Example output** (for the [development quick-start](/development#quick-start) `clicks` table): ```go // Code generated by wavehouse-codegen. DO NOT EDIT. @@ -166,9 +142,7 @@ type ClicksRow struct { } ``` -(Example for the [development quick-start](/development#quick-start) `clicks` table; `received_timestamp` is `*string` + `,omitempty` due to its `DEFAULT` clause.) - -The generator does not special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part just gets its first letter upper-cased. Table and column names are converted to `PascalCase`; leading digits get an `X` prefix (e.g., `2fa_events` $\rightarrow$ `X2faEventsRow`). Columns with `has_default: true` become pointer fields with `,omitempty`: `nil` uses the server default, a pointed-at value is sent — including an explicit `0`/`false`/`""`. +Output is run through `go/format`, and codegen fails loudly if a table or column name would produce invalid Go source. Names become `PascalCase`, with an `X` prefix for a leading digit (`2fa_events` → `X2faEventsRow`); initialisms are not special-cased, so `event_id` becomes `EventId`, not `EventID`. Columns with `has_default: true` become pointer fields with `,omitempty` — as `received_timestamp` does above — where `nil` uses the server default and a pointed-at value is sent, including an explicit `0`/`false`/`""`. **ClickHouse → Go type mapping:** @@ -184,26 +158,28 @@ The generator does not special-case initialisms: `event_id` becomes `EventId`, n | `Decimal*` | `string` | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` (except `Array(UInt8)` $\rightarrow$ `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Array(T)` | `[]T` (except `Array(UInt8)` → `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | | `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | -Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/ops/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`; for the raw-SQL path (`/v1/ops/query`), which quotes 64-bit-and-wider integers, use `map[string]any` with `SQL[Row]`. ## Testing -Unit tests are colocated in `clients/go/` (module `clients/go/go.mod`), separate from the root `WaveHouse` module. The cross-language wire-format **conformance suite** uses `clients/go/conformance_test.go` to replay the shared fixture (`clients/go/testdata/wire_cases.json`), asserting correct HTTP methods, paths, content types, and bodies. The TypeScript half—`tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` (builds TS SDK first)—uses the same fixture; CI runs both to ensure wire format consistency. +Unit tests are colocated in `clients/go/`, which is its own module (`clients/go/go.mod`), separate from the root `WaveHouse` module: ```bash cd clients/go go test ./... ``` -E2E tests (build tag `e2e`) run against a live WaveHouse instance via a dedicated Make target: +The cross-language wire-format **conformance suite** replays a shared fixture (`clients/go/testdata/wire_cases.json`) from `clients/go/conformance_test.go`, asserting HTTP methods, paths, content types, and bodies. The TypeScript half — `tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` — replays the same fixture, and CI runs both to keep the wire formats in step. + +E2E tests (build tag `e2e`) run against a live WaveHouse instance via their own Make target: ```bash WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e ``` -`WAVEHOUSE_URL` defaults to `http://localhost:8080`; optional `WAVEHOUSE_AUTH` is for admin cases. The suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). +`WAVEHOUSE_URL` defaults to `http://localhost:8080`, and the optional `WAVEHOUSE_AUTH` covers the admin cases; the suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index e136c814..c1076d94 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -3,15 +3,13 @@ title: "Go SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." --- -Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/go/queries). Compare with the TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) page — the two implement the same protocol and mostly the same client-side filtering, but connection lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Frames are parsed over `net/http` with no runtime dependencies. Builders and table refs come from [Queries](/sdk/go/queries). The TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) implements the same protocol and mostly the same client-side filtering, but the lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. ## Streaming -Streams use SSE (Server-Sent Events) parsed via `net/http` with zero runtime dependencies. - ### `*StreamController` -Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` returns immediately; the connection opens in a background goroutine. +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). `.Stream` returns immediately; the connection opens in a background goroutine. ```go stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ @@ -39,17 +37,14 @@ unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ }, }) -// Cleanup — removes this subscriber; the connection stays open for any -// others and must still be closed with stream.Close() when you're done -// with the stream itself. +// Removes this subscriber only — the connection stays open for any others +// and still needs stream.Close() when you're done with the stream itself. defer unsub() ``` -Cleanup via `unsub()` removes the subscriber; the connection remains open for others and must be closed with `stream.Close()`. - ### Channel-based consumption — `.Events()` -A read-only channel, closed automatically when the stream shuts down. +A read-only channel, closed automatically when the stream shuts down. It is buffered (256 events) and buffers from `.Stream()` onward, so events arriving before the first `Events()` call are not lost. A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop; the first drop logs via `log`, later drops are silent, and `.Subscribe` callbacks fire regardless. ```go stream := wh.From("clicks").Stream(nil) @@ -67,27 +62,21 @@ for event := range stream.Events() { Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. ::: -The channel is buffered (256 events). A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop. The first drop logs via `log`; later drops are silent (`.Subscribe` callbacks fire regardless). +:::note[`Events()` carries events only] +`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404), so pair `Events()` with a subscriber to learn why a stream ended. +::: ### `.Close()` -Explicitly closes the stream and releases resources. Non-blocking and safe to call from inside a subscriber callback. - -```go -stream.Close() -``` +`stream.Close()` closes the stream and releases its resources. Non-blocking, and safe to call from inside a subscriber callback. ### `.Status()` -Returns the current `StreamStatus`. - -```go -status := stream.Status() -``` +`stream.Status()` returns the current `StreamStatus`: `StatusConnecting`, `StatusLive`, `StatusReconnecting`, or `StatusClosed`. ### `.Connected(ctx)` -Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (e.g., in tests). +Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (in tests, for instance). ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -103,7 +92,7 @@ if err := stream.Connected(ctx); err != nil { | ----- | ---- | ----------- | | `Since` | `string` | RFC3339 timestamp for gap-fill replay | -There's no `Signal`/context field: a stream isn't canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. +There is no `Signal`/context field: a stream is not canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. ### `StreamEvent` @@ -115,33 +104,19 @@ type StreamEvent struct { } ``` -Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value — the ingest handler rewrites them before publishing, so a live frame and a later query can't disagree on the spelling of an instant. Two consequences worth knowing: a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open — a value the server can't parse, or one whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). - -:::note[`Events()` carries events only] -`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. -::: - -:::note[The channel buffers from stream construction] -Events buffer (up to 256) starting at `.Stream()`; events arriving before the first `Events()` call are not lost. -::: +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value, because the ingest handler rewrites them before publishing — a live frame and a later query can't disagree on the spelling of an instant. So a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open: a value the server can't parse, or whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). ### Transport Behavior -| Transport | Reconnect | Protocol | -| --------- | --------- | -------- | -| SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | - -Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). +SSE reconnects automatically with exponential backoff (capped at 30s) and gap-fill replay from the last event ID; HTTP/2 is recommended. Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). -Note that `/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`. A `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. - -Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). The TypeScript SDK streams over `fetch` and authenticates the same way, so this is shared behavior rather than a Go-only property — what Go avoids is the browser's per-domain connection ceiling, not a different auth mechanism. +`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). Delivery across a reconnect is **at-least-once**: the server replays from the last event ID *inclusively*, so the first frame after a gap-fill is usually one you already saw. Replay reaches back only as far as the server's `mq.gap_window_minutes` (15 minutes by default); a longer outage resumes live with a hole. ### Server-Side Policy Filtering -Before anything reaches the client, the server applies the caller's policy to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber — on live frames and on `Since` gap-fill replay alike. The claims are captured from the JWT at connect time. +Before anything reaches the client, the server applies the caller's policy — claims captured from the JWT at connect time — to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber, on live frames and `Since` gap-fill replay alike. Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control](/access-control#row-level-security). @@ -158,25 +133,25 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere. `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively, and `OpIn` accepts any Go slice type (e.g. `[]string`, `[]int`). #### How values are compared The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: -- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload reads `2026-06-21T04:00:00Z` while your filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a row that is chronologically equal. Both sides are parsed as instants instead. -- **Only unambiguous spellings count as instants.** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — guessing UTC would move the instant. Such a constant is not treated as a timestamp. -- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could otherwise admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload may read `2026-06-21T04:00:00Z` while your filter constant names the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a chronologically equal row. Both sides are parsed as instants instead. +- **Only unambiguous spellings count as instants:** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have, so it is not treated as a timestamp. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. - **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. - **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. :::caution[Integer precision above 2^53] -Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond `Number.MAX_SAFE_INTEGER` (2^53) has already lost exactness before any filter runs — the server compares such columns in their exact storage domain, so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond 2^53 has already lost exactness before any filter runs, and the server compares such columns in their exact storage domain — so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. ::: ## Live Queries -Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. +Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream, for seamless initial loads and updates. They exist only on `*QueryBuilder` (there is no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. ```go lq := wh.From("clicks"). @@ -198,7 +173,6 @@ lq := wh.From("clicks"). }, }, nil) -// Cleanup defer lq.Close() ``` @@ -218,7 +192,7 @@ type StreamSubscriber struct { ``` :::note[`Initial` is always untyped] -Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even if you'd use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. +Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even where you would use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. ::: ### How it works @@ -228,22 +202,16 @@ Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `Live 3. Deduplicates buffered events against the maximum `received_timestamp` in the backfill (not necessarily the last row). 4. Flushes remaining buffered events and switches to live mode. -This "stream-first" approach prevents event loss between fetch and stream start. +Subscribing first is what prevents event loss between fetch and stream start. :::caution[Dedup needs `received_timestamp` in the projection] -Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, causing events in the overlap window to be delivered twice (via `Initial` and `Next`). +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, so events in the overlap window are delivered twice — once via `Initial`, once via `Next`. ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive. Consequently, a live query filtering on `OpLike` may exclude rows in the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). - -`OpNotLike` is rejected by `/v1/query` with a `400`, causing `Initial` callbacks to fail. See [Queries](/sdk/go/queries#wherecolumn-op-value). +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback — see [Queries](/sdk/go/queries#wherecolumn-op-value). ::: ### `.Close()` -Shuts down the live query and its underlying stream. Safe to call more than once (idempotent via `sync.Once`). - -```go -lq.Close() -``` +`lq.Close()` shuts down the live query and its underlying stream. Idempotent (guarded by `sync.Once`). diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 652f9ba5..b6030214 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -8,7 +8,7 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — TypeScript client for WaveHouse. One runtime dependency: `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. :::tip[Writing Go instead?] -WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually, but API and lifecycle details differ — Go uses context-first calls and package-level generics, and streams must be closed explicitly. +WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). Both clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually; the [API shapes differ](/sdk/go#differences-from-the-typescript-sdk). ::: ## Installation @@ -559,16 +559,9 @@ The full error-code table lives in [Error Handling](/sdk/reference#error-handlin description="Error codes, AbortController, the full API tree, codegen, and E2E testing." href="/sdk/reference" /> - - -## Go SDK - -Prefer Go? The same server, the same wire format, an idiomatic Go client: - - From 0b8e450388b13f194ebe1cb3ce25c0fd1f9a7e01 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 10:15:39 -0400 Subject: [PATCH 46/59] test(sdk): consolidate Go SDK test setup and collapse sibling cases Shrinks the Go SDK test surface by ~295 lines with no loss of asserted behavior and no change in coverage (go-sdk suite stays at 84.6%). - Hoist one shared request recorder into http_test.go: recordedRequest + recordRequests/recordingCtx/recordingClient replace the per-file mutex-guarded handler captures in table_test.go, namespaces_test.go and query_builder_test.go, plus headerCaptureServer and captureQueryBody. Every capture now crosses the handler->test boundary on a buffered channel, matching the headerCaptureServer pattern. - Collapse sibling tests that differed only by inputs into table-driven tests: doRequest body/auth shapes, doRequest retry policy, NewClient options, parseErrorResponse retryability, TableRef.Insert wire formats, every namespace's method/path, the QueryBuilder request body, the pagination cursor, and the live-query backfill dedup. - Pin the exact StructuredQuery body per builder chain instead of probing one field at a time, and assert the ?table= parameter on every case. - Fold duplicate assertions into their surviving twin: StaticToken, Client.From, compareOrdered, equalValues' nil cases, handleSSEData's malformed frame, the 403 terminal-connect case, and codegen's json.Number import check. --- clients/go/client_test.go | 107 ++-- clients/go/cmd/wavehouse-codegen/main_test.go | 21 +- clients/go/e2e_test.go | 131 ++-- clients/go/errors_test.go | 48 +- clients/go/http_test.go | 383 ++++++----- clients/go/live_query_test.go | 155 ++--- clients/go/namespaces_test.go | 364 ++++++----- clients/go/query_builder_test.go | 605 ++++++++---------- clients/go/stream_test.go | 489 ++++++-------- clients/go/table_test.go | 336 ++++------ 10 files changed, 1172 insertions(+), 1467 deletions(-) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 2ba55833..2440f11b 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -3,35 +3,45 @@ package wavehouse import ( "context" "encoding/json" - "net/http" - "net/http/httptest" "testing" ) -func TestNewClient_Defaults(t *testing.T) { - c := NewClient(Config{BaseURL: "http://localhost:8080"}) - if c.ctx.maxRetries != 2 { - t.Fatalf("want default maxRetries=2, got %d", c.ctx.maxRetries) - } - if c.ctx.baseURL != "http://localhost:8080" { - t.Fatalf("want baseURL, got %s", c.ctx.baseURL) - } -} - -func TestNewClient_StripsTrailingSlashes(t *testing.T) { - c := NewClient(Config{BaseURL: "http://localhost:8080///"}) - if c.ctx.baseURL != "http://localhost:8080" { - t.Fatalf("want stripped URL, got %s", c.ctx.baseURL) - } -} - -func TestNewClient_CustomMaxRetries(t *testing.T) { - c := NewClient(Config{ - BaseURL: "http://localhost:8080", - Options: &ClientOptions{MaxRetries: 5}, - }) - if c.ctx.maxRetries != 5 { - t.Fatalf("want 5, got %d", c.ctx.maxRetries) +func TestNewClient(t *testing.T) { + tests := []struct { + name string + cfg Config + wantBaseURL string + wantMaxRetries int + }{ + { + name: "defaults", + cfg: Config{BaseURL: "http://localhost:8080"}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 2, + }, + { + name: "trailing slashes are stripped", + cfg: Config{BaseURL: "http://localhost:8080///"}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 2, + }, + { + name: "MaxRetries is honored", + cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{MaxRetries: 5}}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 5, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c := NewClient(tt.cfg) + if c.ctx.baseURL != tt.wantBaseURL { + t.Errorf("want baseURL %q, got %q", tt.wantBaseURL, c.ctx.baseURL) + } + if c.ctx.maxRetries != tt.wantMaxRetries { + t.Errorf("want maxRetries %d, got %d", tt.wantMaxRetries, c.ctx.maxRetries) + } + }) } } @@ -57,37 +67,8 @@ func TestNewClient_HasNamespaces(t *testing.T) { } } -func TestClient_From(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - // Verify the table name appears in the URL. - if r.URL.Query().Get("table") != "events" { - t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) - } - _ = json.NewEncoder(w).Encode([]map[string]any{}) - })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) - // Checked, not discarded: if Fetch returns before issuing the request, the - // handler never runs and the table= assertion above proves nothing. - if _, err := c.From("events").Fetch(context.Background()); err != nil { - t.Fatal(err) - } -} - func TestClient_SQL(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/ops/query" { - t.Errorf("want /v1/ops/query, got %s", r.URL.Path) - } - var body map[string]string - _ = json.NewDecoder(r.Body).Decode(&body) - if body["sql"] != "SELECT 1" { - t.Errorf("want sql=SELECT 1, got %s", body["sql"]) - } - _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) - })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + c, reqs := recordingClient(t, jsonResponse([]map[string]any{{"x": 1}})) rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") if err != nil { t.Fatal(err) @@ -95,16 +76,12 @@ func TestClient_SQL(t *testing.T) { if len(rows) != 1 { t.Fatalf("want 1 row, got %d", len(rows)) } -} - -func TestStaticToken(t *testing.T) { - fn := StaticToken("abc") - token, err := fn(context.Background()) - if err != nil { - t.Fatal(err) + got := <-reqs + if got.method != "POST" || got.path != "/v1/ops/query" { + t.Fatalf("want POST /v1/ops/query, got %s %s", got.method, got.path) } - if token != "abc" { - t.Fatalf("want abc, got %s", token) + if body := string(got.body); body != `{"sql":"SELECT 1"}` { + t.Fatalf(`want {"sql":"SELECT 1"}, got %s`, body) } } diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index f6207ddc..c5a04432 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -99,6 +99,7 @@ func TestGenerate_Basic(t *testing.T) { "clicks": {Name: "clicks", Columns: []column{ {Name: "page", Type: "String"}, {Name: "score", Type: "Float64"}, + {Name: "big", Type: "UInt128"}, {Name: "received_timestamp", Type: "DateTime64(3, 'UTC')", HasDefault: true}, }}, }, "myapp") @@ -107,9 +108,11 @@ func TestGenerate_Basic(t *testing.T) { } for _, want := range []string{ "package myapp", + `import "encoding/json"`, // dragged in by the json.Number field below "type ClicksRow struct {", "Page string `json:\"page\"`", "Score float64 `json:\"score\"`", + "Big json.Number `json:\"big\"`", // Defaulted column: pointer + omitempty so an explicit zero still sends. "ReceivedTimestamp *string `json:\"received_timestamp,omitempty\"`", } { @@ -117,6 +120,9 @@ func TestGenerate_Basic(t *testing.T) { t.Errorf("generated output missing %q:\n%s", want, out) } } + if _, err := format.Source([]byte(out)); err != nil { + t.Fatalf("generated output is not valid Go: %v", err) + } } func TestGenerate_FieldCollisionFails(t *testing.T) { @@ -165,18 +171,3 @@ func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { t.Fatalf("128-bit value corrupted: %s", rows[0].Big) } } - -func TestGenerate_JSONNumberImport(t *testing.T) { - out, err := generate(map[string]tableSchema{ - "t": {Name: "t", Columns: []column{{Name: "big", Type: "UInt128"}}}, - }, "main") - if err != nil { - t.Fatal(err) - } - if !strings.Contains(out, `import "encoding/json"`) { - t.Fatalf("json.Number field without encoding/json import:\n%s", out) - } - if _, err := format.Source([]byte(out)); err != nil { - t.Fatalf("generated output is not valid Go: %v", err) - } -} diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 3bce0341..0e5e2475 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -52,6 +52,10 @@ func e2eClient(t *testing.T) *Client { return NewClient(cfg) } +// e2eCtx is the context every e2e call uses. The deadlines that matter are +// scoped where they belong: the reachability probe above and waitForRows below. +var e2eCtx = context.Background() + // marker returns a unique string for the running test, useful for // inserting distinguishable rows that won't collide across parallel runs. func marker(t *testing.T) string { @@ -110,18 +114,16 @@ func waitForRows(t *testing.T, c *Client, table, markerCol, mk string, want int) func TestE2E_HealthCheck(t *testing.T) { c := e2eClient(t) - ctx := context.Background() - if err := c.Sys.Health(ctx); err != nil { + if err := c.Sys.Health(e2eCtx); err != nil { t.Fatalf("Health check failed: %v", err) } } func TestE2E_SchemaList(t *testing.T) { c := e2eClient(t) - ctx := context.Background() - schemas, err := c.Schema.List(ctx) + schemas, err := c.Schema.List(e2eCtx) if err != nil { t.Fatalf("Schema.List failed: %v", err) } @@ -136,63 +138,54 @@ func TestE2E_SchemaList(t *testing.T) { } } -func TestE2E_InsertAndQuery(t *testing.T) { - c := e2eClient(t) - ctx := context.Background() - table, ts := firstTable(t, c) - mk := marker(t) - - row, markerCol := buildMarkerRow(t, ts, mk) - - res, err := c.From(table).Insert(ctx, row) - if err != nil { - t.Fatalf("Insert into %s failed: %v", table, err) - } - if !res.OK { - t.Fatalf("Insert into %s: OK=false", table) - } - - rows := waitForRows(t, c, table, markerCol, mk, 1) - if len(rows) == 0 { - t.Fatal("Query returned zero rows — expected the inserted marker row") - } - got, _ := rows[0][markerCol].(string) - if got != mk { - t.Errorf("marker mismatch: want %q, got %q", mk, got) - } -} - -func TestE2E_BatchInsert(t *testing.T) { - c := e2eClient(t) - ctx := context.Background() - table, ts := firstTable(t, c) - - mk := marker(t) - - // Build 3 rows, each with the same marker so we can count them. - rows := make([]map[string]any, 3) - markerCol := "" - for i := range rows { - rows[i], markerCol = buildMarkerRow(t, ts, mk) - } - - res, err := c.From(table).Insert(ctx, rows) - if err != nil { - t.Fatalf("Batch insert failed: %v", err) - } - if !res.OK { - t.Fatalf("Batch insert: OK=false") - } - - got := waitForRows(t, c, table, markerCol, mk, 3) - if len(got) < 3 { - t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(got)) +// TestE2E_Insert covers both ingest paths against the live server: a bare map +// goes as one JSON body, a slice as an NDJSON batch. +func TestE2E_Insert(t *testing.T) { + tests := []struct { + name string + count int + }{ + {"single row", 1}, + {"batch of three", 3}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c := e2eClient(t) + table, ts := firstTable(t, c) + mk := marker(t) + + // Every row carries the same marker, so one query counts them all. + rows := make([]map[string]any, tt.count) + markerCol := "" + for i := range rows { + rows[i], markerCol = buildMarkerRow(t, ts, mk) + } + var payload any = rows + if tt.count == 1 { + payload = rows[0] // a bare map takes the single-insert path + } + + res, err := c.From(table).Insert(e2eCtx, payload) + if err != nil { + t.Fatalf("Insert into %s failed: %v", table, err) + } + if !res.OK { + t.Fatalf("Insert into %s: OK=false", table) + } + + got := waitForRows(t, c, table, markerCol, mk, tt.count) + if len(got) < tt.count { + t.Fatalf("expected >= %d rows for marker %q, got %d", tt.count, mk, len(got)) + } + if v, _ := got[0][markerCol].(string); v != mk { + t.Errorf("marker mismatch: want %q, got %q", mk, v) + } + }) } } func TestE2E_QueryBuilder(t *testing.T) { c := e2eClient(t) - ctx := context.Background() table, ts := firstTable(t, c) // Pick two columns for a minimal projection. @@ -211,7 +204,7 @@ func TestE2E_QueryBuilder(t *testing.T) { Select(cols...). OrderBy(cols[0], "asc"). Limit(5). - FetchUntyped(ctx) + FetchUntyped(e2eCtx) if err != nil { t.Fatalf("QueryBuilder chain failed: %v", err) } @@ -224,11 +217,10 @@ func TestE2E_QueryBuilder(t *testing.T) { func TestE2E_TypedFetch(t *testing.T) { c := e2eClient(t) - ctx := context.Background() table, _ := firstTable(t, c) q := c.From(table).SelectAll().Limit(3) - page, err := FetchTyped[map[string]any](ctx, q) + page, err := FetchTyped[map[string]any](e2eCtx, q) if err != nil { t.Fatalf("FetchTyped failed: %v", err) } @@ -243,9 +235,8 @@ func TestE2E_TypedFetch(t *testing.T) { func TestE2E_SQLQuery(t *testing.T) { c := e2eClient(t) - ctx := context.Background() - rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") + rows, err := SQL[map[string]any](e2eCtx, c, "SELECT 1 AS n") if err != nil { skipIfUnauthorized(t, err, "SQL query") t.Fatalf("SQL query failed: %v", err) @@ -272,21 +263,20 @@ func TestE2E_SQLQuery(t *testing.T) { func TestE2E_PolicyGetSet(t *testing.T) { c := e2eClient(t) - ctx := context.Background() - pol, err := c.Policy.Get(ctx) + pol, err := c.Policy.Get(e2eCtx) if err != nil { skipIfUnauthorized(t, err, "Policy.Get") t.Fatalf("Policy.Get failed: %v", err) } // Round-trip: set the same policy back. - if err := c.Policy.Set(ctx, pol); err != nil { + if err := c.Policy.Set(e2eCtx, pol); err != nil { t.Fatalf("Policy.Set (round-trip) failed: %v", err) } // Read again and verify tables still match. - pol2, err := c.Policy.Get(ctx) + pol2, err := c.Policy.Get(e2eCtx) if err != nil { t.Fatalf("Policy.Get (after set) failed: %v", err) } @@ -297,7 +287,6 @@ func TestE2E_PolicyGetSet(t *testing.T) { func TestE2E_PipesCRUD(t *testing.T) { c := e2eClient(t) - ctx := context.Background() pipeName := fmt.Sprintf("e2e_test_%d", time.Now().UnixNano()) @@ -306,7 +295,7 @@ func TestE2E_PipesCRUD(t *testing.T) { SQL: "SELECT 1 AS ok", Description: "E2E test pipe — safe to delete", } - if err := c.Pipes.Set(ctx, pipeName, def); err != nil { + if err := c.Pipes.Set(e2eCtx, pipeName, def); err != nil { skipIfUnauthorized(t, err, "Pipes.Set") t.Fatalf("Pipes.Set (create) failed: %v", err) } @@ -317,7 +306,7 @@ func TestE2E_PipesCRUD(t *testing.T) { }) // Get - pipe, err := c.Pipes.Get(ctx, pipeName) + pipe, err := c.Pipes.Get(e2eCtx, pipeName) if err != nil { t.Fatalf("Pipes.Get failed: %v", err) } @@ -326,7 +315,7 @@ func TestE2E_PipesCRUD(t *testing.T) { } // List — verify it appears - pipes, err := c.Pipes.List(ctx) + pipes, err := c.Pipes.List(e2eCtx) if err != nil { t.Fatalf("Pipes.List failed: %v", err) } @@ -342,12 +331,12 @@ func TestE2E_PipesCRUD(t *testing.T) { } // Delete - if err := c.Pipes.Delete(ctx, pipeName); err != nil { + if err := c.Pipes.Delete(e2eCtx, pipeName); err != nil { t.Fatalf("Pipes.Delete failed: %v", err) } // Verify gone — Get should fail. - _, err = c.Pipes.Get(ctx, pipeName) + _, err = c.Pipes.Get(e2eCtx, pipeName) if err == nil { t.Error("Pipes.Get after delete: expected error, got nil") } diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go index 160cb5c4..459cb610 100644 --- a/clients/go/errors_test.go +++ b/clients/go/errors_test.go @@ -46,6 +46,10 @@ func TestParseErrorResponse(t *testing.T) { wantRetry: true, nilDetails: true, }, + // Retryability is decided by status class alone — 429 and 5xx only. + {name: "ForbiddenNotRetryable", status: 403, body: `{"error":"nope"}`, wantMsg: "nope"}, + {name: "TooManyRequestsRetryable", status: 429, body: `{"error":"slow down"}`, wantMsg: "slow down", wantRetry: true}, + {name: "ServiceUnavailableRetryable", status: 503, body: `{"error":"down"}`, wantMsg: "down", wantRetry: true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -74,33 +78,6 @@ func TestParseErrorResponse(t *testing.T) { } } -func TestParseErrorResponse_5xxRetryable(t *testing.T) { - tests := []struct { - name string - status int - retryable bool - }{ - {"BadRequest", 400, false}, - {"Forbidden", 403, false}, - {"TooManyRequests", 429, true}, - {"InternalServerError", 500, true}, - {"ServiceUnavailable", 503, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - res := &http.Response{ - StatusCode: tt.status, - Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Retryable != tt.retryable { - t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) - } - }) - } -} - func TestNetworkError(t *testing.T) { e := networkError(errors.New("connection refused")) if e.Code != "NETWORK_ERROR" { @@ -118,16 +95,13 @@ func TestNetworkError(t *testing.T) { } func TestError_ErrorMethod(t *testing.T) { - e := &Error{Status: 404, Code: "HTTP_404", Message: "not found"} - got := e.Error() - if !strings.Contains(got, "HTTP_404") || !strings.Contains(got, "not found") { - t.Fatalf("unexpected Error() output: %s", got) - } - - e2 := &Error{Status: 0, Code: "NETWORK_ERROR", Message: "timeout"} - got2 := e2.Error() - if !strings.Contains(got2, "NETWORK_ERROR") { - t.Fatalf("unexpected Error() output: %s", got2) + for _, e := range []*Error{ + {Status: 404, Code: "HTTP_404", Message: "not found"}, + {Status: 0, Code: "NETWORK_ERROR", Message: "timeout"}, + } { + if got := e.Error(); !strings.Contains(got, e.Code) || !strings.Contains(got, e.Message) { + t.Errorf("Error() = %q, want it to name both %q and %q", got, e.Code, e.Message) + } } } diff --git a/clients/go/http_test.go b/clients/go/http_test.go index c8b0d4c5..be7b04c3 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -1,12 +1,14 @@ package wavehouse import ( + "bytes" "context" "encoding/json" "errors" "io" "net/http" "net/http/httptest" + "net/url" "sync/atomic" "testing" "time" @@ -21,6 +23,9 @@ func errIs(err error, code string) bool { return false } +// testCtx and queryTestCtx are the two entry points every test in this package +// uses to reach a throwaway server: the bare transport context, and a full +// Client wired to the same server. func testCtx(t *testing.T, handler http.Handler) httpContext { t.Helper() srv := httptest.NewServer(handler) @@ -32,133 +37,100 @@ func testCtx(t *testing.T, handler http.Handler) httpContext { } } -func TestDoRequest_SuccessfulGET(t *testing.T) { - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) - })) - - var result map[string]string - err := doRequest(context.Background(), hctx, requestOptions{ - method: "GET", - path: "/health", - }, &result) - if err != nil { - t.Fatal(err) - } - if result["status"] != "ok" { - t.Fatalf("want ok, got %v", result) - } +func queryTestCtx(t *testing.T, handler http.Handler) *Client { + t.Helper() + srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) + return NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) } -func TestDoRequest_POSTWithBody(t *testing.T) { - var gotBody map[string]string - var gotCT string - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotCT = r.Header.Get("Content-Type") - _ = json.NewDecoder(r.Body).Decode(&gotBody) - w.WriteHeader(200) - })) +// recordedRequest is one request as the test server saw it — everything a test +// might assert on, copied on the server goroutine. +type recordedRequest struct { + method string + path string + query url.Values + header http.Header + body []byte +} - err := doRequest(context.Background(), hctx, requestOptions{ - method: "POST", - path: "/v1/ingest", - body: map[string]string{"page": "/home"}, - }, nil) - if err != nil { - t.Fatal(err) - } - if gotCT != "application/json" { - t.Fatalf("want application/json, got %s", gotCT) - } - if gotBody["page"] != "/home" { - t.Fatalf("want /home, got %v", gotBody) +// jsonBody decodes the recorded body as a JSON object. UseNumber keeps int64 +// cursor values exact on the assertion side too. +func (r recordedRequest) jsonBody(t *testing.T) map[string]any { + t.Helper() + dec := json.NewDecoder(bytes.NewReader(r.body)) + dec.UseNumber() + var m map[string]any + if err := dec.Decode(&m); err != nil { + t.Fatalf("decode request body %q: %v", r.body, err) } + return m } -func TestDoRequest_RawBody(t *testing.T) { - var gotBody string - var gotCT string - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotCT = r.Header.Get("Content-Type") - raw, _ := io.ReadAll(r.Body) - gotBody = string(raw) - _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) - })) - - err := doRequest(context.Background(), hctx, requestOptions{ - method: "POST", - path: "/v1/ingest", - rawBody: `{"page":"/a"}`, - contentType: "application/x-ndjson", - }, nil) - if err != nil { - t.Fatal(err) - } - if gotCT != "application/x-ndjson" { - t.Fatalf("want ndjson content type, got %s", gotCT) - } - if gotBody != `{"page":"/a"}` { - t.Fatalf("want raw body, got %s", gotBody) - } +// recordRequests wraps handler so every request it serves lands on the +// returned buffered channel — a channel, not a shared variable, so -race sees +// the edge between the server goroutine and the test. handler still reads an +// intact Body. +func recordRequests(t *testing.T, handler http.Handler) (http.Handler, <-chan recordedRequest) { + t.Helper() + seen := make(chan recordedRequest, 16) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("read request body: %v", err) + } + seen <- recordedRequest{ + method: r.Method, + path: r.URL.Path, + query: r.URL.Query(), + header: r.Header.Clone(), + body: raw, + } + r.Body = io.NopCloser(bytes.NewReader(raw)) + handler.ServeHTTP(w, r) + }), seen } -func TestDoRequest_AuthInjection(t *testing.T) { - var gotAuth string - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotAuth = r.Header.Get("Authorization") - w.WriteHeader(200) - })) - hctx.auth = StaticToken("my-token") +func recordingCtx(t *testing.T, handler http.Handler) (httpContext, <-chan recordedRequest) { + t.Helper() + h, seen := recordRequests(t, handler) + return testCtx(t, h), seen +} - err := doRequest(context.Background(), hctx, requestOptions{ - method: "GET", - path: "/v1/ops/schema", - }, nil) - if err != nil { - t.Fatal(err) - } - if gotAuth != "Bearer my-token" { - t.Fatalf("want 'Bearer my-token', got %s", gotAuth) - } +func recordingClient(t *testing.T, handler http.Handler) (*Client, <-chan recordedRequest) { + t.Helper() + h, seen := recordRequests(t, handler) + return queryTestCtx(t, h), seen } -func TestDoRequest_4xxNotRetried(t *testing.T) { - var count atomic.Int32 - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - count.Add(1) +// ok200 answers with a bare 200; jsonArray with an empty JSON array — the +// minimal valid reply for a list endpoint. +var ( + ok200 = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + }) + jsonArray = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - w.WriteHeader(404) - _ = json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) - })) - hctx.maxRetries = 2 - - err := doRequest(context.Background(), hctx, requestOptions{ - method: "GET", - path: "/v1/ops/schema", - }, nil) + _, _ = io.WriteString(w, `[]`) + }) +) - if !errIs(err, "HTTP_404") { - t.Fatalf("want HTTP_404 error, got %v", err) - } - if count.Load() != 1 { - t.Fatalf("4xx should not retry, got %d attempts", count.Load()) +// jsonResponse answers any request with body encoded as JSON. +func jsonResponse(body any) http.HandlerFunc { + return func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(body) } } -func TestDoRequest_5xxRetried(t *testing.T) { - var count atomic.Int32 +func TestDoRequest_SuccessfulGET(t *testing.T) { hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - n := count.Add(1) - if n < 3 { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(500) - _ = json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) - return - } - _ = json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) })) - hctx.maxRetries = 2 var result map[string]string err := doRequest(context.Background(), hctx, requestOptions{ @@ -166,38 +138,141 @@ func TestDoRequest_5xxRetried(t *testing.T) { path: "/health", }, &result) if err != nil { - t.Fatalf("want success after retries, got %v", err) + t.Fatal(err) } - if count.Load() != 3 { - t.Fatalf("want 3 attempts, got %d", count.Load()) + if result["status"] != "ok" { + t.Fatalf("want ok, got %v", result) + } +} + +// TestDoRequest_RequestShape: what the transport puts on the wire for each +// body/auth combination. +func TestDoRequest_RequestShape(t *testing.T) { + tests := []struct { + name string + auth func(context.Context) (string, error) + opts requestOptions + wantCT string + wantBody string + wantAuth string + }{ + { + name: "a struct body is marshaled as JSON", + opts: requestOptions{method: "POST", path: "/v1/ingest", body: map[string]string{"page": "/home"}}, + wantCT: "application/json", + wantBody: `{"page":"/home"}`, + }, + { + name: "a raw body keeps the caller's content type", + opts: requestOptions{method: "POST", path: "/v1/ingest", rawBody: `{"page":"/a"}`, contentType: "application/x-ndjson"}, + wantCT: "application/x-ndjson", + wantBody: `{"page":"/a"}`, + }, + { + name: "the auth provider's token becomes a Bearer header", + auth: StaticToken("my-token"), + opts: requestOptions{method: "GET", path: "/v1/ops/schema"}, + wantCT: "application/json", + wantAuth: "Bearer my-token", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + hctx, reqs := recordingCtx(t, ok200) + hctx.auth = tt.auth + + if err := doRequest(context.Background(), hctx, tt.opts, nil); err != nil { + t.Fatal(err) + } + got := <-reqs + if got.method != tt.opts.method || got.path != tt.opts.path { + t.Fatalf("want %s %s, got %s %s", tt.opts.method, tt.opts.path, got.method, got.path) + } + if ct := got.header.Get("Content-Type"); ct != tt.wantCT { + t.Fatalf("want Content-Type %q, got %q", tt.wantCT, ct) + } + if string(got.body) != tt.wantBody { + t.Fatalf("want body %q, got %q", tt.wantBody, got.body) + } + if auth := got.header.Get("Authorization"); auth != tt.wantAuth { + t.Fatalf("want Authorization %q, got %q", tt.wantAuth, auth) + } + }) + } +} + +// TestDoRequest_RetryPolicy: which statuses are retried, how many attempts +// they take, and whether Retry-After is honored. +func TestDoRequest_RetryPolicy(t *testing.T) { + tests := []struct { + name string + status int + retryAfter string + maxRetries int + wantAttempts int32 + wantErrCode string + wantMinElapsed time.Duration + }{ + {name: "4xx is not retried", status: 404, maxRetries: 2, wantAttempts: 1, wantErrCode: "HTTP_404"}, + {name: "5xx retries until it succeeds", status: 500, maxRetries: 2, wantAttempts: 3}, + { + name: "429 waits out Retry-After", status: 429, retryAfter: "1", + maxRetries: 1, wantAttempts: 2, wantMinElapsed: 900 * time.Millisecond, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var calls atomic.Int32 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) >= tt.wantAttempts && tt.wantErrCode == "" { + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + return + } + w.Header().Set("Content-Type", "application/json") + if tt.retryAfter != "" { + w.Header().Set("Retry-After", tt.retryAfter) + } + w.WriteHeader(tt.status) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "boom"}) + })) + hctx.maxRetries = tt.maxRetries + + start := time.Now() + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{method: "GET", path: "/health"}, &result) + if tt.wantErrCode == "" && err != nil { + t.Fatalf("want success after retries, got %v", err) + } + if tt.wantErrCode != "" && !errIs(err, tt.wantErrCode) { + t.Fatalf("want %s error, got %v", tt.wantErrCode, err) + } + if got := calls.Load(); got != tt.wantAttempts { + t.Fatalf("want %d attempts, got %d", tt.wantAttempts, got) + } + if elapsed := time.Since(start); elapsed < tt.wantMinElapsed { + t.Fatalf("Retry-After not honored — retried after only %v", elapsed) + } + }) } } func TestDoRequest_AbortedOnCancel(t *testing.T) { - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { time.Sleep(5 * time.Second) })) ctx, cancel := context.WithCancel(context.Background()) cancel() // cancel immediately - err := doRequest(ctx, hctx, requestOptions{ - method: "GET", - path: "/health", - }, nil) - + err := doRequest(ctx, hctx, requestOptions{method: "GET", path: "/health"}, nil) if !errIs(err, "ABORTED") { t.Fatalf("want ABORTED, got %v", err) } } func TestDoRequest_EmptyResponse(t *testing.T) { - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(200) - })) - var result map[string]string - err := doRequest(context.Background(), hctx, requestOptions{ + err := doRequest(context.Background(), testCtx(t, ok200), requestOptions{ method: "POST", path: "/v1/ops/schema/refresh", }, &result) @@ -283,32 +358,6 @@ func TestRetryAfterDelay(t *testing.T) { } } -func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { - var calls atomic.Int64 - hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - if calls.Add(1) == 1 { - w.Header().Set("Retry-After", "1") - w.WriteHeader(http.StatusTooManyRequests) - return - } - _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) - })) - hctx.maxRetries = 1 - - start := time.Now() - var result map[string]string - err := doRequest(context.Background(), hctx, requestOptions{method: "GET", path: "/x"}, &result) - if err != nil { - t.Fatalf("want success after 429 retry, got %v", err) - } - if got := calls.Load(); got != 2 { - t.Fatalf("want 2 attempts, got %d", got) - } - if elapsed := time.Since(start); elapsed < 900*time.Millisecond { - t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) - } -} - // A BaseURL carrying a path prefix must survive on both transports — the bug // #428 fixed in the TS client, which Go avoids by concatenating rather than // resolving. Guards against a future switch to url.JoinPath/ResolveReference. @@ -336,19 +385,14 @@ func TestBaseURLPathPrefixIsPreserved(t *testing.T) { } } -// headerCaptureServer answers any request with `[]` and hands that request's -// headers back over the channel — a channel, not a shared variable, so -race -// sees the edge between the server goroutine and the test. -func headerCaptureServer(t *testing.T) (*httptest.Server, <-chan http.Header) { +// headerCaptureClient is a Client with configured headers and auth pointed at +// a server that records what it received. +func headerCaptureClient(t *testing.T, opts *ClientOptions, auth func(context.Context) (string, error)) (*Client, <-chan recordedRequest) { t.Helper() - captured := make(chan http.Header, 1) - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - captured <- r.Header.Clone() - w.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(w, `[]`) - })) + h, reqs := recordRequests(t, jsonArray) + srv := httptest.NewServer(h) t.Cleanup(srv.Close) - return srv, captured + return NewClient(Config{BaseURL: srv.URL, Auth: auth, HTTPClient: srv.Client(), Options: opts}), reqs } // TestConfiguredHeadersOnRESTRequests: ClientOptions.Headers apply to every @@ -391,17 +435,11 @@ func TestConfiguredHeadersOnRESTRequests(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - srv, headers := headerCaptureServer(t) - client := NewClient(Config{ - BaseURL: srv.URL, - Auth: tc.auth, - HTTPClient: srv.Client(), - Options: &ClientOptions{Headers: tc.configured}, - }) + client, reqs := headerCaptureClient(t, &ClientOptions{Headers: tc.configured}, tc.auth) if _, err := client.Schema.List(context.Background()); err != nil { t.Fatalf("schema list: %v", err) } - got := <-headers + got := (<-reqs).header if v := got.Values(tc.header); len(v) != 1 { t.Fatalf("want exactly one %s header, got %v", tc.header, v) } @@ -415,20 +453,15 @@ func TestConfiguredHeadersOnRESTRequests(t *testing.T) { // TestConfiguredHeadersAreCopied: mutating the caller's map after NewClient // must not change what later requests send. func TestConfiguredHeadersAreCopied(t *testing.T) { - srv, captured := headerCaptureServer(t) headers := map[string]string{"X-Tenant-Id": "acme"} - client := NewClient(Config{ - BaseURL: srv.URL, - HTTPClient: srv.Client(), - Options: &ClientOptions{Headers: headers}, - }) + client, reqs := headerCaptureClient(t, &ClientOptions{Headers: headers}, nil) headers["X-Tenant-Id"] = "attacker" delete(headers, "X-Tenant-Id") if _, err := client.Schema.List(context.Background()); err != nil { t.Fatalf("schema list: %v", err) } - if v := (<-captured).Get("X-Tenant-Id"); v != "acme" { + if v := (<-reqs).header.Get("X-Tenant-Id"); v != "acme" { t.Fatalf("want the value captured at construction, got %q", v) } } diff --git a/clients/go/live_query_test.go b/clients/go/live_query_test.go index f703a68e..c171b198 100644 --- a/clients/go/live_query_test.go +++ b/clients/go/live_query_test.go @@ -34,94 +34,79 @@ func awaitInitial(t *testing.T, ch <-chan []map[string]any) []map[string]any { } } -func TestLiveQuery_InitialThenLiveWithDedup(t *testing.T) { - sc := bareStream() - fetched := []map[string]any{ - {"page": "/a", "received_timestamp": "2026-01-01T00:00:05Z"}, - // Descending order: the max timestamp is NOT the last row. - {"page": "/b", "received_timestamp": "2026-01-01T00:00:03Z"}, - } - gate := make(chan struct{}) - initialCh := make(chan []map[string]any, 1) - nextCh := make(chan StreamEvent, 8) - - lq := newLiveQuery(sc, - func(context.Context) ([]map[string]any, error) { - <-gate - return fetched, nil +// TestLiveQuery_BackfillThenLive: events arriving while the backfill is in +// flight are buffered, flushed once Initial has fired, and deduplicated +// against the *maximum* backfilled timestamp — not the last row, which a +// descending sort makes the oldest. +func TestLiveQuery_BackfillThenLive(t *testing.T) { + tests := []struct { + name string + backfill []string // received_timestamp per backfilled row, in server order + emit []string // events emitted while the backfill is gated + wantLive string // the only emitted event that survives dedup + }{ + { + name: "the dedup bound is the max backfill timestamp, not the last row", + backfill: []string{"2026-01-01T00:00:05Z", "2026-01-01T00:00:03Z"}, // descending + emit: []string{"2026-01-01T00:00:04Z", "2026-01-01T00:00:06Z"}, + wantLive: "2026-01-01T00:00:06Z", }, - &StreamSubscriber{ - Initial: func(rows []map[string]any, err error) { - if err != nil { - t.Errorf("Initial err: %v", err) - } - initialCh <- rows - }, - Next: func(e StreamEvent) { nextCh <- e }, - }) - defer lq.Close() - - // Buffered while the backfill is in flight; deduped against the *max* - // backfilled timestamp (5Z despite descending order) on flush. - sc.emitEvent(liveEvent("2026-01-01T00:00:04Z")) // ≤ max backfill → skipped - sc.emitEvent(liveEvent("2026-01-01T00:00:06Z")) // newer → delivered - close(gate) - - rows := awaitInitial(t, initialCh) - if len(rows) != 2 { - t.Fatalf("want 2 backfill rows, got %d", len(rows)) - } - - select { - case e := <-nextCh: - if e.Timestamp != "2026-01-01T00:00:06Z" { - t.Fatalf("want the newer event only, got %s", e.Timestamp) - } - case <-time.After(5 * time.Second): - t.Fatal("live event never delivered") - } - select { - case e := <-nextCh: - t.Fatalf("stale event delivered despite dedup: %s", e.Timestamp) - case <-time.After(100 * time.Millisecond): - } -} - -func TestLiveQuery_BuffersDuringBackfill(t *testing.T) { - sc := bareStream() - gate := make(chan struct{}) - initialCh := make(chan []map[string]any, 1) - nextCh := make(chan StreamEvent, 8) - - lq := newLiveQuery(sc, - func(context.Context) ([]map[string]any, error) { - <-gate - return []map[string]any{{"received_timestamp": "2026-01-01T00:00:01Z"}}, nil + { + name: "an event older than the backfill is dropped on flush", + backfill: []string{"2026-01-01T00:00:01Z"}, + emit: []string{"2026-01-01T00:00:02Z", "2026-01-01T00:00:00Z"}, + wantLive: "2026-01-01T00:00:02Z", }, - &StreamSubscriber{ - Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, - Next: func(e StreamEvent) { nextCh <- e }, - }) - defer lq.Close() - - // Events arriving mid-backfill are buffered, then flushed post-Initial. - sc.emitEvent(liveEvent("2026-01-01T00:00:02Z")) - sc.emitEvent(liveEvent("2026-01-01T00:00:00Z")) // older than backfill → dropped in flush - close(gate) - - awaitInitial(t, initialCh) - select { - case e := <-nextCh: - if e.Timestamp != "2026-01-01T00:00:02Z" { - t.Fatalf("want buffered 02Z event, got %s", e.Timestamp) - } - case <-time.After(5 * time.Second): - t.Fatal("buffered event never flushed") } - select { - case e := <-nextCh: - t.Fatalf("pre-backfill event should have been deduped: %s", e.Timestamp) - case <-time.After(100 * time.Millisecond): + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + rows := make([]map[string]any, len(tt.backfill)) + for i, ts := range tt.backfill { + rows[i] = map[string]any{"page": "/a", "received_timestamp": ts} + } + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + sc := bareStream() + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return rows, nil + }, + &StreamSubscriber{ + Initial: func(got []map[string]any, err error) { + if err != nil { + t.Errorf("Initial err: %v", err) + } + initialCh <- got + }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + for _, ts := range tt.emit { + sc.emitEvent(liveEvent(ts)) + } + close(gate) + + if got := awaitInitial(t, initialCh); len(got) != len(rows) { + t.Fatalf("want %d backfill rows, got %d", len(rows), len(got)) + } + select { + case e := <-nextCh: + if e.Timestamp != tt.wantLive { + t.Fatalf("want %s delivered, got %s", tt.wantLive, e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("buffered live event never flushed") + } + select { + case e := <-nextCh: + t.Fatalf("stale event delivered despite dedup: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } + }) } } diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 27453e6b..12364138 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -2,200 +2,194 @@ package wavehouse import ( "context" - "encoding/json" - "net/http" - "sync" "testing" ) -func TestSysNamespace_Health(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/health" { - t.Errorf("want /v1/health, got %s", r.URL.Path) - } - w.WriteHeader(200) - })) - err := c.Sys.Health(context.Background()) - if err != nil { - t.Fatal(err) - } -} - -func TestSchemaNamespace_List(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/ops/schema" { - t.Errorf("want /v1/ops/schema, got %s", r.URL.Path) - } - _ = json.NewEncoder(w).Encode([]TableSchema{ - {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, - }) - })) - schemas, err := c.Schema.List(context.Background()) - if err != nil { - t.Fatal(err) - } - if _, ok := schemas["clicks"]; !ok { - t.Fatal("want clicks in schemas") - } -} - -func TestSchemaNamespace_Refresh(t *testing.T) { - var mu sync.Mutex - var gotMethod string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - gotMethod = r.Method - mu.Unlock() - w.WriteHeader(200) - })) - err := c.Schema.Refresh(context.Background()) - if err != nil { - t.Fatal(err) - } - mu.Lock() - defer mu.Unlock() - if gotMethod != "POST" { - t.Fatalf("want POST, got %s", gotMethod) - } -} - -func TestPolicyNamespace_GetSetValidate(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - switch r.Method { - case "GET": - _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) - case "PUT": - w.WriteHeader(200) - case "POST": - _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) - } - })) - - pol, err := c.Policy.Get(context.Background()) - if err != nil { - t.Fatal(err) - } - if pol.Tables == nil { - t.Fatal("want tables map") - } - - err = c.Policy.Set(context.Background(), pol) - if err != nil { - t.Fatal(err) - } - - v, err := c.Policy.Validate(context.Background(), pol) - if err != nil { - t.Fatal(err) - } - if !v.Valid { - t.Fatal("want valid=true") - } -} - -func TestDLQNamespace(t *testing.T) { - t.Run("List", func(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) - })) - stats, err := c.DLQ.List(context.Background()) - if err != nil { - t.Fatal(err) - } - if stats.Total != 3 { - t.Fatalf("want total=3, got %d", stats.Total) - } - }) - - t.Run("Table", func(t *testing.T) { - var mu sync.Mutex - var gotParam string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - gotParam = r.URL.Query().Get("table") - mu.Unlock() - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) - })) - _, err := c.DLQ.Table(context.Background(), "clicks") - if err != nil { - t.Fatal(err) - } - mu.Lock() - defer mu.Unlock() - if gotParam != "clicks" { - t.Fatalf("want table=clicks, got %s", gotParam) - } - }) -} - -func TestPipesNamespace_CRUD(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - switch r.Method { - case "GET": - if r.URL.Path == "/v1/ops/pipes" { - _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) - } else { - _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) +// TestNamespaces_RequestShape: the method, path and query parameters every +// namespace helper puts on the wire, and the shape it decodes the reply into. +func TestNamespaces_RequestShape(t *testing.T) { + ctx := context.Background() + policy := &Policy{Tables: map[string]TablePolicy{}} + + tests := []struct { + name string + reply any // nil replies with a bare 200 + call func(*testing.T, *Client) + wantMethod string + wantPath string + wantTable string // expected ?table= parameter, if any + }{ + { + name: "Sys.Health", + call: func(t *testing.T, c *Client) { mustNoErr(t, c.Sys.Health(ctx)) }, + wantMethod: "GET", + wantPath: "/v1/health", + }, + { + name: "Schema.List keys tables by name", + reply: []TableSchema{{Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}}, + call: func(t *testing.T, c *Client) { + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatal(err) + } + if _, ok := schemas["clicks"]; !ok { + t.Fatalf("want clicks in schemas, got %v", schemas) + } + }, + wantMethod: "GET", + wantPath: "/v1/ops/schema", + }, + { + name: "Schema.Refresh", + call: func(t *testing.T, c *Client) { mustNoErr(t, c.Schema.Refresh(ctx)) }, + wantMethod: "POST", + wantPath: "/v1/ops/schema/refresh", + }, + { + name: "Policy.Get", + reply: policy, + call: func(t *testing.T, c *Client) { + pol, err := c.Policy.Get(ctx) + if err != nil { + t.Fatal(err) + } + if pol.Tables == nil { + t.Fatal("want a tables map") + } + }, + wantMethod: "GET", + wantPath: "/v1/ops/policy", + }, + { + name: "Policy.Set", + call: func(t *testing.T, c *Client) { mustNoErr(t, c.Policy.Set(ctx, policy)) }, + wantMethod: "PUT", + wantPath: "/v1/ops/policy", + }, + { + name: "Policy.Validate", + reply: ValidationResult{Valid: true}, + call: func(t *testing.T, c *Client) { + v, err := c.Policy.Validate(ctx, policy) + if err != nil { + t.Fatal(err) + } + if !v.Valid { + t.Fatal("want valid=true") + } + }, + wantMethod: "POST", + wantPath: "/v1/ops/policy/validate", + }, + { + name: "DLQ.List totals every table", + reply: DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}, + call: func(t *testing.T, c *Client) { + stats, err := c.DLQ.List(ctx) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } + }, + wantMethod: "GET", + wantPath: "/v1/ops/dlq/stats", + }, + { + name: "DLQ.Table filters by table", + reply: DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}, + call: func(t *testing.T, c *Client) { + _, err := c.DLQ.Table(ctx, "clicks") + mustNoErr(t, err) + }, + wantMethod: "GET", + wantPath: "/v1/ops/dlq/stats", + wantTable: "clicks", + }, + { + name: "Pipes.List", + reply: []Pipe{{Name: "p1", SQL: "SELECT 1"}}, + call: func(t *testing.T, c *Client) { + pipes, err := c.Pipes.List(ctx) + if err != nil { + t.Fatal(err) + } + if len(pipes) != 1 || pipes[0].Name != "p1" { + t.Fatalf("want [p1], got %v", pipes) + } + }, + wantMethod: "GET", + wantPath: "/v1/ops/pipes", + }, + { + name: "Pipes.Get", + reply: Pipe{Name: "p1", SQL: "SELECT 1"}, + call: func(t *testing.T, c *Client) { + p, err := c.Pipes.Get(ctx, "p1") + if err != nil { + t.Fatal(err) + } + if p.Name != "p1" { + t.Fatalf("want p1, got %s", p.Name) + } + }, + wantMethod: "GET", + wantPath: "/v1/ops/pipes/p1", + }, + { + name: "Pipes.Set", + call: func(t *testing.T, c *Client) { mustNoErr(t, c.Pipes.Set(ctx, "p1", PipeDef{SQL: "SELECT 1"})) }, + wantMethod: "PUT", + wantPath: "/v1/ops/pipes/p1", + }, + { + name: "Pipes.Delete", + call: func(t *testing.T, c *Client) { mustNoErr(t, c.Pipes.Delete(ctx, "p1")) }, + wantMethod: "DELETE", + wantPath: "/v1/ops/pipes/p1", + }, + { + name: "PipeRef.Fetch posts the pipe's parameters", + reply: []map[string]any{{"count": 42}}, + call: func(t *testing.T, c *Client) { + rows, err := Fetch[map[string]any](ctx, c.Pipe("top_pages", map[string]any{"limit": 10})) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } + }, + wantMethod: "POST", + wantPath: "/v1/pipes/top_pages", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + respond := ok200 + if tt.reply != nil { + respond = jsonResponse(tt.reply) } - case "PUT": - w.WriteHeader(200) - case "DELETE": - w.WriteHeader(200) - } - })) + c, reqs := recordingClient(t, respond) + tt.call(t, c) - pipes, err := c.Pipes.List(context.Background()) - if err != nil { - t.Fatal(err) - } - if len(pipes) != 1 || pipes[0].Name != "p1" { - t.Fatalf("want [p1], got %v", pipes) - } - - p, err := c.Pipes.Get(context.Background(), "p1") - if err != nil { - t.Fatal(err) - } - if p.Name != "p1" { - t.Fatalf("want p1, got %s", p.Name) - } - - err = c.Pipes.Set(context.Background(), "p1", PipeDef{SQL: "SELECT 1"}) - if err != nil { - t.Fatal(err) - } - - err = c.Pipes.Delete(context.Background(), "p1") - if err != nil { - t.Fatal(err) + got := <-reqs + if got.method != tt.wantMethod || got.path != tt.wantPath { + t.Fatalf("want %s %s, got %s %s", tt.wantMethod, tt.wantPath, got.method, got.path) + } + if tbl := got.query.Get("table"); tbl != tt.wantTable { + t.Fatalf("want table=%q, got %q", tt.wantTable, tbl) + } + }) } } -func TestPipeRef_Fetch(t *testing.T) { - var mu sync.Mutex - var gotPath, gotMethod string - var gotBody map[string]any - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - gotPath = r.URL.Path - gotMethod = r.Method - _ = json.NewDecoder(r.Body).Decode(&gotBody) - mu.Unlock() - _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) - })) - rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) +func mustNoErr(t *testing.T, err error) { + t.Helper() if err != nil { t.Fatal(err) } - if len(rows) != 1 { - t.Fatalf("want 1 row, got %d", len(rows)) - } - mu.Lock() - defer mu.Unlock() - if gotPath != "/v1/pipes/top_pages" { - t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) - } - if gotMethod != "POST" { - t.Fatalf("want POST, got %s", gotMethod) - } } diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 7ccb9027..a9343ca7 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -1,124 +1,148 @@ package wavehouse import ( - "bytes" "context" "encoding/json" "fmt" - "io" "net/http" - "net/http/httptest" "sync" "testing" ) -func queryTestCtx(t *testing.T, handler http.Handler) *Client { - t.Helper() - srv := httptest.NewServer(handler) - t.Cleanup(srv.Close) - return NewClient(Config{ - BaseURL: srv.URL, - HTTPClient: srv.Client(), - Options: &ClientOptions{MaxRetries: 0}, - }) -} - -func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { - t.Helper() - // body is written on the server goroutine and read on the test goroutine; - // the mutex is what makes that visible under -race. - var mu sync.Mutex - var body []byte - wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw, err := io.ReadAll(r.Body) - if err != nil { - t.Errorf("read request body: %v", err) - } - mu.Lock() - body = raw - mu.Unlock() - handler.ServeHTTP(w, r) - }) - c := queryTestCtx(t, wrapper) - return c, func() map[string]any { - mu.Lock() - defer mu.Unlock() - var m map[string]any - _ = json.Unmarshal(body, &m) - return m - } -} - -var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { +// oneRow answers any query with a single-row result set. +var oneRow = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) }) func TestQueryBuilder_Immutability(t *testing.T) { - c := queryTestCtx(t, emptyRows) - b1 := c.From("clicks").Select("page") - b2 := b1.Where("score", OpGt, 10) - if b1 == b2 { + b1 := queryTestCtx(t, oneRow).From("clicks").Select("page") + if b2 := b1.Where("score", OpGt, 10); b1 == b2 { t.Fatal("builder should be immutable — chain methods return new instances") } } -func TestQueryBuilder_SelectColumns(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page", "button").FetchUntyped(context.Background()) - - body := getBody() - cols, ok := body["columns"].([]any) - if !ok || len(cols) != 2 { - t.Fatalf("want [page, button], got %v", body["columns"]) - } -} - -func TestQueryBuilder_SelectAll(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").SelectAll().FetchUntyped(context.Background()) - - body := getBody() - if body["select_all"] != true { - t.Fatalf("want select_all=true, got %v", body) - } -} - -func TestQueryBuilder_BareQueryDefaultsToSelectAll(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select().FetchUntyped(context.Background()) - - body := getBody() - if body["select_all"] != true { - t.Fatalf("bare query should default to select_all, got %v", body) - } -} - -func TestQueryBuilder_AggregationOnlyNoSelectAll(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select().Count("*", "n").FetchUntyped(context.Background()) - - body := getBody() - if body["select_all"] != nil { - t.Fatalf("aggregation-only query should not set select_all, got %v", body) +// TestQueryBuilder_RequestBody pins the exact StructuredQuery each chain puts +// on the wire. The table name always travels as a query parameter, never in +// the body — and CacheTTL is client-side only, so it appears in neither. +func TestQueryBuilder_RequestBody(t *testing.T) { + tests := []struct { + name string + run func(context.Context, *TableRef) error + want string + }{ + { + name: "the Fetch shortcut selects everything", + run: func(ctx context.Context, tr *TableRef) error { _, err := tr.Fetch(ctx); return err }, + want: `{"select_all":true,"limit":1000}`, + }, + { + name: "Select projects the named columns", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.Select("page", "button")) }, + want: `{"columns":["page","button"],"limit":1000}`, + }, + { + name: "SelectAll asks for every readable column", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.SelectAll()) }, + want: `{"select_all":true,"limit":1000}`, + }, + { + name: "a bare query defaults to select_all", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.Select()) }, + want: `{"select_all":true,"limit":1000}`, + }, + { + name: "an aggregation-only query does not set select_all", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.Select().Count("*", "n")) }, + want: `{"aggregations":[{"fn":"count","column":"*","alias":"n"}],"limit":1000}`, + }, + { + name: "Where appends a filter", + run: func(ctx context.Context, tr *TableRef) error { + return runFetch(ctx, tr.Select("page").Where("score", OpGt, 10)) + }, + want: `{"columns":["page"],"filters":[{"column":"score","op":"gt","value":10}],"limit":1000}`, + }, + { + name: "every aggregation helper, with and without an explicit alias", + run: func(ctx context.Context, tr *TableRef) error { + return runFetch(ctx, tr.Select().Count("*", "total").Sum("score", "").Avg("score", ""). + Min("score", "").Max("score", "").CountDistinct("page", ""). + Aggregate("uniqExact", "user_id", "unique_users")) + }, + want: `{"aggregations":[{"fn":"count","column":"*","alias":"total"},` + + `{"fn":"sum","column":"score","alias":"sum_score"},` + + `{"fn":"avg","column":"score","alias":"avg_score"},` + + `{"fn":"min","column":"score","alias":"min_score"},` + + `{"fn":"max","column":"score","alias":"max_score"},` + + `{"fn":"countDistinct","column":"page","alias":"count_distinct_page"},` + + `{"fn":"uniqExact","column":"user_id","alias":"unique_users"}],"limit":1000}`, + }, + { + name: "GroupBy", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.Select("page").GroupBy("page")) }, + want: `{"columns":["page"],"group_by":["page"],"limit":1000}`, + }, + { + name: "OrderBy", + run: func(ctx context.Context, tr *TableRef) error { + return runFetch(ctx, tr.Select("page").OrderBy("page", "desc")) + }, + want: `{"columns":["page"],"order_by":[{"column":"page","dir":"desc"}],"limit":1000}`, + }, + { + name: "Limit overrides the default", + run: func(ctx context.Context, tr *TableRef) error { return runFetch(ctx, tr.Select("page").Limit(50)) }, + want: `{"columns":["page"],"limit":50}`, + }, + { + name: "TimeRange omits an empty until", + run: func(ctx context.Context, tr *TableRef) error { + return runFetch(ctx, tr.Select("page").TimeRange("received_timestamp", "1h", "")) + }, + want: `{"columns":["page"],"limit":1000,"time_range":{"column":"received_timestamp","since":"1h"}}`, + }, + { + name: "a fully chained query composes every clause", + run: func(ctx context.Context, tr *TableRef) error { + return runFetch(ctx, tr.Select("page").Where("score", OpGt, 10).Count("*", "total"). + GroupBy("page").OrderBy("total", "desc").Limit(50). + TimeRange("received_timestamp", "1h", "").CacheTTL(60)) + }, + want: `{"columns":["page"],"aggregations":[{"fn":"count","column":"*","alias":"total"}],` + + `"filters":[{"column":"score","op":"gt","value":10}],"group_by":["page"],` + + `"order_by":[{"column":"total","dir":"desc"}],"limit":50,` + + `"time_range":{"column":"received_timestamp","since":"1h"}}`, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, reqs := recordingClient(t, oneRow) + if err := tt.run(context.Background(), c.From("clicks")); err != nil { + t.Fatal(err) + } + got := <-reqs + if got.method != "POST" || got.path != "/v1/query" { + t.Fatalf("want POST /v1/query, got %s %s", got.method, got.path) + } + if tbl := got.query.Get("table"); tbl != "clicks" { + t.Fatalf("want table=clicks in the query string, got %q", tbl) + } + if string(got.body) != tt.want { + t.Fatalf("body:\n got %s\nwant %s", got.body, tt.want) + } + }) } } -func TestQueryBuilder_Where(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page").Where("score", OpGt, 10).FetchUntyped(context.Background()) - - body := getBody() - filters, ok := body["filters"].([]any) - if !ok || len(filters) != 1 { - t.Fatalf("want 1 filter, got %v", body["filters"]) - } - f := filters[0].(map[string]any) - if f["column"] != "score" || f["op"] != "gt" { - t.Fatalf("want score/gt filter, got %v", f) - } +// runFetch runs the query and discards the page — the request body is what these +// tests assert on. +func runFetch(ctx context.Context, q *QueryBuilder) error { + _, err := q.FetchUntyped(ctx) + return err } +// TestQueryBuilder_AllOperators pins each SDK operator's wire spelling. func TestQueryBuilder_AllOperators(t *testing.T) { ops := []struct { sdk FilterOp @@ -136,157 +160,27 @@ func TestQueryBuilder_AllOperators(t *testing.T) { } for _, tt := range ops { t.Run(tt.wire, func(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) - body := getBody() - filters := body["filters"].([]any) - f := filters[0].(map[string]any) - if f["op"] != tt.wire { - t.Errorf("want wire op %s, got %s", tt.wire, f["op"]) + c, reqs := recordingClient(t, oneRow) + if err := runFetch(context.Background(), c.From("clicks").Select("x").Where("col", tt.sdk, "v")); err != nil { + t.Fatal(err) + } + want := fmt.Sprintf(`{"columns":["x"],"filters":[{"column":"col","op":%q,"value":"v"}],"limit":1000}`, tt.wire) + if got := string((<-reqs).body); got != want { + t.Fatalf("want %s, got %s", want, got) } }) } } -func TestQueryBuilder_Aggregations(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select(). - Count("*", "total"). - Sum("score", ""). - Avg("score", ""). - Min("score", ""). - Max("score", ""). - CountDistinct("page", ""). - Aggregate("uniqExact", "user_id", "unique_users"). - FetchUntyped(context.Background()) - - body := getBody() - aggs, ok := body["aggregations"].([]any) - if !ok || len(aggs) != 7 { - t.Fatalf("want 7 aggregations, got %v", body["aggregations"]) - } -} - -func TestQueryBuilder_GroupBy(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page").GroupBy("page").FetchUntyped(context.Background()) - - body := getBody() - gb, ok := body["group_by"].([]any) - if !ok || len(gb) != 1 || gb[0] != "page" { - t.Fatalf("want [page], got %v", body["group_by"]) - } -} - -func TestQueryBuilder_OrderBy(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page").OrderBy("page", "desc").FetchUntyped(context.Background()) - - body := getBody() - ob := body["order_by"].([]any) - o := ob[0].(map[string]any) - if o["column"] != "page" || o["dir"] != "desc" { - t.Fatalf("want page/desc, got %v", o) - } -} - -func TestQueryBuilder_Limit(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page").Limit(50).FetchUntyped(context.Background()) - - body := getBody() - if body["limit"] != float64(50) { - t.Fatalf("want 50, got %v", body["limit"]) - } -} - -func TestQueryBuilder_TimeRange(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("page"). - TimeRange("received_timestamp", "1h", ""). - FetchUntyped(context.Background()) - - body := getBody() - tr := body["time_range"].(map[string]any) - if tr["column"] != "received_timestamp" || tr["since"] != "1h" { - t.Fatalf("want received_timestamp/1h, got %v", tr) - } -} - -func TestQueryBuilder_Pagination_HasMore(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) - })) - - page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) - if err != nil { - t.Fatal(err) - } - if !page.HasMore { - t.Fatal("want hasMore=true") - } - if page.Next == nil { - t.Fatal("want next function") - } -} - -func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) - })) - - page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) - if err != nil { - t.Fatal(err) - } - if !page.HasMore { - t.Fatal("want hasMore=true") - } - if page.Next != nil { - t.Fatal("want nil next — no order column for cursor") - } -} - -func TestQueryBuilder_ComplexQuery(t *testing.T) { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks"). - Select("page"). - Where("score", OpGt, 10). - Count("*", "total"). - GroupBy("page"). - OrderBy("total", "desc"). - Limit(50). - TimeRange("received_timestamp", "1h", ""). - CacheTTL(60). - FetchUntyped(context.Background()) - - body := getBody() - if body["columns"].([]any)[0] != "page" { - t.Fatal("missing page column") - } - if body["limit"] != float64(50) { - t.Fatal("wrong limit") - } - if body["group_by"].([]any)[0] != "page" { - t.Fatal("wrong group_by") - } -} - -// pagingServer returns limit-sized pages of rows and captures each request -// body, so tests can walk page.Next and inspect the cursor filters sent. -func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, func() []map[string]any) { +// pagingServer answers each request with the next fixture page (an empty page +// once they run out) and records every request, so tests can walk page.Next +// and inspect the cursor filters sent. +func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, <-chan recordedRequest) { t.Helper() var mu sync.Mutex - var bodies []map[string]any call := 0 - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw, _ := io.ReadAll(r.Body) - var body map[string]any - dec := json.NewDecoder(bytes.NewReader(raw)) - dec.UseNumber() // keep int64 cursor values exact on the capture side too - _ = dec.Decode(&body) + return recordingClient(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { mu.Lock() - bodies = append(bodies, body) idx := call call++ mu.Unlock() @@ -296,19 +190,11 @@ func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, func() []map } _ = json.NewEncoder(w).Encode(page) })) - return c, func() []map[string]any { - mu.Lock() - defer mu.Unlock() - return append([]map[string]any(nil), bodies...) - } } -func filtersOf(t *testing.T, body map[string]any) []map[string]any { +func filtersOf(t *testing.T, req recordedRequest) []map[string]any { t.Helper() - raw, ok := body["filters"].([]any) - if !ok { - return nil - } + raw, _ := req.jsonBody(t)["filters"].([]any) out := make([]map[string]any, len(raw)) for i, f := range raw { out[i] = f.(map[string]any) @@ -316,69 +202,77 @@ func filtersOf(t *testing.T, body map[string]any) []map[string]any { return out } -func TestQueryBuilder_Pagination_NextWalksPages(t *testing.T) { - c, getBodies := pagingServer(t, [][]map[string]any{ - {{"id": "a"}, {"id": "b"}}, - {{"id": "c"}, {"id": "d"}}, - {{"id": "e"}}, - }) - - page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) - if err != nil { - t.Fatal(err) - } - page2, err := page.Next(context.Background()) - if err != nil { - t.Fatal(err) - } - if page2.Data[0]["id"] != "c" || !page2.HasMore || page2.Next == nil { - t.Fatalf("unexpected page 2: %+v", page2) - } - page3, err := page2.Next(context.Background()) +// A full page always means HasMore, but Next only exists when there is an +// order column to build a cursor from. The ordered half is covered by +// TestQueryBuilder_PaginationCursor, which walks the cursor it hands back. +func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { + c, _ := pagingServer(t, [][]map[string]any{{{"id": "a"}, {"id": "b"}}}) + page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) if err != nil { t.Fatal(err) } - if len(page3.Data) != 1 || page3.HasMore { - t.Fatalf("unexpected page 3: %+v", page3) - } - - bodies := getBodies() - if len(bodies) != 3 { - t.Fatalf("want 3 requests, got %d", len(bodies)) - } - if f := filtersOf(t, bodies[0]); len(f) != 0 { - t.Fatalf("page 1 must have no cursor filter, got %v", f) + if !page.HasMore { + t.Fatal("a full page means hasMore=true") } - // Page 2 and 3: exactly ONE cursor filter (replaced, not stacked), with - // the ascending op and the previous page's last cursor value. - for i, want := range []string{"b", "d"} { - f := filtersOf(t, bodies[i+1]) - if len(f) != 1 { - t.Fatalf("page %d: want exactly 1 cursor filter, got %v", i+2, f) - } - if f[0]["column"] != "id" || f[0]["op"] != "gt" || f[0]["value"] != want { - t.Fatalf("page %d: unexpected cursor filter %v", i+2, f[0]) - } + if page.Next != nil { + t.Fatal("want nil next — no order column for cursor") } } -func TestQueryBuilder_Pagination_DescUsesLt(t *testing.T) { - c, getBodies := pagingServer(t, [][]map[string]any{ - {{"id": "z"}, {"id": "y"}}, - {{"id": "x"}}, - }) - - page, err := c.From("clicks").Select("id").OrderBy("id", "desc").Limit(2).FetchUntyped(context.Background()) - if err != nil { - t.Fatal(err) - } - if _, err := page.Next(context.Background()); err != nil { - t.Fatal(err) - } - - f := filtersOf(t, getBodies()[1]) - if len(f) != 1 || f[0]["op"] != "lt" || f[0]["value"] != "y" { - t.Fatalf("desc cursor filter wrong: %v", f) +// TestQueryBuilder_PaginationCursor: every follow-up request carries exactly +// ONE cursor filter — replaced, never stacked — holding the previous page's +// last value, with the operator implied by the sort direction. +func TestQueryBuilder_PaginationCursor(t *testing.T) { + tests := []struct { + name string + dir string + op string + pages [][]map[string]any + cursors []string // expected cursor value per follow-up request + }{ + { + name: "ascending walks forward with gt", dir: "asc", op: "gt", + pages: [][]map[string]any{{{"id": "a"}, {"id": "b"}}, {{"id": "c"}, {"id": "d"}}, {{"id": "e"}}}, + cursors: []string{"b", "d"}, + }, + { + name: "descending walks back with lt", dir: "desc", op: "lt", + pages: [][]map[string]any{{{"id": "z"}, {"id": "y"}}, {{"id": "x"}}}, + cursors: []string{"y"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, reqs := pagingServer(t, tt.pages) + page, err := c.From("clicks").Select("id").OrderBy("id", tt.dir).Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if f := filtersOf(t, <-reqs); len(f) != 0 { + t.Fatalf("page 1 must have no cursor filter, got %v", f) + } + for i, want := range tt.cursors { + if page, err = page.Next(context.Background()); err != nil { + t.Fatal(err) + } + if got, want := page.Data[0]["id"], tt.pages[i+1][0]["id"]; got != want { + t.Fatalf("page %d: want first row %v, got %v", i+2, want, got) + } + f := filtersOf(t, <-reqs) + if len(f) != 1 { + t.Fatalf("page %d: want exactly 1 cursor filter, got %v", i+2, f) + } + if f[0]["column"] != "id" || f[0]["op"] != tt.op || f[0]["value"] != want { + t.Fatalf("page %d: unexpected cursor filter %v", i+2, f[0]) + } + } + if page.HasMore { + t.Fatal("the last fixture page is short — want hasMore=false") + } + if n := len(reqs); n != 0 { + t.Fatalf("want no requests beyond the pages walked, got %d more", n) + } + }) } } @@ -400,60 +294,63 @@ func TestQueryBuilder_Pagination_CursorColumnMissingEndsQuietly(t *testing.T) { } } -func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { +// TestQueryBuilder_PaginationCursorPrecision: the cursor value is read back off +// the decoded row, so how the row decoded decides how much precision survives. +// Typed rows keep int64 exactly; the untyped path decodes to float64 and loses +// everything past 2^53 (the same ceiling the TS SDK's JS numbers have). Use +// FetchTyped or codegen structs past 2^53 — documented in queries.md. +func TestQueryBuilder_PaginationCursorPrecision(t *testing.T) { type idRow struct { ID int64 `json:"id"` } - const bigID = int64(9007199254740993) // 2^53 + 1: float64 round-trip corrupts it - c, getBodies := pagingServer(t, [][]map[string]any{ - {{"id": 1}, {"id": bigID}}, - {}, - }) - - q := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2) - page, err := FetchTyped[idRow](context.Background(), q) - if err != nil { - t.Fatal(err) - } - if _, err := page.Next(context.Background()); err != nil { - t.Fatal(err) - } + const bigID = int64(9007199254740993) // 2^53 + 1: a float64 round-trip corrupts it - f := filtersOf(t, getBodies()[1]) - if len(f) != 1 { - t.Fatalf("want 1 cursor filter, got %v", f) - } - // json.Number survives the round-trip; float64 would have sent ...992. - if got := fmt.Sprint(f[0]["value"]); got != "9007199254740993" { - t.Fatalf("cursor value lost precision: %s", got) - } -} - -// TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling documents the known -// ceiling on the untyped path: rows decode to float64, so an integer cursor -// past 2^53 loses precision before pagination sees it (same as the TS SDK's -// JS-number ceiling). Use FetchTyped or codegen structs past 2^53. -func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { - c, getBodies := pagingServer(t, [][]map[string]any{ - {{"id": 1}, {"id": int64(9007199254740993)}}, - {}, - }) - - page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) - if err != nil { - t.Fatal(err) - } - if _, err := page.Next(context.Background()); err != nil { - t.Fatal(err) - } - - f := filtersOf(t, getBodies()[1]) - if len(f) != 1 { - t.Fatalf("want 1 cursor filter, got %v", f) - } - // float64 rounds 2^53+1 down to 2^53 — the documented untyped ceiling. - if got := fmt.Sprint(f[0]["value"]); got != "9007199254740992" { - t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) + tests := []struct { + name string + walk func(context.Context, *QueryBuilder) error + want string + }{ + { + name: "typed rows keep int64 exactly", + walk: func(ctx context.Context, q *QueryBuilder) error { + page, err := FetchTyped[idRow](ctx, q) + if err != nil { + return err + } + _, err = page.Next(ctx) + return err + }, + want: "9007199254740993", + }, + { + name: "untyped rows hit the float64 ceiling", + walk: func(ctx context.Context, q *QueryBuilder) error { + page, err := q.FetchUntyped(ctx) + if err != nil { + return err + } + _, err = page.Next(ctx) + return err + }, + want: "9007199254740992", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, reqs := pagingServer(t, [][]map[string]any{{{"id": 1}, {"id": bigID}}, {}}) + q := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2) + if err := tt.walk(context.Background(), q); err != nil { + t.Fatal(err) + } + <-reqs // page 1 carries no cursor + f := filtersOf(t, <-reqs) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + if got := fmt.Sprint(f[0]["value"]); got != tt.want { + t.Fatalf("cursor value: want %s, got %s (update docs if intentional)", tt.want, got) + } + }) } } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 533c14f2..f837837a 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -15,22 +15,33 @@ import ( "time" ) +// sseSender opens w as a flushed text/event-stream response and returns a +// frame writer — the preamble every SSE handler in this file starts with. The +// writer reports false once the client has gone away. +func sseSender(t *testing.T, w http.ResponseWriter) func(frame string) bool { + fl, ok := w.(http.Flusher) + if !ok { + t.Error("response writer is not a flusher") + return func(string) bool { return false } + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + fl.Flush() + return func(frame string) bool { + _, err := io.WriteString(w, frame) + fl.Flush() + return err == nil + } +} + // sseServer serves the given SSE frames on any request, then holds the // connection open until the client disconnects. func sseServer(t *testing.T, frames []string) *httptest.Server { t.Helper() srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "text/event-stream") - fl, ok := w.(http.Flusher) - if !ok { - t.Error("response writer is not a flusher") - return - } - w.WriteHeader(200) - fl.Flush() + send := sseSender(t, w) for _, f := range frames { - _, _ = io.WriteString(w, f) - fl.Flush() + send(f) } <-r.Context().Done() })) @@ -49,6 +60,36 @@ func streamClient(t *testing.T, srv *httptest.Server) *Client { return NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) } +// firstAPIError subscribes to sc and returns the first error it reports, +// failing the test if none arrives or it isn't a typed *Error. +func firstAPIError(t *testing.T, sc *StreamController) *Error { + t.Helper() + errCh := make(chan error, 4) + sc.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + return apiErr + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + return nil + } +} + +func recvEvent(t *testing.T, ch <-chan StreamEvent) StreamEvent { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for stream event") + return StreamEvent{} + } +} + func TestStream_SubscribeReceivesEvents(t *testing.T) { srv := sseServer(t, []string{ sseFrame("2026-01-01T00:00:01Z", "/home"), @@ -72,17 +113,6 @@ func TestStream_SubscribeReceivesEvents(t *testing.T) { } } -func recvEvent(t *testing.T, ch <-chan StreamEvent) StreamEvent { - t.Helper() - select { - case e := <-ch: - return e - case <-time.After(5 * time.Second): - t.Fatal("timed out waiting for stream event") - return StreamEvent{} - } -} - func TestStream_EventsChannel(t *testing.T) { srv := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) stream := streamClient(t, srv).From("clicks").Stream(nil) @@ -157,21 +187,16 @@ func TestStream_FilteredDeliversMatchesAndProjects(t *testing.T) { // inner stream is still delivering — the send-on-closed-channel regression. func TestStream_FilteredCloseUnderLoad(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "text/event-stream") - fl := w.(http.Flusher) - w.WriteHeader(200) - fl.Flush() + send := sseSender(t, w) for i := 0; ; i++ { select { case <-r.Context().Done(): return default: } - _, err := io.WriteString(w, sseFrame(fmt.Sprintf("2026-01-01T00:00:%02dZ", i%60), "/home")) - if err != nil { + if !send(sseFrame(fmt.Sprintf("2026-01-01T00:00:%02dZ", i%60), "/home")) { return } - fl.Flush() } })) t.Cleanup(srv.Close) @@ -202,21 +227,6 @@ func TestStream_FilteredCloseUnderLoad(t *testing.T) { } } -func TestStream_HandleMalformedSSEData(t *testing.T) { - sc := &StreamController{eventCh: make(chan StreamEvent, 1)} - var gotErr error - sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} - sc.handleSSEData("not json") // must not panic or emit - select { - case e := <-sc.eventCh: - t.Fatalf("malformed data emitted event: %+v", e) - default: - } - if gotErr == nil || !strings.Contains(gotErr.Error(), "malformed SSE message") { - t.Fatalf("want malformed-SSE error via subscriber, got %v", gotErr) - } -} - // TestStream_EventsBufferBeforeFirstEventsCall pins TS parity: the channel // buffers from construction, so events emitted before the first Events() call // are still delivered once the consumer starts reading. @@ -238,7 +248,19 @@ func TestStream_EventsBufferBeforeFirstEventsCall(t *testing.T) { // Client-side filter engine // --------------------------------------------------------------------------- +// TestEvaluateFilter covers the operator semantics the client-side filter +// engine has to reproduce. The timestamp block exists because the server +// canonicalizes every top-level DateTime value to RFC 3339 UTC before +// publishing (#402), so a payload and a caller's filter constant routinely +// spell the same instant differently; comparing those as text disagrees with +// the server's row filter, which compares DateTime columns chronologically. func TestEvaluateFilter(t *testing.T) { + // The canonicalized payload value, and the same instant in +02:00 — which + // sorts ABOVE it lexically ("06" > "04") while being chronologically equal. + const canonical = "2026-06-21T04:00:00Z" + const sameInstantOffset = "2026-06-21T06:00:00+02:00" + const oneSecondLater = "2026-06-21T06:00:01+02:00" + tests := []struct { name string actual any @@ -269,6 +291,43 @@ func TestEvaluateFilter(t *testing.T) { {"NotLike", "abc", "not_like", "x%", true}, {"LikeNonString", 5, "like", "5", false}, {"UnknownOp", "a", "regex", "a", false}, + + // Timestamps compare as instants, not as text. + {"TSEqualAcrossOffsets", canonical, "eq", sameInstantOffset, true}, + {"TSNeqFalseAcrossOffsets", canonical, "neq", sameInstantOffset, false}, + {"TSGteAtSameInstant", canonical, "gte", sameInstantOffset, true}, + {"TSLteAtSameInstant", canonical, "lte", sameInstantOffset, true}, + {"TSGtFalseAtSameInstant", canonical, "gt", sameInstantOffset, false}, + {"TSLtSeesLaterOffsetInstant", canonical, "lt", oneSecondLater, true}, + {"TSGtFalseAgainstLaterInstant", canonical, "gt", oneSecondLater, false}, + {"TSInMatchesAcrossOffsets", canonical, "in", []any{"2020-01-01T00:00:00Z", sameInstantOffset}, true}, + + // Same-offset spellings must keep working exactly as before. + {"TSGtWithinUTC", "2026-06-21T04:00:01Z", "gt", canonical, true}, + {"TSLtWithinUTC", "2026-06-21T03:59:59Z", "lt", canonical, true}, + {"TSEqIdenticalText", canonical, "eq", canonical, true}, + {"TSFractionalSecondsOrder", "2026-06-21T04:00:00.500Z", "gt", canonical, true}, + + // A zone-less constant names an instant only relative to the column's + // declared timezone, which a stream subscriber does not have. It must + // not be silently read as UTC — ordering fails closed. + {"TSZonelessFailsClosedOnGt", canonical, "gt", "2026-06-21 03:00:00", false}, + {"TSZonelessFailsClosedOnLt", canonical, "lt", "2026-06-21 05:00:00", false}, + // A ',' fraction is ISO 8601 but not ClickHouse, so it is not an instant. + {"TSCommaFractionIsNotAnInstant", canonical, "eq", "2026-06-21T04:00:00,000Z", false}, + + // nil only equals nil: a column missing from the payload must not match + // the literal string "" through equalValues' fmt.Sprint fallback. + {"NilEqualsNil", nil, "eq", nil, true}, + {"NilIsNotTheStringNil", nil, "eq", "", false}, + {"TheStringNilIsNotNil", "", "eq", nil, false}, + {"NilIsNotEmptyString", nil, "eq", "", false}, + {"NilIsNotZero", nil, "eq", 0, false}, + + // Non-timestamp strings keep lexicographic ordering; numbers are untouched. + {"PlainStringsOrderLexically", "banana", "gt", "apple", true}, + {"PlainStringsCompareEqual", "apple", "eq", "apple", true}, + {"NumbersOrderNumerically", 100.0, "gt", 9.0, true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -298,18 +357,6 @@ func TestMatchesFilters_AllMustMatch(t *testing.T) { } } -func TestCompareOrdered(t *testing.T) { - if c, ok := compareOrdered(float64(1), 2); !ok || c != -1 { - t.Fatalf("numeric compare: got (%d, %v)", c, ok) - } - if c, ok := compareOrdered("b", "a"); !ok || c != 1 { - t.Fatalf("string compare: got (%d, %v)", c, ok) - } - if _, ok := compareOrdered(map[string]any{}, 1); ok { - t.Fatal("incomparable types must return ok=false") - } -} - func TestToFloat64(t *testing.T) { for _, v := range []any{ float64(1), float32(1), @@ -334,43 +381,94 @@ func TestProjectColumns(t *testing.T) { } } -// TestStream_NonRetryableConnectErrorIsTerminal: a 403 must close the stream -// (no infinite reconnect) and surface the API error to Error subscribers. -func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden) - })) - t.Cleanup(srv.Close) +// TestStream_TerminalConnectFailures: every way a connection can fail in a way +// reconnecting cannot fix. Each case must surface a specific, non-retryable +// code, close the stream, and leave Connected failing — the generic retryable +// SSE_ERROR would spin here instead. +func TestStream_TerminalConnectFailures(t *testing.T) { + tests := []struct { + name string + handler http.HandlerFunc + baseURL string // overrides the test server URL when non-empty + auth func(context.Context) (string, error) + wantCode string + }{ + { + name: "a 403 the caller's credentials cannot fix", + handler: func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden) + }, + wantCode: "HTTP_403", + }, + { + name: "200 that is not an event stream", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(http.StatusOK) + _, _ = io.WriteString(w, "Please sign in") + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "200 with no Content-Type at all", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + w.WriteHeader(http.StatusOK) + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "credentialed request is redirected", + handler: func(w http.ResponseWriter, _ *http.Request) { + http.Redirect(w, &http.Request{}, "https://elsewhere.example/v1/stream", http.StatusFound) + }, + auth: StaticToken("secret-token"), + wantCode: "SSE_REDIRECT", + }, + { + name: "baseURL scheme cannot carry SSE", + handler: func(http.ResponseWriter, *http.Request) {}, + baseURL: "ws://example.invalid", + wantCode: "SSE_CONNECT_ERROR", + }, + } - stream := streamClient(t, srv).From("clicks").Stream(nil) - defer stream.Close() + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(tc.handler) + t.Cleanup(srv.Close) - errCh := make(chan error, 4) - stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + base := srv.URL + if tc.baseURL != "" { + base = tc.baseURL + } + client := NewClient(Config{BaseURL: base, Auth: tc.auth, HTTPClient: srv.Client()}) - select { - case err := <-errCh: - var apiErr *Error - if !errors.As(err, &apiErr) || apiErr.Status != http.StatusForbidden || apiErr.Retryable { - t.Fatalf("want non-retryable HTTP_403, got %v", err) - } - case <-time.After(5 * time.Second): - t.Fatal("error never surfaced") - } + stream := client.From("clicks").Stream(nil) + defer stream.Close() - select { - case <-stream.done: - case <-time.After(5 * time.Second): - t.Fatal("stream never closed after non-retryable connect error") - } - if s := stream.Status(); s != StatusClosed { - t.Fatalf("want closed, got %s", s) - } + apiErr := firstAPIError(t, stream) + if apiErr.Code != tc.wantCode { + t.Fatalf("want code %s, got %s", tc.wantCode, apiErr) + } + if apiErr.Retryable { + t.Fatalf("%s must not be retryable", apiErr.Code) + } - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - if err := stream.Connected(ctx); err == nil { - t.Fatal("Connected must fail on a terminally-closed stream") + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatalf("stream never closed after terminal %s", tc.wantCode) + } + if s := stream.Status(); s != StatusClosed { + t.Fatalf("want closed, got %s", s) + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err == nil { + t.Fatal("Connected must fail on a terminally-closed stream") + } + }) } } @@ -386,13 +484,9 @@ func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { n := len(sinceParams) mu.Unlock() - w.Header().Set("Content-Type", "text/event-stream") - fl := w.(http.Flusher) - w.WriteHeader(200) - fl.Flush() + send := sseSender(t, w) if n == 1 { - _, _ = io.WriteString(w, sseFrame("2026-01-01T00:00:01Z", "/home")) - fl.Flush() + send(sseFrame("2026-01-01T00:00:01Z", "/home")) return // server closes → client must reconnect with since= } <-r.Context().Done() @@ -441,9 +535,7 @@ func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { case gotPath <- r.URL.Path: default: } - w.Header().Set("Content-Type", "text/event-stream") - w.WriteHeader(200) - w.(http.Flusher).Flush() + sseSender(t, w) <-r.Context().Done() }) srv := httptest.NewServer(mux) // anything off-prefix 404s @@ -467,92 +559,6 @@ func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { } } -// TestStream_TerminalConnectFailures: every way a connection can fail in a way -// reconnecting cannot fix. Each case must surface a specific, non-retryable -// code and close the stream — the generic retryable SSE_ERROR would spin here. -func TestStream_TerminalConnectFailures(t *testing.T) { - tests := []struct { - name string - handler http.HandlerFunc - baseURL string // overrides the test server URL when non-empty - auth func(context.Context) (string, error) - wantCode string - }{ - { - name: "200 that is not an event stream", - handler: func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "text/html; charset=utf-8") - w.WriteHeader(http.StatusOK) - _, _ = io.WriteString(w, "Please sign in") - }, - wantCode: "SSE_BAD_CONTENT_TYPE", - }, - { - name: "200 with no Content-Type at all", - handler: func(w http.ResponseWriter, _ *http.Request) { - w.Header()["Content-Type"] = nil - w.WriteHeader(http.StatusOK) - }, - wantCode: "SSE_BAD_CONTENT_TYPE", - }, - { - name: "credentialed request is redirected", - handler: func(w http.ResponseWriter, _ *http.Request) { - http.Redirect(w, &http.Request{}, "https://elsewhere.example/v1/stream", http.StatusFound) - }, - auth: StaticToken("secret-token"), - wantCode: "SSE_REDIRECT", - }, - { - name: "baseURL scheme cannot carry SSE", - handler: func(http.ResponseWriter, *http.Request) {}, - baseURL: "ws://example.invalid", - wantCode: "SSE_CONNECT_ERROR", - }, - } - - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - srv := httptest.NewServer(tc.handler) - t.Cleanup(srv.Close) - - base := srv.URL - if tc.baseURL != "" { - base = tc.baseURL - } - client := NewClient(Config{BaseURL: base, Auth: tc.auth, HTTPClient: srv.Client()}) - - stream := client.From("clicks").Stream(nil) - defer stream.Close() - - errCh := make(chan error, 4) - stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) - - select { - case err := <-errCh: - var apiErr *Error - if !errors.As(err, &apiErr) { - t.Fatalf("want *Error, got %T: %v", err, err) - } - if apiErr.Code != tc.wantCode { - t.Fatalf("want code %s, got %s (%v)", tc.wantCode, apiErr.Code, err) - } - if apiErr.Retryable { - t.Fatalf("%s must not be retryable", apiErr.Code) - } - case <-time.After(5 * time.Second): - t.Fatal("error never surfaced") - } - - select { - case <-stream.done: - case <-time.After(5 * time.Second): - t.Fatalf("stream never closed after terminal %s", tc.wantCode) - } - }) - } -} - // TestStream_RedirectFollowedWhenUncredentialed: the refusal is scoped to // requests carrying a credential. Without one there is nothing to leak or // silently downgrade, so the redirect is followed as usual. @@ -580,16 +586,20 @@ func TestStream_RedirectFollowedWhenUncredentialed(t *testing.T) { } } -// TestStream_MalformedFrameIsTypedAndRetryable: a bad frame must arrive as an -// *Error so errors.As and IsRetryable work on it — a bare fmt.Errorf would -// leave callers string-matching. +// TestStream_MalformedFrameIsTypedAndRetryable: a bad frame must emit no +// event, and must arrive as an *Error so errors.As and IsRetryable work on it +// — a bare fmt.Errorf would leave callers string-matching. func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { srv := sseServer(t, []string{"id: 1\ndata: {not json\n\n"}) stream := streamClient(t, srv).From("clicks").Stream(nil) defer stream.Close() errCh := make(chan error, 4) - stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{ + Next: func(e StreamEvent) { events <- e }, + Error: func(err error) { errCh <- err }, + }) select { case err := <-errCh: @@ -597,8 +607,8 @@ func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { if !errors.As(err, &apiErr) { t.Fatalf("want *Error, got %T: %v", err, err) } - if apiErr.Code != "SSE_PARSE_ERROR" { - t.Fatalf("want SSE_PARSE_ERROR, got %s", apiErr.Code) + if apiErr.Code != "SSE_PARSE_ERROR" || !strings.Contains(apiErr.Message, "malformed SSE message") { + t.Fatalf("want a malformed-SSE parse error, got %s: %v", apiErr.Code, err) } if !IsRetryable(err) { t.Fatal("a malformed frame must stay retryable") @@ -609,6 +619,13 @@ func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { case <-time.After(5 * time.Second): t.Fatal("error never surfaced") } + // The error is emitted from the same handleSSEData call that would have + // emitted an event, so by now a leaked event would already be queued. + select { + case e := <-events: + t.Fatalf("malformed data emitted an event: %+v", e) + default: + } } // TestStream_ConfiguredHeadersReachTheStream: ClientOptions.Headers apply to @@ -620,11 +637,7 @@ func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { case seen <- r.Header.Clone(): default: } - w.Header().Set("Content-Type", "text/event-stream") - w.WriteHeader(http.StatusOK) - if fl, ok := w.(http.Flusher); ok { - fl.Flush() - } + sseSender(t, w) <-r.Context().Done() })) t.Cleanup(srv.Close) @@ -653,95 +666,9 @@ func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { } } -// TestEvaluateFilter_TimestampsCompareAsInstants: the server canonicalizes -// every top-level DateTime value to RFC 3339 UTC before publishing (#402), so -// a payload and a caller's filter constant routinely spell the same instant -// differently. Comparing those as text disagrees with the server's row filter, -// which compares DateTime columns chronologically. -func TestEvaluateFilter_TimestampsCompareAsInstants(t *testing.T) { - // The canonicalized payload value, and the same instant in +02:00 — which - // sorts ABOVE it lexically ("06" > "04") while being chronologically equal. - const canonical = "2026-06-21T04:00:00Z" - const sameInstantOffset = "2026-06-21T06:00:00+02:00" - const oneSecondLater = "2026-06-21T06:00:01+02:00" - - tests := []struct { - name string - actual any - op string - expected any - want bool - }{ - {"equal across offsets", canonical, "eq", sameInstantOffset, true}, - {"neq is false across offsets", canonical, "neq", sameInstantOffset, false}, - {"gte holds at the same instant", canonical, "gte", sameInstantOffset, true}, - {"lte holds at the same instant", canonical, "lte", sameInstantOffset, true}, - {"gt is false at the same instant", canonical, "gt", sameInstantOffset, false}, - {"lt sees a later offset instant", canonical, "lt", oneSecondLater, true}, - {"gt is false against a later instant", canonical, "gt", oneSecondLater, false}, - {"in matches across offsets", canonical, "in", []any{"2020-01-01T00:00:00Z", sameInstantOffset}, true}, - - // Same-offset spellings must keep working exactly as before. - {"gt within UTC", "2026-06-21T04:00:01Z", "gt", canonical, true}, - {"lt within UTC", "2026-06-21T03:59:59Z", "lt", canonical, true}, - {"eq identical text", canonical, "eq", canonical, true}, - - // Sub-second precision survives the round trip. - {"fractional seconds order correctly", "2026-06-21T04:00:00.500Z", "gt", canonical, true}, - - // A zone-less constant names an instant only relative to the column's - // declared timezone, which a stream subscriber does not have. It must - // not be silently read as UTC — ordering fails closed. - {"zone-less constant fails closed on gt", canonical, "gt", "2026-06-21 03:00:00", false}, - {"zone-less constant fails closed on lt", canonical, "lt", "2026-06-21 05:00:00", false}, - - // A ',' fraction is ISO 8601 but not ClickHouse, so it is not an instant. - {"comma fraction is not an instant", canonical, "eq", "2026-06-21T04:00:00,000Z", false}, - - // Non-timestamp strings keep lexicographic ordering. - {"plain strings still order lexically", "banana", "gt", "apple", true}, - {"plain strings still compare equal", "apple", "eq", "apple", true}, - - // Numbers are untouched by any of this. - {"numbers still order numerically", 100.0, "gt", 9.0, true}, - } - - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if got := evaluateFilter(tc.actual, tc.op, tc.expected, nil); got != tc.want { - t.Fatalf("evaluateFilter(%v, %q, %v) = %v, want %v", - tc.actual, tc.op, tc.expected, got, tc.want) - } - }) - } -} - -// TestEqualValues_NilOnlyEqualsNil: a column missing from the payload must not -// match the literal string "" through the fmt.Sprint fallback. -func TestEqualValues_NilOnlyEqualsNil(t *testing.T) { - tests := []struct { - name string - a, b any - want bool - }{ - {"nil equals nil", nil, nil, true}, - {"nil does not equal the string ", nil, "", false}, - {"the string does not equal nil", "", nil, false}, - {"nil does not equal empty string", nil, "", false}, - {"nil does not equal zero", nil, 0, false}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if got := equalValues(tc.a, tc.b); got != tc.want { - t.Fatalf("equalValues(%v, %v) = %v, want %v", tc.a, tc.b, got, tc.want) - } - }) - } -} - // TestStream_FilterMatchesCanonicalizedPayload: the end-to-end shape of the -// same bug — a caller filters on a non-UTC spelling and the server delivers the -// canonicalized one. +// instant-comparison bug — a caller filters on a non-UTC spelling and the +// server delivers the canonicalized one. func TestStream_FilterMatchesCanonicalizedPayload(t *testing.T) { frame := `event: message id: 2026-06-21T04:00:00Z diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 652daf88..154cfd34 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -2,169 +2,151 @@ package wavehouse import ( "context" - "encoding/json" - "io" "net/http" - "sync" "testing" ) -func TestTableRef_InsertSingle(t *testing.T) { - // mu guards handler captures throughout this file: the handler runs on the - // server goroutine and no happens-before edge exists via the TCP socket. - var mu sync.Mutex - var gotBody map[string]any - var gotPath string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - defer mu.Unlock() - gotPath = r.URL.Path - _ = json.NewDecoder(r.Body).Decode(&gotBody) - _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) - })) - result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) - if err != nil { - t.Fatal(err) - } - if !result.OK { - t.Fatal("want ok=true") - } - mu.Lock() - defer mu.Unlock() - if gotPath != "/v1/ingest" { - t.Fatalf("want /v1/ingest, got %s", gotPath) - } - if gotBody["page"] != "/home" { - t.Fatalf("want page=/home, got %v", gotBody) - } -} - -func TestTableRef_InsertBatch(t *testing.T) { - var mu sync.Mutex - var gotCT string - var gotBody string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - defer mu.Unlock() - gotCT = r.Header.Get("Content-Type") - raw, _ := io.ReadAll(r.Body) - gotBody = string(raw) - _ = json.NewEncoder(w).Encode(map[string]any{ - "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, - }) - })) - result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ - {"page": "/a"}, - {"page": "/b"}, - }) - if err != nil { - t.Fatal(err) - } - if !result.OK { - t.Fatal("want ok=true") - } - mu.Lock() - defer mu.Unlock() - if gotCT != "application/x-ndjson" { - t.Fatalf("want ndjson content type, got %s", gotCT) - } - if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { - t.Fatalf("want NDJSON body, got %s", gotBody) - } -} - -// TestTableRef_InsertTypedSlice covers the P1 finding: a typed slice (e.g. a -// generated or user-defined row type such as []ClickRow) must take the batch -// NDJSON path — not fall through to insertSingle, which would send the slice -// as a single JSON body and silently ignore any per-record failures the -// server reports. -func TestTableRef_InsertTypedSlice(t *testing.T) { +// TestTableRef_Insert: which wire format each argument shape takes on the way +// to /v1/ingest, and how the server's reply maps onto InsertResult. +func TestTableRef_Insert(t *testing.T) { type ClickRow struct { Page string `json:"page"` } + const ndjson = `{"page":"/a"}` + "\n" + `{"page":"/b"}` - var mu sync.Mutex - var gotCT string - var gotBody string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - defer mu.Unlock() - gotCT = r.Header.Get("Content-Type") - raw, _ := io.ReadAll(r.Body) - gotBody = string(raw) - _ = json.NewEncoder(w).Encode(map[string]any{ - "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, - }) - })) - result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ - {Page: "/a"}, - {Page: "/b"}, - }) - if err != nil { - t.Fatal(err) - } - mu.Lock() - defer mu.Unlock() - if gotCT != "application/x-ndjson" { - t.Fatalf("want ndjson content type, got %s", gotCT) - } - if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { - t.Fatalf("want NDJSON body, got %s", gotBody) - } - if result.OK { - t.Fatal("want ok=false when a batch record fails") - } - if result.Failed == nil || *result.Failed != 1 { - t.Fatalf("want failed=1, got %v", result.Failed) - } - if result.Total == nil || *result.Total != 2 { - t.Fatalf("want total=2, got %v", result.Total) + wantOK := func(t *testing.T, r *InsertResult) { + t.Helper() + if !r.OK { + t.Fatalf("want ok=true, got %+v", r) + } } -} + batchReply := map[string]any{"total": 2, "succeeded": 2, "failed": 0, "duplicates": 0} -// TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through -// insertSingle rather than being (mis)treated as a slice of per-byte rows. -func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { - var mu sync.Mutex - var gotPath, gotCT, gotBody string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - defer mu.Unlock() - gotPath = r.URL.Path - gotCT = r.Header.Get("Content-Type") - b, _ := io.ReadAll(r.Body) - gotBody = string(b) - _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) - })) - result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) - if err != nil { - t.Fatal(err) - } - if !result.OK { - t.Fatal("want ok=true") - } - mu.Lock() - defer mu.Unlock() - if gotPath != "/v1/ingest" { - t.Fatalf("want /v1/ingest, got %s", gotPath) - } - // The batch path posts to the same URL and also yields ok=true, so the - // wire format is the only thing that distinguishes them: one opaque JSON - // value vs. NDJSON of 16 per-byte rows. - if gotCT != "application/json" { - t.Fatalf("want application/json (single insert), got %q", gotCT) - } - // encoding/json base64s a []byte — documented in queries.md as a value the - // server rejects (use InsertNDJSON for raw bytes). Pinned here because it - // proves the batch path wasn't taken. - if gotBody != `"eyJwYWdlIjoiL2hvbWUifQ=="` { - t.Fatalf("want single base64 value, got %q", gotBody) + tests := []struct { + name string + insert func(context.Context, *TableRef) (*InsertResult, error) + reply map[string]any + wantCT string + wantBody string + check func(*testing.T, *InsertResult) + }{ + { + name: "a single map is sent as one JSON value", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.Insert(ctx, map[string]any{"page": "/home"}) + }, + reply: map[string]any{"ok": true}, + wantCT: "application/json", + wantBody: `{"page":"/home"}`, + check: wantOK, + }, + { + name: "a []map batch is sent as NDJSON", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.Insert(ctx, []map[string]any{{"page": "/a"}, {"page": "/b"}}) + }, + reply: batchReply, + wantCT: "application/x-ndjson", + wantBody: ndjson, + check: wantOK, + }, + { + // A typed slice (a generated or user-defined row type such as + // []ClickRow) must take the batch NDJSON path — not fall through to + // insertSingle, which would send the slice as one JSON body and + // silently ignore any per-record failures the server reports. + name: "a typed slice takes the batch path and surfaces per-record failures", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.Insert(ctx, []ClickRow{{Page: "/a"}, {Page: "/b"}}) + }, + reply: map[string]any{"total": 2, "succeeded": 1, "failed": 1, "duplicates": 0}, + wantCT: "application/x-ndjson", + wantBody: ndjson, + check: func(t *testing.T, r *InsertResult) { + t.Helper() + if r.OK { + t.Fatal("want ok=false when a batch record fails") + } + if r.Failed == nil || *r.Failed != 1 { + t.Fatalf("want failed=1, got %v", r.Failed) + } + if r.Total == nil || *r.Total != 2 { + t.Fatalf("want total=2, got %v", r.Total) + } + }, + }, + { + // The batch path posts to the same URL and also yields ok=true, so + // the wire format is the only thing that distinguishes them: one + // opaque JSON value vs. NDJSON of 16 per-byte rows. encoding/json + // base64s a []byte — documented in queries.md as a value the server + // rejects (use InsertNDJSON for raw bytes); pinned here because it + // proves the batch path wasn't taken. + name: "a []byte stays a single opaque value, not 16 rows", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.Insert(ctx, []byte(`{"page":"/home"}`)) + }, + reply: map[string]any{"ok": true}, + wantCT: "application/json", + wantBody: `"eyJwYWdlIjoiL2hvbWUifQ=="`, + check: wantOK, + }, + { + name: "InsertNDJSON forwards pre-formatted lines verbatim", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.InsertNDJSON(ctx, ndjson) + }, + reply: batchReply, + wantCT: "application/x-ndjson", + wantBody: ndjson, + check: func(t *testing.T, r *InsertResult) { + t.Helper() + if r.Total == nil || *r.Total != 2 { + t.Fatalf("want total=2, got %v", r.Total) + } + }, + }, + { + name: "a deduplicated row reports duplicate=true", + insert: func(ctx context.Context, tr *TableRef) (*InsertResult, error) { + return tr.Insert(ctx, map[string]any{"page": "/dup"}) + }, + reply: map[string]any{"duplicate": true}, + wantCT: "application/json", + wantBody: `{"page":"/dup"}`, + check: func(t *testing.T, r *InsertResult) { + t.Helper() + if r.Duplicate == nil || !*r.Duplicate { + t.Fatal("want duplicate=true") + } + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c, reqs := recordingClient(t, jsonResponse(tt.reply)) + result, err := tt.insert(context.Background(), c.From("clicks")) + if err != nil { + t.Fatal(err) + } + got := <-reqs + if got.path != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", got.path) + } + if ct := got.header.Get("Content-Type"); ct != tt.wantCT { + t.Fatalf("want Content-Type %q, got %q", tt.wantCT, ct) + } + if string(got.body) != tt.wantBody { + t.Fatalf("body:\n got %s\nwant %s", got.body, tt.wantBody) + } + tt.check(t, result) + }) } } func TestTableRef_InsertEmptyBatch(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - t.Fatal("should not make a request for empty batch") + c := queryTestCtx(t, http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + t.Error("should not make a request for empty batch") })) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) if err != nil { @@ -178,49 +160,18 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { } } -func TestTableRef_InsertNDJSON(t *testing.T) { - var mu sync.Mutex - var gotBody string - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - mu.Lock() - defer mu.Unlock() - raw, _ := io.ReadAll(r.Body) - gotBody = string(raw) - _ = json.NewEncoder(w).Encode(map[string]any{ - "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, - }) - })) - ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` - result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) - if err != nil { - t.Fatal(err) - } - if result.Total == nil || *result.Total != 2 { - t.Fatalf("want total=2, got %v", result.Total) - } - mu.Lock() - defer mu.Unlock() - if gotBody != ndjson { - t.Fatalf("want raw NDJSON, got %s", gotBody) - } -} - func TestTableRef_Schema(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Query().Get("table") != "clicks" { - t.Errorf("want table=clicks") - } - _ = json.NewEncoder(w).Encode(TableSchema{ - Name: "clicks", - Columns: []Column{ - {Name: "page", Type: "String"}, - }, - }) + c, reqs := recordingClient(t, jsonResponse(TableSchema{ + Name: "clicks", + Columns: []Column{{Name: "page", Type: "String"}}, })) schema, err := c.From("clicks").Schema(context.Background()) if err != nil { t.Fatal(err) } + if got := (<-reqs).query.Get("table"); got != "clicks" { + t.Fatalf("want table=clicks, got %q", got) + } if schema.Name != "clicks" { t.Fatalf("want clicks, got %s", schema.Name) } @@ -228,16 +179,3 @@ func TestTableRef_Schema(t *testing.T) { t.Fatalf("unexpected columns: %v", schema.Columns) } } - -func TestTableRef_InsertDuplicate(t *testing.T) { - c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) - })) - result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) - if err != nil { - t.Fatal(err) - } - if result.Duplicate == nil || !*result.Duplicate { - t.Fatal("want duplicate=true") - } -} From 29c6b6b9a0623f07988a693ac95cb5dc7ade19ff Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 10:20:38 -0400 Subject: [PATCH 47/59] fix(sdk): replay a missed terminal stream error to late subscribers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit emitError only walked the subscribers registered at the time it fired, so a connection that failed between `Stream(...)` returning and the caller's `Subscribe(...)` call delivered the error to nobody. The natural usage — stream := client.From("clicks").Stream(nil) stream.Subscribe(&wavehouse.StreamSubscriber{Error: ...}) therefore dropped terminal connect errors whenever the connect failed fast enough, which for an unusable scheme is immediate. `make ci` surfaced it as a flake in TestStream_TerminalConnectFailures/baseURL_scheme_cannot_carry_SSE under parallel load; the mechanism is a plain scheduling race, not a test bug. The controller now records the last error, and Subscribe replays it to a subscriber that registered too late — symmetric with the Status callback, which already fires immediately with the current status. Recording and snapshotting share one critical section, so a subscriber racing emitError is served by exactly one of the two paths and never sees the error twice. --- clients/go/stream.go | 20 +++++++++++++++++--- clients/go/stream_test.go | 21 +++++++++++++++++++++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/clients/go/stream.go b/clients/go/stream.go index 3a652be1..8d75acc2 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -28,6 +28,7 @@ type StreamController struct { cancel context.CancelFunc done chan struct{} closed bool + lastErr error // replayed to subscribers that register after it was emitted } // newController builds a controller whose event channel buffers from @@ -57,11 +58,14 @@ func (sc *StreamController) Status() StreamStatus { } // Subscribe registers callbacks and returns an unsubscribe function. The -// subscriber's Status callback fires immediately with the current status. +// subscriber's Status callback fires immediately with the current status, and +// its Error callback fires with the last error if one was already emitted — +// otherwise a connection that fails before Subscribe returns would deliver it +// to nobody. func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { sc.mu.Lock() sc.subscribers = append(sc.subscribers, sub) - currentStatus := sc.status + currentStatus, missedErr := sc.status, sc.lastErr sc.mu.Unlock() // Benign race: setStatus also calls the subscriber, so a stale status here @@ -69,6 +73,9 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { if sub.Status != nil { sub.Status(currentStatus) } + if missedErr != nil && sub.Error != nil { + sub.Error(missedErr) + } return func() { sc.mu.Lock() @@ -192,8 +199,15 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } +// emitError records err for late subscribers and delivers it to the current +// ones. Recording and snapshotting share one critical section so a subscriber +// registering concurrently is served by exactly one of the two paths. func (sc *StreamController) emitError(err error) { - for _, sub := range sc.snapshotSubs() { + sc.mu.Lock() + sc.lastErr = err + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + for _, sub := range subs { if sub.Error != nil { sub.Error(err) } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index f837837a..57e084dc 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -381,6 +381,27 @@ func TestProjectColumns(t *testing.T) { } } +// A connect that fails before Subscribe returns still has to reach the +// subscriber; without the replay in Subscribe this is a scheduling race that +// only shows up on a loaded machine. +func TestStream_SubscribeReplaysAMissedTerminalError(t *testing.T) { + stream := NewClient(Config{BaseURL: "ws://example.invalid"}).From("clicks").Stream(nil) + defer stream.Close() + <-stream.done // the connect has already failed and emitted + + errCh := make(chan error, 1) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) || apiErr.Code != "SSE_CONNECT_ERROR" { + t.Fatalf("want SSE_CONNECT_ERROR, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("a subscriber registered after the failure never saw it") + } +} + // TestStream_TerminalConnectFailures: every way a connection can fail in a way // reconnecting cannot fix. Each case must surface a specific, non-retryable // code, close the stream, and leave Connected failing — the generic retryable From 48a24e8db32b75d0d950321548f8ed2e736e7e32 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 11:01:50 -0400 Subject: [PATCH 48/59] fix(sdk): address the latest CodeRabbit review round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - conformance: the Go harness had no /v1/ingest branch, so both insert cases decoded the default `[]` into InsertResult, failed, and were only logged by logErr — the request assertions still passed, so the divergence was invisible. Both harnesses now answer ingest and health the same way, which is what the TS runner's comment already claimed. (/v1/health never actually failed: Sys.Health passes a nil decode target.) - conformance_ts.mjs: index an aggregation op's args defensively, matching the Go harness's stringArg, so a fixture like {"method":"count"} cannot crash one runner while the other accepts it. - pipes: PipeRef.Stream's doc comment claimed it works wherever the pipe name is also a table name. It does not — the pipe's SQL and params are never sent, so the caller gets that table's raw events. Say so, in the godoc and the docs page, and point at #445. - AGENTS.md: the second coverage summary at line 413 still said "sdk 50%" and omitted go-sdk entirely. --- AGENTS.md | 2 +- clients/go/conformance_test.go | 8 ++++++++ clients/go/pipes.go | 8 +++++--- docs/src/content/docs/sdk/go/pipes.md | 2 +- tests/conformance/conformance_ts.mjs | 2 +- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cf72a163..16de05d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -410,7 +410,7 @@ Internal-only backend changes (middleware refactors, observability internals, de 4. Use `testutil.MakeJWT(t, claims)` for auth tests, `testutil.NewTestSchemaRegistry(t, ...)` for schema-aware tests, `policy.NewMemoryStore(p)` for policy tests, `pipes.NewMemoryStore(queries...)` for pipes tests. 5. Use `testutil.AssertJSONResponse` and `testutil.AssertJSONContains` for HTTP handler assertions. 6. Run `make test` — it gates the unit-test coverage threshold from `.testcoverage.yml`, so a passing run already confirms coverage. -7. Aim for 80%+ coverage on new code. The project-wide CI-enforced minimum is 80% (merged unit + integration + e2e via `.testcoverage.yml`'s `threshold.total`); per-suite minima are unit 80%, integration 20%, e2e 60%, sdk 50%. +7. Aim for 80%+ coverage on new code. The project-wide CI-enforced minimum is 80% (merged unit + integration + e2e via `.testcoverage.yml`'s `threshold.total`); per-suite minima are unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts-unit 40%, ts-e2e 40%, ts-total 50%. ## File Structure diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 0d1324b0..5236ec7f 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -112,6 +112,14 @@ func TestConformance_WireFormat(t *testing.T) { _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) case r.URL.Path == "/v1/ops/pipes" && r.Method == "GET": _ = json.NewEncoder(w).Encode([]Pipe{}) + case strings.HasPrefix(r.URL.Path, "/v1/ingest"): + if r.Header.Get("Content-Type") == "application/x-ndjson" { + _ = json.NewEncoder(w).Encode(InsertResult{}) + } else { + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) + } + case r.URL.Path == "/v1/health": + _ = json.NewEncoder(w).Encode(map[string]any{"status": "ok"}) default: _ = json.NewEncoder(w).Encode([]map[string]any{}) } diff --git a/clients/go/pipes.go b/clients/go/pipes.go index a038ec73..d97bfe43 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -96,9 +96,11 @@ func (p *PipeRef) FetchUntyped(ctx context.Context) ([]map[string]any, error) { return Fetch[map[string]any](ctx, p) } -// Stream opens a live event stream from the pipe's underlying query. It -// subscribes by table name using the pipe's own name, so it only works when -// the pipe name is also a valid table name. +// Stream subscribes to live events using the pipe's name as a table name. The +// pipe's SQL and params are NOT applied: where a table of that name exists the +// caller receives its raw events, and otherwise the stream stays silent. Kept +// for parity with the TypeScript SDK's PipeRef.stream(), which behaves the same +// way; both wait on a pipe-aware stream endpoint (issue #445). func (p *PipeRef) Stream(opts *StreamOptions) *StreamController { return p.createStream(p.name, opts) } diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 7cbee2c8..4585d0bf 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -34,7 +34,7 @@ rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) ### `.Stream(opts)` -Opens a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). It streams by table name using the pipe's own name, so it works only where that name is also a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. +Subscribes to live events using the pipe's name as a table name; see [Streaming](/sdk/go/streaming). The pipe's SQL and params are **not** applied — where a table of that name exists you receive its raw events, and otherwise the stream stays silent rather than erroring. This mirrors the TypeScript SDK's `PipeRef.stream()`; both wait on a pipe-aware stream endpoint ([#445](https://github.com/Wave-RF/WaveHouse/issues/445)). ```go stream := wh.Pipe("top_pages", nil).Stream(nil) diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 54c58424..8db0a291 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -85,7 +85,7 @@ function applyQueryOps(wh, table, operations) { let q = wh.from(table).select(); for (const op of operations) { if (AGGREGATIONS.has(op.method)) { - q = q[op.method](op.args[0] || undefined, op.args[1] || undefined); + q = q[op.method](op.args?.[0] || undefined, op.args?.[1] || undefined); continue; } switch (op.method) { From 6c94796e57d86a8e41c79ee9ee991c5bee4fe7bf Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 11:12:38 -0400 Subject: [PATCH 49/59] fix: address the out-of-diff findings from CodeRabbit review 5020447247 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Makefile: GO_DIRS comes from `go list ./...` on the root module, which never yields the nested clients/go, so `make fmt` reported success on an unformatted SDK file. Only `make verify` caught it, via verify-go-sdk. gofumpt now gets clients/go explicitly; verified with a deliberately misformatted probe file. - scripts/cov: merge-all counted the standalone go-sdk suite in hasAnyCoverage but merge() only iterates goSuites, so go-sdk-only input exited 0 with no threshold applied. It now renders and gates the standalone suites. (`make cov` runs `report`, which already gated them — merge-all has no callers in the build, so this was reachable only by hand.) - docs/sdk/go/streaming.md: SSE_PARSE_ERROR was listed among the reconnect triggers. handleSSEData drops the frame and returns on the same connection, which is what reference.md already said. The two pages now agree with the code. - docs/sdk/go/pipes.md: the .Stream example never closed the controller, unlike every other stream example. --- Makefile | 2 +- docs/src/content/docs/sdk/go/pipes.md | 1 + docs/src/content/docs/sdk/go/streaming.md | 2 +- scripts/cov/main.go | 10 ++++++++++ 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index fbc1fa92..936f1c08 100644 --- a/Makefile +++ b/Makefile @@ -359,7 +359,7 @@ fmt: fmt-go fmt-ts ## Check formatting across Go (gofumpt) + TS (Biome). Run `ma .PHONY: fmt-go fmt-go: - $(call run,gofumpt (Go fmt),$(GOFUMPT) -l $(GO_DIRS) | (! grep .),run make fix to apply formatting) + $(call run,gofumpt (Go fmt),$(GOFUMPT) -l $(GO_DIRS) clients/go | (! grep .),run make fix to apply formatting) .PHONY: fmt-ts fmt-ts: pnpm-install diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 4585d0bf..a75f58ae 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -38,6 +38,7 @@ Subscribes to live events using the pipe's name as a table name; see [Streaming] ```go stream := wh.Pipe("top_pages", nil).Stream(nil) +defer stream.Close() ``` --- diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index c1076d94..1a81a3b3 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -108,7 +108,7 @@ Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3 ### Transport Behavior -SSE reconnects automatically with exponential backoff (capped at 30s) and gap-fill replay from the last event ID; HTTP/2 is recommended. Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). +SSE reconnects automatically with exponential backoff (capped at 30s) and gap-fill replay from the last event ID; HTTP/2 is recommended. Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect — the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). `/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). diff --git a/scripts/cov/main.go b/scripts/cov/main.go index f543676a..f432d1b1 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -150,6 +150,16 @@ func main() { if err := merge(cfg); err != nil { fatal("%v", err) } + // merge() only prints the standalone suites; gate them here too, or + // go-sdk-only input would exit 0 with no threshold applied. + for _, s := range standaloneGoSuites { + if !hasCovdata(filepath.Join(root, s, "data")) { + continue + } + if err := renderSuite(cfg, s); err != nil { + fatal("%v", err) + } + } if err := mergeTS(cfg); err != nil { fatal("%v", err) } From e17152f0dcf04dc25732ad6e0b6dfd47375c9523 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 11:18:39 -0400 Subject: [PATCH 50/59] test(sdk): cover the Go codegen CLI's argv parsing and schema fetch main_test.go exercised the generation helpers only, leaving parseArgs, flagValue, fetchSchemas and sortedKeys with no direct cases (CodeRabbit, PR #434). Adds table-driven coverage for flag values (long and short), defaults, the WAVEHOUSE_AUTH fallback and its --auth override, and, in a re-executed child process, the os.Exit branches for a missing flag value, an unknown argument and --help. fetchSchemas is driven against httptest servers for array- and map-shaped responses, a trimmed trailing slash, malformed JSON, a body that is neither shape, a non-200 status, an unbuildable URL and a transport failure, asserting the request path and that Authorization: Bearer is sent only when auth is supplied. Package coverage 55.8% -> 81.8%; parseArgs, flagValue, fetchSchemas and sortedKeys are now at 100%. Production code unchanged. --- clients/go/cmd/wavehouse-codegen/main_test.go | 244 ++++++++++++++++++ 1 file changed, 244 insertions(+) diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index c5a04432..1b6d0a82 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -1,8 +1,16 @@ package main import ( + "context" "encoding/json" + "errors" "go/format" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "reflect" "strings" "testing" ) @@ -171,3 +179,239 @@ func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { t.Fatalf("128-bit value corrupted: %s", rows[0].Big) } } + +// withArgs points os.Args at args for the duration of the test. parseArgs and +// flagValue read the global directly, so tests driving them must not run in +// parallel. +func withArgs(t *testing.T, args []string) { + t.Helper() + saved := os.Args + t.Cleanup(func() { os.Args = saved }) + os.Args = args +} + +func TestFlagValue(t *testing.T) { + tests := []struct { + name string + args []string + want string + wantI int + }{ + {name: "value follows the flag", args: []string{"cg", "--url", "http://h:9000"}, want: "http://h:9000", wantI: 2}, + // There is no lookahead: a flag-shaped value is consumed as the value. + {name: "flag-shaped value", args: []string{"cg", "--out", "--package"}, want: "--package", wantI: 2}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + withArgs(t, tt.args) + i := 1 + if got := flagValue(&i); got != tt.want { + t.Errorf("flagValue() = %q, want %q", got, tt.want) + } + if i != tt.wantI { + t.Errorf("index advanced to %d, want %d", i, tt.wantI) + } + }) + } +} + +func TestParseArgs(t *testing.T) { + // Every case differs from the defaults parseArgs starts with by a field or + // two, so deriving the wants keeps each row about what it changes. + defaults := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} + allFlags := cliArgs{url: "http://h:9000", out: "./types.go", auth: "argv-token", pkg: "db"} + secondURL, envAuth, argvAuth := defaults, defaults, defaults + secondURL.url = "http://second" + envAuth.auth = "env-token" + argvAuth.auth = "argv-token" + tests := []struct { + name string + args []string + env string // WAVEHOUSE_AUTH + want cliArgs + }{ + {name: "no arguments uses defaults", args: []string{"cg"}, want: defaults}, + { + name: "long flags", + args: []string{"cg", "--url", "http://h:9000", "--out", "./types.go", "--auth", "argv-token", "--package", "db"}, + want: allFlags, + }, + { + name: "short flags", + args: []string{"cg", "-u", "http://h:9000", "-o", "./types.go", "-a", "argv-token", "-p", "db"}, + want: allFlags, + }, + {name: "later flag wins", args: []string{"cg", "-u", "http://first", "--url", "http://second"}, want: secondURL}, + {name: "WAVEHOUSE_AUTH fills an unset --auth", args: []string{"cg"}, env: "env-token", want: envAuth}, + {name: "--auth beats WAVEHOUSE_AUTH", args: []string{"cg", "--auth", "argv-token"}, env: "env-token", want: argvAuth}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Setenv("WAVEHOUSE_AUTH", tt.env) // empty reads back the same as unset + withArgs(t, tt.args) + if got := parseArgs(); got != tt.want { + t.Errorf("parseArgs() = %+v, want %+v", got, tt.want) + } + }) + } +} + +// parseArgsChildEnv carries the argv under test to the re-executed child. +const parseArgsChildEnv = "WAVEHOUSE_CODEGEN_TEST_ARGS" + +// TestParseArgsExitPaths covers the branches that end in os.Exit, which can +// only be observed from outside the process: each case re-runs this test +// binary with the argv under test and asserts on the child's exit code and +// output. +func TestParseArgsExitPaths(t *testing.T) { + if raw, ok := os.LookupEnv(parseArgsChildEnv); ok { + withArgs(t, append([]string{"wavehouse-codegen"}, strings.Fields(raw)...)) + parseArgs() + t.Fatal("parseArgs returned instead of exiting") + } + tests := []struct { + name string + args string + wantCode int + wantOut string + }{ + {name: "missing value for a long flag", args: "--url", wantCode: 2, wantOut: "missing value for --url"}, + {name: "missing value for a short flag", args: "-o", wantCode: 2, wantOut: "missing value for -o"}, + {name: "missing value after a satisfied flag", args: "--url http://h:9000 --package", wantCode: 2, wantOut: "missing value for --package"}, + {name: "unknown argument", args: "--nope", wantCode: 2, wantOut: `unknown argument "--nope"`}, + {name: "bare value with no flag", args: "stray", wantCode: 2, wantOut: `unknown argument "stray"`}, + {name: "help exits cleanly", args: "--help", wantCode: 0, wantOut: "Generate Go types from WaveHouse schema"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // Re-executing this test binary is the standard way to observe + // an os.Exit path; the argv is fixed by the table above. + cmd := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestParseArgsExitPaths$") //nolint:gosec // the command is this test binary, not user input + cmd.Env = append(os.Environ(), parseArgsChildEnv+"="+tt.args) + out, err := cmd.CombinedOutput() + code := 0 + var exitErr *exec.ExitError + switch { + case errors.As(err, &exitErr): + code = exitErr.ExitCode() + case err != nil: + t.Fatalf("run child: %v", err) + } + if code != tt.wantCode { + t.Errorf("child exit code = %d, want %d\n%s", code, tt.wantCode, out) + } + if !strings.Contains(string(out), tt.wantOut) { + t.Errorf("child output missing %q:\n%s", tt.wantOut, out) + } + }) + } +} + +// schemaRequest is what the stub server saw. It travels over a buffered +// channel rather than a shared variable so -race sees the edge between the +// server goroutine and the test. +type schemaRequest struct { + path string + auth string +} + +// schemaServer answers every request with body under status (0 means 200). +func schemaServer(t *testing.T, status int, body string) (string, <-chan schemaRequest) { + t.Helper() + seen := make(chan schemaRequest, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + seen <- schemaRequest{path: r.URL.Path, auth: r.Header.Get("Authorization")} + if status != 0 { + w.WriteHeader(status) + } + _, _ = io.WriteString(w, body) + })) + t.Cleanup(srv.Close) + return srv.URL, seen +} + +func TestFetchSchemas(t *testing.T) { + const clicksJSON = `{"name":"clicks","columns":[{"name":"page","type":"String"},{"name":"ts","type":"DateTime","has_default":true}]}` + clicks := tableSchema{Name: "clicks", Columns: []column{ + {Name: "page", Type: "String"}, + {Name: "ts", Type: "DateTime", HasDefault: true}, + }} + tests := []struct { + name string + suffix string // appended to the stub server's base URL + auth string + status int + body string + want map[string]tableSchema + wantAuth string + wantErr string + }{ + { + name: "array response with bearer auth", auth: "tok-123", body: "[" + clicksJSON + "]", + want: map[string]tableSchema{"clicks": clicks}, wantAuth: "Bearer tok-123", + }, + { + name: "map response without auth", body: `{"clicks":` + clicksJSON + `}`, + want: map[string]tableSchema{"clicks": clicks}, + }, + {name: "trailing slash trimmed from base URL", suffix: "/", body: "[]", want: map[string]tableSchema{}}, + {name: "malformed JSON", body: `[{"name":`, wantErr: "read schema response"}, + {name: "JSON that is neither array nor map", body: `"nope"`, wantErr: "decode schema JSON"}, + {name: "non-200 status", status: http.StatusInternalServerError, body: "boom", wantErr: "schema fetch failed: HTTP 500"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + base, seen := schemaServer(t, tt.status, tt.body) + got, err := fetchSchemas(t.Context(), base+tt.suffix, tt.auth) + switch { + case tt.wantErr != "": + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("fetchSchemas() error = %v, want one containing %q", err, tt.wantErr) + } + case err != nil: + t.Fatalf("fetchSchemas() error = %v", err) + case !reflect.DeepEqual(got, tt.want): + t.Errorf("fetchSchemas() = %+v, want %+v", got, tt.want) + } + select { + case req := <-seen: + if req.path != "/v1/ops/schema" { + t.Errorf("requested path = %q, want /v1/ops/schema", req.path) + } + if req.auth != tt.wantAuth { + t.Errorf("Authorization header = %q, want %q", req.auth, tt.wantAuth) + } + default: + t.Error("stub server never saw a request") + } + }) + } +} + +func TestFetchSchemasRequestErrors(t *testing.T) { + canceled, cancel := context.WithCancel(t.Context()) + cancel() + tests := []struct { + name string + ctx context.Context + baseURL string + wantErr string + }{ + {name: "unparseable base URL", ctx: t.Context(), baseURL: "http://%zz", wantErr: "build schema request"}, + {name: "transport failure", ctx: canceled, baseURL: "http://127.0.0.1:1", wantErr: "fetch schema from"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if _, err := fetchSchemas(tt.ctx, tt.baseURL, ""); err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("fetchSchemas() error = %v, want one containing %q", err, tt.wantErr) + } + }) + } +} + +func TestSortedKeys(t *testing.T) { + got := sortedKeys(map[string]tableSchema{"clicks": {}, "acks": {}, "views": {}}) + if want := []string{"acks", "clicks", "views"}; !reflect.DeepEqual(got, want) { + t.Errorf("sortedKeys() = %v, want %v", got, want) + } +} From 50bdc2f741f749bbf3ffe034a20b106e8f3d44d0 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 13:21:13 -0400 Subject: [PATCH 51/59] refactor(sdk): topic-first SDK docs + one make target family MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses every thread @EricAndrechek left on #434. Docs — the documented decision, restored. PR #313 settled that SDK pages are topic-first: shared usage pages grow as languages land, each language keeps its own setup page, and the topic URLs never churn. The Go SDK's first draft deleted that comment and shipped a parallel /sdk/go/* tree with TypeScript left un-prefixed at the root of /sdk. Undone: - /sdk/{queries,streaming,pipes,admin,reference} are now .mdx, each carrying one block per section — shared prose outside, language-specific code and caveats inside. Nothing was dropped in the merge; both languages' full content is preserved. - /sdk/typescript (was the root /sdk) and /sdk/go are the per-language setup/caveats pages. /sdk is now a language-neutral overview. - The sidebar comment is back, and stronger: it says what a third language costs (one setup page, one TabItem per topic) and what it must not do (a parallel tree; a language at the root of /sdk). - AGENTS.md gains a §SDK docs layout section so the rule is enforceable rather than a comment one agent can delete. - SDK mentions across README, why-wavehouse, getting-started, the docs homepage, the 404 page and the footer drop the per-SDK dependency-count trivia and just name the clients we publish. The dependency posture stays where it is a real difference: the Go page's own comparison list. Make targets — one family, no new ad-hoc names. - test-ts -> test-sdk-ts, test-go-sdk -> test-sdk-go, plus test-sdk which runs both and test-sdk-go-e2e for the live-server suite. - test-conformance-ts is gone as a target: each language's half of the wire-format suite now rides that language's target, which is already how the Go half worked. - `make test` is the test-unit alias it was before this PR. - All four SDK targets use gotestsum and honor ARGS/V=1 like every other Go suite — the nested module has no tool directives, so they resolve the binary with `go tool -n gotestsum` from the root module first. Static checks — one entry point per tool, not per module. - lint-go-sdk and verify-go-sdk are deleted. lint-go now lints both modules (two invocations, because golangci-lint is module-scoped); verify-go-sdk was pure duplication — gofumpt is already fmt-go's job and `go vet` is golangci-lint's govet linter. - tidy covers the nested go.mod too, so verify checks exactly what fix rewrites. That closes the tidy half of #437; vulncheck remains open. - go-mod-download warms both modules, so anything reaching into clients/go inherits the prereq. - fix-go is one step per line instead of a five-command && chain, and a comment says the `cd` is the only difference between its two golangci-lint lines. Coverage — .testcoverage.yml now explains why the Go SDK has one gate key where the TS SDK has three (one instrumented suite vs two), and #518 tracks running the Go e2e suite from the orchestrator so it earns the same shape. Refs #434, #437, #518 --- .claude/commands/cover.md | 4 +- .github/workflows/ci.yml | 4 +- .testcoverage.yml | 11 + AGENTS.md | 16 +- CHANGELOG.md | 2 +- CONTRIBUTING.md | 4 +- Makefile | 164 +++-- README.md | 2 +- clients/go/README.md | 14 +- clients/ts/README.md | 6 +- docs/src/components/Footer.astro | 2 +- docs/src/config/sidebar.ts | 24 +- docs/src/content/docs/404.md | 3 +- docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/development.md | 77 ++- docs/src/content/docs/getting-started.md | 5 +- docs/src/content/docs/index.mdx | 13 +- docs/src/content/docs/pipes.mdx | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/admin.md | 90 --- docs/src/content/docs/sdk/admin.mdx | 190 +++++ .../content/docs/sdk/{go/index.md => go.md} | 40 +- docs/src/content/docs/sdk/go/admin.md | 94 --- docs/src/content/docs/sdk/go/pipes.md | 80 --- docs/src/content/docs/sdk/go/queries.md | 284 -------- docs/src/content/docs/sdk/go/reference.md | 185 ----- docs/src/content/docs/sdk/go/streaming.md | 217 ------ docs/src/content/docs/sdk/index.mdx | 548 ++------------- docs/src/content/docs/sdk/pipes.md | 53 -- docs/src/content/docs/sdk/pipes.mdx | 140 ++++ docs/src/content/docs/sdk/queries.md | 263 ------- docs/src/content/docs/sdk/queries.mdx | 647 ++++++++++++++++++ .../docs/sdk/{reference.md => reference.mdx} | 245 ++++++- docs/src/content/docs/sdk/streaming.md | 216 ------ docs/src/content/docs/sdk/streaming.mdx | 490 +++++++++++++ docs/src/content/docs/sdk/typescript.mdx | 565 +++++++++++++++ docs/src/content/docs/why-wavehouse.md | 2 +- scripts/cov/main.go | 21 +- 38 files changed, 2561 insertions(+), 2166 deletions(-) delete mode 100644 docs/src/content/docs/sdk/admin.md create mode 100644 docs/src/content/docs/sdk/admin.mdx rename docs/src/content/docs/sdk/{go/index.md => go.md} (78%) delete mode 100644 docs/src/content/docs/sdk/go/admin.md delete mode 100644 docs/src/content/docs/sdk/go/pipes.md delete mode 100644 docs/src/content/docs/sdk/go/queries.md delete mode 100644 docs/src/content/docs/sdk/go/reference.md delete mode 100644 docs/src/content/docs/sdk/go/streaming.md delete mode 100644 docs/src/content/docs/sdk/pipes.md create mode 100644 docs/src/content/docs/sdk/pipes.mdx delete mode 100644 docs/src/content/docs/sdk/queries.md create mode 100644 docs/src/content/docs/sdk/queries.mdx rename docs/src/content/docs/sdk/{reference.md => reference.mdx} (50%) delete mode 100644 docs/src/content/docs/sdk/streaming.md create mode 100644 docs/src/content/docs/sdk/streaming.mdx create mode 100644 docs/src/content/docs/sdk/typescript.mdx diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 7154b8de..8aed9ddb 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -13,8 +13,8 @@ Behavior: - **unit**: `make test-unit` (gates per-suite + writes `tmp/coverage/unit/`) - **integration**: `make test-integration` (requires Docker) - **e2e**: `make test-e2e` (requires Docker; orchestrator + cover binary) -- **go-sdk**: `make test-go-sdk` (nested module `clients/go`; gates against `suites.go-sdk`, rendered separately and never merged into the Go total) -- **ts-unit**: `make test-ts` (SDK unit tests + coverage + gate against `suites.ts-unit`) +- **go-sdk**: `make test-sdk-go` (nested module `clients/go`; gates against `suites.go-sdk`, rendered separately and never merged into the Go total) +- **ts-unit**: `make test-sdk-ts` (TS SDK unit tests + coverage + gate against `suites.ts-unit`) - **ts-e2e**: emitted as a side effect of `make test-e2e` (the orchestrator always passes `--coverage` to the e2e vitest run; informational only, no standalone gate) - **ts-total**: `make cov` (runs `cov report` — one consolidated Go + TS summary with per-suite HTML links + all gates; fails if *no* suite has data) - **all**: `make test-all` (every suite sequentially + `make cov`) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c6c4d15f..943aa504 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -204,8 +204,8 @@ jobs: uses: ./.github/actions/setup-env with: go-cache-suffix: "-unit" - - name: Run Go unit tests + SDK vitest + Go SDK tests - run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 + - name: Run Go unit tests + both SDK suites + run: make test-unit test-sdk COV_DEFER=1 # One fragment, three suites (unit, ts-unit, go-sdk) — all deferred to # the `coverage` job. Whole-tree path so a new suite needs no edit here. - name: Upload coverage fragment diff --git a/.testcoverage.yml b/.testcoverage.yml index 0d0595ed..f4705d1a 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -35,6 +35,17 @@ suites: # `-coverpkg=./...` can never reach it: rendered and gated on its own, # never merged into the total above, nothing to add under exclude.paths. # 75 vs 82.7% measured; the headroom is the codegen CLI. Raise as it fills. + # + # ONE key here, three for the TS SDK below, because the TS SDK has two + # coverage-PRODUCING suites and this has one. `ts-unit` is vitest, `ts-e2e` + # is the same SDK source re-measured by the orchestrator's live-server run, + # and `ts-total` exists only because those two have to be merged before a + # meaningful floor can be applied. `make test-sdk-go` is the Go SDK's only + # instrumented suite, so a `go-sdk-total` would be a copy of this number and + # a `go-sdk-e2e` would gate a suite that emits no covdata: `test-sdk-go-e2e` + # drives a server it did not start and runs uninstrumented. Wiring that suite + # into the orchestrator so it earns the same three-gate shape is #518 — add + # the other two keys with it, not before. go-sdk: 75 # TypeScript SDK suites — see scripts/cov for the merge / render logic. # vitest gates via --coverage.thresholds.statements; the merged ts-total diff --git a/AGENTS.md b/AGENTS.md index 16de05d7..80c7c1c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,7 +94,7 @@ make fix # Auto-fix everything fixable (gofumpt, goimports, lint - make test # Go unit tests + coverage gate (alias for test-unit) make test-integration # Go integration tests + gate (Docker; testcontainers) make test-e2e # E2E SDK suite vs the cover binary + gate (Docker; testcontainers) -make test-ts # SDK vitest unit tests + coverage + gate +make test-sdk # Both SDK suites (Go + TS) + coverage + gates make ci # Full pre-push pipeline — run it the documented way (§Local-First Validation) make build # Compile → bin/wavehouse make dev # ClickHouse + hot-reload server on :8080 (Docker) @@ -140,7 +140,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): make ci # Full parity with CI: parallel verify + builds + unit/SDK tests, then integration + E2E + cov ``` -If `make ci` passes locally, your commit has crossed the same gates CI will run — the CI workflow (`.github/workflows/ci.yml`) is a job DAG over the *same Makefile targets* (`verify`, `build-docs`, `test-unit`/`test-ts`, `test-integration`, `test-e2e`, `cov`), just spread across parallel runners. For workflow-only changes, read the YAML diff carefully and run `actionlint` if you have it installed. +If `make ci` passes locally, your commit has crossed the same gates CI will run — the CI workflow (`.github/workflows/ci.yml`) is a job DAG over the *same Makefile targets* (`verify`, `build-docs`, `test-unit`/`test-sdk`, `test-integration`, `test-e2e`, `cov`), just spread across parallel runners. For workflow-only changes, read the YAML diff carefully and run `actionlint` if you have it installed. ### Running `make ci` (for agents) @@ -364,7 +364,7 @@ The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`github.com/W | Backend change | SDK considerations | | -------------- | ------------------ | -| New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update doc pages under `docs/src/content/docs/sdk/` for both `ts/` and `go/`. Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | +| New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update the shared topic page under `docs/src/content/docs/sdk/` — add or extend the `` block so BOTH languages are covered on the same page (never a per-language page tree; see §SDK docs layout). Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | | Change to JWT auth / role extraction | TS: `clients/ts/src/http.ts` + `client.ts`. Go: `clients/go/http.go` + `wavehouse.go`. | | Change to `EventMessage` / ingest event format | Update payload types in both SDKs (some are codegen-regenerated — re-run both codegen CLIs). | | New / changed structured query AST | TS: `clients/ts/src/query-builder.ts`. Go: `clients/go/query_builder.go` + `types.go`. | @@ -377,6 +377,16 @@ Internal-only backend changes (middleware refactors, observability internals, de **The decision test**: would a user's *code* need to change to take advantage of (or be compatible with) this change? If yes, both SDKs need updates. If no (purely internal optimization), no. +### SDK docs layout + +The SDK docs are **topic-first, not language-first** — the decision from PR #313, carried into the Go SDK in PR #434. Adding a third language must not change the shape: + +- **One page per topic, shared by every language**: `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, `/sdk/reference`. Prose that holds for every client stays outside the tabs; the language-specific code and caveats go inside a `` block, one `` per language. The `syncKey` is `lang` everywhere, so a reader picks a language once and it follows them across the whole tree. +- **One setup/caveats page per language**: `/sdk/typescript`, `/sdk/go`. Installation, client construction, auth, and the error model are genuinely per-language and live here. +- **`/sdk` is the language-neutral overview.** No language sits at the root of `/sdk` — that was the trap the Go SDK's first draft fell into, leaving TypeScript un-prefixed and undocumented as a language. +- **Adding a language** = one new setup page + one new `` per topic page + one sidebar entry. It is never a parallel `/sdk//` tree, because that churns the topic URLs and doubles the pages to keep in sync. +- Because tabs need MDX, the topic pages are `.mdx` — remember §Markdown authoring rules: `make fix` does not auto-fix MDX, so unwrap WH001 wrapping by hand and keep a blank line between a JSX tag and a code fence. + ## Common Tasks ### Adding a new API endpoint diff --git a/CHANGELOG.md b/CHANGELOG.md index 849b018f..4705e61c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go.md` (new), `docs/src/content/docs/sdk/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); all four use gotestsum and honor `ARGS`/`V=1` like every other Go suite. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page — `/sdk/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/go`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 015b72b4..9473ca43 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` - - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), their doc trees (`docs/src/content/docs/sdk/` and `.../sdk/go/`), and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (one `` block per topic) plus the per-language setup pages `sdk/typescript.mdx` / `sdk/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). @@ -90,7 +90,7 @@ test(cache): add tiered cache stampede test ## Code Style -- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks the root module; the nested `clients/go` module is checked by `make verify` (its `verify-go-sdk` leaf, which the pre-commit hook and CI run). `make fix` applies gofumpt to both. +- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks both modules — the root one and the nested `clients/go` — as do `make lint` and `make tidy`. `make fix` applies the corresponding fixes to both. - **Linting**: All lint checks in `.golangci.yml` must pass (see `make lint`). - **Naming**: Follow [Go naming conventions](https://go.dev/doc/effective_go#names). - **Interfaces**: Define interfaces where they are consumed, not where they are implemented. diff --git a/Makefile b/Makefile index 936f1c08..ed1916a3 100644 --- a/Makefile +++ b/Makefile @@ -332,6 +332,12 @@ obs-front: ## Start local OTel Front UI # Dynamically find all directories containing Go files, safely ignoring hidden folders like .worktrees GO_DIRS := $(shell go list -f '{{.Dir}}' ./...) +# The Go SDK is a NESTED module (its own go.mod), so it is invisible to the +# root `go list ./...` above — every Go check names it explicitly. Path-based +# tools (gofumpt, goimports) take it as an extra argument; module-scoped tools +# (go mod tidy, golangci-lint) need a second invocation from inside it. +GO_SDK_DIR := clients/go + # fmt / lint / fix: one Biome binary (biome.json) scans the whole workspace # (SDK + e2e + docs); Markdown is owned separately by markdownlint-cli2 (rules in # .markdownlint.json, globs in .markdownlint-cli2.jsonc) — Biome only does @@ -359,22 +365,25 @@ fmt: fmt-go fmt-ts ## Check formatting across Go (gofumpt) + TS (Biome). Run `ma .PHONY: fmt-go fmt-go: - $(call run,gofumpt (Go fmt),$(GOFUMPT) -l $(GO_DIRS) clients/go | (! grep .),run make fix to apply formatting) + $(call run,gofumpt (Go fmt),$(GOFUMPT) -l $(GO_DIRS) $(GO_SDK_DIR) | (! grep .),run make fix to apply formatting) .PHONY: fmt-ts fmt-ts: pnpm-install $(call run,Biome (format),$(PNPM) -s -w run format,run make fix to apply formatting) .PHONY: lint -lint: lint-go lint-go-sdk lint-ts lint-md lint-prose ## Lint across Go (root + clients/go golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. - +lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint, both modules) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. + +# lint-go spans BOTH Go modules, the same way fmt-go does — one entry point per +# tool, not one per module. It takes two invocations rather than two targets +# because golangci-lint is module-scoped: `run ./...` resolves packages through +# the go.mod in its working directory, so the nested SDK needs its own run from +# inside $(GO_SDK_DIR). It picks up this repo's .golangci.yml either way +# (golangci-lint walks up to find the config). .PHONY: lint-go lint-go: $(GOLANGCI_LINT) go-mod-download $(call run,golangci-lint,$(GOLANGCI_LINT) run ./... --allow-parallel-runners,run make fix to auto-fix what is fixable) - -.PHONY: lint-go-sdk -lint-go-sdk: $(GOLANGCI_LINT) - $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./... --allow-parallel-runners,) + $(call run,golangci-lint (Go SDK),cd $(GO_SDK_DIR) && $(GOLANGCI_LINT) run ./... --allow-parallel-runners,run make fix to auto-fix what is fixable) .PHONY: lint-ts lint-ts: pnpm-install @@ -453,15 +462,12 @@ endif # tidy: read-only check via `go mod tidy -diff` (Go 1.23+). Prints the # unified diff that would be applied and exits non-zero if anything is off, # without touching go.mod / go.sum. Safe to run in parallel with fmt/lint. +# Spans both modules (see $(GO_SDK_DIR)) so `make verify` checks exactly what +# `make fix` would rewrite — the nested go.mod included. .PHONY: tidy -tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) +tidy: ## Verify go.mod/go.sum are tidy in both modules (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) - -# verify-go-sdk: nested module at clients/go/ — invisible to root go list. -.PHONY: verify-go-sdk -verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt - $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) - $(call run,gofumpt (Go SDK),$(GOFUMPT) -l clients/go | (! grep .),run make fix to apply formatting) + $(call run,go.mod tidy (Go SDK),cd $(GO_SDK_DIR) && go mod tidy -diff,run make fix to tidy go.mod and go.sum) # fix: apply auto-fixes everywhere, fanned out into three tracks that touch # disjoint files — Go (.go + go.mod/sum), TS/JS/JSON (Biome), Markdown — so they @@ -485,15 +491,19 @@ fix-docs: pnpm-install @$(MAKE) fix-md @$(MAKE) fix-prose +# The gofumpt/goimports passes take $(GO_SDK_DIR) as an extra path argument; +# `go mod tidy` and `golangci-lint` are module-scoped, so the nested SDK gets +# its own `cd $(GO_SDK_DIR) &&` line for each. That `cd` is the only difference +# between the two golangci-lint lines below — they are not a duplicate. .PHONY: fix-go fix-go: $(GOLANGCI_LINT) - @echo "$(CYAN)==> Applying Go auto-fixes (tidy + gofumpt + goimports + lint --fix)...$(RESET)" + @echo "$(CYAN)==> Applying Go auto-fixes (tidy + gofumpt + goimports + lint --fix, both modules)...$(RESET)" @go mod tidy - @$(GOFUMPT) -w $(GO_DIRS) - @$(GOIMPORTS) -w $(GO_DIRS) + @cd $(GO_SDK_DIR) && go mod tidy + @$(GOFUMPT) -w $(GO_DIRS) $(GO_SDK_DIR) + @$(GOIMPORTS) -w $(GO_DIRS) $(GO_SDK_DIR) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners - @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" - @$(GOFUMPT) -w clients/go && $(GOIMPORTS) -w clients/go && cd clients/go && go mod tidy && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @cd $(GO_SDK_DIR) && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install @@ -539,10 +549,11 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (16): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, -# lint-go-sdk + verify-go-sdk (both on the nested clients/go module) on the Go -# side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, -# docs spelling) + test-md-rules (node --test over the WH001/WH002 fixtures) +# Leaves (14): tidy, fmt-go (gofumpt), lint-go (golangci) and vulncheck on the +# Go side — the first three span BOTH modules (root + the nested clients/go +# SDK), so the SDK needs no leaves of its own; lint-ts (biome check) + lint-md +# (markdownlint) + lint-prose (misspell, docs spelling) + test-md-rules +# (node --test over the WH001/WH002 fixtures) # for JS/TS + Markdown + prose; lint-sh (shellcheck), lint-gha (actionlint), # test-classify-paths and test-release-channel for the tooling; # check-docs (astro check — the only leaf that writes, to docs/.astro/, and @@ -558,7 +569,7 @@ verify: ## Run all static checks across the repo (Go + TS + docs, parallelized) @printf "$(GREEN)$(BOLD)✔ All static checks passed$(RESET)\n" .PHONY: verify-parallel -verify-parallel: tidy fmt-go lint-go lint-go-sdk lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths test-md-rules test-release-channel vulncheck check-docs typecheck-ts verify-go-sdk +verify-parallel: tidy fmt-go lint-go lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths test-md-rules test-release-channel vulncheck check-docs typecheck-ts # typecheck-ts: tsc --noEmit on the SDK. Its own target (was inline in verify's # recipe) so it can run as a parallel leaf of verify-parallel. @@ -580,10 +591,18 @@ typecheck-ts: pnpm-install # go-mod-download is a no-doc intermediate target — every Go-toolchain target # (build/test/lint variants) declares it as a prereq so `make -j` doesn't # kick off N parallel `go mod download` calls racing on the module cache. -# Symmetric with pnpm-install for the Node side. +# Symmetric with pnpm-install for the Node side. It warms BOTH modules, so a +# target that reaches into $(GO_SDK_DIR) inherits the same guarantee from the +# one prereq (the SDK is stdlib-only today, so its half is ~free — it is there +# so the invariant holds if that ever changes). Only the SDK half swallows its +# output on success: with no requirements to fetch it prints a "no module +# dependencies to download" line that would otherwise show up ahead of every +# Go target. The root half stays unfiltered so a cold cache's "go: downloading" +# progress is still visible. .PHONY: go-mod-download go-mod-download: @go mod download + @cd $(GO_SDK_DIR) && out=$$(go mod download 2>&1) || { printf '%s\n' "$$out" >&2; exit 1; } .PHONY: $(BINARIES) $(BINARIES): go-mod-download @@ -672,7 +691,7 @@ branding-docs: ## Regenerate docs logo/favicon/OG assets from docs/src/assets/br # # Everything is driven directly via `pnpm --filter ` — no per-subproject # Makefiles. The user-facing Node targets live inline in their natural verb -# sections (build-ts/dev-ts/test-ts/clean-ts; build-docs/dev-docs/preview-docs/ +# sections (build-ts/dev-ts/clean-ts and test-sdk-ts; build-docs/dev-docs/preview-docs/ # branding-docs/clean-docs) and declare pnpm-install as a prereq so a fresh # clone or a changed lockfile is handled lazily. `make tools` does the full # bootstrap. @@ -757,9 +776,9 @@ test-unit: go-mod-download ## Run Go unit tests + render coverage + gate thresho @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render unit; fi # Hidden alias: `make test` matches `go test ./...` muscle memory; test-unit -# is the explicit form. +# is the explicit form. Server unit tests only — the SDK suites are `test-sdk`. .PHONY: test -test: test-unit test-go-sdk +test: test-unit .PHONY: test-integration test-integration: go-mod-download ## Run Go integration tests + render coverage + gate threshold (requires Docker) @@ -787,47 +806,64 @@ test-e2e: build-ts build-cover ## Run E2E SDK suite against cover binary + rende go run ./scripts/orchestrator @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render e2e; fi -# test-ts: vitest unit tests for the SDK, always with v8 coverage. Standalone -# it also gates against suites.ts-unit (via vitest's --coverage.thresholds); -# under COV_DEFER it only collects, leaving the gate to `make cov` (cov report) -# so CI emits one consolidated coverage block. THRESHOLD is read live from -# .testcoverage.yml via scripts/cov; override with -# `make test-ts ARGS='--coverage.thresholds.statements=70'`. -.PHONY: test-ts -test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against suites.ts-unit - @printf "$(CYAN)==> Running SDK unit tests...$(RESET)\n" - @rm -rf tmp/coverage/ts-unit && mkdir -p tmp/coverage/ts-unit - @TS_UNIT_COVERAGE_DIR="$(CURDIR)/tmp/coverage/ts-unit" \ - $(PNPM) --filter $(SDK_NAME) exec vitest run --coverage \ - $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) - @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi - -# test-go-sdk: unit tests for the nested clients/go module — outside -# test-unit's scope, so it needs its own target (-race: the SDK's streaming -# subsystem is the most concurrent code in the repo). Covdata lands in the -# same layout as the root-module suites, so `cov render go-sdk` gates it with -# no new machinery, but it is never merged into the Go total — see the go-sdk +# --- SDK test suites ---------------------------------------------------------- +# One naming family for every SDK suite: `test-sdk` runs them all, `test-sdk-go` +# / `test-sdk-ts` run one language. Each language target also runs ITS HALF of +# the cross-language wire-format conformance suite (both halves replay +# clients/go/testdata/wire_cases.json) — the Go half is an ordinary test file in +# the SDK package, so folding the TS half into test-sdk-ts keeps the two +# symmetric and costs no extra target. +.PHONY: test-sdk +test-sdk: test-sdk-go test-sdk-ts ## Run both SDK test suites (Go + TypeScript) incl. cross-language wire conformance + +# test-sdk-go: unit tests for the nested clients/go module — outside test-unit's +# scope, so it needs its own target (-race: the SDK's streaming subsystem is the +# most concurrent code in the repo). gotestsum comes from the ROOT module's tool +# directives, which `go tool` can only resolve there — hence `go tool -n` to +# resolve the binary before cd-ing into the nested module. Covdata lands in the +# same layout as the root-module suites, so `cov render go-sdk` gates it with no +# new machinery, but it is never merged into the Go total — see the go-sdk # comment in .testcoverage.yml. -.PHONY: test-go-sdk -test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render coverage + gate threshold +.PHONY: test-sdk-go +test-sdk-go: go-mod-download ## Run Go SDK (clients/go, a nested module) unit + conformance tests + render coverage + gate threshold @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" @rm -rf $(COV_GOSDK)/data && mkdir -p $(COV_GOSDK)/data - @cd clients/go && GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" go test -cover -coverpkg=./... -race ./... \ + @gotestsum=$$(go tool -n gotestsum) && cd $(GO_SDK_DIR) && \ + GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" "$$gotestsum" --format $(GOTESTSUM_FMT) -- \ + -cover -coverpkg=./... -race ./... $(ARGS) \ -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi -# test-conformance-ts: TS half of the cross-SDK wire-format conformance suite -# (Go half: clients/go/conformance_test.go); both replay the same fixture. -.PHONY: test-conformance-ts -test-conformance-ts: build-ts ## Run TS SDK wire-format conformance against the shared fixture - @printf "$(CYAN)==> Running TS wire-format conformance...$(RESET)\n" +# test-sdk-ts: vitest unit tests for the TS SDK, always with v8 coverage, then +# the TS half of the wire conformance suite (which replays the fixture through +# the built dist — hence the build-ts prereq). Standalone it also gates against +# suites.ts-unit (via vitest's --coverage.thresholds); under COV_DEFER it only +# collects, leaving the gate to `make cov` (cov report) so CI emits one +# consolidated coverage block. THRESHOLD is read live from .testcoverage.yml via +# scripts/cov; override with +# `make test-sdk-ts ARGS='--coverage.thresholds.statements=70'`. +.PHONY: test-sdk-ts +test-sdk-ts: pnpm-install build-ts ## Run TS SDK vitest unit + conformance tests + coverage + gate against suites.ts-unit + @printf "$(CYAN)==> Running TypeScript SDK tests...$(RESET)\n" + @rm -rf tmp/coverage/ts-unit && mkdir -p tmp/coverage/ts-unit + @TS_UNIT_COVERAGE_DIR="$(CURDIR)/tmp/coverage/ts-unit" \ + $(PNPM) --filter $(SDK_NAME) exec vitest run --coverage \ + $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) + @printf "$(CYAN)==> Running TypeScript wire-format conformance...$(RESET)\n" @node tests/conformance/conformance_ts.mjs + @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi -# test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. -.PHONY: test-go-sdk-e2e -test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) +# test-sdk-go-e2e: the Go SDK against a LIVE server (WAVEHOUSE_URL + +# WAVEHOUSE_AUTH), behind the `e2e` build tag. Unlike test-e2e it brings up +# nothing itself and collects no coverage; wiring it into the orchestrator so it +# runs in CI with a coverage gate — the way the TS SDK's e2e half already does — +# is tracked in #518. +.PHONY: test-sdk-go-e2e +test-sdk-go-e2e: go-mod-download ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" - @cd clients/go && go test -tags e2e -v -count=1 -timeout 60s ./... + @gotestsum=$$(go tool -n gotestsum) && cd $(GO_SDK_DIR) && \ + "$$gotestsum" --format $(GOTESTSUM_FMT) -- \ + -tags e2e -count=1 -timeout 60s ./... $(ARGS) # Aggregator: recipe-based with $(MAKE) calls so suites run sequentially even # under `make -j N`. The suites bind ports / spin testcontainers / start the @@ -835,9 +871,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 - @$(MAKE) test-go-sdk COV_DEFER=1 - @$(MAKE) test-conformance-ts - @$(MAKE) test-ts COV_DEFER=1 + @$(MAKE) test-sdk COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 @$(MAKE) cov @@ -876,7 +910,7 @@ cov: go-mod-download ## Consolidated coverage report (Go + TS) + gate against th # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-conformance-ts +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-sdk .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage @@ -1032,7 +1066,7 @@ clean-all: clean clean-test clean-tools ## Full reset — clean + clean-test + c # tools: bootstrap a fresh clone. # - Installs pinned external binaries to .bin/ (golangci-lint, air, misspell). # - Downloads Go modules so go.mod tool deps are available offline. -# - Installs SDK + E2E pnpm deps so test-ts / test-e2e are runnable +# - Installs SDK + E2E pnpm deps so test-sdk-ts / test-e2e are runnable # without a separate manual setup step. # # Note: go.mod tool deps (gotestsum, gofumpt, etc.) are *downloaded* by diff --git a/README.md b/README.md index 34d16188..45ba081b 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): query builder, live queries, streaming, and schema codegen in both. TypeScript has one runtime dependency (an SSE frame parser, ~1.4 KB gzipped); Go has none. +- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): query builder, live queries, streaming, and schema codegen in both, over one shared wire format. ## How it compares diff --git a/clients/go/README.md b/clients/go/README.md index 68809b8c..f1d7d803 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -2,7 +2,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. Zero third-party runtime dependencies, SSE parser included. -**Full documentation: [wavehouse.dev/sdk/go](https://wavehouse.dev/sdk/go)** +**Full documentation: [wavehouse.dev/sdk/go](https://wavehouse.dev/sdk/go)** (setup); the usage guides below cover both SDKs, tabbed by language. ## Install @@ -47,12 +47,12 @@ func main() { ## Documentation -- [Go SDK](https://wavehouse.dev/sdk/go) — client config, auth, typed rows via generics, error handling. -- [Queries](https://wavehouse.dev/sdk/go/queries) — tables, the query builder, inserts, pagination, raw SQL. -- [Streaming & Live Queries](https://wavehouse.dev/sdk/go/streaming) — SSE streams, client-side filtering, backfill-then-live. -- [Pipes](https://wavehouse.dev/sdk/go/pipes) — execute and manage named query pipes. -- [Admin & System](https://wavehouse.dev/sdk/go/admin) — schema, policy, DLQ stats, health. -- [Reference & CLI](https://wavehouse.dev/sdk/go/reference) — error codes, the full API tree, and the `wavehouse-codegen` struct generator. +- [Go SDK setup](https://wavehouse.dev/sdk/go) — client config, auth, typed rows via generics, error handling. +- [Queries](https://wavehouse.dev/sdk/queries) — tables, the query builder, inserts, pagination, raw SQL. +- [Streaming & Live Queries](https://wavehouse.dev/sdk/streaming) — SSE streams, client-side filtering, backfill-then-live. +- [Pipes](https://wavehouse.dev/sdk/pipes) — execute and manage named query pipes. +- [Admin & System](https://wavehouse.dev/sdk/admin) — schema, policy, DLQ stats, health. +- [Reference & CLI](https://wavehouse.dev/sdk/reference) — error codes, the full API tree, and the `wavehouse-codegen` struct generator. ## License diff --git a/clients/ts/README.md b/clients/ts/README.md index 4ec5a9f2..a30c9ee7 100644 --- a/clients/ts/README.md +++ b/clients/ts/README.md @@ -8,7 +8,7 @@ TypeScript client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — sche npm install @wavehouse/sdk ``` -Requires Node 22 or newer — the only line this SDK is tested against; Node 18 and 20 are past upstream end-of-life. Neither browsers nor Node need a polyfill: streaming runs on `fetch`, like the rest of the SDK — see [Runtime support](https://wavehouse.dev/sdk/#runtime-support). +Requires Node 22 or newer — the only line this SDK is tested against; Node 18 and 20 are past upstream end-of-life. Neither browsers nor Node need a polyfill: streaming runs on `fetch`, like the rest of the SDK — see [Runtime support](https://wavehouse.dev/sdk/typescript#runtime-support). This works in any framework that uses a bundler — React, Vue, Svelte, Angular, Astro, SolidJS, or plain Vite — with `import { createClient } from '@wavehouse/sdk'`. @@ -57,7 +57,7 @@ const wh = createClient({ }); ``` -`baseURL` may include a path prefix (`https://app.example.com/api/wavehouse`) when WaveHouse is served under one — see [Serving under a path prefix](https://wavehouse.dev/sdk#serving-under-a-path-prefix). +`baseURL` may include a path prefix (`https://app.example.com/api/wavehouse`) when WaveHouse is served under one — see [Serving under a path prefix](https://wavehouse.dev/sdk/typescript#serving-under-a-path-prefix). ### Query Data @@ -170,7 +170,7 @@ E2E tests live in `tests/e2e/sdk/` (repo root) and exercise the full pipeline th ## API Reference -See the full [SDK API Reference](https://wavehouse.dev/sdk) for detailed documentation of every method, type, and option. +See the full [SDK documentation](https://wavehouse.dev/sdk/typescript) for detailed documentation of every method, type, and option. ## License diff --git a/docs/src/components/Footer.astro b/docs/src/components/Footer.astro index cfeda0be..87d99617 100644 --- a/docs/src/components/Footer.astro +++ b/docs/src/components/Footer.astro @@ -146,7 +146,7 @@ const trademarkSource = pageText(entry.data, entry.body ?? ""); Getting Started Architecture API Reference - TypeScript SDK + Client SDKs
Operations diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index b4ec5da3..e40f45df 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,12 +25,17 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Separate trees per SDK — API shapes diverge enough that shared - // prose reads worse than dedicated pages. Revisit with tabs if a - // third language lands. - label: "TypeScript SDK", + // Topic-first SDK pages: the shared usage pages carry + // per language and each language keeps its own + // setup/caveats page — the topic URLs never churn as languages are + // added (decision in PR #313, executed for Go in PR #434). A third + // language adds one setup page and a tab per topic; it does NOT add a + // parallel tree, and no language sits at the root of /sdk. + label: "SDKs", items: [ { label: "Overview", slug: "sdk" }, + { label: "TypeScript setup", slug: "sdk/typescript" }, + { label: "Go setup", slug: "sdk/go" }, { label: "Queries", slug: "sdk/queries" }, { label: "Streaming & Live Queries", slug: "sdk/streaming" }, { label: "Pipes", slug: "sdk/pipes" }, @@ -38,17 +43,6 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ { label: "Reference & CLI", slug: "sdk/reference" }, ], }, - { - label: "Go SDK", - items: [ - { label: "Overview", slug: "sdk/go" }, - { label: "Queries", slug: "sdk/go/queries" }, - { label: "Streaming & Live Queries", slug: "sdk/go/streaming" }, - { label: "Pipes", slug: "sdk/go/pipes" }, - { label: "Admin & System", slug: "sdk/go/admin" }, - { label: "Reference & CLI", slug: "sdk/go/reference" }, - ], - }, ], }, { diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md index 0dd4e786..4eec97c6 100644 --- a/docs/src/content/docs/404.md +++ b/docs/src/content/docs/404.md @@ -45,8 +45,7 @@ head: Getting startedInstall, run, and ingest your first event ArchitectureHow the pieces fit together API referenceEndpoints, payloads, and error semantics - TypeScript SDKTyped client for browser and Node - Go SDKTyped client for Go services + Client SDKsTyped TypeScript and Go clients

Followed a link that should have worked? File an issue — broken links are bugs.

diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 7a42189c..f2c323ef 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -451,7 +451,7 @@ curl -X PUT http://localhost:8080/v1/ops/policy \ -d @policy.json ``` -The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`, and the [Go SDK](/sdk/go/admin) as `wh.Policy.Get(ctx)`, `wh.Policy.Set(ctx, policy)`, and `wh.Policy.Validate(ctx, policy)`. +The full request and response shapes for these endpoints live in the [API Reference](/api); both [SDKs](/sdk/admin#policy) wrap them — TypeScript as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`, Go as `wh.Policy.Get(ctx)`, `wh.Policy.Set(ctx, policy)`, and `wh.Policy.Validate(ctx, policy)`. ## Bootstrapping and the policy lifecycle diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index cc07e18f..1fb47543 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -18,7 +18,7 @@ You need these on your `PATH` before any `make` recipe will work end-to-end: | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | | **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file) | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | -| **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | +| **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-sdk-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | ### Auto-installed by `make tools` @@ -41,7 +41,7 @@ node --version # v22.x (matches .nvmrc and CI) pnpm --version # 11.21+ ``` -If any of those are wrong/missing, the Makefile recipes will fail with confusing errors (e.g. `--output-sync` is unrecognized on Make 3.81; `pnpm: command not found` on `make test-ts`). +If any of those are wrong/missing, the Makefile recipes will fail with confusing errors (e.g. `--output-sync` is unrecognized on Make 3.81; `pnpm: command not found` on `make test-sdk-ts`). ### Optional but recommended @@ -192,9 +192,9 @@ There's no bundled playground — point the published `@wavehouse/sdk` client (o WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev ``` -See the [SDK guide](/sdk) for the client API and examples. +See the [SDK guide](/sdk) for the client API and examples in both languages. -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see the [Go SDK docs](/sdk/go). +Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see [Go setup](/sdk/go). ### Validating tokens @@ -286,29 +286,36 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile invokes them as `go tool `, so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. +Every Go suite target — `test-unit`, `test-integration`, `test-sdk-go`, `test-sdk-go-e2e` — uses [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary, and every one of them honors `ARGS="..."` and `V=1`. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile invokes them as `go tool `, so no global installation is needed. The nested `clients/go` module has no `tool` directives of its own, so the SDK targets resolve the binary from the root module with `go tool -n gotestsum` before running it inside `clients/go`. `test-e2e` runs the orchestrator + vitest and `test-sdk-ts` runs vitest plus the Node conformance runner, so neither uses gotestsum. -Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. +Go tests run with the **race detector** (`-race`) enabled by default (including `test-sdk-go` — the SDK's streaming subsystem is highly concurrent; `test-sdk-go-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. ### Quick Reference ```bash -# V=1 gives verbose output on test-unit / test-integration / test-e2e, +# V=1 gives verbose output on every Go suite target and test-e2e, # e.g. `V=1 make test-unit` -# Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` +# Go server unit tests (compact output) — alias for `test-unit` make test -# Run specific root-module test(s) — ARGS reaches test-unit only; the -# test-go-sdk half of `make test` runs its full suite regardless +# Run specific test(s) — ARGS passes through to go test make test ARGS="-run TestValidate" # Go integration tests (requires Docker) make test-integration -# SDK vitest unit tests + coverage + gate against suites.ts-unit -# (`make cov` auto-merges ts-unit + ts-e2e — no separate command) -make test-ts +# Both SDK suites, each including its half of the wire-format +# conformance suite +make test-sdk + +# One SDK at a time +make test-sdk-go # nested clients/go module, -race, gates suites.go-sdk +make test-sdk-ts # vitest + coverage, gates suites.ts-unit + # (`make cov` auto-merges ts-unit + ts-e2e) + +# Go SDK against a live server (nothing is started for you) +WAVEHOUSE_URL=http://localhost:8080 make test-sdk-go-e2e # E2E SDK suite against bin/wavehouse-cov make test-e2e @@ -316,19 +323,19 @@ make test-e2e # All suites sequentially + merged coverage make test-all -# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts + -# test-conformance-ts, then test-integration + test-e2e + cov +# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-sdk, +# then test-integration + test-e2e + cov make ci # Merge available covdata + gate against total threshold make cov ``` -Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-go-sdk`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. `test-go-sdk` is gated but **not** merged: `clients/go` is a nested Go module, invisible to the root module's `-coverpkg=./...`, so its statements can never reach `tmp/coverage/total` — it carries its own `suites.go-sdk` floor instead, the same way the TS SDK carries `ts-*`. The remaining SDK/conformance targets (`test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-sdk-go`, `test-sdk-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. `test-sdk-go` is gated but **not** merged: `clients/go` is a nested Go module, invisible to the root module's `-coverpkg=./...`, so its statements can never reach `tmp/coverage/total` — it carries its own `suites.go-sdk` floor instead, the same way the TS SDK carries `ts-*`. `test-sdk-go-e2e` runs without coverage instrumentation or a per-suite gate; giving it the same unit/e2e/total gate shape the TS SDK has is tracked in [#518](https://github.com/Wave-RF/WaveHouse/issues/518). -**Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. +**Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on every gotestsum target (`test-unit`, `test-integration`, `test-sdk-go`, `test-sdk-go-e2e`), and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-sdk-ts` ignores it. -**Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. +**Extra flags**: every Go suite target plus `test-sdk-ts` accepts `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-sdk-ts`). `test-e2e` ignores it. **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). @@ -337,10 +344,10 @@ Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test- | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | -| SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | -| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`, always includes coverage + gate) | -| Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | -| SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | +| SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-sdk-ts` (always includes coverage + gate) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-sdk-go` (runs with `-race`, always includes coverage + gate) | +| Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Each half rides its language's target — Go in `make test-sdk-go`, TS in `make test-sdk-ts`; `make test-sdk` runs both | +| SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-sdk-go-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | @@ -353,8 +360,8 @@ Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). - **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). -- **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. -- **Go SDK unit test** → add to `clients/go/*_test.go` (nested module — outside `test-unit`'s scope). Run with `make test-go-sdk`. +- **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. (The Go SDK's own e2e suite is not yet driven by that orchestrator — [#518](https://github.com/Wave-RF/WaveHouse/issues/518).) +- **Go SDK unit test** → add to `clients/go/*_test.go` (nested module — outside `test-unit`'s scope). Run with `make test-sdk-go`. - **Wire-format parity case** → when you add or change an endpoint, add an entry to `clients/go/testdata/wire_cases.json` plus its dispatch in both runners (`clients/go/conformance_test.go` and `tests/conformance/conformance_ts.mjs`). Required by the SDK sync rule in `AGENTS.md` / `CONTRIBUTING.md`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). @@ -520,11 +527,11 @@ Run `make help` to see all targets. Key ones: | `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | | `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | | **Static checks** | | -| `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | -| `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | +| `make fmt` | Check formatting across Go (`gofumpt`, both the root module and the nested `clients/go`) + TS (Biome). Run `make fix` to apply. | +| `make tidy` | Verify `go.mod`/`go.sum` are tidy in both modules (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) + Markdown/MDX (markdownlint) + prose (misspell) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + docs type-check (`astro check` — not a full build, so link validation stays CI's job) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: Go (tidy + fmt + lint across both modules; `vulncheck` is still root-only — extending it to the nested module is tracked in [#437](https://github.com/Wave-RF/WaveHouse/issues/437)) + TS (Biome + `tsc` typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + docs type-check (`astro check` — not a full build, so link validation stays CI's job) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`), TS (Biome `--write`), Markdown (markdownlint `--fix`), MDX (`fix-mdx-fences` only — the generic fixers never run over `.mdx`), and docs-prose spelling (misspell, both) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | @@ -532,17 +539,17 @@ Run `make help` to see all targets. Key ones: | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/` | | **Test** | | -| `make test` | Alias for `test-unit` + `test-go-sdk` | +| `make test` | Alias for `test-unit` (Go server unit tests) | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | -| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` + render coverage + gate `suites.go-sdk` (own gate; never merged into the Go total) | -| `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | -| `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | +| `make test-sdk` | Both SDK suites — `test-sdk-go` + `test-sdk-ts` | +| `make test-sdk-go` | Go SDK (`clients/go`, nested module) unit + wire-conformance tests with `-race` + render coverage + gate `suites.go-sdk` (own gate; never merged into the Go total) | +| `make test-sdk-ts` | TS SDK vitest unit + wire-conformance tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | +| `make test-sdk-go-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | -| `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | | `make test-all` | All suites sequentially + merged coverage gate | -| `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | +| `make ci` | Full pipeline: parallel `verify` + builds + `test` / `test-sdk`, then integration + E2E + cov | | **Release** (see [Cutting a release](#cutting-a-release)) | | | `make release-server VERSION=X.Y.Z` | Tag a server release — binaries + container image | | `make release-sdk-ts VERSION=X.Y.Z` | Tag a TypeScript SDK release — npm | @@ -559,7 +566,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -`test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags; `test-unit`, `test-integration`, and `test-e2e` accept `V=1` for verbose output. `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. +Every Go suite target (`test-unit`, `test-integration`, `test-sdk-go`, `test-sdk-go-e2e`) plus `test-sdk-ts` accepts `ARGS="..."` for pass-through flags; those Go targets and `test-e2e` accept `V=1` for verbose output, which `test-sdk-ts` ignores. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management @@ -703,7 +710,7 @@ If the title doesn't match, a sticky comment posts on the PR explaining the form The `main branch protection` ruleset requires one status check to pass before any PR can merge: -- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts test-go-sdk test-conformance-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). +- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-sdk`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 69e0a097..1226670d 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -76,7 +76,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ `POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] -The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming; the [Go SDK](/sdk/go) offers the same builder with generics for typed rows. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +The [client SDKs](/sdk/queries) wrap this endpoint in a chainable query builder — autocomplete on your table names and row types in TypeScript, generics for typed rows in Go — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). ::: ## 5. Subscribe to real-time updates @@ -105,8 +105,7 @@ The handful of things that most often trip up a first session — each is expect - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. - **[API Reference](/api)** — every endpoint, request/response shape, and error code. -- **[TypeScript SDK](/sdk)** — client with query builder, live queries, and codegen; one runtime dependency. -- **[Go SDK](/sdk/go)** — the same surface for Go: context-first, generics for typed rows, codegen CLI. +- **[Client SDKs](/sdk)** — official TypeScript and Go clients: query builder, live queries, streaming, and codegen. - **[Configuration](/configuration)** — full YAML + environment variable reference. - **[Deployment](/deployment)** — Docker images, releases, health checks. - **[Development](/development)** — building from source, running tests, hot-reload workflow. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index da92388a..efb437b2 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -103,7 +103,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - `@wavehouse/sdk` and `github.com/Wave-RF/WaveHouse/clients/go` — type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. One runtime dependency of ~1.4 KB gzipped in TypeScript; none in Go. + `@wavehouse/sdk` and `github.com/Wave-RF/WaveHouse/clients/go` — type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. @@ -111,7 +111,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket -The [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming (examples below are TypeScript). Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set — see the [Go quick start](/sdk/go#quick-start). +The [client SDKs](/sdk) wrap the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming. Examples below are TypeScript; the Go client mirrors the same feature set, and every SDK page carries both languages side by side. @@ -212,15 +212,10 @@ Self-hosting WaveHouse is deliberately boring — one binary, one dependency. Bu href="/api" /> -
diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index dc3ac13d..5b6336cf 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -148,7 +148,7 @@ curl -X PUT http://localhost:8080/v1/ops/pipes/top_pages \ }' ``` -A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`; the [Go SDK](/sdk/go/pipes) as `wh.Pipes.List(ctx)`, `wh.Pipes.Get(ctx, name)`, `wh.Pipes.Set(ctx, name, def)`, and `wh.Pipes.Delete(ctx, name)`. +A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. Both [SDKs](/sdk/pipes#managing-pipe-definitions) expose the same operations — TypeScript as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`, Go as `wh.Pipes.List(ctx)`, `wh.Pipes.Get(ctx, name)`, `wh.Pipes.Set(ctx, name, def)`, and `wh.Pipes.Delete(ctx, name)`. ## Executing a pipe diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index f68dac7e..bc5b35d4 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -82,7 +82,7 @@ handle_path /api/wavehouse/* { The ingress controller forwards the full path unless you ask it to rewrite. Pair a capture-group path (`/api/wavehouse(/|$)(.*)` with `pathType: ImplementationSpecific`) with the `nginx.ingress.kubernetes.io/rewrite-target: /$2` annotation, or the prefix arrives at WaveHouse unstripped. ::: -Point either SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` in TypeScript, `wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wavehouse"})` in Go — both sending REST calls and SSE streams under the prefix ([SDK → Serving under a path prefix](/sdk#serving-under-a-path-prefix), [Go SDK → Creating a client](/sdk/go#creating-a-client)). +Point either SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` in TypeScript, `wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wavehouse"})` in Go — both sending REST calls and SSE streams under the prefix ([TypeScript → Serving under a path prefix](/sdk/typescript#serving-under-a-path-prefix), [Go → Creating a client](/sdk/go#creating-a-client)). ## Request-body size limits diff --git a/docs/src/content/docs/sdk/admin.md b/docs/src/content/docs/sdk/admin.md deleted file mode 100644 index 852ea02b..00000000 --- a/docs/src/content/docs/sdk/admin.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: "TypeScript SDK Admin & System" -description: "Schema introspection, access-control policy, DLQ stats, and health checks in @wavehouse/sdk." ---- - -Operational surfaces of `@wavehouse/sdk`. With one exception, everything on this page sits behind the server's admin gate: the caller must resolve to the admin role (`policy.admin_role`) or present the non-JWT [operator key](/api#authentication) — the SDK has no first-class operator-key option, but [`options.headers`](/sdk#custom-headers) can carry the `X-Operator-Key` header. The exception is `wh.sys.health()`, which calls the public, content-free `/v1/health` route and needs no credentials. See [Access Control](/access-control) for how roles resolve. Examples import from `@wavehouse/sdk` or `https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). - -## Schema — `wh.schema` - -Introspect ClickHouse table schemas. - -```ts -// List all table schemas -const { data: schemas } = await wh.schema.list(); -// schemas: { clicks: { name: 'clicks', columns: [...] }, users: { ... } } - -// Force refresh from ClickHouse -await wh.schema.refresh(); -``` - -Individual table schema is also available via `wh.from('clicks').schema()`. - -> `wh.schema.list()`, `wh.schema.refresh()`, and `wh.from(t).schema()` hit `/v1/ops/schema*`, which are **admin-only** endpoints: the caller must pass the admin gate — resolve to the policy admin role (`admin_role`, `"admin"` by default) or present the non-JWT [operator key](/api#authentication). Unless the deployment deliberately sets `default_role` to the admin role (the loudly-warned dev-only setting), construct the client with an admin-role token — or send the operator key via [`options.headers`](/sdk#custom-headers) — or these calls return `403`. - ---- - -## Policy — `wh.policy` - -Manage Hasura-style access control policies. Requires the admin gate — the admin role (`policy.admin_role`) or the [operator key](/api#authentication). - -```ts -// Get current policy -const { data: policy } = await wh.policy.get(); - -// Update policy -await wh.policy.set({ - default_role: 'viewer', - admin_role: 'admin', - tables: { - clicks: { - select: { - viewer: { - allow_columns: ['page', 'button', 'received_timestamp'], - filter: { tenant_id: { _eq: '{{ jwt.app_metadata.tenant_id }}' } }, - }, - admin: { allow_columns: ['*'] }, - }, - }, - }, -}); - -// Validate without applying (dry run) -const { data } = await wh.policy.validate(policyDraft); -// data: { valid: true } or error with validation details -``` - ---- - -## DLQ — `wh.dlq` - -Dead Letter Queue operations. Requires the admin gate — the admin role (`policy.admin_role`) or the [operator key](/api#authentication). - -```ts -// Get DLQ statistics -const { data } = await wh.dlq.list(); -// data: { tables: { "clicks": 3, "users": 0 }, total: 3 } - -// Stats for a specific table -const { data } = await wh.dlq.table('clicks'); -``` - -`wh.dlq.stream()` exists in the API but is **not yet functional**: there is no server-side DLQ stream today (the SSE bridge only carries `ingest.>` subjects), so it connects and receives no events rather than failing. Live DLQ streaming is tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). - ---- - -## System — `wh.sys` - -Content-free server-online check. - -```ts -// health() hits the public, content-free /v1/health route — 200/503, no body. -// Use it to check a server is reachable before sending data. -const result = await wh.sys.health(); -if (result.ok) { - // server is up and past boot -} -// on failure, result.error carries the reason (network vs. server error) -``` - -> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern, not the client's. Probe `/readyz` directly from your orchestrator if you need it. diff --git a/docs/src/content/docs/sdk/admin.mdx b/docs/src/content/docs/sdk/admin.mdx new file mode 100644 index 00000000..926f6b9e --- /dev/null +++ b/docs/src/content/docs/sdk/admin.mdx @@ -0,0 +1,190 @@ +--- +title: "SDK Admin & System" +description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse SDKs." +--- + +import { Tabs, TabItem } from "@astrojs/starlight/components"; + +Operational surfaces of the SDKs. With one exception, everything on this page sits behind the server's admin gate on `/v1/ops/*`: the caller must resolve to the admin role (`policy.admin_role`, `"admin"` by default) or present the server's non-JWT [operator key](/api#authentication) as an `X-Operator-Key` header. Without one, these calls return `403` — unless the deployment deliberately sets `default_role` to the admin role, a loudly-warned dev-only setting. The exception is the health check, which calls the public, content-free `/v1/health` route and needs no credentials. See [Access Control](/access-control) for how roles resolve. + +Neither SDK has a first-class operator-key option; each carries it as a custom header — [`options.headers`](/sdk/typescript#custom-headers) in TypeScript, [`ClientOptions.Headers`](/sdk/go#clientoptions) in Go. + +## Schema + +Introspect ClickHouse table schemas. A single table's schema is also available from its table ref (see [Queries → Table schema](/sdk/queries#table-schema)). + + + + +```ts +// List all table schemas +const { data: schemas } = await wh.schema.list(); +// schemas: { clicks: { name: 'clicks', columns: [...] }, users: { ... } } + +// Force refresh from ClickHouse +await wh.schema.refresh(); + +// One table +const { data } = await wh.from('clicks').schema(); +``` + + + + +```go +// List all table schemas. +schemas, err := wh.Schema.List(ctx) +// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name + +// Force refresh from ClickHouse. +err = wh.Schema.Refresh(ctx) + +// One table. +schema, err := wh.From("clicks").Schema(ctx) +``` + + + + +--- + +## Policy + +Manage Hasura-style access control policies. + + + + +```ts +// Get current policy +const { data: policy } = await wh.policy.get(); + +// Update policy +await wh.policy.set({ + default_role: 'viewer', + admin_role: 'admin', + tables: { + clicks: { + select: { + viewer: { + allow_columns: ['page', 'button', 'received_timestamp'], + filter: { tenant_id: { _eq: '{{ jwt.app_metadata.tenant_id }}' } }, + }, + admin: { allow_columns: ['*'] }, + }, + }, + }, +}); + +// Validate without applying (dry run) +const { data } = await wh.policy.validate(policyDraft); +// data: { valid: true } or error with validation details +``` + + + + +```go +// Get current policy. +policy, err := wh.Policy.Get(ctx) + +// Update policy. +tenantFilter := "{{ jwt.app_metadata.tenant_id }}" +policyDraft := &wavehouse.Policy{ + DefaultRole: "viewer", + Tables: map[string]wavehouse.TablePolicy{ + "clicks": { + Select: map[string]wavehouse.RolePermissions{ + "viewer": { + AllowColumns: []string{"page", "button", "received_timestamp"}, + Filter: map[string]wavehouse.PolicyFilter{ + "tenant_id": {Eq: &tenantFilter}, + }, + }, + "admin": {AllowColumns: []string{"*"}}, + }, + }, + }, +} +err = wh.Policy.Set(ctx, policyDraft) + +// Validate without applying (dry run). +result, err := wh.Policy.Validate(ctx, policyDraft) +// result.Valid == true, or err wraps the validation failure details +``` + +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator — hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. + + + + +--- + +## Dead Letter Queue + +Dead Letter Queue statistics. The DLQ stream method exists in both API trees but is **not yet functional**: there is no server-side DLQ stream today (the SSE bridge only carries `ingest.>` subjects), so it connects and receives no events rather than failing. Live DLQ streaming is tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). + + + + +```ts +// Get DLQ statistics +const { data } = await wh.dlq.list(); +// data: { tables: { "clicks": 3, "users": 0 }, total: 3 } + +// Stats for a specific table +const { data } = await wh.dlq.table('clicks'); +``` + + + + +```go +// Totals across tables. +stats, err := wh.DLQ.List(ctx) +// stats.Tables: map[string]int{"clicks": 3, "users": 0} +// stats.Total: 3 + +// Stats for a specific table. +stats, err = wh.DLQ.Table(ctx, "clicks") +``` + + + + +--- + +## System + +Content-free server-online check — the one surface on this page that needs no credentials. + + + + +```ts +// health() hits the public, content-free /v1/health route — 200/503, no body. +// Use it to check a server is reachable before sending data. +const result = await wh.sys.health(); +if (result.ok) { + // server is up and past boot +} +// on failure, result.error carries the reason (network vs. server error) +``` + + + + +```go +// Health hits the public, content-free /v1/health route — 200 → nil error, +// any other status (including 503) → a non-nil *wavehouse.Error. +// Use it to check a server is reachable before sending data. +if err := wh.Sys.Health(ctx); err != nil { + // server is unreachable or not yet past boot + log.Println(err) +} +``` + + + + +> Readiness (`/readyz`) is intentionally **not** exposed through either SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern, not the client's. Probe `/readyz` directly from your orchestrator if you need it. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go.md similarity index 78% rename from docs/src/content/docs/sdk/go/index.md rename to docs/src/content/docs/sdk/go.md index e1f496e0..db264d2d 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go.md @@ -1,13 +1,9 @@ --- -title: "Go SDK" -description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." +title: "Go SDK setup" +description: "Installing the WaveHouse Go client, creating a client, typed rows via generics, and the (T, error) model." --- -`github.com/Wave-RF/WaveHouse/clients/go` — a Go client for WaveHouse with zero third-party runtime dependencies, SSE parser included. - -:::tip[Looking for the TypeScript SDK?] -`/sdk/go/*` covers the Go client; the JavaScript/TypeScript client (`@wavehouse/sdk`) starts at [SDK Overview](/sdk). Both speak the same wire format, so concepts carry over — only the [API shapes differ](#differences-from-the-typescript-sdk). -::: +Setup and caveats for `github.com/Wave-RF/WaveHouse/clients/go`, the Go client: installation, client construction, typed rows, and the `(T, error)` model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing TypeScript instead? [TypeScript setup](/sdk/typescript) is the mirror of this page. ## Installation @@ -95,7 +91,7 @@ The default client sets no `Timeout`; use a `context.Context` deadline to preven The 2-retry default applies only when `Config.Options` is `nil`. Passing `&wavehouse.ClientOptions{}` leaves `MaxRetries` at Go's zero value (`0`), which disables retries — set it explicitly. ::: -`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk/typescript#custom-headers) — a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): ```go wh := wavehouse.NewClient(wavehouse.Config{ @@ -110,7 +106,7 @@ wh := wavehouse.NewClient(wavehouse.Config{ The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both — supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. :::note[How the token is transmitted] -The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk/typescript#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. ::: :::caution[A credentialed stream will not follow a redirect] @@ -136,11 +132,11 @@ page, err := wavehouse.FetchTyped[ClickRow](ctx, // page.Data is []ClickRow ``` -Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. `FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods; the untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. +Use the [codegen CLI](/sdk/reference#codegen-cli) to generate row structs from a running server. `FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods; the untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. ## Error Handling -Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, or a bare `error` when there is no body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (`Auth` provider, marshal errors) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) report through callbacks or plain errors; see [Streaming](/sdk/go/streaming). +Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, or a bare `error` when there is no body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (`Auth` provider, marshal errors) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) report through callbacks or plain errors; see [Streaming](/sdk/streaming). ```go page, err := wh.From("clicks").Fetch(ctx) @@ -155,24 +151,26 @@ if err != nil { } ``` -See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry behavior and error codes. +See [Reference → Error Handling](/sdk/reference#error-handling) for retry behavior and error codes. ## Differences from the TypeScript SDK Both SDKs share a wire format and feature set, verified in CI against a shared `wire_cases.json` fixture that asserts equivalent HTTP requests for builder calls. The API shapes differ: - **No `Result` union.** Go returns `(T, error)`; a non-nil `error` is the only failure signal. No `{ok, data, error}` objects, no `error: null` sentinels. -- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` first; use a deadline or `cancel()`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). -- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` take no context; the returned `*StreamController` owns its goroutine and connection until `.Close()` (usually deferred). See [Streaming](/sdk/go/streaming). +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` first; use a deadline or `cancel()`. See [Reference → Context Cancellation](/sdk/reference#cancellation). +- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` take no context; the returned `*StreamController` owns its goroutine and connection until `.Close()` (usually deferred). See [Streaming](/sdk/streaming). - **Generics on package functions.** Go has no type parameters on methods, so use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. - **No implicit "await."** `QueryBuilder` is not `PromiseLike`; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly. - **No third-party dependencies.** Stdlib only, SSE frame parser included. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). -- **Any slice batches.** Reflection lets `[]ClickRow{...}` take the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). +- **Any slice batches.** Reflection lets `[]ClickRow{...}` take the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/queries#inserting-rows). + +## Where to go next -## Explore the Go SDK +The topic pages cover both SDKs, tabbed by language — the tab you pick here follows you across all of them. -- [Queries](/sdk/go/queries) — Tables, chainable query builder, pagination, and raw SQL. -- [Streaming & Live Queries](/sdk/go/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. -- [Pipes](/sdk/go/pipes) — Manage named query pipes. -- [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, API tree, and codegen CLI. +- [Queries](/sdk/queries) — Tables, chainable query builder, inserts, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/pipes) — Execute and manage named query pipes. +- [Admin & System](/sdk/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. +- [Reference & CLI](/sdk/reference) — Error codes, context cancellation, API tree, and codegen CLI. diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md deleted file mode 100644 index 6475a070..00000000 --- a/docs/src/content/docs/sdk/go/admin.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: "Go SDK Admin & System" -description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." ---- - -Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. Every namespace here except `client.Sys.Health` sits behind the server's admin gate on `/v1/ops/*`, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). Without one, these calls return a `*wavehouse.Error` with `Status: 403` — unless the deployment sets `default_role` to admin, which is dev-only. See [Access Control](/access-control), and the TypeScript SDK's [Admin & System](/sdk/admin) page. - -## Schema — `client.Schema` - -Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` all hit `/v1/ops/schema*`. - -```go -// List all table schemas. -schemas, err := wh.Schema.List(ctx) -// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name - -// Force refresh from ClickHouse. -err = wh.Schema.Refresh(ctx) - -// One table: wh.From("clicks").Schema(ctx) -``` - ---- - -## Policy — `client.Policy` - -Manage Hasura-style access control policies. - -```go -// Get current policy. -policy, err := wh.Policy.Get(ctx) - -// Update policy. -tenantFilter := "{{ jwt.app_metadata.tenant_id }}" -policyDraft := &wavehouse.Policy{ - DefaultRole: "viewer", - Tables: map[string]wavehouse.TablePolicy{ - "clicks": { - Select: map[string]wavehouse.RolePermissions{ - "viewer": { - AllowColumns: []string{"page", "button", "received_timestamp"}, - Filter: map[string]wavehouse.PolicyFilter{ - "tenant_id": {Eq: &tenantFilter}, - }, - }, - "admin": {AllowColumns: []string{"*"}}, - }, - }, - }, -} -err = wh.Policy.Set(ctx, policyDraft) - -// Validate without applying (dry run). -result, err := wh.Policy.Validate(ctx, policyDraft) -// result.Valid == true, or err wraps the validation failure details -``` - -`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator — hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. - ---- - -## DLQ — `client.DLQ` - -Dead Letter Queue statistics. - -```go -// Totals across tables. -stats, err := wh.DLQ.List(ctx) -// stats.Tables: map[string]int{"clicks": 3, "users": 0} -// stats.Total: 3 - -// Stats for a specific table. -stats, err = wh.DLQ.Table(ctx, "clicks") -``` - -`wh.DLQ.Stream(opts)` is **not yet functional**: no server-side DLQ stream exists (the SSE bridge carries only `ingest.>` subjects), so it connects and receives nothing. Tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). - ---- - -## System — `client.Sys` - -The one surface on this page that needs no credentials. - -```go -// Health hits the public, content-free /v1/health route — 200 → nil error, -// any other status (including 503) → a non-nil *wavehouse.Error. -// Use it to check a server is reachable before sending data. -if err := wh.Sys.Health(ctx); err != nil { - // server is unreachable or not yet past boot - log.Println(err) -} -``` - -> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern. Probe it directly from your orchestrator. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md deleted file mode 100644 index a75f58ae..00000000 --- a/docs/src/content/docs/sdk/go/pipes.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: "Go SDK Pipes" -description: "Execute and manage named query pipes with the WaveHouse Go SDK." ---- - -Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages their definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. - -## Named Pipes — `client.Pipe(name, params)` - -Returns a `*PipeRef` for a pre-defined named query pipe. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you execute it explicitly with `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. Pass `nil` for `params` if the pipe takes none, or only needs its server-side defaults. - -### `wavehouse.Fetch[Row](ctx, pipeRef)` - -Executes the pipe and decodes results into `[]Row`. A package-level generic function, since Go has no generic methods — the same pattern as `FetchTyped` for queries and `SQL` for raw SQL. - -```go -type TopPage struct { - Page string `json:"page"` - Views int `json:"views"` -} - -rows, err := wavehouse.Fetch[TopPage](ctx, - wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), -) -``` - -### `.FetchUntyped(ctx)` - -The non-generic method form: decodes results into `[]map[string]any`. - -```go -rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) -``` - -### `.Stream(opts)` - -Subscribes to live events using the pipe's name as a table name; see [Streaming](/sdk/go/streaming). The pipe's SQL and params are **not** applied — where a table of that name exists you receive its raw events, and otherwise the stream stays silent rather than erroring. This mirrors the TypeScript SDK's `PipeRef.stream()`; both wait on a pipe-aware stream endpoint ([#445](https://github.com/Wave-RF/WaveHouse/issues/445)). - -```go -stream := wh.Pipe("top_pages", nil).Stream(nil) -defer stream.Close() -``` - ---- - -## Pipes Admin — `client.Pipes` - -Create, read, and delete pipe definitions. These sit behind the [admin gate](/sdk/go/admin) on `/v1/ops/*`. - -```go -// List all pipes. -pipes, err := wh.Pipes.List(ctx) - -// Get a single pipe definition. -pipe, err := wh.Pipes.Get(ctx, "top_pages") - -// Create or update. -err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ - SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", - Parameters: []wavehouse.ParamDef{ - {Name: "limit", Type: "number", Required: false, Default: 100}, - }, - Description: "Top pages by view count", - AllowedRoles: []string{"viewer", "admin"}, -}) - -// Delete. -err = wh.Pipes.Delete(ctx, "old_pipe") -``` - -`PipeDef` is `Pipe` minus `Name`, which the methods take as a path argument: - -```go -type PipeDef struct { - SQL string - Parameters []ParamDef - Description string - AllowedRoles []string -} -``` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md deleted file mode 100644 index 282ba25a..00000000 --- a/docs/src/content/docs/sdk/go/queries.md +++ /dev/null @@ -1,284 +0,0 @@ ---- -title: "Go SDK Queries" -description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." ---- - -Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` first and returns `(T, error)`, the chainable builder methods and `.Stream(opts)` excepted — see [Error Handling](/sdk/go#error-handling). The TypeScript SDK covers the same surface on its [Queries](/sdk/queries) page, with a `Result`-returning, `PromiseLike` builder. - -## Tables — `client.From(table)` - -`From` returns a `*TableRef`. It performs no request, so it is safe to store or pass around. - -```go -clicks := wh.From("clicks") -``` - -### `.Fetch(ctx)` - -Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`) — internally `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. There is no options struct as in the TypeScript SDK's `.fetch(opts?)`: to override the limit or paginate, chain `.SelectAll().Limit(n).OrderBy(...)` yourself ([Query Builder](#query-builder)). Access-control policies restrict the returned columns, and `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). - -```go -page, err := clicks.Fetch(ctx) -if err != nil { - log.Fatal(err) -} -for _, row := range page.Data { - fmt.Println(row["page"]) -} -``` - -### `.Insert(ctx, data)` - -Inserts one or many rows based on the input type: - -- **Map or struct** (excluding slices and `[]byte`): sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. -- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): serialized to NDJSON via reflection and sent as one `application/x-ndjson` request, with per-record outcomes in the result. - -```go -// Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) -res, err := clicks.Insert(ctx, map[string]any{"page": "/home", "button": "cta"}) - -// Many rows (map slice) → one NDJSON request, per-record summary -res, err = clicks.Insert(ctx, []map[string]any{ - {"page": "/home", "button": "cta"}, - {"page": "/about", "button": "nav"}, -}) -// res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results - -// Many rows (typed slice) — same NDJSON path, via reflection -type ClickRow struct { - Page string `json:"page"` - Button string `json:"button"` -} -res, err = clicks.Insert(ctx, []ClickRow{ - {Page: "/home", Button: "cta"}, - {Page: "/about", Button: "nav"}, -}) -``` - -For batches, `res.OK` is `true` only if every record succeeded (`*res.Failed == 0`); check `res.Failed` and `res.Results` (each an `InsertRecordResult{Index, OK, Duplicate, Error}` with a 1-based `Index`) for partial failures. The returned `error` signals whole-request failures instead (network, `404`, `403`, `503`), and empty slices are no-ops. The server itself is format-agnostic — `POST /v1/ingest` also accepts a raw JSON array or a single object, since `Content-Type` is only a hint ([API reference](/api#post-v1ingesttabletable--ingest-data)). - -### `.InsertNDJSON(ctx, ndjson)` - -Inserts pre-formatted NDJSON as a `string` without parsing it into Go values. Returns the same summary as slice `Insert`. - -```go -// From a literal string. -res, err := clicks.InsertNDJSON(ctx, `{"page":"/a"}`+"\n"+`{"page":"/b"}`) - -// From a file on disk. -raw, err := os.ReadFile("events.ndjson") -if err != nil { - log.Fatal(err) -} -res, err = clicks.InsertNDJSON(ctx, string(raw)) -``` - -### `.Schema(ctx)` - -Fetch table column definitions from ClickHouse. Admin-only. - -```go -schema, err := clicks.Schema(ctx) -// schema.Name == "clicks" -// schema.Columns: []Column{{Name: "page", Type: "String", IsNullable: false, HasDefault: false}, ...} -``` - -### `.Select(...columns)` - -Starts a query builder chain — see [Query Builder](#query-builder) for the chainable methods and how to execute it. - -### `.SelectAll()` - -The explicit version of `.Fetch()`: selects every column your role may read. Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles the server expands it to the allowed columns rather than a bare `SELECT *`, so it never bypasses `deny_columns`/`allow_columns` ([Access control](/access-control#column-permissions)). - -```go -page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) -``` - -### `.Stream(opts)` - -Opens a real-time event subscription on the table, e.g. `clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"})`. See [Streaming](/sdk/go/streaming). - -## Query Builder - -Returned by `tableRef.Select(...)` or `tableRef.SelectAll()`. Immutable — every chain method returns a new `*QueryBuilder`, leaving the original unchanged. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: - -```go -page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) -``` - -### Chain Methods - -#### `.Select(...columns)` - -Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*` — use `.SelectAll()` for all columns. - -```go -q := clicks.Select("page").Select("button") // SELECT page, button -``` - -#### `.SelectAll()` - -Same expansion rules as [`tableRef.SelectAll()`](#selectall), from an existing builder. - -```go -q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") -``` - -#### `.Where(column, op, value)` - -Add a filter using `FilterOp` constants: - -```go -clicks.Select("page"). - Where("score", wavehouse.OpGt, 10). - Where("page", wavehouse.OpLike, "/home%") -``` - -| `FilterOp` constant | Backend wire token | Description | -|----------------------|---------------------|--------------| -| `wavehouse.OpEq` | `eq` | Equal | -| `wavehouse.OpNeq` | `neq` | Not equal | -| `wavehouse.OpGt` | `gt` | Greater than | -| `wavehouse.OpGte` | `gte` | Greater than or equal | -| `wavehouse.OpLt` | `lt` | Less than | -| `wavehouse.OpLte` | `lte` | Less than or equal | -| `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | -| `wavehouse.OpLike` | `like` | SQL LIKE pattern | -| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | - -#### Aggregations - -```go -clicks.Select("page"). - Count("*", "total"). // COUNT(*) - Sum("score", "total_score"). // SUM(score) - Avg("score", "avg_score"). // AVG(score) - Min("score", "min_score"). // MIN(score) - Max("score", "max_score"). // MAX(score) - CountDistinct("page", "unique_pages"). - Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn -``` - -`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `.Aggregate(fn, column, alias)` runs a custom function, validated server-side (case-insensitively) against the allowlist `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp` — anything else returns `400 unsupported aggregation function`. With an empty alias, `Count` defaults to `count` (and `column=""` becomes `*`), `Sum`/`Avg`/`Min`/`Max` to `sum_`/`avg_`/`min_`/`max_`, and `CountDistinct` to `count_distinct_`; `Aggregate` has no default and sends `""`. - -#### `.GroupBy(...columns)` - -Group the result set, as in `clicks.Select("page").Count("", "").GroupBy("page")`. - -#### `.OrderBy(column, dir)` - -`dir` defaults to `"asc"` if `""`. - -```go -clicks.Select("page").Count("", "total").OrderBy("total", "desc") -``` - -#### `.Limit(n)` - -Caps the row count, as in `clicks.Select().Limit(100)`. Defaults to `wavehouse.DefaultLimit` (1000); the server also enforces a maximum (`query.default_max_rows`, default 10,000). - -#### `.TimeRange(column, since, until)` - -Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for an open-ended range. - -```go -clicks.Select("page").TimeRange("received_timestamp", "1h", "") -clicks.Select("page").TimeRange( - "received_timestamp", "2026-01-01T00:00:00Z", "2026-02-01T00:00:00Z", -) -``` - -#### `.CacheTTL(seconds)` - -Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time ([#280](https://github.com/Wave-RF/WaveHouse/issues/280)). - -```go -clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 -``` - -### `wavehouse.FetchTyped[Row](ctx, q)` - -Executes the query and decodes rows into `[]Row`. - -```go -type PageCount struct { - Page string `json:"page"` - Count int `json:"total"` -} - -page, err := wavehouse.FetchTyped[PageCount](ctx, - clicks.Select("page").Count("*", "total").GroupBy("page"), -) -// page.Data is []PageCount -``` - -### `.FetchUntyped(ctx)` - -Executes the query and decodes rows into `[]map[string]any`. - -```go -page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) -``` - -### `.Stream(opts)` - -Opens a live stream from the builder's table with client-side filtering and projection. See [Streaming](/sdk/go/streaming). - -### Pagination - -Both fetch methods return a `Page[T]`: - -```go -type Page[T any] struct { - Data []T - HasMore bool - Next func(ctx context.Context) (*Page[T], error) // nil when no cursor is available -} -``` - -`HasMore` is `true` when a `Limit` is set and the results meet it. `Next` walks the **first** `.OrderBy()` column by filtering on the last row's value, so it requires an explicit `.OrderBy()` — without one `Next` is `nil`, and if the order column is missing from `.Select(...)` it returns an empty page. The cursor filter is strict (`gt`/`lt`, no tie-breaker), so rows sharing a boundary value with the last row are skipped: paginate on a per-row-unique column or accept dropped ties, a limitation the TypeScript SDK's `next()` shares ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row; `FetchTyped` with an `int64` field, or codegen structs, keep it exact. - -```go -page, err := clicks.Select(). - OrderBy("received_timestamp", "desc"). - Limit(100). - FetchUntyped(ctx) -if err != nil { - log.Fatal(err) -} - -allRows := append([]map[string]any(nil), page.Data...) -for page.HasMore && page.Next != nil { - page, err = page.Next(ctx) - if err != nil { - log.Fatal(err) - } - allRows = append(allRows, page.Data...) -} -``` - -## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` - -Executes a raw SQL query via the admin-only `/v1/ops/query`. A JWT must resolve to the admin role (`admin_role`, default `"admin"`); requests without a valid token fall back to `default_role` and are rejected unless that role is admin (dev-only). An operator key authorizes `/v1/ops/*` as well, but since `Config.Auth` always sends `Bearer `, pass it by giving `Config.HTTPClient` a `Transport` that sets the `X-Operator-Key` header ([API authentication](/api#authentication)). Use `map[string]any` for dynamic schemas. - -```go -rows, err := wavehouse.SQL[map[string]any](ctx, wh, - "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") - -// Or decode into a struct that matches the projected columns/aliases. -// NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit -// integers (count() is UInt64) — decode them with the `,string` tag, or -// use map[string]any. See Reference → Codegen CLI for the full story. -type PageTotal struct { - Page string `json:"page"` - Total uint64 `json:"total,string"` -} -typed, err := wavehouse.SQL[PageTotal](ctx, wh, - "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") -``` - -:::note[No parameter binding through the SDK] -Positional `?` substitution is unsupported, and the SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` has no hook to add them. Use inline literals, or the structured query builder (`wh.From(table)...`) for safe binding of user input. -::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md deleted file mode 100644 index 4bdb794c..00000000 --- a/docs/src/content/docs/sdk/go/reference.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -title: "Go SDK Reference & CLI" -description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." ---- - -Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference). - -## Context Cancellation - -Non-streaming operations take a `context.Context` as their first argument (the analog of TypeScript's `AbortSignal`). Cancel via a timeout or an explicit `cancel()`; cancellation returns immediately, without retrying, as `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. - -```go -ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) -defer cancel() - -page, err := wh.From("clicks").Fetch(ctx) -var whErr *wavehouse.Error -if errors.As(err, &whErr) && whErr.Code == "ABORTED" { - fmt.Println("Request timed out") -} -``` - -`.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming](/sdk/go/streaming#streamoptions). - -## Error Handling - -The SDK never panics on API or network failures, mirroring the TypeScript SDK's "never throws" guarantee. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less ones (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`, or use `wavehouse.IsRetryable(err)` to shortcut the `errors.As` + `.Retryable` check); client-side failures such as an `Auth` provider or marshal error are plain wrapped errors, so handle the `errors.As == false` case — see the [worked example](/sdk/go#error-handling). Streaming methods (`Stream`, `Subscribe`, `Close`) report through the subscriber's `Error` callback instead, and `Connected(ctx)` returns plain errors. - -| Status | Code | Retryable | Description | -|--------|------|-----------|--------------| -| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Invalid or expired JWT (missing tokens use `default_role`, resulting in success or 403) | -| 403 | `HTTP_403` | No | Insufficient permissions | -| 404 | `HTTP_404` | No | Table or pipe not found | -| 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | -| 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | -| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | -| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | -| 0 | `ABORTED` | No | Request canceled via `context.Context` | -| 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | -| 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | -| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | -| *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | -| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | -| 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | -| 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | -| 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | - -Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup to suppress duplicates. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. - -## Full API Tree - -```text -NewClient(Config) → *Client -├── .From(table) → *TableRef -│ ├── .Fetch(ctx) → (*Page[map[string]any], error) -│ ├── .Select(...cols) → *QueryBuilder -│ │ ├── .Select() .SelectAll() .Where() .Count() .Sum() .Avg() .Min() .Max() -│ │ │ .CountDistinct() .Aggregate() .GroupBy() .OrderBy() -│ │ │ .Limit() .TimeRange() .CacheTTL() -│ │ ├── FetchTyped[Row](ctx, q) → (*Page[Row], error) // package-level generic func -│ │ ├── .FetchUntyped(ctx) → (*Page[map[string]any], error) -│ │ ├── .Stream(opts) → *StreamController -│ │ └── .LiveQuery(sub, opts) → *LiveQueryHandle -│ ├── .SelectAll() → *QueryBuilder -│ ├── .Insert(ctx, data) → (*InsertResult, error) -│ ├── .InsertNDJSON(ctx, ndjson) → (*InsertResult, error) -│ ├── .Schema(ctx) → (*TableSchema, error) -│ └── .Stream(opts) → *StreamController -├── .Pipe(name, params) → *PipeRef -│ ├── Fetch[Row](ctx, p) → ([]Row, error) // package-level generic func -│ ├── .FetchUntyped(ctx) → ([]map[string]any, error) -│ └── .Stream(opts) → *StreamController -├── .Pipes (admin) → *PipesNamespace -│ ├── .List(ctx) → ([]Pipe, error) -│ ├── .Get(ctx, name) → (*Pipe, error) -│ ├── .Set(ctx, name, PipeDef) → error -│ └── .Delete(ctx, name) → error -├── SQL[Row](ctx, client, query) → ([]Row, error) // package-level generic func, admin-only -├── .Schema (admin) → *SchemaNamespace -│ ├── .List(ctx) → (Schemas, error) -│ └── .Refresh(ctx) → error -├── .Policy (admin) → *PolicyNamespace -│ ├── .Get(ctx) → (*Policy, error) -│ ├── .Set(ctx, *Policy) → error -│ └── .Validate(ctx, *Policy) → (*ValidationResult, error) -├── .DLQ (admin) → *DLQNamespace -│ ├── .List(ctx) → (*DLQStats, error) -│ ├── .Table(ctx, name) → (*DLQStats, error) -│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 -└── .Sys → *SysNamespace - └── .Health(ctx) → error - -*StreamController -├── .Subscribe(*StreamSubscriber) → func() // unsubscribe -├── .Events() → <-chan StreamEvent // idiomatic Go alternative to an async iterator -├── .Close() -├── .Status() → StreamStatus -└── .Connected(ctx) → error // Go-only addition, blocks until live -``` - -## Codegen CLI - -Generate Go structs from a running WaveHouse instance with the `wavehouse-codegen` command in `cmd/`: - -```bash -export WAVEHOUSE_AUTH='' # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ - --url http://localhost:8080 \ - --out ./db_types.go \ - --package myapp - -# Or, from inside a checkout of clients/go/: -go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go -``` - -Codegen reads the admin-only `/v1/ops/schema` endpoint, so a non-dev server needs an admin token or returns `403`. Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history and process listings. - -**Options:** - -| Flag | Description | Default | -|------|-------------|---------| -| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | -| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | -| `--package`, `-p` | Go package name for the generated file | `main` | -| `--help`, `-h` | Show usage and exit | — | - -**Example output** (for the [development quick-start](/development#quick-start) `clicks` table): - -```go -// Code generated by wavehouse-codegen. DO NOT EDIT. - -package myapp - -// ClicksRow represents a row in the "clicks" table. -type ClicksRow struct { - Page string `json:"page"` - Button string `json:"button"` - Score float64 `json:"score"` - ReceivedTimestamp *string `json:"received_timestamp,omitempty"` -} -``` - -Output is run through `go/format`, and codegen fails loudly if a table or column name would produce invalid Go source. Names become `PascalCase`, with an `X` prefix for a leading digit (`2fa_events` → `X2faEventsRow`); initialisms are not special-cased, so `event_id` becomes `EventId`, not `EventID`. Columns with `has_default: true` become pointer fields with `,omitempty` — as `received_timestamp` does above — where `nil` uses the server default and a pointed-at value is sent, including an explicit `0`/`false`/`""`. - -**ClickHouse → Go type mapping:** - -| ClickHouse Type | Go Type | -|------------------|---------| -| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | -| `Bool` / `Boolean` | `bool` | -| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | -| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | -| `Float32`, `BFloat16` | `float32` | -| `Float64` | `float64` | -| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` | -| `Decimal*` | `string` | -| `Nullable(T)` | `*T` | -| `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` (except `Array(UInt8)` → `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | -| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | -| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | -| anything unrecognized | `any` | - -Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`; for the raw-SQL path (`/v1/ops/query`), which quotes 64-bit-and-wider integers, use `map[string]any` with `SQL[Row]`. - -## Testing - -Unit tests are colocated in `clients/go/`, which is its own module (`clients/go/go.mod`), separate from the root `WaveHouse` module: - -```bash -cd clients/go -go test ./... -``` - -The cross-language wire-format **conformance suite** replays a shared fixture (`clients/go/testdata/wire_cases.json`) from `clients/go/conformance_test.go`, asserting HTTP methods, paths, content types, and bodies. The TypeScript half — `tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` — replays the same fixture, and CI runs both to keep the wire formats in step. - -E2E tests (build tag `e2e`) run against a live WaveHouse instance via their own Make target: - -```bash -WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e -``` - -`WAVEHOUSE_URL` defaults to `http://localhost:8080`, and the optional `WAVEHOUSE_AUTH` covers the admin cases; the suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md deleted file mode 100644 index 1a81a3b3..00000000 --- a/docs/src/content/docs/sdk/go/streaming.md +++ /dev/null @@ -1,217 +0,0 @@ ---- -title: "Go SDK Streaming & Live Queries" -description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." ---- - -Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Frames are parsed over `net/http` with no runtime dependencies. Builders and table refs come from [Queries](/sdk/go/queries). The TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) implements the same protocol and mostly the same client-side filtering, but the lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. - -## Streaming - -### `*StreamController` - -Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). `.Stream` returns immediately; the connection opens in a background goroutine. - -```go -stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ - Since: "2026-01-01T00:00:00Z", -}) -defer stream.Close() -``` - -### `.Subscribe(sub) → func()` - -Callback-based consumption. Returns an unsubscribe function. The `Status` callback fires immediately with the current status. - -```go -unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ - Next: func(e wavehouse.StreamEvent) { - // e: {Table: "clicks", Timestamp: "2026-...", Data: map[string]any{"page": "/", ...}} - fmt.Println("New event:", e.Data) - }, - Status: func(s wavehouse.StreamStatus) { - // s: StatusConnecting | StatusLive | StatusReconnecting | StatusClosed - updateIndicator(s) - }, - Error: func(err error) { - fmt.Println("Stream error:", err) - }, -}) - -// Removes this subscriber only — the connection stays open for any others -// and still needs stream.Close() when you're done with the stream itself. -defer unsub() -``` - -### Channel-based consumption — `.Events()` - -A read-only channel, closed automatically when the stream shuts down. It is buffered (256 events) and buffers from `.Stream()` onward, so events arriving before the first `Events()` call are not lost. A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop; the first drop logs via `log`, later drops are silent, and `.Subscribe` callbacks fire regardless. - -```go -stream := wh.From("clicks").Stream(nil) -defer stream.Close() - -for event := range stream.Events() { - fmt.Println(event.Table, event.Data) - if shouldStop { - break - } -} -``` - -:::caution[`break` does not close the stream] -Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. -::: - -:::note[`Events()` carries events only] -`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404), so pair `Events()` with a subscriber to learn why a stream ended. -::: - -### `.Close()` - -`stream.Close()` closes the stream and releases its resources. Non-blocking, and safe to call from inside a subscriber callback. - -### `.Status()` - -`stream.Status()` returns the current `StreamStatus`: `StatusConnecting`, `StatusLive`, `StatusReconnecting`, or `StatusClosed`. - -### `.Connected(ctx)` - -Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (in tests, for instance). - -```go -ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) -defer cancel() -if err := stream.Connected(ctx); err != nil { - log.Fatal(err) -} -``` - -### `StreamOptions` - -| Field | Type | Description | -| ----- | ---- | ----------- | -| `Since` | `string` | RFC3339 timestamp for gap-fill replay | - -There is no `Signal`/context field: a stream is not canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. - -### `StreamEvent` - -```go -type StreamEvent struct { - Table string // table name (e.g. "clicks") - Timestamp string // received_timestamp (RFC3339Nano) - Data map[string]any // row data -} -``` - -Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value, because the ingest handler rewrites them before publishing — a live frame and a later query can't disagree on the spelling of an instant. So a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open: a value the server can't parse, or whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). - -### Transport Behavior - -SSE reconnects automatically with exponential backoff (capped at 30s) and gap-fill replay from the last event ID; HTTP/2 is recommended. Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect — the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). - -`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). - -Delivery across a reconnect is **at-least-once**: the server replays from the last event ID *inclusively*, so the first frame after a gap-fill is usually one you already saw. Replay reaches back only as far as the server's `mq.gap_window_minutes` (15 minutes by default); a longer outage resumes live with a hole. - -### Server-Side Policy Filtering - -Before anything reaches the client, the server applies the caller's policy — claims captured from the JWT at connect time — to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber, on live frames and `Since` gap-fill replay alike. - -Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control](/access-control#row-level-security). - -### Client-Side Stream Filtering - -When a `*QueryBuilder` with `.Where()` or `.Select()` calls `.Stream()`, filters are applied client-side: - -```go -stream := wh.From("clicks"). - Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Stream(nil) - -// Only events where page == "/home" are emitted, with only page + button fields -``` - -Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere. `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively, and `OpIn` accepts any Go slice type (e.g. `[]string`, `[]int`). - -#### How values are compared - -The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: - -- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload may read `2026-06-21T04:00:00Z` while your filter constant names the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a chronologically equal row. Both sides are parsed as instants instead. -- **Only unambiguous spellings count as instants:** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have, so it is not treated as a timestamp. -- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. -- **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. -- **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. - -:::caution[Integer precision above 2^53] -Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond 2^53 has already lost exactness before any filter runs, and the server compares such columns in their exact storage domain — so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. -::: - -## Live Queries - -Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream, for seamless initial loads and updates. They exist only on `*QueryBuilder` (there is no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. - -```go -lq := wh.From("clicks"). - SelectAll(). - Where("page", wavehouse.OpEq, "/home"). - OrderBy("received_timestamp", "desc"). - Limit(100). - LiveQuery(&wavehouse.StreamSubscriber{ - Initial: func(rows []map[string]any, err error) { - // Called once with the historical backfill. - setRows(rows) - }, - Next: func(e wavehouse.StreamEvent) { - // Called for each live event after backfill. - addRow(e.Data) - }, - Error: func(err error) { - log.Println(err) - }, - }, nil) - -defer lq.Close() -``` - -### `StreamSubscriber` - -```go -type StreamSubscriber struct { - // Initial is called once with historical backfill data (live queries only). - Initial func(rows []map[string]any, err error) - // Next is called for each live event. - Next func(event StreamEvent) - // Status is called when the connection status changes. - Status func(status StreamStatus) - // Error is called on stream errors. - Error func(err error) -} -``` - -:::note[`Initial` is always untyped] -Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even where you would use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. -::: - -### How it works - -1. Subscribes to the stream immediately and buffers events. -2. Runs `.FetchUntyped(ctx)` for historical data, then calls `sub.Initial(rows, err)`. -3. Deduplicates buffered events against the maximum `received_timestamp` in the backfill (not necessarily the last row). -4. Flushes remaining buffered events and switches to live mode. - -Subscribing first is what prevents event loss between fetch and stream start. - -:::caution[Dedup needs `received_timestamp` in the projection] -Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, so events in the overlap window are delivered twice — once via `Initial`, once via `Next`. -::: - -:::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback — see [Queries](/sdk/go/queries#wherecolumn-op-value). -::: - -### `.Close()` - -`lq.Close()` shuts down the live query and its underlying stream. Idempotent (guarded by `sync.Once`). diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index b6030214..0e3a0a2e 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -1,542 +1,105 @@ --- -title: "TypeScript SDK" -description: "Client SDK — query builder, real-time streaming, codegen." +title: "Client SDKs" +description: "Official WaveHouse clients for TypeScript and Go — query builder, real-time streaming, codegen." --- import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components"; -`@wavehouse/sdk` — TypeScript client for WaveHouse. One runtime dependency: `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. +WaveHouse ships two officially supported clients with full API-tree parity: **TypeScript** (`@wavehouse/sdk`) and **Go** (`github.com/Wave-RF/WaveHouse/clients/go`). Both give you a typed query builder, real-time SSE streaming over header-authenticated HTTP, live queries that backfill history before going live, and a codegen CLI that turns your ClickHouse schema into row types. -:::tip[Writing Go instead?] -WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). Both clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually; the [API shapes differ](/sdk/go#differences-from-the-typescript-sdk). -::: +They speak the same wire format — a shared fixture replayed by both test suites keeps it that way — so the topic pages below cover both languages side by side. Pick a language once and the tabs follow you across every page. -## Installation +## Install - - - -```bash -pnpm add @wavehouse/sdk -``` - - - + + ```bash npm install @wavehouse/sdk ``` - - - -```bash -yarn add @wavehouse/sdk -``` - - - - -```bash -bun add @wavehouse/sdk -``` +Other package managers, the CDN build, and runtime support: [TypeScript setup](/sdk/typescript#installation). - + ```bash -deno add npm:@wavehouse/sdk +go get github.com/Wave-RF/WaveHouse/clients/go ``` - - - No bundler or package manager required — the ES module loads natively in modern browsers: - -```html - -``` - -Pin a version for production (`https://esm.sh/@wavehouse/sdk@0.1.0`); jsDelivr (`.../+esm`) and unpkg (`?module`) serve the same module. For pages that can't use ES modules, the bundled IIFE build at `https://cdn.jsdelivr.net/npm/@wavehouse/sdk` exposes a `WaveHouse` global (`WaveHouse.createClient({ … })`) for a classic ` - -``` - -### Runtime support - -**Browsers** — all SDK features work natively. `fetch`, `AbortController`, and `ReadableStream` are built into every modern browser; no polyfills are required. - -**Node.js** — every feature, streaming included, works in Node 22 and later (the package's minimum, per `engines.node`). 22 is the only line we test against, and older releases are past end-of-life upstream. Streaming runs on the same `fetch` and `ReadableStream` globals as the rest of the SDK, so it needs **no `EventSource` polyfill** — earlier versions did, and that requirement is gone. - -## Quick Start - - - - -```ts -import { createClient } from '@wavehouse/sdk'; - -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); - -// Query -const { data, error } = await wh.from('clicks').select('page').limit(10); - -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); - -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); -``` - - - - -```ts -import { createClient } from '@wavehouse/sdk'; - -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); - -// Query const { data, error } = await wh.from('clicks').select('page').limit(10); - -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); - -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); +if (error) throw new Error(error.message); +console.log(data); ``` - - -```ts -import { createClient } from '@wavehouse/sdk'; - -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); + -// Query -const { data, error } = await wh.from('clicks').select('page').limit(10); - -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); +```go +package main -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); -``` +import ( + "context" + "fmt" + "log" - - - -```ts -import { createClient } from '@wavehouse/sdk'; + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); - -// Query -const { data, error } = await wh.from('clicks').select('page').limit(10); - -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); - -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); -``` - - - - -```ts -import { createClient } from '@wavehouse/sdk'; - -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); - -// Query -const { data, error } = await wh.from('clicks').select('page').limit(10); +func main() { + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "https://.wavehouse.app", + }) -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); - -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); -``` - - - - -```ts -import { createClient } from 'https://esm.sh/@wavehouse/sdk'; - -const wh = createClient({ - baseURL: 'http://localhost:8080', - auth: async () => getAccessToken(), // omit for public/unauthenticated -}); - -// Query -const { data, error } = await wh.from('clicks').select('page').limit(10); - -// Insert -await wh.from('clicks').insert({ page: '/home', button: 'signup' }); - -// Stream -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ - next: (event) => console.log(event.data), - status: (s) => console.log('Stream:', s), -}); + page, err := wh.From("clicks").Select("page").Limit(10).FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } + fmt.Println(page.Data) +} ``` -## Creating a Client - -```ts -import { createClient } from '@wavehouse/sdk'; -import type { Database } from './my-types'; // optional hand-written types - -const wh = createClient({ - baseURL: 'https://.wavehouse.app', - auth: async () => myAuthProvider.getToken(), - options: { - maxRetries: 2, - }, -}); -``` - -### `ClientConfig` - -| Field | Type | Default | Description | -|-------|------|---------|-------------| -| `baseURL` | `string` | — | WaveHouse server URL, optionally including a path prefix (required) | -| `auth` | `() => Promise \| string` | — | Token provider. Omit for public access | -| `options.maxRetries` | `number` | `2` | Retry attempts for failed/5xx **REST** requests; stream reconnects are unbounded ([details](/sdk/streaming#transport-behavior)) | -| `options.headers` | `Record` | — | Headers added to every request ([details](#custom-headers)) | -| `options.fetchOptions` | `RequestInit` | — | Extra `RequestInit` fields merged into every request ([details](#extra-requestinit-fields)) | -| `options.fetch` | `FetchLike` | global `fetch` | HTTP implementation for every request ([details](#supplying-your-own-fetch)) | - -:::note[How the token is transmitted] -The SDK attaches your `auth` token as an `Authorization: Bearer` header on every request, streaming included. It is re-read on each connection attempt, so a stream that outlives its token reconnects with a fresh one instead of replaying an expired credential. - -The token is never placed in the URL. The server still accepts a `?token=` query parameter for clients that cannot set headers — see [API Reference — Authentication](/api#authentication) — but the SDK does not use it, so it never reaches a proxy or CDN access log. -::: - -:::caution[Streaming asks more of a custom `fetch`] -`.stream()` and `.liveQuery()` read the response as it arrives, so an `options.fetch` used with them must return a response with a live streaming body. A response handed back with an absent or already-consumed body fails fast with `SSE_NO_STREAM_BODY`. What the SDK cannot rescue is a wrapper that *awaits* the body before returning — `await res.text()`, or the `res.clone().text()` a logging wrapper reaches for — because on a stream that never ends it never resolves and your function never returns. Both satisfy every REST call, which is what makes the trap easy to walk into. - -`options.headers` is also subject to CORS in a browser: a header outside the safelist joins the preflight, which the origin has to allow. WaveHouse advertises a fixed set, so custom headers reach it cross-origin only when a proxy in front terminates the preflight — the deployment they exist for. Server-side callers never preflight. -::: - -#### Serving under a path prefix - -`baseURL` may carry a path prefix, for a WaveHouse reachable somewhere other than the root of an origin — behind a backend-for-frontend (BFF), an app-server route, or a path-routed ingress. Every request path is appended to it, on both transports: - -```ts -const wh = createClient({ baseURL: 'https://app.example.com/api/wavehouse' }); -// queries → https://app.example.com/api/wavehouse/v1/query -// streams → https://app.example.com/api/wavehouse/v1/stream -``` - -Trailing slashes on `baseURL` are optional, but it must be **absolute** — scheme and host included — and the scheme must be `http` or `https`. A same-origin relative path like `/api/wavehouse` makes every REST call reject with a `TypeError` instead of returning a `Result`, and surfaces on a stream as an `SSE_CONNECT_ERROR` to the subscriber's `error` callback — so build an absolute one: `` `${location.origin}/api/wavehouse` ``. A `ws://`/`wss://` base ends the stream with the same error: streaming is plain HTTP, and there is no WebSocket endpoint. - -The proxy in front must **strip** the prefix before forwarding, since WaveHouse itself always serves at `/v1/…` — see [Behind a reverse proxy → Path prefixes](/reverse-proxy#path-prefixes). - -### Custom headers - -`options.headers` adds headers to every request, streaming included — the usual reason being a header-gated proxy in front of WaveHouse, such as a Cloudflare Access service token: - -```ts -const wh = createClient({ - baseURL: 'https://.wavehouse.app', - options: { - headers: { - 'CF-Access-Client-Id': process.env.CF_ACCESS_CLIENT_ID!, - 'CF-Access-Client-Secret': process.env.CF_ACCESS_CLIENT_SECRET!, - }, - }, -}); -``` - -Header names are matched **case-insensitively**, as HTTP requires, so `authorization` and `Authorization` are the same header rather than two. - -Your headers are applied *underneath* the SDK's own, and a collision means yours is dropped rather than merged: - -- **`auth` keeps `Authorization`.** Setting it here won't displace your token provider. Use `auth` for credentials. -- **`Content-Type` and `Accept` belong to the request.** The SDK knows what it's sending; a global `Content-Type` that outranked it would break requests whose body isn't JSON. - -Nothing is ever comma-joined: on a collision the SDK's value stands alone, and two of your own entries differing only in case collapse to the last one — a header joined rather than replaced is how you end up sending `Content-Type: application/json, image/png`. - -From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`) with no config knob. So custom headers work server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. - -Headers are static. For a credential that rotates per request, wrap the transport with [`options.fetch`](#supplying-your-own-fetch); a callback form is tracked in [#459](https://github.com/Wave-RF/WaveHouse/issues/459). - -### Extra `RequestInit` fields - -`options.fetchOptions` is merged into the `RequestInit` of every request, streaming included — for settings that aren't headers and don't warrant replacing the whole transport: - -```ts -const wh = createClient({ - baseURL: 'https://wavehouse.example.com', - options: { fetchOptions: { cache: 'no-store', mode: 'cors' } }, -}); -``` - -It carries any `RequestInit` field — `mode`, `cache`, `keepalive`, `credentials`, `redirect` — plus runtime-specific extensions such as Next.js's `next: { tags }`, which tags the cached response for on-demand revalidation. Only those runtime extensions need a cast; the standard fields are declared on `RequestInit` already. - -:::caution[Two of those fields don't apply to streams] -On `.stream()` and `.liveQuery()`'s live connection the transport keeps `cache` and `redirect` for itself, because there they're correctness rather than preference. It constrains one more without owning it: `credentials` stays yours, forwarded in a browser and dropped elsewhere because some runtimes throw if it is set at all. See [Supplying your own fetch](#supplying-your-own-fetch) for what each is set to and why. - -Everything else reaches streams normally — including `mode` from the example above. Only `cache` is different there: on a stream it is already `no-store`. -::: - -:::note[`credentials: 'include'` needs a proxy that owns CORS] -Sending cookies cross-origin is a common reason to reach for this, but it can't work against WaveHouse's own CORS: it deliberately never emits `Access-Control-Allow-Credentials` (it's a Bearer-token API), and the default `cors_allowed_origins: "*"` makes `include` a hard browser failure regardless. It applies where a fronting proxy answers CORS itself — see [Behind a reverse proxy](/reverse-proxy#header-and-auth-forwarding). Same-origin deployments already send cookies without it. -::: - -The fields the SDK controls — `method`, `headers`, `body`, and `signal` — always win, so this can't corrupt the request itself. In particular `headers` here is ignored; use `options.headers`, which merges properly. - -### Supplying your own `fetch` - -`options.fetch` replaces the HTTP implementation the SDK uses. Reach for it to route through a proxy, attach client certificates, wrap requests in your own middleware (logging, tracing, circuit breaking), stub HTTP in your own tests without monkey-patching a global, or to work around transport behavior of the runtime you happen to be on: - -```ts -const wh = createClient({ - baseURL: 'https://.wavehouse.app', - options: { - fetch: async (url, init) => { - const started = performance.now(); - const res = await fetch(url, init); - console.log(res.status, url, `${Math.round(performance.now() - started)}ms`); - return res; - }, - }, -}); -``` - -Retries go through the same function, so middleware sees every attempt. - -The SDK always calls your function with a **string** URL and a plain `RequestInit`. Off the response the **REST path** reads `.ok` and `.headers` always, `.text()` on success, and `.status`, `.statusText` plus `.json()` when the response is not `ok` — so a hand-rolled response object needs all six there. A stub used with `.stream()` or `.liveQuery()` needs a different set: `.ok`, `.status`, `.type` (checked for `opaqueredirect`), `.headers` carrying `content-type: text/event-stream`, and a live, unread `.body` — plus `.statusText` and `.json()` if your stub ever answers a stream with a non-`ok` response, which goes through the same error parsing as REST. `.text()` is never called on a stream. - -**On REST**, if your function rejects, the request becomes a `NETWORK_ERROR` result and is retried with backoff — unless the `AbortSignal` you passed has been aborted, in which case the result is `ABORTED` and nothing is retried. That's decided from the signal rather than from what your function threw, so an implementation that signals abort some other way — `AbortSignal.timeout()` raises a `TimeoutError`, `node-fetch` its own `AbortError` class — is reported the same way. On a stream the same rejection is reported to the subscriber as `SSE_NETWORK_ERROR` and re-dialed — including an `AbortError`, since the stream transport applies the same rule: only an abort *it* raised, via `.close()`, ends the stream. See [Error Handling](/sdk/reference#error-handling). - -The option's type is exported as `FetchLike`. It is written out rather than as `typeof fetch`, which resolves differently depending on whether your TypeScript `lib` includes DOM. Its URL parameter is `string` — all the SDK ever passes — which, because function parameters are contravariant, accepts *more* implementations than the wider spelling would: the global `fetch` fits, and so does hand-written `(url: string, init?: RequestInit) => Promise` middleware. Let TypeScript infer your parameters (`(url, init) => …`) and it fits without annotation. - -Streaming goes through your function too. That comes with the extra requirement described above — the response must carry a live streaming body — and the streaming transport keeps ownership of two `RequestInit` fields where the value is load-bearing rather than a preference: `cache` (as an init field; a `Cache-Control` header would fail cross-origin preflight) and `redirect`. It constrains one more without owning it: `credentials` stays yours, forwarded in a browser and dropped elsewhere, because some runtimes throw if it is set at all. - -`redirect` depends on whether the request carries a credential. With one — an `auth` token or configured `headers` — a redirect is refused and reported as `SSE_REDIRECT`, because a cross-origin hop strips `Authorization` (which this endpoint answers with a silently reduced view rather than an error) while forwarding your other headers to wherever the redirect points. Without either, redirects are followed normally, so CDN canonicalization, geo/LB indirection, and an http→https upgrade all work. The test is that concrete pair rather than "is this request authenticated", and **cookies are not part of it**. A cookie isn't stripped the way `Authorization` is — it is re-derived from the cookie store at each hop, so a redirect carries it only while the hop stays inside **both** the request's origin and the cookie's own `Path`. Leaving either loses it. The origin half is the one no cookie attribute can override: under the default `same-origin` credentials mode the browser attaches nothing once a redirect goes cross-origin, whatever the `Domain` — so `app.example.com` → `api.example.com` on a shared `Domain=example.com` session cookie loses it, and so does an http→https upgrade, which is cross-origin by scheme and port. A same-origin rewrite from `/v1/stream` to `/stream` under a `Path=/v1` cookie loses it the other way. This endpoint answers a cookieless request with a reduced view rather than an error ([#478](https://github.com/Wave-RF/WaveHouse/issues/478)). Going cross-origin with `credentials: "include"` usually fails earlier and louder instead: WaveHouse never sends `Access-Control-Allow-Credentials`, so against it the browser blocks the response outright. Behind a proxy that owns CORS and does allow credentials the hop succeeds — and then ordinary cookie scoping decides. `include` *replaces* the origin rule rather than lifting it: the cookie crosses only if its own `Domain` covers the target, so a host-only cookie — what you get with no `Domain` attribute — still stops at its host. If you need to follow a credentialed redirect anyway, supply an `options.fetch` that overrides `redirect`. A `200` that isn't `text/event-stream` is refused too (`SSE_BAD_CONTENT_TYPE`), so an auth gateway's login page fails loudly instead of leaving the stream connected and permanently silent. - -#### Swapping in undici - -The motivating case is a transport bug in the runtime's own HTTP stack, which you can't fix from inside the SDK — for example [undici #5600](https://github.com/nodejs/undici/issues/5600): reusing a keep-alive socket while the event loop is idle stalls the request before it goes out. How bad it gets varies with the runtime and the idle gap: the upstream report measured ~450–465 ms against a 10 ms server, and we have measured anything from ~100 ms to tens of seconds on a server answering instantly. It affects undici 8.8.0–8.9.0, and Node 26 bundles 8.9.0. - -**Upgrading undici is the actual fix** — it landed in 8.10.0. What `options.fetch` buys you is a way to get there without waiting for a new runtime: install undici yourself and route requests through it. The one non-obvious part is that you must pass its dispatcher **explicitly**. - -```ts -import { createClient } from '@wavehouse/sdk'; -import { Agent, fetch as undiciFetch } from 'undici'; // npm install undici — 8.10.0+ - -// Explicit, not implied — see below. -const dispatcher = new Agent(); - -const wh = createClient({ - baseURL: 'https://.wavehouse.app', - options: { - fetch: (url, init) => - undiciFetch(url, { ...init, dispatcher } as never) as unknown as Promise, - }, -}); -``` - -:::caution[Importing a fixed undici is not enough on its own] -undici keeps its connection pool on a shared `globalThis` symbol, and whichever copy loads first claims it. Node claims it for the *bundled* copy the first time anything touches one of its web globals — a `fetch()` call, but equally a `new Headers()` or `new Response()` — not at startup. So which undici owns the pool comes down to a load order you don't really control, and auditing your own code for `fetch` calls won't tell you: any dependency can claim it first, and so can this SDK, which constructs a `Headers` on its abort and retry-exhausted paths. Call `undiciFetch` without a `dispatcher` and it resolves whatever is on that symbol, so requests can go through the buggy pool even though you imported a fixed undici. - -Measured with 8.9.0 loaded first and 8.10.0's `fetch` doing the request, 1.5 s idle gaps against a 10 ms server: - -```text - per-request ms -no explicit dispatcher 21 1514 1495 583 ← still stalling -explicit new Agent() 17 14 12 13 -``` - -An explicit `dispatcher` wins because the request never consults the shared symbol at all. An explicit `setGlobalDispatcher` call (below) wins the other way round — it overwrites the symbol after the first copy claimed it. -::: - -Both casts are load-bearing, for one underlying reason: undici declares its own request/response types, separate from the ones behind your global `fetch`, so the two aren't structurally assignable. `{ ...init, dispatcher } as never` covers the split on `RequestInit`, which differs on `body` and `headers` (`never` is assignable to either spelling, so one snippet compiles whether or not your `lib` includes DOM); and the return cast handles the `Response` mismatch. The URL needs no cast — `FetchLike` declares it as `string`, which undici's `RequestInfo` accepts. They're safe because of the narrow runtime contract above — a string URL and a plain `RequestInit` in, and only the response members enumerated above read back. `undici` is a dependency you add on top of the SDK's single ~1.4 KB one. - -If you're genuinely pinned to an affected undici, the same snippet fixes it — your pinned version, with a dispatcher that never reuses a keep-alive socket: `new Agent({ pipelining: 0 })`. That costs a fresh connection per request, so prefer upgrading. Note that tuning `keepAliveTimeout` does **not** help: the retirement timer is starved by the same idle event loop that triggers the bug, so the socket is still there to be reused. - -To change the transport process-wide instead of per-client, an installed undici's `setGlobalDispatcher(new Agent())` is picked up by Node's built-in global `fetch`, no `options.fetch` needed. That's an explicit write, so it wins the race described above — but only if the *installed* undici is 8.10.0+, since it hands connection handling to that copy. - -### Type-Safe Tables - -Pass a `Database` type to get autocomplete on table names and row types: +## Setup & caveats, per language -```ts -interface Database { - clicks: { page: string; button: string; score: number; received_timestamp: string }; - users: { id: string; name: string; email: string }; -} - -const wh = createClient({ baseURL: '...' }); -const clicks = wh.from('clicks'); // ✅ autocomplete -const { data } = await clicks.select('page', 'button').limit(10); -// data is Array<{ page: string; button: string; score: number; received_timestamp: string }> | null -``` - -Generate the `Database` interface from a running server with the [codegen CLI](/sdk/reference#codegen-cli). - -## Result Type - -Every async SDK operation returns `Result` — a discriminated union that never throws for anything the server returns (caller and environment errors do throw — see [Error Handling](/sdk/reference#error-handling)): +Installation, client construction, auth, custom headers, and the error model are language-specific — each SDK has its own page for them. -```ts -type Result = - | { ok: true; data: T; error: null; hasMore?: boolean; next?: () => Promise> } - | { ok: false; data: null; error: WaveHouseError } - -interface WaveHouseError { - status: number; // HTTP status (0 for network errors) - code: string; // e.g. 'HTTP_400', 'NETWORK_ERROR', 'ABORTED' - message: string; // Human-readable error message - details?: unknown; // Raw response body - retryable: boolean; // Whether SDK would retry this error -} -``` - -Usage pattern — branch on `result.ok` (or destructure `{ data, error }` if you prefer): - -```ts -const result = await wh.from('clicks').select('page').limit(10); -if (result.ok) { - console.log(result.data); // Row[] — TypeScript knows data is non-null here -} else { - console.error(result.error.message); // never throws -} -``` - -The full error-code table lives in [Error Handling](/sdk/reference#error-handling). + + + + -## Explore the SDK +## Topics, both languages - diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md deleted file mode 100644 index cd1fad9e..00000000 --- a/docs/src/content/docs/sdk/pipes.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -title: "TypeScript SDK Pipes" -description: "Execute and manage named query pipes with @wavehouse/sdk." ---- - -Named pipes are server-defined, parameterized queries (see [Named Pipes guide](/pipes)). The SDK executes pipes for allowed roles and manages their definitions under the admin role. Examples import from `@wavehouse/sdk` or `https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). - -## Named Pipes — `wh.pipe(name, params?)` - -Execute a pre-defined named query pipe. Returns a `PipeRef` which is **PromiseLike**. - -```ts -// These are equivalent (PipeRef is PromiseLike): -const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }).fetch(); -const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }); -``` - -### `.fetch(opts?)` - -Execute and return results. Takes `PipeRequestOptions` — `{ signal }` only, narrower than the `.fetch(opts?)` on a [query builder](/sdk/queries), which also accepts `limit`. Passing a `limit` is a compile error rather than a silent no-op. - -`limit` is typed `never` rather than left out, so the rejection also catches a value passed in a variable — leaving it out would only reject an inline object. That cuts both ways: a value *declared* as `RequestOptions` is rejected whether or not it actually carries a limit, since the type permits one. If you share one options object across calls, type it as `PipeRequestOptions` — the table and query-builder `.fetch()` accept that too — or inline `{ signal }` at the pipe call. - -There is no per-call row cap here: the endpoint binds your `params` as the pipe's parameters, so a limit has to be declared in the pipe's SQL as `{{limit}}` (see [Named Pipes](/pipes)) and passed as `wh.pipe(name, { limit })`, as in the example above. - -### `.stream(opts?)` - -Open a live stream (see [Streaming](/sdk/streaming)). - ---- - -## Pipes Admin — `wh.pipes` - -Manage named query pipes. Requires the admin gate — the admin role (`policy.admin_role`) or the [operator key](/api#authentication). - -```ts -// List all pipes -const { data: pipes } = await wh.pipes.list(); - -// Get a single pipe definition -const { data: pipe } = await wh.pipes.get('top_pages'); - -// Create or update -await wh.pipes.set('top_pages', { - sql: 'SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}', - parameters: [{ name: 'limit', type: 'number', required: false, default: 100 }], - description: 'Top pages by view count', - allowed_roles: ['viewer', 'admin'], -}); - -// Delete -await wh.pipes.delete('old_pipe'); -``` diff --git a/docs/src/content/docs/sdk/pipes.mdx b/docs/src/content/docs/sdk/pipes.mdx new file mode 100644 index 00000000..305666b3 --- /dev/null +++ b/docs/src/content/docs/sdk/pipes.mdx @@ -0,0 +1,140 @@ +--- +title: "SDK Pipes" +description: "Execute and manage named query pipes with the WaveHouse SDKs." +--- + +import { Tabs, TabItem } from "@astrojs/starlight/components"; + +Named pipes are server-defined, parameterized queries (see [Named Pipes guide](/pipes)). Both SDKs execute pipes for allowed roles and manage their definitions under the admin role. + +## Executing a pipe + +Reference a pre-defined named query pipe, binding its parameters. + + + + +`wh.pipe(name, params?)` returns a `PipeRef`, which is **PromiseLike**. + +```ts +// These are equivalent (PipeRef is PromiseLike): +const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }).fetch(); +const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }); +``` + +`.fetch(opts?)` takes `PipeRequestOptions` — `{ signal }` only, narrower than the `.fetch(opts?)` on a [query builder](/sdk/queries#executing-the-query), which also accepts `limit`. Passing a `limit` is a compile error rather than a silent no-op. + +`limit` is typed `never` rather than left out, so the rejection also catches a value passed in a variable — leaving it out would only reject an inline object. That cuts both ways: a value *declared* as `RequestOptions` is rejected whether or not it actually carries a limit, since the type permits one. If you share one options object across calls, type it as `PipeRequestOptions` — the table and query-builder `.fetch()` accept that too — or inline `{ signal }` at the pipe call. + + + + +`client.Pipe(name, params)` returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you execute it explicitly. Pass `nil` for `params` if the pipe takes none, or only needs its server-side defaults. + +`wavehouse.Fetch[Row](ctx, pipeRef)` decodes results into `[]Row` — a package-level generic function, since Go has no generic methods (the same pattern as `FetchTyped` for queries and `SQL` for raw SQL). `.FetchUntyped(ctx)` is the non-generic method form, decoding into `[]map[string]any`. + +```go +type TopPage struct { + Page string `json:"page"` + Views int `json:"views"` +} + +rows, err := wavehouse.Fetch[TopPage](ctx, + wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), +) + +untyped, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) +``` + + + + +There is no per-call row cap here: the endpoint binds your params as the pipe's parameters, so a limit has to be declared in the pipe's SQL as `{{limit}}` (see [Named Pipes](/pipes)) and passed as a parameter, as in the examples above. + +## Streaming a pipe + +Open a live stream from a pipe ref (see [Streaming](/sdk/streaming)). The pipe's SQL and params are **not** applied — the pipe's *name* is used as a table name, so where a table of that name exists you receive its raw events, and otherwise the stream stays silent rather than erroring. Both SDKs behave this way; both wait on a pipe-aware stream endpoint ([#445](https://github.com/Wave-RF/WaveHouse/issues/445)). + + + + +```ts +const stream = wh.pipe('top_pages').stream(); +``` + + + + +```go +stream := wh.Pipe("top_pages", nil).Stream(nil) +defer stream.Close() +``` + + + + +--- + +## Managing pipe definitions + +Create, read, and delete pipe definitions. These sit behind the [admin gate](/sdk/admin) on `/v1/ops/*` — the admin role (`policy.admin_role`) or the [operator key](/api#authentication). + + + + +```ts +// List all pipes +const { data: pipes } = await wh.pipes.list(); + +// Get a single pipe definition +const { data: pipe } = await wh.pipes.get('top_pages'); + +// Create or update +await wh.pipes.set('top_pages', { + sql: 'SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}', + parameters: [{ name: 'limit', type: 'number', required: false, default: 100 }], + description: 'Top pages by view count', + allowed_roles: ['viewer', 'admin'], +}); + +// Delete +await wh.pipes.delete('old_pipe'); +``` + + + + +```go +// List all pipes. +pipes, err := wh.Pipes.List(ctx) + +// Get a single pipe definition. +pipe, err := wh.Pipes.Get(ctx, "top_pages") + +// Create or update. +err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + Parameters: []wavehouse.ParamDef{ + {Name: "limit", Type: "number", Required: false, Default: 100}, + }, + Description: "Top pages by view count", + AllowedRoles: []string{"viewer", "admin"}, +}) + +// Delete. +err = wh.Pipes.Delete(ctx, "old_pipe") +``` + +`PipeDef` is `Pipe` minus `Name`, which the methods take as a path argument: + +```go +type PipeDef struct { + SQL string + Parameters []ParamDef + Description string + AllowedRoles []string +} +``` + + + diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md deleted file mode 100644 index 3b7f177b..00000000 --- a/docs/src/content/docs/sdk/queries.md +++ /dev/null @@ -1,263 +0,0 @@ ---- -title: "TypeScript SDK Queries" -description: "Tables, the chainable query builder, pagination, and raw SQL in @wavehouse/sdk." ---- - -Reading and writing data with `@wavehouse/sdk`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every call returns the SDK's [`Result`](/sdk#result-type) — nothing throws for anything the server returns (see [Error Handling](/sdk/reference#error-handling) for the caller and environment errors that do). Examples import from `@wavehouse/sdk` or `https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). - -## Tables — `wh.from(table)` - -`from()` returns a `TableRef` — a reference to a table. It is **NOT thenable**, so it's safe to pass around or store in a variable without triggering requests. - -```ts -const clicks = wh.from('clicks'); -``` - -### `.fetch(opts?)` - -Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read — `.fetch()` is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). - -To paginate, chain an explicit `.orderBy()` — a bare `.fetch()` sends no default order (see [Pagination](#pagination)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. - -```ts -const { data, error, hasMore, next } = await clicks.fetch(); -const { data } = await clicks.fetch({ limit: 50, signal: controller.signal }); -``` - -### `.insert(data, opts?)` - -Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. An **array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. - -```ts -// Single row → { ok: true } (or { ok: true, duplicate: true } when dedup skips it) -const { data, error } = await clicks.insert({ page: '/home', button: 'cta' }); - -// Many rows → one NDJSON request, per-record summary -const { data } = await clicks.insert([ - { page: '/home', button: 'cta' }, - { page: '/about', button: 'nav' }, -]); -// data: { ok, total, succeeded, failed, duplicates, results? } -``` - -For an array insert, `data.ok` is `true` only when every record succeeded (`failed === 0`). Inspect `data.failed` and `data.results` (each `{ index, ok|duplicate|error }`, 1-based `index`) for partial failures — the call's top-level `error` is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty array is a no-op and sends no request. The array path sends one request regardless of size; bounded-concurrency chunking of very large arrays is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). - -> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object directly (the `Content-Type` is only a hint), so non-SDK clients can send whichever shape is convenient. See the [API reference](/api#post-v1ingesttabletable--ingest-data). - -### `.insertNDJSON(source, opts?)` - -Insert pre-formatted NDJSON you already have — a `.ndjson` file, a byte stream, or a string — without first parsing it into objects. Accepts a `string`, `Uint8Array`, `Blob`/`File`, or `ReadableStream`; non-string sources are read fully into memory before sending. Returns the same per-record summary as an array `insert`. - -```ts -// From a string -await clicks.insertNDJSON('{"page":"/a"}\n{"page":"/b"}\n'); - -// From a browser (a File is a Blob) -await clicks.insertNDJSON(fileInput.files[0]); - -// From a Node file (fs.openAsBlob; or read it to a string) -import { openAsBlob } from 'node:fs'; -await clicks.insertNDJSON(await openAsBlob('events.ndjson')); -``` - -### `.schema(opts?)` - -Fetch the table's column definitions from ClickHouse. `.schema()` hits `/v1/ops/schema`, an **admin-only** endpoint: the caller must pass the admin gate — resolve to the policy admin role or present the non-JWT [operator key](/api#authentication) via [`options.headers`](/sdk#custom-headers) — or this returns `403`. - -```ts -const { data } = await clicks.schema(); -// data: { name: 'clicks', columns: [ -// { name: 'page', type: 'String', is_nullable: false, has_default: false }, ... -// ] } -``` - -### `.select(...columns)` - -Start a query builder chain. See [Query Builder](#query-builder). - -```ts -const { data } = await clicks.select('page', 'button').where('page', '=', '/home').limit(10); -``` - -### `.selectAll()` - -Selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). - -```ts -const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); -``` - -### `.stream(opts?)` - -Open a real-time event subscription. See [Streaming](/sdk/streaming). - -```ts -const stream = clicks.stream({ since: '2026-01-01T00:00:00Z' }); -``` - ---- - -## Query Builder - -Returned by `tableRef.select()`. Immutable — every chain method returns a new `QueryBuilder`. The builder is **PromiseLike**, so `await builder` auto-executes `.fetch()`. - -```ts -// These are equivalent: -const result = await clicks.select('page').limit(10).fetch(); -const result = await clicks.select('page').limit(10); // PromiseLike shortcut -``` - -### Chain Methods - -All methods return a new `QueryBuilder` — the original is unchanged. - -#### `.select(...columns)` - -Append columns to the SELECT clause. A literal `'*'` is the column *named* `*`, not a wildcard — use `.selectAll()` for all columns. - -```ts -const q = clicks.select('page').select('button'); // SELECT page, button -``` - -#### `.selectAll()` - -Select every column your role may read. For a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). - -```ts -const q = clicks.selectAll().where('country', '=', 'US'); -``` - -#### `.where(column, op, value)` - -Add a filter condition. SDK operators are translated to backend format. - -```ts -clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') -``` - -| SDK Operator | Backend | Description | -|-------------|---------|-------------| -| `'='` | `eq` | Equal | -| `'!='` | `neq` | Not equal | -| `'>'` | `gt` | Greater than | -| `'>='` | `gte` | Greater than or equal | -| `'<'` | `lt` | Less than | -| `'<='` | `lte` | Less than or equal | -| `'in'` | `in` | Value in array | -| `'like'` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.stream()` / `.liveQuery()` matches case-**insensitively** | -| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | - -#### Aggregations - -```ts -clicks.select('page') - .count('*', 'total') // COUNT(*) - .sum('score', 'total_score') // SUM(score) - .avg('score', 'avg_score') // AVG(score) - .min('score', 'min_score') // MIN(score) - .max('score', 'max_score') // MAX(score) - .countDistinct('page', 'unique_pages') - .aggregate('uniqExact', 'user_id', 'unique_users') // allowlisted fn -``` - -Custom function names pass through `.aggregate(fn, column, alias)` but are validated server-side against a fixed allowlist (matched case-insensitively): `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected with `400 unsupported aggregation function`. - -Each aggregation method signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. - -#### `.groupBy(...columns)` - -```ts -clicks.select('page').count().groupBy('page') -``` - -#### `.orderBy(column, dir?)` - -```ts -clicks.select('page').count('*', 'total').orderBy('total', 'desc') -``` - -`dir` defaults to `'asc'`. - -#### `.limit(n)` - -```ts -clicks.select().limit(100) -``` - -If no limit is specified, `QueryBuilder.DEFAULT_LIMIT` (1000) is applied automatically to prevent unbounded result sets. The server also enforces the configured maximum (`query.default_max_rows`, default 10,000 rows). - -#### `.timeRange(column, since, until?)` - -Filter by a time window. `since` and `until` accept RFC3339 timestamps or relative durations (`'1h'`, `'30m'`, `'7d'`, `'2w'` — day and week suffixes expand to hours, so `'7d'` is `'168h'`). - -```ts -clicks.select('page').timeRange('received_timestamp', '1h') -clicks.select('page').timeRange( - 'received_timestamp', '2026-01-01T00:00:00Z', '2026-02-01T00:00:00Z' -) -``` - -#### `.cacheTTL(seconds)` - -Records a desired result-cache TTL on the builder. **Currently client-side state only** — the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). - -```ts -clicks.select('page').count().cacheTTL(300) // not yet honored server-side — see #280 -``` - -### `.fetch(opts?)` - -Execute the query. Returns `Result` with optional pagination. - -```ts -const { data, error, hasMore, next } = await clicks.select('page').limit(50).fetch(); - -if (hasMore && next) { - const page2 = await next(); // cursor-based pagination -} -``` - -**Options** — `RequestOptions`: - -| Field | Type | Description | -|-------|------|-------------| -| `signal` | `AbortSignal` | Cancel the request | -| `limit` | `number` | Override builder limit for this fetch | - -A pipe's `.fetch()` takes the narrower `PipeRequestOptions` instead — see [Pipes](/sdk/pipes#fetchopts). - -### `.stream(opts?)` - -Open a live stream from the builder's table. See [Streaming](/sdk/streaming). - -### Pagination - -When `limit` is set and the result contains at least `limit` rows, `hasMore` is `true`. Cursor-based pagination's `next()` walks an **order column** — it adds a filter on that column using the last row's value — so `next()` is only attached when the query has an explicit `.orderBy()`. With no order column the result still reports `hasMore` honestly, but `next` is `undefined` (there is no deterministic cursor to build) — add an `.orderBy()` to paginate. - -```ts -let result = await clicks.select().orderBy('received_timestamp', 'desc').limit(100).fetch(); - -const allRows = [...result.data!]; -while (result.hasMore && result.next) { - result = await result.next(); - if (result.data) allRows.push(...result.data); -} -``` - -The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. The Go SDK's `Next` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). - -Rows decode with JSON numbers as JS `number`s, so an integer cursor column past `Number.MAX_SAFE_INTEGER` (2^53) loses exactness and pagination can repeat or skip a row at that scale. The Go SDK's `FetchTyped` with an `int64` field avoids this; there is no JS equivalent short of a string or `bigint` column. - ---- - -## Raw SQL — `wh.sql(query, opts?)` - -Execute a raw SQL query. `/v1/ops/query` is admin-only: the caller must resolve to the policy admin role (`admin_role`, `"admin"` by default). A tokenless request falls back to the `default_role`, so it is rejected with `403` on any policy that doesn't deliberately set `default_role` to the admin role (a loudly-warned dev-only setting); an invalid or expired token is rejected with `401`. The SDK has no first-class option for the server's non-JWT [operator key](/api#authentication) — an operator can send its `X-Operator-Key` header via [`options.headers`](/sdk#custom-headers). - -```ts -const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); -``` - -:::note[No parameter binding through the SDK] -Positional `?` substitution is not supported, and the SDK has no way to forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo) — the proxy doesn't forward arbitrary query-string params and `wh.sql()` doesn't expose a hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder (`wh.from(table)…`). -::: diff --git a/docs/src/content/docs/sdk/queries.mdx b/docs/src/content/docs/sdk/queries.mdx new file mode 100644 index 00000000..3a0acdf4 --- /dev/null +++ b/docs/src/content/docs/sdk/queries.mdx @@ -0,0 +1,647 @@ +--- +title: "SDK Queries" +description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse SDKs." +--- + +import { Tabs, TabItem } from "@astrojs/starlight/components"; + +Reading and writing data: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Both SDKs speak the same wire format and expose the same surface — pick your language with the tabs below and the choice follows you across every page. Setup, client construction, and the error model live on the per-language pages: [TypeScript](/sdk/typescript) and [Go](/sdk/go). + +## Tables + +The entry point is a table reference. It performs no request, so it is safe to store in a variable or pass around. + + + + +`wh.from(table)` returns a `TableRef`. It is **NOT thenable**, so holding one triggers nothing. + +```ts +const clicks = wh.from('clicks'); +``` + +Every call returns the SDK's [`Result`](/sdk/typescript#result-type) — nothing throws for anything the server returns (see [Error Handling](/sdk/reference#error-handling) for the caller and environment errors that do). + + + + +`client.From(table)` returns a `*TableRef`. + +```go +clicks := wh.From("clicks") +``` + +Every request-response operation takes a `context.Context` first and returns `(T, error)` — the chainable builder methods and `.Stream(opts)` excepted. See [Error Handling](/sdk/go#error-handling). + + + + +### Fetching every column + +Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read — this is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. + +To paginate, chain an explicit order — a bare fetch sends no default order (see [Pagination](#pagination)). + + + + +```ts +const { data, error, hasMore, next } = await clicks.fetch(); +const { data } = await clicks.fetch({ limit: 50, signal: controller.signal }); +``` + + + + +`.Fetch(ctx)` is internally `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. There is no options struct as in the TypeScript SDK's `.fetch(opts?)`: to override the limit or paginate, chain `.SelectAll().Limit(n).OrderBy(...)` yourself ([Query Builder](#query-builder)). + +```go +page, err := clicks.Fetch(ctx) +if err != nil { + log.Fatal(err) +} +for _, row := range page.Data { + fmt.Println(row["page"]) +} +``` + + + + +### Inserting rows + +Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. A **slice/array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. + +For a batch, the result is `ok` only when every record succeeded (`failed === 0`). Inspect the failure count and the per-record results (each carrying a **1-based** index) for partial failures — the call's top-level error is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty batch is a no-op and sends no request. The batch path sends one request regardless of size; bounded-concurrency chunking of very large batches is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). + + + + +```ts +// Single row → { ok: true } (or { ok: true, duplicate: true } when dedup skips it) +const { data, error } = await clicks.insert({ page: '/home', button: 'cta' }); + +// Many rows → one NDJSON request, per-record summary +const { data } = await clicks.insert([ + { page: '/home', button: 'cta' }, + { page: '/about', button: 'nav' }, +]); +// data: { ok, total, succeeded, failed, duplicates, results? } +``` + +Each entry of `data.results` is `{ index, ok|duplicate|error }`. + + + + +`.Insert(ctx, data)` picks the path from the input type: a **map or struct** (excluding slices and `[]byte`) goes as JSON; **any slice** (`[]map[string]any`, `[]ClickRow`, …) is serialized to NDJSON via reflection. + +```go +// Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) +res, err := clicks.Insert(ctx, map[string]any{"page": "/home", "button": "cta"}) + +// Many rows (map slice) → one NDJSON request, per-record summary +res, err = clicks.Insert(ctx, []map[string]any{ + {"page": "/home", "button": "cta"}, + {"page": "/about", "button": "nav"}, +}) +// res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results + +// Many rows (typed slice) — same NDJSON path, via reflection +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` +} +res, err = clicks.Insert(ctx, []ClickRow{ + {Page: "/home", Button: "cta"}, + {Page: "/about", Button: "nav"}, +}) +``` + +Each entry of `res.Results` is an `InsertRecordResult{Index, OK, Duplicate, Error}`. + + + + +> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object directly (the `Content-Type` is only a hint), so non-SDK clients can send whichever shape is convenient. See the [API reference](/api#post-v1ingesttabletable--ingest-data). + +### Inserting pre-formatted NDJSON + +Insert NDJSON you already have — a `.ndjson` file, a byte stream, or a string — without first parsing it into objects. Returns the same per-record summary as a batch insert. + + + + +`.insertNDJSON(source, opts?)` accepts a `string`, `Uint8Array`, `Blob`/`File`, or `ReadableStream`; non-string sources are read fully into memory before sending. + +```ts +// From a string +await clicks.insertNDJSON('{"page":"/a"}\n{"page":"/b"}\n'); + +// From a browser (a File is a Blob) +await clicks.insertNDJSON(fileInput.files[0]); + +// From a Node file (fs.openAsBlob; or read it to a string) +import { openAsBlob } from 'node:fs'; +await clicks.insertNDJSON(await openAsBlob('events.ndjson')); +``` + + + + +`.InsertNDJSON(ctx, ndjson)` takes a `string`. + +```go +// From a literal string. +res, err := clicks.InsertNDJSON(ctx, `{"page":"/a"}`+"\n"+`{"page":"/b"}`) + +// From a file on disk. +raw, err := os.ReadFile("events.ndjson") +if err != nil { + log.Fatal(err) +} +res, err = clicks.InsertNDJSON(ctx, string(raw)) +``` + + + + +### Table schema + +Fetch the table's column definitions from ClickHouse. This hits `/v1/ops/schema`, an **admin-only** endpoint: the caller must pass the admin gate — resolve to the policy admin role or present the non-JWT [operator key](/api#authentication) — or it returns `403`. + + + + +Send the operator key via [`options.headers`](/sdk/typescript#custom-headers). + +```ts +const { data } = await clicks.schema(); +// data: { name: 'clicks', columns: [ +// { name: 'page', type: 'String', is_nullable: false, has_default: false }, ... +// ] } +``` + + + + +Send the operator key via [`ClientOptions.Headers`](/sdk/go#clientoptions). + +```go +schema, err := clicks.Schema(ctx) +// schema.Name == "clicks" +// schema.Columns: []Column{{Name: "page", Type: "String", IsNullable: false, HasDefault: false}, ...} +``` + + + + +### Selecting columns + +Starts a query builder chain — see [Query Builder](#query-builder) for the chainable methods and how to execute it. + + + + +```ts +const { data } = await clicks.select('page', 'button').where('page', '=', '/home').limit(10); +``` + + + + +```go +page, err := clicks.Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) +``` + + + + +### Selecting every column explicitly + +`selectAll` selects **every column your role is allowed to read** — the explicit form of what a bare fetch does. Mutually exclusive with an explicit column list and with aggregations; for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). + + + + +```ts +const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); +``` + + + + +```go +page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) +``` + + + + +### Opening a stream + +Open a real-time event subscription on the table. See [Streaming](/sdk/streaming). + + + + +```ts +const stream = clicks.stream({ since: '2026-01-01T00:00:00Z' }); +``` + + + + +```go +stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) +defer stream.Close() +``` + + + + +--- + +## Query Builder + +Returned by the table ref's select methods. Immutable — every chain method returns a new builder, leaving the original unchanged. + + + + +The builder is **PromiseLike**, so `await builder` auto-executes `.fetch()`. + +```ts +// These are equivalent: +const result = await clicks.select('page').limit(10).fetch(); +const result = await clicks.select('page').limit(10); // PromiseLike shortcut +``` + + + + +Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly. + +```go +page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) +``` + + + + +### Appending columns + +Append columns to the SELECT clause. A literal `*` is the column *named* `*`, not a wildcard — use the `selectAll` form for all columns. + + + + +```ts +const q = clicks.select('page').select('button'); // SELECT page, button +``` + + + + +```go +q := clicks.Select("page").Select("button") // SELECT page, button +``` + + + + +### Filtering + +Add a filter condition. Operators are translated to the backend wire format. + + + + +```ts +clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') +``` + +| SDK Operator | Backend | Description | +|-------------|---------|-------------| +| `'='` | `eq` | Equal | +| `'!='` | `neq` | Not equal | +| `'>'` | `gt` | Greater than | +| `'>='` | `gte` | Greater than or equal | +| `'<'` | `lt` | Less than | +| `'<='` | `lte` | Less than or equal | +| `'in'` | `in` | Value in array | +| `'like'` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.stream()` / `.liveQuery()` matches case-**insensitively** | +| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | + + + + +```go +clicks.Select("page"). + Where("score", wavehouse.OpGt, 10). + Where("page", wavehouse.OpLike, "/home%") +``` + +| `FilterOp` constant | Backend wire token | Description | +|----------------------|---------------------|--------------| +| `wavehouse.OpEq` | `eq` | Equal | +| `wavehouse.OpNeq` | `neq` | Not equal | +| `wavehouse.OpGt` | `gt` | Greater than | +| `wavehouse.OpGte` | `gte` | Greater than or equal | +| `wavehouse.OpLt` | `lt` | Less than | +| `wavehouse.OpLte` | `lte` | Less than or equal | +| `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | +| `wavehouse.OpLike` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.Stream()` / `.LiveQuery()` matches case-**insensitively** | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | + + + + +### Aggregations + +Custom function names pass through the generic `aggregate` method but are validated server-side against a fixed allowlist (matched case-insensitively): `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected with `400 unsupported aggregation function`. + + + + +```ts +clicks.select('page') + .count('*', 'total') // COUNT(*) + .sum('score', 'total_score') // SUM(score) + .avg('score', 'avg_score') // AVG(score) + .min('score', 'min_score') // MIN(score) + .max('score', 'max_score') // MAX(score) + .countDistinct('page', 'unique_pages') + .aggregate('uniqExact', 'user_id', 'unique_users') // allowlisted fn +``` + +Each aggregation method signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. + + + + +```go +clicks.Select("page"). + Count("*", "total"). // COUNT(*) + Sum("score", "total_score"). // SUM(score) + Avg("score", "avg_score"). // AVG(score) + Min("score", "min_score"). // MIN(score) + Max("score", "max_score"). // MAX(score) + CountDistinct("page", "unique_pages"). + Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn +``` + +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `.Aggregate` takes `(fn, column, alias)`. With an empty alias, `Count` defaults to `count` (and `column=""` becomes `*`), `Sum`/`Avg`/`Min`/`Max` to `sum_`/`avg_`/`min_`/`max_`, and `CountDistinct` to `count_distinct_`; `Aggregate` has no default and sends `""`. + + + + +### Grouping and ordering + +Order direction defaults to ascending. + + + + +```ts +clicks.select('page').count().groupBy('page') +clicks.select('page').count('*', 'total').orderBy('total', 'desc') +``` + + + + +```go +clicks.Select("page").Count("", "").GroupBy("page") +clicks.Select("page").Count("", "total").OrderBy("total", "desc") +``` + +`OrderBy`'s `dir` defaults to `"asc"` when passed `""`. + + + + +### Limiting + +If no limit is specified, the SDK's default limit (1000) is applied automatically to prevent unbounded result sets. The server also enforces the configured maximum (`query.default_max_rows`, default 10,000 rows). + + + + +```ts +clicks.select().limit(100) // QueryBuilder.DEFAULT_LIMIT otherwise +``` + + + + +```go +clicks.Select().Limit(100) // wavehouse.DefaultLimit otherwise +``` + + + + +### Time ranges + +Filter by a time window. Both bounds accept RFC3339 timestamps or relative durations (`1h`, `30m`, `7d`, `2w` — day and week suffixes expand to hours, so `7d` is `168h`). + + + + +`until` is optional; omit it for an open-ended range. + +```ts +clicks.select('page').timeRange('received_timestamp', '1h') +clicks.select('page').timeRange( + 'received_timestamp', '2026-01-01T00:00:00Z', '2026-02-01T00:00:00Z' +) +``` + + + + +Pass `""` for `until` for an open-ended range. + +```go +clicks.Select("page").TimeRange("received_timestamp", "1h", "") +clicks.Select("page").TimeRange( + "received_timestamp", "2026-01-01T00:00:00Z", "2026-02-01T00:00:00Z", +) +``` + + + + +### Cache TTL + +Records a desired result-cache TTL on the builder. **Currently client-side state only** — the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). + + + + +```ts +clicks.select('page').count().cacheTTL(300) // not yet honored server-side — see #280 +``` + + + + +```go +clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 +``` + + + + +### Executing the query + + + + +`.fetch(opts?)` returns `Result` with optional pagination. + +```ts +const { data, error, hasMore, next } = await clicks.select('page').limit(50).fetch(); + +if (hasMore && next) { + const page2 = await next(); // cursor-based pagination +} +``` + +**Options** — `RequestOptions`: + +| Field | Type | Description | +|-------|------|-------------| +| `signal` | `AbortSignal` | Cancel the request | +| `limit` | `number` | Override builder limit for this fetch | + +A pipe's `.fetch()` takes the narrower `PipeRequestOptions` instead — see [Pipes](/sdk/pipes#executing-a-pipe). + + + + +`wavehouse.FetchTyped[Row](ctx, q)` decodes rows into `[]Row`; `.FetchUntyped(ctx)` decodes into `[]map[string]any`. Both return a `Page[T]`. Typed fetching is a package-level generic function because Go has no generic methods. + +```go +type PageCount struct { + Page string `json:"page"` + Count int `json:"total"` +} + +page, err := wavehouse.FetchTyped[PageCount](ctx, + clicks.Select("page").Count("*", "total").GroupBy("page"), +) +// page.Data is []PageCount + +untyped, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) +``` + + + + +### Pagination + +When a limit is set and the result contains at least that many rows, the result reports that more rows are available. Cursor-based pagination walks an **order column** — it adds a filter on that column using the last row's value — so the `next` cursor is only attached when the query has an explicit order. With no order column the result still reports "has more" honestly, but there is no cursor to build. + +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. Both SDKs share this limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + + + + +```ts +let result = await clicks.select().orderBy('received_timestamp', 'desc').limit(100).fetch(); + +const allRows = [...result.data!]; +while (result.hasMore && result.next) { + result = await result.next(); + if (result.data) allRows.push(...result.data); +} +``` + +Rows decode with JSON numbers as JS `number`s, so an integer cursor column past `Number.MAX_SAFE_INTEGER` (2^53) loses exactness and pagination can repeat or skip a row at that scale. The Go SDK's `FetchTyped` with an `int64` field avoids this; there is no JS equivalent short of a string or `bigint` column. + + + + +Both fetch methods return a `Page[T]`: + +```go +type Page[T any] struct { + Data []T + HasMore bool + Next func(ctx context.Context) (*Page[T], error) // nil when no cursor is available +} +``` + +`Next` walks the **first** `.OrderBy()` column. Without one it is `nil`, and if the order column is missing from `.Select(...)` it returns an empty page. + +```go +page, err := clicks.Select(). + OrderBy("received_timestamp", "desc"). + Limit(100). + FetchUntyped(ctx) +if err != nil { + log.Fatal(err) +} + +allRows := append([]map[string]any(nil), page.Data...) +for page.HasMore && page.Next != nil { + page, err = page.Next(ctx) + if err != nil { + log.Fatal(err) + } + allRows = append(allRows, page.Data...) +} +``` + +On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row; `FetchTyped` with an `int64` field, or codegen structs, keep it exact. + + + + +--- + +## Raw SQL + +Execute a raw SQL query. `/v1/ops/query` is admin-only: the caller must resolve to the policy admin role (`admin_role`, `"admin"` by default). A tokenless request falls back to the `default_role`, so it is rejected with `403` on any policy that doesn't deliberately set `default_role` to the admin role (a loudly-warned dev-only setting); an invalid or expired token is rejected with `401`. + + + + +The SDK has no first-class option for the server's non-JWT [operator key](/api#authentication) — an operator can send its `X-Operator-Key` header via [`options.headers`](/sdk/typescript#custom-headers). + +```ts +const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); +``` + + + + +An operator key authorizes `/v1/ops/*` as well; send it via [`ClientOptions.Headers`](/sdk/go#clientoptions). Use `map[string]any` for dynamic schemas. + +```go +rows, err := wavehouse.SQL[map[string]any](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") + +// Or decode into a struct that matches the projected columns/aliases. +// NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit +// integers (count() is UInt64) — decode them with the `,string` tag, or +// use map[string]any. See Reference → Codegen CLI for the full story. +type PageTotal struct { + Page string `json:"page"` + Total uint64 `json:"total,string"` +} +typed, err := wavehouse.SQL[PageTotal](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") +``` + + + + +:::note[No parameter binding through the SDK] +Positional `?` substitution is not supported, and neither SDK can forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo) — the proxy doesn't forward arbitrary query-string params and the raw-SQL entry point exposes no hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder. +::: diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.mdx similarity index 50% rename from docs/src/content/docs/sdk/reference.md rename to docs/src/content/docs/sdk/reference.mdx index 44832116..094f9fcb 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.mdx @@ -1,13 +1,18 @@ --- -title: "TypeScript SDK Reference & CLI" -description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." +title: "SDK Reference & CLI" +description: "Error codes, cancellation, the full API tree, and the codegen CLIs for the WaveHouse SDKs." --- -Cross-cutting reference for `@wavehouse/sdk`: cancellation, the error model behind every [`Result`](/sdk#result-type), the complete API tree at a glance, and the tooling that ships in the package. +import { Tabs, TabItem } from "@astrojs/starlight/components"; -## AbortController Support +Cross-cutting reference for both SDKs: cancellation, the error model, the complete API tree at a glance, and the tooling each package ships. -All async operations accept an `AbortSignal` for cancellation: +## Cancellation + + + + +All async operations accept an `AbortSignal`: ```ts const controller = new AbortController(); @@ -19,11 +24,37 @@ if (error?.code === 'ABORTED') { } ``` + + + +Non-streaming operations take a `context.Context` as their first argument (the analog of TypeScript's `AbortSignal`). Cancel via a timeout or an explicit `cancel()`; cancellation returns immediately, without retrying, as `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() + +page, err := wh.From("clicks").Fetch(ctx) +var whErr *wavehouse.Error +if errors.As(err, &whErr) && whErr.Code == "ABORTED" { + fmt.Println("Request timed out") +} +``` + +`.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming → Stream options](/sdk/streaming#stream-options). + + + + --- ## Error Handling -The SDK **never throws** for anything the server returns — all API errors come back in `Result.error`. It does throw on caller and environment errors: a non-absolute `baseURL` (REST calls reject with a `TypeError`; streams report `SSE_CONNECT_ERROR` to the subscriber's `error` callback — see [Serving under a path prefix](/sdk#serving-under-a-path-prefix)), `.stream()` / `.liveQuery()` in a runtime with no global `fetch` and no `options.fetch` (see [Runtime support](/sdk#runtime-support)), and an `auth` callback that rejects — a token-refresh failure propagates out of the REST call, and on a stream is reported as a retryable `SSE_AUTH_ERROR`. One more exception escapes an SDK call synchronously, though it is yours rather than ours: your own `status` handler throwing on the first `.subscribe()` or `.liveQuery()`, described under *If your own callback throws* below. +Neither SDK raises for anything the server returns — API errors are values. Both raise (or return plain wrapped errors) for caller and environment faults instead. + + + + +All API errors come back in [`Result.error`](/sdk/typescript#result-type). The SDK does throw on caller and environment errors: a non-absolute `baseURL` (REST calls reject with a `TypeError`; streams report `SSE_CONNECT_ERROR` to the subscriber's `error` callback — see [Serving under a path prefix](/sdk/typescript#serving-under-a-path-prefix)), `.stream()` / `.liveQuery()` in a runtime with no global `fetch` and no `options.fetch` (see [Runtime support](/sdk/typescript#runtime-support)), and an `auth` callback that rejects — a token-refresh failure propagates out of the REST call, and on a stream is reported as a retryable `SSE_AUTH_ERROR`. One more exception escapes an SDK call synchronously, though it is yours rather than ours: your own `status` handler throwing on the first `.subscribe()` or `.liveQuery()`, described under *If your own callback throws* below. | Status | Code | Retryable | Description | |--------|------|-----------|-------------| @@ -71,10 +102,43 @@ Rejected requests surface the real status and message rather than an opaque conn `SSE_AUTH_ERROR` is the one caller-side failure that isn't terminal. A rejecting `auth` callback propagates out of a REST call, but on a stream — where `auth` is invoked on every connection attempt rather than once per stream — a token endpoint having a bad minute is treated as transient and retried, rather than tearing down a stream that is otherwise healthy. + + + +The SDK never panics on API or network failures, mirroring the TypeScript SDK's "never throws" guarantee. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less ones (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`, or use `wavehouse.IsRetryable(err)` to shortcut the `errors.As` + `.Retryable` check); client-side failures such as an `Auth` provider or marshal error are plain wrapped errors, so handle the `errors.As == false` case — see the [worked example](/sdk/go#error-handling). Streaming methods (`Stream`, `Subscribe`, `Close`) report through the subscriber's `Error` callback instead, and `Connected(ctx)` returns plain errors. + +| Status | Code | Retryable | Description | +|--------|------|-----------|--------------| +| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | +| 401 | `HTTP_401` | No | Invalid or expired JWT (missing tokens use `default_role`, resulting in success or 403) | +| 403 | `HTTP_403` | No | Insufficient permissions | +| 404 | `HTTP_404` | No | Table or pipe not found | +| 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | +| 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `ABORTED` | No | Request canceled via `context.Context` | +| 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | +| 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | +| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | +| *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | +| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | +| 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | +| 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | +| 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | + +Retries apply to all HTTP methods, matching the TypeScript SDK's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup to suppress duplicates. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. + + + + --- ## Full API Tree + + + ```text createClient(config) → WaveHouseClient ├── .from(table) → TableRef (NOT thenable) @@ -122,9 +186,71 @@ StreamController (NOT thenable) └── [Symbol.asyncIterator]() → AsyncIterableIterator ``` + + + +```text +NewClient(Config) → *Client +├── .From(table) → *TableRef +│ ├── .Fetch(ctx) → (*Page[map[string]any], error) +│ ├── .Select(...cols) → *QueryBuilder +│ │ ├── .Select() .SelectAll() .Where() .Count() .Sum() .Avg() .Min() .Max() +│ │ │ .CountDistinct() .Aggregate() .GroupBy() .OrderBy() +│ │ │ .Limit() .TimeRange() .CacheTTL() +│ │ ├── FetchTyped[Row](ctx, q) → (*Page[Row], error) // package-level generic func +│ │ ├── .FetchUntyped(ctx) → (*Page[map[string]any], error) +│ │ ├── .Stream(opts) → *StreamController +│ │ └── .LiveQuery(sub, opts) → *LiveQueryHandle +│ ├── .SelectAll() → *QueryBuilder +│ ├── .Insert(ctx, data) → (*InsertResult, error) +│ ├── .InsertNDJSON(ctx, ndjson) → (*InsertResult, error) +│ ├── .Schema(ctx) → (*TableSchema, error) +│ └── .Stream(opts) → *StreamController +├── .Pipe(name, params) → *PipeRef +│ ├── Fetch[Row](ctx, p) → ([]Row, error) // package-level generic func +│ ├── .FetchUntyped(ctx) → ([]map[string]any, error) +│ └── .Stream(opts) → *StreamController +├── .Pipes (admin) → *PipesNamespace +│ ├── .List(ctx) → ([]Pipe, error) +│ ├── .Get(ctx, name) → (*Pipe, error) +│ ├── .Set(ctx, name, PipeDef) → error +│ └── .Delete(ctx, name) → error +├── SQL[Row](ctx, client, query) → ([]Row, error) // package-level generic func, admin-only +├── .Schema (admin) → *SchemaNamespace +│ ├── .List(ctx) → (Schemas, error) +│ └── .Refresh(ctx) → error +├── .Policy (admin) → *PolicyNamespace +│ ├── .Get(ctx) → (*Policy, error) +│ ├── .Set(ctx, *Policy) → error +│ └── .Validate(ctx, *Policy) → (*ValidationResult, error) +├── .DLQ (admin) → *DLQNamespace +│ ├── .List(ctx) → (*DLQStats, error) +│ ├── .Table(ctx, name) → (*DLQStats, error) +│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 +└── .Sys → *SysNamespace + └── .Health(ctx) → error + +*StreamController +├── .Subscribe(*StreamSubscriber) → func() // unsubscribe +├── .Events() → <-chan StreamEvent // idiomatic Go alternative to an async iterator +├── .Close() +├── .Status() → StreamStatus +└── .Connected(ctx) → error // Go-only addition, blocks until live +``` + + + + +--- + ## Codegen CLI -Generate TypeScript types from a running WaveHouse instance. The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`: +Generate typed row definitions from a running WaveHouse instance. Codegen reads the admin-only `/v1/ops/schema` endpoint, so a non-dev server needs an admin token or returns `403`. + + + + +The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`: ```bash npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts @@ -133,7 +259,7 @@ npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts pnpm codegen --url http://localhost:8080 --out ./src/db.d.ts ``` -Codegen reads `/v1/ops/schema`, which is **admin-only**. Against a non-dev server, pass an admin-role token with `--auth ` or the request is denied with `403`. +Pass an admin-role token with `--auth `. **Options:** @@ -174,11 +300,91 @@ export interface ClicksRow { | `Map(K, V)` | `Record` | | `LowCardinality(T)` | same as `T` | -## E2E Testing + + + +The module ships a `wavehouse-codegen` command under `cmd/`: + +```bash +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ + --url http://localhost:8080 \ + --out ./db_types.go \ + --package myapp + +# Or, from inside a checkout of clients/go/: +go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go +``` + +Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history and process listings. + +**Options:** + +| Flag | Description | Default | +|------|-------------|---------| +| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | +| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | +| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | +| `--package`, `-p` | Go package name for the generated file | `main` | +| `--help`, `-h` | Show usage and exit | — | + +**Example output** (for the [development quick-start](/development#quick-start) `clicks` table): + +```go +// Code generated by wavehouse-codegen. DO NOT EDIT. + +package myapp + +// ClicksRow represents a row in the "clicks" table. +type ClicksRow struct { + Page string `json:"page"` + Button string `json:"button"` + Score float64 `json:"score"` + ReceivedTimestamp *string `json:"received_timestamp,omitempty"` +} +``` + +Output is run through `go/format`, and codegen fails loudly if a table or column name would produce invalid Go source. Names become `PascalCase`, with an `X` prefix for a leading digit (`2fa_events` → `X2faEventsRow`); initialisms are not special-cased, so `event_id` becomes `EventId`, not `EventID`. Columns with `has_default: true` become pointer fields with `,omitempty` — as `received_timestamp` does above — where `nil` uses the server default and a pointed-at value is sent, including an explicit `0`/`false`/`""`. + +**ClickHouse → Go type mapping:** + +| ClickHouse Type | Go Type | +|------------------|---------| +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` / `Boolean` | `bool` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `Float32`, `BFloat16` | `float32` | +| `Float64` | `float64` | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` | +| `Decimal*` | `string` | +| `Nullable(T)` | `*T` | +| `LowCardinality(T)` | same as `T` | +| `Array(T)` | `[]T` (except `Array(UInt8)` → `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | +| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | +| anything unrecognized | `any` | + +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`; for the raw-SQL path (`/v1/ops/query`), which quotes 64-bit-and-wider integers, use `map[string]any` with `SQL[Row]`. + + + + +--- + +## Testing + +The cross-language wire-format **conformance suite** replays one shared fixture (`clients/go/testdata/wire_cases.json`) through both SDKs, asserting HTTP methods, paths, content types, and bodies. The Go half is `clients/go/conformance_test.go` (run by `make test-sdk-go`); the TypeScript half is `tests/conformance/conformance_ts.mjs` (run by `make test-sdk-ts`). `make test-sdk` runs both, and so does CI, keeping the wire formats in step. + + + The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) through the SDK, validating both the backend and the client library in one pass. ```bash +# Unit tests + wire conformance +make test-sdk-ts + # Run all E2E tests: the orchestrator boots a ClickHouse testcontainer + # the wavehouse-cov binary, then runs the SDK suite make test-e2e @@ -187,3 +393,24 @@ make test-e2e Test files live in `tests/e2e/sdk/` (each `*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. See [Development Guide — E2E Tests via SDK](/development#e2e-tests-via-sdk) for architecture details and workflow tips. + + + + +Unit tests are colocated in `clients/go/`, which is its own module (`clients/go/go.mod`), separate from the root `WaveHouse` module: + +```bash +# Unit tests + wire conformance + coverage gate +make test-sdk-go +``` + +E2E tests (build tag `e2e`) run against a live WaveHouse instance: + +```bash +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-sdk-go-e2e +``` + +`WAVEHOUSE_URL` defaults to `http://localhost:8080`, and the optional `WAVEHOUSE_AUTH` covers the admin cases; the suite skips if the server is unreachable. Unlike the TypeScript SDK, the Go E2E suite isn't yet driven by the repo's `make test-e2e` orchestrator — wiring it in, with its own coverage gate, is tracked in [#518](https://github.com/Wave-RF/WaveHouse/issues/518). + + + diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md deleted file mode 100644 index c4b7dc69..00000000 --- a/docs/src/content/docs/sdk/streaming.md +++ /dev/null @@ -1,216 +0,0 @@ ---- -title: "TypeScript SDK Streaming & Live Queries" -description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." ---- - -Real-time consumption with `@wavehouse/sdk`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/queries). Examples import from `@wavehouse/sdk` or `https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). - -## Streaming - -Streams are Server-Sent Events over `fetch`. The `auth` token rides in an `Authorization: Bearer` header — the same as every other request, on browsers and servers alike — and is re-read on each connection attempt, so a stream that outlives its token picks up a fresh one. Unauthenticated streams work the same way, minus the header. - -### `StreamController` - -Returned by `.stream()` on `TableRef`, `QueryBuilder`, `PipeRef`, and `DLQNamespace` (the DLQ variant is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). It is **NOT thenable**. - -```ts -const stream = wh.from('clicks').stream({ since: '2026-01-01T00:00:00Z' }); -``` - -### `.subscribe(subscriber)` → `unsubscribe()` - -Callback-based consumption. Returns a cleanup function. - -```ts -const unsub = stream.subscribe({ - next: (event) => { - // event: { table: 'clicks', timestamp: '2026-...', data: { page: '/', ... } } - console.log('New event:', event.data); - }, - status: (state) => { - // state: 'connecting' | 'live' | 'reconnecting' | 'closed' - updateIndicator(state); - }, - error: (err) => { - console.error('Stream error:', err.message); - }, -}); - -// Cleanup — closes the connection if no other subscribers remain -unsub(); -``` - -A handler that throws *during delivery* doesn't end the stream; the exception is logged and the connection keeps running. But delivery of that event stops at the handler that threw — your *later* subscribers, and any concurrent `for await`, do not get it. The first `status` call, the synchronous one `.subscribe()` makes before returning, isn't caught at all and throws back out at you. Wrap your handler bodies in your own `try`/`catch`; see [Error Handling](/sdk/reference#error-handling) for the carve-outs. - -### Async Iterator - -```ts -const stream = wh.from('clicks').stream(); - -for await (const event of stream) { - console.log(event.table, event.data); - if (shouldStop) break; // breaking auto-closes the stream -} -``` - -### `.close()` - -Explicitly close the stream and release all resources. - -```ts -stream.close(); -``` - -### `.status` - -Current connection status: `'connecting' | 'live' | 'reconnecting' | 'closed'`. - -### `.connected(timeoutMs?)` → `Promise` - -Resolves once the stream reaches `live`. Rejects if it is already `closed`, if it closes before connecting, or after `timeoutMs` (default `5000`). Useful when you need the subscription established before doing something that depends on it — inserting a row you expect to see come back, for instance. - -```ts -const stream = wh.from('clicks').stream(); -const unsub = stream.subscribe({ next: (e) => console.log(e) }); -await stream.connected(); // wait until the transport is live -await wh.from('clicks').insert({ page: '/home' }); -``` - -A *timeout* rejection does **not** stop the transport — reconnection is unbounded, so it means "not live yet", not "given up"; call `.close()` if you want it to stop. A rejection because the stream closed is different: there the transport has already stopped. - -One way this rejects against a perfectly healthy stream: if a subscriber you registered *before* calling `.connected()` has a `status` handler that throws, the throw aborts the fan-out before `connected()`'s internal watcher sees `live`, so it times out while `.status` already reads `live`. See [Error Handling](/sdk/reference#error-handling). - -### `StreamOptions` - -| Field | Type | Description | -| ----- | ---- | ----------- | -| `since` | `string` | RFC3339 timestamp for gap-fill replay | -| `signal` | `AbortSignal` | Cancel/close the stream. Wired via `attachSignal()` internally. | - -### `StreamEvent` - -```ts -interface StreamEvent { - table: string; // table name (e.g. 'clicks') - timestamp: string; // received_timestamp (RFC3339Nano) - data: T; // row data -} -``` - -Row values of top-level `DateTime`/`DateTime64` columns inside `data` (not timestamps nested in `Array`/`Map`/`Tuple` columns) arrive in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), matching what `/v1/query` returns for the same row — `new Date(value)` parses correctly with no zone fix-up. Values WaveHouse couldn't canonicalize (ingest is fail-open) stream in the producer's original spelling, and the `/v1/query` match doesn't hold for them: a spelling ClickHouse accepted anyway still queries back in canonical UTC (one it rejected never lands in the table at all), and a zone-less date-time is what `new Date()` reads as *local* time — though a date-only `YYYY-MM-DD` string is read as UTC, an ECMAScript quirk (see [Timestamp canonicalization](/api#timestamp-canonicalization)). - -### Transport Behavior - -| Transport | Reconnect | Protocol | -| --------- | --------- | -------- | -| SSE over `fetch` | Automatic, jittered backoff, resumes via `Last-Event-ID` | HTTP/2 recommended | - -:::note[SSE connection limit] -The SDK warns above 5 concurrent SSE connections — just under the browser's 6-per-domain limit for HTTP/1.1. HTTP/2 multiplexes over one connection, so the limit doesn't apply there. -::: - -A dropped stream reconnects on a jittered exponential backoff, capped at 30s, and resumes with `Last-Event-ID` so the server gap-fills what was missed. The schedule resets only after a connection has *held* for a few seconds, so a server accepting and immediately closing — slow-consumer eviction, a half-broken upstream — can't pin the client at sub-second retries. Reconnection is **unbounded**: a stream keeps re-dialing until it hits a terminal error or you `close()` it. `options.maxRetries` bounds REST requests only and is never consulted here. - -:::caution[Resumption is at-least-once, and time-bounded] -Delivery across a reconnect is **at-least-once**. The `Last-Event-ID` the client sends is the last event's `received_timestamp`, and the server replays from that instant *inclusively* — so the last event you already saw, and anything sharing its timestamp, arrives again. The SDK does not deduplicate live frames — `liveQuery()` makes one pass at the backfill seam, and only under an ascending order ([#449](https://github.com/Wave-RF/WaveHouse/issues/449)) — so key on `timestamp` plus your own row identity if duplicates matter. - -Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration#message-queue-nats) — 15 minutes by default. A drop longer than that resumes with a hole and no signal, because the purged messages are simply gone. -::: - -A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the only 4xx it raises itself is `400` for a missing or empty table name, and only on the stream route — a `404` or `405` means the request never reached it, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. - -Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request — which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk#supplying-your-own-fetch). - -### Server-Side Policy Filtering - -Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims. A stream on a row-policied table therefore delivers only the rows the policy admits for that connection — and, where the server's in-memory comparison can't prove a match, fewer; see [Access control — where each rule is enforced](/access-control#where-each-rule-is-enforced). Claims are captured when the connection opens: a policy change applies from the next live event (an in-flight gap-fill replay finishes under the policy snapshot taken at connect), while token expiry or claim changes take effect on reconnect. - -### Client-Side Stream Filtering - -On top of that, when a `QueryBuilder` with `.where()` filters or `.select()` columns calls `.stream()`, the returned stream applies those filters client-side: - -```ts -const stream = wh.from('clicks') - .select('page', 'button') - .where('page', '=', '/home') - .stream(); - -// Only events where page === '/home' are emitted, with only page + button columns -``` - -Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). - -Two of them need care, for different reasons. `like` matches **case-insensitively** here, while ClickHouse's `LIKE` on the query path is case-sensitive — so inside a single `liveQuery()` the backfill rows arrive through the server's case-sensitive filter and the live frames through this one, and a pattern like `'/Home%'` can admit live events whose historical counterparts the backfill excluded. `not_like` has no server-side counterpart at all: `/v1/query` rejects it with a `400` (see the operator table in [Queries](/sdk/queries)), so a `liveQuery()` filtered on it gets an error `Result` in `initial()`, drops whatever was buffered during the backfill window, and runs live-only from there. - ---- - -## Live Queries - -Live queries combine a historical backfill (`.fetch()`) with a real-time stream, providing a seamless initial load + live updates experience. - -```ts -const lq = wh.from('clicks') - .selectAll() - .where('page', '=', '/home') - .orderBy('received_timestamp', 'desc') - .limit(100) - .liveQuery({ - initial: (result) => { - // Called once with the historical data - setRows(result.data ?? []); - }, - next: (event) => { - // Called for each live event after backfill - addRow(event.data); - }, - error: (err) => console.error(err), - }); - -// Cleanup -lq.close(); -``` - -### `StreamSubscriber` - -```ts -interface StreamSubscriber { - initial?: (result: Result) => void; // Historical backfill result - next: (event: StreamEvent) => void; // Live events - status?: (state: StreamStatus) => void; // Connection state changes - error?: (err: WaveHouseError) => void; // Errors -} -``` - -### How it works - -1. Opens the stream **immediately** and buffers incoming events. -2. Runs the `.fetch()` query for historical data, calls `subscriber.initial()` with the result — unless the fetch itself throws, in which case neither happens (see below). -3. Deduplicates buffered events against the **last row** of the backfill result — which is the newest only when the query orders ascending. A `desc` query (like the example above) puts the oldest row last, so for live frames the boundary is the oldest timestamp and this step filters nothing. With a `since` gap-fill in flight it works the other way — replay lands in the same buffer, so replayed events at or older than that row are dropped before reaching `next()`. A projection that omits `received_timestamp` skips the pass entirely. Tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). -4. Flushes remaining buffered events and switches to live mode. - -This "stream-first" approach is what closes the window between the fetch and the stream starting — events arriving during the fetch are buffered rather than missed. It holds only when the backfill completes cleanly. - -#### When the backfill doesn't complete - -Here is how it doesn't, and what each case looks like from your subscriber. "Buffered events" means the ones that arrived during the fetch window, which step 4 would otherwise have flushed; the column reports how many of them reach you. - -| What happened | `initial()` | Live events | `error` | Buffered events | -|---|---|---|---|---| -| Backfill returns an error `Result` | fires, `result.error` set | delivered | — | none delivered | -| Your `initial()` throws | fires, then throws | delivered | — | none delivered — the flush never starts | -| Your `next()` throws during the flush | fires normally | delivered, after the flush stops | — | none after the throw | -| `auth` rejects, the stream connects anyway | never fires | delivered | — | none delivered | -| `auth` rejects for a few attempts, then recovers | never fires | delivered once it recovers | `SSE_AUTH_ERROR` per failed attempt | none delivered | -| `auth` keeps rejecting | never fires | none — never reaches `live` | `SSE_AUTH_ERROR` per attempt | none delivered | -| Relative `baseURL` | never fires | none — the stream is terminated too | terminal `SSE_CONNECT_ERROR` | none delivered | -| Your `status` handler throws on the first, synchronous call | never fires | none — buffered where you can't reach them | — | none delivered | - -**The `status`-throw row is the odd one out**, and everything below is written for the others. It is the one case where the backfill never *starts*: the throw escapes `liveQuery()` before the constructor reaches it, so you get no handle back, nothing can `.close()` the stream, and — because the buffering phase never ends — every event accumulates in an object you have no reference to, indefinitely. `next()` is never called at all. Passing `opts.signal` is the only way to stop it; not throwing is the fix. Throwing on any *later* `status` transition is isolated and logged like `next`. See [Error Handling](/sdk/reference#error-handling). - -**What to do about the rest.** Check `result.error` inside `initial()`, keep the flush handlers total, and — if a missing backfill matters — treat `initial()` never firing as its own failure. Where `auth` rejects and the stream connects anyway nothing else will tell you, and where the other rows *do* raise an error it names `auth` or the URL, never the backfill. Re-run the fetch then; you never have to work out which row you hit. Leave it a moment first: events reach a stream from the message queue before the ingest worker lands them in ClickHouse, and its per-table batcher flushes on size or a deadline with the insert still to complete after that (see [Ingest pipeline](/ingest-pipeline)), so an immediate re-fetch can miss the newest rows. - -**Why `auth` splits the way it does.** The backfill makes the *first* `auth()` call and gets exactly one shot — the token is minted above the REST retry loop, so a rejection there is never retried and the whole backfill is gone. The stream calls `auth()` again on every connection attempt and treats the same rejection as transient. That asymmetry is the entire reason a live query can end up running normally with no snapshot behind it, and why nothing announces it: `error` never fires, and the only trace is that `initial()` didn't. That is the case worth guarding against, because it is the one that looks like success. - -**What the live stream can lose.** In every row where the backfill runs at all: nothing, except in one window — events arriving after the stream goes `live` but before the backfill fails are buffered, and the failure discards the buffer. On the `auth` rows that window opens only if the rejection arrives *after* the connection does: the backfill fails as soon as its own `auth()` call rejects, while the stream needs a second `auth()` call resolved **and** a connection opened. Which way that goes depends on your token provider, and you don't need to know — the recovery above covers both. - -Tracked in [#473](https://github.com/Wave-RF/WaveHouse/issues/473). diff --git a/docs/src/content/docs/sdk/streaming.mdx b/docs/src/content/docs/sdk/streaming.mdx new file mode 100644 index 00000000..b97dfbe2 --- /dev/null +++ b/docs/src/content/docs/sdk/streaming.mdx @@ -0,0 +1,490 @@ +--- +title: "SDK Streaming & Live Queries" +description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse SDKs." +--- + +import { Tabs, TabItem } from "@astrojs/starlight/components"; + +Real-time consumption: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/queries). Both SDKs implement the same protocol and mostly the same client-side filtering; the lifecycles differ, and each tab says how. + +## Streaming + +Streams are Server-Sent Events over HTTP. The auth token rides in an `Authorization: Bearer` header — the same as every other request — and is re-read on each connection attempt, so a stream that outlives its token picks up a fresh one. Unauthenticated streams work the same way, minus the header. + +### The stream controller + +Returned by the stream method on a table ref, a query builder, a pipe ref, and the DLQ namespace (the DLQ variant is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). + + + + +`.stream()` returns a `StreamController`. It is **NOT thenable**. Streams run over `fetch` rather than `EventSource`, which is what lets them carry the auth header. + +```ts +const stream = wh.from('clicks').stream({ since: '2026-01-01T00:00:00Z' }); +``` + + + + +`.Stream(opts)` returns a `*StreamController` immediately; the connection opens in a background goroutine. Frames are parsed over `net/http` with no runtime dependencies. + +```go +stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() +``` + + + + +### Callback subscriptions + +Callback-based consumption. Subscribing returns a cleanup function that removes just that subscriber. + + + + +```ts +const unsub = stream.subscribe({ + next: (event) => { + // event: { table: 'clicks', timestamp: '2026-...', data: { page: '/', ... } } + console.log('New event:', event.data); + }, + status: (state) => { + // state: 'connecting' | 'live' | 'reconnecting' | 'closed' + updateIndicator(state); + }, + error: (err) => { + console.error('Stream error:', err.message); + }, +}); + +// Cleanup — closes the connection if no other subscribers remain +unsub(); +``` + +A handler that throws *during delivery* doesn't end the stream; the exception is logged and the connection keeps running. But delivery of that event stops at the handler that threw — your *later* subscribers, and any concurrent `for await`, do not get it. The first `status` call, the synchronous one `.subscribe()` makes before returning, isn't caught at all and throws back out at you. Wrap your handler bodies in your own `try`/`catch`; see [Error Handling](/sdk/reference#error-handling) for the carve-outs. + + + + +The `Status` callback fires immediately with the current status. + +```go +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { + // e: {Table: "clicks", Timestamp: "2026-...", Data: map[string]any{"page": "/", ...}} + fmt.Println("New event:", e.Data) + }, + Status: func(s wavehouse.StreamStatus) { + // s: StatusConnecting | StatusLive | StatusReconnecting | StatusClosed + updateIndicator(s) + }, + Error: func(err error) { + fmt.Println("Stream error:", err) + }, +}) + +// Removes this subscriber only — the connection stays open for any others +// and still needs stream.Close() when you're done with the stream itself. +defer unsub() +``` + + + + +### Iterating events + + + + +The controller is an async iterable. + +```ts +const stream = wh.from('clicks').stream(); + +for await (const event of stream) { + console.log(event.table, event.data); + if (shouldStop) break; // breaking auto-closes the stream +} +``` + + + + +`.Events()` returns a read-only channel, closed automatically when the stream shuts down. It is buffered (256 events) and buffers from `.Stream()` onward, so events arriving before the first `Events()` call are not lost. A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop; the first drop logs via `log`, later drops are silent, and `.Subscribe` callbacks fire regardless. + +```go +stream := wh.From("clicks").Stream(nil) +defer stream.Close() + +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) + if shouldStop { + break + } +} +``` + +:::caution[`break` does not close the stream] +Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. +::: + +:::note[`Events()` carries events only] +`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404), so pair `Events()` with a subscriber to learn why a stream ended. +::: + + + + +### Closing a stream + + + + +`stream.close()` explicitly closes the stream and releases all resources. + + + + +`stream.Close()` closes the stream and releases its resources. Non-blocking, and safe to call from inside a subscriber callback. + + + + +### Connection status + + + + +`stream.status` is the current status: `'connecting' | 'live' | 'reconnecting' | 'closed'`. + + + + +`stream.Status()` returns the current `StreamStatus`: `StatusConnecting`, `StatusLive`, `StatusReconnecting`, or `StatusClosed`. + + + + +### Waiting for the stream to go live + +Useful when you need the subscription established before doing something that depends on it — inserting a row you expect to see come back, for instance. + + + + +`.connected(timeoutMs?)` resolves once the stream reaches `live`. Rejects if it is already `closed`, if it closes before connecting, or after `timeoutMs` (default `5000`). + +```ts +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ next: (e) => console.log(e) }); +await stream.connected(); // wait until the transport is live +await wh.from('clicks').insert({ page: '/home' }); +``` + +A *timeout* rejection does **not** stop the transport — reconnection is unbounded, so it means "not live yet", not "given up"; call `.close()` if you want it to stop. A rejection because the stream closed is different: there the transport has already stopped. + +One way this rejects against a perfectly healthy stream: if a subscriber you registered *before* calling `.connected()` has a `status` handler that throws, the throw aborts the fan-out before `connected()`'s internal watcher sees `live`, so it times out while `.status` already reads `live`. See [Error Handling](/sdk/reference#error-handling). + + + + +`.Connected(ctx)` blocks until the stream reaches `StatusLive` or `ctx` is canceled; it returns an error if the stream closes before connecting. The deadline is yours to set — there is no built-in default. + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() +if err := stream.Connected(ctx); err != nil { + log.Fatal(err) +} +``` + + + + +### Stream options + + + + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `since` | `string` | RFC3339 timestamp for gap-fill replay | +| `signal` | `AbortSignal` | Cancel/close the stream. Wired via `attachSignal()` internally. | + + + + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `Since` | `string` | RFC3339 timestamp for gap-fill replay | + +There is no `Signal`/context field: a stream is not canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. + + + + +### The event shape + + + + +```ts +interface StreamEvent { + table: string; // table name (e.g. 'clicks') + timestamp: string; // received_timestamp (RFC3339Nano) + data: T; // row data +} +``` + +Row values of top-level `DateTime`/`DateTime64` columns inside `data` (not timestamps nested in `Array`/`Map`/`Tuple` columns) arrive in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), matching what `/v1/query` returns for the same row — `new Date(value)` parses correctly with no zone fix-up. Values WaveHouse couldn't canonicalize (ingest is fail-open) stream in the producer's original spelling, and the `/v1/query` match doesn't hold for them: a spelling ClickHouse accepted anyway still queries back in canonical UTC (one it rejected never lands in the table at all), and a zone-less date-time is what `new Date()` reads as *local* time — though a date-only `YYYY-MM-DD` string is read as UTC, an ECMAScript quirk (see [Timestamp canonicalization](/api#timestamp-canonicalization)). + + + + +```go +type StreamEvent struct { + Table string // table name (e.g. "clicks") + Timestamp string // received_timestamp (RFC3339Nano) + Data map[string]any // row data +} +``` + +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value, because the ingest handler rewrites them before publishing — a live frame and a later query can't disagree on the spelling of an instant. So a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open: a value the server can't parse, or whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). + + + + +### Transport behavior + +A dropped stream reconnects on a jittered exponential backoff, capped at 30s, and resumes with `Last-Event-ID` so the server gap-fills what was missed. HTTP/2 is recommended. Reconnection is **unbounded**: a stream keeps re-dialing until it hits a terminal error or you close it. The client option that bounds retries applies to REST requests only and is never consulted here. + +:::caution[Resumption is at-least-once, and time-bounded] +Delivery across a reconnect is **at-least-once**. The `Last-Event-ID` the client sends is the last event's `received_timestamp`, and the server replays from that instant *inclusively* — so the last event you already saw, and anything sharing its timestamp, arrives again. Neither SDK deduplicates live frames — a live query makes one pass at the backfill seam only — so key on `timestamp` plus your own row identity if duplicates matter. + +Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration#message-queue-nats) — 15 minutes by default. A drop longer than that resumes with a hole and no signal, because the purged messages are simply gone. +::: + + + + +The backoff schedule resets only after a connection has *held* for a few seconds, so a server accepting and immediately closing — slow-consumer eviction, a half-broken upstream — can't pin the client at sub-second retries. + +:::note[SSE connection limit] +The SDK warns above 5 concurrent SSE connections — just under the browser's 6-per-domain limit for HTTP/1.1. HTTP/2 multiplexes over one connection, so the limit doesn't apply there. +::: + +A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the only 4xx it raises itself is `400` for a missing or empty table name, and only on the stream route — a `404` or `405` means the request never reached it, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. + +Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request — which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk/typescript#supplying-your-own-fetch). + + + + +Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect — the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/reference#error-handling). + +`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Creating a client](/sdk/go#creating-a-client)). + + + + +### Server-side policy filtering + +Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims — on live frames and gap-fill replay alike. Claims are captured when the connection opens: a policy change applies from the next live event (an in-flight gap-fill replay finishes under the policy snapshot taken at connect), while token expiry or claim changes take effect on reconnect. + +Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control — where each rule is enforced](/access-control#where-each-rule-is-enforced). + +### Client-side stream filtering + +On top of that, when a query builder carrying filters or a column projection opens a stream, the returned stream applies those filters client-side. + + + + +```ts +const stream = wh.from('clicks') + .select('page', 'button') + .where('page', '=', '/home') + .stream(); + +// Only events where page === '/home' are emitted, with only page + button columns +``` + +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). + +Two of them need care, for different reasons. `like` matches **case-insensitively** here, while ClickHouse's `LIKE` on the query path is case-sensitive — so inside a single `liveQuery()` the backfill rows arrive through the server's case-sensitive filter and the live frames through this one, and a pattern like `'/Home%'` can admit live events whose historical counterparts the backfill excluded. `not_like` has no server-side counterpart at all: `/v1/query` rejects it with a `400` (see the operator table in [Queries](/sdk/queries#filtering)), so a `liveQuery()` filtered on it gets an error `Result` in `initial()`, drops whatever was buffered during the backfill window, and runs live-only from there. + + + + +```go +stream := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Stream(nil) + +// Only events where page == "/home" are emitted, with only page + button fields +``` + +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere. `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively, and `OpIn` accepts any Go slice type (e.g. `[]string`, `[]int`). + +**How values are compared.** The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: + +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload may read `2026-06-21T04:00:00Z` while your filter constant names the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a chronologically equal row. Both sides are parsed as instants instead. +- **Only unambiguous spellings count as instants:** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have, so it is not treated as a timestamp. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. +- **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. + +:::caution[Integer precision above 2^53] +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond 2^53 has already lost exactness before any filter runs, and the server compares such columns in their exact storage domain — so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +::: + + + + +--- + +## Live Queries + +Live queries combine a historical backfill with a real-time stream, providing a seamless initial load + live updates experience. They exist only on the query builder — there is no table-ref shortcut in either SDK. + + + + +```ts +const lq = wh.from('clicks') + .selectAll() + .where('page', '=', '/home') + .orderBy('received_timestamp', 'desc') + .limit(100) + .liveQuery({ + initial: (result) => { + // Called once with the historical data + setRows(result.data ?? []); + }, + next: (event) => { + // Called for each live event after backfill + addRow(event.data); + }, + error: (err) => console.error(err), + }); + +// Cleanup +lq.close(); +``` + +```ts +interface StreamSubscriber { + initial?: (result: Result) => void; // Historical backfill result + next: (event: StreamEvent) => void; // Live events + status?: (state: StreamStatus) => void; // Connection state changes + error?: (err: WaveHouseError) => void; // Errors +} +``` + + + + +```go +lq := wh.From("clicks"). + SelectAll(). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Called once with the historical backfill. + setRows(rows) + }, + Next: func(e wavehouse.StreamEvent) { + // Called for each live event after backfill. + addRow(e.Data) + }, + Error: func(err error) { + log.Println(err) + }, + }, nil) + +defer lq.Close() +``` + +```go +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} +``` + +:::note[`Initial` is always untyped] +Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even where you would use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. +::: + +`lq.Close()` shuts down the live query and its underlying stream. Idempotent (guarded by `sync.Once`). + + + + +### How it works + +1. Opens the stream **immediately** and buffers incoming events. +2. Runs the backfill query for historical data and hands the result to the `initial` callback. +3. Deduplicates buffered events against the backfill. +4. Flushes remaining buffered events and switches to live mode. + +This "stream-first" approach is what closes the window between the fetch and the stream starting — events arriving during the fetch are buffered rather than missed. It holds only when the backfill completes cleanly. + + + + +Step 2 is skipped entirely if the fetch itself throws (see below). Step 3 deduplicates against the **last row** of the backfill result — which is the newest only when the query orders ascending. A `desc` query (like the example above) puts the oldest row last, so for live frames the boundary is the oldest timestamp and this step filters nothing. With a `since` gap-fill in flight it works the other way — replay lands in the same buffer, so replayed events at or older than that row are dropped before reaching `next()`. A projection that omits `received_timestamp` skips the pass entirely. Tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). + +#### When the backfill doesn't complete + +Here is how it doesn't, and what each case looks like from your subscriber. "Buffered events" means the ones that arrived during the fetch window, which step 4 would otherwise have flushed; the column reports how many of them reach you. + +| What happened | `initial()` | Live events | `error` | Buffered events | +|---|---|---|---|---| +| Backfill returns an error `Result` | fires, `result.error` set | delivered | — | none delivered | +| Your `initial()` throws | fires, then throws | delivered | — | none delivered — the flush never starts | +| Your `next()` throws during the flush | fires normally | delivered, after the flush stops | — | none after the throw | +| `auth` rejects, the stream connects anyway | never fires | delivered | — | none delivered | +| `auth` rejects for a few attempts, then recovers | never fires | delivered once it recovers | `SSE_AUTH_ERROR` per failed attempt | none delivered | +| `auth` keeps rejecting | never fires | none — never reaches `live` | `SSE_AUTH_ERROR` per attempt | none delivered | +| Relative `baseURL` | never fires | none — the stream is terminated too | terminal `SSE_CONNECT_ERROR` | none delivered | +| Your `status` handler throws on the first, synchronous call | never fires | none — buffered where you can't reach them | — | none delivered | + +**The `status`-throw row is the odd one out**, and everything below is written for the others. It is the one case where the backfill never *starts*: the throw escapes `liveQuery()` before the constructor reaches it, so you get no handle back, nothing can `.close()` the stream, and — because the buffering phase never ends — every event accumulates in an object you have no reference to, indefinitely. `next()` is never called at all. Passing `opts.signal` is the only way to stop it; not throwing is the fix. Throwing on any *later* `status` transition is isolated and logged like `next`. See [Error Handling](/sdk/reference#error-handling). + +**What to do about the rest.** Check `result.error` inside `initial()`, keep the flush handlers total, and — if a missing backfill matters — treat `initial()` never firing as its own failure. Where `auth` rejects and the stream connects anyway nothing else will tell you, and where the other rows *do* raise an error it names `auth` or the URL, never the backfill. Re-run the fetch then; you never have to work out which row you hit. Leave it a moment first: events reach a stream from the message queue before the ingest worker lands them in ClickHouse, and its per-table batcher flushes on size or a deadline with the insert still to complete after that (see [Ingest pipeline](/ingest-pipeline)), so an immediate re-fetch can miss the newest rows. + +**Why `auth` splits the way it does.** The backfill makes the *first* `auth()` call and gets exactly one shot — the token is minted above the REST retry loop, so a rejection there is never retried and the whole backfill is gone. The stream calls `auth()` again on every connection attempt and treats the same rejection as transient. That asymmetry is the entire reason a live query can end up running normally with no snapshot behind it, and why nothing announces it: `error` never fires, and the only trace is that `initial()` didn't. That is the case worth guarding against, because it is the one that looks like success. + +**What the live stream can lose.** In every row where the backfill runs at all: nothing, except in one window — events arriving after the stream goes `live` but before the backfill fails are buffered, and the failure discards the buffer. On the `auth` rows that window opens only if the rejection arrives *after* the connection does: the backfill fails as soon as its own `auth()` call rejects, while the stream needs a second `auth()` call resolved **and** a connection opened. Which way that goes depends on your token provider, and you don't need to know — the recovery above covers both. + +Tracked in [#473](https://github.com/Wave-RF/WaveHouse/issues/473). + + + + +Step 2 runs `.FetchUntyped(ctx)` and then calls `sub.Initial(rows, err)`. Step 3 deduplicates against the maximum `received_timestamp` in the backfill (not necessarily the last row). + +:::caution[Dedup needs `received_timestamp` in the projection] +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, so events in the overlap window are delivered twice — once via `Initial`, once via `Next`. +::: + +:::caution[`OpLike` matching differs between backfill and live] +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback — see [Queries → Filtering](/sdk/queries#filtering). +::: + + + diff --git a/docs/src/content/docs/sdk/typescript.mdx b/docs/src/content/docs/sdk/typescript.mdx new file mode 100644 index 00000000..d2edbe56 --- /dev/null +++ b/docs/src/content/docs/sdk/typescript.mdx @@ -0,0 +1,565 @@ +--- +title: "TypeScript SDK setup" +description: "Installing @wavehouse/sdk, creating a client, custom headers and fetch, and the Result type." +--- + +import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components"; + +Setup and caveats for `@wavehouse/sdk`, the TypeScript client: installation, runtimes, client construction, and the `Result` error model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing Go instead? [Go setup](/sdk/go) is the mirror of this page. + +## Installation + + + + +```bash +pnpm add @wavehouse/sdk +``` + + + + +```bash +npm install @wavehouse/sdk +``` + + + + +```bash +yarn add @wavehouse/sdk +``` + + + + +```bash +bun add @wavehouse/sdk +``` + + + + +```bash +deno add npm:@wavehouse/sdk +``` + + + + No bundler or package manager required — the ES module loads natively in modern browsers: + +```html + +``` + +Pin a version for production (`https://esm.sh/@wavehouse/sdk@0.1.0`); jsDelivr (`.../+esm`) and unpkg (`?module`) serve the same module. For pages that can't use ES modules, the bundled IIFE build at `https://cdn.jsdelivr.net/npm/@wavehouse/sdk` exposes a `WaveHouse` global (`WaveHouse.createClient({ … })`) for a classic ` + +``` + +### Runtime support + +**Browsers** — all SDK features work natively. `fetch`, `AbortController`, and `ReadableStream` are built into every modern browser; no polyfills are required. + +**Node.js** — every feature, streaming included, works in Node 22 and later (the package's minimum, per `engines.node`). 22 is the only line we test against, and older releases are past end-of-life upstream. Streaming runs on the same `fetch` and `ReadableStream` globals as the rest of the SDK, so it needs **no `EventSource` polyfill** — earlier versions did, and that requirement is gone. + +## Quick Start + + + + +```ts +import { createClient } from '@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +```ts +import { createClient } from '@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +```ts +import { createClient } from '@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +```ts +import { createClient } from '@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +```ts +import { createClient } from '@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +```ts +import { createClient } from 'https://esm.sh/@wavehouse/sdk'; + +const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated +}); + +// Query +const { data, error } = await wh.from('clicks').select('page').limit(10); + +// Insert +await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + +// Stream +const stream = wh.from('clicks').stream(); +const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), +}); +``` + + + + +## Creating a Client + +```ts +import { createClient } from '@wavehouse/sdk'; +import type { Database } from './my-types'; // optional hand-written types + +const wh = createClient({ + baseURL: 'https://.wavehouse.app', + auth: async () => myAuthProvider.getToken(), + options: { + maxRetries: 2, + }, +}); +``` + +### `ClientConfig` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `baseURL` | `string` | — | WaveHouse server URL, optionally including a path prefix (required) | +| `auth` | `() => Promise \| string` | — | Token provider. Omit for public access | +| `options.maxRetries` | `number` | `2` | Retry attempts for failed/5xx **REST** requests; stream reconnects are unbounded ([details](/sdk/streaming#transport-behavior)) | +| `options.headers` | `Record` | — | Headers added to every request ([details](#custom-headers)) | +| `options.fetchOptions` | `RequestInit` | — | Extra `RequestInit` fields merged into every request ([details](#extra-requestinit-fields)) | +| `options.fetch` | `FetchLike` | global `fetch` | HTTP implementation for every request ([details](#supplying-your-own-fetch)) | + +:::note[How the token is transmitted] +The SDK attaches your `auth` token as an `Authorization: Bearer` header on every request, streaming included. It is re-read on each connection attempt, so a stream that outlives its token reconnects with a fresh one instead of replaying an expired credential. + +The token is never placed in the URL. The server still accepts a `?token=` query parameter for clients that cannot set headers — see [API Reference — Authentication](/api#authentication) — but the SDK does not use it, so it never reaches a proxy or CDN access log. +::: + +:::caution[Streaming asks more of a custom `fetch`] +`.stream()` and `.liveQuery()` read the response as it arrives, so an `options.fetch` used with them must return a response with a live streaming body. A response handed back with an absent or already-consumed body fails fast with `SSE_NO_STREAM_BODY`. What the SDK cannot rescue is a wrapper that *awaits* the body before returning — `await res.text()`, or the `res.clone().text()` a logging wrapper reaches for — because on a stream that never ends it never resolves and your function never returns. Both satisfy every REST call, which is what makes the trap easy to walk into. + +`options.headers` is also subject to CORS in a browser: a header outside the safelist joins the preflight, which the origin has to allow. WaveHouse advertises a fixed set, so custom headers reach it cross-origin only when a proxy in front terminates the preflight — the deployment they exist for. Server-side callers never preflight. +::: + +#### Serving under a path prefix + +`baseURL` may carry a path prefix, for a WaveHouse reachable somewhere other than the root of an origin — behind a backend-for-frontend (BFF), an app-server route, or a path-routed ingress. Every request path is appended to it, on both transports: + +```ts +const wh = createClient({ baseURL: 'https://app.example.com/api/wavehouse' }); +// queries → https://app.example.com/api/wavehouse/v1/query +// streams → https://app.example.com/api/wavehouse/v1/stream +``` + +Trailing slashes on `baseURL` are optional, but it must be **absolute** — scheme and host included — and the scheme must be `http` or `https`. A same-origin relative path like `/api/wavehouse` makes every REST call reject with a `TypeError` instead of returning a `Result`, and surfaces on a stream as an `SSE_CONNECT_ERROR` to the subscriber's `error` callback — so build an absolute one: `` `${location.origin}/api/wavehouse` ``. A `ws://`/`wss://` base ends the stream with the same error: streaming is plain HTTP, and there is no WebSocket endpoint. + +The proxy in front must **strip** the prefix before forwarding, since WaveHouse itself always serves at `/v1/…` — see [Behind a reverse proxy → Path prefixes](/reverse-proxy#path-prefixes). + +### Custom headers + +`options.headers` adds headers to every request, streaming included — the usual reason being a header-gated proxy in front of WaveHouse, such as a Cloudflare Access service token: + +```ts +const wh = createClient({ + baseURL: 'https://.wavehouse.app', + options: { + headers: { + 'CF-Access-Client-Id': process.env.CF_ACCESS_CLIENT_ID!, + 'CF-Access-Client-Secret': process.env.CF_ACCESS_CLIENT_SECRET!, + }, + }, +}); +``` + +Header names are matched **case-insensitively**, as HTTP requires, so `authorization` and `Authorization` are the same header rather than two. + +Your headers are applied *underneath* the SDK's own, and a collision means yours is dropped rather than merged: + +- **`auth` keeps `Authorization`.** Setting it here won't displace your token provider. Use `auth` for credentials. +- **`Content-Type` and `Accept` belong to the request.** The SDK knows what it's sending; a global `Content-Type` that outranked it would break requests whose body isn't JSON. + +Nothing is ever comma-joined: on a collision the SDK's value stands alone, and two of your own entries differing only in case collapse to the last one — a header joined rather than replaced is how you end up sending `Content-Type: application/json, image/png`. + +From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`) with no config knob. So custom headers work server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. + +Headers are static. For a credential that rotates per request, wrap the transport with [`options.fetch`](#supplying-your-own-fetch); a callback form is tracked in [#459](https://github.com/Wave-RF/WaveHouse/issues/459). + +### Extra `RequestInit` fields + +`options.fetchOptions` is merged into the `RequestInit` of every request, streaming included — for settings that aren't headers and don't warrant replacing the whole transport: + +```ts +const wh = createClient({ + baseURL: 'https://wavehouse.example.com', + options: { fetchOptions: { cache: 'no-store', mode: 'cors' } }, +}); +``` + +It carries any `RequestInit` field — `mode`, `cache`, `keepalive`, `credentials`, `redirect` — plus runtime-specific extensions such as Next.js's `next: { tags }`, which tags the cached response for on-demand revalidation. Only those runtime extensions need a cast; the standard fields are declared on `RequestInit` already. + +:::caution[Two of those fields don't apply to streams] +On `.stream()` and `.liveQuery()`'s live connection the transport keeps `cache` and `redirect` for itself, because there they're correctness rather than preference. It constrains one more without owning it: `credentials` stays yours, forwarded in a browser and dropped elsewhere because some runtimes throw if it is set at all. See [Supplying your own fetch](#supplying-your-own-fetch) for what each is set to and why. + +Everything else reaches streams normally — including `mode` from the example above. Only `cache` is different there: on a stream it is already `no-store`. +::: + +:::note[`credentials: 'include'` needs a proxy that owns CORS] +Sending cookies cross-origin is a common reason to reach for this, but it can't work against WaveHouse's own CORS: it deliberately never emits `Access-Control-Allow-Credentials` (it's a Bearer-token API), and the default `cors_allowed_origins: "*"` makes `include` a hard browser failure regardless. It applies where a fronting proxy answers CORS itself — see [Behind a reverse proxy](/reverse-proxy#header-and-auth-forwarding). Same-origin deployments already send cookies without it. +::: + +The fields the SDK controls — `method`, `headers`, `body`, and `signal` — always win, so this can't corrupt the request itself. In particular `headers` here is ignored; use `options.headers`, which merges properly. + +### Supplying your own `fetch` + +`options.fetch` replaces the HTTP implementation the SDK uses. Reach for it to route through a proxy, attach client certificates, wrap requests in your own middleware (logging, tracing, circuit breaking), stub HTTP in your own tests without monkey-patching a global, or to work around transport behavior of the runtime you happen to be on: + +```ts +const wh = createClient({ + baseURL: 'https://.wavehouse.app', + options: { + fetch: async (url, init) => { + const started = performance.now(); + const res = await fetch(url, init); + console.log(res.status, url, `${Math.round(performance.now() - started)}ms`); + return res; + }, + }, +}); +``` + +Retries go through the same function, so middleware sees every attempt. + +The SDK always calls your function with a **string** URL and a plain `RequestInit`. Off the response the **REST path** reads `.ok` and `.headers` always, `.text()` on success, and `.status`, `.statusText` plus `.json()` when the response is not `ok` — so a hand-rolled response object needs all six there. A stub used with `.stream()` or `.liveQuery()` needs a different set: `.ok`, `.status`, `.type` (checked for `opaqueredirect`), `.headers` carrying `content-type: text/event-stream`, and a live, unread `.body` — plus `.statusText` and `.json()` if your stub ever answers a stream with a non-`ok` response, which goes through the same error parsing as REST. `.text()` is never called on a stream. + +**On REST**, if your function rejects, the request becomes a `NETWORK_ERROR` result and is retried with backoff — unless the `AbortSignal` you passed has been aborted, in which case the result is `ABORTED` and nothing is retried. That's decided from the signal rather than from what your function threw, so an implementation that signals abort some other way — `AbortSignal.timeout()` raises a `TimeoutError`, `node-fetch` its own `AbortError` class — is reported the same way. On a stream the same rejection is reported to the subscriber as `SSE_NETWORK_ERROR` and re-dialed — including an `AbortError`, since the stream transport applies the same rule: only an abort *it* raised, via `.close()`, ends the stream. See [Error Handling](/sdk/reference#error-handling). + +The option's type is exported as `FetchLike`. It is written out rather than as `typeof fetch`, which resolves differently depending on whether your TypeScript `lib` includes DOM. Its URL parameter is `string` — all the SDK ever passes — which, because function parameters are contravariant, accepts *more* implementations than the wider spelling would: the global `fetch` fits, and so does hand-written `(url: string, init?: RequestInit) => Promise` middleware. Let TypeScript infer your parameters (`(url, init) => …`) and it fits without annotation. + +Streaming goes through your function too. That comes with the extra requirement described above — the response must carry a live streaming body — and the streaming transport keeps ownership of two `RequestInit` fields where the value is load-bearing rather than a preference: `cache` (as an init field; a `Cache-Control` header would fail cross-origin preflight) and `redirect`. It constrains one more without owning it: `credentials` stays yours, forwarded in a browser and dropped elsewhere, because some runtimes throw if it is set at all. + +`redirect` depends on whether the request carries a credential. With one — an `auth` token or configured `headers` — a redirect is refused and reported as `SSE_REDIRECT`, because a cross-origin hop strips `Authorization` (which this endpoint answers with a silently reduced view rather than an error) while forwarding your other headers to wherever the redirect points. Without either, redirects are followed normally, so CDN canonicalization, geo/LB indirection, and an http→https upgrade all work. The test is that concrete pair rather than "is this request authenticated", and **cookies are not part of it**. A cookie isn't stripped the way `Authorization` is — it is re-derived from the cookie store at each hop, so a redirect carries it only while the hop stays inside **both** the request's origin and the cookie's own `Path`. Leaving either loses it. The origin half is the one no cookie attribute can override: under the default `same-origin` credentials mode the browser attaches nothing once a redirect goes cross-origin, whatever the `Domain` — so `app.example.com` → `api.example.com` on a shared `Domain=example.com` session cookie loses it, and so does an http→https upgrade, which is cross-origin by scheme and port. A same-origin rewrite from `/v1/stream` to `/stream` under a `Path=/v1` cookie loses it the other way. This endpoint answers a cookieless request with a reduced view rather than an error ([#478](https://github.com/Wave-RF/WaveHouse/issues/478)). Going cross-origin with `credentials: "include"` usually fails earlier and louder instead: WaveHouse never sends `Access-Control-Allow-Credentials`, so against it the browser blocks the response outright. Behind a proxy that owns CORS and does allow credentials the hop succeeds — and then ordinary cookie scoping decides. `include` *replaces* the origin rule rather than lifting it: the cookie crosses only if its own `Domain` covers the target, so a host-only cookie — what you get with no `Domain` attribute — still stops at its host. If you need to follow a credentialed redirect anyway, supply an `options.fetch` that overrides `redirect`. A `200` that isn't `text/event-stream` is refused too (`SSE_BAD_CONTENT_TYPE`), so an auth gateway's login page fails loudly instead of leaving the stream connected and permanently silent. + +#### Swapping in undici + +The motivating case is a transport bug in the runtime's own HTTP stack, which you can't fix from inside the SDK — for example [undici #5600](https://github.com/nodejs/undici/issues/5600): reusing a keep-alive socket while the event loop is idle stalls the request before it goes out. How bad it gets varies with the runtime and the idle gap: the upstream report measured ~450–465 ms against a 10 ms server, and we have measured anything from ~100 ms to tens of seconds on a server answering instantly. It affects undici 8.8.0–8.9.0, and Node 26 bundles 8.9.0. + +**Upgrading undici is the actual fix** — it landed in 8.10.0. What `options.fetch` buys you is a way to get there without waiting for a new runtime: install undici yourself and route requests through it. The one non-obvious part is that you must pass its dispatcher **explicitly**. + +```ts +import { createClient } from '@wavehouse/sdk'; +import { Agent, fetch as undiciFetch } from 'undici'; // npm install undici — 8.10.0+ + +// Explicit, not implied — see below. +const dispatcher = new Agent(); + +const wh = createClient({ + baseURL: 'https://.wavehouse.app', + options: { + fetch: (url, init) => + undiciFetch(url, { ...init, dispatcher } as never) as unknown as Promise, + }, +}); +``` + +:::caution[Importing a fixed undici is not enough on its own] +undici keeps its connection pool on a shared `globalThis` symbol, and whichever copy loads first claims it. Node claims it for the *bundled* copy the first time anything touches one of its web globals — a `fetch()` call, but equally a `new Headers()` or `new Response()` — not at startup. So which undici owns the pool comes down to a load order you don't really control, and auditing your own code for `fetch` calls won't tell you: any dependency can claim it first, and so can this SDK, which constructs a `Headers` on its abort and retry-exhausted paths. Call `undiciFetch` without a `dispatcher` and it resolves whatever is on that symbol, so requests can go through the buggy pool even though you imported a fixed undici. + +Measured with 8.9.0 loaded first and 8.10.0's `fetch` doing the request, 1.5 s idle gaps against a 10 ms server: + +```text + per-request ms +no explicit dispatcher 21 1514 1495 583 ← still stalling +explicit new Agent() 17 14 12 13 +``` + +An explicit `dispatcher` wins because the request never consults the shared symbol at all. An explicit `setGlobalDispatcher` call (below) wins the other way round — it overwrites the symbol after the first copy claimed it. +::: + +Both casts are load-bearing, for one underlying reason: undici declares its own request/response types, separate from the ones behind your global `fetch`, so the two aren't structurally assignable. `{ ...init, dispatcher } as never` covers the split on `RequestInit`, which differs on `body` and `headers` (`never` is assignable to either spelling, so one snippet compiles whether or not your `lib` includes DOM); and the return cast handles the `Response` mismatch. The URL needs no cast — `FetchLike` declares it as `string`, which undici's `RequestInfo` accepts. They're safe because of the narrow runtime contract above — a string URL and a plain `RequestInit` in, and only the response members enumerated above read back. `undici` is a dependency you add on top of the SDK's single ~1.4 KB one. + +If you're genuinely pinned to an affected undici, the same snippet fixes it — your pinned version, with a dispatcher that never reuses a keep-alive socket: `new Agent({ pipelining: 0 })`. That costs a fresh connection per request, so prefer upgrading. Note that tuning `keepAliveTimeout` does **not** help: the retirement timer is starved by the same idle event loop that triggers the bug, so the socket is still there to be reused. + +To change the transport process-wide instead of per-client, an installed undici's `setGlobalDispatcher(new Agent())` is picked up by Node's built-in global `fetch`, no `options.fetch` needed. That's an explicit write, so it wins the race described above — but only if the *installed* undici is 8.10.0+, since it hands connection handling to that copy. + +### Type-Safe Tables + +Pass a `Database` type to get autocomplete on table names and row types: + +```ts +interface Database { + clicks: { page: string; button: string; score: number; received_timestamp: string }; + users: { id: string; name: string; email: string }; +} + +const wh = createClient({ baseURL: '...' }); +const clicks = wh.from('clicks'); // ✅ autocomplete +const { data } = await clicks.select('page', 'button').limit(10); +// data is Array<{ page: string; button: string; score: number; received_timestamp: string }> | null +``` + +Generate the `Database` interface from a running server with the [codegen CLI](/sdk/reference#codegen-cli). + +## Result Type + +Every async SDK operation returns `Result` — a discriminated union that never throws for anything the server returns (caller and environment errors do throw — see [Error Handling](/sdk/reference#error-handling)): + +```ts +type Result = + | { ok: true; data: T; error: null; hasMore?: boolean; next?: () => Promise> } + | { ok: false; data: null; error: WaveHouseError } + +interface WaveHouseError { + status: number; // HTTP status (0 for network errors) + code: string; // e.g. 'HTTP_400', 'NETWORK_ERROR', 'ABORTED' + message: string; // Human-readable error message + details?: unknown; // Raw response body + retryable: boolean; // Whether SDK would retry this error +} +``` + +Usage pattern — branch on `result.ok` (or destructure `{ data, error }` if you prefer): + +```ts +const result = await wh.from('clicks').select('page').limit(10); +if (result.ok) { + console.log(result.data); // Row[] — TypeScript knows data is non-null here +} else { + console.error(result.error.message); // never throws +} +``` + +The full error-code table lives in [Error Handling](/sdk/reference#error-handling). + +## Where to go next + +The topic pages cover both SDKs, tabbed by language — the tab you pick here follows you across all of them. + + + + + + + + + diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 8aaf5798..7d64526f 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -157,7 +157,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, one dependency, codegen) + `github.com/Wave-RF/WaveHouse/clients/go` (Go, zero dependencies, codegen) | +| Client SDK | Each team writes one | Official [TypeScript and Go clients](/sdk) — query builder, live queries, codegen | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. diff --git a/scripts/cov/main.go b/scripts/cov/main.go index f432d1b1..007c821e 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -69,6 +69,19 @@ var standaloneGoSuites = []string{"go-sdk"} // paths through the module in its working directory, so it must run there. var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} +// suiteTarget names the make target that populates a suite, for the +// "did you run it?" hint. Only suites whose target isn't `test-` +// need an entry. +var suiteTarget = map[string]string{"go-sdk": "test-sdk-go"} + +// makeTargetFor returns the make target that populates the named suite. +func makeTargetFor(suite string) string { + if t, ok := suiteTarget[suite]; ok { + return t + } + return "test-" + suite +} + // TypeScript SDK suites (vitest). ts-unit comes from clients/ts; ts-e2e // from tests/e2e/sdk run with --coverage. Both produce Istanbul-format // coverage-final.json that `cov ts-merge` combines into ts-total. @@ -225,7 +238,7 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int dir := filepath.Join(root, suite) dataDir := filepath.Join(dir, "data") if !hasCovdata(dataDir) { - return nil, 0, 0, "", fmt.Errorf("no covdata in %s — did make test-%s run?", dataDir, suite) + return nil, 0, 0, "", fmt.Errorf("no covdata in %s — did make %s run?", dataDir, makeTargetFor(suite)) } profile := filepath.Join(dir, "coverage.txt") htmlOut = filepath.Join(dir, "coverage.html") @@ -385,7 +398,7 @@ func merge(c *config) error { // // Layout under tmp/coverage/: // -// ts-unit/coverage-final.json ← `make test-ts` +// ts-unit/coverage-final.json ← `make test-sdk-ts` // ts-e2e/coverage-final.json ← `make test-e2e` // ts-merge-input/ ← scratch dir (both renamed json files) // ts-total/ ← merged JSON + requested reports @@ -469,11 +482,11 @@ func mergeTS(c *config) error { filepath.Join(root, name, "coverage-final.json")) } else { fmt.Printf(" %s✗%s %-9s (no coverage-final.json; run `make %s` to include)\n", - yellow, reset, name, ternary(name == "ts-unit", "test-ts", "test-e2e")) + yellow, reset, name, ternary(name == "ts-unit", "test-sdk-ts", "test-e2e")) } } if len(merged) == 0 { - fmt.Printf(" %sno TS coverage data — skipping ts-merge (run `make test-ts` and/or `make test-e2e` to populate)%s\n", yellow, reset) + fmt.Printf(" %sno TS coverage data — skipping ts-merge (run `make test-sdk-ts` and/or `make test-e2e` to populate)%s\n", yellow, reset) return nil } From 23c58e2314a9bd2acdf9d90c0273a6b83d5bb894 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 25 Aug 2026 13:53:29 -0400 Subject: [PATCH 52/59] fix(sdk): address pre-push review findings on the SDK docs merge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs review (1 MUST, 2 SHOULD, 4 MAY): - MUST: the topic-page merge hoisted `Last-Event-ID` into shared prose, claiming it for both clients. Only the TS SDK sends that header (clients/ts/src/stream/sse.ts:393); Go folds the last event id back into `?since=` (clients/go/stream.go:246,294). Same server behavior — internal/api/stream.go prefers the header and falls back to since — but wrong for anyone configuring a gateway header allow-list or reading a capture. Both spellings are now named. - development.md's file tree still said "hence the *-go-sdk make targets"; no target matches that glob any more. - typescript.mdx had a dangling back-reference to "the SDK's single ~1.4 KB one" — the concision pass removed its antecedent, leaving the TS setup page never stating the dependency. Restored there, which is where per-language detail belongs; the overview pages stay trimmed. - Restored the builder-level "Streaming from a builder" pointer, the one substantive thing the merge dropped. - `failed === 0` was JS syntax sitting in language-neutral prose. - go.md linked "Reference → Context Cancellation" at a heading this PR renamed to "Cancellation". - The Go live-query tab said the stream-first guarantee "holds only when the backfill completes cleanly" and never said what happens when it doesn't. Code review (0 MUST, 3 SHOULD, 2 MAY): - test-sdk-go was the only Go suite target with no -timeout, on the most concurrent code in the repo under -race — a deadlocked SSE controller would have burned half of CI's 20-minute unit cap before dumping goroutines. Pinned to 60s, matching test-sdk-go-e2e. - .testcoverage.yml's go-sdk floor was justified by "75 vs 82.7% measured; the headroom is the codegen CLI" — staled by this PR's own e17152f, which added those tests. Now 87.1% (SDK 88.5%, codegen 81.0%), and the comment says why the floor stays a regression floor rather than a target. - scripts/cov had three open-coded suite->target hints, two of which this PR hand-edited for the rename. All three now route through makeTargetFor, so the next rename touches one map; `ternary` was its last caller and goes with it. The map gains ts-e2e -> test-e2e, which is not test-. - Two comments still pointed at scripts/coverage.sh, which is scripts/cov now. The PR description is stale for the same reasons and is updated separately. Refs #434 --- .testcoverage.yml | 7 +++++-- Makefile | 4 ++-- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go.md | 4 ++-- docs/src/content/docs/sdk/queries.mdx | 6 +++++- docs/src/content/docs/sdk/streaming.mdx | 6 ++++-- docs/src/content/docs/sdk/typescript.mdx | 4 +++- scripts/cov/main.go | 26 ++++++++---------------- 8 files changed, 31 insertions(+), 28 deletions(-) diff --git a/.testcoverage.yml b/.testcoverage.yml index f4705d1a..309a5d77 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -23,7 +23,7 @@ local-prefix: github.com/Wave-RF/WaveHouse threshold: total: 80 -# Per-suite coverage thresholds (custom key — read by scripts/coverage.sh, +# Per-suite coverage thresholds (custom key — read by scripts/cov, # silently ignored by go-test-coverage). Single source of truth for what # percentage each suite must hit before it counts as passing. # @@ -34,7 +34,10 @@ suites: # Go SDK (clients/go) — a NESTED module, so the root module's # `-coverpkg=./...` can never reach it: rendered and gated on its own, # never merged into the total above, nothing to add under exclude.paths. - # 75 vs 82.7% measured; the headroom is the codegen CLI. Raise as it fills. + # 75 vs 87.1% measured (SDK 88.5%, codegen CLI 81.0%). The gap is deliberate + # slack, matching how every other floor here is set (ts-unit is 40 against a + # measured 76%): these are regression floors, not targets, so ordinary churn + # doesn't red the build. Raise it when a real regression slips under 75. # # ONE key here, three for the TS SDK below, because the TS SDK has two # coverage-PRODUCING suites and this has one. `ts-unit` is vitest, `ts-e2e` diff --git a/Makefile b/Makefile index ed1916a3..89cf29f7 100644 --- a/Makefile +++ b/Makefile @@ -192,7 +192,7 @@ ACTIONLINT_VERSION := v1.7.12 ACTIONLINT := $(LOCAL_BIN)/actionlint-$(ACTIONLINT_VERSION) # --- Coverage Directories ----------------------------------------------------- -# One path per suite. Internal layout (managed by scripts/coverage.sh): +# One path per suite. Internal layout (managed by scripts/cov): # $(COV_X)/data/ binary covdata (covmeta.* / covcounters.*) # $(COV_X)/coverage.txt rendered textfmt profile # $(COV_X)/coverage.html rendered HTML report @@ -830,7 +830,7 @@ test-sdk-go: go-mod-download ## Run Go SDK (clients/go, a nested module) unit + @rm -rf $(COV_GOSDK)/data && mkdir -p $(COV_GOSDK)/data @gotestsum=$$(go tool -n gotestsum) && cd $(GO_SDK_DIR) && \ GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" "$$gotestsum" --format $(GOTESTSUM_FMT) -- \ - -cover -coverpkg=./... -race ./... $(ARGS) \ + -cover -coverpkg=./... -race -timeout 60s ./... $(ARGS) \ -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 1fb47543..5374685d 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -477,7 +477,7 @@ WaveHouse/ ├── clients/ # Official SDKs │ ├── ts/ # TypeScript SDK (@wavehouse/sdk) │ └── go/ # Go SDK — a NESTED Go module (own go.mod, invisible -│ # to root `go list`; hence the *-go-sdk make targets) +│ # to root `go list`; hence the test-sdk-go* targets) ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) │ ├── conformance/ # TS half of the cross-SDK wire-format conformance suite diff --git a/docs/src/content/docs/sdk/go.md b/docs/src/content/docs/sdk/go.md index db264d2d..1b23d411 100644 --- a/docs/src/content/docs/sdk/go.md +++ b/docs/src/content/docs/sdk/go.md @@ -158,7 +158,7 @@ See [Reference → Error Handling](/sdk/reference#error-handling) for retry beha Both SDKs share a wire format and feature set, verified in CI against a shared `wire_cases.json` fixture that asserts equivalent HTTP requests for builder calls. The API shapes differ: - **No `Result` union.** Go returns `(T, error)`; a non-nil `error` is the only failure signal. No `{ok, data, error}` objects, no `error: null` sentinels. -- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` first; use a deadline or `cancel()`. See [Reference → Context Cancellation](/sdk/reference#cancellation). +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` first; use a deadline or `cancel()`. See [Reference → Cancellation](/sdk/reference#cancellation). - **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` take no context; the returned `*StreamController` owns its goroutine and connection until `.Close()` (usually deferred). See [Streaming](/sdk/streaming). - **Generics on package functions.** Go has no type parameters on methods, so use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. - **No implicit "await."** `QueryBuilder` is not `PromiseLike`; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly. @@ -173,4 +173,4 @@ The topic pages cover both SDKs, tabbed by language — the tab you pick here fo - [Streaming & Live Queries](/sdk/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. - [Pipes](/sdk/pipes) — Execute and manage named query pipes. - [Admin & System](/sdk/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/reference) — Error codes, context cancellation, API tree, and codegen CLI. +- [Reference & CLI](/sdk/reference) — Error codes, cancellation, the full API tree, and the codegen CLIs. diff --git a/docs/src/content/docs/sdk/queries.mdx b/docs/src/content/docs/sdk/queries.mdx index 3a0acdf4..f1ff48aa 100644 --- a/docs/src/content/docs/sdk/queries.mdx +++ b/docs/src/content/docs/sdk/queries.mdx @@ -72,7 +72,7 @@ for _, row := range page.Data { Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. A **slice/array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. -For a batch, the result is `ok` only when every record succeeded (`failed === 0`). Inspect the failure count and the per-record results (each carrying a **1-based** index) for partial failures — the call's top-level error is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty batch is a no-op and sends no request. The batch path sends one request regardless of size; bounded-concurrency chunking of very large batches is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). +For a batch, the result is `ok` only when every record succeeded (no records failed). Inspect the failure count and the per-record results (each carrying a **1-based** index) for partial failures — the call's top-level error is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty batch is a no-op and sends no request. The batch path sends one request regardless of size; bounded-concurrency chunking of very large batches is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). @@ -543,6 +543,10 @@ untyped, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUnty +### Streaming from a builder + +A builder can also open a stream instead of fetching — its filters and column projection are then applied client-side to the live events, which is the only place that client-side filtering happens. See [Streaming → Client-side stream filtering](/sdk/streaming#client-side-stream-filtering). + ### Pagination When a limit is set and the result contains at least that many rows, the result reports that more rows are available. Cursor-based pagination walks an **order column** — it adds a filter on that column using the last row's value — so the `next` cursor is only attached when the query has an explicit order. With no order column the result still reports "has more" honestly, but there is no cursor to build. diff --git a/docs/src/content/docs/sdk/streaming.mdx b/docs/src/content/docs/sdk/streaming.mdx index b97dfbe2..05c89bf4 100644 --- a/docs/src/content/docs/sdk/streaming.mdx +++ b/docs/src/content/docs/sdk/streaming.mdx @@ -260,10 +260,10 @@ Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3 ### Transport behavior -A dropped stream reconnects on a jittered exponential backoff, capped at 30s, and resumes with `Last-Event-ID` so the server gap-fills what was missed. HTTP/2 is recommended. Reconnection is **unbounded**: a stream keeps re-dialing until it hits a terminal error or you close it. The client option that bounds retries applies to REST requests only and is never consulted here. +A dropped stream reconnects on a jittered exponential backoff, capped at 30s, and resumes from the last event it saw so the server gap-fills what was missed. HTTP/2 is recommended. Reconnection is **unbounded**: a stream keeps re-dialing until it hits a terminal error or you close it. The client option that bounds retries applies to REST requests only and is never consulted here. :::caution[Resumption is at-least-once, and time-bounded] -Delivery across a reconnect is **at-least-once**. The `Last-Event-ID` the client sends is the last event's `received_timestamp`, and the server replays from that instant *inclusively* — so the last event you already saw, and anything sharing its timestamp, arrives again. Neither SDK deduplicates live frames — a live query makes one pass at the backfill seam only — so key on `timestamp` plus your own row identity if duplicates matter. +Delivery across a reconnect is **at-least-once**. The resume point the client sends is the last event's `received_timestamp` — a `Last-Event-ID` header in TypeScript, a `?since=` query parameter in Go; the server accepts either and replays from that instant *inclusively* — so the last event you already saw, and anything sharing its timestamp, arrives again. Neither SDK deduplicates live frames — a live query makes one pass at the backfill seam only — so key on `timestamp` plus your own row identity if duplicates matter. Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration#message-queue-nats) — 15 minutes by default. A drop longer than that resumes with a hole and no signal, because the purged messages are simply gone. ::: @@ -486,5 +486,7 @@ Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback — see [Queries → Filtering](/sdk/queries#filtering). ::: +**When the backfill doesn't complete.** `Initial(rows, err)` fires with the error, the events buffered during the fetch window are discarded, and live delivery continues from there — so a live query can end up running normally with no snapshot behind it. Check `err` inside `Initial` and re-run the fetch if a missing backfill matters; leave it a moment first, since events reach a stream from the message queue before the ingest worker lands them in ClickHouse (see [Ingest pipeline](/ingest-pipeline)). + diff --git a/docs/src/content/docs/sdk/typescript.mdx b/docs/src/content/docs/sdk/typescript.mdx index d2edbe56..083a9710 100644 --- a/docs/src/content/docs/sdk/typescript.mdx +++ b/docs/src/content/docs/sdk/typescript.mdx @@ -7,6 +7,8 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components Setup and caveats for `@wavehouse/sdk`, the TypeScript client: installation, runtimes, client construction, and the `Result` error model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing Go instead? [Go setup](/sdk/go) is the mirror of this page. +`@wavehouse/sdk` has exactly one runtime dependency — `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. + ## Installation @@ -472,7 +474,7 @@ explicit new Agent() 17 14 12 13 An explicit `dispatcher` wins because the request never consults the shared symbol at all. An explicit `setGlobalDispatcher` call (below) wins the other way round — it overwrites the symbol after the first copy claimed it. ::: -Both casts are load-bearing, for one underlying reason: undici declares its own request/response types, separate from the ones behind your global `fetch`, so the two aren't structurally assignable. `{ ...init, dispatcher } as never` covers the split on `RequestInit`, which differs on `body` and `headers` (`never` is assignable to either spelling, so one snippet compiles whether or not your `lib` includes DOM); and the return cast handles the `Response` mismatch. The URL needs no cast — `FetchLike` declares it as `string`, which undici's `RequestInfo` accepts. They're safe because of the narrow runtime contract above — a string URL and a plain `RequestInit` in, and only the response members enumerated above read back. `undici` is a dependency you add on top of the SDK's single ~1.4 KB one. +Both casts are load-bearing, for one underlying reason: undici declares its own request/response types, separate from the ones behind your global `fetch`, so the two aren't structurally assignable. `{ ...init, dispatcher } as never` covers the split on `RequestInit`, which differs on `body` and `headers` (`never` is assignable to either spelling, so one snippet compiles whether or not your `lib` includes DOM); and the return cast handles the `Response` mismatch. The URL needs no cast — `FetchLike` declares it as `string`, which undici's `RequestInfo` accepts. They're safe because of the narrow runtime contract above — a string URL and a plain `RequestInit` in, and only the response members enumerated above read back. `undici` is a dependency you add on top of `eventsource-parser`, the SDK's single ~1.4 KB runtime dependency. If you're genuinely pinned to an affected undici, the same snippet fixes it — your pinned version, with a dispatcher that never reuses a keep-alive socket: `new Agent({ pipelining: 0 })`. That costs a fresh connection per request, so prefer upgrading. Note that tuning `keepAliveTimeout` does **not** help: the retirement timer is starved by the same idle event loop that triggers the bug, so the socket is still there to be reused. diff --git a/scripts/cov/main.go b/scripts/cov/main.go index 007c821e..02c88979 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -71,8 +71,13 @@ var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} // suiteTarget names the make target that populates a suite, for the // "did you run it?" hint. Only suites whose target isn't `test-` -// need an entry. -var suiteTarget = map[string]string{"go-sdk": "test-sdk-go"} +// need an entry. One source of truth: every hint in this file routes +// through makeTargetFor, so a target rename touches only this map. +var suiteTarget = map[string]string{ + "go-sdk": "test-sdk-go", + "ts-unit": "test-sdk-ts", + "ts-e2e": "test-e2e", // the orchestrator run, not a target of its own +} // makeTargetFor returns the make target that populates the named suite. func makeTargetFor(suite string) string { @@ -320,11 +325,7 @@ func merge(c *config) error { dirs = append(dirs, d) fmt.Printf(" %s✔%s %-13s %s\n", green, reset, s, d) } else { - hint := "test-" + s - if s == "unit" { - hint = "test" - } - fmt.Printf(" %s✗%s %-13s (no covdata; run `make %s` to include)\n", yellow, reset, s, hint) + fmt.Printf(" %s✗%s %-13s (no covdata; run `make %s` to include)\n", yellow, reset, s, makeTargetFor(s)) } } if len(dirs) == 0 { @@ -482,7 +483,7 @@ func mergeTS(c *config) error { filepath.Join(root, name, "coverage-final.json")) } else { fmt.Printf(" %s✗%s %-9s (no coverage-final.json; run `make %s` to include)\n", - yellow, reset, name, ternary(name == "ts-unit", "test-sdk-ts", "test-e2e")) + yellow, reset, name, makeTargetFor(name)) } } if len(merged) == 0 { @@ -768,15 +769,6 @@ func formatPctBare(covered, total int) string { // tsHTML is the vitest/nyc HTML report path for a TS suite. func tsHTML(suite string) string { return filepath.Join(root, suite, "index.html") } -// ternary returns a if cond else b. Used inline to keep the merge log -// branching from sprawling into a 5-line if/else. -func ternary[T any](cond bool, a, b T) T { - if cond { - return a - } - return b -} - // copyFile streams src → dst, creating dst and overwriting if it exists. // Used to stage coverage-final.json files under suite-prefixed names // before nyc merge so the inputs land in one directory. From 4094272bc7d23b42fbd6876b9241f1a73a847294 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 13:43:41 -0400 Subject: [PATCH 53/59] style(docs): updating copywriting on some pages --- docs/src/content/docs/sdk/{ => setup}/go.md | 0 docs/src/content/docs/sdk/{ => setup}/typescript.mdx | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename docs/src/content/docs/sdk/{ => setup}/go.md (100%) rename docs/src/content/docs/sdk/{ => setup}/typescript.mdx (100%) diff --git a/docs/src/content/docs/sdk/go.md b/docs/src/content/docs/sdk/setup/go.md similarity index 100% rename from docs/src/content/docs/sdk/go.md rename to docs/src/content/docs/sdk/setup/go.md diff --git a/docs/src/content/docs/sdk/typescript.mdx b/docs/src/content/docs/sdk/setup/typescript.mdx similarity index 100% rename from docs/src/content/docs/sdk/typescript.mdx rename to docs/src/content/docs/sdk/setup/typescript.mdx From f778e415e287d461e4f680ebb6d94af3775348fd Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 13:44:12 -0400 Subject: [PATCH 54/59] style(docs): updating copywriting on some pages again --- AGENTS.md | 4 +- CHANGELOG.md | 4 +- CONTRIBUTING.md | 2 +- clients/go/README.md | 4 +- clients/ts/README.md | 6 +- docs/src/config/sidebar.ts | 21 +++-- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/admin.mdx | 16 ++-- docs/src/content/docs/sdk/index.mdx | 23 ++--- docs/src/content/docs/sdk/pipes.mdx | 10 +-- docs/src/content/docs/sdk/queries.mdx | 60 ++++++------- docs/src/content/docs/sdk/reference.mdx | 54 ++++++------ docs/src/content/docs/sdk/setup/go.md | 34 ++++---- docs/src/content/docs/sdk/setup/index.md | 13 +++ .../src/content/docs/sdk/setup/typescript.mdx | 4 +- docs/src/content/docs/sdk/streaming.mdx | 84 +++++++++---------- 17 files changed, 180 insertions(+), 163 deletions(-) create mode 100644 docs/src/content/docs/sdk/setup/index.md diff --git a/AGENTS.md b/AGENTS.md index 80c7c1c5..5b9d735b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -382,9 +382,9 @@ Internal-only backend changes (middleware refactors, observability internals, de The SDK docs are **topic-first, not language-first** — the decision from PR #313, carried into the Go SDK in PR #434. Adding a third language must not change the shape: - **One page per topic, shared by every language**: `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, `/sdk/reference`. Prose that holds for every client stays outside the tabs; the language-specific code and caveats go inside a `` block, one `` per language. The `syncKey` is `lang` everywhere, so a reader picks a language once and it follows them across the whole tree. -- **One setup/caveats page per language**: `/sdk/typescript`, `/sdk/go`. Installation, client construction, auth, and the error model are genuinely per-language and live here. +- **One setup/caveats page per language, under `/sdk/setup/`**: `/sdk/setup/typescript`, `/sdk/setup/go`. Installation, client construction, auth, and the error model are genuinely per-language and live here. `/sdk/setup` itself is the hub — a table of every client (package, install command, version floor) plus cards to each language's page. - **`/sdk` is the language-neutral overview.** No language sits at the root of `/sdk` — that was the trap the Go SDK's first draft fell into, leaving TypeScript un-prefixed and undocumented as a language. -- **Adding a language** = one new setup page + one new `` per topic page + one sidebar entry. It is never a parallel `/sdk//` tree, because that churns the topic URLs and doubles the pages to keep in sync. +- **Adding a language** = one new `/sdk/setup/` page + a row in the `/sdk/setup` table + one new `` per topic page + one entry in the sidebar's nested `Setup` group (which mirrors the `/sdk/setup/` route segment). It is never a parallel `/sdk//` tree, because that churns the topic URLs and doubles the pages to keep in sync. - Because tabs need MDX, the topic pages are `.mdx` — remember §Markdown authoring rules: `make fix` does not auto-fix MDX, so unwrap WH001 wrapping by hand and keep a blank line between a JSX tag and a code fence. ## Common Tasks diff --git a/CHANGELOG.md b/CHANGELOG.md index 4705e61c..a3743330 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go.md` (new), `docs/src/content/docs/sdk/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); all four use gotestsum and honor `ARGS`/`V=1` like every other Go suite. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page — `/sdk/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/go`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); all four use gotestsum and honor `ARGS`/`V=1` like every other Go suite. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page — `/sdk/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/go`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. @@ -18,7 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed -- **Go SDK client-side stream filters compare timestamps as instants, not as text** (`clients/go/stream.go`, `clients/go/stream_test.go`, `docs/src/content/docs/sdk/go/streaming.md`): the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing (#402), so a payload reads `2026-06-21T04:00:00Z` while a caller's filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Compared as text those disagree in both directions — lexically the payload sorts *below* the constant, so `OpGte` withheld a row that was chronologically equal. Both sides now parse as instants, mirroring the server's row-filter rule for DateTime columns. Only unambiguous spellings count (RFC 3339 with an explicit offset or `Z`): a zone-less constant names an instant only relative to the column's declared timezone, which a stream subscriber doesn't have, so reading it as UTC would move the instant. Ordering an instant against a non-instant now fails closed rather than falling back to text comparison. Also fixes a missing column matching the literal string `""` through the equality fallback. The TypeScript SDK's `matchesFilters` has the same text-comparison behavior and needs the same change for parity. +- **Go SDK client-side stream filters compare timestamps as instants, not as text** (`clients/go/stream.go`, `clients/go/stream_test.go`, `docs/src/content/docs/sdk/streaming.mdx`): the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing (#402), so a payload reads `2026-06-21T04:00:00Z` while a caller's filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Compared as text those disagree in both directions — lexically the payload sorts *below* the constant, so `OpGte` withheld a row that was chronologically equal. Both sides now parse as instants, mirroring the server's row-filter rule for DateTime columns. Only unambiguous spellings count (RFC 3339 with an explicit offset or `Z`): a zone-less constant names an instant only relative to the column's declared timezone, which a stream subscriber doesn't have, so reading it as UTC would move the instant. Ordering an instant against a non-instant now fails closed rather than falling back to text comparison. Also fixes a missing column matching the literal string `""` through the equality fallback. The TypeScript SDK's `matchesFilters` has the same text-comparison behavior and needs the same change for parity. - **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9473ca43..4092c665 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` - - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (one `` block per topic) plus the per-language setup pages `sdk/typescript.mdx` / `sdk/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (one `` block per topic) plus the per-language setup pages `sdk/setup/typescript.mdx` / `sdk/setup/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). diff --git a/clients/go/README.md b/clients/go/README.md index f1d7d803..52a12f5f 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -2,7 +2,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. Zero third-party runtime dependencies, SSE parser included. -**Full documentation: [wavehouse.dev/sdk/go](https://wavehouse.dev/sdk/go)** (setup); the usage guides below cover both SDKs, tabbed by language. +**Full documentation: [wavehouse.dev/sdk/go](https://wavehouse.dev/sdk/setup/go)** (setup); the usage guides below cover both SDKs, tabbed by language. ## Install @@ -47,7 +47,7 @@ func main() { ## Documentation -- [Go SDK setup](https://wavehouse.dev/sdk/go) — client config, auth, typed rows via generics, error handling. +- [Go SDK setup](https://wavehouse.dev/sdk/setup/go) — client config, auth, typed rows via generics, error handling. - [Queries](https://wavehouse.dev/sdk/queries) — tables, the query builder, inserts, pagination, raw SQL. - [Streaming & Live Queries](https://wavehouse.dev/sdk/streaming) — SSE streams, client-side filtering, backfill-then-live. - [Pipes](https://wavehouse.dev/sdk/pipes) — execute and manage named query pipes. diff --git a/clients/ts/README.md b/clients/ts/README.md index a30c9ee7..ce03eaea 100644 --- a/clients/ts/README.md +++ b/clients/ts/README.md @@ -8,7 +8,7 @@ TypeScript client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — sche npm install @wavehouse/sdk ``` -Requires Node 22 or newer — the only line this SDK is tested against; Node 18 and 20 are past upstream end-of-life. Neither browsers nor Node need a polyfill: streaming runs on `fetch`, like the rest of the SDK — see [Runtime support](https://wavehouse.dev/sdk/typescript#runtime-support). +Requires Node 22 or newer — the only line this SDK is tested against; Node 18 and 20 are past upstream end-of-life. Neither browsers nor Node need a polyfill: streaming runs on `fetch`, like the rest of the SDK — see [Runtime support](https://wavehouse.dev/sdk/setup/typescript#runtime-support). This works in any framework that uses a bundler — React, Vue, Svelte, Angular, Astro, SolidJS, or plain Vite — with `import { createClient } from '@wavehouse/sdk'`. @@ -57,7 +57,7 @@ const wh = createClient({ }); ``` -`baseURL` may include a path prefix (`https://app.example.com/api/wavehouse`) when WaveHouse is served under one — see [Serving under a path prefix](https://wavehouse.dev/sdk/typescript#serving-under-a-path-prefix). +`baseURL` may include a path prefix (`https://app.example.com/api/wavehouse`) when WaveHouse is served under one — see [Serving under a path prefix](https://wavehouse.dev/sdk/setup/typescript#serving-under-a-path-prefix). ### Query Data @@ -170,7 +170,7 @@ E2E tests live in `tests/e2e/sdk/` (repo root) and exercise the full pipeline th ## API Reference -See the full [SDK documentation](https://wavehouse.dev/sdk/typescript) for detailed documentation of every method, type, and option. +See the full [SDK documentation](https://wavehouse.dev/sdk/setup/typescript) for detailed documentation of every method, type, and option. ## License diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index e40f45df..94d2513d 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -26,16 +26,25 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ { label: "API Reference", slug: "api" }, { // Topic-first SDK pages: the shared usage pages carry - // per language and each language keeps its own - // setup/caveats page — the topic URLs never churn as languages are + // per language, and setup — the one genuinely + // per-language part — is grouped under /sdk/setup/, with the sidebar + // mirroring that segment. The topic URLs never churn as languages are // added (decision in PR #313, executed for Go in PR #434). A third - // language adds one setup page and a tab per topic; it does NOT add a - // parallel tree, and no language sits at the root of /sdk. + // language adds one /sdk/setup/ page, a row in the /sdk/setup + // table, a tab per topic, and one entry in the Setup group below; it + // does NOT add a parallel tree, and no language sits at the root + // of /sdk. label: "SDKs", items: [ { label: "Overview", slug: "sdk" }, - { label: "TypeScript setup", slug: "sdk/typescript" }, - { label: "Go setup", slug: "sdk/go" }, + { + label: "Setup", + items: [ + { label: "All SDKs", slug: "sdk/setup" }, + { label: "TypeScript", slug: "sdk/setup/typescript" }, + { label: "Go", slug: "sdk/setup/go" }, + ], + }, { label: "Queries", slug: "sdk/queries" }, { label: "Streaming & Live Queries", slug: "sdk/streaming" }, { label: "Pipes", slug: "sdk/pipes" }, diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 5374685d..74a4c01e 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -194,7 +194,7 @@ WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev See the [SDK guide](/sdk) for the client API and examples in both languages. -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see [Go setup](/sdk/go). +Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see [Go setup](/sdk/setup/go). ### Validating tokens diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index bc5b35d4..539f9e1b 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -82,7 +82,7 @@ handle_path /api/wavehouse/* { The ingress controller forwards the full path unless you ask it to rewrite. Pair a capture-group path (`/api/wavehouse(/|$)(.*)` with `pathType: ImplementationSpecific`) with the `nginx.ingress.kubernetes.io/rewrite-target: /$2` annotation, or the prefix arrives at WaveHouse unstripped. ::: -Point either SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` in TypeScript, `wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wavehouse"})` in Go — both sending REST calls and SSE streams under the prefix ([TypeScript → Serving under a path prefix](/sdk/typescript#serving-under-a-path-prefix), [Go → Creating a client](/sdk/go#creating-a-client)). +Point either SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` in TypeScript, `wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wavehouse"})` in Go — both sending REST calls and SSE streams under the prefix ([TypeScript → Serving under a path prefix](/sdk/setup/typescript#serving-under-a-path-prefix), [Go → Creating a client](/sdk/setup/go#creating-a-client)). ## Request-body size limits diff --git a/docs/src/content/docs/sdk/admin.mdx b/docs/src/content/docs/sdk/admin.mdx index 926f6b9e..eb0bccb6 100644 --- a/docs/src/content/docs/sdk/admin.mdx +++ b/docs/src/content/docs/sdk/admin.mdx @@ -5,9 +5,9 @@ description: "Schema introspection, access-control policy, DLQ stats, and health import { Tabs, TabItem } from "@astrojs/starlight/components"; -Operational surfaces of the SDKs. With one exception, everything on this page sits behind the server's admin gate on `/v1/ops/*`: the caller must resolve to the admin role (`policy.admin_role`, `"admin"` by default) or present the server's non-JWT [operator key](/api#authentication) as an `X-Operator-Key` header. Without one, these calls return `403` — unless the deployment deliberately sets `default_role` to the admin role, a loudly-warned dev-only setting. The exception is the health check, which calls the public, content-free `/v1/health` route and needs no credentials. See [Access Control](/access-control) for how roles resolve. +Operational surfaces of the SDKs. With one exception, everything on this page sits behind the server's admin gate on `/v1/ops/*`: the caller must resolve to the admin role (`policy.admin_role`, `"admin"` by default) or present the server's non-JWT [operator key](/api#authentication) as an `X-Operator-Key` header. Without one, these calls return `403`, unless the deployment deliberately sets `default_role` to the admin role, a loudly-warned dev-only setting. The exception is the health check, which calls the public, content-free `/v1/health` route and needs no credentials. See [Access Control](/access-control) for how roles resolve. -Neither SDK has a first-class operator-key option; each carries it as a custom header — [`options.headers`](/sdk/typescript#custom-headers) in TypeScript, [`ClientOptions.Headers`](/sdk/go#clientoptions) in Go. +Neither SDK has a first-class operator-key option; each carries it as a custom header: [`options.headers`](/sdk/setup/typescript#custom-headers) in TypeScript, [`ClientOptions.Headers`](/sdk/setup/go#clientoptions) in Go. ## Schema @@ -34,7 +34,7 @@ const { data } = await wh.from('clicks').schema(); ```go // List all table schemas. schemas, err := wh.Schema.List(ctx) -// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name +// schemas is wavehouse.Schemas: map[string]TableSchema, keyed by table name // Force refresh from ClickHouse. err = wh.Schema.Refresh(ctx) @@ -113,7 +113,7 @@ result, err := wh.Policy.Validate(ctx, policyDraft) // result.Valid == true, or err wraps the validation failure details ``` -`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator — hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator, hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. @@ -156,13 +156,13 @@ stats, err = wh.DLQ.Table(ctx, "clicks") ## System -Content-free server-online check — the one surface on this page that needs no credentials. +Content-free server-online check: the one surface on this page that needs no credentials. ```ts -// health() hits the public, content-free /v1/health route — 200/503, no body. +// health() hits the public, content-free /v1/health route: 200/503, no body. // Use it to check a server is reachable before sending data. const result = await wh.sys.health(); if (result.ok) { @@ -175,7 +175,7 @@ if (result.ok) { ```go -// Health hits the public, content-free /v1/health route — 200 → nil error, +// Health hits the public, content-free /v1/health route: 200 → nil error, // any other status (including 503) → a non-nil *wavehouse.Error. // Use it to check a server is reachable before sending data. if err := wh.Sys.Health(ctx); err != nil { @@ -187,4 +187,4 @@ if err := wh.Sys.Health(ctx); err != nil { -> Readiness (`/readyz`) is intentionally **not** exposed through either SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern, not the client's. Probe `/readyz` directly from your orchestrator if you need it. +> Readiness (`/readyz`) is intentionally **not** exposed through either SDK: it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern, not the client's. Probe `/readyz` directly from your orchestrator if you need it. diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 0e3a0a2e..f51a1026 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -1,13 +1,13 @@ --- title: "Client SDKs" -description: "Official WaveHouse clients for TypeScript and Go — query builder, real-time streaming, codegen." +description: "Official WaveHouse clients for TypeScript and Go: query builder, real-time streaming, codegen." --- import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components"; WaveHouse ships two officially supported clients with full API-tree parity: **TypeScript** (`@wavehouse/sdk`) and **Go** (`github.com/Wave-RF/WaveHouse/clients/go`). Both give you a typed query builder, real-time SSE streaming over header-authenticated HTTP, live queries that backfill history before going live, and a codegen CLI that turns your ClickHouse schema into row types. -They speak the same wire format — a shared fixture replayed by both test suites keeps it that way — so the topic pages below cover both languages side by side. Pick a language once and the tabs follow you across every page. +They speak the same wire format, and a shared fixture replayed by both test suites keeps it that way, so the topic pages below cover both languages side by side. Pick a language once and the tabs follow you across every page. ## Install @@ -18,7 +18,7 @@ They speak the same wire format — a shared fixture replayed by both test suite npm install @wavehouse/sdk ``` -Other package managers, the CDN build, and runtime support: [TypeScript setup](/sdk/typescript#installation). +Other package managers, the CDN build, and runtime support: [TypeScript setup](/sdk/setup/typescript#installation). @@ -27,7 +27,7 @@ Other package managers, the CDN build, and runtime support: [TypeScript setup](/ go get github.com/Wave-RF/WaveHouse/clients/go ``` -Import path, Go version floor, and client construction: [Go setup](/sdk/go#installation). +Import path, Go version floor, and client construction: [Go setup](/sdk/setup/go#installation). @@ -77,20 +77,15 @@ func main() { -## Setup & caveats, per language +## Setup -Installation, client construction, auth, custom headers, and the error model are language-specific — each SDK has its own page for them. +Installation, client construction, auth, and the error model are the only language-specific parts. [`/sdk/setup`](/sdk/setup) has the package name, install command, and version floor for each client. - diff --git a/docs/src/content/docs/sdk/pipes.mdx b/docs/src/content/docs/sdk/pipes.mdx index 305666b3..a4466ad2 100644 --- a/docs/src/content/docs/sdk/pipes.mdx +++ b/docs/src/content/docs/sdk/pipes.mdx @@ -22,16 +22,16 @@ const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 5 const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }); ``` -`.fetch(opts?)` takes `PipeRequestOptions` — `{ signal }` only, narrower than the `.fetch(opts?)` on a [query builder](/sdk/queries#executing-the-query), which also accepts `limit`. Passing a `limit` is a compile error rather than a silent no-op. +`.fetch(opts?)` takes `PipeRequestOptions`: `{ signal }` only, narrower than the `.fetch(opts?)` on a [query builder](/sdk/queries#executing-the-query), which also accepts `limit`. Passing a `limit` is a compile error rather than a silent no-op. -`limit` is typed `never` rather than left out, so the rejection also catches a value passed in a variable — leaving it out would only reject an inline object. That cuts both ways: a value *declared* as `RequestOptions` is rejected whether or not it actually carries a limit, since the type permits one. If you share one options object across calls, type it as `PipeRequestOptions` — the table and query-builder `.fetch()` accept that too — or inline `{ signal }` at the pipe call. +`limit` is typed `never` rather than left out, so the rejection also catches a value passed in a variable, where leaving it out would only reject an inline object. That cuts both ways: a value *declared* as `RequestOptions` is rejected whether or not it actually carries a limit, since the type permits one. If you share one options object across calls, type it as `PipeRequestOptions` (the table and query-builder `.fetch()` accept that too), or inline `{ signal }` at the pipe call. `client.Pipe(name, params)` returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you execute it explicitly. Pass `nil` for `params` if the pipe takes none, or only needs its server-side defaults. -`wavehouse.Fetch[Row](ctx, pipeRef)` decodes results into `[]Row` — a package-level generic function, since Go has no generic methods (the same pattern as `FetchTyped` for queries and `SQL` for raw SQL). `.FetchUntyped(ctx)` is the non-generic method form, decoding into `[]map[string]any`. +`wavehouse.Fetch[Row](ctx, pipeRef)` decodes results into `[]Row`. It is a package-level generic function, since Go has no generic methods (the same pattern as `FetchTyped` for queries and `SQL` for raw SQL). `.FetchUntyped(ctx)` is the non-generic method form, decoding into `[]map[string]any`. ```go type TopPage struct { @@ -53,7 +53,7 @@ There is no per-call row cap here: the endpoint binds your params as the pipe's ## Streaming a pipe -Open a live stream from a pipe ref (see [Streaming](/sdk/streaming)). The pipe's SQL and params are **not** applied — the pipe's *name* is used as a table name, so where a table of that name exists you receive its raw events, and otherwise the stream stays silent rather than erroring. Both SDKs behave this way; both wait on a pipe-aware stream endpoint ([#445](https://github.com/Wave-RF/WaveHouse/issues/445)). +Open a live stream from a pipe ref (see [Streaming](/sdk/streaming)). The pipe's SQL and params are **not** applied: the pipe's *name* is used as a table name, so where a table of that name exists you receive its raw events, and otherwise the stream stays silent rather than erroring. Both SDKs behave this way; both wait on a pipe-aware stream endpoint ([#445](https://github.com/Wave-RF/WaveHouse/issues/445)). @@ -77,7 +77,7 @@ defer stream.Close() ## Managing pipe definitions -Create, read, and delete pipe definitions. These sit behind the [admin gate](/sdk/admin) on `/v1/ops/*` — the admin role (`policy.admin_role`) or the [operator key](/api#authentication). +Create, read, and delete pipe definitions. These sit behind the [admin gate](/sdk/admin) on `/v1/ops/*`: the admin role (`policy.admin_role`) or the [operator key](/api#authentication). diff --git a/docs/src/content/docs/sdk/queries.mdx b/docs/src/content/docs/sdk/queries.mdx index f1ff48aa..438f66c6 100644 --- a/docs/src/content/docs/sdk/queries.mdx +++ b/docs/src/content/docs/sdk/queries.mdx @@ -5,7 +5,7 @@ description: "Tables, the chainable query builder, pagination, and raw SQL in th import { Tabs, TabItem } from "@astrojs/starlight/components"; -Reading and writing data: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Both SDKs speak the same wire format and expose the same surface — pick your language with the tabs below and the choice follows you across every page. Setup, client construction, and the error model live on the per-language pages: [TypeScript](/sdk/typescript) and [Go](/sdk/go). +Reading and writing data: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Both SDKs speak the same wire format and expose the same surface. Pick your language with the tabs below and the choice follows you across every page. Setup, client construction, and the error model live on the per-language pages: [TypeScript](/sdk/setup/typescript) and [Go](/sdk/setup/go). ## Tables @@ -20,7 +20,7 @@ The entry point is a table reference. It performs no request, so it is safe to s const clicks = wh.from('clicks'); ``` -Every call returns the SDK's [`Result`](/sdk/typescript#result-type) — nothing throws for anything the server returns (see [Error Handling](/sdk/reference#error-handling) for the caller and environment errors that do). +Every call returns the SDK's [`Result`](/sdk/setup/typescript#result-type): nothing throws for anything the server returns (see [Error Handling](/sdk/reference#error-handling) for the caller and environment errors that do). @@ -31,16 +31,16 @@ Every call returns the SDK's [`Result`](/sdk/typescript#result-type) — noth clicks := wh.From("clicks") ``` -Every request-response operation takes a `context.Context` first and returns `(T, error)` — the chainable builder methods and `.Stream(opts)` excepted. See [Error Handling](/sdk/go#error-handling). +Every request-response operation takes a `context.Context` first and returns `(T, error)`, with the chainable builder methods and `.Stream(opts)` excepted. See [Error Handling](/sdk/setup/go#error-handling). ### Fetching every column -Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read — this is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. +Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read. This is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. -To paginate, chain an explicit order — a bare fetch sends no default order (see [Pagination](#pagination)). +To paginate, chain an explicit order: a bare fetch sends no default order (see [Pagination](#pagination)). @@ -70,9 +70,9 @@ for _, row := range page.Data { ### Inserting rows -Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. A **slice/array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. +Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. A **slice/array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch: per-record outcomes come back in the result. -For a batch, the result is `ok` only when every record succeeded (no records failed). Inspect the failure count and the per-record results (each carrying a **1-based** index) for partial failures — the call's top-level error is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty batch is a no-op and sends no request. The batch path sends one request regardless of size; bounded-concurrency chunking of very large batches is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). +For a batch, the result is `ok` only when every record succeeded (no records failed). Inspect the failure count and the per-record results (each carrying a **1-based** index) for partial failures. The call's top-level error is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty batch is a no-op and sends no request. The batch path sends one request regardless of size; bounded-concurrency chunking of very large batches is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). @@ -107,7 +107,7 @@ res, err = clicks.Insert(ctx, []map[string]any{ }) // res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results -// Many rows (typed slice) — same NDJSON path, via reflection +// Many rows (typed slice): same NDJSON path, via reflection type ClickRow struct { Page string `json:"page"` Button string `json:"button"` @@ -168,12 +168,12 @@ res, err = clicks.InsertNDJSON(ctx, string(raw)) ### Table schema -Fetch the table's column definitions from ClickHouse. This hits `/v1/ops/schema`, an **admin-only** endpoint: the caller must pass the admin gate — resolve to the policy admin role or present the non-JWT [operator key](/api#authentication) — or it returns `403`. +Fetch the table's column definitions from ClickHouse. This hits `/v1/ops/schema`, an **admin-only** endpoint: the caller must pass the admin gate by resolving to the policy admin role or presenting the non-JWT [operator key](/api#authentication), or it returns `403`. -Send the operator key via [`options.headers`](/sdk/typescript#custom-headers). +Send the operator key via [`options.headers`](/sdk/setup/typescript#custom-headers). ```ts const { data } = await clicks.schema(); @@ -185,7 +185,7 @@ const { data } = await clicks.schema(); -Send the operator key via [`ClientOptions.Headers`](/sdk/go#clientoptions). +Send the operator key via [`ClientOptions.Headers`](/sdk/setup/go#clientoptions). ```go schema, err := clicks.Schema(ctx) @@ -198,7 +198,7 @@ schema, err := clicks.Schema(ctx) ### Selecting columns -Starts a query builder chain — see [Query Builder](#query-builder) for the chainable methods and how to execute it. +Starts a query builder chain. See [Query Builder](#query-builder) for the chainable methods and how to execute it. @@ -222,7 +222,7 @@ page, err := clicks.Select("page", "button"). ### Selecting every column explicitly -`selectAll` selects **every column your role is allowed to read** — the explicit form of what a bare fetch does. Mutually exclusive with an explicit column list and with aggregations; for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). +`selectAll` selects **every column your role is allowed to read**: the explicit form of what a bare fetch does. Mutually exclusive with an explicit column list and with aggregations; for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). @@ -267,7 +267,7 @@ defer stream.Close() ## Query Builder -Returned by the table ref's select methods. Immutable — every chain method returns a new builder, leaving the original unchanged. +Returned by the table ref's select methods. Immutable: every chain method returns a new builder, leaving the original unchanged. @@ -294,7 +294,7 @@ page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) ### Appending columns -Append columns to the SELECT clause. A literal `*` is the column *named* `*`, not a wildcard — use the `selectAll` form for all columns. +Append columns to the SELECT clause. A literal `*` is the column *named* `*`, not a wildcard. Use the `selectAll` form for all columns. @@ -334,7 +334,7 @@ clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') | `'<='` | `lte` | Less than or equal | | `'in'` | `in` | Value in array | | `'like'` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.stream()` / `.liveQuery()` matches case-**insensitively** | -| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `'not_like'` | `not_like` | SQL NOT LIKE, **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | @@ -355,7 +355,7 @@ clicks.Select("page"). | `wavehouse.OpLte` | `lte` | Less than or equal | | `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | | `wavehouse.OpLike` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.Stream()` / `.LiveQuery()` matches case-**insensitively** | -| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE, **client-side only**; `/v1/query` rejects this token | @@ -447,7 +447,7 @@ clicks.Select().Limit(100) // wavehouse.DefaultLimit otherwise ### Time ranges -Filter by a time window. Both bounds accept RFC3339 timestamps or relative durations (`1h`, `30m`, `7d`, `2w` — day and week suffixes expand to hours, so `7d` is `168h`). +Filter by a time window. Both bounds accept RFC3339 timestamps or relative durations (`1h`, `30m`, `7d`, `2w`; day and week suffixes expand to hours, so `7d` is `168h`). @@ -478,20 +478,20 @@ clicks.Select("page").TimeRange( ### Cache TTL -Records a desired result-cache TTL on the builder. **Currently client-side state only** — the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). +Records a desired result-cache TTL on the builder. **Currently client-side state only**: the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). ```ts -clicks.select('page').count().cacheTTL(300) // not yet honored server-side — see #280 +clicks.select('page').count().cacheTTL(300) // not yet honored server-side, see #280 ``` ```go -clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 +clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side, see #280 ``` @@ -512,14 +512,14 @@ if (hasMore && next) { } ``` -**Options** — `RequestOptions`: +**Options** (`RequestOptions`): | Field | Type | Description | |-------|------|-------------| | `signal` | `AbortSignal` | Cancel the request | | `limit` | `number` | Override builder limit for this fetch | -A pipe's `.fetch()` takes the narrower `PipeRequestOptions` instead — see [Pipes](/sdk/pipes#executing-a-pipe). +A pipe's `.fetch()` takes the narrower `PipeRequestOptions` instead. See [Pipes](/sdk/pipes#executing-a-pipe). @@ -545,13 +545,13 @@ untyped, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUnty ### Streaming from a builder -A builder can also open a stream instead of fetching — its filters and column projection are then applied client-side to the live events, which is the only place that client-side filtering happens. See [Streaming → Client-side stream filtering](/sdk/streaming#client-side-stream-filtering). +A builder can also open a stream instead of fetching: its filters and column projection are then applied client-side to the live events, which is the only place that client-side filtering happens. See [Streaming → Client-side stream filtering](/sdk/streaming#client-side-stream-filtering). ### Pagination -When a limit is set and the result contains at least that many rows, the result reports that more rows are available. Cursor-based pagination walks an **order column** — it adds a filter on that column using the last row's value — so the `next` cursor is only attached when the query has an explicit order. With no order column the result still reports "has more" honestly, but there is no cursor to build. +When a limit is set and the result contains at least that many rows, the result reports that more rows are available. Cursor-based pagination walks an **order column** (it adds a filter on that column using the last row's value), so the `next` cursor is only attached when the query has an explicit order. With no order column the result still reports "has more" honestly, but there is no cursor to build. -The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. Both SDKs share this limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker, so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. Both SDKs share this limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). @@ -616,7 +616,7 @@ Execute a raw SQL query. `/v1/ops/query` is admin-only: the caller must resolve -The SDK has no first-class option for the server's non-JWT [operator key](/api#authentication) — an operator can send its `X-Operator-Key` header via [`options.headers`](/sdk/typescript#custom-headers). +The SDK has no first-class option for the server's non-JWT [operator key](/api#authentication): an operator can send its `X-Operator-Key` header via [`options.headers`](/sdk/setup/typescript#custom-headers). ```ts const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); @@ -625,7 +625,7 @@ const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY -An operator key authorizes `/v1/ops/*` as well; send it via [`ClientOptions.Headers`](/sdk/go#clientoptions). Use `map[string]any` for dynamic schemas. +An operator key authorizes `/v1/ops/*` as well; send it via [`ClientOptions.Headers`](/sdk/setup/go#clientoptions). Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, @@ -633,7 +633,7 @@ rows, err := wavehouse.SQL[map[string]any](ctx, wh, // Or decode into a struct that matches the projected columns/aliases. // NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit -// integers (count() is UInt64) — decode them with the `,string` tag, or +// integers (count() is UInt64), so decode them with the `,string` tag, or // use map[string]any. See Reference → Codegen CLI for the full story. type PageTotal struct { Page string `json:"page"` @@ -647,5 +647,5 @@ typed, err := wavehouse.SQL[PageTotal](ctx, wh, :::note[No parameter binding through the SDK] -Positional `?` substitution is not supported, and neither SDK can forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo) — the proxy doesn't forward arbitrary query-string params and the raw-SQL entry point exposes no hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder. +Positional `?` substitution is not supported, and neither SDK can forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo): the proxy doesn't forward arbitrary query-string params and the raw-SQL entry point exposes no hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder. ::: diff --git a/docs/src/content/docs/sdk/reference.mdx b/docs/src/content/docs/sdk/reference.mdx index 094f9fcb..cd277b5f 100644 --- a/docs/src/content/docs/sdk/reference.mdx +++ b/docs/src/content/docs/sdk/reference.mdx @@ -49,17 +49,17 @@ if errors.As(err, &whErr) && whErr.Code == "ABORTED" { ## Error Handling -Neither SDK raises for anything the server returns — API errors are values. Both raise (or return plain wrapped errors) for caller and environment faults instead. +Neither SDK raises for anything the server returns: API errors are values. Both raise (or return plain wrapped errors) for caller and environment faults instead. -All API errors come back in [`Result.error`](/sdk/typescript#result-type). The SDK does throw on caller and environment errors: a non-absolute `baseURL` (REST calls reject with a `TypeError`; streams report `SSE_CONNECT_ERROR` to the subscriber's `error` callback — see [Serving under a path prefix](/sdk/typescript#serving-under-a-path-prefix)), `.stream()` / `.liveQuery()` in a runtime with no global `fetch` and no `options.fetch` (see [Runtime support](/sdk/typescript#runtime-support)), and an `auth` callback that rejects — a token-refresh failure propagates out of the REST call, and on a stream is reported as a retryable `SSE_AUTH_ERROR`. One more exception escapes an SDK call synchronously, though it is yours rather than ours: your own `status` handler throwing on the first `.subscribe()` or `.liveQuery()`, described under *If your own callback throws* below. +All API errors come back in [`Result.error`](/sdk/setup/typescript#result-type). The SDK does throw on caller and environment errors: a non-absolute `baseURL` (REST calls reject with a `TypeError`; streams report `SSE_CONNECT_ERROR` to the subscriber's `error` callback; see [Serving under a path prefix](/sdk/setup/typescript#serving-under-a-path-prefix)), `.stream()` / `.liveQuery()` in a runtime with no global `fetch` and no `options.fetch` (see [Runtime support](/sdk/setup/typescript#runtime-support)), and an `auth` callback that rejects: a token-refresh failure propagates out of the REST call, and on a stream is reported as a retryable `SSE_AUTH_ERROR`. One more exception escapes an SDK call synchronously, though it is yours rather than ours: your own `status` handler throwing on the first `.subscribe()` or `.liveQuery()`, described under *If your own callback throws* below. | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | On REST, a present-but-invalid or expired JWT that a gate then denied. **WaveHouse itself** never returns `401` for a *missing* token — that resolves to `default_role`, and a denial is `403`. On a stream it is always from something in front, since `/v1/stream` is ungated | +| 401 | `HTTP_401` | No | On REST, a present-but-invalid or expired JWT that a gate then denied. **WaveHouse itself** never returns `401` for a *missing* token: that resolves to `default_role`, and a denial is `403`. On a stream it is always from something in front, since `/v1/stream` is ungated | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | @@ -70,42 +70,42 @@ All API errors come back in [`Result.error`](/sdk/typescript#result-type). The S | 0 | `SSE_AUTH_ERROR` | Yes | The `auth` callback threw while minting a token for an attempt | | 0 | `SSE_NETWORK_ERROR` | Yes | Stream request failed to reach the server | | 0 | `SSE_READ_ERROR` | Yes | Stream was interrupted mid-read | -| 0 | `SSE_PARSE_ERROR` | Yes (usually nothing to re-dial) | Unparseable frame — reported and skipped, the connection continues. The buffer cap reports here too, then ends the connection (see below) | +| 0 | `SSE_PARSE_ERROR` | Yes (usually nothing to re-dial) | Unparseable frame, reported and skipped; the connection continues. The buffer cap reports here too, then ends the connection (see below) | | *(response status)* | `SSE_NO_STREAM_BODY` | No | A configured `options.fetch` returned a response with no readable body | | *(0 in a browser, the 3xx in Node)* | `SSE_REDIRECT` | No | The stream endpoint redirected; point `baseURL` at the final URL | -| *(response status)* | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — usually a gateway's login page | -| *(response status)* | `HTTP_4xx` / `HTTP_5xx` | Per status | The stream request was rejected — same codes as REST | +| *(response status)* | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream`, usually a gateway's login page | +| *(response status)* | `HTTP_4xx` / `HTTP_5xx` | Per status | The stream request was rejected, same codes as REST | The `SSE_*` codes arrive on the subscriber's `error` callback rather than in a `Result.error`, since a stream has no single result to carry them. Both transports act on `retryable`; what differs is when the error reaches you. All four combinations, with the two exceptions noted under the table: | | Retryable | Not retryable | |---|---|---| -| **REST** | retried up to `maxRetries`, then returned — the flag records what was already tried | returned on the first attempt, no backoff (every `4xx`) | +| **REST** | retried up to `maxRetries`, then returned; the flag records what was already tried | returned on the first attempt, no backoff (every `4xx`) | | **Stream** | reported *before* the transport re-dials, which it does indefinitely | reported once; the stream closes and stays closed | -On REST, `ABORTED` is the one error raised *by* a backoff rather than by an attempt: aborting during a retry sleep ends the call immediately with `ABORTED` instead of waiting the delay out. On a stream, `SSE_PARSE_ERROR` is flagged retryable, but an ordinary bad frame is skipped in place — no re-dial, no close. The one that *does* re-dial is the buffer-cap overflow described below. +On REST, `ABORTED` is the one error raised *by* a backoff rather than by an attempt: aborting during a retry sleep ends the call immediately with `ABORTED` instead of waiting the delay out. On a stream, `SSE_PARSE_ERROR` is flagged retryable, but an ordinary bad frame is skipped in place: no re-dial, no close. The one that *does* re-dial is the buffer-cap overflow described below. -On a stream, a retryable failure is re-dialed on a jittered exponential backoff (capped at 30s, and reset only once a connection has held for a few seconds — so a server that accepts and instantly closes still backs off), with the `status` callback moving `reconnecting` → `live`. +On a stream, a retryable failure is re-dialed on a jittered exponential backoff (capped at 30s, and reset only once a connection has held for a few seconds, so a server that accepts and instantly closes still backs off), with the `status` callback moving `reconnecting` → `live`. -Rejected requests surface the real status and message rather than an opaque connection failure — in a browser going cross-origin, though, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers — `Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes; a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead — indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round — the exception being a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The one 4xx it raises itself is `400` for a missing or empty `table`, and only on the stream route: a `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front — an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. +Rejected requests surface the real status and message rather than an opaque connection failure. In a browser going cross-origin, though, that only holds when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead, indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round. The exception is a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The one 4xx it raises itself is `400` for a missing or empty `table`, and only on the stream route: a `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front: an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. -`SSE_PARSE_ERROR` is the one code that isn't a connection outcome: it's reported and *skipped*, and the connection keeps reading — one bad frame shouldn't cost you the stream. For an ordinary bad frame its `retryable: true` is therefore vestigial — nothing is re-dialed. Two exceptions, one to each half of that reported-and-skipped rule. A frame whose `data` isn't valid JSON is skipped but never *reported* — `console.warn` and dropped, with no `error` callback. The parser's 16 MiB buffer cap is reported but not *skipped*: an overflow terminates the parser, so the transport stops reading and reconnects rather than feeding it again. +`SSE_PARSE_ERROR` is the one code that isn't a connection outcome: it's reported and *skipped*, and the connection keeps reading: one bad frame shouldn't cost you the stream. For an ordinary bad frame its `retryable: true` is therefore vestigial: nothing is re-dialed. Two exceptions, one to each half of that reported-and-skipped rule. A frame whose `data` isn't valid JSON is skipped but never *reported*: `console.warn` and dropped, with no `error` callback. The parser's 16 MiB buffer cap is reported but not *skipped*: an overflow terminates the parser, so the transport stops reading and reconnects rather than feeding it again. **If your own callback throws.** For anything delivered *through the transport* — `next`, `status`, or `error` — a throw never ends the stream: it is isolated and logged to the console, and never routed to your `error` callback, so a handler that swallows its own failures fails silently. **Wrap your handler bodies in your own `try`/`catch`.** That isolation is a backstop against one bad callback killing the connection, not a promise that throwing is harmless. Four paths sit outside it, all tracked in [#473](https://github.com/Wave-RF/WaveHouse/issues/473): -- **The first `status` call** — `.subscribe()`'s, and the one `liveQuery()` makes internally on your behalf. It fires synchronously with the current state before the transport is involved, and is unguarded, so the throw propagates back out of the call *you* made, unlogged. Out of `.subscribe()` that leaves your subscriber registered with no unsubscribe function returned. Out of `liveQuery()` it is worse: you get no handle at all, so there is nothing to `.close()`; the backfill never starts, so `initial()` is never called; and the stream opened a moment earlier keeps running — connected, reconnecting on its own — with nothing referencing it. Because the backfill never ran, the buffering phase also never ends — your `next()` is never called at all, and every event the stream receives is appended to a buffer you have no way to reach. Passing `opts.signal` is the only way to stop it. Since this call always happens, it is the *first* thing a throwing `status` handler does. -- **Delivery to other consumers stops at the one that threw** — later subscribers and any concurrent `for await`. For the iterator the event is *dropped*, not queued, because the fan-out runs before the event is handed to a waiter or buffered. And if the throw lands on the terminal `closed` status, the iterator is never marked done either, so a `for await` waits forever against a dead stream until something calls `.close()`. Order-dependent and invisible. +- **The first `status` call**: `.subscribe()`'s, and the one `liveQuery()` makes internally on your behalf. It fires synchronously with the current state before the transport is involved, and is unguarded, so the throw propagates back out of the call *you* made, unlogged. Out of `.subscribe()` that leaves your subscriber registered with no unsubscribe function returned. Out of `liveQuery()` it is worse: you get no handle at all, so there is nothing to `.close()`; the backfill never starts, so `initial()` is never called; and the stream opened a moment earlier keeps running (connected, reconnecting on its own) with nothing referencing it. Because the backfill never ran, the buffering phase also never ends: your `next()` is never called at all, and every event the stream receives is appended to a buffer you have no way to reach. Passing `opts.signal` is the only way to stop it. Since this call always happens, it is the *first* thing a throwing `status` handler does. +- **Delivery to other consumers stops at the one that threw**: later subscribers and any concurrent `for await`. For the iterator the event is *dropped*, not queued, because the fan-out runs before the event is handed to a waiter or buffered. And if the throw lands on the terminal `closed` status, the iterator is never marked done either, so a `for await` waits forever against a dead stream until something calls `.close()`. Order-dependent and invisible. - **A throwing `status` handler subscribed before `.connected()`** makes that promise reject with its timeout error against a stream that is already `live`, because the throw aborts the fan-out before `connected()`'s internal watcher is reached. - **Inside `liveQuery()`**, a throw from `initial()`, or from `next()` during the backfill flush, is absorbed by the backfill's own error path, with no log and no signal. The cost differs: `initial()` throws *before* the flush starts, so every buffered event is lost, while a `next()` throw mid-flush loses only the remainder. Either way, later live events reach `next()` as normal, with a further throw isolated and logged. -`SSE_AUTH_ERROR` is the one caller-side failure that isn't terminal. A rejecting `auth` callback propagates out of a REST call, but on a stream — where `auth` is invoked on every connection attempt rather than once per stream — a token endpoint having a bad minute is treated as transient and retried, rather than tearing down a stream that is otherwise healthy. +`SSE_AUTH_ERROR` is the one caller-side failure that isn't terminal. A rejecting `auth` callback propagates out of a REST call, but on a stream, where `auth` is invoked on every connection attempt rather than once per stream, a token endpoint having a bad minute is treated as transient and retried, rather than tearing down a stream that is otherwise healthy. -The SDK never panics on API or network failures, mirroring the TypeScript SDK's "never throws" guarantee. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less ones (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`, or use `wavehouse.IsRetryable(err)` to shortcut the `errors.As` + `.Retryable` check); client-side failures such as an `Auth` provider or marshal error are plain wrapped errors, so handle the `errors.As == false` case — see the [worked example](/sdk/go#error-handling). Streaming methods (`Stream`, `Subscribe`, `Close`) report through the subscriber's `Error` callback instead, and `Connected(ctx)` returns plain errors. +The SDK never panics on API or network failures, mirroring the TypeScript SDK's "never throws" guarantee. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less ones (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`, or use `wavehouse.IsRetryable(err)` to shortcut the `errors.As` + `.Retryable` check); client-side failures such as an `Auth` provider or marshal error are plain wrapped errors, so handle the `errors.As == false` case; see the [worked example](/sdk/setup/go#error-handling). Streaming methods (`Stream`, `Subscribe`, `Close`) report through the subscriber's `Error` callback instead, and `Connected(ctx)` returns plain errors. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -120,9 +120,9 @@ The SDK never panics on API or network failures, mirroring the TypeScript SDK's | 0 | `ABORTED` | No | Request canceled via `context.Context` | | 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | | 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | -| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | +| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https`; retrying cannot fix it | | *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | -| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | +| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream`; something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | | 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | | 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | | 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | @@ -156,7 +156,7 @@ createClient(config) → WaveHouseClient │ ├── .schema() → Promise> (admin) │ └── .stream(opts?) → StreamController ├── .pipe(name, params?) → PipeRef (PromiseLike) -│ ├── .fetch(opts?) → Promise> // { signal } only — no limit +│ ├── .fetch(opts?) → Promise> // { signal } only, no limit │ └── .stream(opts?) → StreamController ├── .pipes (admin) │ ├── .list() → Promise> @@ -174,7 +174,7 @@ createClient(config) → WaveHouseClient ├── .dlq (admin) │ ├── .list() → Promise> │ ├── .table(name) → Promise> -│ └── .stream() → StreamController // not yet functional server-side — #197 +│ └── .stream() → StreamController // not yet functional server-side, #197 └── .sys └── .health() → Promise> @@ -226,7 +226,7 @@ NewClient(Config) → *Client ├── .DLQ (admin) → *DLQNamespace │ ├── .List(ctx) → (*DLQStats, error) │ ├── .Table(ctx, name) → (*DLQStats, error) -│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 +│ └── .Stream(opts) → *StreamController // not yet functional server-side, #197 └── .Sys → *SysNamespace └── .Health(ctx) → error @@ -267,7 +267,7 @@ Pass an admin-role token with `--auth `. |------|-------------|---------| | `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | | `--out`, `-o` | Output .d.ts file path | `./wavehouse.d.ts` | -| `--auth`, `-a` | Bearer token (if auth required) | — | +| `--auth`, `-a` | Bearer token (if auth required) | none | **Example output:** @@ -293,10 +293,10 @@ export interface ClicksRow { |----------------|-----------------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | | `UInt*`, `Int*`, `Float*` | `number` | -| `Decimal*` | `number` *(generated)* — but `/v1/query` returns Decimals as **quoted strings**, so treat the field as `string` until codegen is fixed ([#453](https://github.com/Wave-RF/WaveHouse/issues/453)) | +| `Decimal*` | `number` *(generated)*, but `/v1/query` returns Decimals as **quoted strings**, so treat the field as `string` until codegen is fixed ([#453](https://github.com/Wave-RF/WaveHouse/issues/453)) | | `Bool` | `boolean` | | `Nullable(T)` | `T \| null` | -| `Array(T)` | `T[]` — except `Array(UInt8)`, which `/v1/query` base64-encodes, so the generated `number[]` is a `string` at runtime ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Array(T)` | `T[]`, except `Array(UInt8)`, which `/v1/query` base64-encodes, so the generated `number[]` is a `string` at runtime ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | | `Map(K, V)` | `Record` | | `LowCardinality(T)` | same as `T` | @@ -326,7 +326,7 @@ Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history | `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | | `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | | `--package`, `-p` | Go package name for the generated file | `main` | -| `--help`, `-h` | Show usage and exit | — | +| `--help`, `-h` | Show usage and exit | none | **Example output** (for the [development quick-start](/development#quick-start) `clicks` table): @@ -390,9 +390,9 @@ make test-sdk-ts make test-e2e ``` -Test files live in `tests/e2e/sdk/` (each `*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. +Test files live in `tests/e2e/sdk/` (each `*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers`, a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. -See [Development Guide — E2E Tests via SDK](/development#e2e-tests-via-sdk) for architecture details and workflow tips. +See [Development Guide: E2E Tests via SDK](/development#e2e-tests-via-sdk) for architecture details and workflow tips. @@ -410,7 +410,7 @@ E2E tests (build tag `e2e`) run against a live WaveHouse instance: WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-sdk-go-e2e ``` -`WAVEHOUSE_URL` defaults to `http://localhost:8080`, and the optional `WAVEHOUSE_AUTH` covers the admin cases; the suite skips if the server is unreachable. Unlike the TypeScript SDK, the Go E2E suite isn't yet driven by the repo's `make test-e2e` orchestrator — wiring it in, with its own coverage gate, is tracked in [#518](https://github.com/Wave-RF/WaveHouse/issues/518). +`WAVEHOUSE_URL` defaults to `http://localhost:8080`, and the optional `WAVEHOUSE_AUTH` covers the admin cases; the suite skips if the server is unreachable. Unlike the TypeScript SDK, the Go E2E suite isn't yet driven by the repo's `make test-e2e` orchestrator. Wiring it in, with its own coverage gate, is tracked in [#518](https://github.com/Wave-RF/WaveHouse/issues/518). diff --git a/docs/src/content/docs/sdk/setup/go.md b/docs/src/content/docs/sdk/setup/go.md index 1b23d411..4597f7aa 100644 --- a/docs/src/content/docs/sdk/setup/go.md +++ b/docs/src/content/docs/sdk/setup/go.md @@ -3,7 +3,7 @@ title: "Go SDK setup" description: "Installing the WaveHouse Go client, creating a client, typed rows via generics, and the (T, error) model." --- -Setup and caveats for `github.com/Wave-RF/WaveHouse/clients/go`, the Go client: installation, client construction, typed rows, and the `(T, error)` model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing TypeScript instead? [TypeScript setup](/sdk/typescript) is the mirror of this page. +Setup and caveats for `github.com/Wave-RF/WaveHouse/clients/go`, the Go client: installation, client construction, typed rows, and the `(T, error)` model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries). Writing TypeScript instead? [TypeScript setup](/sdk/setup/typescript) is the mirror of this page. ## Installation @@ -69,13 +69,13 @@ wh := wavehouse.NewClient(wavehouse.Config{ | Field | Type | Default | Description | |-------|------|---------|-------------| -| `BaseURL` | `string` | — | Required WaveHouse server URL, optionally with a path prefix. A trailing `/` is trimmed and every request path is appended on both transports, so a server under `https://app.example.com/wavehouse` works as-is. | +| `BaseURL` | `string` | none | Required WaveHouse server URL, optionally with a path prefix. A trailing `/` is trimmed and every request path is appended on both transports, so a server under `https://app.example.com/wavehouse` works as-is. | | `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider called before each request. `nil` means unauthenticated access. | | `Options` | `*ClientOptions` | `nil` | Transport tuning (see below). | | `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports. | :::caution[Timeouts: use contexts, not `http.Client.Timeout`] -The default client sets no `Timeout`; use a `context.Context` deadline to prevent hangs. Leave `Timeout` unset on a custom `HTTPClient` too — it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. +The default client sets no `Timeout`; use a `context.Context` deadline to prevent hangs. Leave `Timeout` unset on a custom `HTTPClient` too: it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. ::: ### `ClientOptions` @@ -83,34 +83,34 @@ The default client sets no `Timeout`; use a `context.Context` deadline to preven | Field | Type | Default | Description | |-------|------|---------|-------------| | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | -| `Headers` | `map[string]string` | `nil` | Sent on every request the client makes — REST calls and SSE streams alike. | +| `Headers` | `map[string]string` | `nil` | Sent on every request the client makes: REST calls and SSE streams alike. | -`*Client` is safe for concurrent use — state is immutable after `NewClient` and builder chains copy — provided your `Auth` function is concurrency-safe. +`*Client` is safe for concurrent use (state is immutable after `NewClient` and builder chains copy), provided your `Auth` function is concurrency-safe. :::caution[`Options` opts you out of the default, not just in] -The 2-retry default applies only when `Config.Options` is `nil`. Passing `&wavehouse.ClientOptions{}` leaves `MaxRetries` at Go's zero value (`0`), which disables retries — set it explicitly. +The 2-retry default applies only when `Config.Options` is `nil`. Passing `&wavehouse.ClientOptions{}` leaves `MaxRetries` at Go's zero value (`0`), which disables retries: set it explicitly. ::: -`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk/typescript#custom-headers) — a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk/setup/typescript#custom-headers): a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): ```go wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "http://localhost:8080", Options: &wavehouse.ClientOptions{ - MaxRetries: 2, // Options opts out of the default — set it explicitly. + MaxRetries: 2, // Options opts out of the default: set it explicitly. Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, }, }) ``` -The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both — supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both: supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. :::note[How the token is transmitted] -The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk/typescript#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk/setup/typescript#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. ::: :::caution[A credentialed stream will not follow a redirect] -When the stream request carries a credential — an `Auth` token or a `ClientOptions.Headers` entry — the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. +When the stream request carries a credential (an `Auth` token or a `ClientOptions.Headers` entry), the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. ::: Use `https://` for any authenticated server outside a trusted network. The SDK allows `http://` for local development and private networks, but bearer tokens over plaintext are insecure. @@ -167,10 +167,10 @@ Both SDKs share a wire format and feature set, verified in CI against a shared ` ## Where to go next -The topic pages cover both SDKs, tabbed by language — the tab you pick here follows you across all of them. +The topic pages cover both SDKs, tabbed by language. The tab you pick here follows you across all of them. -- [Queries](/sdk/queries) — Tables, chainable query builder, inserts, pagination, and raw SQL. -- [Streaming & Live Queries](/sdk/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. -- [Pipes](/sdk/pipes) — Execute and manage named query pipes. -- [Admin & System](/sdk/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/reference) — Error codes, cancellation, the full API tree, and the codegen CLIs. +- [Queries](/sdk/queries): Tables, chainable query builder, inserts, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/streaming): SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/pipes): Execute and manage named query pipes. +- [Admin & System](/sdk/admin): Schema introspection, access-control policy, DLQ stats, and health checks. +- [Reference & CLI](/sdk/reference): Error codes, cancellation, the full API tree, and the codegen CLIs. diff --git a/docs/src/content/docs/sdk/setup/index.md b/docs/src/content/docs/sdk/setup/index.md new file mode 100644 index 00000000..cd1f47be --- /dev/null +++ b/docs/src/content/docs/sdk/setup/index.md @@ -0,0 +1,13 @@ +--- +title: "SDK setup" +description: "Package names, install commands, and version floors for the WaveHouse TypeScript and Go clients." +--- + +Setup is the only per-language part. Everything after it is documented once per topic, both languages side by side, starting at [Queries](/sdk/queries). + +| SDK | Package | Install | Requires | +| --- | ------- | ------- | -------- | +| [TypeScript](/sdk/setup/typescript) | `@wavehouse/sdk` | `npm install @wavehouse/sdk` | Node 22+, or any modern browser | +| [Go](/sdk/setup/go) | `github.com/Wave-RF/WaveHouse/clients/go` | `go get github.com/Wave-RF/WaveHouse/clients/go` | Go 1.24+ | + +Both have full API-tree parity, so pick the language your service already uses. diff --git a/docs/src/content/docs/sdk/setup/typescript.mdx b/docs/src/content/docs/sdk/setup/typescript.mdx index 083a9710..5683d081 100644 --- a/docs/src/content/docs/sdk/setup/typescript.mdx +++ b/docs/src/content/docs/sdk/setup/typescript.mdx @@ -5,7 +5,7 @@ description: "Installing @wavehouse/sdk, creating a client, custom headers and f import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components"; -Setup and caveats for `@wavehouse/sdk`, the TypeScript client: installation, runtimes, client construction, and the `Result` error model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing Go instead? [Go setup](/sdk/go) is the mirror of this page. +Setup and caveats for `@wavehouse/sdk`, the TypeScript client: installation, runtimes, client construction, and the `Result` error model. Usage is documented per topic, both languages side by side, starting at [Queries](/sdk/queries) — writing Go instead? [Go setup](/sdk/setup/go) is the mirror of this page. `@wavehouse/sdk` has exactly one runtime dependency — `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. @@ -562,6 +562,6 @@ The topic pages cover both SDKs, tabbed by language — the tab you pick here fo diff --git a/docs/src/content/docs/sdk/streaming.mdx b/docs/src/content/docs/sdk/streaming.mdx index 05c89bf4..68743878 100644 --- a/docs/src/content/docs/sdk/streaming.mdx +++ b/docs/src/content/docs/sdk/streaming.mdx @@ -13,7 +13,7 @@ Streams are Server-Sent Events over HTTP. The auth token rides in an `Authorizat ### The stream controller -Returned by the stream method on a table ref, a query builder, a pipe ref, and the DLQ namespace (the DLQ variant is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). +Returned by the stream method on a table ref, a query builder, a pipe ref, and the DLQ namespace (the DLQ variant is not yet functional server-side, [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). @@ -61,11 +61,11 @@ const unsub = stream.subscribe({ }, }); -// Cleanup — closes the connection if no other subscribers remain +// Cleanup: closes the connection if no other subscribers remain unsub(); ``` -A handler that throws *during delivery* doesn't end the stream; the exception is logged and the connection keeps running. But delivery of that event stops at the handler that threw — your *later* subscribers, and any concurrent `for await`, do not get it. The first `status` call, the synchronous one `.subscribe()` makes before returning, isn't caught at all and throws back out at you. Wrap your handler bodies in your own `try`/`catch`; see [Error Handling](/sdk/reference#error-handling) for the carve-outs. +A handler that throws *during delivery* doesn't end the stream; the exception is logged and the connection keeps running. But delivery of that event stops at the handler that threw: your *later* subscribers, and any concurrent `for await`, do not get it. The first `status` call, the synchronous one `.subscribe()` makes before returning, isn't caught at all and throws back out at you. Wrap your handler bodies in your own `try`/`catch`; see [Error Handling](/sdk/reference#error-handling) for the carve-outs. @@ -87,7 +87,7 @@ unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ }, }) -// Removes this subscriber only — the connection stays open for any others +// Removes this subscriber only; the connection stays open for any others // and still needs stream.Close() when you're done with the stream itself. defer unsub() ``` @@ -129,7 +129,7 @@ for event := range stream.Events() { ``` :::caution[`break` does not close the stream] -Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. +Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption: the background goroutine and HTTP connection persist. Always `defer stream.Close()`. ::: :::note[`Events()` carries events only] @@ -171,7 +171,7 @@ Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop cl ### Waiting for the stream to go live -Useful when you need the subscription established before doing something that depends on it — inserting a row you expect to see come back, for instance. +Useful when you need the subscription established before doing something that depends on it, such as inserting a row you expect to see come back. @@ -185,14 +185,14 @@ await stream.connected(); // wait until the transport is live await wh.from('clicks').insert({ page: '/home' }); ``` -A *timeout* rejection does **not** stop the transport — reconnection is unbounded, so it means "not live yet", not "given up"; call `.close()` if you want it to stop. A rejection because the stream closed is different: there the transport has already stopped. +A *timeout* rejection does **not** stop the transport: reconnection is unbounded, so it means "not live yet", not "given up"; call `.close()` if you want it to stop. A rejection because the stream closed is different: there the transport has already stopped. One way this rejects against a perfectly healthy stream: if a subscriber you registered *before* calling `.connected()` has a `status` handler that throws, the throw aborts the fan-out before `connected()`'s internal watcher sees `live`, so it times out while `.status` already reads `live`. See [Error Handling](/sdk/reference#error-handling). -`.Connected(ctx)` blocks until the stream reaches `StatusLive` or `ctx` is canceled; it returns an error if the stream closes before connecting. The deadline is yours to set — there is no built-in default. +`.Connected(ctx)` blocks until the stream reaches `StatusLive` or `ctx` is canceled; it returns an error if the stream closes before connecting. The deadline is yours to set: there is no built-in default. ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -222,7 +222,7 @@ if err := stream.Connected(ctx); err != nil { | ----- | ---- | ----------- | | `Since` | `string` | RFC3339 timestamp for gap-fill replay | -There is no `Signal`/context field: a stream is not canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. +There is no `Signal`/context field: a stream is not canceled by passing a `context.Context` into `.Stream()`. Call `.Close()` instead. @@ -240,7 +240,7 @@ interface StreamEvent { } ``` -Row values of top-level `DateTime`/`DateTime64` columns inside `data` (not timestamps nested in `Array`/`Map`/`Tuple` columns) arrive in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), matching what `/v1/query` returns for the same row — `new Date(value)` parses correctly with no zone fix-up. Values WaveHouse couldn't canonicalize (ingest is fail-open) stream in the producer's original spelling, and the `/v1/query` match doesn't hold for them: a spelling ClickHouse accepted anyway still queries back in canonical UTC (one it rejected never lands in the table at all), and a zone-less date-time is what `new Date()` reads as *local* time — though a date-only `YYYY-MM-DD` string is read as UTC, an ECMAScript quirk (see [Timestamp canonicalization](/api#timestamp-canonicalization)). +Row values of top-level `DateTime`/`DateTime64` columns inside `data` (not timestamps nested in `Array`/`Map`/`Tuple` columns) arrive in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), matching what `/v1/query` returns for the same row, so `new Date(value)` parses correctly with no zone fix-up. Values WaveHouse couldn't canonicalize (ingest is fail-open) stream in the producer's original spelling, and the `/v1/query` match doesn't hold for them: a spelling ClickHouse accepted anyway still queries back in canonical UTC (one it rejected never lands in the table at all), and a zone-less date-time is what `new Date()` reads as *local* time, though a date-only `YYYY-MM-DD` string is read as UTC, an ECMAScript quirk (see [Timestamp canonicalization](/api#timestamp-canonicalization)). @@ -253,7 +253,7 @@ type StreamEvent struct { } ``` -Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value, because the ingest handler rewrites them before publishing — a live frame and a later query can't disagree on the spelling of an instant. So a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open: a value the server can't parse, or whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value, because the ingest handler rewrites them before publishing: a live frame and a later query can't disagree on the spelling of an instant. So a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open: a value the server can't parse, or whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). @@ -263,9 +263,9 @@ Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3 A dropped stream reconnects on a jittered exponential backoff, capped at 30s, and resumes from the last event it saw so the server gap-fills what was missed. HTTP/2 is recommended. Reconnection is **unbounded**: a stream keeps re-dialing until it hits a terminal error or you close it. The client option that bounds retries applies to REST requests only and is never consulted here. :::caution[Resumption is at-least-once, and time-bounded] -Delivery across a reconnect is **at-least-once**. The resume point the client sends is the last event's `received_timestamp` — a `Last-Event-ID` header in TypeScript, a `?since=` query parameter in Go; the server accepts either and replays from that instant *inclusively* — so the last event you already saw, and anything sharing its timestamp, arrives again. Neither SDK deduplicates live frames — a live query makes one pass at the backfill seam only — so key on `timestamp` plus your own row identity if duplicates matter. +Delivery across a reconnect is **at-least-once**. The resume point the client sends is the last event's `received_timestamp`: a `Last-Event-ID` header in TypeScript, a `?since=` query parameter in Go. The server accepts either and replays from that instant *inclusively*, so the last event you already saw, and anything sharing its timestamp, arrives again. Neither SDK deduplicates live frames (a live query makes one pass at the backfill seam only), so key on `timestamp` plus your own row identity if duplicates matter. -Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration#message-queue-nats) — 15 minutes by default. A drop longer than that resumes with a hole and no signal, because the purged messages are simply gone. +Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration#message-queue-nats): 15 minutes by default. A drop longer than that resumes with a hole and no signal, because the purged messages are simply gone. ::: @@ -274,28 +274,28 @@ Replay is also bounded by the server's [`mq.gap_window_minutes`](/configuration# The backoff schedule resets only after a connection has *held* for a few seconds, so a server accepting and immediately closing — slow-consumer eviction, a half-broken upstream — can't pin the client at sub-second retries. :::note[SSE connection limit] -The SDK warns above 5 concurrent SSE connections — just under the browser's 6-per-domain limit for HTTP/1.1. HTTP/2 multiplexes over one connection, so the limit doesn't apply there. +The SDK warns above 5 concurrent SSE connections, just under the browser's 6-per-domain limit for HTTP/1.1. HTTP/2 multiplexes over one connection, so the limit doesn't apply there. ::: -A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the only 4xx it raises itself is `400` for a missing or empty table name, and only on the stream route — a `404` or `405` means the request never reached it, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. +A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure. In a browser going cross-origin, that only holds when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead, indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the only 4xx it raises itself is `400` for a missing or empty table name, and only on the stream route: a `404` or `405` means the request never reached it, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away. Note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. -Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request — which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk/typescript#supplying-your-own-fetch). +Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request, which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk/setup/typescript#supplying-your-own-fetch). -Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect — the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/reference#error-handling). +Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect: the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them; see the [error-code table](/sdk/reference#error-handling). -`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Creating a client](/sdk/go#creating-a-client)). +`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue (`ClientOptions.MaxRetries` bounds request retries only, not stream reconnects), so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Creating a client](/sdk/setup/go#creating-a-client)). ### Server-side policy filtering -Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims — on live frames and gap-fill replay alike. Claims are captured when the connection opens: a policy change applies from the next live event (an in-flight gap-fill replay finishes under the policy snapshot taken at connect), while token expiry or claim changes take effect on reconnect. +Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims, on live frames and gap-fill replay alike. Claims are captured when the connection opens: a policy change applies from the next live event (an in-flight gap-fill replay finishes under the policy snapshot taken at connect), while token expiry or claim changes take effect on reconnect. -Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control — where each rule is enforced](/access-control#where-each-rule-is-enforced). +Two things follow. **Event-id gaps are normal on a filtered stream**: a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove (an unresolvable claim, a type it can't compare) withholds the row rather than passing it. See [Access control: where each rule is enforced](/access-control#where-each-rule-is-enforced). ### Client-side stream filtering @@ -313,9 +313,9 @@ const stream = wh.from('clicks') // Only events where page === '/home' are emitted, with only page + button columns ``` -Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like`: the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). -Two of them need care, for different reasons. `like` matches **case-insensitively** here, while ClickHouse's `LIKE` on the query path is case-sensitive — so inside a single `liveQuery()` the backfill rows arrive through the server's case-sensitive filter and the live frames through this one, and a pattern like `'/Home%'` can admit live events whose historical counterparts the backfill excluded. `not_like` has no server-side counterpart at all: `/v1/query` rejects it with a `400` (see the operator table in [Queries](/sdk/queries#filtering)), so a `liveQuery()` filtered on it gets an error `Result` in `initial()`, drops whatever was buffered during the backfill window, and runs live-only from there. +Two of them need care, for different reasons. `like` matches **case-insensitively** here, while ClickHouse's `LIKE` on the query path is case-sensitive, so inside a single `liveQuery()` the backfill rows arrive through the server's case-sensitive filter and the live frames through this one, and a pattern like `'/Home%'` can admit live events whose historical counterparts the backfill excluded. `not_like` has no server-side counterpart at all: `/v1/query` rejects it with a `400` (see the operator table in [Queries](/sdk/queries#filtering)), so a `liveQuery()` filtered on it gets an error `Result` in `initial()`, drops whatever was buffered during the backfill window, and runs live-only from there. @@ -329,18 +329,18 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere. `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively, and `OpIn` accepts any Go slice type (e.g. `[]string`, `[]int`). +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike`: the `FilterOp` set `.Where()` takes everywhere. `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively, and `OpIn` accepts any Go slice type (e.g. `[]string`, `[]int`). **How values are compared.** The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: -- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload may read `2026-06-21T04:00:00Z` while your filter constant names the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a chronologically equal row. Both sides are parsed as instants instead. +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload may read `2026-06-21T04:00:00Z` while your filter constant names the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions: lexically the payload sorts *below* the constant, so `OpGte` would miss a chronologically equal row. Both sides are parsed as instants instead. - **Only unambiguous spellings count as instants:** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have, so it is not treated as a timestamp. -- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could admit rows the query path excludes. The usual cause is a zone-less filter constant: give it an offset. - **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. - **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. :::caution[Integer precision above 2^53] -Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond 2^53 has already lost exactness before any filter runs, and the server compares such columns in their exact storage domain — so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond 2^53 has already lost exactness before any filter runs, and the server compares such columns in their exact storage domain, so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. ::: @@ -350,7 +350,7 @@ Event data decodes through `encoding/json` into `map[string]any`, so JSON number ## Live Queries -Live queries combine a historical backfill with a real-time stream, providing a seamless initial load + live updates experience. They exist only on the query builder — there is no table-ref shortcut in either SDK. +Live queries combine a historical backfill with a real-time stream, providing a seamless initial load + live updates experience. They exist only on the query builder: there is no table-ref shortcut in either SDK. @@ -441,12 +441,12 @@ Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `Live 3. Deduplicates buffered events against the backfill. 4. Flushes remaining buffered events and switches to live mode. -This "stream-first" approach is what closes the window between the fetch and the stream starting — events arriving during the fetch are buffered rather than missed. It holds only when the backfill completes cleanly. +This "stream-first" approach is what closes the window between the fetch and the stream starting: events arriving during the fetch are buffered rather than missed. It holds only when the backfill completes cleanly. -Step 2 is skipped entirely if the fetch itself throws (see below). Step 3 deduplicates against the **last row** of the backfill result — which is the newest only when the query orders ascending. A `desc` query (like the example above) puts the oldest row last, so for live frames the boundary is the oldest timestamp and this step filters nothing. With a `since` gap-fill in flight it works the other way — replay lands in the same buffer, so replayed events at or older than that row are dropped before reaching `next()`. A projection that omits `received_timestamp` skips the pass entirely. Tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). +Step 2 is skipped entirely if the fetch itself throws (see below). Step 3 deduplicates against the **last row** of the backfill result, which is the newest only when the query orders ascending. A `desc` query (like the example above) puts the oldest row last, so for live frames the boundary is the oldest timestamp and this step filters nothing. With a `since` gap-fill in flight it works the other way: replay lands in the same buffer, so replayed events at or older than that row are dropped before reaching `next()`. A projection that omits `received_timestamp` skips the pass entirely. Tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). #### When the backfill doesn't complete @@ -454,22 +454,22 @@ Here is how it doesn't, and what each case looks like from your subscriber. "Buf | What happened | `initial()` | Live events | `error` | Buffered events | |---|---|---|---|---| -| Backfill returns an error `Result` | fires, `result.error` set | delivered | — | none delivered | -| Your `initial()` throws | fires, then throws | delivered | — | none delivered — the flush never starts | -| Your `next()` throws during the flush | fires normally | delivered, after the flush stops | — | none after the throw | -| `auth` rejects, the stream connects anyway | never fires | delivered | — | none delivered | +| Backfill returns an error `Result` | fires, `result.error` set | delivered | n/a | none delivered | +| Your `initial()` throws | fires, then throws | delivered | n/a | none delivered, the flush never starts | +| Your `next()` throws during the flush | fires normally | delivered, after the flush stops | n/a | none after the throw | +| `auth` rejects, the stream connects anyway | never fires | delivered | n/a | none delivered | | `auth` rejects for a few attempts, then recovers | never fires | delivered once it recovers | `SSE_AUTH_ERROR` per failed attempt | none delivered | -| `auth` keeps rejecting | never fires | none — never reaches `live` | `SSE_AUTH_ERROR` per attempt | none delivered | -| Relative `baseURL` | never fires | none — the stream is terminated too | terminal `SSE_CONNECT_ERROR` | none delivered | -| Your `status` handler throws on the first, synchronous call | never fires | none — buffered where you can't reach them | — | none delivered | +| `auth` keeps rejecting | never fires | none, never reaches `live` | `SSE_AUTH_ERROR` per attempt | none delivered | +| Relative `baseURL` | never fires | none, the stream is terminated too | terminal `SSE_CONNECT_ERROR` | none delivered | +| Your `status` handler throws on the first, synchronous call | never fires | none, buffered where you can't reach them | n/a | none delivered | **The `status`-throw row is the odd one out**, and everything below is written for the others. It is the one case where the backfill never *starts*: the throw escapes `liveQuery()` before the constructor reaches it, so you get no handle back, nothing can `.close()` the stream, and — because the buffering phase never ends — every event accumulates in an object you have no reference to, indefinitely. `next()` is never called at all. Passing `opts.signal` is the only way to stop it; not throwing is the fix. Throwing on any *later* `status` transition is isolated and logged like `next`. See [Error Handling](/sdk/reference#error-handling). -**What to do about the rest.** Check `result.error` inside `initial()`, keep the flush handlers total, and — if a missing backfill matters — treat `initial()` never firing as its own failure. Where `auth` rejects and the stream connects anyway nothing else will tell you, and where the other rows *do* raise an error it names `auth` or the URL, never the backfill. Re-run the fetch then; you never have to work out which row you hit. Leave it a moment first: events reach a stream from the message queue before the ingest worker lands them in ClickHouse, and its per-table batcher flushes on size or a deadline with the insert still to complete after that (see [Ingest pipeline](/ingest-pipeline)), so an immediate re-fetch can miss the newest rows. +**What to do about the rest.** Check `result.error` inside `initial()`, keep the flush handlers total, and if a missing backfill matters, treat `initial()` never firing as its own failure. Where `auth` rejects and the stream connects anyway nothing else will tell you, and where the other rows *do* raise an error it names `auth` or the URL, never the backfill. Re-run the fetch then; you never have to work out which row you hit. Leave it a moment first: events reach a stream from the message queue before the ingest worker lands them in ClickHouse, and its per-table batcher flushes on size or a deadline with the insert still to complete after that (see [Ingest pipeline](/ingest-pipeline)), so an immediate re-fetch can miss the newest rows. -**Why `auth` splits the way it does.** The backfill makes the *first* `auth()` call and gets exactly one shot — the token is minted above the REST retry loop, so a rejection there is never retried and the whole backfill is gone. The stream calls `auth()` again on every connection attempt and treats the same rejection as transient. That asymmetry is the entire reason a live query can end up running normally with no snapshot behind it, and why nothing announces it: `error` never fires, and the only trace is that `initial()` didn't. That is the case worth guarding against, because it is the one that looks like success. +**Why `auth` splits the way it does.** The backfill makes the *first* `auth()` call and gets exactly one shot: the token is minted above the REST retry loop, so a rejection there is never retried and the whole backfill is gone. The stream calls `auth()` again on every connection attempt and treats the same rejection as transient. That asymmetry is the entire reason a live query can end up running normally with no snapshot behind it, and why nothing announces it: `error` never fires, and the only trace is that `initial()` didn't. That is the case worth guarding against, because it is the one that looks like success. -**What the live stream can lose.** In every row where the backfill runs at all: nothing, except in one window — events arriving after the stream goes `live` but before the backfill fails are buffered, and the failure discards the buffer. On the `auth` rows that window opens only if the rejection arrives *after* the connection does: the backfill fails as soon as its own `auth()` call rejects, while the stream needs a second `auth()` call resolved **and** a connection opened. Which way that goes depends on your token provider, and you don't need to know — the recovery above covers both. +**What the live stream can lose.** In every row where the backfill runs at all: nothing, except in one window: events arriving after the stream goes `live` but before the backfill fails are buffered, and the failure discards the buffer. On the `auth` rows that window opens only if the rejection arrives *after* the connection does: the backfill fails as soon as its own `auth()` call rejects, while the stream needs a second `auth()` call resolved **and** a connection opened. Which way that goes depends on your token provider, and you don't need to know: the recovery above covers both. Tracked in [#473](https://github.com/Wave-RF/WaveHouse/issues/473). @@ -479,14 +479,14 @@ Tracked in [#473](https://github.com/Wave-RF/WaveHouse/issues/473). Step 2 runs `.FetchUntyped(ctx)` and then calls `sub.Initial(rows, err)`. Step 3 deduplicates against the maximum `received_timestamp` in the backfill (not necessarily the last row). :::caution[Dedup needs `received_timestamp` in the projection] -Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, so events in the overlap window are delivered twice — once via `Initial`, once via `Next`. +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, so events in the overlap window are delivered twice: once via `Initial`, once via `Next`. ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback — see [Queries → Filtering](/sdk/queries#filtering). +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive, so a live query filtering on `OpLike` may exclude rows from the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). `OpNotLike` is rejected by `/v1/query` with a `400`, failing the `Initial` callback; see [Queries → Filtering](/sdk/queries#filtering). ::: -**When the backfill doesn't complete.** `Initial(rows, err)` fires with the error, the events buffered during the fetch window are discarded, and live delivery continues from there — so a live query can end up running normally with no snapshot behind it. Check `err` inside `Initial` and re-run the fetch if a missing backfill matters; leave it a moment first, since events reach a stream from the message queue before the ingest worker lands them in ClickHouse (see [Ingest pipeline](/ingest-pipeline)). +**When the backfill doesn't complete.** `Initial(rows, err)` fires with the error, the events buffered during the fetch window are discarded, and live delivery continues from there, so a live query can end up running normally with no snapshot behind it. Check `err` inside `Initial` and re-run the fetch if a missing backfill matters; leave it a moment first, since events reach a stream from the message queue before the ingest worker lands them in ClickHouse (see [Ingest pipeline](/ingest-pipeline)). From ce066c9696bb3277d35f8433ad663ba65b0d12e0 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 13:55:36 -0400 Subject: [PATCH 55/59] fix(sdk): address CodeRabbit findings on the SDK docs merge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four threads, all raised against f778e41 and all correct. Two of them are cross-language leaks from the topic-page merge, which is now the second time that restructure has produced this class of bug (the first was `Last-Event-ID` in 23c58e2): nothing type-checks the snippets inside a ``, so a Go identifier surviving into the TypeScript tab builds clean. - admin.mdx's TypeScript tab called `wh.policy.validate(policyDraft)` against a variable it never declared. `policyDraft` exists only in the Go tab. Declared it and passed it to both `set` and `validate`, mirroring the Go tab's shape. - streaming.mdx's Go tab said `Auth` provider errors are retryable as `SSE_ERROR`. clients/go/stream.go:310 returns `SSE_AUTH_ERROR`; `SSE_ERROR` (:270) is the fallback for a failure that isn't already an *Error. The same page two paragraphs earlier, and the error table in reference.mdx, both had it right, so the page contradicted itself. - .claude/commands/cover.md said `make test-all` runs "every suite". It runs test-unit, test-sdk, test-integration, test-e2e and cov — test-sdk is test-sdk-go + test-sdk-ts, so test-sdk-go-e2e is never invoked. Named the exclusion and why it's excluded. - parseArgs and flagValue read os.Args directly, and the test helper mutated the global to drive them, against the `**/*.go` guideline that dependencies are passed explicitly. Both now take argv, and parseArgs takes the resolved WAVEHOUSE_AUTH value, so main owns the two reads of process state. withArgs and its t.Setenv are gone. Not done: CodeRabbit's stated motive for the last one was enabling t.Parallel(). There is no t.Parallel() anywhere in clients/go, so adding it here would invent a convention the package doesn't have. The injection fixes the guideline violation on its own. --- .claude/commands/cover.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 33 ++++++++++--------- clients/go/cmd/wavehouse-codegen/main_test.go | 20 ++--------- docs/src/content/docs/sdk/admin.mdx | 5 +-- docs/src/content/docs/sdk/streaming.mdx | 2 +- 5 files changed, 26 insertions(+), 36 deletions(-) diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 8aed9ddb..a760c242 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -17,7 +17,7 @@ Behavior: - **ts-unit**: `make test-sdk-ts` (TS SDK unit tests + coverage + gate against `suites.ts-unit`) - **ts-e2e**: emitted as a side effect of `make test-e2e` (the orchestrator always passes `--coverage` to the e2e vitest run; informational only, no standalone gate) - **ts-total**: `make cov` (runs `cov report` — one consolidated Go + TS summary with per-suite HTML links + all gates; fails if *no* suite has data) -- **all**: `make test-all` (every suite sequentially + `make cov`) +- **all**: `make test-all` (every coverage-producing suite sequentially + `make cov`; `test-sdk-go-e2e` is excluded, it needs a live server it did not start and runs uninstrumented) After the run completes: diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 536488ac..28a87c6f 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -26,30 +26,33 @@ type cliArgs struct { pkg string } -// flagValue consumes and returns the value following os.Args[*i], exiting +// flagValue consumes and returns the value following argv[*i], exiting // rather than silently falling back to the default when it is missing. -func flagValue(i *int) string { - flag := os.Args[*i] +func flagValue(argv []string, i *int) string { + flag := argv[*i] *i++ - if *i >= len(os.Args) { + if *i >= len(argv) { fmt.Fprintf(os.Stderr, "Error: missing value for %s (use --help)\n", flag) os.Exit(2) } - return os.Args[*i] + return argv[*i] } -func parseArgs() cliArgs { +// parseArgs takes argv (including argv[0]) and the WAVEHOUSE_AUTH value rather +// than reading os.Args / the environment itself, so tests drive it with their +// own fixtures instead of mutating process-global state. +func parseArgs(argv []string, envAuth string) cliArgs { args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} - for i := 1; i < len(os.Args); i++ { - switch os.Args[i] { + for i := 1; i < len(argv); i++ { + switch argv[i] { case "--url", "-u": - args.url = flagValue(&i) + args.url = flagValue(argv, &i) case "--out", "-o": - args.out = flagValue(&i) + args.out = flagValue(argv, &i) case "--auth", "-a": - args.auth = flagValue(&i) + args.auth = flagValue(argv, &i) case "--package", "-p": - args.pkg = flagValue(&i) + args.pkg = flagValue(argv, &i) case "--help", "-h": fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema @@ -63,12 +66,12 @@ Options: --help, -h Show this help`) os.Exit(0) default: - fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", os.Args[i]) + fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", argv[i]) os.Exit(2) } } if args.auth == "" { - args.auth = os.Getenv("WAVEHOUSE_AUTH") + args.auth = envAuth } return args } @@ -314,7 +317,7 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { } func main() { - args := parseArgs() + args := parseArgs(os.Args, os.Getenv("WAVEHOUSE_AUTH")) fmt.Printf("Fetching schema from %s...\n", args.url) schemas, err := fetchSchemas(context.Background(), args.url, args.auth) diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 1b6d0a82..125a6fc8 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -180,16 +180,6 @@ func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { } } -// withArgs points os.Args at args for the duration of the test. parseArgs and -// flagValue read the global directly, so tests driving them must not run in -// parallel. -func withArgs(t *testing.T, args []string) { - t.Helper() - saved := os.Args - t.Cleanup(func() { os.Args = saved }) - os.Args = args -} - func TestFlagValue(t *testing.T) { tests := []struct { name string @@ -203,9 +193,8 @@ func TestFlagValue(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - withArgs(t, tt.args) i := 1 - if got := flagValue(&i); got != tt.want { + if got := flagValue(tt.args, &i); got != tt.want { t.Errorf("flagValue() = %q, want %q", got, tt.want) } if i != tt.wantI { @@ -247,9 +236,7 @@ func TestParseArgs(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - t.Setenv("WAVEHOUSE_AUTH", tt.env) // empty reads back the same as unset - withArgs(t, tt.args) - if got := parseArgs(); got != tt.want { + if got := parseArgs(tt.args, tt.env); got != tt.want { t.Errorf("parseArgs() = %+v, want %+v", got, tt.want) } }) @@ -265,8 +252,7 @@ const parseArgsChildEnv = "WAVEHOUSE_CODEGEN_TEST_ARGS" // output. func TestParseArgsExitPaths(t *testing.T) { if raw, ok := os.LookupEnv(parseArgsChildEnv); ok { - withArgs(t, append([]string{"wavehouse-codegen"}, strings.Fields(raw)...)) - parseArgs() + parseArgs(append([]string{"wavehouse-codegen"}, strings.Fields(raw)...), "") t.Fatal("parseArgs returned instead of exiting") } tests := []struct { diff --git a/docs/src/content/docs/sdk/admin.mdx b/docs/src/content/docs/sdk/admin.mdx index eb0bccb6..d736cbda 100644 --- a/docs/src/content/docs/sdk/admin.mdx +++ b/docs/src/content/docs/sdk/admin.mdx @@ -60,7 +60,7 @@ Manage Hasura-style access control policies. const { data: policy } = await wh.policy.get(); // Update policy -await wh.policy.set({ +const policyDraft = { default_role: 'viewer', admin_role: 'admin', tables: { @@ -74,7 +74,8 @@ await wh.policy.set({ }, }, }, -}); +}; +await wh.policy.set(policyDraft); // Validate without applying (dry run) const { data } = await wh.policy.validate(policyDraft); diff --git a/docs/src/content/docs/sdk/streaming.mdx b/docs/src/content/docs/sdk/streaming.mdx index 68743878..198afa32 100644 --- a/docs/src/content/docs/sdk/streaming.mdx +++ b/docs/src/content/docs/sdk/streaming.mdx @@ -286,7 +286,7 @@ Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR` and `SSE_READ_ERROR`). `SSE_PARSE_ERROR` is retryable but does *not* reconnect: the offending frame is dropped and the same connection carries on. Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them; see the [error-code table](/sdk/reference#error-handling). -`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue (`ClientOptions.MaxRetries` bounds request retries only, not stream reconnects), so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Creating a client](/sdk/setup/go#creating-a-client)). +`/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`; a `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_AUTH_ERROR`) and reconnects continue (`ClientOptions.MaxRetries` bounds request retries only, not stream reconnects), so call `.Close()` if the provider fails permanently. Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Creating a client](/sdk/setup/go#creating-a-client)). From ff35c8b1b60075b764edbb2d99b4bc17f6a13a92 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 14:53:05 -0400 Subject: [PATCH 56/59] fix(sdk): harden wavehouse-codegen and correct three stale docs claims CodeRabbit's 18:11Z review on ce066c9: two inline threads plus six findings GitHub could not post inline, which live only in the review body and never show up as unresolved threads. Seven of the eight are taken here; the type dedup is deliberately not (see below). Code injection via schema names (CWE-94) was real and demonstrable, not theoretical. pascalCase splits only on space/_/-/., so tabs and newlines survive it, and a keyword inside a part keeps its lowercase; the result is written unquoted and format.Source only *parses*. A column named with a tab-separated payload closed the struct and appended a top-level `func Pwn() string { return "owned" }` that format.Source accepted with no error. Type and field identifiers are now validated before emission, which also covers the backtick case (a backtick in a name closes the struct tag's raw string literal). The regression test uses that exact payload. --package is validated on the same path -- not a trust boundary, but the same class of broken output. - chTypeToGo mapped Map(Array(...), String) to `map[[]T]V`, which parses and then fails to compile in the caller's build. ClickHouse restricts Map keys to comparable types so a real server shouldn't send this, but format.Source can't catch it. Falls back to `any`. - fetchSchemas sent the bearer token over whatever scheme the URL carried (CWE-319), and left redirects to net/http, which drops Authorization only on a host change and ignores scheme and port (CWE-522) -- so an https->http hop on the same host handed the token over in cleartext. Credentials now require https or loopback, and any redirect changing scheme/host/port is refused. This matches the stance the SDK already takes on streams (SSE_REDIRECT). Documented in reference.mdx, since it's user-visible CLI behavior; both documented invocations use http://localhost and are unaffected. - CHANGELOG still gave the setup routes as /sdk/typescript and /sdk/go after f778e41 moved them under /sdk/setup/, contradicting the paths listed earlier in its own entry, and claimed all four SDK targets use gotestsum and honor V=1 -- test-sdk-ts is vitest. - CONTRIBUTING said the topic pages carry "one block per topic". They carry one per section (queries.mdx has 19), which was deliberate so the ToC doesn't list every heading twice. - development.md had two `clients/` entries in the file tree; I added one without removing the old one, which listed only ts/. Folded the surviving detail (pnpm workspace) into the current entry. Not done: reusing Column/TableSchema from clients/go/types.go in the CLI. That makes cmd/ depend on the SDK package, which is a design change rather than a fix, and this PR is already large. go-sdk coverage 87.1% -> 87.4%. --- CHANGELOG.md | 2 +- CONTRIBUTING.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 92 +++++++++- clients/go/cmd/wavehouse-codegen/main_test.go | 163 ++++++++++++++++++ docs/src/content/docs/development.md | 4 +- docs/src/content/docs/sdk/reference.mdx | 2 + 6 files changed, 257 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3743330..dee0513b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); all four use gotestsum and honor `ARGS`/`V=1` like every other Go suite. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page — `/sdk/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/go`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); the Go suites use gotestsum and honor `ARGS`/`V=1` like every other Go suite, while `test-sdk-ts` runs vitest. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language-specific section, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page under `/sdk/setup/` — `/sdk/setup/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/setup/go`, indexed by `/sdk/setup`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4092c665..97e59621 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` - - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (one `` block per topic) plus the per-language setup pages `sdk/setup/typescript.mdx` / `sdk/setup/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (a `` block per language-specific section, with one `` per language) plus the per-language setup pages `sdk/setup/typescript.mdx` / `sdk/setup/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 28a87c6f..4e60272a 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -9,9 +9,12 @@ package main import ( "context" "encoding/json" + "errors" "fmt" "go/format" + "net" "net/http" + "net/url" "os" "slices" "strings" @@ -87,16 +90,54 @@ type tableSchema struct { Columns []column `json:"columns"` } +// requireSecureAuthURL refuses to put a bearer token on the wire in cleartext. +// Loopback is exempt: it is the default target and never leaves the machine. +func requireSecureAuthURL(u *url.URL) error { + if u.Scheme == "https" || isLoopback(u.Hostname()) { + return nil + } + return fmt.Errorf("refusing to send credentials to %s over %s: use an https:// URL, or drop --auth/WAVEHOUSE_AUTH", u.Host, u.Scheme) +} + +func isLoopback(host string) bool { + if host == "localhost" { + return true + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} + +// refuseCredentialedRedirect stops a redirect from carrying the bearer token +// somewhere it wasn't meant for. net/http drops Authorization only when the +// host changes, ignoring scheme and port, so its default policy would hand the +// token to a plaintext hop on the same host. The SDK takes the same stance on +// streams (SSE_REDIRECT). +func refuseCredentialedRedirect(req *http.Request, via []*http.Request) error { + if len(via) >= 10 { + return errors.New("stopped after 10 redirects") + } + prev := via[len(via)-1].URL + if req.URL.Scheme != prev.Scheme || req.URL.Host != prev.Host { + return fmt.Errorf("refusing redirect from %s://%s to %s://%s while sending credentials", + prev.Scheme, prev.Host, req.URL.Scheme, req.URL.Host) + } + return nil +} + func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { url := strings.TrimRight(baseURL, "/") + "/v1/ops/schema" req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return nil, fmt.Errorf("build schema request for %s: %w", url, err) } + client := &http.Client{Timeout: 30 * time.Second} if auth != "" { + if err := requireSecureAuthURL(req.URL); err != nil { + return nil, err + } req.Header.Set("Authorization", "Bearer "+auth) + client.CheckRedirect = refuseCredentialedRedirect } - client := &http.Client{Timeout: 30 * time.Second} resp, err := client.Do(req) if err != nil { return nil, fmt.Errorf("fetch schema from %s: %w", url, err) @@ -202,6 +243,13 @@ func chTypeToGo(chType string) string { if comma != -1 { k := chTypeToGo(strings.TrimSpace(inner[:comma])) v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) + // ClickHouse restricts Map keys to comparable types, so this only + // fires on something a real server shouldn't send — but `map[[]T]V` + // parses fine and then fails to compile, and format.Source only + // parses. Fall back rather than emit source that can't build. + if !comparableGoType(k) { + return "any" + } return "map[" + k + "]" + v } return "map[string]any" @@ -209,6 +257,12 @@ func chTypeToGo(chType string) string { return "any" } +// comparableGoType reports whether t is usable as a Go map key. chTypeToGo +// only ever returns primitives, slices ("[]T", json.RawMessage), or maps. +func comparableGoType(t string) bool { + return !strings.HasPrefix(t, "[]") && !strings.HasPrefix(t, "map[") && t != "json.RawMessage" +} + func findTopLevelComma(s string) int { depth := 0 for i := range len(s) { @@ -260,9 +314,34 @@ func sortedKeys(m map[string]tableSchema) []string { return names } +// validGoIdent reports whether s is a legal Go identifier. Schema names reach +// the generated source as identifiers via pascalCase, which only splits on +// space/_/-/. — tabs, newlines, braces and backticks survive it verbatim. Since +// identifiers are written unquoted and format.Source only *parses*, a crafted +// column name can otherwise close the struct and append arbitrary top-level +// declarations to a file the caller then compiles. +func validGoIdent(s string) bool { + if s == "" { + return false + } + for i, r := range s { + switch { + case r == '_' || unicode.IsLetter(r): + case i > 0 && unicode.IsDigit(r): + default: + return false + } + } + return true +} + func generate(schemas map[string]tableSchema, pkg string) (string, error) { var sb strings.Builder + if !validGoIdent(pkg) { + return "", fmt.Errorf("--package %q is not a valid Go package name", pkg) + } + names := sortedKeys(schemas) // json.Number fields (128/256-bit integer columns) need the import. @@ -279,12 +358,16 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { sb.WriteString("import \"encoding/json\"\n\n") } - // pascalCase is not injective and format.Source only parses, so a - // collision would otherwise be written out as a non-compiling file. + // pascalCase is neither injective nor escaping, and format.Source only + // parses: an unchecked name is a non-compiling file at best and injected + // source at worst. seenTypes := make(map[string]string, len(names)) for _, name := range names { schema := schemas[name] typeName := pascalCase(name) + "Row" + if !validGoIdent(typeName) { + return "", fmt.Errorf("table %q does not map to a usable Go type name (got %q)", name, typeName) + } if prev, dup := seenTypes[typeName]; dup { return "", fmt.Errorf("tables %q and %q both map to type %q; rename one or generate separately", prev, name, typeName) } @@ -294,6 +377,9 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { for _, col := range schema.Columns { goType := chTypeToGo(col.Type) fieldName := pascalCase(col.Name) + if !validGoIdent(fieldName) { + return "", fmt.Errorf("table %q: column %q does not map to a usable Go field name (got %q)", name, col.Name, fieldName) + } if prev, dup := seenFields[fieldName]; dup { return "", fmt.Errorf("table %q: columns %q and %q both map to field %q", name, prev, col.Name, fieldName) } diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 125a6fc8..2e1a02c9 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -8,6 +8,7 @@ import ( "io" "net/http" "net/http/httptest" + "net/url" "os" "os/exec" "reflect" @@ -401,3 +402,165 @@ func TestSortedKeys(t *testing.T) { t.Errorf("sortedKeys() = %v, want %v", got, want) } } + +// injectedColumnName is the payload that reached valid Go before identifiers +// were validated: pascalCase splits on space/_/-/. only, so tabs and newlines +// pass through and keywords inside a part keep their lowercase. It closed the +// struct and appended a function that format.Source accepted without error. +const injectedColumnName = "x string\n}\n\nfunc\tPwn()\tstring\t{\n\treturn\t\"owned\"\n}\n\ntype\tT2Row\tstruct\t{\n\tY" + +func TestGenerate_RejectsInjectedNames(t *testing.T) { + tests := []struct { + name string + schemas map[string]tableSchema + wantErr string + }{ + { + name: "column name breaks out of the struct", + schemas: map[string]tableSchema{"t": {Name: "t", Columns: []column{{Name: injectedColumnName, Type: "String"}}}}, + wantErr: "usable Go field name", + }, + { + name: "backtick in a column name would close the struct tag", + schemas: map[string]tableSchema{"t": {Name: "t", Columns: []column{{Name: "a`b", Type: "String"}}}}, + wantErr: "usable Go field name", + }, + { + name: "table name breaks out of the declaration", + schemas: map[string]tableSchema{injectedColumnName: {Name: injectedColumnName, Columns: []column{{Name: "a", Type: "String"}}}}, + wantErr: "usable Go type name", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + out, err := generate(tt.schemas, "main") + if err == nil { + t.Fatalf("generate() succeeded; output was:\n%s", out) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("error = %v, want it to mention %q", err, tt.wantErr) + } + }) + } +} + +func TestGenerate_RejectsInvalidPackageName(t *testing.T) { + _, err := generate(map[string]tableSchema{}, "main\n\nfunc\tPwn()\t{}") + if err == nil || !strings.Contains(err.Error(), "valid Go package name") { + t.Fatalf("want invalid-package error, got %v", err) + } +} + +func TestValidGoIdent(t *testing.T) { + tests := map[string]bool{ + "Page": true, "X2fa": true, "_x": true, "A1_b2": true, + "": false, "2fa": false, "A`b": false, "A B": false, "A\tB": false, "A\nB": false, "A{}": false, + } + for in, want := range tests { + if got := validGoIdent(in); got != want { + t.Errorf("validGoIdent(%q) = %v, want %v", in, got, want) + } + } +} + +// Go map keys must be comparable. ClickHouse won't emit these key types, but +// format.Source only parses, so an unguarded `map[[]T]V` would be written out +// and then fail to compile in the caller's build. +func TestChTypeToGo_NonComparableMapKey(t *testing.T) { + tests := map[string]string{ + "Map(Array(Map(Float64, String)), String)": "any", + "Map(Array(UInt8), String)": "any", + "Map(Map(String, String), String)": "any", + // Comparable keys are unaffected. + "Map(String, String)": "map[string]string", + "Map(LowCardinality(String), Int8)": "map[string]int8", + } + for in, want := range tests { + if got := chTypeToGo(in); got != want { + t.Errorf("chTypeToGo(%q) = %q, want %q", in, got, want) + } + } +} + +func TestRequireSecureAuthURL(t *testing.T) { + tests := []struct { + raw string + wantErr bool + }{ + {raw: "https://wh.example.com/v1/ops/schema"}, + {raw: "http://localhost:8080/v1/ops/schema"}, + {raw: "http://127.0.0.1:8080/v1/ops/schema"}, + {raw: "http://[::1]:8080/v1/ops/schema"}, + {raw: "http://wh.example.com/v1/ops/schema", wantErr: true}, + {raw: "http://10.0.0.5:8080/v1/ops/schema", wantErr: true}, + } + for _, tt := range tests { + t.Run(tt.raw, func(t *testing.T) { + u, err := url.Parse(tt.raw) + if err != nil { + t.Fatal(err) + } + if err := requireSecureAuthURL(u); (err != nil) != tt.wantErr { + t.Errorf("requireSecureAuthURL(%q) error = %v, wantErr %v", tt.raw, err, tt.wantErr) + } + }) + } +} + +func TestFetchSchemas_RefusesCleartextCredentials(t *testing.T) { + // Fails before any connection is attempted, so the unroutable host is safe. + _, err := fetchSchemas(t.Context(), "http://wh.example.com", "tok-123") + if err == nil || !strings.Contains(err.Error(), "refusing to send credentials") { + t.Fatalf("want cleartext-credential refusal, got %v", err) + } + // Without credentials there is nothing to protect, so the scheme check + // must not fire; this one fails at connect instead. + if _, err := fetchSchemas(t.Context(), "http://wh.invalid", ""); err != nil && + strings.Contains(err.Error(), "refusing to send credentials") { + t.Errorf("unauthenticated request was refused for its scheme: %v", err) + } +} + +func TestFetchSchemas_RedirectsWithCredentials(t *testing.T) { + const body = `{"clicks":{"name":"clicks","columns":[{"name":"page","type":"String"}]}}` + + t.Run("cross-origin redirect is refused", func(t *testing.T) { + // A second server means a different port, so this also covers the + // port change that net/http's own policy ignores. + dst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, body) + })) + defer dst.Close() + src := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, dst.URL+"/v1/ops/schema", http.StatusFound) + })) + defer src.Close() + + if _, err := fetchSchemas(t.Context(), src.URL, "tok-123"); err == nil || + !strings.Contains(err.Error(), "refusing redirect") { + t.Fatalf("want redirect refusal, got %v", err) + } + }) + + t.Run("same-origin redirect is followed", func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/moved" { + http.Redirect(w, r, "/moved", http.StatusFound) + return + } + if got := r.Header.Get("Authorization"); got != "Bearer tok-123" { + t.Errorf("Authorization after same-origin redirect = %q", got) + } + _, _ = io.WriteString(w, body) + })) + defer srv.Close() + + got, err := fetchSchemas(t.Context(), srv.URL, "tok-123") + if err != nil { + t.Fatalf("fetchSchemas: %v", err) + } + if _, ok := got["clicks"]; !ok { + t.Errorf("schemas = %v, want a clicks entry", got) + } + }) +} diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 74a4c01e..ea553c81 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -475,7 +475,7 @@ WaveHouse/ │ ├── stream/ # SSE fan-out: Hub, Subscriber queue, Bucket, keepalive wheel │ └── testutil/ # Shared test helpers and mocks ├── clients/ # Official SDKs -│ ├── ts/ # TypeScript SDK (@wavehouse/sdk) +│ ├── ts/ # TypeScript SDK (@wavehouse/sdk, pnpm workspace) │ └── go/ # Go SDK — a NESTED Go module (own go.mod, invisible │ # to root `go list`; hence the test-sdk-go* targets) ├── tests/ # Integration & E2E tests @@ -484,8 +484,6 @@ WaveHouse/ │ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) │ ├── fixtures/ # ClickHouse DDL + config/policy fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) -├── clients/ # Client SDKs -│ └── ts/ # TypeScript SDK (@wavehouse/sdk, pnpm workspace) ├── deployments/ │ ├── compose/ # Docker Compose files (standalone.yaml, dependencies.yaml) │ ├── Dockerfile # Runtime image diff --git a/docs/src/content/docs/sdk/reference.mdx b/docs/src/content/docs/sdk/reference.mdx index cd277b5f..a5c48eb2 100644 --- a/docs/src/content/docs/sdk/reference.mdx +++ b/docs/src/content/docs/sdk/reference.mdx @@ -318,6 +318,8 @@ go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history and process listings. +When a token is supplied, the URL must be `https://` or a loopback host: the CLI refuses to send credentials in cleartext, and refuses any redirect that changes scheme, host, or port rather than let `net/http` carry the token along (it drops `Authorization` only on a host change, ignoring scheme and port). Unauthenticated runs are unrestricted. + **Options:** | Flag | Description | Default | From 607d00004a36d1057194e66ea5864488bf49575a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 17:15:20 -0400 Subject: [PATCH 57/59] fix(sdk): gate generated map keys on encoding/json, not Go comparability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit's 20:24Z review on ff35c8b. All three findings hold. The map-key guard added in ff35c8b tested the wrong property. Go comparability keeps `map[[]T]V` out of the output, but `map[float64]V` and `map[bool]V` are comparable, compile, and then fail at Marshal and Unmarshal: encoding/json supports only string-kinded and integer-kinded keys plus encoding.TextMarshaler implementors. That is worse than the non-compiling case, because it surfaces at runtime in the caller's service rather than in their build. chTypeToGo returns no TextMarshaler, so the check is now a closed allow-list of its own outputs, which also covers `*T` from Nullable — comparable, and equally unmarshalable. TestJSONMapKeyMatchesEncodingJSON pins that allow-list to what encoding/json actually does rather than to a reading of its docs: it marshals a real map for each candidate key type and asserts the outcome agrees with jsonMapKey. - validGoIdent accepted "_" and keywords like "type". Neither can arise for a type or field name (pascalCase capitalizes the first rune of each part, and strips underscores entirely), but --package is passed through to the `package` clause verbatim, so `--package type` wrote a file that could not parse. Replaced the hand-rolled loop with token.IsIdentifier, which excludes keywords, plus an explicit "_" rejection since the stdlib check admits it. - CONTRIBUTING listed the two setup pages as `sdk/setup/typescript.mdx` and `sdk/setup/go.md` in a bullet whose other entries are repo-root paths, so neither resolved from the repository root. AGENTS.md is unaffected: it names the routes, not the files. go-sdk coverage holds at 87.4%. --- CONTRIBUTING.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 44 ++++++------- clients/go/cmd/wavehouse-codegen/main_test.go | 63 ++++++++++++++++--- 3 files changed, 79 insertions(+), 30 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 97e59621..3ce264fd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` - - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (a `` block per language-specific section, with one `` per language) plus the per-language setup pages `sdk/setup/typescript.mdx` / `sdk/setup/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), the shared topic pages under `docs/src/content/docs/sdk/` (a `` block per language-specific section, with one `` per language) plus the per-language setup pages `docs/src/content/docs/sdk/setup/typescript.mdx` / `docs/src/content/docs/sdk/setup/go.md`, and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 4e60272a..0ab2e28a 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -12,6 +12,7 @@ import ( "errors" "fmt" "go/format" + "go/token" "net" "net/http" "net/url" @@ -243,11 +244,11 @@ func chTypeToGo(chType string) string { if comma != -1 { k := chTypeToGo(strings.TrimSpace(inner[:comma])) v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) - // ClickHouse restricts Map keys to comparable types, so this only - // fires on something a real server shouldn't send — but `map[[]T]V` - // parses fine and then fails to compile, and format.Source only - // parses. Fall back rather than emit source that can't build. - if !comparableGoType(k) { + // Go comparability is not enough: `map[[]T]V` parses and fails to + // compile, and `map[float64]V` compiles and then fails at + // Marshal/Unmarshal, which is worse. format.Source only parses, so + // neither is caught downstream. Fall back instead. + if !jsonMapKey(k) { return "any" } return "map[" + k + "]" + v @@ -257,10 +258,20 @@ func chTypeToGo(chType string) string { return "any" } -// comparableGoType reports whether t is usable as a Go map key. chTypeToGo -// only ever returns primitives, slices ("[]T", json.RawMessage), or maps. -func comparableGoType(t string) bool { - return !strings.HasPrefix(t, "[]") && !strings.HasPrefix(t, "map[") && t != "json.RawMessage" +// jsonMapKey reports whether t is usable as a map key that encoding/json can +// round-trip. It supports string-kinded and integer-kinded keys plus +// encoding.TextMarshaler implementors; chTypeToGo returns none of the latter, +// so this is the complete allow-list of its outputs. Everything else it can +// return — bool, the floats, any, *T, []T, json.RawMessage, nested maps — is +// either non-comparable or silently unmarshalable. +func jsonMapKey(t string) bool { + switch t { + case "string", "json.Number", + "int8", "int16", "int32", "int64", + "uint8", "uint16", "uint32", "uint64": + return true + } + return false } func findTopLevelComma(s string) int { @@ -321,18 +332,9 @@ func sortedKeys(m map[string]tableSchema) []string { // column name can otherwise close the struct and append arbitrary top-level // declarations to a file the caller then compiles. func validGoIdent(s string) bool { - if s == "" { - return false - } - for i, r := range s { - switch { - case r == '_' || unicode.IsLetter(r): - case i > 0 && unicode.IsDigit(r): - default: - return false - } - } - return true + // token.IsIdentifier rejects the empty string and keywords; "_" satisfies + // it but is not a usable package or field name. + return s != "_" && token.IsIdentifier(s) } func generate(schemas map[string]tableSchema, pkg string) (string, error) { diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 2e1a02c9..afae26ed 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -445,9 +445,19 @@ func TestGenerate_RejectsInjectedNames(t *testing.T) { } func TestGenerate_RejectsInvalidPackageName(t *testing.T) { - _, err := generate(map[string]tableSchema{}, "main\n\nfunc\tPwn()\t{}") - if err == nil || !strings.Contains(err.Error(), "valid Go package name") { - t.Fatalf("want invalid-package error, got %v", err) + for _, pkg := range []string{ + "main\n\nfunc\tPwn()\t{}", // injection + "type", // a keyword is not a package name + "_", // blank identifier + "", // empty + "2fa", // leading digit + } { + t.Run(pkg, func(t *testing.T) { + if _, err := generate(map[string]tableSchema{}, pkg); err == nil || + !strings.Contains(err.Error(), "valid Go package name") { + t.Fatalf("generate(pkg=%q) error = %v, want an invalid-package error", pkg, err) + } + }) } } @@ -455,6 +465,8 @@ func TestValidGoIdent(t *testing.T) { tests := map[string]bool{ "Page": true, "X2fa": true, "_x": true, "A1_b2": true, "": false, "2fa": false, "A`b": false, "A B": false, "A\tB": false, "A\nB": false, "A{}": false, + // Keywords and the blank identifier are not usable names. + "type": false, "func": false, "_": false, } for in, want := range tests { if got := validGoIdent(in); got != want { @@ -463,17 +475,27 @@ func TestValidGoIdent(t *testing.T) { } } -// Go map keys must be comparable. ClickHouse won't emit these key types, but -// format.Source only parses, so an unguarded `map[[]T]V` would be written out -// and then fail to compile in the caller's build. -func TestChTypeToGo_NonComparableMapKey(t *testing.T) { +// A generated map key has to survive two hurdles: Go comparability (or the +// file won't compile) and encoding/json support (or it compiles and then fails +// at Marshal/Unmarshal). format.Source only parses, so neither is caught +// downstream — chTypeToGo falls back to `any` rather than emit either. +func TestChTypeToGo_UnsupportedMapKey(t *testing.T) { tests := map[string]string{ + // Not comparable — would not compile. "Map(Array(Map(Float64, String)), String)": "any", "Map(Array(UInt8), String)": "any", "Map(Map(String, String), String)": "any", - // Comparable keys are unaffected. + // Comparable, but encoding/json cannot use them as keys. + "Map(Float64, String)": "any", + "Map(Float32, String)": "any", + "Map(Bool, String)": "any", + "Map(Nullable(String), String)": "any", + // Supported keys are unaffected. "Map(String, String)": "map[string]string", "Map(LowCardinality(String), Int8)": "map[string]int8", + "Map(UInt64, String)": "map[uint64]string", + "Map(UInt128, String)": "map[json.Number]string", + "Map(DateTime64(3, 'UTC'), Int32)": "map[string]int32", } for in, want := range tests { if got := chTypeToGo(in); got != want { @@ -482,6 +504,31 @@ func TestChTypeToGo_NonComparableMapKey(t *testing.T) { } } +// TestJSONMapKeyMatchesEncodingJSON pins the allow-list to encoding/json's +// actual behavior rather than a reading of its docs: every type jsonMapKey +// admits must round-trip as a map key, and the ones it rejects must not. +func TestJSONMapKeyMatchesEncodingJSON(t *testing.T) { + // Non-comparable candidates ([]T, json.RawMessage, nested maps) can't be + // written as a map key at all, so they never reach encoding/json. + probes := map[string]any{ + "string": map[string]string{"k": "v"}, + "json.Number": map[json.Number]string{"1": "v"}, + "int64": map[int64]string{1: "v"}, + "uint8": map[uint8]string{1: "v"}, + "bool": map[bool]string{true: "v"}, + "float64": map[float64]string{1.5: "v"}, + "any": map[any]string{"k": "v"}, + } + for typ, v := range probes { + t.Run(typ, func(t *testing.T) { + _, err := json.Marshal(v) + if want := jsonMapKey(typ); (err == nil) != want { + t.Errorf("json.Marshal(map[%s]...) error = %v, but jsonMapKey(%q) = %v", typ, err, typ, want) + } + }) + } +} + func TestRequireSecureAuthURL(t *testing.T) { tests := []struct { raw string From 654924ad5edeb3a808e5b5659d5d09df1a87b570 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 17:32:28 -0400 Subject: [PATCH 58/59] refactor(sdk)!: drop the Go codegen CLI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The TypeScript codegen it was built for parity with is itself almost untested and unused, so matching it bought a second copy of a feature nobody runs. It was also the most defect-dense code on the branch: the last three CodeRabbit rounds were all codegen findings (identifier injection, non-comparable map keys, JSON-incompatible map keys, cleartext credentials, redirect credential leakage), and it was the lowest-coverage package in the module. Removing it also removes a security surface that only existed because the CLI compiled untrusted schema names into Go source the caller then built. - Deleted clients/go/cmd/ entirely (main.go, main_test.go). - /sdk/reference's "Codegen CLI" section becomes "Row types". The TypeScript tab keeps the CLI; the Go tab documents hand-written row structs and keeps the ClickHouse-to-Go field table, which is still load-bearing without codegen and has no other home. It gains the encoding/json map-key constraint learned in 607d000: `map[K]V` is only usable where K is a string or integer type. - Anchor moved to #row-types; both inbound links updated. The TS setup page still points at the codegen CLI, which still exists. - Swept the "codegen" claim out of the language-neutral surfaces that asserted it for both SDKs: /sdk, the docs landing page, README, why-wavehouse's comparison table, getting-started, AGENTS.md §SDK Sync, and the CHANGELOG entry. TypeScript's own codegen docs are untouched. go-sdk coverage 87.4% -> 88.5%: the codegen CLI was the package holding the module's number down. Branch diff 11,667 -> 10,580 lines. --- .testcoverage.yml | 8 +- AGENTS.md | 8 +- CHANGELOG.md | 2 +- README.md | 2 +- clients/go/README.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 444 ------------- clients/go/cmd/wavehouse-codegen/main_test.go | 613 ------------------ clients/go/query_builder_test.go | 2 +- clients/go/stream.go | 2 +- clients/go/stream_test.go | 2 +- docs/src/content/docs/getting-started.md | 2 +- docs/src/content/docs/index.mdx | 4 +- docs/src/content/docs/sdk/index.mdx | 6 +- docs/src/content/docs/sdk/queries.mdx | 2 +- docs/src/content/docs/sdk/reference.mdx | 52 +- docs/src/content/docs/sdk/setup/go.md | 4 +- .../src/content/docs/sdk/setup/typescript.mdx | 2 +- docs/src/content/docs/why-wavehouse.md | 4 +- 18 files changed, 37 insertions(+), 1124 deletions(-) delete mode 100644 clients/go/cmd/wavehouse-codegen/main.go delete mode 100644 clients/go/cmd/wavehouse-codegen/main_test.go diff --git a/.testcoverage.yml b/.testcoverage.yml index 309a5d77..cc5fa11b 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -34,10 +34,10 @@ suites: # Go SDK (clients/go) — a NESTED module, so the root module's # `-coverpkg=./...` can never reach it: rendered and gated on its own, # never merged into the total above, nothing to add under exclude.paths. - # 75 vs 87.1% measured (SDK 88.5%, codegen CLI 81.0%). The gap is deliberate - # slack, matching how every other floor here is set (ts-unit is 40 against a - # measured 76%): these are regression floors, not targets, so ordinary churn - # doesn't red the build. Raise it when a real regression slips under 75. + # 75 vs 88.5% measured. The gap is deliberate slack, matching how every other + # floor here is set (ts-unit is 40 against a measured 76%): these are + # regression floors, not targets, so ordinary churn doesn't red the build. + # Raise it when a real regression slips under 75. # # ONE key here, three for the TS SDK below, because the TS SDK has two # coverage-PRODUCING suites and this has one. `ts-unit` is vitest, `ts-e2e` diff --git a/AGENTS.md b/AGENTS.md index 5b9d735b..0596a01d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/ops` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/ops` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Row visibility has the same one-source guarantee (#319): `Evaluate` resolves a role's row-`filter` once (`resolvePredicates`), and both surfaces consume that single resolution — the query path renders it to SQL (`predicatesToSQL`), the stream evaluates it in memory per subscriber (`ResolvedPermissions.RowVisible`, whose type-aware comparison fails closed on anything it can't prove about the ingested payload — `policy.ColumnSpec`, with `DateTime`/`DateTime64` operands compared as instants through the ingest grammar (`discovery.Column.TimeParser`) and claim constants rendered canonically and digit-exact by the one shared rule `policy.CanonicalScalar` (#457 — which also refuses a float64 at/past 2^53 rather than match a neighboring ID, and whose ok=false — an absent claim, a structured value, no canonical form — makes the predicate match no rows on BOTH surfaces: `1 = 0` in SQL, every row withheld in memory); numeric comparison runs in the column's STORAGE domain (`policy.NumericSpec`, classified by `discovery.NumericStorageOf` — Float width rounding, Decimal scale truncation, integer exactness, both operands narrowed as ClickHouse narrows stored value and bound constant, out-of-range operands refused rather than modeled; the `tests/integration` differential oracle holds in-range verdicts equal to a live ClickHouse's and the never-admit-where-SQL-hides direction for the refused out-of-range ones); an event whose insert later fails into the DLQ is the one residual payload-vs-stored asymmetry, documented in the access-control enforcement caution) — so row visibility can't drift either. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`, package `wavehouse`, in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Each ships a typed query builder, real-time SSE over header-authenticated HTTP, live queries (incrementable/decomposable/poll aggregation), and a codegen CLI. TypeScript has exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got. Go has zero third-party runtime dependencies (stdlib-only, hand-rolled SSE framing). See §SDK Sync. +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`, package `wavehouse`, in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Each ships a typed query builder, real-time SSE over header-authenticated HTTP, live queries (incrementable/decomposable/poll aggregation). TypeScript adds a codegen CLI; Go row structs are hand-written. TypeScript has exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got. Go has zero third-party runtime dependencies (stdlib-only, hand-rolled SSE framing). See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -366,12 +366,12 @@ The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`github.com/W | -------------- | ------------------ | | New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update the shared topic page under `docs/src/content/docs/sdk/` — add or extend the `` block so BOTH languages are covered on the same page (never a per-language page tree; see §SDK docs layout). Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | | Change to JWT auth / role extraction | TS: `clients/ts/src/http.ts` + `client.ts`. Go: `clients/go/http.go` + `wavehouse.go`. | -| Change to `EventMessage` / ingest event format | Update payload types in both SDKs (some are codegen-regenerated — re-run both codegen CLIs). | +| Change to `EventMessage` / ingest event format | Update payload types in both SDKs (re-run the TS codegen CLI where types are generated; Go row structs are hand-written). | | New / changed structured query AST | TS: `clients/ts/src/query-builder.ts`. Go: `clients/go/query_builder.go` + `types.go`. | | Change to live-query aggregation classification | TS: `clients/ts/src/stream/`. Go: `clients/go/live_query.go`. | | Named pipes API change | TS: `clients/ts/src/pipes.ts`. Go: `clients/go/pipes.go`. | | Policy / access-control change | TS: `clients/ts/src/policy.ts`. Go: `clients/go/policy.go`. | -| ClickHouse schema-driven type changes | Re-run both SDK codegen CLIs; commit regenerated types. | +| ClickHouse schema-driven type changes | Re-run the TS SDK codegen CLI and commit regenerated types; Go row structs are hand-written by the consumer. | Internal-only backend changes (middleware refactors, observability internals, dedup implementation, sweeper logic, NATS plumbing) generally don't need SDK updates. Use judgement — table above is the source of truth; nothing automated nudges you. @@ -427,7 +427,7 @@ The SDK docs are **topic-first, not language-first** — the decision from PR #3 ```text cmd/ → Binary entry points (thin — just wiring) clients/ts/ → TypeScript SDK (@wavehouse/sdk) -clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) — nested module; file layout mirrors clients/ts/, plus cmd/wavehouse-codegen/ and testdata/wire_cases.json +clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) — nested module; file layout mirrors clients/ts/, plus testdata/wire_cases.json internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/policy/pipes endpoints) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) internal/cache/ → Caching (interface + L1/L2/tiered implementations) diff --git a/CHANGELOG.md b/CHANGELOG.md index dee0513b..54457952 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); the Go suites use gotestsum and honor `ARGS`/`V=1` like every other Go suite, while `test-sdk-ts` runs vitest. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language-specific section, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page under `/sdk/setup/` — `/sdk/setup/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/setup/go`, indexed by `/sdk/setup`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`). Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); the Go suites use gotestsum and honor `ARGS`/`V=1` like every other Go suite, while `test-sdk-ts` runs vitest. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language-specific section, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page under `/sdk/setup/` — `/sdk/setup/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/setup/go`, indexed by `/sdk/setup`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. diff --git a/README.md b/README.md index 45ba081b..a54bab37 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): query builder, live queries, streaming, and schema codegen in both, over one shared wire format. +- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): query builder, live queries, streaming, and typed rows, over one shared wire format. ## How it compares diff --git a/clients/go/README.md b/clients/go/README.md index 52a12f5f..f1b67448 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -52,7 +52,7 @@ func main() { - [Streaming & Live Queries](https://wavehouse.dev/sdk/streaming) — SSE streams, client-side filtering, backfill-then-live. - [Pipes](https://wavehouse.dev/sdk/pipes) — execute and manage named query pipes. - [Admin & System](https://wavehouse.dev/sdk/admin) — schema, policy, DLQ stats, health. -- [Reference & CLI](https://wavehouse.dev/sdk/reference) — error codes, the full API tree, and the `wavehouse-codegen` struct generator. +- [Reference & CLI](https://wavehouse.dev/sdk/reference) — error codes, the full API tree, and the ClickHouse-to-Go field mapping for row structs. ## License diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go deleted file mode 100644 index 0ab2e28a..00000000 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ /dev/null @@ -1,444 +0,0 @@ -// Command wavehouse-codegen reads a WaveHouse server's /v1/ops/schema endpoint -// and generates Go struct definitions for use with the wavehouse SDK. -// -// Usage: -// -// WAVEHOUSE_AUTH= wavehouse-codegen --url http://localhost:8080 --out ./db.go -package main - -import ( - "context" - "encoding/json" - "errors" - "fmt" - "go/format" - "go/token" - "net" - "net/http" - "net/url" - "os" - "slices" - "strings" - "time" - "unicode" -) - -type cliArgs struct { - url string - out string - auth string - pkg string -} - -// flagValue consumes and returns the value following argv[*i], exiting -// rather than silently falling back to the default when it is missing. -func flagValue(argv []string, i *int) string { - flag := argv[*i] - *i++ - if *i >= len(argv) { - fmt.Fprintf(os.Stderr, "Error: missing value for %s (use --help)\n", flag) - os.Exit(2) - } - return argv[*i] -} - -// parseArgs takes argv (including argv[0]) and the WAVEHOUSE_AUTH value rather -// than reading os.Args / the environment itself, so tests drive it with their -// own fixtures instead of mutating process-global state. -func parseArgs(argv []string, envAuth string) cliArgs { - args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} - for i := 1; i < len(argv); i++ { - switch argv[i] { - case "--url", "-u": - args.url = flagValue(argv, &i) - case "--out", "-o": - args.out = flagValue(argv, &i) - case "--auth", "-a": - args.auth = flagValue(argv, &i) - case "--package", "-p": - args.pkg = flagValue(argv, &i) - case "--help", "-h": - fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema - -Options: - --url, -u WaveHouse base URL (default: http://localhost:8080) - --out, -o Output .go file path (default: ./wavehouse_types.go) - --auth, -a Bearer token for authenticated /v1/ops/schema endpoint - (prefer the WAVEHOUSE_AUTH env var — argv leaks into - shell history and process listings) - --package, -p Go package name (default: main) - --help, -h Show this help`) - os.Exit(0) - default: - fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", argv[i]) - os.Exit(2) - } - } - if args.auth == "" { - args.auth = envAuth - } - return args -} - -type column struct { - Name string `json:"name"` - Type string `json:"type"` - HasDefault bool `json:"has_default"` -} - -type tableSchema struct { - Name string `json:"name"` - Columns []column `json:"columns"` -} - -// requireSecureAuthURL refuses to put a bearer token on the wire in cleartext. -// Loopback is exempt: it is the default target and never leaves the machine. -func requireSecureAuthURL(u *url.URL) error { - if u.Scheme == "https" || isLoopback(u.Hostname()) { - return nil - } - return fmt.Errorf("refusing to send credentials to %s over %s: use an https:// URL, or drop --auth/WAVEHOUSE_AUTH", u.Host, u.Scheme) -} - -func isLoopback(host string) bool { - if host == "localhost" { - return true - } - ip := net.ParseIP(host) - return ip != nil && ip.IsLoopback() -} - -// refuseCredentialedRedirect stops a redirect from carrying the bearer token -// somewhere it wasn't meant for. net/http drops Authorization only when the -// host changes, ignoring scheme and port, so its default policy would hand the -// token to a plaintext hop on the same host. The SDK takes the same stance on -// streams (SSE_REDIRECT). -func refuseCredentialedRedirect(req *http.Request, via []*http.Request) error { - if len(via) >= 10 { - return errors.New("stopped after 10 redirects") - } - prev := via[len(via)-1].URL - if req.URL.Scheme != prev.Scheme || req.URL.Host != prev.Host { - return fmt.Errorf("refusing redirect from %s://%s to %s://%s while sending credentials", - prev.Scheme, prev.Host, req.URL.Scheme, req.URL.Host) - } - return nil -} - -func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { - url := strings.TrimRight(baseURL, "/") + "/v1/ops/schema" - req, err := http.NewRequestWithContext(ctx, "GET", url, nil) - if err != nil { - return nil, fmt.Errorf("build schema request for %s: %w", url, err) - } - client := &http.Client{Timeout: 30 * time.Second} - if auth != "" { - if err := requireSecureAuthURL(req.URL); err != nil { - return nil, err - } - req.Header.Set("Authorization", "Bearer "+auth) - client.CheckRedirect = refuseCredentialedRedirect - } - resp, err := client.Do(req) - if err != nil { - return nil, fmt.Errorf("fetch schema from %s: %w", url, err) - } - defer func() { _ = resp.Body.Close() }() - if resp.StatusCode != 200 { - return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) - } - - // The server returns either []tableSchema or map[string]tableSchema. - var raw json.RawMessage - if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { - return nil, fmt.Errorf("read schema response: %w", err) - } - var arr []tableSchema - if err := json.Unmarshal(raw, &arr); err == nil { - m := make(map[string]tableSchema, len(arr)) - for _, t := range arr { - m[t.Name] = t - } - return m, nil - } - var m map[string]tableSchema - if err := json.Unmarshal(raw, &m); err != nil { - return nil, fmt.Errorf("decode schema JSON: %w", err) - } - return m, nil -} - -// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to -// a Go type name suitable for a JSON struct field. The mapping targets what -// round-trips through the server's JSON, not clickhouse-go's native scan -// types, and keeps generated output free of non-stdlib imports. -func chTypeToGo(chType string) string { - // Unwrap Nullable → pointer. - if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { - inner := chType[9 : len(chType)-1] - return "*" + chTypeToGo(inner) - } - // Unwrap LowCardinality. - if strings.HasPrefix(chType, "LowCardinality(") && strings.HasSuffix(chType, ")") { - return chTypeToGo(chType[15 : len(chType)-1]) - } - // Unwrap SimpleAggregateFunction(func, InnerType): the wire value is just - // InnerType, the function name only describes how merges combine rows. - if strings.HasPrefix(chType, "SimpleAggregateFunction(") && strings.HasSuffix(chType, ")") { - inner := chType[len("SimpleAggregateFunction(") : len(chType)-1] - if comma := findTopLevelComma(inner); comma != -1 { - return chTypeToGo(strings.TrimSpace(inner[comma+1:])) - } - return "any" - } - // String-like. - switch { - case chType == "String", - strings.HasPrefix(chType, "FixedString("), - chType == "UUID", - strings.HasPrefix(chType, "DateTime"), - strings.HasPrefix(chType, "Date"), - strings.HasPrefix(chType, "Time"), // Time/Time64 time-of-day types - strings.HasPrefix(chType, "Enum8("), - strings.HasPrefix(chType, "Enum16("), - chType == "IPv4", - chType == "IPv6": - return "string" - case chType == "Bool", chType == "Boolean": - return "bool" - } - // Generated structs target /v1/query and /v1/pipes/*, where the server - // re-marshals values so 64-bit integers arrive as unquoted JSON numbers. - // /v1/ops/query forwards ClickHouse's own JSON, which quotes them — use - // SQL[map[string]any] there. - if mapped, ok := map[string]string{ - "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", - "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", - "Float32": "float32", "Float64": "float64", "BFloat16": "float32", - }[chType]; ok { - return mapped - } - switch { - case strings.HasPrefix(chType, "Decimal"): - return "string" // marshaled as a quoted string on the structured path - case strings.HasPrefix(chType, "UInt128"), - strings.HasPrefix(chType, "UInt256"), - strings.HasPrefix(chType, "Int128"), - strings.HasPrefix(chType, "Int256"): - // Arbitrary-width unquoted JSON numbers; int64/uint64 would overflow. - return "json.Number" - } - if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { - inner := chTypeToGo(chType[6 : len(chType)-1]) - // Array(UInt8) is asymmetric on the wire — ingest takes a JSON array, - // /v1/query returns base64 (#436) — and only json.RawMessage - // round-trips both without a decode error. - if inner == "uint8" { - return "json.RawMessage" - } - return "[]" + inner - } - if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { - inner := chType[4 : len(chType)-1] - comma := findTopLevelComma(inner) - if comma != -1 { - k := chTypeToGo(strings.TrimSpace(inner[:comma])) - v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) - // Go comparability is not enough: `map[[]T]V` parses and fails to - // compile, and `map[float64]V` compiles and then fails at - // Marshal/Unmarshal, which is worse. format.Source only parses, so - // neither is caught downstream. Fall back instead. - if !jsonMapKey(k) { - return "any" - } - return "map[" + k + "]" + v - } - return "map[string]any" - } - return "any" -} - -// jsonMapKey reports whether t is usable as a map key that encoding/json can -// round-trip. It supports string-kinded and integer-kinded keys plus -// encoding.TextMarshaler implementors; chTypeToGo returns none of the latter, -// so this is the complete allow-list of its outputs. Everything else it can -// return — bool, the floats, any, *T, []T, json.RawMessage, nested maps — is -// either non-comparable or silently unmarshalable. -func jsonMapKey(t string) bool { - switch t { - case "string", "json.Number", - "int8", "int16", "int32", "int64", - "uint8", "uint16", "uint32", "uint64": - return true - } - return false -} - -func findTopLevelComma(s string) int { - depth := 0 - for i := range len(s) { - switch s[i] { - case '(': - depth++ - case ')': - depth-- - case ',': - if depth == 0 { - return i - } - } - } - return -1 -} - -func pascalCase(s string) string { - parts := strings.FieldsFunc(s, func(r rune) bool { - return r == '_' || r == '-' || r == ' ' || r == '.' - }) - var sb strings.Builder - for _, p := range parts { - if len(p) == 0 { - continue - } - runes := []rune(p) - runes[0] = unicode.ToUpper(runes[0]) - sb.WriteString(string(runes)) - } - result := sb.String() - if result == "" { - return result - } - // Go identifiers can't start with a digit, so "2fa_events" needs a prefix - // to stay a valid exported name. - if unicode.IsDigit(rune(result[0])) { // digits are ASCII; no rune-slice needed - result = "X" + result - } - return result -} - -func sortedKeys(m map[string]tableSchema) []string { - names := make([]string, 0, len(m)) - for name := range m { - names = append(names, name) - } - slices.Sort(names) - return names -} - -// validGoIdent reports whether s is a legal Go identifier. Schema names reach -// the generated source as identifiers via pascalCase, which only splits on -// space/_/-/. — tabs, newlines, braces and backticks survive it verbatim. Since -// identifiers are written unquoted and format.Source only *parses*, a crafted -// column name can otherwise close the struct and append arbitrary top-level -// declarations to a file the caller then compiles. -func validGoIdent(s string) bool { - // token.IsIdentifier rejects the empty string and keywords; "_" satisfies - // it but is not a usable package or field name. - return s != "_" && token.IsIdentifier(s) -} - -func generate(schemas map[string]tableSchema, pkg string) (string, error) { - var sb strings.Builder - - if !validGoIdent(pkg) { - return "", fmt.Errorf("--package %q is not a valid Go package name", pkg) - } - - names := sortedKeys(schemas) - - // json.Number fields (128/256-bit integer columns) need the import. - needsJSON := false - for _, name := range names { - for _, col := range schemas[name].Columns { - if strings.Contains(chTypeToGo(col.Type), "json.") { - needsJSON = true - } - } - } - fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) - if needsJSON { - sb.WriteString("import \"encoding/json\"\n\n") - } - - // pascalCase is neither injective nor escaping, and format.Source only - // parses: an unchecked name is a non-compiling file at best and injected - // source at worst. - seenTypes := make(map[string]string, len(names)) - for _, name := range names { - schema := schemas[name] - typeName := pascalCase(name) + "Row" - if !validGoIdent(typeName) { - return "", fmt.Errorf("table %q does not map to a usable Go type name (got %q)", name, typeName) - } - if prev, dup := seenTypes[typeName]; dup { - return "", fmt.Errorf("tables %q and %q both map to type %q; rename one or generate separately", prev, name, typeName) - } - seenTypes[typeName] = name - fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) - seenFields := make(map[string]string, len(schema.Columns)) - for _, col := range schema.Columns { - goType := chTypeToGo(col.Type) - fieldName := pascalCase(col.Name) - if !validGoIdent(fieldName) { - return "", fmt.Errorf("table %q: column %q does not map to a usable Go field name (got %q)", name, col.Name, fieldName) - } - if prev, dup := seenFields[fieldName]; dup { - return "", fmt.Errorf("table %q: columns %q and %q both map to field %q", name, prev, col.Name, fieldName) - } - seenFields[fieldName] = col.Name - jsonTag := col.Name - if col.HasDefault { - // Pointer + omitempty means nil omits the field so the server - // default applies, while a pointer to the zero value still - // sends an explicit 0/false/"". - jsonTag += ",omitempty" - if !strings.HasPrefix(goType, "*") { - goType = "*" + goType - } - } - fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) - } - sb.WriteString("}\n\n") - } - - return sb.String(), nil -} - -func main() { - args := parseArgs(os.Args, os.Getenv("WAVEHOUSE_AUTH")) - fmt.Printf("Fetching schema from %s...\n", args.url) - - schemas, err := fetchSchemas(context.Background(), args.url, args.auth) - if err != nil { - fmt.Fprintf(os.Stderr, "Error: %v\n", err) - os.Exit(1) - } - if len(schemas) == 0 { - fmt.Fprintln(os.Stderr, "No tables found. Is WaveHouse running with tables in ClickHouse?") - os.Exit(1) - } - - names := sortedKeys(schemas) - fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) - - output, err := generate(schemas, args.pkg) - if err != nil { - fmt.Fprintf(os.Stderr, "Error: %v\n", err) - os.Exit(1) - } - - // A format failure means the generated source is not valid Go; don't write - // unusable output and claim success. - formatted, err := format.Source([]byte(output)) - if err != nil { - fmt.Fprintf(os.Stderr, "Error: generated code is not valid Go: %v\n", err) - os.Exit(1) - } - - if err := os.WriteFile(args.out, formatted, 0o600); err != nil { - fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) - os.Exit(1) - } - - fmt.Printf("✓ Types written to %s\n", args.out) -} diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go deleted file mode 100644 index afae26ed..00000000 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ /dev/null @@ -1,613 +0,0 @@ -package main - -import ( - "context" - "encoding/json" - "errors" - "go/format" - "io" - "net/http" - "net/http/httptest" - "net/url" - "os" - "os/exec" - "reflect" - "strings" - "testing" -) - -func TestChTypeToGo(t *testing.T) { - tests := []struct { - ch string - want string - }{ - {"String", "string"}, - {"FixedString(16)", "string"}, - {"UUID", "string"}, - {"DateTime64(3, 'UTC')", "string"}, - {"Date", "string"}, - {"Time64(3)", "string"}, - {"Enum8('a' = 1)", "string"}, - {"IPv4", "string"}, - {"Bool", "bool"}, - {"Boolean", "bool"}, - {"UInt8", "uint8"}, - {"UInt16", "uint16"}, - {"UInt32", "uint32"}, - {"Int8", "int8"}, - {"Int32", "int32"}, - {"Float32", "float32"}, - {"BFloat16", "float32"}, - {"Float64", "float64"}, - // /v1/query re-marshals server-side: 64-bit ints arrive unquoted. - {"UInt64", "uint64"}, - {"Int64", "int64"}, - {"UInt128", "json.Number"}, - {"Int256", "json.Number"}, - {"Decimal(18, 4)", "string"}, - {"Nullable(Int32)", "*int32"}, - {"Nullable(Int64)", "*int64"}, - {"LowCardinality(String)", "string"}, - {"LowCardinality(Nullable(String))", "*string"}, - {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, - {"SimpleAggregateFunction(any)", "any"}, - {"Array(String)", "[]string"}, - {"Array(Nullable(Int32))", "[]*int32"}, - // []uint8 is []byte → base64 on marshal; RawMessage round-trips both - // the ingest array form and the (currently base64) query response. - {"Array(UInt8)", "json.RawMessage"}, - {"Map(String, UInt32)", "map[string]uint32"}, - {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, - {"Tuple(String, UInt8)", "any"}, - {"SomethingNew", "any"}, - } - for _, tt := range tests { - if got := chTypeToGo(tt.ch); got != tt.want { - t.Errorf("chTypeToGo(%q) = %q, want %q", tt.ch, got, tt.want) - } - } -} - -func TestPascalCase(t *testing.T) { - tests := []struct { - in string - want string - }{ - {"clicks", "Clicks"}, - {"user_id", "UserId"}, - {"received_timestamp", "ReceivedTimestamp"}, - {"multi-part.name here", "MultiPartNameHere"}, - {"2fa_events", "X2faEvents"}, // leading digit gets the X prefix - {"", ""}, - } - for _, tt := range tests { - if got := pascalCase(tt.in); got != tt.want { - t.Errorf("pascalCase(%q) = %q, want %q", tt.in, got, tt.want) - } - } -} - -func TestFindTopLevelComma(t *testing.T) { - tests := []struct { - in string - want int - }{ - {"String, UInt32", 6}, - {"Map(String, String), UInt8", 19}, - {"NoComma", -1}, - } - for _, tt := range tests { - if got := findTopLevelComma(tt.in); got != tt.want { - t.Errorf("findTopLevelComma(%q) = %d, want %d", tt.in, got, tt.want) - } - } -} - -func TestGenerate_Basic(t *testing.T) { - out, err := generate(map[string]tableSchema{ - "clicks": {Name: "clicks", Columns: []column{ - {Name: "page", Type: "String"}, - {Name: "score", Type: "Float64"}, - {Name: "big", Type: "UInt128"}, - {Name: "received_timestamp", Type: "DateTime64(3, 'UTC')", HasDefault: true}, - }}, - }, "myapp") - if err != nil { - t.Fatal(err) - } - for _, want := range []string{ - "package myapp", - `import "encoding/json"`, // dragged in by the json.Number field below - "type ClicksRow struct {", - "Page string `json:\"page\"`", - "Score float64 `json:\"score\"`", - "Big json.Number `json:\"big\"`", - // Defaulted column: pointer + omitempty so an explicit zero still sends. - "ReceivedTimestamp *string `json:\"received_timestamp,omitempty\"`", - } { - if !strings.Contains(out, want) { - t.Errorf("generated output missing %q:\n%s", want, out) - } - } - if _, err := format.Source([]byte(out)); err != nil { - t.Fatalf("generated output is not valid Go: %v", err) - } -} - -func TestGenerate_FieldCollisionFails(t *testing.T) { - _, err := generate(map[string]tableSchema{ - "t": {Name: "t", Columns: []column{ - {Name: "user_id", Type: "String"}, - {Name: "userId", Type: "String"}, - }}, - }, "main") - if err == nil || !strings.Contains(err.Error(), "UserId") { - t.Fatalf("want field-collision error naming UserId, got %v", err) - } -} - -func TestGenerate_TypeCollisionFails(t *testing.T) { - _, err := generate(map[string]tableSchema{ - "2fa": {Name: "2fa", Columns: []column{{Name: "a", Type: "String"}}}, - "x2fa": {Name: "x2fa", Columns: []column{{Name: "a", Type: "String"}}}, - }, "main") - if err == nil || !strings.Contains(err.Error(), "X2faRow") { - t.Fatalf("want type-collision error naming X2faRow, got %v", err) - } -} - -// TestGeneratedShapeDecodesStructuredQueryPayload asserts the mapping choices -// actually decode what /v1/query emits: the server scans ClickHouse values -// into Go types and re-marshals, so 64-bit ints are unquoted numbers, -// 128/256-bit ints are unquoted arbitrary-width numbers, and Decimals are -// quoted strings. -func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { - type row struct { - ID uint64 `json:"id"` - Delta int64 `json:"delta"` - Big json.Number `json:"big"` - Price string `json:"price"` - } - payload := `[{"id":18446744073709551615,"delta":-9007199254740993,"big":170141183460469231731687303715884105727,"price":"12.3400"}]` - var rows []row - if err := json.Unmarshal([]byte(payload), &rows); err != nil { - t.Fatalf("generated shape failed to decode /v1/query payload: %v", err) - } - if rows[0].ID != 18446744073709551615 || rows[0].Delta != -9007199254740993 { - t.Fatalf("64-bit values corrupted: %+v", rows[0]) - } - if rows[0].Big.String() != "170141183460469231731687303715884105727" { - t.Fatalf("128-bit value corrupted: %s", rows[0].Big) - } -} - -func TestFlagValue(t *testing.T) { - tests := []struct { - name string - args []string - want string - wantI int - }{ - {name: "value follows the flag", args: []string{"cg", "--url", "http://h:9000"}, want: "http://h:9000", wantI: 2}, - // There is no lookahead: a flag-shaped value is consumed as the value. - {name: "flag-shaped value", args: []string{"cg", "--out", "--package"}, want: "--package", wantI: 2}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - i := 1 - if got := flagValue(tt.args, &i); got != tt.want { - t.Errorf("flagValue() = %q, want %q", got, tt.want) - } - if i != tt.wantI { - t.Errorf("index advanced to %d, want %d", i, tt.wantI) - } - }) - } -} - -func TestParseArgs(t *testing.T) { - // Every case differs from the defaults parseArgs starts with by a field or - // two, so deriving the wants keeps each row about what it changes. - defaults := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} - allFlags := cliArgs{url: "http://h:9000", out: "./types.go", auth: "argv-token", pkg: "db"} - secondURL, envAuth, argvAuth := defaults, defaults, defaults - secondURL.url = "http://second" - envAuth.auth = "env-token" - argvAuth.auth = "argv-token" - tests := []struct { - name string - args []string - env string // WAVEHOUSE_AUTH - want cliArgs - }{ - {name: "no arguments uses defaults", args: []string{"cg"}, want: defaults}, - { - name: "long flags", - args: []string{"cg", "--url", "http://h:9000", "--out", "./types.go", "--auth", "argv-token", "--package", "db"}, - want: allFlags, - }, - { - name: "short flags", - args: []string{"cg", "-u", "http://h:9000", "-o", "./types.go", "-a", "argv-token", "-p", "db"}, - want: allFlags, - }, - {name: "later flag wins", args: []string{"cg", "-u", "http://first", "--url", "http://second"}, want: secondURL}, - {name: "WAVEHOUSE_AUTH fills an unset --auth", args: []string{"cg"}, env: "env-token", want: envAuth}, - {name: "--auth beats WAVEHOUSE_AUTH", args: []string{"cg", "--auth", "argv-token"}, env: "env-token", want: argvAuth}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - if got := parseArgs(tt.args, tt.env); got != tt.want { - t.Errorf("parseArgs() = %+v, want %+v", got, tt.want) - } - }) - } -} - -// parseArgsChildEnv carries the argv under test to the re-executed child. -const parseArgsChildEnv = "WAVEHOUSE_CODEGEN_TEST_ARGS" - -// TestParseArgsExitPaths covers the branches that end in os.Exit, which can -// only be observed from outside the process: each case re-runs this test -// binary with the argv under test and asserts on the child's exit code and -// output. -func TestParseArgsExitPaths(t *testing.T) { - if raw, ok := os.LookupEnv(parseArgsChildEnv); ok { - parseArgs(append([]string{"wavehouse-codegen"}, strings.Fields(raw)...), "") - t.Fatal("parseArgs returned instead of exiting") - } - tests := []struct { - name string - args string - wantCode int - wantOut string - }{ - {name: "missing value for a long flag", args: "--url", wantCode: 2, wantOut: "missing value for --url"}, - {name: "missing value for a short flag", args: "-o", wantCode: 2, wantOut: "missing value for -o"}, - {name: "missing value after a satisfied flag", args: "--url http://h:9000 --package", wantCode: 2, wantOut: "missing value for --package"}, - {name: "unknown argument", args: "--nope", wantCode: 2, wantOut: `unknown argument "--nope"`}, - {name: "bare value with no flag", args: "stray", wantCode: 2, wantOut: `unknown argument "stray"`}, - {name: "help exits cleanly", args: "--help", wantCode: 0, wantOut: "Generate Go types from WaveHouse schema"}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - // Re-executing this test binary is the standard way to observe - // an os.Exit path; the argv is fixed by the table above. - cmd := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestParseArgsExitPaths$") //nolint:gosec // the command is this test binary, not user input - cmd.Env = append(os.Environ(), parseArgsChildEnv+"="+tt.args) - out, err := cmd.CombinedOutput() - code := 0 - var exitErr *exec.ExitError - switch { - case errors.As(err, &exitErr): - code = exitErr.ExitCode() - case err != nil: - t.Fatalf("run child: %v", err) - } - if code != tt.wantCode { - t.Errorf("child exit code = %d, want %d\n%s", code, tt.wantCode, out) - } - if !strings.Contains(string(out), tt.wantOut) { - t.Errorf("child output missing %q:\n%s", tt.wantOut, out) - } - }) - } -} - -// schemaRequest is what the stub server saw. It travels over a buffered -// channel rather than a shared variable so -race sees the edge between the -// server goroutine and the test. -type schemaRequest struct { - path string - auth string -} - -// schemaServer answers every request with body under status (0 means 200). -func schemaServer(t *testing.T, status int, body string) (string, <-chan schemaRequest) { - t.Helper() - seen := make(chan schemaRequest, 1) - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - seen <- schemaRequest{path: r.URL.Path, auth: r.Header.Get("Authorization")} - if status != 0 { - w.WriteHeader(status) - } - _, _ = io.WriteString(w, body) - })) - t.Cleanup(srv.Close) - return srv.URL, seen -} - -func TestFetchSchemas(t *testing.T) { - const clicksJSON = `{"name":"clicks","columns":[{"name":"page","type":"String"},{"name":"ts","type":"DateTime","has_default":true}]}` - clicks := tableSchema{Name: "clicks", Columns: []column{ - {Name: "page", Type: "String"}, - {Name: "ts", Type: "DateTime", HasDefault: true}, - }} - tests := []struct { - name string - suffix string // appended to the stub server's base URL - auth string - status int - body string - want map[string]tableSchema - wantAuth string - wantErr string - }{ - { - name: "array response with bearer auth", auth: "tok-123", body: "[" + clicksJSON + "]", - want: map[string]tableSchema{"clicks": clicks}, wantAuth: "Bearer tok-123", - }, - { - name: "map response without auth", body: `{"clicks":` + clicksJSON + `}`, - want: map[string]tableSchema{"clicks": clicks}, - }, - {name: "trailing slash trimmed from base URL", suffix: "/", body: "[]", want: map[string]tableSchema{}}, - {name: "malformed JSON", body: `[{"name":`, wantErr: "read schema response"}, - {name: "JSON that is neither array nor map", body: `"nope"`, wantErr: "decode schema JSON"}, - {name: "non-200 status", status: http.StatusInternalServerError, body: "boom", wantErr: "schema fetch failed: HTTP 500"}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - base, seen := schemaServer(t, tt.status, tt.body) - got, err := fetchSchemas(t.Context(), base+tt.suffix, tt.auth) - switch { - case tt.wantErr != "": - if err == nil || !strings.Contains(err.Error(), tt.wantErr) { - t.Fatalf("fetchSchemas() error = %v, want one containing %q", err, tt.wantErr) - } - case err != nil: - t.Fatalf("fetchSchemas() error = %v", err) - case !reflect.DeepEqual(got, tt.want): - t.Errorf("fetchSchemas() = %+v, want %+v", got, tt.want) - } - select { - case req := <-seen: - if req.path != "/v1/ops/schema" { - t.Errorf("requested path = %q, want /v1/ops/schema", req.path) - } - if req.auth != tt.wantAuth { - t.Errorf("Authorization header = %q, want %q", req.auth, tt.wantAuth) - } - default: - t.Error("stub server never saw a request") - } - }) - } -} - -func TestFetchSchemasRequestErrors(t *testing.T) { - canceled, cancel := context.WithCancel(t.Context()) - cancel() - tests := []struct { - name string - ctx context.Context - baseURL string - wantErr string - }{ - {name: "unparseable base URL", ctx: t.Context(), baseURL: "http://%zz", wantErr: "build schema request"}, - {name: "transport failure", ctx: canceled, baseURL: "http://127.0.0.1:1", wantErr: "fetch schema from"}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - if _, err := fetchSchemas(tt.ctx, tt.baseURL, ""); err == nil || !strings.Contains(err.Error(), tt.wantErr) { - t.Fatalf("fetchSchemas() error = %v, want one containing %q", err, tt.wantErr) - } - }) - } -} - -func TestSortedKeys(t *testing.T) { - got := sortedKeys(map[string]tableSchema{"clicks": {}, "acks": {}, "views": {}}) - if want := []string{"acks", "clicks", "views"}; !reflect.DeepEqual(got, want) { - t.Errorf("sortedKeys() = %v, want %v", got, want) - } -} - -// injectedColumnName is the payload that reached valid Go before identifiers -// were validated: pascalCase splits on space/_/-/. only, so tabs and newlines -// pass through and keywords inside a part keep their lowercase. It closed the -// struct and appended a function that format.Source accepted without error. -const injectedColumnName = "x string\n}\n\nfunc\tPwn()\tstring\t{\n\treturn\t\"owned\"\n}\n\ntype\tT2Row\tstruct\t{\n\tY" - -func TestGenerate_RejectsInjectedNames(t *testing.T) { - tests := []struct { - name string - schemas map[string]tableSchema - wantErr string - }{ - { - name: "column name breaks out of the struct", - schemas: map[string]tableSchema{"t": {Name: "t", Columns: []column{{Name: injectedColumnName, Type: "String"}}}}, - wantErr: "usable Go field name", - }, - { - name: "backtick in a column name would close the struct tag", - schemas: map[string]tableSchema{"t": {Name: "t", Columns: []column{{Name: "a`b", Type: "String"}}}}, - wantErr: "usable Go field name", - }, - { - name: "table name breaks out of the declaration", - schemas: map[string]tableSchema{injectedColumnName: {Name: injectedColumnName, Columns: []column{{Name: "a", Type: "String"}}}}, - wantErr: "usable Go type name", - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - out, err := generate(tt.schemas, "main") - if err == nil { - t.Fatalf("generate() succeeded; output was:\n%s", out) - } - if !strings.Contains(err.Error(), tt.wantErr) { - t.Errorf("error = %v, want it to mention %q", err, tt.wantErr) - } - }) - } -} - -func TestGenerate_RejectsInvalidPackageName(t *testing.T) { - for _, pkg := range []string{ - "main\n\nfunc\tPwn()\t{}", // injection - "type", // a keyword is not a package name - "_", // blank identifier - "", // empty - "2fa", // leading digit - } { - t.Run(pkg, func(t *testing.T) { - if _, err := generate(map[string]tableSchema{}, pkg); err == nil || - !strings.Contains(err.Error(), "valid Go package name") { - t.Fatalf("generate(pkg=%q) error = %v, want an invalid-package error", pkg, err) - } - }) - } -} - -func TestValidGoIdent(t *testing.T) { - tests := map[string]bool{ - "Page": true, "X2fa": true, "_x": true, "A1_b2": true, - "": false, "2fa": false, "A`b": false, "A B": false, "A\tB": false, "A\nB": false, "A{}": false, - // Keywords and the blank identifier are not usable names. - "type": false, "func": false, "_": false, - } - for in, want := range tests { - if got := validGoIdent(in); got != want { - t.Errorf("validGoIdent(%q) = %v, want %v", in, got, want) - } - } -} - -// A generated map key has to survive two hurdles: Go comparability (or the -// file won't compile) and encoding/json support (or it compiles and then fails -// at Marshal/Unmarshal). format.Source only parses, so neither is caught -// downstream — chTypeToGo falls back to `any` rather than emit either. -func TestChTypeToGo_UnsupportedMapKey(t *testing.T) { - tests := map[string]string{ - // Not comparable — would not compile. - "Map(Array(Map(Float64, String)), String)": "any", - "Map(Array(UInt8), String)": "any", - "Map(Map(String, String), String)": "any", - // Comparable, but encoding/json cannot use them as keys. - "Map(Float64, String)": "any", - "Map(Float32, String)": "any", - "Map(Bool, String)": "any", - "Map(Nullable(String), String)": "any", - // Supported keys are unaffected. - "Map(String, String)": "map[string]string", - "Map(LowCardinality(String), Int8)": "map[string]int8", - "Map(UInt64, String)": "map[uint64]string", - "Map(UInt128, String)": "map[json.Number]string", - "Map(DateTime64(3, 'UTC'), Int32)": "map[string]int32", - } - for in, want := range tests { - if got := chTypeToGo(in); got != want { - t.Errorf("chTypeToGo(%q) = %q, want %q", in, got, want) - } - } -} - -// TestJSONMapKeyMatchesEncodingJSON pins the allow-list to encoding/json's -// actual behavior rather than a reading of its docs: every type jsonMapKey -// admits must round-trip as a map key, and the ones it rejects must not. -func TestJSONMapKeyMatchesEncodingJSON(t *testing.T) { - // Non-comparable candidates ([]T, json.RawMessage, nested maps) can't be - // written as a map key at all, so they never reach encoding/json. - probes := map[string]any{ - "string": map[string]string{"k": "v"}, - "json.Number": map[json.Number]string{"1": "v"}, - "int64": map[int64]string{1: "v"}, - "uint8": map[uint8]string{1: "v"}, - "bool": map[bool]string{true: "v"}, - "float64": map[float64]string{1.5: "v"}, - "any": map[any]string{"k": "v"}, - } - for typ, v := range probes { - t.Run(typ, func(t *testing.T) { - _, err := json.Marshal(v) - if want := jsonMapKey(typ); (err == nil) != want { - t.Errorf("json.Marshal(map[%s]...) error = %v, but jsonMapKey(%q) = %v", typ, err, typ, want) - } - }) - } -} - -func TestRequireSecureAuthURL(t *testing.T) { - tests := []struct { - raw string - wantErr bool - }{ - {raw: "https://wh.example.com/v1/ops/schema"}, - {raw: "http://localhost:8080/v1/ops/schema"}, - {raw: "http://127.0.0.1:8080/v1/ops/schema"}, - {raw: "http://[::1]:8080/v1/ops/schema"}, - {raw: "http://wh.example.com/v1/ops/schema", wantErr: true}, - {raw: "http://10.0.0.5:8080/v1/ops/schema", wantErr: true}, - } - for _, tt := range tests { - t.Run(tt.raw, func(t *testing.T) { - u, err := url.Parse(tt.raw) - if err != nil { - t.Fatal(err) - } - if err := requireSecureAuthURL(u); (err != nil) != tt.wantErr { - t.Errorf("requireSecureAuthURL(%q) error = %v, wantErr %v", tt.raw, err, tt.wantErr) - } - }) - } -} - -func TestFetchSchemas_RefusesCleartextCredentials(t *testing.T) { - // Fails before any connection is attempted, so the unroutable host is safe. - _, err := fetchSchemas(t.Context(), "http://wh.example.com", "tok-123") - if err == nil || !strings.Contains(err.Error(), "refusing to send credentials") { - t.Fatalf("want cleartext-credential refusal, got %v", err) - } - // Without credentials there is nothing to protect, so the scheme check - // must not fire; this one fails at connect instead. - if _, err := fetchSchemas(t.Context(), "http://wh.invalid", ""); err != nil && - strings.Contains(err.Error(), "refusing to send credentials") { - t.Errorf("unauthenticated request was refused for its scheme: %v", err) - } -} - -func TestFetchSchemas_RedirectsWithCredentials(t *testing.T) { - const body = `{"clicks":{"name":"clicks","columns":[{"name":"page","type":"String"}]}}` - - t.Run("cross-origin redirect is refused", func(t *testing.T) { - // A second server means a different port, so this also covers the - // port change that net/http's own policy ignores. - dst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _, _ = io.WriteString(w, body) - })) - defer dst.Close() - src := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - http.Redirect(w, r, dst.URL+"/v1/ops/schema", http.StatusFound) - })) - defer src.Close() - - if _, err := fetchSchemas(t.Context(), src.URL, "tok-123"); err == nil || - !strings.Contains(err.Error(), "refusing redirect") { - t.Fatalf("want redirect refusal, got %v", err) - } - }) - - t.Run("same-origin redirect is followed", func(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/moved" { - http.Redirect(w, r, "/moved", http.StatusFound) - return - } - if got := r.Header.Get("Authorization"); got != "Bearer tok-123" { - t.Errorf("Authorization after same-origin redirect = %q", got) - } - _, _ = io.WriteString(w, body) - })) - defer srv.Close() - - got, err := fetchSchemas(t.Context(), srv.URL, "tok-123") - if err != nil { - t.Fatalf("fetchSchemas: %v", err) - } - if _, ok := got["clicks"]; !ok { - t.Errorf("schemas = %v, want a clicks entry", got) - } - }) -} diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index a9343ca7..733ba5ba 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -298,7 +298,7 @@ func TestQueryBuilder_Pagination_CursorColumnMissingEndsQuietly(t *testing.T) { // the decoded row, so how the row decoded decides how much precision survives. // Typed rows keep int64 exactly; the untyped path decodes to float64 and loses // everything past 2^53 (the same ceiling the TS SDK's JS numbers have). Use -// FetchTyped or codegen structs past 2^53 — documented in queries.md. +// FetchTyped with an int64 field past 2^53 — documented in queries.mdx. func TestQueryBuilder_PaginationCursorPrecision(t *testing.T) { type idRow struct { ID int64 `json:"id"` diff --git a/clients/go/stream.go b/clients/go/stream.go index 8d75acc2..bb6050dc 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -692,7 +692,7 @@ func toFloat64(v any) (float64, bool) { f, err := n.Float64() return f, err == nil } - // All int/uint widths in two cases (codegen structs use the narrow ones). + // All int/uint widths in two cases (row structs often use the narrow ones). rv := reflect.ValueOf(v) switch { case rv.CanInt(): diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 57e084dc..89fe3f84 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -278,7 +278,7 @@ func TestEvaluateFilter(t *testing.T) { {"Lt", float64(9), "lt", 10, true}, {"LteString", "a", "lte", "b", true}, {"GtIncomparable", "a", "gt", 10, false}, - // Narrow/unsigned codegen-struct fields must compare, not silently drop. + // Narrow/unsigned row-struct fields must compare, not silently drop. {"GtUnsignedOperand", float64(10), "gt", uint32(5), true}, {"InAnySlice", "b", "in", []any{"a", "b"}, true}, {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 1226670d..3339b598 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -105,7 +105,7 @@ The handful of things that most often trip up a first session — each is expect - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. - **[API Reference](/api)** — every endpoint, request/response shape, and error code. -- **[Client SDKs](/sdk)** — official TypeScript and Go clients: query builder, live queries, streaming, and codegen. +- **[Client SDKs](/sdk)** — official TypeScript and Go clients: query builder, live queries, streaming, and typed rows. - **[Configuration](/configuration)** — full YAML + environment variable reference. - **[Deployment](/deployment)** — Docker images, releases, health checks. - **[Development](/development)** — building from source, running tests, hot-reload workflow. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index efb437b2..1275e4fb 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -103,7 +103,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - `@wavehouse/sdk` and `github.com/Wave-RF/WaveHouse/clients/go` — type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. + `@wavehouse/sdk` and `github.com/Wave-RF/WaveHouse/clients/go` — type-safe query builders, live queries, real-time streaming, and typed rows, speaking one shared wire format. @@ -213,7 +213,7 @@ Self-hosting WaveHouse is deliberately boring — one binary, one dependency. Bu /> diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index f51a1026..1baff9db 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -1,11 +1,11 @@ --- title: "Client SDKs" -description: "Official WaveHouse clients for TypeScript and Go: query builder, real-time streaming, codegen." +description: "Official WaveHouse clients for TypeScript and Go: query builder, real-time streaming, typed rows." --- import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components"; -WaveHouse ships two officially supported clients with full API-tree parity: **TypeScript** (`@wavehouse/sdk`) and **Go** (`github.com/Wave-RF/WaveHouse/clients/go`). Both give you a typed query builder, real-time SSE streaming over header-authenticated HTTP, live queries that backfill history before going live, and a codegen CLI that turns your ClickHouse schema into row types. +WaveHouse ships two officially supported clients with full API-tree parity: **TypeScript** (`@wavehouse/sdk`) and **Go** (`github.com/Wave-RF/WaveHouse/clients/go`). Both give you a typed query builder, real-time SSE streaming over header-authenticated HTTP, live queries that backfill history before going live, and typed rows: TypeScript generates them from your ClickHouse schema, Go takes a struct you write. They speak the same wire format, and a shared fixture replayed by both test suites keeps it that way, so the topic pages below cover both languages side by side. Pick a language once and the tabs follow you across every page. @@ -114,7 +114,7 @@ Installation, client construction, auth, and the error model are the only langua /> diff --git a/docs/src/content/docs/sdk/queries.mdx b/docs/src/content/docs/sdk/queries.mdx index 438f66c6..c8384bce 100644 --- a/docs/src/content/docs/sdk/queries.mdx +++ b/docs/src/content/docs/sdk/queries.mdx @@ -602,7 +602,7 @@ for page.HasMore && page.Next != nil { } ``` -On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row; `FetchTyped` with an `int64` field, or codegen structs, keep it exact. +On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row; `FetchTyped` with an `int64` field keeps it exact. diff --git a/docs/src/content/docs/sdk/reference.mdx b/docs/src/content/docs/sdk/reference.mdx index a5c48eb2..63f3d3eb 100644 --- a/docs/src/content/docs/sdk/reference.mdx +++ b/docs/src/content/docs/sdk/reference.mdx @@ -243,14 +243,14 @@ NewClient(Config) → *Client --- -## Codegen CLI +## Row types -Generate typed row definitions from a running WaveHouse instance. Codegen reads the admin-only `/v1/ops/schema` endpoint, so a non-dev server needs an admin token or returns `403`. +Both SDKs decode rows into a type you supply. TypeScript generates that type from a running server; Go asks you to write the struct. -The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`: +The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`. It reads the admin-only `/v1/ops/schema` endpoint, so a non-dev server needs an admin token or returns `403`: ```bash npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts @@ -303,41 +303,12 @@ export interface ClicksRow { -The module ships a `wavehouse-codegen` command under `cmd/`: - -```bash -export WAVEHOUSE_AUTH='' # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ - --url http://localhost:8080 \ - --out ./db_types.go \ - --package myapp - -# Or, from inside a checkout of clients/go/: -go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go -``` - -Prefer `WAVEHOUSE_AUTH` over `--auth ` to keep tokens out of shell history and process listings. - -When a token is supplied, the URL must be `https://` or a loopback host: the CLI refuses to send credentials in cleartext, and refuses any redirect that changes scheme, host, or port rather than let `net/http` carry the token along (it drops `Authorization` only on a host change, ignoring scheme and port). Unauthenticated runs are unrestricted. - -**Options:** - -| Flag | Description | Default | -|------|-------------|---------| -| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | -| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | -| `--package`, `-p` | Go package name for the generated file | `main` | -| `--help`, `-h` | Show usage and exit | none | - -**Example output** (for the [development quick-start](/development#quick-start) `clicks` table): +There is no Go codegen CLI. Write the row struct and tag each field with its column name; `FetchTyped[Row]`, `Fetch[Row]`, and `SQL[Row]` decode into whatever shape you define, and `map[string]any` with the untyped methods covers a dynamic or one-off schema. ```go -// Code generated by wavehouse-codegen. DO NOT EDIT. - package myapp -// ClicksRow represents a row in the "clicks" table. +// ClicksRow is one row of the "clicks" table. type ClicksRow struct { Page string `json:"page"` Button string `json:"button"` @@ -346,9 +317,9 @@ type ClicksRow struct { } ``` -Output is run through `go/format`, and codegen fails loudly if a table or column name would produce invalid Go source. Names become `PascalCase`, with an `X` prefix for a leading digit (`2fa_events` → `X2faEventsRow`); initialisms are not special-cased, so `event_id` becomes `EventId`, not `EventID`. Columns with `has_default: true` become pointer fields with `,omitempty` — as `received_timestamp` does above — where `nil` uses the server default and a pointed-at value is sent, including an explicit `0`/`false`/`""`. +Give a column with a `DEFAULT` a pointer field with `,omitempty`, as `received_timestamp` has above: `nil` omits the field so the server default applies, while a pointer to the zero value still sends an explicit `0`/`false`/`""`. -**ClickHouse → Go type mapping:** +**ClickHouse → Go field types:** | ClickHouse Type | Go Type | |------------------|---------| @@ -362,12 +333,11 @@ Output is run through `go/format`, and codegen fails loudly if a table or column | `Decimal*` | `string` | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` (except `Array(UInt8)` → `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | -| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | -| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | -| anything unrecognized | `any` | +| `Array(T)` | `[]T`, except `Array(UInt8)` → `json.RawMessage` ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Map(K, V)` | `map[K]V`, but only where `K` is a string or integer type: `encoding/json` cannot use `bool`, float, or pointer map keys, so reach for `map[string]any` there | +| `SimpleAggregateFunction(fn, T)` | same as `T` | -Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`; for the raw-SQL path (`/v1/ops/query`), which quotes 64-bit-and-wider integers, use `map[string]any` with `SQL[Row]`. +Go keeps ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. These types target `/v1/query` and `/v1/pipes/*`; for the raw-SQL path (`/v1/ops/query`), which quotes 64-bit-and-wider integers, use `map[string]any` with `SQL[Row]`. diff --git a/docs/src/content/docs/sdk/setup/go.md b/docs/src/content/docs/sdk/setup/go.md index 4597f7aa..9c400ca8 100644 --- a/docs/src/content/docs/sdk/setup/go.md +++ b/docs/src/content/docs/sdk/setup/go.md @@ -132,7 +132,7 @@ page, err := wavehouse.FetchTyped[ClickRow](ctx, // page.Data is []ClickRow ``` -Use the [codegen CLI](/sdk/reference#codegen-cli) to generate row structs from a running server. `FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods; the untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. +Row structs are hand-written; see [Row types](/sdk/reference#row-types) for the ClickHouse-to-Go field mapping. `FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods; the untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. ## Error Handling @@ -173,4 +173,4 @@ The topic pages cover both SDKs, tabbed by language. The tab you pick here follo - [Streaming & Live Queries](/sdk/streaming): SSE streams, client-side filtering, and backfill-then-live queries. - [Pipes](/sdk/pipes): Execute and manage named query pipes. - [Admin & System](/sdk/admin): Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/reference): Error codes, cancellation, the full API tree, and the codegen CLIs. +- [Reference & CLI](/sdk/reference): Error codes, cancellation, the full API tree, and row types. diff --git a/docs/src/content/docs/sdk/setup/typescript.mdx b/docs/src/content/docs/sdk/setup/typescript.mdx index 5683d081..31245dd3 100644 --- a/docs/src/content/docs/sdk/setup/typescript.mdx +++ b/docs/src/content/docs/sdk/setup/typescript.mdx @@ -496,7 +496,7 @@ const { data } = await clicks.select('page', 'button').limit(10); // data is Array<{ page: string; button: string; score: number; received_timestamp: string }> | null ``` -Generate the `Database` interface from a running server with the [codegen CLI](/sdk/reference#codegen-cli). +Generate the `Database` interface from a running server with the [codegen CLI](/sdk/reference#row-types). ## Result Type diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 7d64526f..ae86ce2a 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -157,7 +157,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | Official [TypeScript and Go clients](/sdk) — query builder, live queries, codegen | +| Client SDK | Each team writes one | Official [TypeScript and Go clients](/sdk) — query builder, live queries, typed rows | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. @@ -197,7 +197,7 @@ Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and | Thundering-herd coalescing | ✗ | Custom | ✓ | ✓ Ristretto + singleflight | | Row/column policies with JWT claims | ✗ | Custom | Tokens only | ✓ Hasura-style | | Named parameterized pipes | ✗ | Custom | ✓ | ✓ stored in NATS KV | -| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ TypeScript + Go SDKs | +| Type-safe client SDK | ✗ | Per team | Partial | ✓ TypeScript + Go SDKs | | Cost model | Infra only | Infra + eng time | Per-vCPU SaaS | Infra only | ## Part IV — End-to-end data journey From edbc0b3cb43b8b940329ac10cecc88073c0555e9 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Thu, 27 Aug 2026 20:05:40 -0400 Subject: [PATCH 59/59] fix(sdk)!: keep the MaxRetries default; refuse credentialed REST redirects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two ultrareview findings, both confirmed against the code. MaxRetries was an int, so its zero value was indistinguishable from an explicit 0, and the `>= 0` gate meant any caller who built ClientOptions for another reason ran with retries off. Headers is exactly that caller: the option the docs push operators toward for X-Operator-Key. Three of our own tests were silently running with retries disabled and passed only because they never reach the retry path. The gate could not simply become `> 0` — http_test.go and conformance_test.go set MaxRetries: 0 deliberately, and that gate would have handed them 2. MaxRetries is now *int: nil means unset, and Ptr(0) is a choice. Nothing is released, so the shape is free to change now. Two decisions beyond the finding: - Added `func Ptr[T any](v T) *T`. Without it every caller needs a throwaway variable, and it retires the `func strPtr(s string) *string` snippet the policy docs told readers to hand-roll for PolicyFilter. - Negative values clamp to 0. The old `>= 0` gate quietly turned a negative into the default; a pointer removes that, and an unclamped -1 makes maxAttempts 0, skipping the request and returning nil error. Second finding: doRequest had no redirect guard, while stream.go has refused 3xx on credentialed connects since it was written. net/http strips only its own four sensitive headers across hosts and forwards configured ones verbatim, so a 302 handed X-Operator-Key to whatever Location named, up to 10 hops. Same threat model, same SDK, opposite postures. doRequest now installs the same CheckRedirect on the same condition, on a copied client so a caller's own CheckRedirect survives, and a refused 3xx returns a non-retryable REDIRECT rather than a bare HTTP_302 — the REST analogue of SSE_REDIRECT. TestRESTRedirectsWithCredentials pins all three cases and asserts the redirect target received nothing. It also pins the scoping: an unauthenticated request still follows redirects. Docs: setup/go.md's ClientOptions table, the REDIRECT row in the reference error table, admin.mdx's pointer advice, and the CHANGELOG. setup/go.md also carried a :::caution titled "Options opts you out of the default, not just in", which documented this bug as intended behavior and told readers to set MaxRetries explicitly. Deleted: it is now false. go-sdk coverage 88.5% -> 88.8%. --- CHANGELOG.md | 2 +- clients/go/client_test.go | 23 ++++++++- clients/go/conformance_test.go | 2 +- clients/go/e2e_test.go | 2 +- clients/go/errors.go | 2 +- clients/go/http.go | 30 +++++++++++- clients/go/http_test.go | 63 ++++++++++++++++++++++++- clients/go/wavehouse.go | 21 +++++++-- docs/src/content/docs/sdk/admin.mdx | 2 +- docs/src/content/docs/sdk/reference.mdx | 1 + docs/src/content/docs/sdk/setup/go.md | 11 ++--- 11 files changed, 138 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54457952..b19ac533 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`). Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); the Go suites use gotestsum and honor `ARGS`/`V=1` like every other Go suite, while `test-sdk-ts` runs vitest. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language-specific section, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page under `/sdk/setup/` — `/sdk/setup/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/setup/go`, indexed by `/sdk/setup`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/setup/go.md` (new), `docs/src/content/docs/sdk/setup/typescript.mdx` (was `sdk/index.mdx`), `docs/src/content/docs/sdk/{index,queries,streaming,pipes,admin,reference}.mdx`, `docs/src/config/sidebar.ts`, `docs/src/components/Footer.astro`, `Makefile`, `.testcoverage.yml`, `scripts/cov/main.go`, `.github/workflows/ci.yml`, `.claude/commands/cover.md`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy,index}.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`). A request carrying a credential — a bearer token or a configured header — never follows a redirect, on REST and SSE alike, because `net/http` strips only its own four sensitive headers across hosts and forwards configured ones verbatim; the refusal surfaces as `REDIRECT` (REST) or `SSE_REDIRECT` (stream). `MaxRetries` is a `*int` so that options constructed for `Headers` alone keep the default of 2 instead of reading Go's zero value as "no retries"; `wavehouse.Ptr` supplies the pointer. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`) and TS (`tests/conformance/conformance_ts.mjs`) — each riding its own language's SDK target, both run by CI's unit job and local `make ci`. **Make targets follow one SDK family**: `test-sdk` runs both suites, `test-sdk-go` / `test-sdk-ts` run one (`test-ts` is renamed to the latter), and `test-sdk-go-e2e` drives a live server (`WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`); the Go suites use gotestsum and honor `ARGS`/`V=1` like every other Go suite, while `test-sdk-ts` runs vitest. `make test` stays the `test-unit` alias it has always been. The nested module needs no targets of its own for static checks: `fmt-go`, `lint-go`, `tidy`, and `fix-go` each span both modules (`vulncheck` is still root-only — [#437](https://github.com/Wave-RF/WaveHouse/issues/437)). **Docs are topic-first, not per-language** (the decision from [#313](https://github.com/Wave-RF/WaveHouse/pull/313)): `/sdk/queries`, `/sdk/streaming`, `/sdk/pipes`, `/sdk/admin`, and `/sdk/reference` each carry a `` block per language-specific section, so the topic URLs never churn as SDKs are added, and each language keeps a setup/caveats page under `/sdk/setup/` — `/sdk/setup/typescript` (moved off the root `/sdk`, which is now a language-neutral overview) and `/sdk/setup/go`, indexed by `/sdk/setup`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 2440f11b..d2ad710f 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -27,10 +27,31 @@ func TestNewClient(t *testing.T) { }, { name: "MaxRetries is honored", - cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{MaxRetries: 5}}, + cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{MaxRetries: Ptr(5)}}, wantBaseURL: "http://localhost:8080", wantMaxRetries: 5, }, + { + // The regression a plain int caused: options set for an unrelated + // reason silently disabled retries, because 0 is both "unset" and + // "none". Headers is the field the docs push operators toward. + name: "options set only for Headers keep the default", + cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{Headers: map[string]string{"X-Operator-Key": "k"}}}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 2, + }, + { + name: "Ptr(0) disables retries", + cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{MaxRetries: Ptr(0)}}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 0, + }, + { + name: "a negative MaxRetries clamps to 0", + cfg: Config{BaseURL: "http://localhost:8080", Options: &ClientOptions{MaxRetries: Ptr(-3)}}, + wantBaseURL: "http://localhost:8080", + wantMaxRetries: 0, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 5236ec7f..81496a59 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -129,7 +129,7 @@ func TestConformance_WireFormat(t *testing.T) { c := NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), - Options: &ClientOptions{MaxRetries: 0}, + Options: &ClientOptions{MaxRetries: Ptr(0)}, }) ctx := context.Background() diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 0e5e2475..ef9ae886 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -29,7 +29,7 @@ func e2eClient(t *testing.T) *Client { cfg := Config{ BaseURL: base, - Options: &ClientOptions{MaxRetries: 1}, + Options: &ClientOptions{MaxRetries: Ptr(1)}, } if tok := os.Getenv("WAVEHOUSE_AUTH"); tok != "" { cfg.Auth = StaticToken(tok) diff --git a/clients/go/errors.go b/clients/go/errors.go index dfa0e7f6..6a321798 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -12,7 +12,7 @@ import ( // [errors.As] to extract it from wrapped errors. type Error struct { Status int `json:"status"` // 0 for network/abort errors - Code string `json:"code"` // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" + Code string `json:"code"` // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED", "REDIRECT" Message string `json:"message"` Details map[string]any `json:"details,omitempty"` // full parsed error body, when present Retryable bool `json:"retryable"` diff --git a/clients/go/http.go b/clients/go/http.go index eec76588..b8657acc 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -81,6 +81,21 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a } } + // Never follow a redirect while carrying a credential: net/http drops + // Authorization across hosts but forwards custom headers verbatim, so a 302 + // would hand a configured X-Operator-Key to whatever Location names. The + // SSE path takes the same stance (SSE_REDIRECT in stream.go). Copied so a + // caller-supplied client keeps its own CheckRedirect. + client := hctx.httpClient + credentialed := authHeader != "" || len(hctx.headers) > 0 + if credentialed { + c := *client + c.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + client = &c + } + var lastErr error maxAttempts := hctx.maxRetries + 1 @@ -103,7 +118,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a req.Header.Set("Authorization", authHeader) } - res, err := hctx.httpClient.Do(req) + res, err := client.Do(req) if err != nil { if ctx.Err() != nil { return errAborted @@ -141,6 +156,19 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a return nil } + if credentialed && res.StatusCode >= 300 && res.StatusCode < 400 { + loc := res.Header.Get("Location") + _ = res.Body.Close() + return &Error{ + Status: res.StatusCode, + Code: "REDIRECT", + Message: fmt.Sprintf( + "request redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", + loc), + Retryable: false, + } + } + apiErr := parseErrorResponse(res) _ = res.Body.Close() diff --git a/clients/go/http_test.go b/clients/go/http_test.go index be7b04c3..ca5423a9 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -44,7 +44,7 @@ func queryTestCtx(t *testing.T, handler http.Handler) *Client { return NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), - Options: &ClientOptions{MaxRetries: 0}, + Options: &ClientOptions{MaxRetries: Ptr(0)}, }) } @@ -465,3 +465,64 @@ func TestConfiguredHeadersAreCopied(t *testing.T) { t.Fatalf("want the value captured at construction, got %q", v) } } + +// TestRESTRedirectsWithCredentials pins the REST half of the invariant that +// TestStream_TerminalConnectFailures pins for SSE: net/http strips only its own +// four sensitive headers across hosts and forwards configured ones verbatim, so +// following a 3xx while credentialed would hand X-Operator-Key (or a bearer +// token, on a same-host hop) to whatever Location names. +func TestRESTRedirectsWithCredentials(t *testing.T) { + // The redirect target records anything that reaches it. Nothing should. + var leaked http.Header + dst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + leaked = r.Header.Clone() + _, _ = io.WriteString(w, `[]`) + })) + defer dst.Close() + + redirector := func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, dst.URL+r.URL.Path, http.StatusFound) + } + + tests := []struct { + name string + opts *ClientOptions + auth func(context.Context) (string, error) + wantErr bool + }{ + {name: "configured header", opts: &ClientOptions{Headers: map[string]string{"X-Operator-Key": "secret"}}, wantErr: true}, + {name: "bearer token", auth: StaticToken("secret"), wantErr: true}, + // Nothing to protect, so the ordinary redirect-following stands. + {name: "no credential follows the redirect", wantErr: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + leaked = nil + src := httptest.NewServer(http.HandlerFunc(redirector)) + defer src.Close() + + client := NewClient(Config{BaseURL: src.URL, Auth: tt.auth, Options: tt.opts, HTTPClient: src.Client()}) + _, err := client.Schema.List(context.Background()) + + if !tt.wantErr { + if err != nil { + t.Fatalf("uncredentialed request should follow the redirect, got %v", err) + } + return + } + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want a *wavehouse.Error, got %v", err) + } + if apiErr.Code != "REDIRECT" { + t.Errorf("want code REDIRECT, got %q (%v)", apiErr.Code, apiErr) + } + if apiErr.Retryable { + t.Error("a refused redirect is terminal, not retryable") + } + if leaked != nil { + t.Errorf("credentialed request reached the redirect target: %v", leaked) + } + }) + } +} diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 74c78938..ad1b7930 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -19,6 +19,9 @@ import ( "strings" ) +// defaultMaxRetries is used when [ClientOptions.MaxRetries] is nil. +const defaultMaxRetries = 2 + // Config configures a [Client]. type Config struct { // BaseURL of the WaveHouse server (e.g. "http://localhost:8080"). @@ -38,8 +41,11 @@ type Config struct { // ClientOptions tunes transport behavior. type ClientOptions struct { // MaxRetries is the maximum number of retry attempts for retryable errors. - // Total attempts = MaxRetries + 1. Default: 2. - MaxRetries int + // Total attempts = MaxRetries + 1. nil means unset and uses the default of + // 2; use [Ptr] to set it, including Ptr(0) to turn retries off. A pointer + // is required because a plain int could not tell an explicit 0 from a + // caller who set only Headers. Negative values are treated as 0. + MaxRetries *int // Headers are sent on every REST call and SSE stream. They are applied // before the SDK's own headers, so Authorization, Accept, Content-Type and @@ -47,6 +53,11 @@ type ClientOptions struct { Headers map[string]string } +// Ptr returns a pointer to v, for the optional fields where nil means "unset" +// and a pointer to the zero value is a deliberate choice — [ClientOptions.MaxRetries] +// and the [PolicyFilter] operators. wavehouse.Ptr(0) disables retries. +func Ptr[T any](v T) *T { return &v } + // StaticToken returns an Auth function that always returns the same token. func StaticToken(token string) func(context.Context) (string, error) { return func(context.Context) (string, error) { return token, nil } @@ -65,9 +76,9 @@ type Client struct { // NewClient creates a new WaveHouse client. func NewClient(cfg Config) *Client { - maxRetries := 2 - if cfg.Options != nil && cfg.Options.MaxRetries >= 0 { - maxRetries = cfg.Options.MaxRetries + maxRetries := defaultMaxRetries + if cfg.Options != nil && cfg.Options.MaxRetries != nil { + maxRetries = max(*cfg.Options.MaxRetries, 0) } // Copy so a later mutation of the caller's map can't reach into requests. diff --git a/docs/src/content/docs/sdk/admin.mdx b/docs/src/content/docs/sdk/admin.mdx index d736cbda..1199f6a7 100644 --- a/docs/src/content/docs/sdk/admin.mdx +++ b/docs/src/content/docs/sdk/admin.mdx @@ -114,7 +114,7 @@ result, err := wh.Policy.Validate(ctx, policyDraft) // result.Valid == true, or err wraps the validation failure details ``` -`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator, hence the `tenantFilter` variable above, or a `func strPtr(s string) *string { return &s }` helper. +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, so an empty string is distinguishable from an absent operator, hence the `tenantFilter` variable above, or `wavehouse.Ptr("...")`. diff --git a/docs/src/content/docs/sdk/reference.mdx b/docs/src/content/docs/sdk/reference.mdx index 63f3d3eb..810ad68d 100644 --- a/docs/src/content/docs/sdk/reference.mdx +++ b/docs/src/content/docs/sdk/reference.mdx @@ -118,6 +118,7 @@ The SDK never panics on API or network failures, mirroring the TypeScript SDK's | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | +| *3xx* | `REDIRECT` | No | The request carried a credential and was redirected; the SDK refused to follow it | | 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | | 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | | 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https`; retrying cannot fix it | diff --git a/docs/src/content/docs/sdk/setup/go.md b/docs/src/content/docs/sdk/setup/go.md index 9c400ca8..23332923 100644 --- a/docs/src/content/docs/sdk/setup/go.md +++ b/docs/src/content/docs/sdk/setup/go.md @@ -82,28 +82,23 @@ The default client sets no `Timeout`; use a `context.Context` deadline to preven | Field | Type | Default | Description | |-------|------|---------|-------------| -| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | +| `MaxRetries` | `*int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). `nil` takes the default; `wavehouse.Ptr(0)` turns retries off. A negative value is treated as `0`. | | `Headers` | `map[string]string` | `nil` | Sent on every request the client makes: REST calls and SSE streams alike. | `*Client` is safe for concurrent use (state is immutable after `NewClient` and builder chains copy), provided your `Auth` function is concurrency-safe. -:::caution[`Options` opts you out of the default, not just in] -The 2-retry default applies only when `Config.Options` is `nil`. Passing `&wavehouse.ClientOptions{}` leaves `MaxRetries` at Go's zero value (`0`), which disables retries: set it explicitly. -::: - `Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk/setup/typescript#custom-headers): a gateway credential, a tenant selector, or tracing metadata with no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): ```go wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "http://localhost:8080", Options: &wavehouse.ClientOptions{ - MaxRetries: 2, // Options opts out of the default: set it explicitly. - Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, + Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, }, }) ``` -The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both: supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any collision, matched case-insensitively and replacing rather than appending. The map is copied at `NewClient`, so later mutation changes nothing. Because a configured header can carry a credential, the SDK refuses to follow a redirect on any request that carries one, on REST calls and streams alike: `net/http` strips only its own four sensitive headers across hosts and forwards yours verbatim. A refused REST redirect surfaces as a non-retryable `REDIRECT`; a stream reports `SSE_REDIRECT`. There is no Go field for `options.fetch` or `options.fetchOptions` because `Config.HTTPClient` covers both: supply your own `*http.Client`, or a custom `http.RoundTripper` on its `Transport`. :::note[How the token is transmitted] The SDK sends `Authorization: Bearer ` on every request, SSE streams included, and never uses a `?token=` query fallback. The TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is shared behavior rather than a Go-only property (see its [equivalent note](/sdk/setup/typescript#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive.