Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ go vet ./...
```

Default tests use owned loopback fixtures, without databases.
[Real backend tests](README.md#examples-and-validation) need the `integration` tag
[Real backend tests](docs/development.md#real-backend-tests) need the `integration` tag
and operator-provided fixtures.

## Prepare a pull request

- Keep the change focused and explain the observed problem and resulting behavior.
- Add a regression test when fixing behavior.
- Update the README and [examples](examples) when the API changes.
- Update the README, [docs](docs) and [examples](examples) when the API changes.
- Add system integration, backend fault tests and throughput benchmarks to [weir-tests](https://github.com/batchstream/weir-tests).
591 changes: 178 additions & 413 deletions README.md

Large diffs are not rendered by default.

62 changes: 62 additions & 0 deletions docs/connections.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Connections

How a Client finds servers, keeps connections healthy and closes.

## Address and reconnection

Point a client at a Weir application address and send typed business requests to
the Stores hosted there. IP and DNS addresses work in Kubernetes and other
deployments.

`New` takes a `host:port` address and rejects anything else. It connects on
first use and reconnects after failures. When a DNS name resolves to several
Weir servers, streams are spread round-robin across them, so every server behind
the address must host the Stores the client uses. Each Read, Mutate, record
stream or Native call uses a gRPC stream of its own; single-record calls share
streams, as described under [Streaming and deadlines](streaming.md). Use a name
that lists every server, such as a Kubernetes headless Service; a ClusterIP
Service resolves to one virtual IP. The Client looks the name up again every 5
seconds. A server that joins receives new calls once connected, and shared
streams move to it until every server carries about the same number. A server
that stops resolving, as a Kubernetes Pod does once it starts terminating,
receives no new records; its streams finish the records they carry and close.
So scaling a headless Service up or down, or rolling out a new version, does not
fail calls, however long they run, provided a server stops resolving at least a
few seconds before it stops taking records, as the Weir chart's preStop delay
arranges. A server that does not host the requested
Store fails the call with NotFound.

## Reuse and close

Reuse the client across goroutines and
close it after callers finish.
Calls started after `Close` return `ErrClosed`; calls still running end with a
gRPC cancellation error. A call of a Client whose context ends returns the
context's error, so test it with `errors.Is` rather than its gRPC status; the
package operation functions return the gRPC status. A Client sends a record again only when a server refused it
before it began, as described under
[Resending refused records](streaming.md#resending-refused-records).

## Kubernetes DNS names

In Kubernetes, give the Service's full name with a trailing dot, such as
`weir.application.svc.cluster.local.:7447`. Without the dot, a name with fewer
than five dots is first tried under each of the Pod's DNS search domains. In a
Pod with four search domains, one lookup of `weir.application.svc` sends 6
queries and `weir.application.svc.cluster.local` sends 10, against 2 with the
dot; every Client repeats its lookup every 5 seconds.

## Keepalive

A Client pings a server it has heard nothing from for 10 seconds and drops the
connection if no answer comes within 5 seconds, so calls leave a server they
can no longer reach, such as one behind a broken network path or a hung
process, even while its name still resolves. Calls waiting on that connection
end with an error, and a mutation sent on it has an unknown outcome. Servers
must accept pings this often, as those of the Weir chart 0.15.0-rc.1 and later
do; an older server ends such connections, after which the Client pings less
often for the rest of its life. Upgrade servers before Clients.

## Network access

The Weir application listener uses plaintext gRPC; restrict its network access.
59 changes: 59 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Development

How to test the SDK and publish a version.

## Module dependencies

The SDK depends on the stable `github.com/batchstream/weir-protocol v0.17.0`
release. The independent protocol repository owns public schemas, generated
protobuf types and shared validation. Both Weir and this SDK consume
it; neither the protocol nor SDK module depends on the server. The SDK does not
generate schemas.

## Examples

[read](../examples/read/main.go) performs a typed read, [basic](../examples/basic/main.go)
mutates a document then reads multiple resources in one stream, and
[native](../examples/native/main.go) consumes
native responses. These examples use SDK request and result types.

## Tests

```sh
go test -race ./...
go vet ./...
```

Default tests use owned loopback fixtures, without databases or production I/O.
CI checks typed operation dispatch, acknowledgements with transport errors,
malformed/empty native chunks, incremental consumption,
completion/cancellation, refusal-only resends and deadline admission. It also checks that
the module graph does not contain the Weir server, that `go.mod` replaces no
module and that the protocol is a released `vMAJOR.MINOR.PATCH` version.

## Release

After a reviewed change is merged, an authorized maintainer publishes a stable
version with `gh workflow run release.yml --ref main -f version=vX.Y.Z`. The
workflow runs CI on the `main` commit, then tags that commit and creates its
GitHub release; it refuses a version already tagged at another commit.

## Real backend tests

Real backend tests require an explicit tag and operator-provided fixtures:

```sh
WEIR_ADDRESS=127.0.0.1:7447 \
WEIR_MONGO_STORE=mongo \
WEIR_MONGO_RESOURCE=weir_example/records \
WEIR_SEARCH_STORE=search \
WEIR_SEARCH_RESOURCE=weir_example \
go test -race -tags integration -run 'Test(Mongo|Search)(Lifecycle|Streaming)' -v .
```

Pre-create disposable collections/indexes. Each test creates and cleans up only
its generated records. Lifecycle tests cover CRUD, backend expressions and
Native; streaming tests verify same-resource ordering through an individual
failure, Lua persistence, and repeated and missing reads. System integration tests,
backend fault tests and throughput benchmarks belong to the independent
[weir-tests](https://github.com/batchstream/weir-tests) repository.
36 changes: 36 additions & 0 deletions docs/lua.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Lua transforms

How to change a record with a Lua function through `AtomicTransform`.

`LuaTransform` accepts UTF-8 Source and optional Input. Source must return exactly
one function, called with ordinary Lua tables as `function(current, incoming)`.
Missing current or omitted Input is nil. Fields, arrays, comparisons and loops
use ordinary Lua syntax; `weir.null()` represents explicit document null.
For example, this merge can create a record on its first call:

```go
input := &weir.Document{ContentType: "application/json", Data: []byte(`{"state":"ready"}`)}
source := []byte(`return function(current, incoming)
current = current or weir.object()
for key, item in pairs(incoming or {}) do current[key] = item end
return current
end`)
lua := &weir.LuaTransform{Source: source, Input: input}
request := &weir.AtomicTransformRequest{Resource: "records/s:first", Lua: lua}
options := weir.AtomicTransformOptions{StoreName: "search", Request: request}
result, err := client.AtomicTransform(ctx, options)
```

Mongo Input uses BSON. Lua source and document values have no Weir size, depth,
or node-count gate. The callback must return exactly one object to create/replace,
or `weir.keep()`, `weir.delete()` or `weir.reject(message)`. Reject returns
NOT_APPLIED with PRECONDITION_FAILED; nil, missing/multiple returns and scalar
results are errors. `weir.object()` and `weir.array()` distinguish empty containers.
Lua integers preserve 64 bits, and `weir.time.now()` returns one fixed UTC timestamp
per operation, including confirmed conflict retries. This function entry point,
return contract, helper behavior and document conversion follow the protocol's
Lua contract. See the server's
[Lua guide](https://github.com/batchstream/weir/blob/main/docs/lua.md).
Concurrent first creation can trigger a bounded fresh read/evaluation after a
confirmed conflict. An ambiguous write or commit acknowledgement never triggers
automatic replay.
75 changes: 75 additions & 0 deletions docs/native.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Native requests and responses

How to send a backend's own commands, such as queries, scans and counts.

## Requests

`NativeRequest` has Resource and a required Request Document. Its ContentType
identifies an adapter-owned format; Data carries opaque bytes and may be empty. The public protocol and SDK do not select or interpret backends.
A new Store can use its own content type without changing the schema or SDK.
Backend queries, counts and traversal use Native with caller-owned backend
commands, pagination and cleanup.
Native imposes no command, method, path or parameter allowlist. The backend
handles authorization and operation semantics. Resource is an optional canonical
prefix; empty addresses the backend root when supported. MongoDB accepts a
database target or an empty Resource with `$db` in its `application/bson` command
document. Search uses `application/http` and an optional resource prefix.

```go
document := &weir.Document{ContentType: "application/vnd.example.command", Data: commandBytes}
request := &weir.NativeRequest{Resource: "records", Request: document}
options := weir.NativeOptions{StoreName: "example", Request: request}
options.Consume = func(ctx context.Context, response *weir.NativeResponse, data []byte) error {
return consumeNativeBytes(ctx, response, data)
}
result, err := client.Native(ctx, options)
```

## Responses

Consume receives nonempty response chunks and finishes before the next chunk.
`NativeResult.Response` also retains metadata for an empty body when no callback
runs. Response.Metadata is an optional opaque Document;
Response.BodyContentType describes the separately streamed body. Metadata may
use a different content type from the request. Do not mutate response metadata
in a callback. The SDK hides the wire Head/Chunk/End phases.

## HTTP helpers

For HTTP-backed adapters, `NewHTTPNativeRequest(resource, request)` consumes and
closes a standard `net/http.Request` body and serializes an `application/http`
Document. It emits an HTTP/1.1 origin-form target, exact Content-Length and no
default User-Agent. Methods, escaped paths, queries and application headers are
preserved. Pass an empty resource for backend-root operations. HTTP request bodies
and headers have no SDK size gate. Trailers are rejected. The Host header never
chooses the Store or backend destination.

```go
httpRequest, err := http.NewRequestWithContext(ctx, http.MethodGet, "/_doc/example", nil)
if err != nil { return err }
request, err := weir.NewHTTPNativeRequest("records", httpRequest)
if err != nil { return err }
options := weir.NativeOptions{StoreName: "search", Request: request}
options.Consume = func(ctx context.Context, response *weir.NativeResponse, data []byte) error {
return consumeNativeBytes(ctx, response, data)
}
result, err := client.Native(ctx, options)
```

`ParseHTTPNativeResponse(response)` optionally parses application/http metadata
containing a complete status line and CRLF header block, with no body bytes or
transfer framing. It returns HTTPNativeResponse with StatusCode and Headers;
response body bytes continue through Native's chunk consumer. These helpers are
ordinary HTTP codecs, independent of any Store type or protobuf backend fields.

## Completion evidence

NativeResult distinguishes response evidence from the Go error. A confirmed
NativeNotStarted means the backend operation did not start. NativeResponseIncomplete
means it started without a complete response and establishes no write outcome.
NativeCompletionUnconfirmed means no terminal was received; Response may still be
available. NativeResponseComplete means all response bytes arrived. Its status/body
can report a backend error such as HTTP 404 with no Go error; interpret that data
separately. A validated terminal is retained alongside a later RPC error. A
Client sends a Native request again only after a refused NativeNotStarted without
a Response.
Loading