Skip to content
Merged
63 changes: 38 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,30 @@ Typed Go client for [Weir](https://github.com/batchstream/weir).

| Method | Use |
| --- | --- |
| `ReadOne` | Read one record. |
| `Read` | Read many records. Results return in input order. |
| `Create` | Write one record that must not exist. |
| `Replace` | Write one record that must exist. |
| `Put` | Create or replace one record. |
| `Delete` | Delete one record. |
| `Mutate` | Write many records in one call. Not a transaction. |
| `AtomicTransform` | Change one record with a Lua function or a backend expression. |
| `ReadStream`, `MutateStream` | Send and receive records incrementally. |
| `Native` | Send a backend's own command. Use it for queries, scans and counts. |
| `Read(ctx, store, options...)` | Read records. Results return in input order. |
| `Mutate(ctx, store, options...)` | Write records. Not a transaction. |
| `Native(ctx, store, request, options...)` | Send a backend's own command. Use it for queries, scans and counts. |

### Options

| Option | Applies to | Effect |
| --- | --- | --- |
| `WithDialOptions` | `New` | gRPC options for the connection, such as an interceptor or a custom dialer. |
| `WithRequests` | `Read`, `Mutate` | The records to send. |
| `WithNext` | `Read`, `Mutate` | A callback that produces more records, after those of `WithRequests`. |
| `WithReadConsume`, `WithMutateConsume` | `Read`, `Mutate` | A callback that receives each result as it arrives, instead of the call returning them. |
| `WithQueryVisible` | `Mutate` | Wait until normal queries can see the mutations. |
| `WithChunks` | `Native` | A callback that receives the response body in chunks, instead of `Body` holding it. |

### Mutation actions

| Action | Use |
| --- | --- |
| `MutationCreate` | Write a record that must not exist. |
| `MutationReplace` | Write a record that must exist. |
| `MutationPut` | Create or replace a record. |
| `MutationDelete` | Delete a record. |
| `MutationAtomicTransform` | Change a record with a Lua function or a backend expression. |

### Backend formats

Expand Down Expand Up @@ -58,8 +72,8 @@ documents. Your code owns the document bytes.
4. Never automatically resend a mutation that is `MutationApplied` or
`MutationUnknown`, or that has no result. It has applied or may have applied.
Reconcile it first.
5. A write may not be visible to the next Search query. Set
`MutationQueryVisible` when it must be.
5. A write may not be visible to the next Search query. Pass
`WithQueryVisible` when it must be.
6. The application listener uses plaintext gRPC. Restrict its network access.

## Requirements
Expand All @@ -84,7 +98,7 @@ This writes one record to MongoDB through Weir and reads it back.
mkdir hello
cd hello
go mod init example.com/hello
go get github.com/batchstream/weir-go@v0.23.1 go.mongodb.org/mongo-driver/v2/bson
go get github.com/batchstream/weir-go@v0.24.0 go.mongodb.org/mongo-driver/v2/bson
```

**Step 3.** Save this as `main.go`.
Expand Down Expand Up @@ -122,24 +136,23 @@ func main() {
log.Fatal(err)
}
document := &weir.Document{ContentType: "application/bson", Data: data}
writeRequest := &weir.WriteRequest{Resource: resource, Document: document}
writeOptions := weir.WriteOptions{StoreName: storeName, Request: writeRequest}
written, err := client.Put(ctx, writeOptions)
write := &weir.MutateRequest{Resource: resource, Action: weir.MutationPut, Document: document}
written, err := client.Mutate(ctx, storeName, weir.WithRequests(write))
if err != nil {
log.Fatal(err)
}
if written.GetFailure() != nil {
log.Fatal(written.GetFailure())
if written[0].GetFailure() != nil {
log.Fatal(written[0].GetFailure())
}
fmt.Println("outcome:", written.GetOutcome())
fmt.Println("outcome:", written[0].GetOutcome())

// Read the record back.
readRequest := &weir.ReadRequest{Resource: resource}
readOptions := weir.ReadOneOptions{StoreName: storeName, Request: readRequest}
result, err := client.ReadOne(ctx, readOptions)
read := &weir.ReadRequest{Resource: resource}
results, err := client.Read(ctx, storeName, weir.WithRequests(read))
if err != nil {
log.Fatal(err)
}
result := results[0]
if result.Failure != nil {
log.Fatal(result.Failure)
}
Expand Down Expand Up @@ -176,18 +189,18 @@ message: hi

| Guide | Contents |
| --- | --- |
| [Connections](docs/connections.md) | Addresses, Kubernetes DNS names, keepalive and closing |
| [Connections](docs/connections.md) | Addresses, gRPC options, Kubernetes DNS names, keepalive and closing |
| [Operations](docs/operations.md) | Reads, writes, visibility, results and outcomes |
| [Lua](docs/lua.md) | Lua transforms |
| [Native](docs/native.md) | Backend commands and HTTP helpers |
| [Streaming](docs/streaming.md) | Streams, deadlines and resending refused records |
| [Streaming](docs/streaming.md) | Callbacks, streams, deadlines and resending refused records |
| [Development](docs/development.md) | Tests, real backend tests and releases |
| [API reference](https://pkg.go.dev/github.com/batchstream/weir-go) | Every exported type and function |

| Example | Shows |
| --- | --- |
| [read](examples/read/main.go) | A typed read |
| [basic](examples/basic/main.go) | A mutation, then many reads in one stream |
| [basic](examples/basic/main.go) | A mutation, then many reads in one call |
| [native](examples/native/main.go) | A native command and its response |

## Development
Expand Down
113 changes: 2 additions & 111 deletions api.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package weir

import (
"context"

"github.com/batchstream/weir-protocol/api/protocol"
pb "github.com/batchstream/weir-protocol/api/weir/v1"
)
Expand All @@ -19,10 +17,6 @@ type FailureCode = pb.FailureCode
type MutationResult = pb.MutationResult
type MutationOutcome = pb.MutationOutcome

// MutationOptions controls completion uniformly for every mutation action.
// A nil Options or zero Visibility waits for application acknowledgement.
type MutationOptions = pb.MutationOptions
type MutationVisibility = pb.MutationVisibility
type LuaTransform = pb.LuaTransform
type ReadRequest = pb.ReadRequest

Expand All @@ -39,9 +33,6 @@ const (
MutationApplied = pb.MutationOutcome_APPLIED
MutationUnknown = pb.MutationOutcome_UNKNOWN

MutationApplicationAcknowledged = pb.MutationVisibility_APPLICATION_ACKNOWLEDGED
MutationQueryVisible = pb.MutationVisibility_QUERY_VISIBLE

NativeCompletionUnconfirmed = pb.NativeCompletion_NATIVE_COMPLETION_UNSPECIFIED
NativeNotStarted = pb.NativeCompletion_NATIVE_NOT_STARTED
NativeResponseComplete = pb.NativeCompletion_RESPONSE_COMPLETE
Expand All @@ -64,50 +55,18 @@ const (
// EncodeSegment encodes one decoded resource path segment, including slashes.
func EncodeSegment(segment string) string { return protocol.EncodeSegment(segment) }

type WriteRequest struct {
Resource string
Document *Document
Options *MutationOptions
}

type DeleteRequest struct {
Resource string
Options *MutationOptions
}

// AtomicTransformRequest requires exactly one of Lua and BackendExpression.
type AtomicTransformRequest struct {
Resource string
Lua *LuaTransform
BackendExpression *Document
Options *MutationOptions
}

// NativeResult retains validated response and completion evidence. Completion is
// NativeCompletionUnconfirmed until a terminal acknowledgement arrives; Response
// may already be available. An RPC error never revokes a confirmed terminal.
// Completion describes response transport, not whether backend writes applied.
// Body holds the response body of a call without WithChunks, as far as it arrived.
type NativeResult struct {
Response *NativeResponse
Body []byte
Completion NativeCompletion
Failure *Failure
}

// ReadOneOptions describes a single read. Use ReadOptions to share one RPC across
// multiple independent requests in the same Store.
type ReadOneOptions struct {
StoreName string
Request *ReadRequest
}

// ReadOptions describes a finite read sequence addressed to one Store. Resources
// are canonical paths relative to StoreName. Results have the input order. Do
// not mutate requests or their document bytes until the call returns.
type ReadOptions struct {
StoreName string
Requests []*ReadRequest
}

// MutationAction selects one business operation without a protobuf oneof.
type MutationAction uint8

Expand All @@ -121,80 +80,12 @@ const (

// MutateRequest describes one mutation. Create, Put and Replace require Document;
// Delete accepts no payload; AtomicTransform requires Lua or BackendExpression.
// Options applies to every action. MutationQueryVisible requires visibility to
// subsequent normal queries before an APPLIED result without Failure. APPLIED
// with Failure confirms application only. Nil Options or zero Visibility waits
// for application acknowledgement without a separate visibility guarantee.
type MutateRequest struct {
Resource string
Action MutationAction
Document *Document
Lua *LuaTransform
BackendExpression *Document
Options *MutationOptions
}

// MutateOptions describes independent mutations addressed to one Store. It is
// not a transaction. Same-resource requests execute in input order; different
// resources may execute concurrently. Cross-RPC ordering follows the database.
// Do not mutate requests, options or their document bytes until the call returns.
type MutateOptions struct {
StoreName string
Requests []*MutateRequest
}

// ReadStreamOptions produces reads without retaining every input or result. Next
// returns io.EOF after its final item. Items are validated before
// they are sent; an invalid later item does not undo earlier requests.
// Consume receives each confirmed result as its record completes; the index
// identifies the input, starting at 1.
// Next and Consume run concurrently; synchronize any shared application state.
// Both callbacks must honor their context and return promptly. Keep yielded
// requests and document bytes immutable until the corresponding Consume call.
type ReadStreamOptions struct {
StoreName string
Next func(context.Context) (*ReadRequest, error)
Consume func(context.Context, uint64, *ReadResult) error
}

// MutateStreamOptions incrementally produces mutations to one Store. Next
// returns io.EOF after the final item. Consume receives confirmed results as
// their records complete, identified by input index; a later error does not
// revoke those acknowledgements. Items not consumed remain unacknowledged and
// must never be automatically replayed.
// Next and Consume run concurrently; synchronize any shared application state.
// Both callbacks must honor their context and return promptly. Keep yielded
// requests, options and document bytes immutable until the corresponding
// Consume call.
type MutateStreamOptions struct {
StoreName string
Next func(context.Context) (*MutateRequest, error)
Consume func(context.Context, uint64, *MutationResult) error
}

type WriteOptions struct {
StoreName string
Request *WriteRequest
}

type DeleteOptions struct {
StoreName string
Request *DeleteRequest
}

type AtomicTransformOptions struct {
StoreName string
Request *AtomicTransformRequest
}

// Native consumes only nonempty response chunks with their validated metadata.
// Consume completes before the next chunk and must honor its context. An empty
// response invokes no callback; its metadata is available in NativeResult. Do
// not mutate response metadata, which remains attached to the final result.
type NativeOptions struct {
StoreName string
Request *NativeRequest
Consume func(context.Context, *NativeResponse, []byte) error
}

// ReadResult contains exactly one of Document, Missing, or Failure. Missing
Expand Down
68 changes: 17 additions & 51 deletions client.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ import (
"google.golang.org/grpc"
)

// execute consumes one Native request incrementally through its terminal event
// and final gRPC status.
func execute(ctx context.Context, client pb.StoreServiceClient, request *pb.ExecuteRequest, consume func(context.Context, *pb.Event) error) (resultErr error) {
// execute sends one request on a gRPC stream of its own and passes its events to
// consume, which reports the terminal one, then awaits the final gRPC status.
func execute(ctx context.Context, client pb.StoreServiceClient, request *pb.ExecuteRequest, consume func(context.Context, *pb.Event) (bool, error)) (resultErr error) {
if client == nil || consume == nil {
return errors.New("Execute requires a client and consumer")
}
Expand All @@ -32,7 +32,7 @@ func execute(ctx context.Context, client pb.StoreServiceClient, request *pb.Exec
return err
}
// Commit this attempt before consuming any event; grpc must not transparently
// replay a Native request that could mutate its backend.
// replay a request that could mutate its backend.
_ = stream.Context()
done := make(chan struct{})
var sendErr error
Expand All @@ -58,11 +58,11 @@ func execute(ctx context.Context, client pb.StoreServiceClient, request *pb.Exec
resultErr = errors.Join(resultErr, sendErr)
}
}()
state := nativeStreamState{}
terminal := false
for {
message, err := stream.Recv()
if errors.Is(err, io.EOF) {
if !state.terminal {
if !terminal {
return errors.New("Execute ended without a terminal business event")
}
// A peer can finish before Send returns to its caller. Await the
Expand All @@ -77,53 +77,19 @@ func execute(ctx context.Context, client pb.StoreServiceClient, request *pb.Exec
return err
}
for _, response := range message.Responses {
if err := state.consume(ctx, response, consume); err != nil {
if terminal {
return errors.New("Execute event after terminal business event")
}
if err := protocol.ValidateExecuteResponse(response); err != nil {
return err
}
if response.Index != 1 {
return errors.New("unexpected Execute response ordinal")
}
terminal, err = consume(ctx, response.Event)
if err != nil {
return err
}
}
}
}

type nativeStreamState struct {
terminal bool
head bool
}

// consume checks one Native response in stream order and passes its event on.
func (s *nativeStreamState) consume(ctx context.Context, response *pb.ExecuteResponse, consume func(context.Context, *pb.Event) error) error {
if s.terminal {
return errors.New("Execute event after terminal business event")
}
if err := protocol.ValidateExecuteResponse(response); err != nil {
return err
}
if response.Index != 1 {
return errors.New("unexpected Native response ordinal")
}
if err := s.validate(response.Event); err != nil {
return err
}
return consume(ctx, response.Event)
}

func (s *nativeStreamState) validate(event *pb.Event) error {
switch value := event.Value.(type) {
case *pb.Event_Head:
if s.head {
return errors.New("invalid Native response head")
}
s.head = true
case *pb.Event_Chunk:
if !s.head {
return errors.New("Native chunk before head")
}
case *pb.Event_NativeEnd:
if value.NativeEnd.Completion == NativeResponseComplete && !s.head || value.NativeEnd.Completion == NativeNotStarted && s.head {
return errors.New("invalid Native completion")
}
s.terminal = true
default:
return errors.New("unexpected Native event")
}
return nil
}
Loading