Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ err := ctx.RunFlow(request)

Flows and steps are registered under their bare Go function names, and can be called by that name as a string. Two functions with the same name in different packages collide, and `Serve` reports it. Use `NewFlowHandler` and `NewStepHandler` to choose a name, which closures and the instantiations of a generic function need too.

`string` and `[]byte` travel as raw bytes and everything else as JSON, so read a value back as the type it was written.
Strings and byte slices, including types built on them, travel as raw bytes unless the type has its own JSON or text methods. Everything else travels as JSON, so read a value back as the type it was written. Adding or removing those methods changes how such a type encodes, so let flows carrying it finish first.

### Parallel steps and cleanup

Expand Down Expand Up @@ -143,7 +143,7 @@ A step that is already running only finds out about the cancellation through `ct

| Option | Applies to | Effect |
|---|---|---|
| `WithMaxAttempts(n)` | both | Total attempts, including the first. Zero means unlimited. |
| `WithMaxAttempts(n)` | both | Total attempts, including the first. Zero means the default, and `gush.UnlimitedAttempts` keeps retrying. Any other negative number is an error. |
| `WithInitialInterval(d)` | both | Delay before the first retry. |
| `WithBackoffCoefficient(f)` | both | Multiplier on the delay after each retry. |
| `WithMaxInterval(d)` | both | Cap on the delay between retries. |
Expand All @@ -154,9 +154,10 @@ A step that is already running only finds out about the cancellation through `ct
| `WithFairKey(key)` | both | Group to schedule fairly against, such as a tenant. |
| `WithLogAttribute(k, v)` | both | Attribute on every log line of this run and its children. |
| `WithLogAttributes(m)` | both | The same, for several attributes. |
| `WithID(id)` | flow | Flow ID. Otherwise one is generated, which `StartFlow` returns. A client's `RunFlow` or `StartFlow` joins a flow already running with this ID and ignores this run's input, and a child flow gets an error. |
| `WithID(id)` | flow | Flow ID. Otherwise one is generated, which `StartFlow` returns. A client's `RunFlow` or `StartFlow` joins a flow already running with this ID and ignores this run's flow, input and settings, so concurrent callers share one run. A child flow gets an error that `gush.IsFlowRunningError` recognizes. |
| `WithErrorIfRunning()` | flow | A client's `RunFlow` or `StartFlow` returns that same error instead of joining. |
| `WithTimeout(d)` | flow | Limit on the whole execution, including every retry. |
| `WithConfig(c)` | flow | Starts from a `FlowConfig`, such as one decoded from a queue. Fields left out of the JSON keep their defaults. Whoever applies it chooses `RunFlow` or `StartFlow`. |
| `WithConfig(c)` | flow | Starts from a `FlowConfig`, such as one decoded from a queue. Settings left out of the JSON, or zero, use their defaults, and durations in JSON are integer nanoseconds. Whoever applies it chooses `RunFlow` or `StartFlow`. |
| `WithQueue(q)` | step | Runs on another queue. |
| `WithAttemptTimeout(d)` | step | Limit on each attempt, not the step as a whole. |
| `WithHeartbeatTimeout(d)` | step | Fails an attempt that stops calling `ctx.Heartbeat()`. |
Expand Down Expand Up @@ -227,22 +228,22 @@ err = client.Resume(ctx, resumeToken, decision, nil)

**Set `WithAttemptTimeout` to the longest acceptable wait.** The step's timeout keeps running while it is suspended, and when it expires the step retries and runs its code again, sending a second request. The default is 2 minutes and 10 attempts. Use `WithMaxAttempts(1)` unless running the code twice is safe.

An error passed to `Resume` counts as a failed attempt, so a plain error runs the code again too. To reject for good, pass `gush.NewPermanentError("rejected", "Rejected", nil)`.
An error passed to `Resume` fails the step for good, so a rejection does not send the request again. To run the code again under the step's retry policy, pass a `gush.NewApplicationError`. A canceled error cancels the step if it is being canceled, and otherwise fails it for good too.

## Changing running flows

Flows can run for a long time, and a replay must take the same path it took originally. `Version` keeps old executions on the old code.

```go
v := ctx.Version("payment-provider", 1)
v := ctx.Version("payment-provider", gush.DefaultFlowVersion, 1)
if v == gush.DefaultFlowVersion {
err = ctx.RunStep(OldCharge, input, &output)
} else {
err = ctx.RunStep(NewCharge, input, &output)
}
```

`Version` accepts every recorded version, so once an old branch is deleted, an execution that recorded it runs the newer one. To make such an execution fail instead, call the SDK's `workflow.GetVersion` with a higher minimum.
The second argument is the oldest version the code still supports. Once no execution can reach an old branch, delete it and raise that minimum, here with `ctx.Version("payment-provider", 1, 1)`. An execution that recorded something older then stops making progress, with its flow tasks failing, until code that supports it is deployed. It never runs a branch it did not take.

## Long-running flows

Expand Down Expand Up @@ -279,7 +280,7 @@ Lines written while Temporal replays a flow are dropped, so each appears once. H

### Export over OTLP

`gush.Dial` starts exporting when `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and `Close` flushes. A process that logs without a client calls `gush.InitLogExport()` itself. [OpenTelemetry's other variables](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) set the headers and TLS, and the protocol is `http/protobuf` unless they ask for `grpc`:
`gush.Dial` starts exporting when `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and `Close` flushes. A process that logs without a client calls `gush.InitLogExport()` itself. [OpenTelemetry's other variables](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) set the headers and TLS. The protocol is `http/protobuf` unless they ask for `grpc`, and any other protocol is an error:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://collector:4317
Expand Down
87 changes: 59 additions & 28 deletions codec.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package gush

import (
"bytes"
"encoding"
"encoding/base64"
"encoding/json"
"fmt"
Expand All @@ -16,33 +17,60 @@ var (
zstdEncoder, _ = zstd.NewWriter(nil)
)

// conversionMethods are the interfaces through which a type chooses its own JSON form.
var conversionMethods = []reflect.Type{
reflect.TypeFor[json.Marshaler](),
reflect.TypeFor[json.Unmarshaler](),
reflect.TypeFor[encoding.TextMarshaler](),
reflect.TypeFor[encoding.TextUnmarshaler](),
}

// usedAsIs reports whether values of a type are their own encoding: strings and byte
// slices, including types built on them, unless the type chooses its own JSON form.
func usedAsIs(valueType reflect.Type) bool {
isBytes := valueType.Kind() == reflect.Slice && valueType.Elem() == reflect.TypeFor[byte]()
if valueType.Kind() != reflect.String && !isBytes {
return false
}

pointer := reflect.PointerTo(valueType)
for _, methods := range conversionMethods {
if pointer.Implements(methods) {
return false
}
}
return true
}

// Encode serializes a value the way flows and steps do for their input and output.
// []byte and string are used as they are, including through a pointer, a nil value or
// nil pointer encodes to nil, and everything else becomes JSON without HTML escaping.
// Strings and byte slices are used as they are, including through a pointer and including
// types built on them, unless the type has its own JSON or text methods. A nil value or nil
// pointer encodes to nil, and everything else becomes JSON without HTML escaping.
//
// A string type of your own is not a string here, so it becomes JSON with its quotes. Read
// a value back as the type it was written.
// Adding or removing those methods changes how a type built on a string or bytes encodes, so
// a flow in flight then reads its old value wrongly. Drain such flows first.
func Encode(value any) ([]byte, error) {
if value == nil {
return nil, nil
}

reflected := reflect.ValueOf(value)
if reflected.Kind() == reflect.Pointer && reflected.IsNil() {
return nil, nil
if reflected.Kind() == reflect.Pointer {
if reflected.IsNil() {
return nil, nil
}
typed, ok := value.(*any)
if ok {
return Encode(*typed)
}
reflected = reflected.Elem()
}

switch typed := value.(type) {
case []byte:
return typed, nil
case *[]byte:
return *typed, nil
case string:
return []byte(typed), nil
case *string:
return []byte(*typed), nil
case *any:
return Encode(*typed)
if usedAsIs(reflected.Type()) {
if reflected.Kind() == reflect.String {
return []byte(reflected.String()), nil
}
return reflected.Bytes(), nil
}

buffer := &bytes.Buffer{}
Expand All @@ -55,27 +83,30 @@ func Encode(value any) ([]byte, error) {
return bytes.TrimSuffix(buffer.Bytes(), []byte("\n")), nil
}

// Decode deserializes data produced by Encode into target. *string and *[]byte receive
// the bytes as they are, empty data zeroes target, and everything else is decoded as JSON.
// *any receives what the JSON holds, or a string when the data is not JSON.
// Decode deserializes data produced by Encode into target. A pointer to anything Encode uses
// as it is receives the bytes as they are, empty data zeroes target, and everything else is
// decoded as JSON. *any receives what the JSON holds, or a string when the data is not JSON.
func Decode(data []byte, target any) error {
if len(data) == 0 {
setToZeroValue(target)
return nil
}

switch typed := target.(type) {
case *string:
typed, ok := target.(*any)
if ok && typed != nil && !json.Valid(data) {
*typed = string(data)
return nil
case *[]byte:
*typed = bytes.Clone(data)
return nil
case *any:
if !json.Valid(data) {
*typed = string(data)
}

reflected := reflect.ValueOf(target)
if reflected.Kind() == reflect.Pointer && !reflected.IsNil() && usedAsIs(reflected.Type().Elem()) {
element := reflected.Elem()
if element.Kind() == reflect.String {
element.SetString(string(data))
return nil
}
element.SetBytes(bytes.Clone(data))
return nil
}
return json.Unmarshal(data, target)
}
Expand Down
120 changes: 120 additions & 0 deletions codec_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
package gush

import (
"encoding/json"
"reflect"
"strings"
"testing"

"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -71,6 +74,123 @@ func TestEncode(t *testing.T) {
})
}

type (
codecStatus string
codecBytes []byte
)

// codecTextStatus chooses its own JSON form through its text methods.
type codecTextStatus string

func (status codecTextStatus) MarshalText() ([]byte, error) {
return []byte("text:" + string(status)), nil
}

func (status *codecTextStatus) UnmarshalText(data []byte) error {
*status = codecTextStatus(strings.TrimPrefix(string(data), "text:"))
return nil
}

type (
codecMarshalsJSON string
codecUnmarshalsJSON string
codecMarshalsText string
codecUnmarshalsText string
)

func (codecMarshalsJSON) MarshalJSON() ([]byte, error) { return nil, nil }

func (*codecUnmarshalsJSON) UnmarshalJSON([]byte) error { return nil }

func (codecMarshalsText) MarshalText() ([]byte, error) { return nil, nil }

func (*codecUnmarshalsText) UnmarshalText([]byte) error { return nil }

func TestEncodeAndDecodeTypesBuiltOnStringsAndBytes(t *testing.T) {
t.Run("a string type has no quotes", func(t *testing.T) {
status := codecStatus("done")
for _, value := range []any{status, &status} {
raw, err := Encode(value)
require.NoError(t, err)
require.Equal(t, "done", string(raw))
}
})

t.Run("a string type reads back as itself or as a string", func(t *testing.T) {
raw, err := Encode(codecStatus("done"))
require.NoError(t, err)

var status codecStatus
require.NoError(t, Decode(raw, &status))
require.Equal(t, codecStatus("done"), status)

var plain string
require.NoError(t, Decode(raw, &plain))
require.Equal(t, "done", plain)
})

t.Run("a string reads back as a string type", func(t *testing.T) {
var status codecStatus
require.NoError(t, Decode([]byte("done"), &status))
require.Equal(t, codecStatus("done"), status)
})

t.Run("a bytes type is not base64", func(t *testing.T) {
original := codecBytes("raw bytes")
raw, err := Encode(original)
require.NoError(t, err)
require.Equal(t, "raw bytes", string(raw))

var decoded codecBytes
require.NoError(t, Decode(raw, &decoded))
require.Equal(t, original, decoded)

raw[0] = 'x'
require.Equal(t, byte('r'), decoded[0], "Decode must copy the data")
})

t.Run("a type with text methods stays JSON", func(t *testing.T) {
raw, err := Encode(codecTextStatus("done"))
require.NoError(t, err)
require.Equal(t, `"text:done"`, string(raw))

var status codecTextStatus
require.NoError(t, Decode(raw, &status))
require.Equal(t, codecTextStatus("done"), status)
})

t.Run("any one conversion method keeps a type JSON", func(t *testing.T) {
types := []reflect.Type{
reflect.TypeFor[codecMarshalsJSON](),
reflect.TypeFor[codecUnmarshalsJSON](),
reflect.TypeFor[codecMarshalsText](),
reflect.TypeFor[codecUnmarshalsText](),
}
for _, valueType := range types {
require.False(t, usedAsIs(valueType), "%v", valueType)
}
require.True(t, usedAsIs(reflect.TypeFor[codecStatus]()))
})

t.Run("json.RawMessage stays JSON", func(t *testing.T) {
raw, err := Encode(json.RawMessage(`{"a": 1}`))
require.NoError(t, err)
require.Equal(t, `{"a":1}`, string(raw))

var message json.RawMessage
require.NoError(t, Decode(raw, &message))
require.Equal(t, `{"a":1}`, string(message))
})

t.Run("a nil pointer target is an error", func(t *testing.T) {
var status *codecStatus
require.Error(t, Decode([]byte("done"), status))

var anything *any
require.Error(t, Decode([]byte("not json"), anything))
})
}

func TestDecode(t *testing.T) {
t.Run("[]byte assigned directly", func(t *testing.T) {
m := make([]byte, 0)
Expand Down
1 change: 1 addition & 0 deletions config_json_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func TestFlowConfigMarshalUsesSnakeCaseJSONTags(t *testing.T) {
expectedKeys := []string{
"id",
"timeout",
"error_if_running",
"priority",
"fair_key",
"initial_interval",
Expand Down
13 changes: 7 additions & 6 deletions env.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,14 @@ const (
envOTLPProtocol = "OTEL_EXPORTER_OTLP_PROTOCOL"
)

// firstEnv returns the first non-empty value among the named environment variables.
func firstEnv(names ...string) string {
for _, name := range names {
value := os.Getenv(name)
// firstEnv returns the first of the named environment variables that has a value, and that
// value.
func firstEnv(names ...string) (name, value string) {
for _, candidate := range names {
value := os.Getenv(candidate)
if value != "" {
return value
return candidate, value
}
}
return ""
return "", ""
}
8 changes: 8 additions & 0 deletions error.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,14 @@ func IsPermanentError(err error) bool {
return false
}

// IsFlowRunningError reports whether a flow did not start because one with the same ID is
// running. A child flow gets this error, and so does a client using WithErrorIfRunning. A
// step using such a client must check the error itself, because a flow cannot recognize it
// after it crosses the step.
func IsFlowRunningError(err error) bool {
return temporal.IsWorkflowExecutionAlreadyStartedError(err)
}

// IsPanicError reports whether err is a PanicError.
func IsPanicError(err error) bool {
return temporal.IsPanicError(err)
Expand Down
Loading