diff --git a/README.md b/README.md index 9a2ec9a..d08a023 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ err := ctx.RunFlow(request) Flows and steps are registered under their bare Go function names, and can be called by that name as a string. Two functions with the same name in different packages collide, and `Serve` reports it. Use `NewFlowHandler` and `NewStepHandler` to choose a name, which closures and the instantiations of a generic function need too. -`string` and `[]byte` travel as raw bytes and everything else as JSON, so read a value back as the type it was written. +Strings and byte slices, including types built on them, travel as raw bytes unless the type has its own JSON or text methods. Everything else travels as JSON, so read a value back as the type it was written. Adding or removing those methods changes how such a type encodes, so let flows carrying it finish first. ### Parallel steps and cleanup @@ -143,7 +143,7 @@ A step that is already running only finds out about the cancellation through `ct | Option | Applies to | Effect | |---|---|---| -| `WithMaxAttempts(n)` | both | Total attempts, including the first. Zero means unlimited. | +| `WithMaxAttempts(n)` | both | Total attempts, including the first. Zero means the default, and `gush.UnlimitedAttempts` keeps retrying. Any other negative number is an error. | | `WithInitialInterval(d)` | both | Delay before the first retry. | | `WithBackoffCoefficient(f)` | both | Multiplier on the delay after each retry. | | `WithMaxInterval(d)` | both | Cap on the delay between retries. | @@ -154,9 +154,10 @@ A step that is already running only finds out about the cancellation through `ct | `WithFairKey(key)` | both | Group to schedule fairly against, such as a tenant. | | `WithLogAttribute(k, v)` | both | Attribute on every log line of this run and its children. | | `WithLogAttributes(m)` | both | The same, for several attributes. | -| `WithID(id)` | flow | Flow ID. Otherwise one is generated, which `StartFlow` returns. A client's `RunFlow` or `StartFlow` joins a flow already running with this ID and ignores this run's input, and a child flow gets an error. | +| `WithID(id)` | flow | Flow ID. Otherwise one is generated, which `StartFlow` returns. A client's `RunFlow` or `StartFlow` joins a flow already running with this ID and ignores this run's flow, input and settings, so concurrent callers share one run. A child flow gets an error that `gush.IsFlowRunningError` recognizes. | +| `WithErrorIfRunning()` | flow | A client's `RunFlow` or `StartFlow` returns that same error instead of joining. | | `WithTimeout(d)` | flow | Limit on the whole execution, including every retry. | -| `WithConfig(c)` | flow | Starts from a `FlowConfig`, such as one decoded from a queue. Fields left out of the JSON keep their defaults. Whoever applies it chooses `RunFlow` or `StartFlow`. | +| `WithConfig(c)` | flow | Starts from a `FlowConfig`, such as one decoded from a queue. Settings left out of the JSON, or zero, use their defaults, and durations in JSON are integer nanoseconds. Whoever applies it chooses `RunFlow` or `StartFlow`. | | `WithQueue(q)` | step | Runs on another queue. | | `WithAttemptTimeout(d)` | step | Limit on each attempt, not the step as a whole. | | `WithHeartbeatTimeout(d)` | step | Fails an attempt that stops calling `ctx.Heartbeat()`. | @@ -227,14 +228,14 @@ err = client.Resume(ctx, resumeToken, decision, nil) **Set `WithAttemptTimeout` to the longest acceptable wait.** The step's timeout keeps running while it is suspended, and when it expires the step retries and runs its code again, sending a second request. The default is 2 minutes and 10 attempts. Use `WithMaxAttempts(1)` unless running the code twice is safe. -An error passed to `Resume` counts as a failed attempt, so a plain error runs the code again too. To reject for good, pass `gush.NewPermanentError("rejected", "Rejected", nil)`. +An error passed to `Resume` fails the step for good, so a rejection does not send the request again. To run the code again under the step's retry policy, pass a `gush.NewApplicationError`. A canceled error cancels the step if it is being canceled, and otherwise fails it for good too. ## Changing running flows Flows can run for a long time, and a replay must take the same path it took originally. `Version` keeps old executions on the old code. ```go -v := ctx.Version("payment-provider", 1) +v := ctx.Version("payment-provider", gush.DefaultFlowVersion, 1) if v == gush.DefaultFlowVersion { err = ctx.RunStep(OldCharge, input, &output) } else { @@ -242,7 +243,7 @@ if v == gush.DefaultFlowVersion { } ``` -`Version` accepts every recorded version, so once an old branch is deleted, an execution that recorded it runs the newer one. To make such an execution fail instead, call the SDK's `workflow.GetVersion` with a higher minimum. +The second argument is the oldest version the code still supports. Once no execution can reach an old branch, delete it and raise that minimum, here with `ctx.Version("payment-provider", 1, 1)`. An execution that recorded something older then stops making progress, with its flow tasks failing, until code that supports it is deployed. It never runs a branch it did not take. ## Long-running flows @@ -279,7 +280,7 @@ Lines written while Temporal replays a flow are dropped, so each appears once. H ### Export over OTLP -`gush.Dial` starts exporting when `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and `Close` flushes. A process that logs without a client calls `gush.InitLogExport()` itself. [OpenTelemetry's other variables](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) set the headers and TLS, and the protocol is `http/protobuf` unless they ask for `grpc`: +`gush.Dial` starts exporting when `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and `Close` flushes. A process that logs without a client calls `gush.InitLogExport()` itself. [OpenTelemetry's other variables](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) set the headers and TLS. The protocol is `http/protobuf` unless they ask for `grpc`, and any other protocol is an error: ```sh OTEL_EXPORTER_OTLP_ENDPOINT=https://collector:4317 diff --git a/codec.go b/codec.go index 39736b0..8c70130 100644 --- a/codec.go +++ b/codec.go @@ -2,6 +2,7 @@ package gush import ( "bytes" + "encoding" "encoding/base64" "encoding/json" "fmt" @@ -16,33 +17,60 @@ var ( zstdEncoder, _ = zstd.NewWriter(nil) ) +// conversionMethods are the interfaces through which a type chooses its own JSON form. +var conversionMethods = []reflect.Type{ + reflect.TypeFor[json.Marshaler](), + reflect.TypeFor[json.Unmarshaler](), + reflect.TypeFor[encoding.TextMarshaler](), + reflect.TypeFor[encoding.TextUnmarshaler](), +} + +// usedAsIs reports whether values of a type are their own encoding: strings and byte +// slices, including types built on them, unless the type chooses its own JSON form. +func usedAsIs(valueType reflect.Type) bool { + isBytes := valueType.Kind() == reflect.Slice && valueType.Elem() == reflect.TypeFor[byte]() + if valueType.Kind() != reflect.String && !isBytes { + return false + } + + pointer := reflect.PointerTo(valueType) + for _, methods := range conversionMethods { + if pointer.Implements(methods) { + return false + } + } + return true +} + // Encode serializes a value the way flows and steps do for their input and output. -// []byte and string are used as they are, including through a pointer, a nil value or -// nil pointer encodes to nil, and everything else becomes JSON without HTML escaping. +// Strings and byte slices are used as they are, including through a pointer and including +// types built on them, unless the type has its own JSON or text methods. A nil value or nil +// pointer encodes to nil, and everything else becomes JSON without HTML escaping. // -// A string type of your own is not a string here, so it becomes JSON with its quotes. Read -// a value back as the type it was written. +// Adding or removing those methods changes how a type built on a string or bytes encodes, so +// a flow in flight then reads its old value wrongly. Drain such flows first. func Encode(value any) ([]byte, error) { if value == nil { return nil, nil } reflected := reflect.ValueOf(value) - if reflected.Kind() == reflect.Pointer && reflected.IsNil() { - return nil, nil + if reflected.Kind() == reflect.Pointer { + if reflected.IsNil() { + return nil, nil + } + typed, ok := value.(*any) + if ok { + return Encode(*typed) + } + reflected = reflected.Elem() } - switch typed := value.(type) { - case []byte: - return typed, nil - case *[]byte: - return *typed, nil - case string: - return []byte(typed), nil - case *string: - return []byte(*typed), nil - case *any: - return Encode(*typed) + if usedAsIs(reflected.Type()) { + if reflected.Kind() == reflect.String { + return []byte(reflected.String()), nil + } + return reflected.Bytes(), nil } buffer := &bytes.Buffer{} @@ -55,27 +83,30 @@ func Encode(value any) ([]byte, error) { return bytes.TrimSuffix(buffer.Bytes(), []byte("\n")), nil } -// Decode deserializes data produced by Encode into target. *string and *[]byte receive -// the bytes as they are, empty data zeroes target, and everything else is decoded as JSON. -// *any receives what the JSON holds, or a string when the data is not JSON. +// Decode deserializes data produced by Encode into target. A pointer to anything Encode uses +// as it is receives the bytes as they are, empty data zeroes target, and everything else is +// decoded as JSON. *any receives what the JSON holds, or a string when the data is not JSON. func Decode(data []byte, target any) error { if len(data) == 0 { setToZeroValue(target) return nil } - switch typed := target.(type) { - case *string: + typed, ok := target.(*any) + if ok && typed != nil && !json.Valid(data) { *typed = string(data) return nil - case *[]byte: - *typed = bytes.Clone(data) - return nil - case *any: - if !json.Valid(data) { - *typed = string(data) + } + + reflected := reflect.ValueOf(target) + if reflected.Kind() == reflect.Pointer && !reflected.IsNil() && usedAsIs(reflected.Type().Elem()) { + element := reflected.Elem() + if element.Kind() == reflect.String { + element.SetString(string(data)) return nil } + element.SetBytes(bytes.Clone(data)) + return nil } return json.Unmarshal(data, target) } diff --git a/codec_test.go b/codec_test.go index e942caf..4f8d345 100644 --- a/codec_test.go +++ b/codec_test.go @@ -1,6 +1,9 @@ package gush import ( + "encoding/json" + "reflect" + "strings" "testing" "github.com/stretchr/testify/require" @@ -71,6 +74,123 @@ func TestEncode(t *testing.T) { }) } +type ( + codecStatus string + codecBytes []byte +) + +// codecTextStatus chooses its own JSON form through its text methods. +type codecTextStatus string + +func (status codecTextStatus) MarshalText() ([]byte, error) { + return []byte("text:" + string(status)), nil +} + +func (status *codecTextStatus) UnmarshalText(data []byte) error { + *status = codecTextStatus(strings.TrimPrefix(string(data), "text:")) + return nil +} + +type ( + codecMarshalsJSON string + codecUnmarshalsJSON string + codecMarshalsText string + codecUnmarshalsText string +) + +func (codecMarshalsJSON) MarshalJSON() ([]byte, error) { return nil, nil } + +func (*codecUnmarshalsJSON) UnmarshalJSON([]byte) error { return nil } + +func (codecMarshalsText) MarshalText() ([]byte, error) { return nil, nil } + +func (*codecUnmarshalsText) UnmarshalText([]byte) error { return nil } + +func TestEncodeAndDecodeTypesBuiltOnStringsAndBytes(t *testing.T) { + t.Run("a string type has no quotes", func(t *testing.T) { + status := codecStatus("done") + for _, value := range []any{status, &status} { + raw, err := Encode(value) + require.NoError(t, err) + require.Equal(t, "done", string(raw)) + } + }) + + t.Run("a string type reads back as itself or as a string", func(t *testing.T) { + raw, err := Encode(codecStatus("done")) + require.NoError(t, err) + + var status codecStatus + require.NoError(t, Decode(raw, &status)) + require.Equal(t, codecStatus("done"), status) + + var plain string + require.NoError(t, Decode(raw, &plain)) + require.Equal(t, "done", plain) + }) + + t.Run("a string reads back as a string type", func(t *testing.T) { + var status codecStatus + require.NoError(t, Decode([]byte("done"), &status)) + require.Equal(t, codecStatus("done"), status) + }) + + t.Run("a bytes type is not base64", func(t *testing.T) { + original := codecBytes("raw bytes") + raw, err := Encode(original) + require.NoError(t, err) + require.Equal(t, "raw bytes", string(raw)) + + var decoded codecBytes + require.NoError(t, Decode(raw, &decoded)) + require.Equal(t, original, decoded) + + raw[0] = 'x' + require.Equal(t, byte('r'), decoded[0], "Decode must copy the data") + }) + + t.Run("a type with text methods stays JSON", func(t *testing.T) { + raw, err := Encode(codecTextStatus("done")) + require.NoError(t, err) + require.Equal(t, `"text:done"`, string(raw)) + + var status codecTextStatus + require.NoError(t, Decode(raw, &status)) + require.Equal(t, codecTextStatus("done"), status) + }) + + t.Run("any one conversion method keeps a type JSON", func(t *testing.T) { + types := []reflect.Type{ + reflect.TypeFor[codecMarshalsJSON](), + reflect.TypeFor[codecUnmarshalsJSON](), + reflect.TypeFor[codecMarshalsText](), + reflect.TypeFor[codecUnmarshalsText](), + } + for _, valueType := range types { + require.False(t, usedAsIs(valueType), "%v", valueType) + } + require.True(t, usedAsIs(reflect.TypeFor[codecStatus]())) + }) + + t.Run("json.RawMessage stays JSON", func(t *testing.T) { + raw, err := Encode(json.RawMessage(`{"a": 1}`)) + require.NoError(t, err) + require.Equal(t, `{"a":1}`, string(raw)) + + var message json.RawMessage + require.NoError(t, Decode(raw, &message)) + require.Equal(t, `{"a":1}`, string(message)) + }) + + t.Run("a nil pointer target is an error", func(t *testing.T) { + var status *codecStatus + require.Error(t, Decode([]byte("done"), status)) + + var anything *any + require.Error(t, Decode([]byte("not json"), anything)) + }) +} + func TestDecode(t *testing.T) { t.Run("[]byte assigned directly", func(t *testing.T) { m := make([]byte, 0) diff --git a/config_json_test.go b/config_json_test.go index 5945a92..63cc6fc 100644 --- a/config_json_test.go +++ b/config_json_test.go @@ -34,6 +34,7 @@ func TestFlowConfigMarshalUsesSnakeCaseJSONTags(t *testing.T) { expectedKeys := []string{ "id", "timeout", + "error_if_running", "priority", "fair_key", "initial_interval", diff --git a/env.go b/env.go index 8d7f1f2..5224fdb 100644 --- a/env.go +++ b/env.go @@ -12,13 +12,14 @@ const ( envOTLPProtocol = "OTEL_EXPORTER_OTLP_PROTOCOL" ) -// firstEnv returns the first non-empty value among the named environment variables. -func firstEnv(names ...string) string { - for _, name := range names { - value := os.Getenv(name) +// firstEnv returns the first of the named environment variables that has a value, and that +// value. +func firstEnv(names ...string) (name, value string) { + for _, candidate := range names { + value := os.Getenv(candidate) if value != "" { - return value + return candidate, value } } - return "" + return "", "" } diff --git a/error.go b/error.go index 85014f7..8b49a69 100644 --- a/error.go +++ b/error.go @@ -40,6 +40,14 @@ func IsPermanentError(err error) bool { return false } +// IsFlowRunningError reports whether a flow did not start because one with the same ID is +// running. A child flow gets this error, and so does a client using WithErrorIfRunning. A +// step using such a client must check the error itself, because a flow cannot recognize it +// after it crosses the step. +func IsFlowRunningError(err error) bool { + return temporal.IsWorkflowExecutionAlreadyStartedError(err) +} + // IsPanicError reports whether err is a PanicError. func IsPanicError(err error) bool { return temporal.IsPanicError(err) diff --git a/error_test.go b/error_test.go index 41b9016..24c3531 100644 --- a/error_test.go +++ b/error_test.go @@ -1,11 +1,14 @@ package gush import ( + "context" "errors" "fmt" + "strings" "testing" "go.temporal.io/api/enums/v1" + "go.temporal.io/api/serviceerror" "go.temporal.io/sdk/temporal" ) @@ -279,3 +282,124 @@ func TestAbortKeepsTheMessageReadable(t *testing.T) { }) } } + +func TestIsFlowRunningErrorIgnoresOtherErrors(t *testing.T) { + for _, err := range []error{nil, errors.New("other"), NewApplicationError("other", "Other")} { + if IsFlowRunningError(err) { + t.Errorf("IsFlowRunningError(%v) = true", err) + } + } +} + +func TestResumeError(t *testing.T) { + t.Run("no error stays nil", func(t *testing.T) { + if err := resumeError(nil); err != nil { + t.Errorf("resumeError(nil) = %v", err) + } + }) + + t.Run("a plain error ends the step", func(t *testing.T) { + got := resumeError(errors.New("rejected")) + + if !IsPermanentError(got) { + t.Errorf("resumeError() = %v, want a permanent error", got) + } + if got.Error() == "" || !strings.Contains(got.Error(), "rejected") { + t.Errorf("resumeError() lost the message: %v", got) + } + }) + + t.Run("an application error keeps its retryability", func(t *testing.T) { + retry := NewApplicationError("ask again", "AskAgain") + if got := resumeError(retry); !IsApplicationError(got) || IsPermanentError(got) { + t.Errorf("resumeError(retryable) = %v", got) + } + + wrapped := fmt.Errorf("reviewer: %w", retry) + if got := resumeError(wrapped); !IsApplicationError(got) || IsPermanentError(got) { + t.Errorf("resumeError(wrapped retryable) = %v", got) + } + + permanent := NewPermanentError("no", "No", nil) + if got := resumeError(permanent); !IsPermanentError(got) { + t.Errorf("resumeError(permanent) = %v", got) + } + }) + + t.Run("a canceled error asks to cancel the step", func(t *testing.T) { + canceled := []error{temporal.NewCanceledError(), context.Canceled, fmt.Errorf("review: %w", context.Canceled)} + for _, err := range canceled { + if !isCancellation(err) { + t.Errorf("isCancellation(%v) = false", err) + } + } + other := []error{nil, errors.New("rejected"), NewApplicationError("retry", "Retry")} + for _, err := range other { + if isCancellation(err) { + t.Errorf("isCancellation(%v) = true", err) + } + } + }) +} + +func TestResumeErrorDropsACanceledCause(t *testing.T) { + permanent := NewPermanentError("rejected", "Rejected", context.Canceled) + wrapped := fmt.Errorf("review: %w", permanent) + options := temporal.ApplicationErrorOptions{Cause: context.Canceled} + retryable := temporal.NewApplicationErrorWithOptions("retry", "Retry", options) + tests := map[string]struct { + err error + message string + permanent bool + }{ + "permanent": {permanent, "rejected", true}, + "wrapped": {wrapped, "review: rejected", true}, + "retryable": {retryable, "retry", false}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + got := resumeError(test.err) + + if errors.Is(got, context.Canceled) || temporal.IsCanceledError(got) { + t.Errorf("resumeError() = %v still reads as canceled", got) + } + var appErr *ApplicationError + if !errors.As(got, &appErr) { + t.Fatalf("resumeError() = %v", got) + } + if appErr.Message() != test.message || appErr.NonRetryable() != test.permanent { + t.Errorf("message = %q, permanent = %v", appErr.Message(), appErr.NonRetryable()) + } + }) + } +} + +func TestResumeErrorKeepsAnotherCause(t *testing.T) { + cause := errors.New("reviewer unavailable") + permanent := NewPermanentError("rejected", "Rejected", cause) + + if got := resumeError(permanent); !errors.Is(got, cause) { + t.Errorf("resumeError() = %v lost its cause", got) + } +} + +func TestIsFlowRunningError(t *testing.T) { + started := serviceerror.NewWorkflowExecutionAlreadyStarted("running", "", "") + if !IsFlowRunningError(started) { + t.Errorf("IsFlowRunningError(%v) = false", started) + } + wrapped := fmt.Errorf("start: %w", started) + if !IsFlowRunningError(wrapped) { + t.Errorf("IsFlowRunningError(%v) = false", wrapped) + } + + // What a flow sees after the error crosses a step, or when it comes from a flow that ran. + acrossAStep := NewApplicationError("running", "WorkflowExecutionAlreadyStarted") + if IsFlowRunningError(acrossAStep) { + t.Errorf("IsFlowRunningError(%v) = true", acrossAStep) + } + if IsFlowRunningError(errors.New("plain")) { + t.Error("IsFlowRunningError(plain) = true") + } +} diff --git a/flow.go b/flow.go index 7a64883..f0053d4 100644 --- a/flow.go +++ b/flow.go @@ -99,6 +99,11 @@ func (c *Client) startFlow(ctx context.Context, request FlowRequest, config *Flo return nil, err } + err = config.validate() + if err != nil { + return nil, err + } + startOptions := newStartWorkflowOptions(request.Queue, config) return c.temporalClient.ExecuteWorkflow(ctx, startOptions, flowName, payload, config.LogAttributes) } diff --git a/flow_config.go b/flow_config.go index 0ca2e92..51c72ae 100644 --- a/flow_config.go +++ b/flow_config.go @@ -1,22 +1,16 @@ package gush import ( - "bytes" - "encoding/json" "maps" - "reflect" "slices" "time" ) // FlowConfig holds the settings of one flow run. Its JSON form is stable, so a config can -// be queued and applied later with WithConfig. A field left out of the JSON keeps its -// default, while a zero written out is taken as it is, so "max_attempts": 0 still -// means unlimited. Waiting is not part of a config: whoever applies it chooses RunFlow or +// be queued and applied later with WithConfig. A setting left out of the JSON, or zero, +// uses its default, "max_attempts": -1 means unlimited, and durations are integer +// nanoseconds. Waiting is not part of a config: whoever applies it chooses RunFlow or // StartFlow. -// -// Like time.Time, it decodes itself from JSON, so a struct that embeds it would decode -// through that method and lose its own fields. Hold it in a named field to decode it. type FlowConfig struct { CommonConfig @@ -25,6 +19,10 @@ type FlowConfig struct { // Timeout limits the whole execution, including every retry. Zero means unlimited. Timeout time.Duration `json:"timeout"` + + // ErrorIfRunning makes a client's RunFlow or StartFlow fail rather than join a flow + // already running with this ID. See WithErrorIfRunning. + ErrorIfRunning bool `json:"error_if_running"` } // DefaultFlowConfig returns the settings a flow run starts from: one attempt, medium @@ -34,36 +32,12 @@ func DefaultFlowConfig() *FlowConfig { config.InitialInterval = 1 * time.Second config.BackoffCoefficient = 2.0 config.MaxInterval = 100 * time.Second - config.MaxAttempts = 1 + config.MaxAttempts = defaultFlowAttempts config.Priority = PriorityMedium config.LogAttributes = make(map[string]any) return config } -// UnmarshalJSON starts an empty config from DefaultFlowConfig, because an attempt count left -// out of the JSON would otherwise read as zero, which means unlimited retries. Priority -// stays empty unless the JSON sets it, so that WithConfig keeps what a child flow inherits. -// A config that already holds settings keeps those the JSON leaves out. -func (config *FlowConfig) UnmarshalJSON(data []byte) error { - type plainFlowConfig FlowConfig - decoded := plainFlowConfig(*config) - if reflect.ValueOf(*config).IsZero() { - decoded = plainFlowConfig(*DefaultFlowConfig()) - decoded.Priority = "" - } - - decoder := json.NewDecoder(bytes.NewReader(data)) - decoder.UseNumber() - err := decoder.Decode(&decoded) - if err != nil { - return err - } - resolveNumbers(decoded.LogAttributes) - - *config = FlowConfig(decoded) - return nil -} - // Apply applies options to config in order. func (config *FlowConfig) Apply(options ...FlowOption) { for _, option := range options { @@ -105,7 +79,10 @@ func WithConfig(config *FlowConfig) FlowOption { // WithID sets the flow ID. Without it one is generated, which StartFlow returns. While a // flow with this ID is running, a client's RunFlow or StartFlow joins it instead of starting -// another and ignores this run's input, and a child flow gets an error. +// another and ignores this run's flow, input and settings, so concurrent callers can share +// one run. RunFlow then returns that flow's result. Use +// WithErrorIfRunning to be told instead. A child flow always gets an error, because a flow +// cannot wait for a flow it did not start; check it with IsFlowRunningError. func WithID(flowID string) FlowOption { apply := func(config *FlowConfig) { config.ID = flowID @@ -113,6 +90,15 @@ func WithID(flowID string) FlowOption { return flowOption(apply) } +// WithErrorIfRunning makes a client's RunFlow or StartFlow return an error, which +// IsFlowRunningError recognizes, rather than join a flow already running with this ID. +func WithErrorIfRunning() FlowOption { + apply := func(config *FlowConfig) { + config.ErrorIfRunning = true + } + return flowOption(apply) +} + // WithTimeout limits the whole flow execution, including every retry. func WithTimeout(timeout time.Duration) FlowOption { apply := func(config *FlowConfig) { diff --git a/flow_config_test.go b/flow_config_test.go index 705daaf..fd243ab 100644 --- a/flow_config_test.go +++ b/flow_config_test.go @@ -243,26 +243,28 @@ func decodedConfig(t *testing.T, raw string) *gush.FlowConfig { return config } -func TestPartialJSONConfigKeepsTheDefaults(t *testing.T) { - config := decodedConfig(t, `{"id":"flow-1","priority":"high"}`) - defaults := gush.DefaultFlowConfig() +// A struct holding a config beside its own fields, such as a queued message, decodes both. +func TestAStructEmbeddingAConfigDecodesItsOwnFields(t *testing.T) { + type queuedFlow struct { + gush.FlowConfig + Tenant string `json:"tenant"` + } + raw := `{"id":"flow-1","max_attempts":3,"tenant":"acme","log_attributes":{"user_id":9007199254740993}}` - if config.ID != "flow-1" || config.Priority != gush.PriorityHigh { - t.Errorf("set fields = %q, %q", config.ID, config.Priority) + var queued queuedFlow + err := gush.Decode([]byte(raw), &queued) + if err != nil { + t.Fatalf("Decode() error = %v", err) } - if config.MaxAttempts != defaults.MaxAttempts { - t.Errorf("attempts = %d, want the default %d", config.MaxAttempts, defaults.MaxAttempts) + + if queued.Tenant != "acme" { + t.Errorf("tenant = %q", queued.Tenant) } - if config.InitialInterval != defaults.InitialInterval || config.MaxInterval != defaults.MaxInterval { - t.Errorf("intervals = %v, %v", config.InitialInterval, config.MaxInterval) + if queued.ID != "flow-1" || queued.MaxAttempts != 3 { + t.Errorf("config = %q, %d", queued.ID, queued.MaxAttempts) } -} - -func TestJSONConfigTakesAWrittenZero(t *testing.T) { - config := decodedConfig(t, `{"max_attempts":0}`) - - if config.MaxAttempts != 0 { - t.Errorf("attempts = %d, want the written 0, which means unlimited", config.MaxAttempts) + if queued.LogAttributes["user_id"] != int64(9007199254740993) { + t.Errorf("user_id = %v (%T)", queued.LogAttributes["user_id"], queued.LogAttributes["user_id"]) } } @@ -328,6 +330,20 @@ func TestWithConfigOverridesWhatItSets(t *testing.T) { } } +func TestJSONConfigMergesLogAttributes(t *testing.T) { + config := gush.DefaultFlowConfig() + config.LogAttributes["kept"] = "value" + + err := gush.Decode([]byte(`{"log_attributes":{"added":1}}`), config) + if err != nil { + t.Fatalf("Decode() error = %v", err) + } + + if config.LogAttributes["kept"] != "value" || config.LogAttributes["added"] != int64(1) { + t.Errorf("attributes = %v", config.LogAttributes) + } +} + func TestJSONConfigDecodesOverExistingSettings(t *testing.T) { config := gush.DefaultFlowConfig() config.ID = "kept" diff --git a/flow_context.go b/flow_context.go index 6ad9d0f..a2bef17 100644 --- a/flow_context.go +++ b/flow_context.go @@ -10,8 +10,8 @@ import ( // abortedFlowErrorType is the ApplicationError type of a flow's Abort. const abortedFlowErrorType = "FlowAborted" -// DefaultFlowVersion is what Version returns for a flow that started before the change -// it guards. +// DefaultFlowVersion is what Version returns for an execution that passed the call before +// it was added. const DefaultFlowVersion = int(workflow.DefaultVersion) // FlowContext is what a flow receives. Its Context field is the Temporal SDK's own context, @@ -111,6 +111,10 @@ func (flowCtx *FlowContext) RunStep(step any, input, output any, options ...Step if err != nil { return flowCtx.Abort(err) } + err = config.validate() + if err != nil { + return flowCtx.Abort(err) + } payload, err := EncodePayload(input) if err != nil { return flowCtx.Abort(err) @@ -136,6 +140,10 @@ func (flowCtx *FlowContext) RunStep(step any, input, output any, options ...Step // priority, fair key and log attributes unless options override them. func (flowCtx *FlowContext) RunFlow(request FlowRequest, options ...FlowOption) error { config := flowCtx.childFlowConfig(options) + err := config.validate() + if err != nil { + return flowCtx.Abort(err) + } childOptions := newChildWorkflowOptions(request.Queue, config) childContext := workflow.WithChildOptions(flowCtx.Context, childOptions) @@ -168,6 +176,10 @@ func (flowCtx *FlowContext) StartFlow(request FlowRequest, options ...FlowOption return "", err } config := flowCtx.childFlowConfig(options) + err = config.validate() + if err != nil { + return "", flowCtx.Abort(err) + } // Canceling a context also cancels the children started on it. detached, _ := workflow.NewDisconnectedContext(flowCtx.Context) @@ -209,12 +221,13 @@ func executeChildFlow(ctx workflow.Context, request FlowRequest, config *FlowCon // Version lets flow code change without breaking executions already running the old code. // The first execution of a call records maxVersion in the flow's history, and replays -// return that recorded value. Flows that started before the call existed get -// DefaultFlowVersion. +// return that recorded value. An execution that passed this point before the call existed +// gets DefaultFlowVersion, and one that reaches it later records maxVersion however long ago +// it started. // // To replace a step: // -// v := ctx.Version("payment-provider", 1) +// v := ctx.Version("payment-provider", gush.DefaultFlowVersion, 1) // if v == gush.DefaultFlowVersion { // err = ctx.RunStep(OldCharge, input, &output) // } else { @@ -222,12 +235,12 @@ func executeChildFlow(ctx workflow.Context, request FlowRequest, config *FlowCon // } // // To change the same code again, keep the tag, raise maxVersion and add a branch. Once no -// execution can reach an old branch, delete it but keep the Version call. Version accepts -// every recorded version, so a stale execution would then run the newer branch. To make one -// fail instead, call the SDK's workflow.GetVersion with a higher minimum. -func (flowCtx *FlowContext) Version(tag string, maxVersion int) int { - minSupported := workflow.Version(DefaultFlowVersion) - version := workflow.GetVersion(flowCtx.Context, tag, minSupported, workflow.Version(maxVersion)) +// execution can reach an old branch, delete it, keep the Version call and raise minVersion +// to the oldest version left. An execution that recorded an older version then stops making +// progress, with its flow tasks failing, until code that supports it is deployed. It never +// runs a branch it did not take. +func (flowCtx *FlowContext) Version(tag string, minVersion, maxVersion int) int { + version := workflow.GetVersion(flowCtx.Context, tag, workflow.Version(minVersion), workflow.Version(maxVersion)) return int(version) } diff --git a/flow_test.go b/flow_test.go index a05c5bc..0b3f9d5 100644 --- a/flow_test.go +++ b/flow_test.go @@ -35,6 +35,41 @@ func TestRunFlowSendsNoRetryTypesToTemporal(t *testing.T) { } } +func TestJSONConfigRunsWithTheAttemptsItAsksFor(t *testing.T) { + tests := map[string]struct { + raw string + expected int32 + }{ + "left out": {raw: `{"id":"flow-1"}`, expected: 1}, + "zero": {raw: `{"max_attempts":0}`, expected: 1}, + "a number": {raw: `{"max_attempts":3}`, expected: 3}, + "unlimited": {raw: `{"max_attempts":-1}`, expected: 0}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + config := &FlowConfig{} + err := Decode([]byte(test.raw), config) + if err != nil { + t.Fatalf("Decode() error = %v", err) + } + + capture := &startCaptureClient{} + flowClient := &Client{temporalClient: capture} + request := FlowRequest{Queue: "queue", Flow: "handler"} + err = flowClient.RunFlow(context.Background(), request, WithConfig(config)) + if !errors.Is(err, errStartCaptured) { + t.Fatalf("RunFlow() error = %v", err) + } + + got := capture.options.RetryPolicy.MaximumAttempts + if got != test.expected { + t.Errorf("attempts sent to Temporal = %d, want %d", got, test.expected) + } + }) + } +} + func TestStartFlowRejectsAFlowThatCannotRun(t *testing.T) { flowClient := &Client{temporalClient: &startCaptureClient{}} diff --git a/function_name.go b/function_name.go index 1cf1ded..dd975ad 100644 --- a/function_name.go +++ b/function_name.go @@ -13,18 +13,17 @@ var typeArguments = regexp.MustCompile(`\[[^\]]*\]`) // functionName returns the name a flow or step function registers under, which is its // bare Go name without package, receiver or type arguments, so every instantiation of a -// generic function shares one name. A string is its own name, and anything that is not a -// function has no name. +// generic function shares one name. A string, or a type built on one, is its own name, and +// anything that is not a function has no name. func functionName(function any) string { - name, ok := function.(string) - if ok { - return name - } if function == nil { return "" } value := reflect.ValueOf(function) + if value.Kind() == reflect.String { + return value.String() + } if value.Kind() != reflect.Func || value.Pointer() == 0 { return "" } diff --git a/function_name_test.go b/function_name_test.go index 4239066..33e6764 100644 --- a/function_name_test.go +++ b/function_name_test.go @@ -46,6 +46,14 @@ type genericTestHandlers[T any] struct{} func (genericTestHandlers[T]) Run(*StepContext) (any, error) { return nil, nil } +type flowNameForTest string + +func TestFunctionNameOfATypeBuiltOnAString(t *testing.T) { + if got := functionName(flowNameForTest("Named")); got != "Named" { + t.Errorf("functionName() = %q", got) + } +} + func TestFunctionNameOfGenericFunctions(t *testing.T) { if name := functionName(genericTestStep[int]); name != "genericTestStep" { t.Errorf("generic function = %q", name) diff --git a/integration_test.go b/integration_test.go index 0431672..def55b0 100644 --- a/integration_test.go +++ b/integration_test.go @@ -21,6 +21,8 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" logcollector "go.opentelemetry.io/proto/otlp/collector/logs/v1" + "go.temporal.io/api/enums/v1" + "go.temporal.io/sdk/client" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/testsuite" "go.temporal.io/sdk/worker" @@ -81,13 +83,14 @@ func runWithTemporal(m *testing.M) int { serveContext, stopServing := context.WithCancel(context.Background()) testWorker := gush.NewWorker(testQueue). - WithFlows(Double, FailingFlow, ParentOfFailingFlow, FailingStepFlow, PlainFailureFlow, AbortFlow, ApprovalFlow, + WithFlows(CancelableApprovalFlow, InvalidStepAttemptsFlow, InvalidChildAttemptsFlow, InvalidStartAttemptsFlow, + HighInitialIntervalFlow, Double, FailingFlow, ParentOfFailingFlow, FailingStepFlow, PlainFailureFlow, AbortFlow, ApprovalFlow, ExpiringApprovalFlow, TimeoutFlow, HeartbeatFlow, SilentFlow, ReplayFlow, AttributesFlow, AttributesChildFlow, WaitingFlow, InheritanceFlow, VersionFlow, SmallFlow, BadStepFlow, ChildOnOwnQueueFlow, WrappedAbortFlow, WrappingFlow, ParallelFlow, CleanupFlow, CountdownFlow, AttributeEchoFlow, CancelOneStepFlow, ChargeThenRefundFlow, DetachingParentFlow, DetailReadingFlow, AttributeRecordingFlow, InheritingDetachingParentFlow, CanceledDetachingFlow, HoldingEchoFlow, InvalidChildStartFlow, DuplicateChildStartFlow, - IdleFlow, IdleDetachingParentFlow, UnpolledChildStartFlow). + IdleFlow, IdleDetachingParentFlow, UnpolledChildStartFlow, RetriableApprovalFlow, RaisedVersionFlow, ChildIDInUseFlow). WithSteps(DoubleStep, FailingStep, PlainFailureStep, AbortStep, ApprovalStep, SlowStep, HeartbeatStep, SilentStep, PurgeCacheStep, AttributesStep, InheritanceStep, WrappedAbortStep, DecliningStep, CleanupStep, AttributeEchoStep, ChargingStep, RefundStep, DetailedStep). WithContext(serveContext) @@ -384,6 +387,16 @@ func ApprovalFlow(ctx *gush.FlowContext) (any, error) { return decision, err } +func RetriableApprovalFlow(ctx *gush.FlowContext) (any, error) { + var decision string + err := ctx.RunStep(ApprovalStep, string(ctx.Input), &decision, + gush.WithAttemptTimeout(30*time.Second), + gush.WithInitialInterval(time.Millisecond), + gush.WithMaxAttempts(2), + ) + return decision, err +} + func TestSuspendedStepFinishesWithResume(t *testing.T) { ctx := testContext(t) key := uniqueKey(t) @@ -645,7 +658,7 @@ func TestOverlongFairKeyIsCutToFit(t *testing.T) { } func VersionFlow(ctx *gush.FlowContext) (any, error) { - return ctx.Version("change", 2), nil + return ctx.Version("change", gush.DefaultFlowVersion, 2), nil } func TestVersionReturnsTheCurrentVersionForNewFlows(t *testing.T) { @@ -657,6 +670,59 @@ func TestVersionReturnsTheCurrentVersionForNewFlows(t *testing.T) { assert.Equal(t, 2, output) } +// RaisedVersionFlow records version 1 and then requires 2, as code does after deleting the +// branch an execution already took. +func RaisedVersionFlow(ctx *gush.FlowContext) (any, error) { + ctx.Version("raised", gush.DefaultFlowVersion, 1) + return ctx.Version("raised", 2, 2), nil +} + +func TestVersionStopsAnExecutionBelowTheMinimum(t *testing.T) { + ctx := testContext(t) + flowID := uniqueKey(t) + _, err := testClient.StartFlow(ctx, flowRequest(RaisedVersionFlow, nil, nil), gush.WithID(flowID)) + require.NoError(t, err) + defer func() { _ = testClient.TerminateFlow(ctx, flowID, "test") }() + + waiting, cancel := context.WithTimeout(ctx, 3*time.Second) + defer cancel() + err = testClient.RunFlow(waiting, flowRequest(RaisedVersionFlow, nil, nil), gush.WithID(flowID)) + + require.Error(t, err) + require.ErrorIs(t, waiting.Err(), context.DeadlineExceeded, "the flow ended instead of stopping: %v", err) + flow, err := testClient.GetFlow(ctx, flowID) + require.NoError(t, err) + assert.Equal(t, "Running", flow.Status) + assert.True(t, flowTaskFailed(t, flowID, "TMPRL1100"), "the flow is not stopped by its version") +} + +// dialTemporal returns the SDK's own client, for what gush does not expose. +func dialTemporal(t *testing.T) client.Client { + t.Helper() + options := client.Options{HostPort: testHostPort} + temporalClient, err := client.Dial(options) + require.NoError(t, err) + t.Cleanup(temporalClient.Close) + return temporalClient +} + +// flowTaskFailed reports whether a flow's history has a failed flow task with this text. +func flowTaskFailed(t *testing.T, flowID, text string) bool { + t.Helper() + temporalClient := dialTemporal(t) + + events := temporalClient.GetWorkflowHistory(testContext(t), flowID, "", false, enums.HISTORY_EVENT_FILTER_TYPE_ALL_EVENT) + for events.HasNext() { + event, err := events.Next() + require.NoError(t, err) + failed := event.GetWorkflowTaskFailedEventAttributes() + if failed != nil && strings.Contains(failed.GetFailure().GetMessage(), text) { + return true + } + } + return false +} + func SmallFlow(*gush.FlowContext) (any, error) { return "done", nil } @@ -1207,6 +1273,60 @@ func TestStartFlowJoinsARunningFlow(t *testing.T) { assert.Equal(t, "first", output) } +func TestWithErrorIfRunningRefusesARunningFlow(t *testing.T) { + ctx := testContext(t) + flowID := uniqueKey(t) + + _, err := testClient.StartFlow(ctx, flowRequest(HoldingEchoFlow, "first", nil), gush.WithID(flowID)) + require.NoError(t, err) + + _, err = testClient.StartFlow(ctx, flowRequest(HoldingEchoFlow, "second", nil), + gush.WithID(flowID), gush.WithErrorIfRunning()) + assert.True(t, gush.IsFlowRunningError(err), "%v", err) + + err = testClient.RunFlow(ctx, flowRequest(HoldingEchoFlow, "third", nil), + gush.WithID(flowID), gush.WithErrorIfRunning()) + assert.True(t, gush.IsFlowRunningError(err), "%v", err) + + var output string + err = testClient.RunFlow(ctx, flowRequest(HoldingEchoFlow, "fourth", &output), gush.WithID(flowID)) + require.NoError(t, err) + assert.Equal(t, "first", output) + + _, err = testClient.StartFlow(ctx, flowRequest(SmallFlow, nil, nil), + gush.WithID(flowID), gush.WithErrorIfRunning()) + assert.NoError(t, err, "a flow that has finished does not count as running") +} + +func ChildIDInUseFlow(ctx *gush.FlowContext) (any, error) { + var childID string + err := ctx.Decode(&childID) + if err != nil { + return nil, err + } + + request := gush.FlowRequest{Flow: WaitingFlow} + runErr := ctx.RunFlow(request, gush.WithID(childID)) + _, startErr := ctx.StartFlow(request, gush.WithID(childID)) + recognized := []bool{gush.IsFlowRunningError(runErr), gush.IsFlowRunningError(startErr)} + return recognized, nil +} + +func TestAChildReusingARunningIDGetsTheSameError(t *testing.T) { + ctx := testContext(t) + childID := uniqueKey(t) + + _, err := testClient.StartFlow(ctx, flowRequest(IdleFlow, nil, nil), gush.WithID(childID)) + require.NoError(t, err) + defer func() { _ = testClient.TerminateFlow(ctx, childID, "test") }() + + var recognized []bool + err = testClient.RunFlow(ctx, flowRequest(ChildIDInUseFlow, childID, &recognized)) + + require.NoError(t, err) + assert.Equal(t, []bool{true, true}, recognized) +} + func InvalidChildStartFlow(ctx *gush.FlowContext) (any, error) { request := gush.FlowRequest{Flow: 123} return ctx.StartFlow(request) @@ -1468,6 +1588,43 @@ func TestWaitForCancelHoldsCleanupUntilTheStepEnds(t *testing.T) { assert.EqualValues(t, 1, attemptsOf(key+"-seen").Load(), "the refund started before the charge ended") } +func TestResumeWithAPlainErrorEndsTheStep(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + + result := make(chan error, 1) + go func() { result <- testClient.RunFlow(ctx, flowRequest(RetriableApprovalFlow, key, nil)) }() + + token := <-resumeTokens + require.NoError(t, testClient.Resume(ctx, token, nil, errors.New("rejected by reviewer"))) + + err := <-result + require.Error(t, err) + assert.True(t, gush.IsPermanentError(err), "%v", err) + assert.Equal(t, "[StepAborted] rejected by reviewer", err.Error()) + assert.EqualValues(t, 1, attemptsOf(key).Load()) +} + +func TestResumeWithAnApplicationErrorRunsTheStepAgain(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + + var decision string + result := make(chan error, 1) + go func() { result <- testClient.RunFlow(ctx, flowRequest(RetriableApprovalFlow, key, &decision)) }() + + token := <-resumeTokens + retry := gush.NewApplicationError("ask again", "AskAgain") + require.NoError(t, testClient.Resume(ctx, token, nil, retry)) + + token = <-resumeTokens + require.NoError(t, testClient.Resume(ctx, token, "approved", nil)) + + require.NoError(t, <-result) + assert.Equal(t, "approved", decision) + assert.EqualValues(t, 2, attemptsOf(key).Load()) +} + func TestResumeKeepsAWrappedErrorsType(t *testing.T) { ctx := testContext(t) key := uniqueKey(t) @@ -1485,3 +1642,167 @@ func TestResumeKeepsAWrappedErrorsType(t *testing.T) { assert.True(t, gush.IsPermanentError(err)) assert.Equal(t, "[Rejected] reviewer said no: rejected by reviewer", err.Error()) } + +func CancelableApprovalFlow(ctx *gush.FlowContext) (any, error) { + var decision string + err := ctx.RunStep(ApprovalStep, string(ctx.Input), &decision, + gush.WithAttemptTimeout(30*time.Second), + gush.WithMaxAttempts(1), + gush.WithWaitForCancel(), + ) + return decision, err +} + +func TestResumeWithACanceledErrorEndsAStepNotBeingCanceled(t *testing.T) { + tests := map[string]error{ + "context canceled": context.Canceled, + "wrapped": fmt.Errorf("review: %w", context.Canceled), + "temporal": temporal.NewCanceledError(), + } + + for name, canceled := range tests { + t.Run(name, func(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + result := make(chan error, 1) + go func() { result <- testClient.RunFlow(ctx, flowRequest(RetriableApprovalFlow, key, nil)) }() + token := <-resumeTokens + + err := testClient.Resume(ctx, token, nil, canceled) + + require.NoError(t, err) + err = <-result + require.Error(t, err) + assert.True(t, gush.IsPermanentError(err), "%v", err) + assert.EqualValues(t, 1, attemptsOf(key).Load()) + }) + } +} + +func TestResumeWithACanceledErrorCancelsAStepBeingCanceled(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + flowID := key + "-flow" + _, err := testClient.StartFlow(ctx, flowRequest(CancelableApprovalFlow, key, nil), gush.WithID(flowID)) + require.NoError(t, err) + token := <-resumeTokens + require.NoError(t, testClient.CancelFlow(ctx, flowID)) + temporalClient := dialTemporal(t) + require.Eventually(t, func() bool { + description, err := temporalClient.DescribeWorkflowExecution(ctx, flowID, "") + if err != nil { + return false + } + for _, step := range description.GetPendingActivities() { + if step.GetState() == enums.PENDING_ACTIVITY_STATE_CANCEL_REQUESTED { + return true + } + } + return false + }, 10*time.Second, 10*time.Millisecond) + + err = testClient.Resume(ctx, token, nil, context.Canceled) + + require.NoError(t, err) + require.Eventually(t, func() bool { + flow, err := testClient.GetFlow(ctx, flowID) + return err == nil && flow.Status == "Canceled" + }, 10*time.Second, 50*time.Millisecond) +} + +func TestResumeWithAnApplicationErrorCausedByACancellation(t *testing.T) { + permanent := gush.NewPermanentError("rejected", "Rejected", context.Canceled) + tests := map[string]struct { + err error + message string + }{ + "permanent": {permanent, "[Rejected] rejected"}, + "wrapped": {fmt.Errorf("review: %w", permanent), "[Rejected] review: rejected"}, + "aborted": {gush.AsStepContext(context.Background()).Abort(context.Canceled), "[StepAborted] context canceled"}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + result := make(chan error, 1) + go func() { result <- testClient.RunFlow(ctx, flowRequest(RetriableApprovalFlow, key, nil)) }() + token := <-resumeTokens + + err := testClient.Resume(ctx, token, nil, test.err) + + require.NoError(t, err) + err = <-result + assert.True(t, gush.IsPermanentError(err), "%v", err) + assert.Equal(t, test.message, err.Error()) + assert.EqualValues(t, 1, attemptsOf(key).Load()) + }) + } + + t.Run("retryable", func(t *testing.T) { + ctx := testContext(t) + key := uniqueKey(t) + result := make(chan error, 1) + go func() { result <- testClient.RunFlow(ctx, flowRequest(RetriableApprovalFlow, key, nil)) }() + token := <-resumeTokens + options := temporal.ApplicationErrorOptions{Cause: context.Canceled} + retry := temporal.NewApplicationErrorWithOptions("retry", "Retry", options) + + err := testClient.Resume(ctx, token, nil, retry) + + require.NoError(t, err) + token = <-resumeTokens + require.NoError(t, testClient.Resume(ctx, token, "approved", nil)) + require.NoError(t, <-result) + assert.EqualValues(t, 2, attemptsOf(key).Load()) + }) +} + +func InvalidStepAttemptsFlow(ctx *gush.FlowContext) (any, error) { + input := order{} + return nil, ctx.RunStep(DoubleStep, input, nil, gush.WithMaxAttempts(-2)) +} + +func InvalidChildAttemptsFlow(ctx *gush.FlowContext) (any, error) { + request := gush.FlowRequest{Flow: SmallFlow} + return nil, ctx.RunFlow(request, gush.WithMaxAttempts(-2)) +} + +func InvalidStartAttemptsFlow(ctx *gush.FlowContext) (any, error) { + request := gush.FlowRequest{Flow: SmallFlow} + _, err := ctx.StartFlow(request, gush.WithMaxAttempts(-2)) + return nil, err +} + +func TestNegativeAttemptsOtherThanUnlimitedAreAnError(t *testing.T) { + ctx := testContext(t) + + err := testClient.RunFlow(ctx, flowRequest(SmallFlow, nil, nil), gush.WithMaxAttempts(-2)) + assert.ErrorContains(t, err, "max attempts -2") + _, err = testClient.StartFlow(ctx, flowRequest(SmallFlow, nil, nil), gush.WithMaxAttempts(-2)) + assert.ErrorContains(t, err, "max attempts -2") + + flows := []any{InvalidStepAttemptsFlow, InvalidChildAttemptsFlow, InvalidStartAttemptsFlow} + for _, flow := range flows { + request := flowRequest(flow, nil, nil) + err = testClient.RunFlow(ctx, request) + assert.True(t, gush.IsPermanentError(err), "%v", err) + assert.ErrorContains(t, err, "max attempts -2") + } +} + +func HighInitialIntervalFlow(ctx *gush.FlowContext) (any, error) { + var output order + input := order{ID: "high-interval", Amount: 21} + err := ctx.RunStep(DoubleStep, input, &output, gush.WithInitialInterval(200*time.Second)) + return output, err +} + +func TestStepInitialIntervalAboveTheDefaultCapRuns(t *testing.T) { + var output order + + err := testClient.RunFlow(testContext(t), flowRequest(HighInitialIntervalFlow, nil, &output)) + + require.NoError(t, err) + assert.Equal(t, 42, output.Amount) +} diff --git a/log_attribute.go b/log_attribute.go index 5a02192..3131eb5 100644 --- a/log_attribute.go +++ b/log_attribute.go @@ -4,6 +4,7 @@ import ( "bytes" "encoding/json" "fmt" + "maps" "math" "strconv" @@ -63,6 +64,24 @@ func unsignedAttr(key string, value uint64) LogAttribute { return attribute.Int64(key, int64(value)) } +// LogAttributes are the fields attached to every log record of a run. +type LogAttributes map[string]any + +// UnmarshalJSON adds the attributes in data to these, keeping numbers exact, and null clears +// them. Plain JSON decoding would turn every number into a float64, which damages large IDs. +func (attributes *LogAttributes) UnmarshalJSON(data []byte) error { + decoded, err := decodeLogAttributes(data) + if err != nil { + return err + } + if decoded == nil || *attributes == nil { + *attributes = decoded + return nil + } + maps.Copy(*attributes, decoded) + return nil +} + // decodeLogAttributes reads attributes as they arrive from another worker. The SDK's own // decoding would turn every number into a float64, which damages large IDs. func decodeLogAttributes(raw json.RawMessage) (map[string]any, error) { diff --git a/log_export.go b/log_export.go index 8a5cfa0..1cae980 100644 --- a/log_export.go +++ b/log_export.go @@ -131,28 +131,36 @@ func releaseLogExport() { // reads OpenTelemetry's standard variables itself. It accepts a malformed endpoint, such as // collector:4318, and then loses every record, so that is an error here. func newLogExporterFromEnv(ctx context.Context) (sdklog.Exporter, error) { - endpoint := firstEnv(envOTLPLogsEndpoint, envOTLPEndpoint) + variable, endpoint := firstEnv(envOTLPLogsEndpoint, envOTLPEndpoint) if endpoint == "" { return nil, nil } parsed, err := url.Parse(endpoint) if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" { - return nil, fmt.Errorf("gush: OTLP log endpoint %q must be an http:// or https:// URL with a host", endpoint) + return nil, fmt.Errorf("gush: %s=%q must be an http:// or https:// URL with a host", variable, endpoint) } - if firstEnv(envOTLPLogsProtocol, envOTLPProtocol) == "grpc" { + variable, protocol := firstEnv(envOTLPLogsProtocol, envOTLPProtocol) + switch strings.ToLower(protocol) { + case "grpc": exporter, err := otlploggrpc.New(ctx) if err != nil { return nil, fmt.Errorf("gush: create OTLP/gRPC log exporter: %w", err) } return exporter, nil + case "", "http/protobuf": + return newHTTPLogExporter(ctx, parsed) + default: + return nil, fmt.Errorf("gush: %s=%q is not supported, use grpc or http/protobuf", variable, protocol) } +} +func newHTTPLogExporter(ctx context.Context, endpoint *url.URL) (sdklog.Exporter, error) { // The exporter appends /v1/logs to the general variable as written, so a trailing slash // would post to //v1/logs. The logs variable is used as written, slash included. var options []otlploghttp.Option - if os.Getenv(envOTLPLogsEndpoint) == "" && strings.HasSuffix(parsed.Path, "/") { - logsPath := strings.TrimRight(parsed.Path, "/") + "/v1/logs" + if os.Getenv(envOTLPLogsEndpoint) == "" && strings.HasSuffix(endpoint.Path, "/") { + logsPath := strings.TrimRight(endpoint.Path, "/") + "/v1/logs" options = append(options, otlploghttp.WithURLPath(logsPath)) } diff --git a/log_export_test.go b/log_export_test.go index 61d9a9e..49380eb 100644 --- a/log_export_test.go +++ b/log_export_test.go @@ -362,6 +362,55 @@ func TestRejectsAnEndpointThatIsNotAURL(t *testing.T) { require.Error(t, err) assert.Contains(t, err.Error(), "must be an http:// or https:// URL") + assert.Contains(t, err.Error(), test.variable) + assert.Same(t, unconfiguredLogProvider, global.GetLoggerProvider()) + }) + } +} + +func TestProtocolIgnoresCase(t *testing.T) { + clearLogEnv(t) + endpoint, collected := startGRPCCollector(t) + t.Setenv(envOTLPEndpoint, "http://"+endpoint) + t.Setenv(envOTLPProtocol, "GRPC") + + exportOneRecord(t, "uppercase protocol") + + assertCollected(t, collected, "uppercase protocol") +} + +func TestExportsWithTheHTTPProtocolNamed(t *testing.T) { + clearLogEnv(t) + server, collected, _ := startHTTPCollector(t) + t.Setenv(envOTLPEndpoint, server.URL) + t.Setenv(envOTLPProtocol, "http/protobuf") + + exportOneRecord(t, "named protocol") + + assertCollected(t, collected, "named protocol") +} + +func TestRejectsAProtocolItCannotSend(t *testing.T) { + tests := map[string]struct { + variable string + protocol string + }{ + "a typo": {variable: envOTLPProtocol, protocol: "gprc"}, + "json": {variable: envOTLPProtocol, protocol: "http/json"}, + "logs variable": {variable: envOTLPLogsProtocol, protocol: "tcp"}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + clearLogEnv(t) + t.Setenv(envOTLPEndpoint, "http://collector:4318") + t.Setenv(test.variable, test.protocol) + + _, err := InitLogExport() + + require.Error(t, err) + assert.Contains(t, err.Error(), "is not supported") + assert.Contains(t, err.Error(), test.variable) assert.Same(t, unconfiguredLogProvider, global.GetLoggerProvider()) }) } diff --git a/options.go b/options.go index 293869a..2dd5876 100644 --- a/options.go +++ b/options.go @@ -17,7 +17,7 @@ type CommonConfig struct { FairKey string `json:"fair_key"` // LogAttributes are attached to every log record and inherited by child flows and steps. - LogAttributes map[string]any `json:"log_attributes"` + LogAttributes LogAttributes `json:"log_attributes"` } // FlowOption customizes a flow run. @@ -79,8 +79,9 @@ func WithMaxInterval(interval time.Duration) Option { return commonOption(apply) } -// WithMaxAttempts sets the total number of attempts, including the first. Zero means -// unlimited. +// WithMaxAttempts sets the total number of attempts, including the first. Zero means one for +// a flow and ten for a step. Use UnlimitedAttempts to keep retrying. Any other negative +// number makes RunFlow, StartFlow or RunStep return an error. func WithMaxAttempts(attempts int32) Option { apply := func(config *CommonConfig) { config.MaxAttempts = attempts diff --git a/retry.go b/retry.go index 104be5e..2d3011e 100644 --- a/retry.go +++ b/retry.go @@ -1,12 +1,23 @@ package gush import ( + "fmt" "time" "go.temporal.io/sdk/temporal" ) -// RetryPolicy defines how flows and steps are retried after they fail. +// UnlimitedAttempts as MaxAttempts keeps retrying until something else ends the run, such as +// a timeout. +const UnlimitedAttempts int32 = -1 + +const ( + defaultFlowAttempts int32 = 1 + defaultStepAttempts int32 = 10 +) + +// RetryPolicy defines how flows and steps are retried after they fail. Every setting left +// at zero uses its default. type RetryPolicy struct { // InitialInterval is the delay before the first retry. Zero means 1s. InitialInterval time.Duration `json:"initial_interval"` @@ -15,32 +26,58 @@ type RetryPolicy struct { // Zero means 2.0. BackoffCoefficient float64 `json:"backoff_coefficient"` - // MaxInterval caps the delay between retries. Zero means 100 times InitialInterval. + // MaxInterval caps the delay between retries. Zero means 100 times InitialInterval, and a + // cap below InitialInterval is raised to it. MaxInterval time.Duration `json:"max_interval"` - // MaxAttempts is the total number of attempts, including the first. Zero means - // unlimited. + // MaxAttempts is the total number of attempts, including the first. Zero means one for + // a flow and ten for a step. Use UnlimitedAttempts to keep retrying. Any other negative + // number is an error. MaxAttempts int32 `json:"max_attempts"` // NoRetry lists the ApplicationError types that fail immediately. NoRetry []string `json:"no_retry"` } +// validate rejects settings Temporal cannot run, which would leave a flow Running with its +// tasks failing, and an attempt count that only a mistake produces, such as a budget computed +// below zero, which must not turn into retrying forever. +func (policy RetryPolicy) validate() error { + switch { + case policy.MaxAttempts < UnlimitedAttempts: + return fmt.Errorf("gush: max attempts %d must be positive, zero for the default, or UnlimitedAttempts", policy.MaxAttempts) + case policy.BackoffCoefficient != 0 && policy.BackoffCoefficient < 1: + return fmt.Errorf("gush: backoff coefficient %v must be at least 1", policy.BackoffCoefficient) + case policy.InitialInterval < 0: + return fmt.Errorf("gush: initial interval %v must not be negative", policy.InitialInterval) + case policy.MaxInterval < 0: + return fmt.Errorf("gush: max interval %v must not be negative", policy.MaxInterval) + } + return nil +} + +// intervals returns the first delay and the cap, applying the defaults Temporal gives zero +// settings. Temporal rejects a cap below the first delay. +func (policy RetryPolicy) intervals() (initial, maximum time.Duration) { + initial = policy.InitialInterval + if initial == 0 { + initial = time.Second + } + maximum = policy.MaxInterval + if maximum == 0 { + maximum = 100 * initial + } + return initial, max(maximum, initial) +} + // totalBackoff returns the time spent waiting between attempts if all MaxAttempts are used, // applying the same defaults Temporal does to zero settings. func (policy RetryPolicy) totalBackoff() time.Duration { - interval := policy.InitialInterval - if interval == 0 { - interval = time.Second - } + interval, maxInterval := policy.intervals() coefficient := policy.BackoffCoefficient if coefficient == 0 { coefficient = 2.0 } - maxInterval := policy.MaxInterval - if maxInterval == 0 { - maxInterval = 100 * interval - } var total time.Duration for retries := policy.MaxAttempts - 1; retries > 0; retries-- { @@ -56,12 +93,19 @@ func (policy RetryPolicy) totalBackoff() time.Duration { return total } -func (policy RetryPolicy) temporalPolicy() *temporal.RetryPolicy { +// temporalPolicy returns the policy Temporal runs, which reads zero attempts as unlimited. +func (policy RetryPolicy) temporalPolicy(defaultAttempts int32) *temporal.RetryPolicy { + attempts := policy.MaxAttempts + if attempts == 0 { + attempts = defaultAttempts + } + + _, maxInterval := policy.intervals() temporalPolicy := &temporal.RetryPolicy{ InitialInterval: policy.InitialInterval, BackoffCoefficient: policy.BackoffCoefficient, - MaximumInterval: policy.MaxInterval, - MaximumAttempts: policy.MaxAttempts, + MaximumInterval: maxInterval, + MaximumAttempts: max(attempts, 0), NonRetryableErrorTypes: policy.NoRetry, } return temporalPolicy diff --git a/retry_internal_test.go b/retry_internal_test.go new file mode 100644 index 0000000..7ee5175 --- /dev/null +++ b/retry_internal_test.go @@ -0,0 +1,73 @@ +package gush + +import ( + "testing" + "time" +) + +func TestRetryPolicyValidate(t *testing.T) { + invalid := map[string]RetryPolicy{ + "negative attempts": {MaxAttempts: -2}, + "a large negative": {MaxAttempts: -100}, + "small coefficient": {BackoffCoefficient: 0.5}, + "negative coefficient": {BackoffCoefficient: -1}, + "negative initial": {InitialInterval: -time.Second}, + "negative maximum": {MaxInterval: -time.Second}, + } + for name, policy := range invalid { + if policy.validate() == nil { + t.Errorf("%s: validate() = nil", name) + } + } + + valid := map[string]RetryPolicy{ + "empty": {}, + "unlimited": {MaxAttempts: UnlimitedAttempts}, + "a number": {MaxAttempts: 3}, + "a coefficient of 1": {BackoffCoefficient: 1}, + "a cap below first": {InitialInterval: time.Minute, MaxInterval: time.Second}, + } + for name, policy := range valid { + if err := policy.validate(); err != nil { + t.Errorf("%s: validate() = %v", name, err) + } + } +} + +func TestTemporalPolicyAttempts(t *testing.T) { + tests := map[int32]int32{0: 10, 3: 3, UnlimitedAttempts: 0} + for attempts, expected := range tests { + policy := RetryPolicy{MaxAttempts: attempts} + if got := policy.temporalPolicy(defaultStepAttempts).MaximumAttempts; got != expected { + t.Errorf("attempts %d = %d, want %d", attempts, got, expected) + } + } +} + +func TestTemporalPolicyRaisesACapBelowTheFirstDelay(t *testing.T) { + tests := map[string]struct { + policy RetryPolicy + expected time.Duration + }{ + "above the cap": {RetryPolicy{InitialInterval: 200 * time.Second, MaxInterval: 100 * time.Second}, 200 * time.Second}, + "below the cap": {RetryPolicy{InitialInterval: time.Second, MaxInterval: 100 * time.Second}, 100 * time.Second}, + "above the default": {RetryPolicy{MaxInterval: time.Millisecond}, time.Second}, + "no cap": {RetryPolicy{InitialInterval: 2 * time.Second}, 200 * time.Second}, + "nothing set": {RetryPolicy{}, 100 * time.Second}, + } + for name, test := range tests { + if got := test.policy.temporalPolicy(defaultStepAttempts).MaximumInterval; got != test.expected { + t.Errorf("%s: MaximumInterval = %v, want %v", name, got, test.expected) + } + } +} + +func TestTotalBackoffWithACapBelowTheFirstDelay(t *testing.T) { + policy := RetryPolicy{InitialInterval: 200 * time.Second, MaxInterval: 100 * time.Second, MaxAttempts: 3} + + got := policy.totalBackoff() + + if expected := 400 * time.Second; got != expected { + t.Errorf("totalBackoff() = %v, want %v", got, expected) + } +} diff --git a/step.go b/step.go index 6c7b968..5c39d6e 100644 --- a/step.go +++ b/step.go @@ -6,6 +6,7 @@ import ( "fmt" "go.temporal.io/api/serviceerror" + "go.temporal.io/sdk/temporal" ) // errInvalidResumeToken marks a token that could not be decoded. @@ -62,8 +63,11 @@ type StepInfo struct { // Resume finishes a suspended step with a result or, if stepErr is set, an error. See // StepContext.Suspend. // -// An error counts as a failed attempt, so a retryable one runs the step's code again under -// its retry policy. Pass a NewPermanentError to fail the step for good. +// An error fails the step for good, because running its code again would repeat whatever +// it asked for, such as a person's approval. To run the code again under the step's retry +// policy, pass a NewApplicationError, which is a request to retry. A canceled error cancels +// the step if it is being canceled, and otherwise fails it for good too. An ApplicationError +// caused by a cancellation is recorded without that cause or its details. func (c *Client) Resume(ctx context.Context, resumeToken string, result any, stepErr error) error { token, err := decodeResumeToken(resumeToken) if err != nil { @@ -73,7 +77,59 @@ func (c *Client) Resume(ctx context.Context, resumeToken string, result any, ste if err != nil { return err } - return c.temporalClient.CompleteActivity(ctx, token, payload, temporalError(stepErr)) + if !isCancellation(stepErr) { + return c.temporalClient.CompleteActivity(ctx, token, payload, resumeError(stepErr)) + } + + err = c.temporalClient.CompleteActivity(ctx, token, payload, stepErr) + var rejected *serviceerror.InvalidArgument + if !errors.As(err, &rejected) { + return err + } + // The server accepts a cancellation only from a step that was asked for one. Only the + // text goes along, because the SDK would read a wrapped cause as a cancellation again. + failure := errors.New(stepErr.Error()) + return c.temporalClient.CompleteActivity(ctx, token, payload, resumeError(failure)) +} + +// isCancellation reports whether an error passed to Resume asks to cancel the step. +func isCancellation(err error) bool { + if err == nil || temporal.IsApplicationError(err) { + return false + } + return temporal.IsCanceledError(err) || errors.Is(err, context.Canceled) +} + +// resumeError returns what Temporal records for an error passed to Resume that does not +// cancel the step. An ApplicationError says for itself whether to retry, and anything else +// ends the step like Abort. +func resumeError(err error) error { + if err == nil { + return nil + } + if !temporal.IsApplicationError(err) { + return abortError(abortedStepErrorType, []error{err}) + } + return withoutCanceledCause(temporalError(err)) +} + +// withoutCanceledCause drops the cause and details of an ApplicationError caused by a +// cancellation. The SDK would report it as a cancellation, which the server refuses unless +// the step is being canceled, and the error would lose its say over retrying. +func withoutCanceledCause(err error) error { + if !temporal.IsCanceledError(err) && !errors.Is(err, context.Canceled) { + return err + } + var appErr *ApplicationError + if !errors.As(err, &appErr) { + return err + } + options := temporal.ApplicationErrorOptions{ + NonRetryable: appErr.NonRetryable(), + NextRetryDelay: appErr.NextRetryDelay(), + Category: appErr.Category(), + } + return temporal.NewApplicationErrorWithOptions(appErr.Message(), appErr.Type(), options) } // IsResumeTokenError reports whether Resume failed because its token is invalid or the diff --git a/step_config.go b/step_config.go index 1f6252a..71a1d41 100644 --- a/step_config.go +++ b/step_config.go @@ -32,7 +32,7 @@ func DefaultStepConfig() *StepConfig { config.InitialInterval = 1 * time.Second config.BackoffCoefficient = 2.0 config.MaxInterval = 100 * time.Second - config.MaxAttempts = 10 + config.MaxAttempts = defaultStepAttempts config.Priority = PriorityMedium config.LogAttributes = make(map[string]any) return config diff --git a/temporal_options.go b/temporal_options.go index ed45276..06bd6ba 100644 --- a/temporal_options.go +++ b/temporal_options.go @@ -30,9 +30,11 @@ func newStartWorkflowOptions(queue string, config *FlowConfig) client.StartWorkf options := client.StartWorkflowOptions{ ID: config.ID, TaskQueue: queue, - RetryPolicy: config.temporalPolicy(), + RetryPolicy: config.temporalPolicy(defaultFlowAttempts), Priority: newTemporalPriority(&config.CommonConfig), WorkflowExecutionTimeout: config.Timeout, + + WorkflowExecutionErrorWhenAlreadyStarted: config.ErrorIfRunning, } return options } @@ -41,7 +43,7 @@ func newChildWorkflowOptions(queue string, config *FlowConfig) workflow.ChildWor options := workflow.ChildWorkflowOptions{ WorkflowID: config.ID, TaskQueue: queue, - RetryPolicy: config.temporalPolicy(), + RetryPolicy: config.temporalPolicy(defaultFlowAttempts), Priority: newTemporalPriority(&config.CommonConfig), WorkflowExecutionTimeout: config.Timeout, VersioningIntent: defaultCommandVersioningIntent(), @@ -57,11 +59,12 @@ func newDetachedChildWorkflowOptions(queue string, config *FlowConfig) workflow. } func newActivityOptions(config *StepConfig) workflow.ActivityOptions { + retryPolicy := config.temporalPolicy(defaultStepAttempts) options := workflow.ActivityOptions{ TaskQueue: config.Queue, StartToCloseTimeout: config.AttemptTimeout, HeartbeatTimeout: config.HeartbeatTimeout, - RetryPolicy: config.temporalPolicy(), + RetryPolicy: retryPolicy, Priority: newTemporalPriority(&config.CommonConfig), VersioningIntent: defaultCommandVersioningIntent(), WaitForCancellation: config.WaitForCancel, @@ -73,9 +76,11 @@ func newActivityOptions(config *StepConfig) workflow.ActivityOptions { // The overall limit stops a task lingering once no worker is left. It must not cut // retries short, so it covers every attempt's pickup wait and run plus the delays // between them. Unlimited attempts have no such total, so they get no overall limit. - if config.MaxAttempts > 0 { + if retryPolicy.MaximumAttempts > 0 { + resolved := config.RetryPolicy + resolved.MaxAttempts = retryPolicy.MaximumAttempts perAttempt := config.PickupTimeout + config.AttemptTimeout - options.ScheduleToCloseTimeout = time.Duration(config.MaxAttempts)*perAttempt + config.totalBackoff() + options.ScheduleToCloseTimeout = time.Duration(resolved.MaxAttempts)*perAttempt + resolved.totalBackoff() } } return options diff --git a/temporal_options_test.go b/temporal_options_test.go index 3fd3fb7..45670af 100644 --- a/temporal_options_test.go +++ b/temporal_options_test.go @@ -106,13 +106,52 @@ func TestPickupTimeoutDoesNotCutRetriesShort(t *testing.T) { func TestPickupTimeoutWithUnlimitedAttemptsHasNoOverallLimit(t *testing.T) { config := DefaultStepConfig() config.PickupTimeout = time.Minute - config.MaxAttempts = 0 + config.MaxAttempts = UnlimitedAttempts options := newActivityOptions(config) if options.ScheduleToCloseTimeout != 0 { t.Errorf("schedule to close = %v", options.ScheduleToCloseTimeout) } + if options.RetryPolicy.MaximumAttempts != 0 { + t.Errorf("attempts sent to Temporal = %d", options.RetryPolicy.MaximumAttempts) + } +} + +func TestZeroAttemptsMeansTheDefaultOnAFlow(t *testing.T) { + config := DefaultFlowConfig() + config.MaxAttempts = 0 + + policies := map[string]*temporal.RetryPolicy{ + "flow start": newStartWorkflowOptions("queue", config).RetryPolicy, + "child flow": newChildWorkflowOptions("queue", config).RetryPolicy, + "detached child": newDetachedChildWorkflowOptions("queue", config).RetryPolicy, + } + + for path, got := range policies { + if got.MaximumAttempts != defaultFlowAttempts { + t.Errorf("%s: attempts sent to Temporal = %d", path, got.MaximumAttempts) + } + } +} + +func TestZeroAttemptsMeansTheDefaultOnAStep(t *testing.T) { + withDefault := DefaultStepConfig() + withDefault.PickupTimeout = time.Minute + + zero := DefaultStepConfig() + zero.PickupTimeout = time.Minute + zero.MaxAttempts = 0 + + expected := newActivityOptions(withDefault) + options := newActivityOptions(zero) + + if options.RetryPolicy.MaximumAttempts != defaultStepAttempts { + t.Errorf("attempts sent to Temporal = %d", options.RetryPolicy.MaximumAttempts) + } + if options.ScheduleToCloseTimeout != expected.ScheduleToCloseTimeout { + t.Errorf("schedule to close = %v, want %v", options.ScheduleToCloseTimeout, expected.ScheduleToCloseTimeout) + } } func TestOnlyADetachedChildFlowOutlivesItsParent(t *testing.T) {