Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ integration:
integration-previous-server:
WEIR_TEST_BINARY="$(CURDIR)/.tools/previous/weir" go test -race -tags=integration,previous -count=1 -timeout=10m -v ./tests/integration

# GOFLAGS, not a flag of go test, so that the binaries the tests build use the same SDK.
# GOFLAGS, not a flag of go test, so that any binary a test builds uses the same SDK.
# The SDK brings the protocol version it was released with, as it does for a caller.
integration-previous-sdk:
mkdir -p .tools
Expand All @@ -27,7 +27,7 @@ integration-previous-sdk:
version=$$(jq -er .weir_go_previous versions.json) && \
protocol=$$(go mod download -json github.com/batchstream/weir-go@$$version | jq -er .GoMod | xargs awk '$$1 == "github.com/batchstream/weir-protocol" {print $$2}') && \
go get -modfile=.tools/previous-sdk.mod github.com/batchstream/weir-go@$$version github.com/batchstream/weir-protocol@$$protocol
GOFLAGS=-modfile="$(CURDIR)/.tools/previous-sdk.mod" WEIR_TEST_BINARY="$(CURDIR)/.tools/weir" go test -race -tags=integration,previous -count=1 -timeout=10m -v ./tests/integration
GOFLAGS=-modfile="$(CURDIR)/.tools/previous-sdk.mod" WEIR_TEST_BINARY="$(CURDIR)/.tools/weir" go test -race -tags=integration,previous,previous_sdk -count=1 -timeout=10m -v ./tests/integration

benchmark:
go run ./cmd/weir-lab -mode saturation -weir "$(CURDIR)/.tools/weir" -backend all -database-cpus 1 -client-processes 4 -concurrency-levels 4,8,32,128,256 -batch-sizes 1 -records 8192 -write-percent 0 -rounds 3 -warmup-duration 10s -duration 20s -output results/local/benchmark
Expand Down
9 changes: 9 additions & 0 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ only the current versions can pass goes in a file with a `!previous` build
constraint so that the rest keeps running, until the previous pin has that
behaviour too.

The SDK's API changed at v0.24.0, so `integration-previous-sdk` also sets the
`previous_sdk` tag. `sdk_current_test.go` and `sdk_previous_test.go` hold the calls
that differ. The soak and benchmark tests use packages built on the current API
only, so they do not run on the previous SDK, and neither does the native write
evidence subtest, which lives with them. That leaves the previous SDK's `MutateStream`
and query-visible writes untested against a newer server. When `weir_go_previous`
reaches v0.24.0, delete the tag, `sdk_previous_test.go` and the constraints on those
tests.

## Output

All generated output belongs under the ignored `results/` directory, conventionally
Expand Down
14 changes: 7 additions & 7 deletions docs/soak.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@ fresh MongoDB database and Search index, drops them at the end unless
## Profiles

Each worker uses the SDK in one way, its profile, and `-profiles` sets their
shares of a backend's workers. The default exercises every call:
shares of a backend's workers. The default runs all of them:

| Profile | Writes | Reads |
| --- | --- | --- |
| `single` | `Put` | `ReadOne` |
| `visible` | query-visible `Put` | `ReadOne` |
| `lua` | `AtomicTransform` | `ReadOne` |
| `batch` | `Mutate` of `-batch-size` keys | `Read` of as many |
| `stream` | a `MutateStream` held for `-stream-lifetime` | a `ReadStream` held as long |
| `native` | `Put` | the backend's own read through `Native` |
| `single` | a `Mutate` of one `MutationPut` | a `Read` of one key |
| `visible` | the same, with `WithQueryVisible` | a `Read` of one key |
| `lua` | a `Mutate` of one `MutationAtomicTransform` | a `Read` of one key |
| `batch` | a `Mutate` of `-batch-size` keys | a `Read` of as many |
| `stream` | a `Mutate` with `WithNext` and `WithMutateConsume`, held for `-stream-lifetime` | a `Read` with `WithNext` and `WithReadConsume`, held as long |
| `native` | a `Mutate` of one `MutationPut` | the backend's own read through `Native` |

`-read-percent` chooses between a worker's reads and writes; a worker in the
`stream` profile alternates, because each of its turns lasts for many records,
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/batchstream/weir-tests
go 1.27.3

require (
github.com/batchstream/weir-go v0.23.0
github.com/batchstream/weir-go v0.24.0
github.com/batchstream/weir-protocol v0.17.0
go.mongodb.org/mongo-driver/v2 v2.9.1
google.golang.org/grpc v1.83.2
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
github.com/batchstream/weir-go v0.23.0 h1:2PMFV1L2dtVzzORzu+ccwe7TfF3M8VsW96yNaPWtYqA=
github.com/batchstream/weir-go v0.23.0/go.mod h1:tW3APP6fK6A+dVLzwcgkRECIDzlhv0hnhTCyJghcjxw=
github.com/batchstream/weir-go v0.24.0 h1:Ij8r7ceDOvEVduNOfnpSpN7mlUocF2H+e2fsp4ejiTY=
github.com/batchstream/weir-go v0.24.0/go.mod h1:tW3APP6fK6A+dVLzwcgkRECIDzlhv0hnhTCyJghcjxw=
github.com/batchstream/weir-protocol v0.17.0 h1:0gfA9nIoRu+6Lhzmtpxv4HruaB4dBDuyAL45x+9OF6g=
github.com/batchstream/weir-protocol v0.17.0/go.mod h1:05pTsdu4nJkyZjvdz67jILOiuEicw0Vq88M0ucD4GVY=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
Expand Down
16 changes: 12 additions & 4 deletions internal/benchmark/crawler_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -204,9 +204,12 @@ func (c *crawlerClient) executeMutations(ctx context.Context, mutations []crawle
if err != nil {
return false, err
}
opts := weir.MutateOptions{StoreName: "search", Requests: requests}
options := []weir.MutateOption{weir.WithRequests(requests...)}
if crawlerQueryVisible(c.config) {
options = append(options, weir.WithQueryVisible())
}
ledger.record(len(requests))
replies, err := c.sdk.Mutate(ctx, opts)
replies, err := c.sdk.Mutate(ctx, "search", options...)
unknown := len(replies) != len(requests)
var failures []error
if err != nil {
Expand Down Expand Up @@ -271,6 +274,12 @@ func (c *crawlerClient) executeMutations(ctx context.Context, mutations []crawle
return false, nil
}

// crawlerQueryVisible reports whether a call waits for query visibility. Only the
// partial-update profile may wait for application acknowledgement instead.
func crawlerQueryVisible(config crawlerClientConfig) bool {
return config.Visibility != "application_acknowledged"
}

func crawlerWeirRequests(config crawlerClientConfig, mutations []crawlerMutation) ([]*weir.MutateRequest, error) {
if config.InputShape == "single" && len(mutations) != 1 {
return nil, errors.New("single-record SDK call contains a client batch")
Expand Down Expand Up @@ -298,8 +307,7 @@ func crawlerWeirRequests(config crawlerClientConfig, mutations []crawlerMutation
}
input := &weir.Document{ContentType: "application/json", Data: raw}
program := &weir.LuaTransform{Source: []byte(source), Input: input}
visibility := &weir.MutationOptions{Visibility: weir.MutationQueryVisible}
request := &weir.MutateRequest{Resource: crawlerIndex(config.Namespace, mutation.Product) + "/s:" + mutation.ID, Action: weir.MutationAtomicTransform, Lua: program, Options: visibility}
request := &weir.MutateRequest{Resource: crawlerIndex(config.Namespace, mutation.Product) + "/s:" + mutation.ID, Action: weir.MutationAtomicTransform, Lua: program}
requests[index] = request
}
return requests, nil
Expand Down
6 changes: 2 additions & 4 deletions internal/benchmark/crawler_completion_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ import (
"sync/atomic"
"testing"
"time"

weir "github.com/batchstream/weir-go"
)

func TestPartialUpdateApplicationAcknowledgedMatchesNativeAndPreservesPadding(t *testing.T) {
Expand Down Expand Up @@ -47,8 +45,8 @@ func TestPartialUpdateApplicationAcknowledgedMatchesNativeAndPreservesPadding(t
if err != nil || len(requests) != 1 {
t.Fatal(requests, err)
}
if requests[0].Options.Visibility != weir.MutationApplicationAcknowledged || requests[0].Lua != nil {
t.Fatal("Weir did not use the same application completion boundary", requests[0])
if requests[0].Lua != nil {
t.Fatal("Weir partial update used Lua", requests[0])
}
var payload map[string]any
if err := json.Unmarshal(requests[0].BackendExpression.Data, &payload); err != nil || !reflect.DeepEqual(payload["doc"], mutation.Document) {
Expand Down
36 changes: 34 additions & 2 deletions internal/benchmark/crawler_single_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"time"

weir "github.com/batchstream/weir-go"
"github.com/batchstream/weir-tests/internal/wiretest"
)

func TestCrawlerSingleNativeHasAtomicEndpointAndConcurrentSellerFanout(t *testing.T) {
Expand Down Expand Up @@ -125,6 +126,37 @@ func TestCrawlerSingleMissingAcknowledgementStopsCausalCallsWithoutReplay(t *tes
}
}

func TestCrawlerCallsSendTheirCompletionOnTheWire(t *testing.T) {
tests := []struct {
name string
profile string
visibility string
mutation crawlerMutation
queryVisible bool
}{
{name: "lua", mutation: crawlerPage(0, 1, 1)[0], queryVisible: true},
{name: "partial update", profile: "partial-update", mutation: partialUpdateMutation(1, 2), queryVisible: true},
{name: "partial update acknowledged", profile: "partial-update", visibility: "application_acknowledged", mutation: partialUpdateMutation(1, 2)},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
server := &wiretest.Recorder{}
config := crawlerClientConfig{InputShape: "single", Namespace: "fixture", PageSize: 1, Profile: test.profile, Visibility: test.visibility}
client := &crawlerClient{config: config, sdk: wiretest.Client(t, server)}
mutations := []crawlerMutation{test.mutation}
var ledger crawlerApplicationLedger
unknown, err := client.executeMutations(t.Context(), mutations, &ledger)
if err != nil || unknown {
t.Fatal(unknown, err)
}
received := server.Mutations()
if len(received) != 1 || wiretest.QueryVisible(received[0]) != test.queryVisible {
t.Fatalf("query visible=%v, sent %v", test.queryVisible, received)
}
})
}
}

func TestCrawlerSingleSDKConstructsExactlyOneRecordAndRejectsBatches(t *testing.T) {
config := crawlerClientConfig{InputShape: "single", Namespace: "fixture", PageSize: 1}
mutation := crawlerPage(0, 1, 1)[0]
Expand All @@ -134,8 +166,8 @@ func TestCrawlerSingleSDKConstructsExactlyOneRecordAndRejectsBatches(t *testing.
t.Fatal("single SDK call was not one record", requests, err)
}
request := requests[0]
if request.Options == nil || request.Options.Visibility != weir.MutationQueryVisible || request.Action != weir.MutationAtomicTransform || request.Lua == nil || request.Lua.Input == nil {
t.Fatal("single SDK call lost Lua or completion semantics", request)
if request.Action != weir.MutationAtomicTransform || request.Lua == nil || request.Lua.Input == nil {
t.Fatal("single SDK call lost Lua", request)
}
var actual crawlerMutation
if err := json.Unmarshal(request.Lua.Input.Data, &actual); err != nil || !reflect.DeepEqual(actual, mutation) {
Expand Down
7 changes: 1 addition & 6 deletions internal/benchmark/partial_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,6 @@ func partialUpdateRequest(config crawlerClientConfig, mutation crawlerMutation)
return nil, err
}
expression := &weir.Document{ContentType: "application/vnd.weir.search-update.v1+json", Data: raw}
completion := weir.MutationQueryVisible
if config.Visibility == "application_acknowledged" {
completion = weir.MutationApplicationAcknowledged
}
visibility := &weir.MutationOptions{Visibility: completion}
request := &weir.MutateRequest{Resource: crawlerIndex(config.Namespace, true) + "/s:" + mutation.ID, Action: weir.MutationAtomicTransform, BackendExpression: expression, Options: visibility}
request := &weir.MutateRequest{Resource: crawlerIndex(config.Namespace, true) + "/s:" + mutation.ID, Action: weir.MutationAtomicTransform, BackendExpression: expression}
return request, nil
}
4 changes: 2 additions & 2 deletions internal/benchmark/partial_update_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ func TestPartialUpdateNativeAndWeirUseIdenticalSingleDocWithoutLua(t *testing.T)
t.Fatal(requests, err)
}
request := requests[0]
if request.Lua != nil || request.BackendExpression == nil || request.BackendExpression.ContentType != "application/vnd.weir.search-update.v1+json" || request.Action != weir.MutationAtomicTransform || request.Options.Visibility != weir.MutationQueryVisible {
t.Fatal("mechanism included Lua or changed completion semantics", request)
if request.Lua != nil || request.BackendExpression == nil || request.BackendExpression.ContentType != "application/vnd.weir.search-update.v1+json" || request.Action != weir.MutationAtomicTransform {
t.Fatal("mechanism included Lua", request)
}
var payload map[string]any
if err := json.Unmarshal(request.BackendExpression.Data, &payload); err != nil || len(payload) != 1 || !reflect.DeepEqual(payload["doc"], mutation.Document) {
Expand Down
2 changes: 1 addition & 1 deletion internal/soak/soak.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ func Run(ctx context.Context, config Config) (resultErr error) {
backend: run.config.Kind,
storeName: run.config.Store,
store: run.store,
client: client,
client: sdkRecords{client: client},
keys: run.keys[index*run.config.KeysPerWorker : (index+1)*run.config.KeysPerWorker],
padding: padding,
profile: profileOf(config.Profiles, index, run.config.Workers),
Expand Down
Loading
Loading