Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,17 +45,17 @@ The opt-in `integration` build tag starts MongoDB, Elasticsearch, two Store
owners and a discovery node. Tagged tests require a prepared `WEIR_TEST_BINARY`.
Coverage includes discovery through every node, direct owner connections, typed
CRUD and Lua transformations, independent persistence checks, same-Store calls
containing many records, slice preflight and per-record validation, finite Scan continuation, Native response evidence,
containing many records, slice preflight and per-record validation, typed Scan filters and include/exclude projections, cross-owner continuation, Native response evidence,
cancellation, owner restart, directory convergence and persistent client recovery.
Separate multi-process tests verify independent single-record callers.
One-RPC reads deliver 64 MiB and 256 MiB across 1024 distinct documents while
Lua tests cover missing and existing documents, explicit and implicit decisions, first creation, and concurrent read-modify-write through both owners. Missing records and missing backend targets have distinct outcomes; empty scans succeed and complete HTTP 404 responses retain Native completion evidence. Separate multi-process tests verify independent single-record callers.
One-RPC reads deliver 64 MiB and 256 MiB across 1024 and 4096 distinct documents while
checking bounded result credits and sampled owner RSS. Interrupted streams retain
confirmed record outcomes; unconfirmed mutations remain indeterminate.

Each Execute request carries one record and its consecutive index. The client
streams records as they are produced; server aggregation stays within the Store.
Later invalid records stop the stream while preserving earlier acknowledged writes.
Document and Native body formats use the public `ContentType` fields.
Documents use explicit `ContentType` fields. Native selects either a BSON MongoDB command or a typed Search HTTP request; SDK consumers receive chunks with validated response metadata.

Services bind to loopback and own exclusive namespaces. Cleanup verifies exact
process and container ownership. Fixture logs, manifests and cleanup receipts
Expand Down Expand Up @@ -111,6 +111,9 @@ adapter time, rejections and operations per adapter invocation. MongoDB also
records physical command deltas. Adapter invocations can split by namespace,
action and byte limits; their counts are distinct from database wire commands.
Missing samples and reset counters are reported as unavailable.
`business_operation_timeouts` counts failed logical operations whose request context
expired before the call returned; an expired bulk context counts every failed operation
in that batch. This diagnostic does not override confirmed mutation evidence.

A database-saturated comparison requires both paths to qualify independently:
at least five CPU intervals cover 80% of measurement time, mean CPU reaches 90%
Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ module github.com/batchstream/weir-tests
go 1.27.1

require (
github.com/batchstream/weir-go v0.7.0
github.com/batchstream/weir-protocol v0.5.0
github.com/batchstream/weir-go v0.8.0
github.com/batchstream/weir-protocol v0.6.0
go.mongodb.org/mongo-driver/v2 v2.9.1
google.golang.org/grpc v1.83.2
)
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
github.com/batchstream/weir-go v0.7.0 h1:PfuZaRH6A8sMOZ5qa0pyLna8A9qsBOydAX3yH2v2t4s=
github.com/batchstream/weir-go v0.7.0/go.mod h1:3hOZQtFCmS7ZbWuM32jOQxdL1sKQD+hdk1d6U5bcglo=
github.com/batchstream/weir-protocol v0.5.0 h1:ARPy3pcEL0EB5pJsGL10R2/MyH+gb0TqO31t18rOrPE=
github.com/batchstream/weir-protocol v0.5.0/go.mod h1:hHmwPoE9yHYAz7Auc3a14Ijo/uxAef4sQ9nBtyXNcGc=
github.com/batchstream/weir-go v0.8.0 h1:NB1K66a61ceLnE7GVUw3N5VfK19D5tunqXzuK59Xkxw=
github.com/batchstream/weir-go v0.8.0/go.mod h1:nNqdGY1ajxYNkRLJWUprF+vQebxd2fOuq0ePAgo2rXs=
github.com/batchstream/weir-protocol v0.6.0 h1:Mpminz34U6IFmqJ0eIl5YA3ClyTok+sxnuLqvIBZHKU=
github.com/batchstream/weir-protocol v0.6.0/go.mod h1:hHmwPoE9yHYAz7Auc3a14Ijo/uxAef4sQ9nBtyXNcGc=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
Expand Down
2 changes: 1 addition & 1 deletion internal/benchmark/client_worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ func (s *durationState) nextSingle(index int) workload.Operation {
if write {
worker.revisions[worker.cursor] = 1 - worker.revisions[worker.cursor]
}
operation := workload.Operation{Worker: index, Sequence: worker.sequence, Record: record, Write: write, Revision: worker.revisions[worker.cursor]}
operation := workload.Operation{Record: record, Write: write, Revision: worker.revisions[worker.cursor]}
worker.cursor = (worker.cursor + 1) % (worker.end - worker.start)
worker.sequence++
return operation
Expand Down
4 changes: 4 additions & 0 deletions internal/benchmark/runner.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,9 @@ func measure(ctx context.Context, executor workload.Executor, plan *workload.Pla
began := time.Now()
outcome := executor.Execute(operationCtx, operation)
worker.hist.add(time.Since(began))
if outcome.Status != workload.Success && errors.Is(operationCtx.Err(), context.DeadlineExceeded) {
worker.result.Timeouts++
}
cancel()
worker.result.Attempted++
worker.result.RequestBytes += outcome.RequestBytes
Expand Down Expand Up @@ -236,6 +239,7 @@ func measure(ctx context.Context, executor workload.Executor, plan *workload.Pla
result := Result{Path: executor.Name(), Planned: uint64(plan.Count), ElapsedNS: elapsed.Nanoseconds(), Evidence: executor.Evidence()}
var hist histogram
for _, worker := range workers {
result.Timeouts += worker.result.Timeouts
result.Attempted += worker.result.Attempted
result.Succeeded += worker.result.Succeeded
result.Errors += worker.result.Errors
Expand Down
30 changes: 25 additions & 5 deletions internal/benchmark/runner_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,17 @@ import (
"os"
"path/filepath"
"reflect"
"sync/atomic"
"testing"
"time"

"github.com/batchstream/weir-tests/internal/workload"
)

type fakeExecutor struct {
mode string
delay time.Duration
mode string
delay time.Duration
sequence atomic.Uint64
}

func (f *fakeExecutor) Name() string { return "direct" }
Expand All @@ -30,8 +32,10 @@ func (f *fakeExecutor) Execute(ctx context.Context, operation workload.Operation
select {
case <-timer.C:
case <-ctx.Done():
outcome := workload.Outcome{Status: workload.Indeterminate, Error: ctx.Err().Error()}
return outcome
if f.mode != "success_after_deadline" {
outcome := workload.Outcome{Status: workload.Indeterminate, Error: ctx.Err().Error()}
return outcome
}
}
}
outcome := workload.Outcome{Status: workload.Success, RequestBytes: 3, ResponseBytes: 7}
Expand All @@ -41,7 +45,7 @@ func (f *fakeExecutor) Execute(ctx context.Context, operation workload.Operation
outcome.Error = "post-write acknowledgement failure"
}
if f.mode == "mixed" {
switch operation.Sequence % 3 {
switch (f.sequence.Add(1) - 1) % 3 {
case 1:
outcome.Status, outcome.Error = workload.Failed, "known failure"
case 2:
Expand Down Expand Up @@ -213,3 +217,19 @@ func TestWriteReportPreservesRawEvidenceAndNoInvalidRatio(t *testing.T) {
t.Fatal("accepted report path collision")
}
}

func TestMeasureCountsOperationDeadlines(t *testing.T) {
executor := &fakeExecutor{delay: time.Second}
result := measure(t.Context(), executor, testPlan(t, 8), time.Millisecond)
if result.Timeouts != 8 || result.Attempted != 8 || result.Indeterminate != 8 {
t.Fatalf("deadline accounting: %+v", result)
}
}

func TestExpiredContextDoesNotEraseConfirmedSuccess(t *testing.T) {
executor := &fakeExecutor{mode: "success_after_deadline", delay: time.Second}
result := measure(t.Context(), executor, testPlan(t, 8), time.Millisecond)
if result.Timeouts != 0 || result.Succeeded != 8 || !result.complete() {
t.Fatalf("confirmed success was counted as timeout: %+v", result)
}
}
7 changes: 6 additions & 1 deletion internal/benchmark/saturation.go
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ func (s *durationState) next(workerIndex int) []workload.Operation {
if write {
worker.revisions[worker.cursor] = 1 - worker.revisions[worker.cursor]
}
operation := workload.Operation{Worker: workerIndex, Sequence: worker.sequence, Record: record, Write: write, Revision: worker.revisions[worker.cursor]}
operation := workload.Operation{Record: record, Write: write, Revision: worker.revisions[worker.cursor]}
operations[index] = operation
worker.cursor = (worker.cursor + 1) % (worker.end - worker.start)
worker.sequence++
Expand Down Expand Up @@ -288,6 +288,7 @@ func measureDuration(ctx context.Context, options durationMeasureOptions) Satura
began := time.Now()
outcomes := options.Executor.ExecuteBatch(operationCtx, operations)
elapsed := time.Since(began)
timedOut := errors.Is(operationCtx.Err(), context.DeadlineExceeded)
cancel()
requests[index]++
if len(outcomes) != len(operations) {
Expand All @@ -298,6 +299,9 @@ func measureDuration(ctx context.Context, options durationMeasureOptions) Satura
}
}
for operationIndex, outcome := range outcomes {
if timedOut && outcome.Status != workload.Success {
worker.result.Timeouts++
}
worker.hist.add(elapsed)
worker.result.Attempted++
if operations[operationIndex].Write {
Expand Down Expand Up @@ -391,6 +395,7 @@ func measureDuration(ctx context.Context, options durationMeasureOptions) Satura
result.Resources.MeasurementNS = elapsed.Nanoseconds()
var hist histogram
for index, worker := range workers {
result.Timeouts += worker.result.Timeouts
result.Attempted += worker.result.Attempted
result.Succeeded += worker.result.Succeeded
result.Errors += worker.result.Errors
Expand Down
14 changes: 14 additions & 0 deletions internal/benchmark/saturation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -234,3 +234,17 @@ func TestSaturationReportRecordsClientAndBackendRequestBudgets(t *testing.T) {
t.Fatal("human-readable report hid the request budgets", markdown)
}
}

func TestBulkDurationCountsLogicalOperationsInExpiredBatches(t *testing.T) {
config := workload.Config{Backend: "search", StoreName: "search", Namespace: "weirtest_0123456789abcdef01234567", Records: 8, Concurrency: 2}
dataset, err := workload.New(config)
if err != nil {
t.Fatal(err)
}
executor := &fakeExecutor{delay: time.Second}
opts := durationMeasureOptions{Executor: executor, State: newDurationState(dataset, 2, 4, 0), Duration: time.Second, Timeout: time.Millisecond}
result := measureDuration(t.Context(), opts)
if result.Timeouts != 8 || result.Attempted != 8 || result.Indeterminate != 8 || result.Requests != 2 {
t.Fatalf("expired batches must count logical operations: %+v", result)
}
}
4 changes: 1 addition & 3 deletions internal/workload/dataset.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ type Config struct {
}

type Operation struct {
Worker int
Sequence int
Record int
Write bool
Revision int
Expand Down Expand Up @@ -161,7 +159,7 @@ func (d *Dataset) Plan(options PlanOptions) (*Plan, error) {
} else {
plan.Reads++
}
operation := Operation{Worker: worker, Sequence: sequence, Record: record, Write: write, Revision: plan.Expected[record]}
operation := Operation{Record: record, Write: write, Revision: plan.Expected[record]}
operations[sequence] = operation
}
plan.Workers[worker] = operations
Expand Down
5 changes: 1 addition & 4 deletions internal/workload/dataset_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,11 @@ func TestDeterministicPlanOwnsWorkerIDsAndReadRevisions(t *testing.T) {
actual := make([]int, dataset.Config.Records)
reads, writes, count := 0, 0, 0
for worker, operations := range first.Workers {
for sequence, operation := range operations {
for _, operation := range operations {
if owner, exists := owners[operation.Record]; exists && owner != worker {
t.Fatal("record shared by workers")
}
owners[operation.Record] = worker
if operation.Worker != worker || operation.Sequence != sequence {
t.Fatal("sequence metadata mismatch")
}
if operation.Write {
actual[operation.Record] = 1 - actual[operation.Record]
writes++
Expand Down
2 changes: 1 addition & 1 deletion internal/workload/lua.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import (
)

func (d *Dataset) luaSource() string {
// The fixture stores BSON int32 and JSON 0/1, both read as lua.v1 int32.
// The fixture stores BSON int32 and JSON 0/1, both read as Lua int32.
return `return weir.replace(weir.set(current, "revision", weir.sub(weir.i32("1"), weir.get(current, "revision"))))`
}

Expand Down
4 changes: 2 additions & 2 deletions internal/workload/paths.go
Original file line number Diff line number Diff line change
Expand Up @@ -482,8 +482,8 @@ func (p *weirPath) Execute(ctx context.Context, operation Operation) Outcome {
}
if operation.Write {
if p.dataset.Config.LuaMutations {
program := &weir.ProgramTransform{Runtime: "lua.v1", Source: []byte(p.dataset.luaSource())}
request := &weir.AtomicTransformRequest{Resource: p.dataset.Resource(operation.Record), Program: program}
program := &weir.LuaTransform{Source: []byte(p.dataset.luaSource())}
request := &weir.AtomicTransformRequest{Resource: p.dataset.Resource(operation.Record), Lua: program}
opts := weir.AtomicTransformOptions{StoreName: p.dataset.Config.StoreName, Request: request}
result, err := p.client.AtomicTransform(ctx, opts)
return mutationOutcome(result, len(program.Source), err)
Expand Down
10 changes: 5 additions & 5 deletions tests/integration/batch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (

func testPublicBatch(t *testing.T, ctx context.Context, client *weir.Client, backend *backendData) {
t.Helper()
secondary := provisionBatchTarget(t, ctx, backend)
secondary := backend.provisionTarget(t, ctx, "batch_extra")
targets := []*backendData{backend, secondary}
const count = 48
mutations := make([]*weir.MutateRequest, count)
Expand Down Expand Up @@ -95,15 +95,15 @@ func testPublicBatch(t *testing.T, ctx context.Context, client *weir.Client, bac
t.Log("48 streamed SDK mutations across two same-Store targets, ordered results including precondition/missing, duplicate URI order, and slice preflight verified")
}

func provisionBatchTarget(t *testing.T, ctx context.Context, base *backendData) *backendData {
func (base *backendData) provisionTarget(t *testing.T, ctx context.Context, suffix string) *backendData {
t.Helper()
name := base.collection + "_batch_extra"
name := base.collection + "_" + suffix
if base.name == "mongo" {
database := base.mongo.Database()
if err := database.CreateCollection(ctx, "batch_extra"); err != nil {
if err := database.CreateCollection(ctx, suffix); err != nil {
t.Fatal("create second owned Mongo collection:", err)
}
secondary := &backendData{name: base.name, collection: database.Name() + "/batch_extra", mongo: database.Collection("batch_extra")}
secondary := &backendData{name: base.name, collection: database.Name() + "/" + suffix, mongo: database.Collection(suffix)}
return secondary
}
secondary := &backendData{name: base.name, collection: name, searchURL: base.searchURL, http: base.http}
Expand Down
Loading
Loading