Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ jobs:

qualification:
name: External test suite
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@ca8862a8531925da8eee74c8b13f77440ef5d827
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@c95932c11d922345deec8e03895129c33ee885b6
with:
suite_ref: ca8862a8531925da8eee74c8b13f77440ef5d827
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
sink_ref: ${{ github.event.pull_request.head.sha || github.sha }}
sdk_ref: ${{ github.head_ref || 'main' }}
4 changes: 2 additions & 2 deletions .github/workflows/release-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@ jobs:
qualify:
name: Public production qualification
needs: release
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@ca8862a8531925da8eee74c8b13f77440ef5d827
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@c95932c11d922345deec8e03895129c33ee885b6
with:
suite_ref: ca8862a8531925da8eee74c8b13f77440ef5d827
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
sink_ref: ${{ needs.release.outputs.revision }}

binaries:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/reliability.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permissions:

jobs:
soak:
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@ca8862a8531925da8eee74c8b13f77440ef5d827
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@c95932c11d922345deec8e03895129c33ee885b6
with:
suite_ref: ca8862a8531925da8eee74c8b13f77440ef5d827
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
sink_ref: ${{ github.sha }}
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,8 +154,8 @@ use the lowercase spelling shown below. Storage names are also case-sensitive.
| `batching.max_wait` | duration string | No | `2ms` | Positive Go duration within the bounds below | Maximum collection delay measured from the first request in a batch. |
| `batching.max_operations` | positive integer | No | `32` | Positive integer | Operation target for one automatically formed batch; a larger valid RPC still executes alone. See [choosing limits](batching.md#choosing-limits). |
| `batching.max_bytes` | byte size | No | `16MiB` | Size greater than `0B` | Encoded-byte target for one automatically formed batch; one larger valid RPC still runs alone. |
| `batching.queue.max_operations` | positive integer | No | max(`10000`, `batching.max_operations`) | Integer at least `batching.max_operations` | Maximum operations waiting in each method queue. |
| `batching.queue.max_bytes` | byte size | No | max(`128MiB`, `grpc.max_receive_message_bytes`) | Size at least `grpc.max_receive_message_bytes` and `batching.max_bytes` | Maximum encoded request bytes waiting in each method queue. |
| `batching.queue.max_operations` | positive integer | No | max(`10000`, `batching.max_operations`) | Integer at least `batching.max_operations` | Maximum operations waiting in each batch-method queue; also the request count bound for the shared Query/Count admission FIFO. |
| `batching.queue.max_bytes` | byte size | No | max(`128MiB`, `grpc.max_receive_message_bytes`) | Size at least `grpc.max_receive_message_bytes` and `batching.max_bytes` | Maximum encoded request bytes waiting in each batch-method queue and in the separate Query/Count admission FIFO. Query/Count waiting ends only with capacity or caller-context cancellation/deadline, not a server-added timeout. |
| `execution.merge.lua.timeout` | duration string | No | `100ms` | Positive Go duration within the bounds below | Maximum wall-clock duration of one Lua execution. |
| `execution.merge.lua.max_source_bytes` | byte size | No | `64KiB` | Size greater than `0B` | Maximum Lua source size per merge operation. |
| `execution.merge.lua.max_result_bytes` | byte size | No | `16MiB` | Size greater than `0B` | Maximum input/current document and encoded result bytes; expanded output is also bounded before conversion. |
Expand Down
50 changes: 38 additions & 12 deletions docs/design/store-backpressure.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Gateway/Engine/Worker scaling adds application processing capacity. Store
backpressure independently regulates the work those processes send to the
database. A growing request queue or Kafka lag never raises the Store window.
database. Queue demand can request growth, but only healthy backend completions
raise the Store window; queue growth or Kafka lag alone cannot do so.
The controller does not consume replica counts, HPA signals, CPU, memory,
connection counts, Ping results, or discovery information.

Expand All @@ -20,15 +21,15 @@ flowchart LR
G[Gateway] --> Q[Engine bounded method queues]
Q -->|TryAcquire before dequeue| C[Local Store controller]
K[Kafka backlog] -->|Wait before poll; Acquire before attempt| C
N[Native request] -->|Admit or reject before execution| C
R[Query / Count] -->|Bounded FIFO; caller context only| C
N[Execute / Scan] -->|Admit or reject before execution| C
C --> E[Sequential admitted execution]
E --> O[Storage observation decorator]
O --> S[Storage / NativeStorage]
O -->|Real results and execution latency| C
```

There is no new work queue and no goroutine in the controller. Engine's three
existing dispatch loops select dependency-ready work, then try admission **before**
Engine's three existing batch dispatch loops select dependency-ready work, then try admission **before**
releasing queue bytes/operations, recording active records, or launching execution.
When blocked, they continue handling arrivals, cancellation, record completions and
shutdown while waiting for a capacity notification or a cooldown timer. Pending
Expand All @@ -45,9 +46,25 @@ avoids moving a bounded queue into unbounded semaphore waiters, and avoids
deadlocking a Merge which already owns its execution slot.

Read, Write, Delete, Execute, Query, Count and Scan share the same controller.
Native methods have no existing bounded waiting queue: they fail admission
immediately when the window is occupied or paused. Engine marks this specific
rejection `sink-forward-not-started`; a backend error with the same gRPC code
Query and Count share a bounded admission FIFO. A full execution window or an
overload cooldown leaves those requests waiting rather than rejecting a healthy
burst. `batching.queue.max_operations` bounds waiting requests (one command per
request), and `batching.queue.max_bytes` bounds retained encoded request bytes.
These are the same configured limits as the batch queues, not a second queue on
top of an existing Read/Write/Delete queue. Only the head competes for a Store
permit; no goroutine is spawned by admission. Query/Count are FIFO with respect
to each other, not globally ordered with the independent batch dispatch loops.

There is **no new server-side admission timeout**. A caller with no deadline can
wait until capacity is available or its context is canceled. A caller deadline
is propagated unchanged, and cancellation removes queue charges without entering
the backend. Only a full waiting queue rejects with `RESOURCE_EXHAUSTED`.
Encoded bytes are a bounded accounting unit, not an exact Go heap estimate;
process memory admission and gRPC message limits remain independent safeguards.
Admission waiting never shortens the lifetime of an already executing permit.

Execute and Scan keep their fail-fast admission semantics. Engine marks a Store
admission or read-queue rejection `sink-forward-not-started`; a backend error with the same gRPC code
does not receive that evidence. Async publication bypasses Store admission;
Kafka producer bounds still apply, and Worker controls the later application.

Expand All @@ -62,14 +79,16 @@ mutation waves sequentially; this feature does not add parallel poll processing.

## Window algorithm

The algorithm has a fixed-size state and one short mutex-protected update per
Store observation. It starts on the first admission demand, so constructing an
idle process early cannot defeat startup staggering.
The window algorithm has fixed-size feedback state and one short mutex-protected
update per Store observation, alongside the bounded Query/Count FIFO. Engine
actively completes randomized initialization in `Run` **before serving readiness
or RPCs**, without issuing a synthetic database request. Worker retains its
first-poll initialization. Subsequent saturation or cooldown does not flap readiness.

| Event | Behavior |
| --- | --- |
| First demand | Window zero for a random 0.5–1.5 seconds, then one execution |
| Healthy, saturated work | Grow only after at least `max(4, window)` healthy samples and a jittered 125–375 ms control interval |
| Initialization | Stagger for a random 0.5–1.5 seconds, then open `min(4, maximum)` executions before Engine readiness |
| Healthy demand | Saturation or queued Query/Count requests request growth; require at least `max(4, window)` healthy samples and a jittered 125–375 ms control interval |
| Initial growth | Add `max(1, window/2)` below a slow-start threshold of 16 |
| After congestion | Threshold becomes half the previous window; subsequent growth is additive by one |
| Sustained latency inflation | Halve the window, retaining at least one execution so successful work continues and the baseline can adapt |
Expand Down Expand Up @@ -190,6 +209,13 @@ A seeded discrete-event simulation runs 1, 8 and 100 independent controllers
against a shared backend with capacity 8 → 2 → 8, checking recovery, bounded
overload traffic, startup staggering and progress across instances. It is a
repeatable algorithm regression, not a production throughput benchmark.
The initial window of four retains this startup-overshoot gate: an eight-slot
initial window failed the 100-instance regression. Healthy burst tests also
require progress without rejection, rather than treating a tiny initial window
as sufficient protection. Deterministic virtual-time tests hold a caller for an
hour without a server-added deadline, cancel the head/middle/tail of the FIFO,
exercise byte/count bounds and cooldown, and verify exact permit/queue cleanup.
Race tests concurrently release permits, cancel callers and collect metrics.
Kafka component tests use an in-process broker to verify that paused intake
does not spend processing timeout, report unhealthy, advance offsets, or produce
dead letters; another test verifies a real processor's overload/cooldown retry.
Expand Down
16 changes: 13 additions & 3 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,16 +191,26 @@ and memory measurements do not feed these metrics or the congestion algorithm.
| `sink_store_feedback_total` | counter | `method`, `signal=healthy,congested,ignored`; one sample per Store call |
| `sink_store_backend_duration_seconds_total` | counter | `method`; elapsed Store time excluding admission and downstream Emit waiting |
| `sink_store_latency_baseline_seconds` | gauge | `method`, `batch_size=1,2_32,33_128,129_plus`; zero before learning |
| `sink_store_admission_queued_requests` | gauge | Query/Count callers waiting for a Store permit |
| `sink_store_admission_queued_bytes` | gauge | Encoded Query/Count request bytes retained while waiting |
| `sink_store_admission_waits_total` | counter | `outcome=admitted,canceled,rejected`; bounded read-queue outcomes |
| `sink_store_admission_queue_duration_seconds` | histogram | `outcome=admitted,canceled`; caller-controlled read admission wait, not backend time |
| `sink_store_permit_hold_duration_seconds` | histogram | Execution permit lifetime, including retries/cursor sends but excluding admission wait |
| `sink_store_emit_wait_duration_seconds_total` | counter | `method`; downstream Query/Scan callback time excluded from backend feedback |

Methods are the fixed set `read`, `write`, `write_visible`, `delete`,
`delete_visible`, `execute`, `query`, `count`, `scan`. Opaque Execute commands
use success/error feedback only, so their latency baseline remains zero.
The collector creates **80 series per process**, independent of request paths,
The collector creates **136 series per process**, independent of request paths,
document identities, query text, and error messages. Deployment/target labels
and historical Pod churn remain outside that count.

Compare backend execution time with `sink_batcher_request_queue_duration_seconds` to
distinguish backend slowdown from admission waiting. A mean Store duration is
Compare backend execution time with `sink_batcher_request_queue_duration_seconds`
and `sink_store_admission_queue_duration_seconds` to distinguish backend slowdown
from admission waiting. Compare permit-hold time with backend and Emit time to
identify slow consumers that retain open cursors without overloading the backend.
These counters describe different scopes; do not add Gateway/Engine observations
as if they were independent requests. A mean Store duration is
`rate(sink_store_backend_duration_seconds_total[5m])` divided by
`sum without (signal) (rate(sink_store_feedback_total[5m]))`. These count interface
calls, not database commands or records. In-flight executions may temporarily
Expand Down
111 changes: 111 additions & 0 deletions internal/app/admission_startup_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
package app

import (
"context"
"net"
"net/http"
"sync"
"testing"
"testing/synctest"
"time"

"github.com/batchstream/sink/internal/backpressure"
"github.com/batchstream/sink/internal/config"
)

type startupListener struct {
entered chan time.Time
closed chan struct{}
once sync.Once
}

func (l *startupListener) Accept() (net.Conn, error) {
l.entered <- time.Now()
<-l.closed
return nil, net.ErrClosed
}

func (l *startupListener) Close() error {
l.once.Do(func() { close(l.closed) })
return nil
}

func (l *startupListener) Addr() net.Addr {
address := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1)}
return address
}

func TestEngineCompletesStoreStartupBeforeServingReadiness(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
opts := backpressure.Options{Store: "primary", Role: "engine", MaxConcurrent: 64}
controller, err := backpressure.New(opts)
if err != nil {
t.Fatal(err)
}
listener := &startupListener{entered: make(chan time.Time, 1), closed: make(chan struct{})}
healthServer := &http.Server{Handler: http.NotFoundHandler()}
settings := config.Config{Mode: config.ModeEngine, ShutdownTimeout: time.Second}
app := &Application{config: settings, admission: controller, healthListener: listener, healthServer: healthServer}
defer app.Close()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
start := time.Now()
go func() { done <- app.Run(ctx) }()
synctest.Wait()
time.Sleep(400 * time.Millisecond)
select {
case <-listener.entered:
t.Fatal("readiness was served before Store startup completed")
default:
}
served := <-listener.entered
if served.Sub(start) < 500*time.Millisecond || served.Sub(start) > 1500*time.Millisecond {
t.Fatalf("startup staggering changed: %s", served.Sub(start))
}
var permits []*backpressure.Permit
for range 4 {
permit, _, _ := controller.TryAcquire()
if permit == nil {
t.Fatal("ready Engine still has a zero/one startup window")
}
permits = append(permits, permit)
}
for _, permit := range permits {
permit.Release()
}
cancel()
if err := <-done; err != nil {
t.Fatal(err)
}
})
}

func TestEngineStartupHonorsProcessCancellation(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
opts := backpressure.Options{Store: "primary", Role: "engine"}
controller, err := backpressure.New(opts)
if err != nil {
t.Fatal(err)
}
listener := &startupListener{entered: make(chan time.Time, 1), closed: make(chan struct{})}
healthServer := &http.Server{Handler: http.NotFoundHandler()}
settings := config.Config{Mode: config.ModeEngine, ShutdownTimeout: time.Second}
app := &Application{config: settings, admission: controller, healthListener: listener, healthServer: healthServer}
defer app.Close()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan error, 1)
go func() { done <- app.Run(ctx) }()
synctest.Wait()
cancel()
if err := <-done; err != nil {
t.Fatal(err)
}
select {
case <-listener.entered:
t.Fatal("canceled Engine started serving during initialization")
default:
}
})
}
6 changes: 5 additions & 1 deletion internal/app/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,11 @@ func New(ctx context.Context, opts Options) (*Application, error) {
app.Close()
}
}()
admissionOptions := backpressure.Options{Store: loaded.Storage.Name, Role: string(loaded.Mode), MaxConcurrent: loaded.Service.StoreMaxConcurrent}
admissionOptions := backpressure.Options{
Store: loaded.Storage.Name, Role: string(loaded.Mode), MaxConcurrent: loaded.Service.StoreMaxConcurrent,
MaxQueuedRequests: loaded.Service.Batching.Queue.MaxOperations,
MaxQueuedBytes: loaded.Service.Batching.Queue.MaxBytes,
}
app.admission, err = backpressure.New(admissionOptions)
if err != nil {
return nil, err
Expand Down
8 changes: 8 additions & 0 deletions internal/app/lifecycle.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"net/http"
"time"

"github.com/batchstream/sink/internal/config"
"github.com/batchstream/sink/internal/logging"
searchstorage "github.com/batchstream/sink/internal/storage/search"
"google.golang.org/grpc"
Expand All @@ -23,6 +24,13 @@ func (app *Application) Run(ctx context.Context) error {
cancel()
app.background.Wait()
}()
// Finish randomized Store startup before either readiness or RPCs are served.
// Routine saturation and subsequent cooldowns must not flap readiness.
if app.admission != nil && app.config.Mode == config.ModeEngine {
if err := app.admission.Wait(runContext); err != nil {
return nil
}
}
if app.gateway != nil {
app.background.Go(func() { app.gateway.Run(runContext) })
}
Expand Down
27 changes: 27 additions & 0 deletions internal/backpressure/cold_start_regression_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package backpressure

import (
"strconv"
"testing"
"testing/synctest"
)

func TestHealthyStartupOpensBoundedWindow(t *testing.T) {
for _, maximum := range []int{1, 4, 64} {
t.Run(strconv.Itoa(maximum), func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
opts := Options{Store: "primary", Role: "engine", MaxConcurrent: maximum}
controller, err := New(opts)
if err != nil {
t.Fatal(err)
}
if err := controller.Wait(t.Context()); err != nil {
t.Fatal(err)
}
if want := min(4, maximum); controller.limit != want {
t.Fatalf("healthy initial window = %d, want %d", controller.limit, want)
}
})
})
}
}
Loading
Loading