Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ jobs:

qualification:
name: External test suite
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@c95932c11d922345deec8e03895129c33ee885b6
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@bb4db31e534339a4d0da7134fd703ccdb84f4707
with:
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
suite_ref: bb4db31e534339a4d0da7134fd703ccdb84f4707
sink_ref: ${{ github.event.pull_request.head.sha || github.sha }}
sdk_ref: ${{ github.head_ref || 'main' }}
4 changes: 2 additions & 2 deletions .github/workflows/release-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@ jobs:
qualify:
name: Public production qualification
needs: release
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@c95932c11d922345deec8e03895129c33ee885b6
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@bb4db31e534339a4d0da7134fd703ccdb84f4707
with:
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
suite_ref: bb4db31e534339a4d0da7134fd703ccdb84f4707
sink_ref: ${{ needs.release.outputs.revision }}

binaries:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/reliability.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permissions:

jobs:
soak:
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@c95932c11d922345deec8e03895129c33ee885b6
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@bb4db31e534339a4d0da7134fd703ccdb84f4707
with:
suite_ref: c95932c11d922345deec8e03895129c33ee885b6
suite_ref: bb4db31e534339a4d0da7134fd703ccdb84f4707
sink_ref: ${{ github.sha }}
3 changes: 3 additions & 0 deletions configs/engine.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ batching:
producer:
max_buffered_bytes: 64MiB
execution:
queue:
max_tasks: 10000 # Ready batches and individual native calls; not operation count.
max_bytes: 128MiB # Shared waiting bytes across collection and admission.
merge:
max_attempts: 3 # Revision-conflict attempts.
lua:
Expand Down
4 changes: 3 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,9 @@ batches. Automatic mutation batches also share adapter resource and completion
mode, limiting refresh waits to their own dataset. Completed write document
chains release their scheduling dependencies; each original RPC returns when
all of its results are final. MongoDB can use collection-level bulk operations, and Elasticsearch or
OpenSearch can use `_mget` and `_bulk`. Queue limits and adaptive Store admission constrain dispatched work. A shared
OpenSearch can use `_mget` and `_bulk`. Ready batches and Native calls share one bounded FIFO; queue limits and adaptive
Store admission constrain dispatched work. Encoded waiting bytes remain charged
once across batch collection and admission, while a batch counts as one ready task. A shared
local window covers record and Native methods; its feedback comes only from real
Store operations. See [Store backpressure](design/store-backpressure.md).

Expand Down
13 changes: 9 additions & 4 deletions docs/batching.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ is always active; every single-store `Read` uses this path. `Write` and
`Delete` use it for `WAIT_UNTIL_APPLIED` and `WAIT_UNTIL_VISIBLE`;
`RETURN_AFTER_ACCEPTED` bypasses it because Kafka already batches asynchronous
mutations. Read, write, and delete have independent queues within each store.
Record dependencies stay method-local; Store admission is shared across all
methods. A congested Store pauses subsequent dispatch for that Store.
Record dependencies stay method-local. Ready batches and all Native calls enter
one FIFO before obtaining Store execution permits. A congested Store pauses subsequent dispatch for that Store.

`batching` configures batch and queue limits; batching is always active.
`batching` configures formation targets and collection bounds; batching is always
active. `execution.queue` separately bounds ready tasks and shared waiting bytes.

The first queued request starts `batching.max_wait`.
Collection stops when that timer expires or adding another request would cross
Expand Down Expand Up @@ -50,7 +51,11 @@ bound Store and never bypasses that check through the batching layer. Independen
Store groups follow the bounded-fanout policy in the [runtime guide](store-isolation.md).

Read, Write and Delete each have one bounded queue in an Engine process. Queue
operation and byte limits apply per method; the process has three such queues.
operation and byte limits apply per method; the process has three such collection
queues. Each retains at most one ready batch in the shared admission FIFO without
blocking its event loop. If that FIFO is full, already accepted work stays
upstream. `execution.queue.max_bytes` counts waiting bytes across both stages
once, while `execution.queue.max_tasks` counts ready tasks, not operations.
Other Stores run in separate Engine processes. A
new single-store request that would cross its queue's limit fails with gRPC
`RESOURCE_EXHAUSTED` and is not applied. Requests canceled before dispatch are
Expand Down
13 changes: 9 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ SDK examples, connection discovery, replica changes and deployment boundaries.
## Synchronous request batching

`batching` sets the collection targets and the per-method
`queue` limits. Batching is always active. See [batching behavior](batching.md)
`queue` limits. `execution.queue` sets ready-task capacity and shared waiting
bytes independently. Batching is always active. See [batching behavior](batching.md)
for queue admission, ordering, execution budgets, and completion boundaries.

## Prometheus metrics
Expand Down Expand Up @@ -150,12 +151,14 @@ use the lowercase spelling shown below. Storage names are also case-sensitive.
| `storage.search.api_key` | string | No | empty | Any API key accepted by the search service | API key used instead of basic authentication. |
| `request.max_operations` | positive integer | No | `1000` | Integer greater than `0` | Maximum operation count accepted in one Read, Write, or Delete batch request. |
| `max_concurrent` | positive integer | No | `64` for MongoDB; `128` for Elasticsearch/OpenSearch | Integer from `1` through `4096` | Optional adaptive-window ceiling per Engine/Worker process for this Store. Not a backend capacity estimate or replica-based quota. |
| `execution.queue.max_tasks` | positive integer | No | `10000` | Integer greater than `0`; Engine only | Maximum ready tasks in the single Store admission FIFO. A collected batch is one task, not one task per operation. |
| `execution.queue.max_bytes` | byte size | No | max(`128MiB`, `grpc.max_receive_message_bytes`, `batching.max_bytes`) | Covers one legal gRPC request and one batch; Engine only | Shared encoded waiting bytes across all collection queues and Native/batch admission tasks, charged once. Waiting ends only with execution capacity or caller cancellation/deadline. |
| `execution.merge.max_attempts` | positive integer | No | `3` | Integer greater than `0` | Maximum revision-conflict attempts for Merge and folded conditional Put chains. |
| `batching.max_wait` | duration string | No | `2ms` | Positive Go duration within the bounds below | Maximum collection delay measured from the first request in a batch. |
| `batching.max_operations` | positive integer | No | `32` | Positive integer | Operation target for one automatically formed batch; a larger valid RPC still executes alone. See [choosing limits](batching.md#choosing-limits). |
| `batching.max_bytes` | byte size | No | `16MiB` | Size greater than `0B` | Encoded-byte target for one automatically formed batch; one larger valid RPC still runs alone. |
| `batching.queue.max_operations` | positive integer | No | max(`10000`, `batching.max_operations`) | Integer at least `batching.max_operations` | Maximum operations waiting in each batch-method queue; also the request count bound for the shared Query/Count admission FIFO. |
| `batching.queue.max_bytes` | byte size | No | max(`128MiB`, `grpc.max_receive_message_bytes`) | Size at least `grpc.max_receive_message_bytes` and `batching.max_bytes` | Maximum encoded request bytes waiting in each batch-method queue and in the separate Query/Count admission FIFO. Query/Count waiting ends only with capacity or caller-context cancellation/deadline, not a server-added timeout. |
| `batching.queue.max_operations` | positive integer | No | max(`10000`, `batching.max_operations`) | Integer at least `batching.max_operations` | Maximum operations waiting in each batch-method collection queue. Not an admission task limit or concurrency estimate. |
| `batching.queue.max_bytes` | byte size | No | max(`128MiB`, `grpc.max_receive_message_bytes`) | Size at least `grpc.max_receive_message_bytes` and `batching.max_bytes` | Maximum encoded request bytes retained by each batch-method collection queue, including its selected admission candidate. The Store-wide execution.queue.max_bytes budget also applies. |
| `execution.merge.lua.timeout` | duration string | No | `100ms` | Positive Go duration within the bounds below | Maximum wall-clock duration of one Lua execution. |
| `execution.merge.lua.max_source_bytes` | byte size | No | `64KiB` | Size greater than `0B` | Maximum Lua source size per merge operation. |
| `execution.merge.lua.max_result_bytes` | byte size | No | `16MiB` | Size greater than `0B` | Maximum input/current document and encoded result bytes; expanded output is also bounded before conversion. |
Expand Down Expand Up @@ -224,7 +227,9 @@ available ceiling. This catches undersized configurations; it does not guarantee
against OOM under concurrent load. See the [sizing formula and runtime behavior](design/process-memory-admission.md)
and [memory metrics and KEDA](observability.md#memory-capacity-and-keda).

Use `memory` for process pressure and `batching.queue` for Engine waiting queues.
Use `memory` for process pressure, `batching.queue` for per-method collection
bounds, and `execution.queue` for ready-task capacity and shared waiting bytes.
Do not derive task limits or execution concurrency from operation counts.

Store admission is shared across Read/Write/Delete/Native operations. It counts
admitted Storage executions; adapter group fanout remains unchanged. Sink sets
Expand Down
Loading
Loading