Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ jobs:

qualification:
name: External test suite
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@924aa80684708a06685625b8161fc3529c218529
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@b053b392ede7f05581921f5cbb41f69d03ece9bb
with:
suite_ref: 924aa80684708a06685625b8161fc3529c218529
suite_ref: b053b392ede7f05581921f5cbb41f69d03ece9bb
sink_ref: ${{ github.event.pull_request.head.sha || github.sha }}
sdk_ref: ${{ github.head_ref || 'main' }}
4 changes: 2 additions & 2 deletions .github/workflows/release-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ env:
jobs:
qualify:
name: Public production qualification
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@e084c611c2ce6efb2e2175cbcfc931dd497c9811
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@b053b392ede7f05581921f5cbb41f69d03ece9bb
with:
suite_ref: e084c611c2ce6efb2e2175cbcfc931dd497c9811
suite_ref: b053b392ede7f05581921f5cbb41f69d03ece9bb
sink_ref: ${{ github.event.release.tag_name }}

binaries:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/reliability.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permissions:

jobs:
soak:
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@e084c611c2ce6efb2e2175cbcfc931dd497c9811
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@b053b392ede7f05581921f5cbb41f69d03ece9bb
with:
suite_ref: e084c611c2ce6efb2e2175cbcfc931dd497c9811
suite_ref: b053b392ede7f05581921f5cbb41f69d03ece9bb
sink_ref: ${{ github.sha }}
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ quickstart. Deployment settings and their validation rules live in the
New deployments use Gateway for routing, one Engine process per Store, and one
Worker role per Store. Engine and Worker replicas scale independently and share
the execution core; Worker directly accesses its database. Gateway does not open
database or Kafka connections. The seven public RPCs remain unchanged.
database or Kafka connections. Read, Write, Query and Scan stream results; Delete, Execute and Count return scalar responses.

See [isolated topology, budgets and migration](store-isolation.md) for the private
forwarding contract and cross-Store request behavior. Execution details below
Expand Down
10 changes: 5 additions & 5 deletions docs/batching.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ alone does not release an executing document. Backend bulk calls still return
together; Sink cannot acknowledge an item whose backend result is not yet known.

Gateway splits cross-Store requests before forwarding them. Engine accepts only its
bound Store and never bypasses that check through the batching layer. Budget-sensitive
Store groups follow the scheduling policy in the [runtime guide](store-isolation.md).
bound Store and never bypasses that check through the batching layer. Independent
Store groups follow the bounded-fanout policy in the [runtime guide](store-isolation.md).

Read, Write and Delete each have one bounded queue in an Engine process. Queue
operation and byte limits apply per method; the process has three such queues.
Expand All @@ -61,9 +61,9 @@ Execution follows the callers' deadlines and cancellation; there is no default
whole-request timeout.
Admitted batches continue through memory pressure. The entire collected batch
executes without independent snapshot/output byte quotas or memory-based splitting.
Record dependencies and adapter/completion grouping still apply. Public response
allowances remain per original RPC; returned documents must fit the gRPC ceiling
before their own commit. Kafka producer buffers and Lua sandbox limits remain.
Record dependencies and adapter/completion grouping still apply. Streaming RPCs
submit one existing microbatch at a time and check each returned result against
the local gRPC ceiling before its own commit. Kafka producer buffers and Lua sandbox limits remain.
Queue limits count waiting work only, excluding dispatched batches. Graceful
shutdown drains gRPC calls before stopping batch dispatchers.
See [memory admission](design/demand-based-admission.md).
Expand Down
8 changes: 4 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ use duration strings such as `2ms`, `30s`, or `1m30s`. The `_bytes` field names
describe what the limit measures; they do not require writing raw byte counts.

This is a breaking YAML change. See the [migration guide](configuration-migration.md)
for the complete old-to-new mapping. The gRPC and SDK contracts are unchanged.
for the complete old-to-new mapping. The streaming gRPC and SDK contracts require coordinated upgrades.

## One Store per Engine or Worker

Expand Down Expand Up @@ -186,9 +186,9 @@ bounds a Lua invocation, and `consumer.processing_timeout` bounds a Worker proce
round; none of these is a whole-RPC deadline.

There is no `request.max_response_bytes` or `request.max_read_bytes` knob. Gateway
uses `grpc.max_send_message_bytes` as the transport ceiling and reserves encoded
result overhead before forwarding. Returned documents share the remaining budget
across all Store groups of the original RPC. Oversized returned writes are rejected
uses `grpc.max_send_message_bytes` as the per-message transport ceiling. Each
streamed result is checked independently; no response allowance is transferred
between Gateway and Engine. Oversized returned writes are rejected
before their own commit. Intermediate snapshots and candidate outputs use process
memory protection without independent quotas. Collected batches execute together.

Expand Down
6 changes: 3 additions & 3 deletions docs/design/demand-based-admission.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ splitting. Record ordering, adapter grouping, completion modes and conflict
retries still determine execution groups. The waiting queue retains its encoded
byte and operation limits; dispatched work no longer counts toward those limits.

The gRPC message ceiling still limits public responses. Each original RPC keeps
its response allowance across Stores, batching and retries. A requested returned
document must fit before its own write commits. This is a wire-format limit,
The gRPC message ceiling limits each streamed result independently. Gateway
and Engine do not exchange response allowances. A requested returned document
must fit the Engine limit before its own write commits. This is a wire-format limit,
separate from process memory protection. Lua sandbox and Kafka buffer limits also
remain in effect.

Expand Down
Loading
Loading