Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ jobs:

qualification:
name: External test suite
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@e6e4058ba7d05aef6154f348526c86acc109ff1c
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@959a7e74d77f308e7167878ab175f4266c7ee11f
with:
suite_ref: e6e4058ba7d05aef6154f348526c86acc109ff1c
suite_ref: 959a7e74d77f308e7167878ab175f4266c7ee11f
sink_ref: ${{ github.event.pull_request.head.sha || github.sha }}
sdk_ref: ${{ github.head_ref || 'main' }}
4 changes: 2 additions & 2 deletions .github/workflows/release-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ env:
jobs:
qualify:
name: Public production qualification
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@e6e4058ba7d05aef6154f348526c86acc109ff1c
uses: batchstream/sink-production-suite/.github/workflows/release-qualification.yml@959a7e74d77f308e7167878ab175f4266c7ee11f
with:
suite_ref: e6e4058ba7d05aef6154f348526c86acc109ff1c
suite_ref: 959a7e74d77f308e7167878ab175f4266c7ee11f
sink_ref: ${{ github.event.release.tag_name }}

binaries:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/reliability.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permissions:

jobs:
soak:
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@e6e4058ba7d05aef6154f348526c86acc109ff1c
uses: batchstream/sink-production-suite/.github/workflows/reliability.yml@959a7e74d77f308e7167878ab175f4266c7ee11f
with:
suite_ref: e6e4058ba7d05aef6154f348526c86acc109ff1c
suite_ref: 959a7e74d77f308e7167878ab175f4266c7ee11f
sink_ref: ${{ github.sha }}
2 changes: 0 additions & 2 deletions configs/stores/primary.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ storage:
uri: mongodb://mongodb:27017 # Example endpoint; required unless uri_file is set.
# uri_file: /etc/sink-secrets/mongodb-uri # Omit uri when using its file alternative.
metadata_field: __sink # Must match across roles and replicas.
max_concurrent_writes: 64 # Per process; shared by Engine and Worker configuration.
max_concurrent_groups: 16
# For elasticsearch/opensearch, replace mongodb with search:
# search:
# endpoints: [http://opensearch:9200]
Expand Down
4 changes: 0 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,6 @@ storage:
mongodb:
uri: mongodb://mongodb:27017
metadata_field: __sink
max_concurrent_writes: 64
max_concurrent_groups: 16
kafka:
enabled: true
brokers: [kafka:9092]
Expand Down Expand Up @@ -144,8 +142,6 @@ use the lowercase spelling shown below. Storage names are also case-sensitive.
| `storage.driver` | enum string | Yes | none | `mongodb`, `elasticsearch`, `opensearch` | Adapter used by this storage instance. See [Storage driver values](#storage-driver-values). |
| `storage.mongodb.uri` | string | Conditionally | none | Valid MongoDB connection string | Required when the entry's driver is `mongodb`. |
| `storage.mongodb.metadata_field` | string | No | `__sink` | Any valid MongoDB field except `_id`; cannot contain `.`, `$`, or a null byte | Reserved top-level field where Sink stores internal metadata such as the record revision; removed from documents returned to clients. |
| `storage.mongodb.max_concurrent_writes` | positive integer | No | `64` | Integer greater than `0` | Maximum concurrent MongoDB conditional writes. |
| `storage.mongodb.max_concurrent_groups` | positive integer | No | `16` | Integer greater than `0` | Maximum collection groups executed concurrently across all calls to one store. |
| `storage.search.endpoints` | list of strings | Conditionally | none | One or more HTTP(S) endpoints | Required for `elasticsearch` and `opensearch`. |
| `storage.search.username` | string | Conditionally | empty | Any username accepted by the search service | Basic-auth username. Must be configured together with `password`. |
| `storage.search.password` | string | Conditionally | empty | Any password accepted by the search service | Basic-auth password. Must be configured together with `username`. |
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ result reduced median allocated bytes as follows (five samples):
| MongoDB | 64 KiB | 386,795 | 306,979 | 20.6% |
| MongoDB | 1 MiB | 5,909,091 | 4,576,522 | 22.6% |

Repeated results still own separate mutable documents and revisions. The read
Repeated storage results still own separate mutable documents and internal revisions. The read
measurements include the local harness and do not establish database throughput.
Encoding queue messages directly into their final envelope reduced allocations
from two to one and allocated bytes by 50% for 4 KiB, 64 KiB, and 1 MiB payloads.
Expand Down
14 changes: 7 additions & 7 deletions docs/merge-folding.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ own boundary. See [record addresses](record-addresses.md).
| One Create or Replace | One direct conditional write, as before |
| Repeated Create/Replace/Upsert, or Put mixed with Merge | Read once, evaluate in order, commit the final successful document once |
| Merge only | Read once, execute each Lua program in order, commit once |
| Write requesting `return_document` | Commit that operation independently and return its own logical document and revision; other operations may still fold |
| Write requesting `return_document` | Commit that operation independently and return its own logical document; other operations may still fold |
| Repeated Read | Fetch once, return separate result objects from that observation |
| Repeated synchronous Delete | Delete once, return the same outcome to every operation |

Expand All @@ -41,7 +41,7 @@ The folding rules below apply to operations that share a commit.
`return_document` guarantees an independent commit for the operation carrying
the option, retaining the batcher's ordering barrier. It does not guarantee
separate commits for every other operation in the same-address chain. A trailing
run without returned documents may still fold and share a revision. Only APPLIED
run without returned documents may still fold into one commit. Only APPLIED
operations requesting a document receive one. Asynchronous completion rejects this option.
Returned documents are the logical Put/Merge outputs; backend-generated fields
and ingest transformations are excluded. See [returned writes](native-access.md#returned-writes).
Expand All @@ -56,8 +56,8 @@ and ingest transformations are excluded. See [returned writes](native-access.md#
4. A failed condition or Lua program leaves the working state unchanged. Commit
the final successful document using the original snapshot's revision, or a
record-not-exists condition for an initially absent document.
5. After a successful commit, successful operations return APPLIED with the
same final revision. Preserve the individually evaluated failures and original
5. After a successful commit, successful operations return APPLIED. The revision
remains internal to Sink. Preserve the individually evaluated failures and original
operation indexes/RPC boundaries. If no operation succeeds, return the
evaluated failures without writing.

Expand Down Expand Up @@ -101,7 +101,7 @@ requirements still apply; folding does not provide exactly-once execution or
batch transactions.

Reads share one backend observation per address but produce independent payload
and revision copies. Every streamed result, including repetitions, must fit its
copies. Every streamed result, including repetitions, must fit its
own message ceiling. Internal batch callers retain their local response limits.
The backend's unique-document read remains bounded by the active microbatch. Missing/error results are returned to all matching
operations.
Expand Down Expand Up @@ -136,7 +136,7 @@ conflict recomputation, failed/lost commits, output and admission bounds, read
response copies and repeated-key budgets, full address isolation, cross-RPC
folding, and preservation of asynchronous intents. Existing completion-mode and
cancellation regressions remain applicable. Tagged MongoDB/Search integration
cases exercise mixed writes, final revisions, BSON dates, and search visibility.
cases exercise mixed writes, final committed state, BSON dates, and search visibility.
Additional regressions cover multiline Bulk JSON, bounded Replace revision
retries (including unknown acknowledgements and successful siblings), per-RPC
budget isolation, memory-limited splitting, and cross-batch dependencies and
Expand Down Expand Up @@ -213,7 +213,7 @@ Reproduce with `GOMAXPROCS=4 go test ./internal/service -run '^$'
benchmark and test helpers into a detached baseline worktree. Race tests separately
cover concurrent servers, whole-run conflict retries, cancellation, failed commits,
and lost acknowledgements. Tagged integration tests check real backend visibility,
concurrent folded updates, shared revisions, and BSON types.
concurrent folded updates, shared commits, and BSON types.

### Contended commits

Expand Down
15 changes: 8 additions & 7 deletions docs/native-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,20 +405,21 @@ retries. Older servers without the detail keep single-attempt behavior.

Set `WriteOperation.return_document` (SDK `WithReturnedDocument()` or
`Record.ReturnDocument`) to receive the operation's logical Put/Merge document
alongside its successful revision. Only APPLIED results include documents.
after its successful commit. Only APPLIED results include documents.
Record revisions remain internal to Sink and are not returned by Read or Write.
`RETURN_AFTER_ACCEPTED` with this option is rejected before publishing anything;
use `WAIT_UNTIL_APPLIED` or `WAIT_UNTIL_VISIBLE`.

Sink takes the document from the successful commit candidate, never from a later
Read. Every operation requesting a returned document commits independently of
preceding and following operations. Other operations can still share a folded
commit and revision; setting the option on one operation does not promise
commit; setting the option on one operation does not promise
independent commits for the entire same-address chain. For three increments
where only the first requests a document, the first can return value 1 and
revision A while the remaining two share the commit of value 3 and revision B.
Request a document on each operation when each needs its own committed output
and revision. A conflicting attempt's speculative output is not returned. A
definite CAS conflict may recompute against a fresh snapshot; transport failures
where only the first requests a document, the first can return value 1 while
the remaining two share the commit of value 3.
Request a document on each operation when each needs its own committed output.
A conflicting attempt's speculative output is not returned. A definite CAS
conflict may recompute against a fresh snapshot; transport failures
and unknown commit outcomes are not automatically replayed.

The returned document is the logical document submitted to the backend. Sink
Expand Down
2 changes: 1 addition & 1 deletion docs/reliability.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ retries safe.

Returned-document chains execute in ordered rounds: independent direct Puts
commit before a round starts conditional reads, without concurrent unbudgeted
snapshots. Each returned operation retains its own committed revision/document.
snapshots. Each returned operation retains its own committed document.
Scan has separate byte, request and per-store sublimits in addition to global
admission. Defaults reserve half the configured capacity for ordinary work
(count limits round up to one for a total limit of one). Two default BSON scans
Expand Down
1 change: 0 additions & 1 deletion examples/kubernetes/sink.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ grpc: {address: ":8080", max_receive_message_bytes: 64MiB, max_send_message_byte
health: {address: ":8081"}
prometheus: {enabled: true, address: ":9090"}
execution:
mongodb: {max_concurrent_writes: 64, max_concurrent_groups: 16}
merge: {max_attempts: 3}
batching:
max_wait: 2ms
Expand Down
2 changes: 0 additions & 2 deletions examples/quickstart/store.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ storage:
uri: mongodb://mongo:27017/?replicaSet=rs0 # Example endpoint; required unless uri_file is set.
# uri_file: /etc/sink-secrets/mongodb-uri # Omit uri when using its file alternative.
metadata_field: __sink # Must match across roles and replicas.
max_concurrent_writes: 64 # Per process; shared by Engine and Worker configuration.
max_concurrent_groups: 16
# For elasticsearch/opensearch, replace mongodb with search:
# search:
# endpoints: [http://opensearch:9200]
Expand Down
Loading
Loading