Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.

Separate role configuration and simplify process memory protection - #112

Merged
liran merged 9 commits into
mainfrom
randy/role-config-redesign
Sep 20, 2026
Merged

liran merged 9 commits into
mainfrom
randy/role-config-redesign

Conversation

@liran

@liran liran commented Sep 19, 2026 •

Copy link
Copy Markdown
Collaborator

Separate reusable Store connection/Kafka policy from Gateway, Engine and Worker files through --store-config. Gateway owns public validation, caller contexts control request lifetime, Worker owns consumption, and MongoDB concurrency belongs to the shared Store. Kafka policy applies to source and DLQ with minimum ISR 1; batching defaults to 32 operations and 2 ms. Replace allocation accounting and snapshot/output quotas with process memory watermarks (80% reject/pause, 70% resume). Admitted batches finish together; startup panics with a sizing breakdown if estimated minimum working memory cannot fit below the high watermark. Remove obsolete admission/reservation metrics and update alerts, KEDA queries, examples and migration documentation. Watermarks reduce overload but do not guarantee against OOM.

Validated with unit/race/coverage gates, lint, component regressions, real MongoDB/Elasticsearch/OpenSearch integration, public conformance plus a separate startup-panic subprocess run (48 top-level tests), Chart rendering/config checks, and a four-case fixed-resource container smoke. Live Engine/Gateway metrics expose the seven replacement memory families without the retired families. Qualification: batchstream/sink-production-suite#28; Chart: batchstream/sink-charts#12; SDK: batchstream/sink-go#24. Private forwarding is version 6 and requires matching Gateway/Engine builds; this is a breaking configuration migration.

@liran liran changed the title Separate shared Store configuration from Gateway, Engine and Worker tuning Separate role configuration and simplify process memory protection Sep 19, 2026
@liran
liran merged commit d9adcc8 into main Sep 20, 2026
15 checks passed
@liran
liran deleted the randy/role-config-redesign branch September 20, 2026 00:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant