Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 42 additions & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,25 @@ concurrency:
cancel-in-progress: true

jobs:
resolve-server:
runs-on: ubuntu-latest
outputs:
revision: ${{ steps.resolve.outputs.revision }}
steps:
- id: resolve
env:
CANDIDATE_REF: ${{ github.head_ref || 'main' }}
run: |
remote=https://github.com/batchstream/sink.git
revision=$(git ls-remote --heads "$remote" "refs/heads/$CANDIDATE_REF" | awk '{print $1}')
if [[ -z "$revision" ]]; then
revision=$(git ls-remote --heads "$remote" refs/heads/main | awk '{print $1}')
fi
[[ "$revision" =~ ^[0-9a-f]{40}$ ]]
echo "revision=$revision" >> "$GITHUB_OUTPUT"

validate:
needs: resolve-server
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
Expand All @@ -38,10 +56,20 @@ jobs:
helm template sink ./charts/sink --kube-version 1.30.0 \
-f examples/cluster-values.yaml -f examples/logging-values.yaml --set metrics.enabled=true \
| kubeconform -strict -summary -kubernetes-version 1.30.0
- name: Check complete configurations with released Sink
run: python tests/check_runtime.py
- name: Check out matching Sink candidate
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: batchstream/sink
path: sink-candidate
ref: ${{ needs.resolve-server.outputs.revision }}
- name: Check generated files with candidate parser
run: |
go -C sink-candidate build -o "$RUNNER_TEMP/sink" ./cmd/sink
python tests/check_runtime.py --sink-binary "$RUNNER_TEMP/sink"


qualification:
needs: resolve-server
name: Kubernetes qualification (${{ matrix.scenario }})
strategy:
fail-fast: false
Expand All @@ -65,8 +93,19 @@ jobs:
run: |
python -m pip install -r requirements-dev.txt
go install sigs.k8s.io/kind@v0.33.0
- name: Check out matching Sink candidate
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: batchstream/sink
path: sink-candidate
ref: ${{ needs.resolve-server.outputs.revision }}
- name: Build compatible rollout candidates
run: |
docker build --build-arg VERSION=chart-candidate-a -t sink-chart-candidate:a sink-candidate
docker build --build-arg VERSION=chart-candidate-b -t sink-chart-candidate:b sink-candidate
- name: Qualify rollouts, Store lifecycle and KEDA with business traffic
run: make integration SCENARIO=${{ matrix.scenario }}
run: python tests/integration/run.py --scenario ${{ matrix.scenario }} --sink-image sink-chart-candidate:a --upgrade-image sink-chart-candidate:b

- name: Preserve qualification evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
Expand Down
49 changes: 29 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ configuration with shared Store settings and managed operational defaults;
external Secrets contain credential values only.

One release deploys a Sink cluster: a shared Gateway and independently configured
Engine and Worker Deployments for each Store. Chart `0.7.0` generates runtime
configuration and references individual credential keys in external Secrets. It targets Sink **0.18.0**, SDK **0.8.0**, and
Engine and Worker Deployments for each Store. Chart `0.8.0` generates runtime
configuration and references individual credential keys in external Secrets. It targets Sink **0.19.0**, SDK **0.8.0**, and
Kubernetes **1.30+** with native lifecycle sleep hooks enabled.

- Stable Store names, generated headless discovery, staged activation and retirement.
Expand All @@ -24,6 +24,18 @@ budget assumptions, not a zero-error guarantee under arbitrary DNS/backend failu

## Install

This configuration requires Sink 0.19.0. Until that release is published, build
its matching candidate and override `image.repository`, `image.tag`, and
`image.digest: ""`. Do not pair Chart 0.8 with a 0.18 or older image.

Each Store gets one `store.yaml` ConfigMap shared by Engine and Worker. Both roles
load it using `--store-config`; their own `sink.yaml` holds only role tuning.
Use `gateway.runtime.forwarding` / `request`, `engineDefaults.runtime.execution` /
`batching` / `producer`, and `workerDefaults.runtime.execution` / `consumer`.
Consumer group belongs in `stores.<name>.worker.runtime.consumer.group_id`.
There is no service-wide request timeout; callers control request lifetime.


Create the release namespace and provision `sink-mongo-v1` and `sink-archive-v1`
Secrets there. Each example references a key named `uri` containing its complete
MongoDB URI. Use your secret manager; never commit credentials to values files.
Expand Down Expand Up @@ -127,21 +139,18 @@ Rendering tests are offline. `make integration` is an explicit Docker/Kind test;
see [tests/integration](tests/integration/README.md). CI also checks Kubernetes
schemas and complete example configurations against the pinned Sink image.

### Demand-based memory admission

The pinned Sink 0.18.0 image supports demand-based memory admission through
`gateway.runtime.memory`, `engineDefaults.runtime.memory`, and
`workerDefaults.runtime.memory`, with per-Store overrides under each role.
The map supports `max_bytes`, `burst_percent` (1–99), and `wait_timeout`.
Leave it empty for server automatic sizing and its measured 10% reserve default.
Empty maps are omitted from generated YAML, which also permits explicit legacy
image overrides during staged upgrades. Keep these fields empty on Sink 0.16.0.

[The memory KEDA overlay](examples/memory-keda-values.yaml) supplies managed-capacity
pressure and temporary-rejection signals with `metricType: Value`. Install KEDA,
configure Prometheus scraping and adjust job/namespace selectors to one Gateway
Deployment before enabling it. Engine selectors must also isolate the Store;
Workers should retain Kafka lag triggers. Missing metrics must remain a scaler
error rather than zero load. Upgrade Engines before Gateways for the private
framed-response protocol; roll back Gateways first. Applying the example changes
scaling configuration; plan and verify the rollout using the operational runbook.
### Process memory admission

Sink 0.19.0 uses high/low watermarks through `gateway.runtime.memory`,
`engineDefaults.runtime.memory`, and `workerDefaults.runtime.memory`, with per-Store
overrides. Supported fields are `max_bytes`, `high_watermark_percent` (default 80),
and `low_watermark_percent` (default 70). Require `0 < low < high < 100`.
An empty map selects server defaults. Startup panics when estimated minimum
working memory does not fit below the high watermark; larger gRPC/Lua/Kafka
buffers may require larger Pod limits. See [server sizing](https://github.com/batchstream/sink/blob/main/docs/design/demand-based-admission.md).

[The memory KEDA overlay](examples/memory-keda-values.yaml) uses observed memory
pressure and RPC rejection ratios with `metricType: Value`. Configure Prometheus
selectors for one role/Deployment/Store and retain Worker Kafka lag triggers.
Missing data remains a scaler error. Gateway and Engine must use matching private
protocol versions; incompatible versions require an isolated cutover.
4 changes: 2 additions & 2 deletions charts/sink/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ apiVersion: v2
name: sink
description: Store-isolated Sink clusters with bounded rollouts and independent autoscaling.
type: application
version: 0.7.0
appVersion: "0.18.0"
version: 0.8.0
appVersion: "0.19.0"
kubeVersion: ">=1.30.0-0"
home: https://github.com/batchstream/sink
sources:
Expand Down
12 changes: 6 additions & 6 deletions charts/sink/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ helm.sh/chart: {{ printf "%s-%s" .root.Chart.Name .root.Chart.Version | quote }}
{{- end -}}
{{- define "sink.preStop" -}}
{{- $minimum := 0 -}}
{{- if eq .role "engine" -}}{{- $minimum = add (include "sink.discoverySeconds" .root | int) .root.Values.requestTimeoutSeconds .root.Values.discovery.safetyMarginSeconds -}}{{- end -}}
{{- if eq .role "gateway" -}}{{- $minimum = add .root.Values.discovery.clientWithdrawalSeconds .root.Values.requestTimeoutSeconds .root.Values.discovery.safetyMarginSeconds -}}{{- end -}}
{{- if eq .role "engine" -}}{{- $minimum = add (include "sink.discoverySeconds" .root | int) .root.Values.discovery.safetyMarginSeconds -}}{{- end -}}
{{- if eq .role "gateway" -}}{{- $minimum = add .root.Values.discovery.clientWithdrawalSeconds .root.Values.discovery.safetyMarginSeconds -}}{{- end -}}
{{- $seconds := $minimum -}}
{{- if ne .settings.pod.preStopSeconds nil -}}{{- $seconds = int .settings.pod.preStopSeconds -}}{{- end -}}
{{- if lt (int $seconds) (int $minimum) -}}{{- fail (printf "%s/%s preStopSeconds must be >= %d (discovery + request + margin)" .role .store $minimum) -}}{{- end -}}
{{- if lt (int $seconds) (int $minimum) -}}{{- fail (printf "%s/%s preStopSeconds must be >= %d (discovery + margin)" .role .store $minimum) -}}{{- end -}}
{{- $seconds -}}
{{- end -}}
{{- define "sink.grace" -}}
Expand All @@ -88,10 +88,10 @@ helm.sh/chart: {{ printf "%s-%s" .root.Chart.Name .root.Chart.Version | quote }}
{{- $routes = append $routes (dict "store" $store "target" (printf "dns:///%s.%s.svc.%s:8080" $name $.Release.Namespace $.Values.clusterDomain) "tls" (dict "insecure" true)) -}}
{{- end -}}
{{- end -}}
{{- $gateway := mergeOverwrite (deepCopy .Values.gateway.runtime.gateway) (dict "routes" $routes "dns_refresh_interval" (printf "%ds" (int .Values.discovery.dnsRefreshSeconds))) -}}
{{- $service := mergeOverwrite (deepCopy .Values.gateway.runtime.service) (dict "request" (dict "timeout" (printf "%ds" (int .Values.requestTimeoutSeconds)))) -}}
{{- $gateway := mergeOverwrite (deepCopy .Values.gateway.runtime.forwarding) (dict "routes" $routes "dns_refresh_interval" (printf "%ds" (int .Values.discovery.dnsRefreshSeconds))) -}}
{{- $grpc := mergeOverwrite (deepCopy .Values.gateway.runtime.grpc) (dict "address" ":8080") -}}
{{- $config := (dict "mode" "gateway" "gateway" $gateway "service" $service "grpc" $grpc "health" (dict "address" ":8081") "prometheus" (dict "enabled" .Values.metrics.enabled "address" ":9090") "shutdown_timeout" (printf "%ds" (int .Values.gateway.pod.shutdownTimeoutSeconds))) -}}
{{- $config := (dict "mode" "gateway" "forwarding" $gateway "grpc" $grpc "health" (dict "address" ":8081") "prometheus" (dict "enabled" .Values.metrics.enabled "address" ":9090") "shutdown_timeout" (printf "%ds" (int .Values.gateway.pod.shutdownTimeoutSeconds))) -}}
{{- with .Values.gateway.runtime.request -}}{{- $_ := set $config "request" . -}}{{- end -}}
{{- with .Values.gateway.runtime.memory -}}{{- $_ := set $config "memory" . -}}{{- end -}}
{{- with .Values.gateway.runtime.logging -}}{{- $_ := set $config "logging" . -}}{{- end -}}
{{- toYaml $config -}}
Expand Down
31 changes: 21 additions & 10 deletions charts/sink/templates/_store.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@
{{- end -}}

{{- define "sink.storeConfig" -}}
{{- $storage := dict "name" .store "driver" .storage.driver -}}
{{- $storage := dict "driver" .storage.driver -}}
{{- if eq .storage.driver "mongodb" -}}
{{- $mongo := dict "metadata_field" (default "__sink" .storage.mongodb.metadataField) "max_concurrent_writes" (default 64 .storage.mongodb.maxConcurrentWrites) "max_concurrent_groups" (default 16 .storage.mongodb.maxConcurrentGroups) -}}
{{- $mongo := dict "metadata_field" (default "__sink" .storage.mongodb.metadataField) -}}
{{- $_ := set $mongo "max_concurrent_writes" (default 64 .storage.mongodb.maxConcurrentWrites) -}}
{{- $_ := set $mongo "max_concurrent_groups" (default 16 .storage.mongodb.maxConcurrentGroups) -}}
{{- $_ := set $mongo "uri_file" "/etc/sink-secrets/mongodb-uri" -}}
{{- $_ := set $storage "mongodb" $mongo -}}
{{- else -}}
Expand All @@ -31,15 +33,24 @@
{{- end -}}
{{- $_ := set $storage "search" $search -}}
{{- end -}}
{{- $config := dict "name" .store "storage" $storage -}}
{{- if .kafka -}}
{{- $kafka := deepCopy (default dict .kafka.runtime) -}}
{{- $managed := dict "enabled" true "brokers" .kafka.brokers "topic" (dict "name" .kafka.topic "partitions" .kafka.partitions "replication_factor" (default 3 .kafka.replicationFactor) "min_insync_replicas" (default 2 .kafka.minInSyncReplicas)) "consumer" (dict "group_id" .kafka.consumerGroup) -}}
{{- $_ := set $storage "kafka" (mergeOverwrite $kafka $managed) -}}
{{- end -}}
{{- $service := mergeOverwrite (deepCopy .settings.runtime.service) (dict "request" (dict "timeout" (printf "%ds" (int .root.Values.requestTimeoutSeconds)))) -}}
{{- $grpc := mergeOverwrite (deepCopy .settings.runtime.grpc) (dict "address" ":8080") -}}
{{- $config := (dict "mode" .role "storage" $storage "service" $service "grpc" $grpc "health" (dict "address" ":8081") "prometheus" (dict "enabled" .root.Values.metrics.enabled "address" ":9090") "shutdown_timeout" (printf "%ds" (int .settings.pod.shutdownTimeoutSeconds))) -}}
{{- with .settings.runtime.memory -}}{{- $_ := set $config "memory" . -}}{{- end -}}
{{- with .settings.runtime.logging -}}{{- $_ := set $config "logging" . -}}{{- end -}}
{{- $managed := dict "enabled" true "brokers" .kafka.brokers "partitions" .kafka.partitions "replication_factor" (default 3 .kafka.replicationFactor) "min_insync_replicas" (default 1 .kafka.minInSyncReplicas) "topic" (dict "name" .kafka.topic) -}}
{{- $_ := set $config "kafka" (mergeOverwrite $kafka $managed) -}}
{{- end -}}
{{- toYaml $config -}}
{{- end -}}

{{- define "sink.roleConfig" -}}
{{- $config := deepCopy .settings.runtime -}}
{{- $_ := set $config "mode" .role -}}
{{- $_ := set $config "health" (dict "address" ":8081") -}}
{{- $_ := set $config "prometheus" (dict "enabled" .root.Values.metrics.enabled "address" ":9090") -}}
{{- $_ := set $config "shutdown_timeout" (printf "%ds" (int .settings.pod.shutdownTimeoutSeconds)) -}}
{{- if eq .role "engine" -}}
{{- $_ := set $config "grpc" (mergeOverwrite (default dict $config.grpc) (dict "address" ":8080")) -}}
{{- end -}}
{{- range $key, $value := $config -}}{{- if empty $value -}}{{- $_ := unset $config $key -}}{{- end -}}{{- end -}}
{{- toYaml $config -}}
{{- end -}}
2 changes: 1 addition & 1 deletion charts/sink/templates/autoscaling.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ spec:
{{- end }}
metadata:
bootstrapServers: {{ join "," $c.kafka.brokers | quote }}
consumerGroup: {{ $c.kafka.consumerGroup | quote }}
consumerGroup: {{ $c.settings.runtime.consumer.group_id | quote }}
topic: {{ $c.kafka.topic | quote }}
lagThreshold: {{ default 1000 $c.kafka.lagThreshold | quote }}
activationLagThreshold: "0"
Expand Down
25 changes: 21 additions & 4 deletions charts/sink/templates/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,22 @@
{{- $engine := mergeOverwrite (deepCopy $.Values.engineDefaults) (default dict $settings.engine) }}
{{- $worker := mergeOverwrite (deepCopy $.Values.workerDefaults) (default dict $settings.worker) }}
{{- $kafka := omit (default dict $settings.kafka) "lagThreshold" "runtime" }}
{{- if $kafka }}{{- $_ := set $kafka "consumerGroup" (default "" $worker.runtime.consumer.group_id) }}{{- end }}
{{- $kafkaRuntime := default dict (default dict $settings.kafka).runtime }}
{{- $deadLetter := default dict $kafkaRuntime.dead_letter }}
{{- with $deadLetter.topic }}{{- $_ := set $kafka "deadLetterTopic" . }}{{- end }}
{{- $engineConfig := include "sink.storeConfig" (dict "root" $ "role" "engine" "store" $store "settings" $engine "storage" $settings.storage "kafka" (default dict $settings.kafka)) }}
{{- with $deadLetter.name }}{{- $_ := set $kafka "deadLetterTopic" . }}{{- end }}
{{- $engineConfig := include "sink.roleConfig" (dict "root" $ "role" "engine" "store" $store "settings" $engine "storage" $settings.storage "kafka" (default dict $settings.kafka)) }}
{{- $entry := dict "state" $settings.state "kafka" $kafka "workerEnabled" $worker.enabled "engineHash" (toJson (dict "engine" $engine "storage" $settings.storage "kafkaRuntime" (default dict $settings.kafka) "credentialRevision" (default "" $settings.credentialRevision) "runtime" $engineConfig "image" $.Values.image "discovery" $.Values.discovery) | sha256sum) }}
{{- $_ := set $current $store $entry }}
{{- if $kafka }}
{{- range $pair := list (dict "name" "topic" "value" $kafka.topic "seen" $topics) (dict "name" "deadLetterTopic" "value" (default (printf "%s.dlq" (trim $kafka.topic)) $deadLetter.topic) "seen" $topics) (dict "name" "consumerGroup" "value" $kafka.consumerGroup "seen" $groups) }}
{{- range $pair := list (dict "name" "topic" "value" $kafka.topic "seen" $topics) (dict "name" "deadLetterTopic" "value" (default (printf "%s.dlq" (trim $kafka.topic)) $deadLetter.name) "seen" $topics) (dict "name" "consumerGroup" "value" $kafka.consumerGroup "seen" $groups) }}
{{- $identity := trim $pair.value }}
{{- if $identity }}
{{- if hasKey $pair.seen $identity }}{{- fail (printf "Kafka %s must be unique per Store" $pair.name) }}{{- end }}
{{- $_ := set $pair.seen $identity true }}
{{- end }}
{{- end }}
{{- end }}
{{- if $hasInventory }}
{{- $old := get $previous $store }}
{{- if not $old }}
Expand Down Expand Up @@ -102,6 +105,20 @@ metadata:
{{- include "sink.labels" $c | nindent 4 }}
data:
sink.yaml: |
{{- include "sink.storeConfig" $c | nindent 4 }}
{{- include "sink.roleConfig" $c | nindent 4 }}
{{- end }}
{{- end }}

{{- range $store, $settings := .Values.stores }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "sink.componentName" (dict "root" $ "role" "store" "store" $store) }}
labels:
app.kubernetes.io/instance: {{ $.Release.Name | quote }}
app.kubernetes.io/managed-by: {{ $.Release.Service }}
data:
store.yaml: |
{{- include "sink.storeConfig" (dict "store" $store "storage" $settings.storage "kafka" (default dict $settings.kafka)) | nindent 4 }}
{{- end }}
14 changes: 10 additions & 4 deletions charts/sink/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ spec:
{{- if eq $c.role "gateway" }}
sink.batchstream.io/config-checksum: {{ include "sink.gatewayConfig" $ | sha256sum | quote }}
{{- else }}
sink.batchstream.io/config-checksum: {{ include "sink.storeConfig" $c | sha256sum | quote }}
sink.batchstream.io/config-checksum: {{ printf "%s\n%s" (include "sink.roleConfig" $c) (include "sink.storeConfig" $c) | sha256sum | quote }}
sink.batchstream.io/credential-revision: {{ $c.credentialRevision | quote }}
{{- end }}
{{- with $s.pod.annotations }}{{ toYaml . | nindent 8 }}{{- end }}
Expand Down Expand Up @@ -82,7 +82,7 @@ spec:
- name: sink
image: {{ if $image.digest }}{{ printf "%s@%s" $image.repository $image.digest | quote }}{{ else }}{{ printf "%s:%s" $image.repository $image.tag | quote }}{{ end }}
imagePullPolicy: {{ $image.pullPolicy }}
args: ["--config", "/etc/sink/sink.yaml"]
args: ["--config", "/etc/sink/sink.yaml"{{ if ne $c.role "gateway" }}, "--store-config", "/etc/sink/store.yaml"{{ end }}]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
Expand Down Expand Up @@ -146,8 +146,14 @@ spec:
{{- end }}
volumes:
- name: config
configMap:
name: {{ $name }}
projected:
sources:
- configMap:
name: {{ $name }}
{{- if ne $c.role "gateway" }}
- configMap:
name: {{ include "sink.componentName" (dict "root" $ "role" "store" "store" $c.store) }}
{{- end }}
{{- if ne $c.role "gateway" }}
{{- $sources := include "sink.secretSources" $c | fromJsonArray }}
{{- if $sources }}
Expand Down
Loading
Loading