Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ For vulnerabilities, follow [SECURITY.md](SECURITY.md).
## Work locally

Install Helm, Python 3 and the Go version the chart's workflows use, then follow
[Develop and release](README.md#develop-and-release):
[Development](docs/development.md):

```sh
python3 -m venv .venv
Expand All @@ -32,5 +32,5 @@ rendered alerts and the autoscaling query; without it, that test is skipped.

- Keep the change focused and explain the observed problem and resulting behavior.
- Add a regression test when fixing behavior.
- Update `values.schema.json`, the [examples](examples) and the README when values change.
- When pinning a new server, pin it in [weir-tests](https://github.com/batchstream/weir-tests) first, then change the source annotation, `appVersion`, image digest, workflow pins, the README's references and the values in `tests/test_chart.py` together.
- Update `values.schema.json`, the [examples](examples), the README and [docs](docs) when values change.
- When pinning a new server, pin it in [weir-tests](https://github.com/batchstream/weir-tests) first, then change the source annotation, `appVersion`, image digest, workflow pins, the references in the README and [docs](docs) and the values in `tests/test_chart.py` together.
182 changes: 126 additions & 56 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ vulnerabilities even if you cannot reproduce them on `main`.
## Deployment guidance

Application traffic uses plaintext HTTP/2 inside a trusted, isolated network. See
[Network and operations](README.md#network-and-operations).
[Network](docs/operations.md#network).
15 changes: 15 additions & 0 deletions docs/autoscaling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Autoscaling

How to scale a Store group with KEDA and Prometheus.

## Example overlay

Overlay [examples/prometheus-autoscaling.yaml](../examples/prometheus-autoscaling.yaml) on the `mongo` release in namespace `application`. Adjust the namespace, release identity, Prometheus endpoint and scraper labels for your deployment. Its query sums this exact group's Weir process CPU; `AverageValue` lets HPA divide that total across replicas, and the threshold of 1.4 cores is 70% of the default per-replica limit, so change it with `resources`. Scale on Weir's own CPU, not on queued operations or active executions: those rise whenever the backend slows down, and replicas added then bring no capacity, so a queue trigger holds the group at its maximum. Each replica runs fewer record batches at once while executions queue in its backend, so the group's combined requests follow what the backend can take rather than the replica count; a new replica starts at two batches per operation for a little over one probe interval after its first execution, so replicas added in front of a queueing backend do not flood it; the dashboard shows that limit and the latency ratio behind it. Set `maxReplicaCount` from the backend's connection budget: each replica runs up to 16 record batches and 64 Natives per Store at once.

## Query guard and coverage

The example guards its aggregate with `and on() (min(up{namespace="application",weir_group="application/mongo"}) == 1)`. A failed discovered scrape target or no discovered targets produces an empty result; `ignoreNullValues: "false"` then leaves KEDA in error instead of treating an incomplete lower workload as a valid scale-down signal. The HTTP timeout is 10 seconds, and no `or vector(0)` supplies a false zero. This guard covers the targets Prometheus has discovered; it cannot prove that scraper configuration includes every group Pod. EndpointSlice discovery keeps a terminating Pod as a target until it is deleted, so its last scrape after exit fails and briefly turns the query into an error; drop endpoints whose `terminating` condition is true, as the default Endpoints discovery already does. Before enabling autoscaling, compare the per-Pod targets with all running/Ready group replicas and verify the labels, `process_cpu_seconds_total` and `up` are present; repeat that coverage check after scale and rollout changes. Configure an existing `TriggerAuthentication`/`ClusterTriggerAuthentication` through `authenticationRef` if needed. No controller credentials are rendered or read by this Chart. KEDA's operator, metrics server and relevant network rules must already allow its Prometheus query. See [the Prometheus scaler contract](https://keda.sh/docs/2.20/scalers/prometheus/) and [ScaledObject specification](https://keda.sh/docs/2.20/reference/scaledobject-spec/).

## KEDA settings and turning autoscaling on

`autoscaling.triggers` accepts native KEDA triggers, including Prometheus and other supported scalers. Prometheus triggers require metrics to be enabled; CRDs/controllers are external prerequisites. `pollingInterval`, `cooldownPeriod` and `scaleDown` configure polling and HPA scale-down behavior. The cooldown setting concerns KEDA activation/scale-to-zero; ordinary reduction between running replica counts is controlled by HPA behavior. Minimum replicas must stay at least one: SDK connections cannot wake a group with zero replicas. A PDB requires at least two minimum replicas. Every upgrade or rollback that turns autoscaling on removes the Chart's replica count, so Kubernetes resets the Deployment to one replica until KEDA's HPA scales it to `minReplicaCount`: from three replicas with a minimum of two, two Pods were terminating within 20 seconds and the second Ready replica returned after about a minute. Turn autoscaling on when one replica can carry the load, with `minReplicaCount` at least the running count. `kubectl apply` and GitOps tools reset the count the same way when they turn autoscaling on. Upgrades that keep autoscaling on leave the count to KEDA. Verify the ScaledObject becomes Ready and HPA reaches its minimum before admitting production traffic; Helm readiness alone does not establish controller readiness. Disabling KEDA resumes the configured `replicaCount`.
52 changes: 52 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Configuration

How to declare Store groups, connect clients and supply credentials.

## Store groups

One Helm release is one Store group. Every replica hosts the same Store definitions; a group can host several Stores, and each group is released and scaled independently. Replicas share nothing with each other. Each Pod runs one Weir process with Lua inside that process. Backends, collections, indices, credentials and backups are provisioned separately.

### Stores

Declare this group's Stores directly in `stores`:

```yaml
stores:
- name: primary
mongodb:
uri: mongodb://mongo:27017/?directConnection=true
- name: search
search:
url: http://search:9200
batching:
max_wait: 5ms
```

Each entry uses Weir's own Store schema: `name`, exactly one of `mongodb.uri` or `search.url`, optional file-based `authentication`, optional `tls.ca_file` and optional `batching` (`max_operations`, `max_bytes`, `max_wait`, `max_in_flight`, `min_in_flight`). Database, collection and index targets come from request URIs; Chart configuration declares the backend connection for each Store. Multiple replicas within a group use identical Stores and settings. A request for a Store the group does not host fails with NotFound.

### Install several groups

Install [examples/mongo.yaml](../examples/mongo.yaml) and [examples/search.yaml](../examples/search.yaml) as separate groups in the same namespace:

```sh
helm upgrade --install mongo ./charts/weir --namespace application \
--kube-context your-context -f examples/mongo.yaml --atomic --wait --timeout 5m
helm upgrade --install search ./charts/weir --namespace application \
--kube-context your-context -f examples/search.yaml --atomic --wait --timeout 5m
```

## Client connections

Clients use [weir-go](https://github.com/batchstream/weir-go) v0.23.1, which speaks this server's protocol, and connect with `weir.New` to the group's headless Service by its full name with a trailing dot, `<release>-weir.<namespace>.svc.cluster.local.:<service.port>`, with the cluster's own domain if it is not `cluster.local`; without the dot every lookup first tries the Pod's DNS search domains. The name resolves to every Ready Pod, and the SDK spreads business RPCs across them. The SDK looks the name up every 5 seconds, so Pods added by a scale-out or rollout take their share of calls within seconds of becoming Ready, and a terminating Pod stops receiving calls once it leaves DNS, before SIGTERM. It also pings a replica it has heard nothing from for 10 seconds and leaves one that does not answer; this server accepts those pings, so upgrade the group before its clients. Generated `[::]` listeners accept IPv4 and IPv6 on the supported Go runtime. Group identity is `namespace/release`, so metric labels remain stable when `fullnameOverride` changes. The Service selects only its release. Allow clients to reach each group's application port.

## Generated configuration

The Chart generates `weir.yaml` with the listeners, `shutdown_timeout` and `stores`. The Chart renders one `weir.yaml` into its ConfigMap and mounts it read-only at `/etc/weir`. Its schema rejects unknown keys; the selected Weir binary validates backend semantics. Weir fixes transport, memory-pressure, Lua, queue and retry policy internally, so only listeners, the shutdown drain and Stores are configurable. Weir sets its Go memory limit to 90% of its container's cgroup v2 `memory.max` and refuses new work as live memory nears it, so keep a memory limit in `resources`. Without a limit, or on cgroup v1 nodes, Weir does not shed load.

## Credentials

Helm retains values in release history. Inline backend credentials are rejected, and so is any `@` in `mongodb.uri` or `search.url`, since it can only carry userinfo. MongoDB and Search use `authentication.username_file/password_file`; mount an externally managed Secret read-only using `extraVolumes/extraVolumeMounts` and reference it by absolute path, since relative paths resolve from `/etc/weir`; [examples/credential-files.yaml](../examples/credential-files.yaml) shows the layout. MongoDB requires SCRAM-SHA-256, authSource and TLS in the URI when authenticating. Both adapters can use `tls.ca_file`, or system trust when it is absent. Credential and CA files are read at startup; roll the Pods after rotating them, for example by changing a `podAnnotations` entry.

## Validate before deployment

Validate the generated `weir.yaml` with `weir check --config ...` from the pinned Weir source before deployment. Credential-file validation reads those supplied files and requires their authorization. ConfigMap updates automatically change the Pod checksum.
27 changes: 27 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Development

How to test the chart and publish a version.

## Test and package

```sh
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt
GOBIN="$PWD/.bin" go install github.com/batchstream/weir/cmd/weir@SOURCE_REVISION
make test PYTHON=.venv/bin/python WEIR_BIN="$PWD/.bin/weir"
make package PYTHON=.venv/bin/python WEIR_BIN="$PWD/.bin/weir"
```

`SOURCE_REVISION` is the `weir.batchstream.io/source-revision` annotation in `Chart.yaml`. `make test` lints the Chart, renders every example and validates the generated configuration with the selected Weir binary, including a scale check with many Stores and replicas and an isolated loopback-only serve, HTTP probe and shutdown run. With an official `promtool` on `PATH` or named by `PROMTOOL`, it also runs the rendered alerts and the autoscaling query against their PromQL fixtures; CI installs a checksum-verified promtool. It opens no backend connections, reads no existing Secrets and does not exercise CNI enforcement, Prometheus collection or KEDA scaling in a real cluster.

## Upgrade test

CI also runs `scripts/kind-upgrade.sh` on every change. In a kind cluster of its own, under load from `weir-soak` with `-no-errors` on both the current and the previous SDK, it upgrades a release from the Chart of the previous server through the Chart a change replaces to the new one, back and forward again at one replica, replaces the replica with one that starts 30 seconds late, scales to three replicas, takes the three back and forward again and scales back down. The change fails if a rollout does not complete, a caller saw any error or an outcome did not hold. While the previous SDK is v0.22.0 its client skips held streams, which that version fails in a rollout. Every rollout setting stays the Chart's default, and the test measures no throughput. The job fails if the weir-tests revision it builds the clients from pins other servers than these two Charts, so pin a new server there first. To run it locally, build `weir-soak` from [weir-tests](https://github.com/batchstream/weir-tests) for the cluster's platform and check out the commit to upgrade from; the script uses a kubeconfig of its own and ignores your current context and Helm settings. A second binary, built with the previous SDK and its protocol version as in the workflow, is optional; `PREVIOUS_PROFILES` sets its profiles, and `THROUGH_CHECKOUT` a Chart to pass through:

```sh
scripts/kind-upgrade.sh FROM_CHECKOUT . path/to/weir-soak [path/to/previous-weir-soak]
```

## Release

A reviewed `weir-VERSION` tag on `main` publishes the OCI Chart and a GitHub release with the archive and SHA256SUMS. The workflow refuses a version whose OCI Chart already exists and creates the release before pushing, so a rerun never overwrites a published Chart. If the push fails after the release exists, verify the release archive against its SHA256SUMS and `helm push` it by hand. Source annotation, image digest and CI source pins must identify the same server revision.
15 changes: 15 additions & 0 deletions docs/monitoring.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Monitoring

How to collect metrics and enable alerts and the dashboard.

## Enable metrics

Enable `metrics.enabled` and authorize scraper sources in `metrics.ingress`. The diagnostics listener is always bound for the probes; `metrics.enabled` adds the NetworkPolicy ingress for scrapers and a separate metrics Service. Reference the diagnostics port by its `metrics` name in ingress rules so a changed `diagnosticsPort` keeps matching. With default NetworkPolicy, metrics enablement requires explicit scraper rules. Annotations can support an existing scraper; `metrics.serviceMonitor.enabled` creates a ServiceMonitor when Prometheus Operator is already installed. Its selector matches exactly this release's metrics Service, and Prometheus discovers each Pod endpoint separately. `metrics.serviceMonitor.labels` must match your Prometheus instance's monitor selector; Chart identity/component labels cannot be overridden.

## Labels and scrapers

ServiceMonitor target relabeling supplies `namespace` and `weir_group="namespace/release"`. Native Weir metrics also carry `store` where applicable. With a custom scraper, supply the same stable labels and avoid duplicate scrape jobs. Scrape all group replicas; querying only the ordinary metrics ClusterIP would sample one Pod.

## Alerts and dashboard

`metrics.prometheusRule.enabled` adds a PrometheusRule with this group's alerts, and `metrics.grafanaDashboard.enabled` a ConfigMap holding a Grafana dashboard for the group. Both require `metrics.enabled`; their `labels` must match your Prometheus rule selector and Grafana's dashboard sidecar. Every expression selects the group by `namespace` and `weir_group`, so a custom scraper must supply those labels. [examples/prometheus-alerts.yaml](../examples/prometheus-alerts.yaml) enables the ServiceMonitor, alerts and dashboard together. `WeirGroupUnavailable` is critical: Prometheus sees no Ready replica of the group, which also covers failed scrapes. The other alerts are warnings: a replica Prometheus cannot scrape, sustained memory pressure, admission or Store rejections other than draining, unknown mutation outcomes, more than 5% of records ending with a failure, more than 5% of Native calls ending without a complete response, a p99 queue wait above one second and stalled RPCs. Each alert links to its runbook in Weir's [operations guide](https://github.com/batchstream/weir/blob/main/docs/operations.md#alerts).
Loading
Loading