Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 6 additions & 48 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,65 +37,23 @@ jobs:
with:
name: weir-chart
path: dist/*.tgz
upgrade:
rollout:
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5
with:
go-version: '1.27.3'
cache: false
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
with:
version: v3.17.0
- name: Install kind
run: GOBIN="$PWD/.bin" go install sigs.k8s.io/kind@v0.33.0
- name: Build the soak client with the current and the previous SDK
shell: bash
run: |
bin=$PWD/.bin
git clone -q https://github.com/batchstream/weir-tests "$RUNNER_TEMP/weir-tests"
cd "$RUNNER_TEMP/weir-tests"
git checkout -q 1347bcdb3351852b7df484e778a757d0d2e80ff3
CGO_ENABLED=0 go build -o "$bin/weir-soak" ./cmd/weir-soak
version=$(jq -er .weir_go_previous versions.json)
protocol=$(go mod download -json "github.com/batchstream/weir-go@$version" | jq -er .GoMod | xargs awk '$1 == "github.com/batchstream/weir-protocol" {print $2}')
go get "github.com/batchstream/weir-go@$version" "github.com/batchstream/weir-protocol@$protocol"
CGO_ENABLED=0 go build -o "$bin/weir-soak-previous" ./cmd/weir-soak
sdk() { go version -m "$bin/$1" | awk '$2 == "github.com/batchstream/weir-go" {print $3}'; }
echo "soak clients use $(sdk weir-soak) and $(sdk weir-soak-previous)"
[ "$(sdk weir-soak)" != "$(sdk weir-soak-previous)" ]
# Held streams of v0.22.0 fail in a rollout; v0.23.0 fixed them.
if [ "$version" = v0.22.0 ]; then
echo 'PREVIOUS_PROFILES=single=40,visible=10,lua=15,batch=30,native=5' >> "$GITHUB_ENV"
fi
- name: Extract the charts of the previous server and the commit this change replaces
shell: bash
env:
BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
- name: Install kind and the soak client
run: |
digest() { git show "$1:charts/weir/values.yaml" | sed -n 's/^ digest: //p'; }
current=$(digest HEAD)
previous=
for commit in $(git log --first-parent --format=%H); do
if [ "$(digest "$commit")" != "$current" ]; then
previous=$commit
break
fi
done
[ -n "$previous" ] || { echo "no earlier commit on this branch pins another server" >&2; exit 1; }
echo "previous server $(digest "$previous") in $previous"
mkdir -p "$RUNNER_TEMP/previous" "$RUNNER_TEMP/base"
git archive "$previous" | tar -x -C "$RUNNER_TEMP/previous"
git archive "$BASE" | tar -x -C "$RUNNER_TEMP/base"
revision() { sed -n 's/^ weir.batchstream.io\/source-revision: "\(.*\)"$/\1/p' "$1/charts/weir/Chart.yaml"; }
pins=$RUNNER_TEMP/weir-tests/versions.json
[ "$(jq -er .weir_source "$pins")" = "$(revision .)" ] || { echo "weir-tests pins another current server than this chart" >&2; exit 1; }
[ "$(jq -er .weir_previous_source "$pins")" = "$(revision "$RUNNER_TEMP/previous")" ] || { echo "weir-tests pins another previous server than the previous chart" >&2; exit 1; }
- name: Upgrade from the previous server under load from both SDKs
GOBIN="$PWD/.bin" go install sigs.k8s.io/kind@v0.33.0
GOBIN="$PWD/.bin" CGO_ENABLED=0 go install github.com/batchstream/weir-tests/cmd/weir-soak@df28116e9c492e15db3ef19ab6a70bd2ea3110a3
- name: Roll the chart out under load
env:
KIND: .bin/kind
run: THROUGH_CHECKOUT="$RUNNER_TEMP/base" scripts/kind-upgrade.sh "$RUNNER_TEMP/previous" . .bin/weir-soak .bin/weir-soak-previous
run: scripts/kind-rollout.sh .bin/weir-soak
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ rendered alerts and the autoscaling query; without it, that test is skipped.
- Keep the change focused and explain the observed problem and resulting behavior.
- Add a regression test when fixing behavior.
- Update `values.schema.json`, the [examples](examples), the README and [docs](docs) when values change.
- When pinning a new server, pin it in [weir-tests](https://github.com/batchstream/weir-tests) first, then change the source annotation, `appVersion`, image digest, workflow pins, the references in the README and [docs](docs) and the values in `tests/test_chart.py` together.
- When pinning a new server, change the source annotation, `appVersion`, image digest, workflow pins (including the weir-tests revision of the rollout test's soak client, whose SDK must speak the new server's protocol), the references in the README and [docs](docs) and the values in `tests/test_chart.py` together.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ MongoDB and Search data plane, on Kubernetes.
| Chart | `0.20.0-rc.1` |
| Server source | [commit `5107986b882d`](https://github.com/batchstream/weir/commit/5107986b882da60751403c7e05bbbebdb12b3cdb) |
| Server image | `ghcr.io/batchstream/weir@sha256:b5d949cefb0ba01e18cff2b5210cef954bc387c20cffa87784466e793e0e088e` |
| Go SDK | [weir-go](https://github.com/batchstream/weir-go) v0.23.1 |
| Go SDK | [weir-go](https://github.com/batchstream/weir-go) v0.24.1 |

`Chart.yaml` records the server source and `values.yaml` pins the image. Both the
validation and release workflows install that exact source with `go install`.
Expand Down Expand Up @@ -128,7 +128,7 @@ must exist first.
| [Monitoring](docs/monitoring.md) | Metrics, labels, alerts and the Grafana dashboard |
| [Autoscaling](docs/autoscaling.md) | KEDA and Prometheus setup |
| [Operations](docs/operations.md) | Network, rolling updates, capacity, shutdown and probes |
| [Development](docs/development.md) | Chart tests, the upgrade test and releases |
| [Development](docs/development.md) | Chart tests, the rollout test and releases |

| Example | Shows |
| --- | --- |
Expand All @@ -149,7 +149,7 @@ make test PYTHON=.venv/bin/python WEIR_BIN="$PWD/.bin/weir"
```

`SOURCE_REVISION` is the `weir.batchstream.io/source-revision` annotation in
`Chart.yaml`. See [Development](docs/development.md) for packaging and the upgrade
`Chart.yaml`. See [Development](docs/development.md) for packaging and the rollout
test.

- Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ helm upgrade --install search ./charts/weir --namespace application \

## Client connections

Clients use [weir-go](https://github.com/batchstream/weir-go) v0.23.1, which speaks this server's protocol, and connect with `weir.New` to the group's headless Service by its full name with a trailing dot, `<release>-weir.<namespace>.svc.cluster.local.:<service.port>`, with the cluster's own domain if it is not `cluster.local`; without the dot every lookup first tries the Pod's DNS search domains. The name resolves to every Ready Pod, and the SDK spreads business RPCs across them. The SDK looks the name up every 5 seconds, so Pods added by a scale-out or rollout take their share of calls within seconds of becoming Ready, and a terminating Pod stops receiving calls once it leaves DNS, before SIGTERM. It also pings a replica it has heard nothing from for 10 seconds and leaves one that does not answer; this server accepts those pings, so upgrade the group before its clients. Generated `[::]` listeners accept IPv4 and IPv6 on the supported Go runtime. Group identity is `namespace/release`, so metric labels remain stable when `fullnameOverride` changes. The Service selects only its release. Allow clients to reach each group's application port.
Clients use [weir-go](https://github.com/batchstream/weir-go) v0.24.1, which speaks this server's protocol, and connect with `weir.New` to the group's headless Service by its full name with a trailing dot, `<release>-weir.<namespace>.svc.cluster.local.:<service.port>`, with the cluster's own domain if it is not `cluster.local`; without the dot every lookup first tries the Pod's DNS search domains. The name resolves to every Ready Pod, and the SDK spreads business RPCs across them. The SDK looks the name up every 5 seconds, so Pods added by a scale-out or rollout take their share of calls within seconds of becoming Ready, and a terminating Pod stops receiving calls once it leaves DNS, before SIGTERM. It also pings a replica it has heard nothing from for 10 seconds and leaves one that does not answer; this server accepts those pings. Generated `[::]` listeners accept IPv4 and IPv6 on the supported Go runtime. Group identity is `namespace/release`, so metric labels remain stable when `fullnameOverride` changes. The Service selects only its release. Allow clients to reach each group's application port.

## Generated configuration

Expand Down
7 changes: 4 additions & 3 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,13 @@ make package PYTHON=.venv/bin/python WEIR_BIN="$PWD/.bin/weir"

`SOURCE_REVISION` is the `weir.batchstream.io/source-revision` annotation in `Chart.yaml`. `make test` lints the Chart, renders every example and validates the generated configuration with the selected Weir binary, including a scale check with many Stores and replicas and an isolated loopback-only serve, HTTP probe and shutdown run. With an official `promtool` on `PATH` or named by `PROMTOOL`, it also runs the rendered alerts and the autoscaling query against their PromQL fixtures; CI installs a checksum-verified promtool. It opens no backend connections, reads no existing Secrets and does not exercise CNI enforcement, Prometheus collection or KEDA scaling in a real cluster.

## Upgrade test
## Rollout test

CI also runs `scripts/kind-upgrade.sh` on every change. In a kind cluster of its own, under load from `weir-soak` with `-no-errors` on both the current and the previous SDK, it upgrades a release from the Chart of the previous server through the Chart a change replaces to the new one, back and forward again at one replica, replaces the replica with one that starts 30 seconds late, scales to three replicas, takes the three back and forward again and scales back down. The change fails if a rollout does not complete, a caller saw any error or an outcome did not hold. While the previous SDK is v0.22.0 its client skips held streams, which that version fails in a rollout. Every rollout setting stays the Chart's default, and the test measures no throughput. The job fails if the weir-tests revision it builds the clients from pins other servers than these two Charts, so pin a new server there first. To run it locally, build `weir-soak` from [weir-tests](https://github.com/batchstream/weir-tests) for the cluster's platform and check out the commit to upgrade from; the script uses a kubeconfig of its own and ignores your current context and Helm settings. A second binary, built with the previous SDK and its protocol version as in the workflow, is optional; `PREVIOUS_PROFILES` sets its profiles, and `THROUGH_CHECKOUT` a Chart to pass through:
CI also runs `scripts/kind-rollout.sh` on every change. In a kind cluster of its own, under load from `weir-soak` with `-no-errors`, it rolls a release of this checkout's Chart out twice at one replica, replaces the replica with one that starts 30 seconds late, scales to three replicas, rolls the three out twice and scales back down. The change fails if a rollout does not complete, a caller saw any error or an outcome did not hold. Every rollout setting stays the Chart's default, and the test measures no throughput. To run it locally, build `weir-soak` for the cluster's platform from a [weir-tests](https://github.com/batchstream/weir-tests) checkout at the revision `ci.yaml` installs. The script uses a kubeconfig of its own and ignores your current context and Helm settings. For kind on an arm64 Mac:

```sh
scripts/kind-upgrade.sh FROM_CHECKOUT . path/to/weir-soak [path/to/previous-weir-soak]
(cd path/to/weir-tests && GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o weir-soak ./cmd/weir-soak)
scripts/kind-rollout.sh path/to/weir-tests/weir-soak
```

## Release
Expand Down
Loading
Loading