Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -344,8 +344,9 @@ jobs:
case "$version" in *"\$Format:"*) echo 'Chart version was not materialized'; exit 1;; esac
helm lint helm-chart
helm package helm-chart --destination release-artifacts
mv "release-artifacts/manticoresearch-${version}.tgz" "release-artifacts/streams-${version}.tgz"
echo "version=${version}" >> "$GITHUB_OUTPUT"
echo "asset=release-artifacts/manticoresearch-${version}.tgz" >> "$GITHUB_OUTPUT"
echo "asset=release-artifacts/streams-${version}.tgz" >> "$GITHUB_OUTPUT"
- name: Create GitHub Release
uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v2.3.2
with:
Expand Down
167 changes: 34 additions & 133 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,155 +1,56 @@
# Kafka UI
# Manticore Streams

**Attention!!!** Min kubernetes version: 1.20
### Kafka Installation
Manticore Streams is a Kubernetes application for filtering Apache Kafka topics with Manticore Search. Define filtering rules in the UI, process incoming documents at scale, and write matched documents to another Kafka topic.

If you don't have it, you install and run Kafka Server
## Requirements

```bash
helm repo add bitnami https://charts.bitnami.com/bitnami
helm --namespace kafka install --set listeners.client.protocol=PLAINTEXT --set listeners.controller.protocol=PLAINTEXT my-kafka bitnami/kafka
```

Don't worry about topics, Stream handler app create it automatically.

In that case Kafka host will `my-kafka.kafka.svc.cluster.local` (inner Kubernetes). It will need us on goals creation stage
- A Kubernetes cluster
- Helm 3
- An Apache Kafka cluster reachable from the Kubernetes workloads

If you see that either `pod/my-kafka-0` or `pod/my-kafka-zookeeper-0` is not running in `kubectl get all -n kafka` try:
```
kubectl edit statefulset.apps/my-kafka-zookeeper -n kafka
kubectl edit statefulset.apps/my-kafka -n kafka
```
To make it run under root:
```
securityContext:
fsGroup: 0
runAsUser: 0
```
The release chart uses public images from `ghcr.io/manticoresoftware/streams`; no image-pull secret is required.

**Exposing**
## Install

For exposing Kafka outside cluster we've modify Kafka helm chart config:
1. Download the `streams-<version>.tgz` chart asset for the desired version from the [GitHub Releases page](https://github.com/manticoresoftware/streams/releases).
2. Install it with Helm. Choose a release name, namespace, administrator credentials, and the hostname served by your ingress controller:

```sh
helm install manticore-streams ./streams-<version>.tgz \
--namespace manticore-streams \
--create-namespace \
--set ui.admin.email='admin@example.com' \
--set ui.admin.pass='change-this-password' \
--set ingress.hosts[0].host='streams.example.com' \
--set ingress.hosts[0].paths[0]='/'
```
externalAccess.enabled=true
externalAccess.service.type=NodePort
externalAccess.serivce.nodePorts[0]='30001'
```
It's work only if we will edit `values.yaml`.
Don't try to specify parameters by adding `--set` arguments to `helm install` command. This will not work!!

### MKC Installation

When Kafka server are runned, we can start deploying `MKC` chart:

The chart defaults to private images hosted at `ghcr.io/manticoresoftware/streams`. Before installing, create an image-pull secret in the release namespace with a GitHub token that has `read:packages` access:

`kubectl create secret docker-registry registry-manticore-streams --namespace={namespace} --docker-server=ghcr.io --docker-username={github-user} --docker-password={github-token}`

To use a differently named secret, set `imagePullSecrets` in your values file.

For installing you must specify auth l/p and ingress host of you project

**Helm 2**

`helm install --name {name} --namespace {namesace}
--set ui.admin.email="user@manticoresearch.com"
--set ui.admin.pass="myNewPassword"
--set ingress.hosts[0].host="kafka.manticoresearch.com"
--set ingress.hosts[0].paths[0]="/" ./manticoresearch-{version}.tgz`

**Helm 3**

`helm install {name} --namespace {namesace}
--set ui.admin.email="user@manticoresearch.com"
--set ui.admin.pass="myNewPassword"
--set ingress.hosts[0].host="kafka.manticoresearch.com"
--set ingress.hosts[0].paths[0]="/" ./manticoresearch-{version}.tgz`


To deploy without an ingress controller, add `--set ingress.enabled=false` and expose the UI service by the mechanism appropriate for your cluster.

The release package uses the `values.yaml` embedded when it was built. To use your own edited values file, pass it explicitly with `-f values.yaml`:
Verify that the workloads are running:

For **Helm 2**

`helm install --name {name} --namespace {namespace} -f values.yaml ./manticoresearch-{version}.tgz`

For **Helm 3**

`helm install {name} --namespace {namespace} -f values.yaml ./manticoresearch-{version}.tgz`

Important parameters there:

* ui.admin.email - email for super admin auth
* ui.admin.pass - password for super admin auth
* ingress.hosts - set host for UI access

That's all


### Removing

For **Helm 3** removing just run `helm uninstall {name} -n {namespace}`

**Helm 2** can't handle dynamically created pods, do after removing we must clean up k8s:

**Warning!!!**: don't use these commands in "default" namespace.
In case if you installed the chart to that namespace, you must remove it remove "by hand"!!
```
kubectl -n {namespace} delete all --all
kubectl -n {namespace} delete cm --all
kubectl -n {namespace} delete pvc --all
```sh
kubectl get pods --namespace manticore-streams
```

In some cases helm can not correctly remove chart and on reinstalling you can get error:
After signing in to the UI, configure sources, destinations, and streams. The Kafka brokers and topics you configure must be reachable from the Manticore Streams namespace.

```
Error: rendered manifests contain a resource that already exists. Unable to continue with install:
existing resource conflict: kind: SomeResouce, namespace: , name: ResourceName
```
## Upgrade and uninstall

You must remove it yourself:
Download the desired release asset, then upgrade the existing release with the same values:

```
kubectl delete all -n {namespace} --all
kubectl delete pvc -n {namespace} --all
kubectl delete sa ui-admin-sa-{namespace} -n {namespace}
kubectl delete ClusterRole ui-admin-role-{namespace} -n {namespace}
kubectl delete ClusterRoleBinding ui-admin-{namespace} -n {namespace}
kubectl delete ingress kafka-pq -n {namespace}
```sh
helm upgrade manticore-streams ./streams-<version>.tgz \
--namespace manticore-streams \
--reuse-values
```

To remove the release:

### Our local cluster tests:
1) Must make vpn to kuber01
2) Run
```sh
helm uninstall manticore-streams --namespace manticore-streams
```
kafkacat -b kafka01.example.com:9092,kafka02.example.com:9092,kafka03.example.com:9092 \
-G mkc20200116 sina.firehose.out | kafkacat -b 78.47.167.154:30001 -t my-docs
```

### Api tokens

For using API first generate token from `/tokens` section.
## Documentation

After, add headers to your request:
```
Authorization: Bearer $TOKEN
Accept: application/json
```

Endpoints:
```
/api/admin/source/getList
/api/admin/source/add
/api/admin/source/delete
/api/admin/destination/getList
/api/admin/destination/add
/api/admin/destination/delete
/api/manager/getRulesList
/api/manager/addRule
/api/manager/deleteRule
/api/admin/process/add
/api/admin/process/assignUser
/api/admin/process/unassignUser
/api/admin/process/remove/{id}
See [docs/README.md](docs/README.md) for the full documentation index, including chart values, Kafka setup guidance, and operating Manticore Streams.
1 change: 0 additions & 1 deletion dev-environment/k8s_tests/dev-server-k3d.sh
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,6 @@ chart_overrides() {
printf '%s\n' \
'--set' 'ingress.enabled=false' \
'--set' 'podSecurityPolicy.enabled=false' \
'--set-json' 'imagePullSecrets=[]' \
'--set' 'worker.resources.requests.cpu=250m' \
'--set' 'worker.resources.requests.memory=512Mi' \
'--set' 'worker.resources.limits.cpu=1' \
Expand Down
5 changes: 0 additions & 5 deletions docs/HelmVariables.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,5 @@
# Helm Variables

### Container registry section
Parameter | Description | Default
--------- | ----------- | -------
`imagePullSecrets` | Kubernetes image-pull secret names for private GHCR images | `[{name: registry-manticore-streams}]`

### UI section
Parameter | Description | Default
--------- | ----------- | -------
Expand Down
117 changes: 43 additions & 74 deletions docs/Installation.md
Original file line number Diff line number Diff line change
@@ -1,98 +1,67 @@
# Installation

Manticore Streams for now supports only Apache Kafka as a source of an input stream. Read
this [article](InstallFromScratch/Kafka.md) about installation of Apache Kafka
Manticore Streams consumes Apache Kafka topics. Before installing it, make sure that Kafka is reachable from the Kubernetes namespace where Manticore Streams will run. See [Kafka setup](InstallFromScratch/Kafka.md) for a basic Kubernetes example.

---
## Requirements

# Cloning
- Kubernetes
- Helm 3
- An ingress controller when `ingress.enabled` is left at its default of `true`
- An Apache Kafka cluster reachable from the Manticore Streams workloads

### ⚠️ Attention! ⚠️
## Download the chart

You can't use **GIT clone**. Download the `.tgz` chart asset from the GitHub Release for the desired version instead. The release package has matching chart and image versions already materialized.
Download the `streams-<version>.tgz` asset for the desired version from [GitHub Releases](https://github.com/manticoresoftware/streams/releases). The package contains matching chart and image versions. Images are public at `ghcr.io/manticoresoftware/streams`, so no image-pull secret is required.

## Container images
## Install with Helm

The chart defaults to private images hosted at `ghcr.io/manticoresoftware/streams`. Before installing, create an image-pull secret in the release namespace with a GitHub token that has `read:packages` access:

`kubectl create secret docker-registry registry-manticore-streams --namespace={namespace} --docker-server=ghcr.io --docker-username={github-user} --docker-password={github-token}`

To use a differently named secret, set `imagePullSecrets` in your values file.

## Helm

<!-- example helm-install -->
For installing, you must specify login and password for the admin, and ingress host of you project

Important parameters there:

* ui.admin.email - email for super admin auth
* ui.admin.pass - password for super admin auth
* ingress.hosts - set host for UI access

<!-- intro -->

### Helm 3

<!-- request Helm 3 -->
Choose a release name, namespace, administrator credentials, and ingress hostname. The following command creates the namespace when it does not already exist:

```sh
helm install manticore-streams ./streams-<version>.tgz \
--namespace manticore-streams \
--create-namespace \
--set ui.admin.email='admin@example.com' \
--set ui.admin.pass='change-this-password' \
--set ingress.hosts[0].host='streams.example.com' \
--set ingress.hosts[0].paths[0]='/'
```
helm install {name} --namespace {namespace} \
--set ui.admin.email="user@manticoresearch.com" \
--set ui.admin.pass="myNewPassword" \
--set ingress.hosts[0].host="kafka.manticoresearch.com" \
--set ingress.hosts[0].paths[0]="/" ./manticoresearch-{version}.tgz
```

<!-- intro -->

### Helm 2
To deploy without an ingress controller, add `--set ingress.enabled=false` and expose the UI service using the mechanism appropriate for the cluster.

<!-- request Helm 2 -->
Check deployment status:

```sh
kubectl get pods --namespace manticore-streams
```
helm install --name {name} --namespace {namespace} \
--set ui.admin.email="user@manticoresearch.com" \
--set ui.admin.pass="myNewPassword" \
--set ingress.hosts[0].host="kafka.manticoresearch.com" \
--set ingress.hosts[0].paths[0]="/" ./manticoresearch-{version}.tgz
```

<!-- end -->
<!-- example helm-filled-install -->
The release package uses the `values.yaml` embedded when it was built. To use your own edited values file, pass it explicitly with `-f values.yaml`:

<!-- intro -->

### Helm 3

<!-- request Helm 3 -->

```helm install {name} --namespace {namespace} -f values.yaml ./manticoresearch-{version}.tgz```
Once the UI is available, sign in using the administrator credentials and configure Kafka sources, destinations, and streams.

<!-- intro -->
## Custom values

### Helm 2
To keep installation settings in a file, put them in `values.yaml` and pass the file to Helm:

<!-- request Helm 2 -->

```helm install --name {name} --namespace {namespace} -f values.yaml ./manticoresearch-{version}.tgz```

<!-- end -->
```sh
helm install manticore-streams ./streams-<version>.tgz \
--namespace manticore-streams \
--create-namespace \
--values values.yaml
```

**That's all**
See [Helm chart variables](HelmVariables.md) for the available settings.

___
## Upgrade and uninstall

Check for deploy:
Upgrade an existing deployment with a downloaded release asset:

```sh
helm upgrade manticore-streams ./streams-<version>.tgz \
--namespace manticore-streams \
--reuse-values
```
$ kubectl get po -n {namespace}
NAME READY STATUS RESTARTS AGE
mkc-columnar-0 1/1 Running 0 1m
mkc-scaler-69554f869b-g7b5f 1/1 Running 0 1m
mkc-ui-0 1/1 Running 0 1m
mkc-ui-mysql-0 1/1 Running 0 1m
```

After our project has been deployed we can start to create streams
Remove the release with:

```sh
helm uninstall manticore-streams --namespace manticore-streams
```
2 changes: 1 addition & 1 deletion docs/Restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

Backups are stored in the `backup / data` folder
* database.sql - backup of UI pod
* indexes-mkc-m{N}-manticore.tar.gz - stream's backup where stream id = `N`
* indexes-ms-m{N}-manticore.tar.gz - stream's backup where stream id = `N`


Recovery takes place gradually, stream after stream.
Expand Down
4 changes: 0 additions & 4 deletions helm-chart/templates/deployment-scaler.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,6 @@ spec:
mountPath: /var/www/html
resources:
{{- toYaml .Values.scaler.nginx.resources | nindent 10 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: config-volume
configMap:
Expand Down
4 changes: 0 additions & 4 deletions helm-chart/templates/deployment-ui.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -189,10 +189,6 @@ spec:
subPath: nginx.conf
resources:
{{- toYaml .Values.ui.nginx.resources | nindent 10 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: process-config
configMap:
Expand Down
Loading
Loading