Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions demos/ach/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,13 @@ oc create project ach

From now on, all the commands must be run into this `ach` project.

=== Open Data Hub
Open Data Hub(ODH) will be deployed using ODH operator. Create an instance of ODH by using the Kfdef definition in /odh. This is the version 0.9 of ODH.
[bash]
----
oc apply -f odh/kfdef-v0.9.yaml
----

=== Kafka

Usign the AMQ Stream or Strimzi operator, we can create a basic Kafka cluster, along with the needed topics.
Expand All @@ -94,7 +101,7 @@ Those are the secrets that will be used by the various pods we will create.

First, edit the file `secrets.yaml` and replace the values for `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` that you got when creating the RGW user.

If you have followed the previous instructions for creating the Object Store, the service should have been created with the right name, `http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage` so you should not need to change it.
If you have followed the previous instructions for creating the Object Store, the service should have been created with the right name, `http://rook-ceph-rgw-s3a.openshift-storage` so you should not need to change it.

You can leave the database parameters as they are, but you must use those when creating the database...

Expand Down Expand Up @@ -268,4 +275,4 @@ In the **tools** folder you will also find:
* ach-bd.txt: various SQL commands to create and (re)initialize tables used in the auxiliary database (used to store the number of processed files)
* amq-streams-ocs4.yaml: definition od the KafkaCluster created with the AMQStreams operator, using OCS4 storage for persistency
* grafana-prometheus-datasource.yaml: datasource for the Grafana Operator to connect to OpenShift Prometheus. You will have to replace the secret (basicAuth) by the one used by the built-in OpenShift Grafana.
* kafdrop.yaml: installation of Kafdrop to monitor your Kafka/AMQStreams cluster
* kafdrop.yaml: installation of Kafdrop to monitor your Kafka/AMQStreams cluster
4 changes: 2 additions & 2 deletions demos/ach/bucket_notifications.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"metadata": {},
"outputs": [],
"source": [
"endpoint_url = 'http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage'\n",
"endpoint_url = 'http://rook-ceph-rgw-s3a.openshift-storage'\n",
"aws_access_key_id = 'replace_me'\n",
"aws_secret_access_key = 'replace_me'"
]
Expand Down Expand Up @@ -230,4 +230,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
4 changes: 2 additions & 2 deletions demos/ach/grafana-mysql-datasource.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ spec:
- type: mysql
name: MySQL
access: proxy
url: ach-db.ach
url: achdbservice.ach
database: achdb
user: achuser
password: achpassword
name: mysql-grafanadatasource.yaml
name: mysql-grafanadatasource.yaml
2 changes: 1 addition & 1 deletion demos/ach/kafkasource-odfi-split.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
consumerGroup: odfi-split
bootstrapServers:
- my-cluster-kafka-bootstrap.kafka:9092
- my-cluster-kafka-bootstrap.ach:9092
topics:
- merchant-upload
sink:
Expand Down
2 changes: 1 addition & 1 deletion demos/ach/kafkasource-rdfi-process.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
consumerGroup: rdfi-process
bootstrapServers:
- my-cluster-kafka-bootstrap.kafka:9092
- my-cluster-kafka-bootstrap.ach:9092
topics:
- rdfi
sink:
Expand Down
2 changes: 1 addition & 1 deletion demos/ach/kafkasource-rdfi-split.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
namespace: ach
spec:
bootstrapServers:
- 'my-cluster-kafka-bootstrap.kafka:9092'
- 'my-cluster-kafka-bootstrap.ach:9092'
consumerGroup: rdfi-split
sink:
ref:
Expand Down
123 changes: 123 additions & 0 deletions demos/ach/odh/kfdef-v0.9.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef
metadata:
name: opendatahub
namespace: ach
spec:
applications:
- kustomizeConfig:
repoRef:
name: manifests
path: odh-common
name: odh-common
#Note: AI Library components requires seldon do be deployed first, make sure to place seldon before AIlibrary
- kustomizeConfig:
repoRef:
name: manifests
path: odhseldon/cluster
name: odhseldon
- kustomizeConfig:
repoRef:
name: manifests
path: ai-library/cluster
name: ai-library-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: ai-library/operator
# Note: In order to utilize ai-library, you also need to have Seldon installed
name: ai-library-operator
- kustomizeConfig:
parameters:
# Note: The admin username is admin
- name: SUPERSET_ADMIN_PASSWORD
value: admin
repoRef:
name: manifests
path: superset
name: superset
- kustomizeConfig:
parameters:
- name: namespace
value: openshift-operators
repoRef:
name: manifests
path: kafka/cluster
name: strimzi-operator
- kustomizeConfig:
repoRef:
name: manifests
path: kafka/kafka
name: kafka-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: grafana/cluster
name: grafana-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: grafana/grafana
name: grafana-instance
- kustomizeConfig:
repoRef:
name: manifests
path: radanalyticsio/spark/cluster
name: radanalyticsio-spark-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: prometheus/cluster
name: prometheus-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: prometheus/operator
name: prometheus-operator
- kustomizeConfig:
parameters:
- name: s3_endpoint_url
value: "s3.odh.com"
repoRef:
name: manifests
path: jupyterhub/jupyterhub
name: jupyterhub
- kustomizeConfig:
overlays:
#- cuda
- additional
repoRef:
name: manifests
path: jupyterhub/notebook-images
name: notebook-images
- kustomizeConfig:
repoRef:
name: manifests
path: airflow/operator
name: airflow-operator
- kustomizeConfig:
repoRef:
name: manifests
path: airflow/cluster
name: airflow-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: odhargo/cluster
name: odhargo-cluster
- kustomizeConfig:
repoRef:
name: manifests
path: odhargo/odhargo
name: odhargo
- kustomizeConfig:
repoRef:
name: manifests
path: odh-dashboard
name: odh-dashboard
repos:
- name: kf-manifests
uri: https://github.com/opendatahub-io/manifests/tarball/v1.0-branch-openshift
- name: manifests
uri: https://github.com/opendatahub-io/odh-manifests/tarball/v0.9.0
version: v0.9.0