Scalingo Operator v1.2.0

The Scalingo Operator can deploy and undeploy PostgreSQL instances hosted on dedicated resources on Scalingo platform from a Kubernetes cluster.

Definitions

Few helpful definitions to start with.

CRD (Custom Resource Definition)

A CRD is a Kubernetes object used to define a new custom resource type in the Kubernetes API. It tells the API server: “Here is a new kind (e.g., MyApp) and its schema.”

Example: config/crd/bases/databases.scalingo.com_postgresqls.yaml

CR (Custom Resource)

A CR is an actual instance of the type defined by the CRD.

Examples:

• config/samples/databases_v1_postgresql.yaml
• doc/examples/custom_resources/cr-postgresql.starter.yaml
• doc/examples/custom_resources/cr-postgresql.enterprise.yaml

Usage

As a pre-requisite, first store your Scalingo API token in your cluster secrets. Then, execute the Kubernetes operator on your cluster and deploy the expected database resource.

Create Secret

# create secret
SCALINGO_TOKEN="tk-my_token"
kubectl create secret generic scalingo \
    --from-literal=api_token="$SCALINGO_TOKEN"

# verification
kubectl describe secret scalingo

Deploy the Operator

Gather and save the latest installer from the Scalingo Operator Github releases: https://github.com/Scalingo/scalingo-operator/releases

The installer file is install.yaml.

Then deploy the Operator controller on your Kubernetes cluster:

# install operator controller manager
kubectl apply --filename install.yaml

# verification
kubectl get all --namespace scalingo-operator-system

# follow operator logs
kubectl logs deploy/scalingo-operator-controller-manager --namespace scalingo-operator-system --follow

Deploy Database Resource

Once the operator is deployed and running, deploy the database resource using its descriptor.

Using PostgreSQL sample example:

kubectl apply --filename config/samples/databases_v1_postgresql.yaml

Read Database URL

At deployment, the database secret is written in Kubernetes secrets. The secret name and key are defined in Custom Resource connInfoSecretTarget fields.

To read the secret:

$NAME=my-postgresql-secret  # read from CR spec.connInfoSecretTarget.name
$PREFIX=PG                  # read from CR spec.connInfoSecretTarget.prefix

# list all secrets in all namespaces
kubectl get secrets -A

# secret details
kubectl describe secret $NAME

# get secret field
kubectl get secret $NAME -o jsonpath='{.data}' | grep $PREFIX

# gather the returned base64 content and decode, example:
echo "ZGJfY29ubmVjdGlvbl9zdHJpbmc=" | base64 --decode

Deploy Multiple Databases Resources

Every database resource is identified by its meta.name and it must use its own database name and database connection information.

Ensure to use distinct values for these fields in a new database resource descriptor:

Field	Description
meta.name	Database resource unique name
spec.name	Scalingo database unique name
spec.connInfoSecretTarget.name	Secret database connection informations

Then deploy using the new database resource descriptor:

kubectl apply --filename new_database_v1_postgresql.yaml

Modify database

To apply database modification or change, modify the resource descriptor accordingly and use the command:

kubectl apply --filename modified_database_v1_postgresql.yaml

Since version v1.1.0-alpha1, the firewall rules are applied and can be modified.

Since version v1.2.0-alpha1, the plan can be changed. The plan change is a long operation (~20 minutes) and implies provisioning. While provisioning, no other plan change is possible.

Undeploy Database

Use almost the same command than deploy, with the same descriptor file: replace apply by delete.

Using PostgreSQL sample example:

kubectl delete --filename config/samples/databases_v1_postgresql.yaml

Undeploy the Operator

Not necessary, if Operator undeploy is needed execute the opposite deploy commands.

# Replace my-namespace and my-operator using your own names
kubectl delete deploy/scalingo-operator-controller-manager --namespace scalingo-operator-system
kubectl delete namespace scalingo-operator-system
kubectl delete --kustomize config/crd

# Verifications
kubectl get deploy,pods --namespace scalingo-operator-system
kubectl get namespaces
kubectl get crds -A

Architecture

This application follows a (slightly adapted version of) the clean architecture pattern.

Layers

• Boundaries > Out: output calls to go-scalingo API
• Controller (> In): handles input Kubernetes requests
• Usecases: business rules
• Domain: business entities

Note

Controller is generated by Kubebuilder, ready to handle Kubernetes requests. For this reason it is kept as is, ensuring the Controllers plus the Boundaries > In layers.

Development

Install Environment

sudo snap install microk8s --classic
sudo snap install kubectl --classic

microk8s.kubectl config view --raw > $HOME/.kube/microk8s.config

# then add in ~/.zshrc
export  KUBECONFIG=$HOME/.kube/config
export  KUBECONFIG=$KUBECONFIG:$HOME/.kube/microk8s.config

# verification: both commands must return the same informations
microk8s.kubectl config view
kubectl config view

# ensure some microk8s modules are enabled
microk8s enable dns
microk8s enable rbac
microk8s status

Download Kubebuilder and Install Locally

curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

Kubebuilder Commands

Operator Creation Example

These commands were executed to create this operator:

kubebuilder init --domain scalingo.com --repo github.com/Scalingo/scalingo-operator
kubebuilder create api --group databases --version v1 --kind PostgreSQL
make manifests
make install

Build Commands

# generate api/v1/zz_generated.deepcopy.go
make generate

# generate the CRD manifests under config/crd/bases and a sample for it under config/samples
make manifests

Execute Command

# deploy the CRD on local cluster
make install

# verify using
kubectl get crd

# set authentication URL (osc-fr1 region example)
export SCALINGO_AUTH_URL="https://auth.scalingo.com"

# execute
make run

List make Targets

To list the make targets with their description:

make help

Tips and Tricks

Force-delete the CRD

In case the kubernetes delete CRD hangs indefinetly, as with such command:

kubectl delete crd postgresqls.databases.scalingo.com

to force the deletion, remove the Finalizers manually:

# example
$KIND=PostgreSQL
$NAME=postgresql-sample
$CRD=postgresqls.databases.scalingo.com

# edit the current CRD descriptor, and remove this block:
#
#   metadata:
#       finalizers: []
#
kubectl edit $KIND $NAME

# then, retry the delete
kubectl delete crd $CRD

Force-pull the Operator image

Set the Manager parameter imagePullPolicy to Always (default value being IfNotPresent).

In config/manager/manager.yaml, add this line:

imagePullPolicy: Always

bellow:

image: controller:latest

then, execute make deploy IMG=....

Useful Links

• Kubebuilder book
• MicroK8s
• Kubernetes operator
• Kubernetes Custom Resource Definition

Release a New Version

Warning

Add any new supported spec in config/samples/databases_v1_postgresql.yaml.

Bump new version number in:

• config/manager/kustomization.yaml, field newTag, do not forget the prefix v,
• CHANGELOG.md,
• README.md: in main title and VERSION.

Commit, tag and create a new release:

VERSION="1.2.0"

git switch --create release/${VERSION}
git add .
git commit --message="feat: bump v${VERSION}"
git push --set-upstream origin release/${VERSION}
gh pr create --reviewer=scalingo/team-ist --fill-first

Once the pull request merged, you can tag the main branch:

git tag v${VERSION}
git push origin main v${VERSION}

After tagging the branch, the process to release the archive with the installer is automatically handled by GitHub Actions. The release will be published on the Scalingo Operator page.

Integration Tests

Integration tests are minimal and remain in progress.

Execute with:

go test -tag integration ./...