[mweibel]

This change implements two new features in separate commits:

Pool Member Network for LoadBalancer

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: CloudscaleCluster
metadata:
  name: "${CLUSTER_NAME}"
  namespace: "${NAMESPACE}"
spec:
  # [...] rest of the spec [...]
  controlPlaneLoadBalancer:
    # network omitted -> public VIP (external API endpoint)
    poolMemberNetwork: "${CLUSTER_NAME}-cp"

When specifying a poolMemberNetwork on the controlPlaneLoadBalancer, the LoadBalancer Members get created with the Network->Subnet specified to what has been configured. This allows us to better specify the LoadBalancer frontend and backend networks. The existing network setting will place the LoadBalancer frontend on that network when specified. If not specified, the LB will get a VIP from the public network instead.

Router
Implements the newly released Router feature for CAPCS.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: CloudscaleCluster
metadata:
  name: "${CLUSTER_NAME}"
  namespace: "${NAMESPACE}"
spec:
  # [...] rest of the spec [...]
  routers:
  - name: "${CLUSTER_NAME}-router"
    internetGateway: true  # enables SNAT for outbound internet access
    interfaces:
    - network: "${CLUSTER_NAME}"

A CloudscaleCluster can now be set up with 0-n routers, allowing for much more flexible and private network topologies. The accompanying new template router-nat places the control-plane and worker nodes in a private network and does not attach them to the public network. Using the router with internetGateway: true and set as the default gateway of the specified network, will enable the nodes to communicate with the internet while ingress is only permitted via the LB -> Kubernetes API server.

Reviewer notes
The govulncheck failure is because the specified Go version has some potential vulnerabilities fixed in a newer version. Fixing this would be part of a separate PR.

[alakae]

should it read topology-router-nat?

Or ## Using the topology-router-nat flavor ?

[alakae]

after discussion, what about two sections:

## Using Templates with Topology/ClusterClass

## topology-router-nat specific properties

[mweibel]

adjusted +/- as discussed

[alakae]

I'd not have dropped the details on networkName/networkCIDR/routerName) - maybe recover them into a NAT/router section. What do you think?

[alakae]

not strictly related to this PR, but makes following getting-started.md slightly easier to follow. feel free to apply, if you agree.

diff --git a/docs/getting-started.md b/docs/getting-started.md
index ed9d208..3b0827d 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -212,6 +212,13 @@ Watch progress:
 clusterctl describe cluster my-cluster
 ```
 
+Wait for the control plane to come up enough to hand out a kubeconfig. This
+doesn't need CCM or CNI yet:
+
+```bash
+kubectl wait --for=condition=ControlPlaneInitialized cluster/my-cluster --timeout=15m
+```
+
 ## 8. Install the cloudscale CCM
 
 CAPCS only provisions infrastructure. For a working cluster
@@ -236,17 +243,16 @@ the CCM starts and removes the taint — that is expected.
 
 ## 9. Install a CNI (Cilium)
 
-Fetch the workload cluster's kubeconfig and point `KUBECONFIG` at it:
+Fetch the workload cluster's kubeconfig:
 
 ```bash
 clusterctl get kubeconfig my-cluster > ~/.kube/my-cluster.yaml
-export KUBECONFIG=~/.kube/my-cluster.yaml
 ```
 
 Install Cilium with defaults:
 
 ```bash
-cilium install
+KUBECONFIG=~/.kube/my-cluster.yaml cilium install --wait
 ```
 
 Nodes become `Ready` once Cilium is up. Any standard CNI works — Cilium is just
@@ -256,26 +262,34 @@ the example.
 
 ```bash
 # management cluster
-clusterctl describe cluster <name>
+clusterctl describe cluster my-cluster
 
 # workload cluster
-kubectl get nodes
-kubectl -n kube-system get pods
+KUBECONFIG=~/.kube/my-cluster.yaml kubectl get nodes
+KUBECONFIG=~/.kube/my-cluster.yaml kubectl -n kube-system get pods
 ```
 
 All nodes should report `Ready`. In `kube-system` you should see
 `cloudscale-cloud-controller-manager-*` on each control plane node and
 `cilium-*` on every node.
 
+Now wait for the cluster to report fully available.
+
+```bash
+kubectl wait --for=condition=Available cluster/my-cluster --timeout=15m
+```
+
 If something is stuck, see [troubleshooting](troubleshooting.md).
 
 ## Optional: persistent storage (cloudscale CSI)
 
 For `PersistentVolumeClaim` support, install the
 [cloudscale CSI driver](https://github.com/cloudscale-ch/csi-cloudscale) on the
-**workload** cluster. With `KUBECONFIG` still pointing at the workload cluster:
+**workload** cluster:
 
 ```bash
+export KUBECONFIG=~/.kube/my-cluster.yaml
+
 kubectl -n kube-system create secret generic cloudscale \
   --from-literal=access-token="$CLOUDSCALE_API_TOKEN"
 

[alakae]

What is the added value of this property?

Can't users just use it like this

spec:
  credentialsRef:
    name: my-cluster-credentials
  region: lpg
  networks:
  - cidr: 10.10.0.0/24
    gatewayAddress: 10.10.0.99
    name: my-cluster
  routers:
  - interfaces:
    - network: my-cluster
      address: 10.10.0.99
#      configureSubnetGateway: true # <- without this
    internetGateway: true
    name: my-cluster-router

[mweibel]

it's more of a convenience: address is defaulted if not set, and together with configureSubnetGateway: true|false, the user can influence whether the address is set as the gateway address or not.

[mweibel]

after discussion: we'll leave it away. There are good benefits of it but also a few cons, and in the spirit of having a minimal API: let's leave it away, instead improve docs.