diff --git a/docs/cli.md b/docs/cli.md index 4ff9991f..06165ecf 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -236,9 +236,10 @@ The flags pin a value regardless of the discovery, with or without | `--coding-harnesses ` | The coding Harnesses `--coding` renders, by runtime: `claude`, `codex`, `omp` or several (`claude,codex,omp`), each named after its runtime; `""` returns to `claude` alone. `codex` needs a kagent line whose chart takes `runtime: codex`; `omp` (oh-my-pi, a BYO Harness on the kagent line's `omp-harness` adapter) one whose chart takes `runtime: byo` and an image from `--coding-images`. A dev build of a coding Harness's image swaps in through `platform.devImages` `claude-harness` / `codex-harness` / `omp-harness`; `claude-harness` and `codex-harness` need `kagent` from the same build, whose controller pins the CLI version ([Coding Harness targets](platform.md#coding-harness-targets)). | | `--coding-images =` | The image of a coding Harness the chart has none of its own for, pinned by digest (`platform.coding.images`): `omp=gsoci.azurecr.io/giantswarm/kagent/omp-harness@sha256:…`, the `runtimeImages.ompHarness.digest` the release's kagent chart stamps. A dev image (`platform.devImages` `omp-harness`) wins over it. | | `--serving[=false]` | Serve models on llm-d in the lab, or turn it off: the KServe llmisvc controller and its CRDs, the well-known runtime configs, the connectivity chart's serving slice with the models Gateway, model-manager's `kserve` backend and one CPU preset of the lab's (needs agents; installs cert-manager; agent-platform 4.44.0 or newer). See [Model serving on llm-d](models.md#model-serving-on-llm-d-platformserving). | -| `--workspaces[=false]` | Install the workspace storage, or turn it off: an in-cluster NFS server, the NFS CSI driver `nfs.csi.k8s.io` with its controller behind an Envoy mTLS proxy only Agent Substrate's API server may reach (its pod-identity certificate), and the read-write-many StorageClass `agentlab-workspaces`; needs the host kernel's `nfsd` and `nfs` modules (available, not loaded by hand: the kernel loads them). Needs the agents (`--agents`): the workspaces live in kagent's namespace, and the lab refuses them without it. See [Workspaces](workspaces.md). | +| `--workspaces[=false]` | Install the workspace storage, or turn it off: an in-cluster NFS server, the NFS CSI driver `nfs.csi.k8s.io` with its controller behind an Envoy mTLS proxy only Agent Substrate's API server may reach (its pod-identity certificate), and the read-write-many StorageClass `agentlab-workspaces`; needs the host kernel's `nfsd` and `nfs` modules (available, not loaded by hand: the kernel loads them); under rootless Podman on Linux the hostpath CSI driver backs the volumes instead (`--workspaces-storage`). Needs the agents (`--agents`): the workspaces live in kagent's namespace, and the lab refuses them without it. See [Workspaces](workspaces.md). | | `--workspaces-provider ` | The workspace-manager's provider instances: `github` (the default, `""` too), a GitHub App of the lab's own on github.com, the instance once the App's ids are set and the Secret `agent-platform/workspace-github` carries `private-key` and `client-secret`; `gitlab`, a GitLab's OAuth application; or both. With workspaces on, the base URL `https://workspace-manager.:` and its route are in place regardless; each instance's App registers muster's `/oauth/proxy/callback`. See [Workspaces](workspaces.md#the-provider-instances). | | `--workspaces-csi-companion[=false]` | With workspaces on, the connectivity chart's CSI companion is Substrate's workspace driver (`platform.workspaces.csiCompanion`): its `efs` preset running the lab's NFS plugin, the lab's NFS driver serving the volumes as `efs.csi.aws.com` — the lab path for proving a companion chart change; needs agent-platform 4.128.0 or newer. Switching takes the workspace storage apart first, its volumes with it. See [The CSI companion](workspaces.md#the-csi-companion). | +| `--workspaces-storage ` | What backs the workspaces' volumes (`platform.workspaces.storage`): `nfs`, an NFS server in the cluster and csi-driver-nfs, which needs a rootful engine on Linux; `hostpath`, csi-driver-host-path, a directory of the single node per volume, bind-mounted, which runs under rootless Podman; `""` (the default) picks `hostpath` under rootless Podman on Linux for the single-node lab without the companion and `nfs` elsewhere. `hostpath` refuses `substrateNodes` above 0 and `--workspaces-csi-companion`. See [Rootless Podman: the hostpath storage](workspaces.md#rootless-podman-the-hostpath-storage). | | `--workspaces-cleanup-after ` | How long a Session's directory on a workspace volume outlives its last turn (`platform.workspaces.sessionsCleanupAfter`, the workspace-manager's `sessions.cleanupAfter`): a Go duration; `""` returns to the chart's 30 days. | | `--workspaces-refresh-after ` | The refresh threshold (`platform.workspaces.syncRefreshAfter`, the workspace-manager's `sync.refreshAfter`): a Session started on a workspace whose last sync is older requests a sync; a Go duration; `""` returns to the manager's 1h. | | `--workspaces-github-app-id`, `--workspaces-github-client-id` | The public ids of the `github` instance's App (`platform.workspaces.github`); its private key and client secret never pass through agentlab. | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index a6782e29..3b80dcf6 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -172,6 +172,30 @@ plumbing for model servers under A new node leaves them alone; if `up` still reports a change, the warning names the key and the command that restores it. +## Rootless Podman's kernel limits + +Under rootless Podman on Linux every container of the lab, the kind node's +own included, runs as your user, so two per-user kernel defaults bound the +whole lab. Neither can be raised from inside the node. + +- **A pod fails to start with `unable to create session key: disk quota + exceeded`.** Every container gets a session keyring, counted against your + user's key quota (`kernel.keys.maxkeys` 200 and `maxbytes` 20000 by + default; `/proc/key-users` shows the use). A full lab comes close to it on + its own, and leftover containers of other tools (testcontainers whose + reaper never ran) count too. Raise it and keep it: + `printf 'kernel.keys.maxkeys = 20000\nkernel.keys.maxbytes = 400000\n' | sudo tee /etc/sysctl.d/90-rootless-keys.conf && sudo sysctl --system`. +- **Services stop reaching their pods while the pods answer on their own + addresses.** kube-proxy logs `Failed to execute iptables-restore` with + `sendmsg() failed: Message too long` and keeps the rules it last wrote, so + every Service whose pods changed since points at addresses that are gone + (an edge's connect to the kagent controller times out, muster cannot reach + agent-manager). Inside a user namespace iptables cannot force a socket + buffer above `net.core.wmem_default` (212992 bytes by default), and the + lab's ruleset outgrows it. Raising it on the host takes effect in the + running node at kube-proxy's next sync: + `printf 'net.core.wmem_default = 4194304\nnet.core.rmem_default = 4194304\n' | sudo tee /etc/sysctl.d/91-rootless-netlink.conf && sudo sysctl --system`. + ## vm-manager - **`vm-manager-test` fails at `get_vm_attestation` with `golden mismatch: diff --git a/docs/workspaces.md b/docs/workspaces.md index b1b2273a..ed5f4b95 100644 --- a/docs/workspaces.md +++ b/docs/workspaces.md @@ -52,6 +52,50 @@ namespace, or with the cluster on `down`) and the export's bytes with it. A server that stopped first would leave a node's NFS client waiting on it for good, a node that can then be neither stopped nor removed. +### Rootless Podman: the hostpath storage + +Under rootless Podman on Linux the nodes run in your user's user namespace, +where the kernel mounts neither the `nfsd` filesystem nor an NFS export: the +`nfs-server` pod logs `Unable to access /proc/fs/nfsd` and never listens, and +every NFS mount is refused. `platform.workspaces.storage` picks what backs the +volumes: + +| Value | What | +|---|---| +| `nfs` | The NFS server and csi-driver-nfs above | +| `hostpath` | kubernetes-csi/csi-driver-host-path v1.17.1, as Substrate's own `ate-setup csi hostpath` installs it: every volume a directory of the node's `/var/lib/agentlab-workspaces` (beside the driver's `state.json`), published with a bind mount, which a user namespace allows | +| empty (the default) | `hostpath` under rootless Podman on Linux for the single-node lab without the companion, `nfs` everywhere else | + +`agentlab configure --workspaces-storage ` sets it; `""` +returns to the default. The hostpath storage: + +| Piece | What | +|---|---| +| The `csi-hostpath-node` DaemonSet (namespace `agentlab-workspaces`) | The driver `hostpath.csi.k8s.io` on the control plane, serving the controller and the node service on `/var/lib/kubelet/plugins/csi-hostpath/csi.sock`, with the provisioner, the resizer and the registrar beside it; Substrate's `/var/lib/ate` mounted `Bidirectional` | +| The controller endpoint | The `csi-hostpath-controller` Deployment: the same Envoy mTLS proxy as the NFS controller's, alone, dialling the driver's socket on the node, behind the Service `csi-hostpath-controller` (port 50051, TLS server name `csi-hostpath-controller.agentlab-workspaces.svc`) | +| The StorageClass `agentlab-workspaces` | Read-write-many on the one node, `Immediate` binding, delete-reclaiming, expandable; no mount options | +| The `CSIDriverConfig` `hostpath.csi.k8s.io` | As the NFS driver's, from the same `workspaces:` values | + +A volume is a directory of one node, so the hostpath storage needs the +single-node lab: `substrateNodes` above 0 is refused with it, and so is +`csiCompanion`, whose preset mounts the lab's NFS volumes. Nothing is +installed on the host and no kernel module is needed. The plugin image is +`registry.k8s.io`'s, pinned by digest (gsoci carries no mirror of it). + +The hostpath driver keeps one staging path per volume on a node and refuses +a second (`volume … is already staged at …`). Two orchestrators stage on the +node, each at its own path: the kubelet for a pod's claim and atelet for an +actor's existing volume. So a pod and an actor cannot hold one volume at the +same time on this storage: a workspace sync beside a running Session, or +`workspaces-test --storage-only`'s report pod reading the volume its actors +hold. Actors sharing a volume among themselves (a Session's directory +read-write and the mirrors read-only, two Sessions of one workspace) need a +Substrate that stages a multi-node volume once per node; earlier releases +stage it once per target and are refused at the second mount. The claims, +the sessions' sub-paths and the controller endpoint's refusal pass on any +release. csi-driver-nfs advertises no staging, which is why the NFS storage +meets neither limit; the whole workspace flow needs it. + ### Turning it on ```sh diff --git a/internal/config/config.go b/internal/config/config.go index 5cc12865..0375d482 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -702,8 +702,8 @@ type Tenants struct { // Workspaces configures the workspace storage in the lab. type Workspaces struct { - // On, `agentlab platform` installs the NFS server, the NFS CSI driver - // behind its mTLS proxy and the StorageClass before the chart, and + // On, `agentlab platform` installs the storage (Storage), its CSI + // driver behind its mTLS proxy and the StorageClass before the chart, and // renders the chart's `workspaces:` block when the chart carries the // key. Enabled bool `yaml:"enabled"` @@ -747,6 +747,15 @@ type Workspaces struct { // the preset's driver name. The lab path for proving a companion chart // change (docs/workspaces.md); needs a chart that carries the key. CSICompanion bool `yaml:"csiCompanion,omitempty"` + // Storage is what backs the workspaces' volumes: WorkspaceStorageNFS, an + // NFS server in the cluster and csi-driver-nfs, or + // WorkspaceStorageHostPath, csi-driver-host-path, a directory of the one + // node per volume, bind-mounted where it is published. Empty picks + // hostpath under rootless Podman on Linux, where the kernel refuses + // nfsd and NFS mounts to the node's user namespace, and nfs elsewhere. + // hostpath needs the single-node lab (substrateNodes 0) and refuses + // csiCompanion, whose preset mounts the lab's NFS volumes. + Storage string `yaml:"storage,omitempty"` } // WorkspaceGitLab is the GitLab behind the workspace-manager's provider @@ -893,6 +902,12 @@ const ( WorkspaceGitLabPasswordKey = "person-password" // #nosec G101 -- Secret KEY name, not a credential ) +// The workspace storages platform.workspaces.storage names. +const ( + WorkspaceStorageNFS = "nfs" + WorkspaceStorageHostPath = "hostpath" +) + // WorkspaceGrantProviderApps is the grantProvider that pins no instance: // one muster provider App per instance, side by side. const WorkspaceGrantProviderApps = "apps" @@ -2392,6 +2407,18 @@ func (c *Config) Validate() error { if c.Platform.Workspaces.Enabled && c.Platform.Enabled && !c.Platform.Agents { return fmt.Errorf("platform.workspaces requires platform.agents (the workspaces live in kagent's namespace, where every Session runs and mounts its workspace) — `agentlab configure --agents`, or `--workspaces=false`") } + switch storage := c.Platform.Workspaces.Storage; storage { + case "", WorkspaceStorageNFS: + case WorkspaceStorageHostPath: + if c.SubstrateNodes > 0 { + return fmt.Errorf("platform.workspaces.storage %q publishes a volume as a directory of one node, and substrateNodes %d adds workers whose actors could not mount it: `agentlab configure --workspaces-storage %s`, or substrateNodes 0", storage, c.SubstrateNodes, WorkspaceStorageNFS) + } + if c.Platform.Workspaces.CSICompanion { + return fmt.Errorf("platform.workspaces.storage %q and csiCompanion: the companion's preset mounts the lab's NFS volumes: `agentlab configure --workspaces-storage %s`, or `--workspaces-csi-companion=false`", storage, WorkspaceStorageNFS) + } + default: + return fmt.Errorf("platform.workspaces.storage %q: want %q, %q, or empty (hostpath under rootless Podman on Linux, nfs elsewhere)", storage, WorkspaceStorageNFS, WorkspaceStorageHostPath) + } if names, err := parseWorkspaceProviders(c.Platform.Workspaces.Provider); err != nil { return err } else if grant := c.Platform.Workspaces.GrantProvider; grant == WorkspaceGrantProviderApps && len(names) < 2 { diff --git a/internal/config/workspaces_test.go b/internal/config/workspaces_test.go index bdc88474..d02b7da4 100644 --- a/internal/config/workspaces_test.go +++ b/internal/config/workspaces_test.go @@ -233,3 +233,38 @@ func TestValidateProofRepositories(t *testing.T) { } } } + +// The workspace storage: empty, nfs and hostpath accepted; hostpath refused +// beside substrate workers and beside the CSI companion; any other value +// refused. +func TestWorkspacesStorage(t *testing.T) { + for _, tc := range []struct { + name string + storage string + substrateNodes int + companion bool + wantErr string + }{ + {"empty", "", 0, false, ""}, + {"nfs", WorkspaceStorageNFS, 2, true, ""}, + {"hostpath", WorkspaceStorageHostPath, 0, false, ""}, + {"hostpath with workers", WorkspaceStorageHostPath, 1, false, "substrateNodes 1"}, + {"hostpath with the companion", WorkspaceStorageHostPath, 0, true, "csiCompanion"}, + {"another storage", "ceph", 0, false, `platform.workspaces.storage "ceph"`}, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := Default() + cfg.Platform.Workspaces.Enabled = true + cfg.Platform.Workspaces.Storage = tc.storage + cfg.Platform.Workspaces.CSICompanion = tc.companion + cfg.SubstrateNodes = tc.substrateNodes + err := cfg.Validate() + switch { + case tc.wantErr == "" && err != nil: + t.Errorf("Validate: %v, want accepted", err) + case tc.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tc.wantErr)): + t.Errorf("Validate: %v, want an error naming %q", err, tc.wantErr) + } + }) + } +} diff --git a/internal/lab/main_test.go b/internal/lab/main_test.go new file mode 100644 index 00000000..b764411e --- /dev/null +++ b/internal/lab/main_test.go @@ -0,0 +1,14 @@ +package lab + +import ( + "os" + "testing" +) + +// TestMain fixes the engine to one that is not rootless Podman, so what the +// lab picks from it (workspacesStorageFor) does not follow the machine the +// tests run on; a test of the rootless branch sets it itself. +func TestMain(m *testing.M) { + rootlessPodmanOnLinux = func() bool { return false } + os.Exit(m.Run()) +} diff --git a/internal/lab/runtime.go b/internal/lab/runtime.go index e59802a4..01260aef 100644 --- a/internal/lab/runtime.go +++ b/internal/lab/runtime.go @@ -2,6 +2,7 @@ package lab import ( "os" + "runtime" "strconv" "strings" "sync" @@ -59,6 +60,14 @@ func MinPublishablePort() int { return unprivilegedPortStart(os.ReadFile) } +// rootlessPodmanOnLinux reports whether the lab's nodes are rootless Podman +// containers on this machine's kernel: they run in the invoking user's user +// namespace, where the kernel mounts neither the nfsd filesystem nor an NFS +// export. +var rootlessPodmanOnLinux = func() bool { + return runtime.GOOS == goosLinux && dockerIsPodman() && os.Geteuid() != 0 +} + // unprivilegedPortStart reads net.ipv4.ip_unprivileged_port_start. An // unreadable or unparsable sysctl means the kernel default holds. func unprivilegedPortStart(readFile func(string) ([]byte, error)) int { diff --git a/internal/lab/templates/workspaces.yaml.tmpl b/internal/lab/templates/workspaces.yaml.tmpl index fabdd54e..f60f834d 100644 --- a/internal/lab/templates/workspaces.yaml.tmpl +++ b/internal/lab/templates/workspaces.yaml.tmpl @@ -15,6 +15,14 @@ # the chart's companion is Substrate's driver (platform.workspaces.csiCompanion), # so the volumes are the ones the companion mounts. # +# With the hostpath storage (platform.workspaces.storage hostpath) the hostpath +# CSI driver takes the NFS server's and csi-driver-nfs's place: every volume a +# directory of {{ .Workspaces.ExportDir }} on the one node, published with a bind +# mount, which a rootless node's user namespace allows where the kernel refuses +# it nfsd and NFS mounts. Its DaemonSet runs the driver (controller and node +# service on one socket) with the provisioner, the resizer and the registrar; +# the controller Deployment is the mTLS proxy alone, dialling that socket. +# # Order matters twice: the controller's Service comes before its Deployment, # since Substrate's service-DNS signer certifies a pod for the Service that # selects it; the NFS server comes before the driver, whose provisioner mounts @@ -30,6 +38,7 @@ metadata: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged +{{- if not .Workspaces.HostPath }} --- # The NFS server: the kernel's nfsd (the kind node shares the host's kernel, # whose nfsd module it loads on the first export), exporting @@ -99,8 +108,7 @@ spec: path: {{ .Workspaces.ExportDir }} type: DirectoryOrCreate --- -# The driver's accounts and roles (csi-driver-nfs deploy/rbac-csi-nfs.yaml, -# without the snapshot rules: the lab installs no snapshot CRDs). +# The driver's accounts (csi-driver-nfs deploy/rbac-csi-nfs.yaml). apiVersion: v1 kind: ServiceAccount metadata: @@ -112,11 +120,23 @@ kind: ServiceAccount metadata: name: csi-nfs-node namespace: {{ .Workspaces.Namespace }} +{{- else }} --- +# The hostpath driver's account: its DaemonSet runs the provisioner and the +# resizer beside the driver. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Workspaces.ProvisionerAccount }} + namespace: {{ .Workspaces.Namespace }} +{{- end }} +--- +# The provisioner's and the resizer's roles, without the snapshot rules: the +# lab installs no snapshot CRDs. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: agentlab-workspaces-csi-nfs-provisioner + name: {{ .Workspaces.RBACName }}-provisioner rules: - apiGroups: [""] resources: ["persistentvolumes"] @@ -146,20 +166,20 @@ rules: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: agentlab-workspaces-csi-nfs-provisioner + name: {{ .Workspaces.RBACName }}-provisioner subjects: - kind: ServiceAccount - name: csi-nfs-controller + name: {{ .Workspaces.ProvisionerAccount }} namespace: {{ .Workspaces.Namespace }} roleRef: kind: ClusterRole - name: agentlab-workspaces-csi-nfs-provisioner + name: {{ .Workspaces.RBACName }}-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: agentlab-workspaces-csi-nfs-resizer + name: {{ .Workspaces.RBACName }}-resizer rules: - apiGroups: [""] resources: ["persistentvolumes"] @@ -180,17 +200,18 @@ rules: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: agentlab-workspaces-csi-nfs-resizer + name: {{ .Workspaces.RBACName }}-resizer subjects: - kind: ServiceAccount - name: csi-nfs-controller + name: {{ .Workspaces.ProvisionerAccount }} namespace: {{ .Workspaces.Namespace }} roleRef: kind: ClusterRole - name: agentlab-workspaces-csi-nfs-resizer + name: {{ .Workspaces.RBACName }}-resizer apiGroup: rbac.authorization.k8s.io --- -# The driver's registration with the cluster (deploy/csi-nfs-driverinfo.yaml): +# The driver's registration with the cluster (deploy/csi-nfs-driverinfo.yaml; +# the hostpath driver's likewise, its attach step off as the driver runs it): # no attach step, POSIX permissions applied by the driver. apiVersion: storage.k8s.io/v1 kind: CSIDriver @@ -212,13 +233,13 @@ spec: apiVersion: v1 kind: ConfigMap metadata: - name: csi-nfs-proxy + name: {{ .Workspaces.ProxyConfig }} namespace: {{ .Workspaces.Namespace }} data: envoy.yaml: | node: - id: csi-nfs-proxy - cluster: csi-nfs-proxy + id: {{ .Workspaces.ProxyConfig }} + cluster: {{ .Workspaces.ProxyConfig }} static_resources: listeners: - name: csi_tls_listener @@ -300,6 +321,279 @@ spec: port: {{ .Workspaces.ControllerPort }} targetPort: {{ .Workspaces.ProxyPort }} protocol: TCP +{{- if .Workspaces.HostPath }} +--- +# The controller endpoint of the hostpath storage: the mTLS proxy alone, on +# the node whose driver socket it dials (the DaemonSet's, at +# {{ .Workspaces.NodeSocketDir }}), only with Substrate, whose signers alone +# certify it. Privileged like upstream's sidecars, which reach the driver's +# socket on an SELinux host only so. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Workspaces.Controller }} + namespace: {{ .Workspaces.Namespace }} +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: {{ .Workspaces.Controller }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ .Workspaces.Controller }} + spec: + priorityClassName: system-cluster-critical + nodeSelector: + kubernetes.io/hostname: {{ .Workspaces.Node }} + tolerations: + - key: node-role.kubernetes.io/control-plane + operator: Exists + effect: NoSchedule + containers: + - name: envoy + image: {{ .Workspaces.Images.Proxy }} + imagePullPolicy: IfNotPresent + command: + - /usr/local/bin/envoy + args: + - -c + - /etc/envoy/envoy.yaml + securityContext: + privileged: true + runAsUser: 0 + ports: + - containerPort: {{ .Workspaces.ProxyPort }} + name: grpc + protocol: TCP + volumeMounts: + - mountPath: /etc/envoy + name: envoy-config + readOnly: true + - mountPath: /csi + name: socket-dir + - mountPath: /run/servicedns + name: servicedns-certs + readOnly: true + - mountPath: /run/podidentity-ca + name: podidentity-ca + readOnly: true + volumes: + - name: socket-dir + hostPath: + path: {{ .Workspaces.NodeSocketDir }} + type: DirectoryOrCreate + - name: envoy-config + configMap: + name: {{ .Workspaces.ProxyConfig }} + - name: servicedns-certs + projected: + sources: + - podCertificate: + signerName: servicedns.podcert.ate.dev/identity + keyType: ECDSAP256 + credentialBundlePath: credential-bundle.pem + - name: podidentity-ca + projected: + sources: + - clusterTrustBundle: + signerName: podidentity.podcert.ate.dev/identity + labelSelector: + matchLabels: + podcert.ate.dev/canarying: live + path: trust-bundle.pem +--- +# The hostpath driver (deploy/csi-hostpath-plugin.yaml without the attacher, +# the snapshotter and the health monitor): the driver serving the controller +# and the node service on {{ .Workspaces.NodeSocketDir }}/csi.sock, registered +# with the kubelet there, keeping each volume as a directory of +# {{ .Workspaces.ExportDir }} beside its state.json, and with Substrate's +# directory {{ .Workspaces.SubstrateDir }} mounted Bidirectional, so a bind +# mount the driver makes for atelet's target path is one the actor's sandbox +# sees. The sidecars are privileged as upstream's are, for the driver's socket +# on an SELinux host. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ .Workspaces.NodePlugin }} + namespace: {{ .Workspaces.Namespace }} +spec: + updateStrategy: + rollingUpdate: + maxUnavailable: 1 + type: RollingUpdate + selector: + matchLabels: + app.kubernetes.io/name: {{ .Workspaces.NodePlugin }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ .Workspaces.NodePlugin }} + spec: + serviceAccountName: {{ .Workspaces.ProvisionerAccount }} + priorityClassName: system-node-critical + nodeSelector: + kubernetes.io/hostname: {{ .Workspaces.Node }} + tolerations: + - operator: "Exists" + containers: + - name: hostpath + image: {{ .Workspaces.Images.HostPathPlugin }} + imagePullPolicy: IfNotPresent + args: + - "--drivername={{ .Workspaces.Driver }}" + - "--v=5" + - "--endpoint=$(CSI_ENDPOINT)" + - "--nodeid=$(KUBE_NODE_NAME)" + - "--statedir=/csi-data-dir" + env: + - name: CSI_ENDPOINT + value: unix:///csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + securityContext: + privileged: true + ports: + - containerPort: 9898 + name: healthz + protocol: TCP + livenessProbe: + failureThreshold: 5 + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 2 + volumeMounts: + - mountPath: /csi + name: socket-dir + - mountPath: /var/lib/kubelet/pods + mountPropagation: Bidirectional + name: pods-mount-dir + - mountPath: /var/lib/kubelet/plugins + mountPropagation: Bidirectional + name: plugins-dir + - mountPath: /csi-data-dir + name: data-dir + - mountPath: /dev + name: dev-dir + - mountPath: {{ .Workspaces.SubstrateDir }} + mountPropagation: Bidirectional + name: substrate-dir + - name: csi-provisioner + image: {{ .Workspaces.Images.Provisioner }} + imagePullPolicy: IfNotPresent + args: + - "-v=2" + - "--csi-address=/csi/csi.sock" + - "--leader-election" + - "--leader-election-namespace=$(POD_NAMESPACE)" + - "--extra-create-metadata=true" + - "--feature-gates=Topology=true,VolumeAttributesClass=false" + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + securityContext: + privileged: true + volumeMounts: + - mountPath: /csi + name: socket-dir + - name: csi-resizer + image: {{ .Workspaces.Images.Resizer }} + imagePullPolicy: IfNotPresent + args: + - "-v=2" + - "-csi-address=/csi/csi.sock" + - "-leader-election" + - "--leader-election-namespace=$(POD_NAMESPACE)" + - "-handle-volume-inuse-error=false" + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + securityContext: + privileged: true + volumeMounts: + - mountPath: /csi + name: socket-dir + - name: node-driver-registrar + image: {{ .Workspaces.Images.Registrar }} + imagePullPolicy: IfNotPresent + args: + - --v=2 + - --csi-address=/csi/csi.sock + - --kubelet-registration-path={{ .Workspaces.NodeSocketDir }}/csi.sock + env: + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + securityContext: + privileged: true + volumeMounts: + - mountPath: /csi + name: socket-dir + - mountPath: /registration + name: registration-dir + - name: liveness-probe + image: {{ .Workspaces.Images.LivenessProbe }} + imagePullPolicy: IfNotPresent + args: + - --csi-address=/csi/csi.sock + - --health-port=9898 + volumeMounts: + - mountPath: /csi + name: socket-dir + volumes: + - name: socket-dir + hostPath: + path: {{ .Workspaces.NodeSocketDir }} + type: DirectoryOrCreate + - name: pods-mount-dir + hostPath: + path: /var/lib/kubelet/pods + type: DirectoryOrCreate + - name: plugins-dir + hostPath: + path: /var/lib/kubelet/plugins + type: Directory + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry + type: Directory + - name: data-dir + hostPath: + path: {{ .Workspaces.ExportDir }} + type: DirectoryOrCreate + - name: dev-dir + hostPath: + path: /dev + type: Directory + - name: substrate-dir + hostPath: + path: {{ .Workspaces.SubstrateDir }} + type: DirectoryOrCreate +--- +# The class the chart's workspaces values name: a directory of the node per +# volume, bind-mounted; read-write-many on the one node, POSIX modes and +# symbolic links as git needs them. Expandable in name, as the NFS class is. +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ .Workspaces.StorageClass }} +provisioner: {{ .Workspaces.Driver }} +reclaimPolicy: Delete +volumeBindingMode: Immediate +allowVolumeExpansion: true +{{- else }} --- # The controller (deploy/csi-nfs-controller.yaml without the snapshotter): # the provisioner and resizer sidecars for the cluster's PVCs, the liveness @@ -472,7 +766,7 @@ spec: emptyDir: {} - name: envoy-config configMap: - name: csi-nfs-proxy + name: {{ .Workspaces.ProxyConfig }} - name: servicedns-certs projected: sources: @@ -638,3 +932,4 @@ allowVolumeExpansion: true mountOptions: - nfsvers=4.1 - hard +{{- end }} diff --git a/internal/lab/tenantstest.go b/internal/lab/tenantstest.go index 00412c65..74caaf89 100644 --- a/internal/lab/tenantstest.go +++ b/internal/lab/tenantstest.go @@ -1090,7 +1090,7 @@ func credentialProviderGrants(ctx context.Context) (map[string][]string, error) // ClusterRole: a RoleBinding to a missing ClusterRole is accepted and grants // nothing, so the calls would fail as a bare 403. func checkTenantRole(role string) error { - gvr, err := gvrFor("clusterroles.rbac.authorization.k8s.io") + gvr, err := gvrFor(clusterRolesResource) if err != nil { return err } diff --git a/internal/lab/workspaces.go b/internal/lab/workspaces.go index f1d38391..19dd6513 100644 --- a/internal/lab/workspaces.go +++ b/internal/lab/workspaces.go @@ -151,24 +151,131 @@ const ( nfsPluginRepository = "giantswarm/nfsplugin" ) -// workspacesCSIDrivers are the names the lab's NFS driver serves volumes -// under: its own, and the companion preset's. -var workspacesCSIDrivers = []string{workspacesCSIDriver, companionCSIDriver} +// The hostpath storage (platform.workspaces.storage hostpath), the shape of +// agent-substrate/substrate's own `ate-setup csi hostpath`: csi-driver-host-path +// keeps every volume as a directory of the one node and publishes it with a +// bind mount, which a rootless node's user namespace allows where the kernel +// refuses it nfsd and NFS mounts. One DaemonSet runs the driver, which serves +// the controller and the node service on one socket, with the provisioner, +// the resizer and the registrar beside it; the controller Deployment is the +// mTLS proxy alone, dialling that socket on the node. +const ( + hostPathCSIDriver = "hostpath.csi.k8s.io" + hostPathController = "csi-hostpath-controller" + hostPathControllerService = "csi-hostpath-controller" + hostPathNodePlugin = "csi-hostpath-node" + hostPathNodeSocketDir = "/var/lib/kubelet/plugins/csi-hostpath" + // hostPathPluginImage has no gsoci mirror; registry.k8s.io's, pinned by + // digest. + hostPathPluginImage = "registry.k8s.io/sig-storage/hostpathplugin:v1.17.1@sha256:ec6ded2430d4c5c6251e8d3a6ca55c675d6bd1b1ee8c012eab13aaa81be9c967" +) + +// workspacesCSIDrivers are the names the lab's drivers serve volumes under: +// the NFS driver's own, the companion preset's, and the hostpath driver's. +var workspacesCSIDrivers = []string{workspacesCSIDriver, companionCSIDriver, hostPathCSIDriver} // isWorkspacesCSIDriver reports whether a CSI driver name is one the lab's -// NFS driver serves volumes under. +// drivers serve volumes under. func isWorkspacesCSIDriver(name string) bool { return slices.Contains(workspacesCSIDrivers, name) } -// workspacesCSIDriverFor is the name the lab's NFS driver serves the -// volumes under: the companion preset's driver while the companion is -// Substrate's driver, csi-driver-nfs's own otherwise. -func workspacesCSIDriverFor(cfg *config.Config) string { +// workspacesStorageFor is the storage the lab's workspace volumes are made +// of: platform.workspaces.storage, or while it is empty hostpath under +// rootless Podman on Linux for the single-node lab without the companion, +// nfs otherwise. +func workspacesStorageFor(cfg *config.Config) string { + if storage := cfg.Platform.Workspaces.Storage; storage != "" { + return storage + } + if rootlessPodmanOnLinux() && cfg.SubstrateNodes == 0 && !cfg.WorkspacesCSICompanion() { + return config.WorkspaceStorageHostPath + } + return config.WorkspaceStorageNFS +} + +// workspacesBackend names the objects of one storage: the driver the class +// provisions with, the controller Deployment behind its Service, the node +// plugin DaemonSet and its socket directory, and whether an NFS server runs. +type workspacesBackend struct { + storage, driver string + controller, controllerService, nodePlugin string + nodeSocketDir string + nfsServer bool +} + +func workspacesBackendFor(cfg *config.Config) workspacesBackend { + if workspacesStorageFor(cfg) == config.WorkspaceStorageHostPath { + return hostPathBackend + } + b := nfsBackend if cfg.WorkspacesCSICompanion() { - return companionCSIDriver + b.driver = companionCSIDriver + } + return b +} + +// workspacesBackendOfDriver is the backend whose class provisions with a +// driver, read off the cluster: the hostpath driver's, or the NFS one's. +func workspacesBackendOfDriver(driver string) workspacesBackend { + if driver == hostPathCSIDriver { + return hostPathBackend + } + b := nfsBackend + if driver != "" { + b.driver = driver + } + return b +} + +var ( + nfsBackend = workspacesBackend{ + storage: config.WorkspaceStorageNFS, driver: workspacesCSIDriver, + controller: workspacesController, controllerService: workspacesControllerService, nodePlugin: workspacesNodePlugin, + nodeSocketDir: workspacesNodeSocketDir, nfsServer: true, + } + hostPathBackend = workspacesBackend{ + storage: config.WorkspaceStorageHostPath, driver: hostPathCSIDriver, + controller: hostPathController, controllerService: hostPathControllerService, nodePlugin: hostPathNodePlugin, + nodeSocketDir: hostPathNodeSocketDir, + } +) + +// serverName is the DNS name the service-DNS signer certifies for the +// controller's proxy, ..svc: the TLS server name +// ate-api-server verifies. Spelled out — Substrate's examples name +// ..svc.cluster.local, which the certificate does not +// carry, and an unset name would skip the verification. +func (b workspacesBackend) serverName() string { + return b.controllerService + "." + workspacesNamespace + ".svc" +} + +// controllerEndpoint is the CSIDriverConfig's controllerEndpoint: the +// controller Service's DNS name and port. +func (b workspacesBackend) controllerEndpoint() string { + return fmt.Sprintf("tcp://%s.%s.svc.cluster.local:%d", b.controllerService, workspacesNamespace, workspacesControllerPort) +} + +// provisionerAccount is the ServiceAccount the provisioner runs as: the NFS +// controller's, or the hostpath node plugin's, whose pod runs it. +func (b workspacesBackend) provisionerAccount() string { + if b.nfsServer { + return b.controller } - return workspacesCSIDriver + return b.nodePlugin +} + +// nodeSocket is the CSIDriverConfig's nodeSocketOverride. +func (b workspacesBackend) nodeSocket() string { + return "unix://" + b.nodeSocketDir + "/csi.sock" +} + +// workspacesCSIDriverFor is the name the lab's driver serves the volumes +// under: the hostpath driver's with that storage, the companion preset's +// driver while the companion is Substrate's driver, csi-driver-nfs's own +// otherwise. +func workspacesCSIDriverFor(cfg *config.Config) string { + return workspacesBackendFor(cfg).driver } // companionValues are the connectivity chart's @@ -205,16 +312,24 @@ const ( ) type workspacesImages struct { - NFSServer, NFSPlugin, Registrar, LivenessProbe, Provisioner, Resizer, Proxy string + NFSServer, NFSPlugin, HostPathPlugin, Registrar, LivenessProbe, Provisioner, Resizer, Proxy string } // workspacesValues are the names workspaces.yaml.tmpl and the values // template's `workspaces:` block render. type workspacesValues struct { + // HostPath renders the hostpath storage in place of the NFS server and + // csi-driver-nfs (workspacesStorageFor). + HostPath bool Namespace, Driver, DriverVersion, StorageClass string NFSServer, NFSServerHost, ExportDir string ControllerService, Controller, NodePlugin string - ControllerPort, ProxyPort int + // RBACName names the provisioner's and the resizer's cluster roles + // (-provisioner, -resizer), ProvisionerAccount the + // ServiceAccount the provisioner runs as, ProxyConfig the proxy's + // ConfigMap. + RBACName, ProvisionerAccount, ProxyConfig string + ControllerPort, ProxyPort int // ControllerEndpoint, ServerName and NodeSocket are the CSIDriverConfig's // controllerEndpoint, tls.serverName and nodeSocketOverride. ControllerEndpoint, ServerName, NodeSocket string @@ -234,23 +349,28 @@ type workspacesValues struct { } func workspacesValuesFor(cfg *config.Config) workspacesValues { + b := workspacesBackendFor(cfg) return workspacesValues{ + HostPath: b.storage == config.WorkspaceStorageHostPath, Namespace: workspacesNamespace, - Driver: workspacesCSIDriverFor(cfg), + Driver: b.driver, DriverVersion: workspacesDriverVersion, StorageClass: workspacesStorageClass, NFSServer: workspacesNFSServer, NFSServerHost: workspacesNFSServer + "." + workspacesNamespace + ".svc.cluster.local", ExportDir: workspacesExportDir, - ControllerService: workspacesControllerService, - Controller: workspacesController, - NodePlugin: workspacesNodePlugin, + ControllerService: b.controllerService, + Controller: b.controller, + NodePlugin: b.nodePlugin, + RBACName: "agentlab-workspaces-csi-" + b.storage, + ProvisionerAccount: b.provisionerAccount(), + ProxyConfig: "csi-" + b.storage + "-proxy", ControllerPort: workspacesControllerPort, ProxyPort: workspacesProxyPort, - ControllerEndpoint: fmt.Sprintf("tcp://%s.%s.svc.cluster.local:%d", workspacesControllerService, workspacesNamespace, workspacesControllerPort), - ServerName: workspacesControllerServerName(), - NodeSocket: "unix://" + workspacesNodeSocketDir + "/csi.sock", - NodeSocketDir: workspacesNodeSocketDir, + ControllerEndpoint: b.controllerEndpoint(), + ServerName: b.serverName(), + NodeSocket: b.nodeSocket(), + NodeSocketDir: b.nodeSocketDir, SubstrateDir: substrateVolumesDir, ClientSPIFFEID: ateAPIServerSPIFFEID, Node: WorkspacesNode(cfg), @@ -258,13 +378,14 @@ func workspacesValuesFor(cfg *config.Config) workspacesValues { OrganizationGroups: config.WorkspaceOrganizationGroups, Companion: companionFor(cfg), Images: workspacesImages{ - NFSServer: nfsServerImage, - NFSPlugin: nfsPluginImage, - Registrar: csiRegistrarImage, - LivenessProbe: csiLivenessProbeImage, - Provisioner: csiProvisionerImage, - Resizer: csiResizerImage, - Proxy: csiProxyImage, + NFSServer: nfsServerImage, + NFSPlugin: nfsPluginImage, + HostPathPlugin: hostPathPluginImage, + Registrar: csiRegistrarImage, + LivenessProbe: csiLivenessProbeImage, + Provisioner: csiProvisionerImage, + Resizer: csiResizerImage, + Proxy: csiProxyImage, }, } } @@ -300,24 +421,16 @@ func substrateDriverEndpointFor(cfg *config.Config) substrateDriverEndpoint { proxyPort: companionProxyPort, } } + b := workspacesBackendFor(cfg) return substrateDriverEndpoint{ namespace: workspacesNamespace, - selector: "app.kubernetes.io/name=" + workspacesController, - controller: workspacesController, - serverName: workspacesControllerServerName(), + selector: "app.kubernetes.io/name=" + b.controller, + controller: b.controller, + serverName: b.serverName(), proxyPort: workspacesProxyPort, } } -// workspacesControllerServerName is the DNS name the service-DNS signer -// certifies for the proxy, ..svc: the TLS server name -// ate-api-server verifies. Spelled out — Substrate's examples name -// ..svc.cluster.local, which the certificate does not -// carry, and an unset name would skip the verification. -func workspacesControllerServerName() string { - return workspacesControllerService + "." + workspacesNamespace + ".svc" -} - // WorkspacesNode is the node the NFS server and the driver's controller run // on: the control plane, whose disk holds the export. The node plugin runs on // every node, so the actors' workers may be anywhere. @@ -487,10 +600,15 @@ func valuesCarry(values map[string]any, path []string) bool { func workspacesUp(cfg *config.Config) error { ctx := context.Background() node := WorkspacesNode(cfg) - driver := workspacesCSIDriverFor(cfg) - step("Installing the workspace storage (an NFS server on %s, the NFS CSI driver %s as %s behind its mTLS proxy, the read-write-many StorageClass %s)", node, workspacesDriverVersion, driver, workspacesStorageClass) - if err := workspacesHostPreflight(); err != nil { - return err + b := workspacesBackendFor(cfg) + driver := b.driver + if b.nfsServer { + step("Installing the workspace storage (an NFS server on %s, the NFS CSI driver %s as %s behind its mTLS proxy, the read-write-many StorageClass %s)", node, workspacesDriverVersion, driver, workspacesStorageClass) + if err := workspacesHostPreflight(); err != nil { + return err + } + } else { + step("Installing the workspace storage (the hostpath CSI driver on %s, %s of the node per volume, behind its mTLS proxy, the read-write-many StorageClass %s)", node, workspacesExportDir, workspacesStorageClass) } // A class of the other driver name (platform.workspaces.csiCompanion // switched) cannot be changed in place: a StorageClass's provisioner is @@ -514,10 +632,22 @@ func workspacesUp(cfg *config.Config) error { if _, err := applyManifests(ctx, manifest); err != nil { return fmt.Errorf("the workspace storage: %w", err) } + if !b.nfsServer { + if err := waitDaemonSetReady(ctx, workspacesNamespace, b.nodePlugin, workspacesRolloutTimeout); err != nil { + return fmt.Errorf("the hostpath CSI driver: %w (`kubectl -n %s logs ds/%s --all-containers`)", err, workspacesNamespace, b.nodePlugin) + } + note("hostpath CSI driver %s Ready on %s, its volumes under %s (Substrate's %s mounted Bidirectional); StorageClass %s (read-write-many on the one node)", + driver, node, workspacesExportDir, substrateVolumesDir, workspacesStorageClass) + return nil + } if err := waitDeploymentRolledOut(ctx, workspacesNamespace, workspacesNFSServer, workspacesRolloutTimeout); err != nil { - return fmt.Errorf("the NFS server: %w (the kind node loads the host kernel's nfsd for it; `kubectl -n %s logs deploy/%s`)", err, workspacesNamespace, workspacesNFSServer) + hint := "the kind node loads the host kernel's nfsd for it" + if rootlessPodmanOnLinux() { + hint = "rootless Podman's node cannot start the kernel's nfsd: `agentlab configure --workspaces-storage hostpath`" + } + return fmt.Errorf("the NFS server: %w (%s; `kubectl -n %s logs deploy/%s`)", err, hint, workspacesNamespace, workspacesNFSServer) } - if err := waitDaemonSetReady(ctx, workspacesNamespace, workspacesNodePlugin, workspacesRolloutTimeout); err != nil { + if err := waitDaemonSetReady(ctx, workspacesNamespace, b.nodePlugin, workspacesRolloutTimeout); err != nil { return fmt.Errorf("the NFS CSI node plugin: %w", err) } note("NFS server exporting %s of %s; node plugin %s Ready on every node (Substrate's %s mounted Bidirectional); StorageClass %s (read-write-many, NFSv4.1)", @@ -554,12 +684,13 @@ func workspacesHostPreflight() error { // do. func waitWorkspacesProxy(ctx context.Context, cfg *config.Config) error { step("Waiting for the workspace storage's controller and its mTLS proxy (their certificate from Substrate's service-DNS signer)") - if err := waitDeploymentRolledOut(ctx, workspacesNamespace, workspacesController, workspacesRolloutTimeout); err != nil { - return fmt.Errorf("the NFS CSI controller: %w (the signer issues a certificate only to a pod the Service %s selects; `agentlab pods -n %s`, `kubectl -n %s describe deploy/%s`)", - err, workspacesControllerService, workspacesNamespace, workspacesNamespace, workspacesController) + b := workspacesBackendFor(cfg) + if err := waitDeploymentRolledOut(ctx, workspacesNamespace, b.controller, workspacesRolloutTimeout); err != nil { + return fmt.Errorf("the CSI controller: %w (the signer issues a certificate only to a pod the Service %s selects; `agentlab pods -n %s`, `kubectl -n %s describe deploy/%s`)", + err, b.controllerService, workspacesNamespace, workspacesNamespace, b.controller) } note("controller Ready: its proxy serves %s:%d with the certificate of %s, admitting %s alone", - workspacesControllerService, workspacesControllerPort, workspacesControllerServerName(), ateAPIServerSPIFFEID) + b.controllerService, workspacesControllerPort, b.serverName(), ateAPIServerSPIFFEID) if cfg.WorkspacesCSICompanion() { return waitWorkspacesCompanion(ctx) } @@ -586,21 +717,20 @@ func waitWorkspacesCompanion(ctx context.Context) error { // workspacesCSIDriverConfig is the CSIDriverConfig that registers the lab's // driver with Substrate, as the chart's workspaces.substrate.csiDriver // values render it: the lab applies it while the chart cannot. -func workspacesCSIDriverConfig() *unstructured.Unstructured { - v := workspacesValuesFor(config.Default()) +func workspacesCSIDriverConfig(b workspacesBackend) *unstructured.Unstructured { u := &unstructured.Unstructured{Object: map[string]any{}} u.SetAPIVersion("ate.dev/v1alpha1") u.SetKind("CSIDriverConfig") - u.SetName(workspacesCSIDriver) + u.SetName(b.driver) u.SetLabels(map[string]string{managedByLabel: managedByAgentlabValue}) u.Object[crSpec] = map[string]any{ - "driverName": workspacesCSIDriver, - "controllerEndpoint": v.ControllerEndpoint, - "nodeSocketOverride": v.NodeSocket, + "driverName": b.driver, + "controllerEndpoint": b.controllerEndpoint(), + "nodeSocketOverride": b.nodeSocket(), "tls": map[string]any{ "enabled": true, "usePodIdentity": true, - "serverName": v.ServerName, + "serverName": b.serverName(), }, } return u @@ -662,23 +792,24 @@ func ensureWorkspacesCSIDriverConfig(ctx context.Context, cfg *config.Config, ch if err != nil { return fmt.Errorf("the CSIDriverConfig API of Substrate is not served (%v): the agents runtime brings it; is platform.agents on and the substrate-crds component installed?", err) } - if obj, err := getObject(ctx, gvr, "", workspacesCSIDriver); err == nil && csiDriverConfigRegistrant(obj) == registrantChart { - if !waitChartCSIDriverConfigGone(ctx, gvr, csiDriverConfigHandoverTimeout) { - note("CSIDriverConfig %s is the chart's and stayed through %s although the chart renders none: the lab applies none", workspacesCSIDriver, csiDriverConfigHandoverTimeout) + b := workspacesBackendFor(cfg) + if obj, err := getObject(ctx, gvr, "", b.driver); err == nil && csiDriverConfigRegistrant(obj) == registrantChart { + if !waitChartCSIDriverConfigGone(ctx, gvr, b.driver, csiDriverConfigHandoverTimeout) { + note("CSIDriverConfig %s is the chart's and stayed through %s although the chart renders none: the lab applies none", b.driver, csiDriverConfigHandoverTimeout) return nil } - note("the earlier chart's CSIDriverConfig %s is gone with its upgrade", workspacesCSIDriver) + note("the earlier chart's CSIDriverConfig %s is gone with its upgrade", b.driver) } k, err := labKube() if err != nil { return err } - res, err := k.apply(ctx, workspacesCSIDriverConfig()) + res, err := k.apply(ctx, workspacesCSIDriverConfig(b)) if err != nil { - return fmt.Errorf("the CSIDriverConfig %s: %w", workspacesCSIDriver, err) + return fmt.Errorf("the CSIDriverConfig %s: %w", b.driver, err) } note("%s (the chart renders no CSIDriverConfig for %s, so the lab registers the driver with Substrate itself: controllerEndpoint %s, tls.serverName %s)", - res, workspacesCSIDriver, workspacesValuesFor(config.Default()).ControllerEndpoint, workspacesControllerServerName()) + res, b.driver, b.controllerEndpoint(), b.serverName()) return nil } @@ -689,9 +820,9 @@ var csiDriverConfigHandoverTimeout = fluxActionTimeout // waitChartCSIDriverConfigGone waits until no Helm-managed CSIDriverConfig // of the driver is live; false when one still is after timeout. -func waitChartCSIDriverConfigGone(ctx context.Context, gvr schema.GroupVersionResource, timeout time.Duration) bool { +func waitChartCSIDriverConfigGone(ctx context.Context, gvr schema.GroupVersionResource, driver string, timeout time.Duration) bool { err := wait.PollUntilContextTimeout(ctx, 2*time.Second, timeout, true, func(ctx context.Context) (bool, error) { - obj, err := getObject(ctx, gvr, "", workspacesCSIDriver) + obj, err := getObject(ctx, gvr, "", driver) if apierrors.IsNotFound(err) { return true, nil } @@ -769,30 +900,46 @@ func removeLabWorkspacesCSIDriverConfig(ctx context.Context) error { if err != nil { return nil } - obj, err := getObject(ctx, gvr, "", workspacesCSIDriver) - if err != nil { + for _, driver := range []string{workspacesCSIDriver, hostPathCSIDriver} { + obj, err := getObject(ctx, gvr, "", driver) if apierrors.IsNotFound(err) { - return nil + continue + } + if err != nil { + return err + } + if obj.GetLabels()[managedByLabel] != managedByAgentlabValue { + continue + } + note("removing the CSIDriverConfig %s the lab applied", driver) + if err := deleteObject(ctx, gvr, "", driver, fixtureDeleteWait); err != nil { + return err } - return err - } - if obj.GetLabels()[managedByLabel] != managedByAgentlabValue { - return nil } - note("removing the CSIDriverConfig %s the lab applied", workspacesCSIDriver) - return deleteObject(ctx, gvr, "", workspacesCSIDriver, fixtureDeleteWait) + return nil } +// The resources of the drivers' cluster roles. +const ( + clusterRolesResource = "clusterroles.rbac.authorization.k8s.io" + clusterRoleBindingsResource = "clusterrolebindings.rbac.authorization.k8s.io" +) + // The cluster-scoped objects of the workspace storage, as kubectl resource // arguments and names: what workspacesDown removes after the namespace. var workspacesClusterObjects = []struct{ resource, name string }{ {"storageclasses.storage.k8s.io", workspacesStorageClass}, {csiDriversResource, workspacesCSIDriver}, {csiDriversResource, companionCSIDriver}, - {"clusterrolebindings.rbac.authorization.k8s.io", "agentlab-workspaces-csi-nfs-provisioner"}, - {"clusterroles.rbac.authorization.k8s.io", "agentlab-workspaces-csi-nfs-provisioner"}, - {"clusterrolebindings.rbac.authorization.k8s.io", "agentlab-workspaces-csi-nfs-resizer"}, - {"clusterroles.rbac.authorization.k8s.io", "agentlab-workspaces-csi-nfs-resizer"}, + {csiDriversResource, hostPathCSIDriver}, + {clusterRoleBindingsResource, "agentlab-workspaces-csi-nfs-provisioner"}, + {clusterRolesResource, "agentlab-workspaces-csi-nfs-provisioner"}, + {clusterRoleBindingsResource, "agentlab-workspaces-csi-nfs-resizer"}, + {clusterRolesResource, "agentlab-workspaces-csi-nfs-resizer"}, + {clusterRoleBindingsResource, "agentlab-workspaces-csi-hostpath-provisioner"}, + {clusterRolesResource, "agentlab-workspaces-csi-hostpath-provisioner"}, + {clusterRoleBindingsResource, "agentlab-workspaces-csi-hostpath-resizer"}, + {clusterRolesResource, "agentlab-workspaces-csi-hostpath-resizer"}, } // workspacesDown removes the workspace storage from a lab that has it: the @@ -914,11 +1061,19 @@ func readWorkspacesStatus(ctx context.Context) (*WorkspacesStatus, error) { if err != nil { return nil, err } - s := &WorkspacesStatus{} - for _, w := range []struct { + s := &WorkspacesStatus{Driver: liveWorkspacesCSIDriver(ctx)} + b := workspacesBackendOfDriver(s.Driver) + workloads := []struct { name string into *string - }{{workspacesNFSServer, &s.NFSServer}, {workspacesController, &s.Controller}} { + }{{b.controller, &s.Controller}} + if b.nfsServer { + workloads = append(workloads, struct { + name string + into *string + }{workspacesNFSServer, &s.NFSServer}) + } + for _, w := range workloads { d, err := k.clientset.AppsV1().Deployments(workspacesNamespace).Get(ctx, w.name, metav1.GetOptions{}) switch { case err != nil: @@ -930,12 +1085,12 @@ func readWorkspacesStatus(ctx context.Context) (*WorkspacesStatus, error) { *w.into = fmt.Sprintf("%d of %d replicas available", d.Status.AvailableReplicas, d.Status.Replicas) } s.Node = d.Spec.Template.Spec.NodeSelector["kubernetes.io/hostname"] - if w.name == workspacesController { + if w.name == b.controller { s.Proxy = slices.ContainsFunc(d.Spec.Template.Spec.Containers, func(c corev1.Container) bool { return c.Name == workspacesProxyContainer }) } } } - ds, err := k.clientset.AppsV1().DaemonSets(workspacesNamespace).Get(ctx, workspacesNodePlugin, metav1.GetOptions{}) + ds, err := k.clientset.AppsV1().DaemonSets(workspacesNamespace).Get(ctx, b.nodePlugin, metav1.GetOptions{}) switch { case err != nil: s.NodePlugin = readState(err) @@ -944,7 +1099,6 @@ func readWorkspacesStatus(ctx context.Context) (*WorkspacesStatus, error) { default: s.NodePlugin = daemonSetStatus(ds) } - s.Driver = liveWorkspacesCSIDriver(ctx) s.StorageClass = s.Driver != "" s.Companion = readCompanionState(ctx, k) s.CSIDriverConfig = readWorkspacesRegistrant(ctx, cmp.Or(s.Driver, workspacesCSIDriver)) @@ -1013,6 +1167,10 @@ func (s *WorkspacesStatus) String() string { } line := fmt.Sprintf("NFS server %s on %s; CSI controller %s %s %s; node plugin %s; %s; %s", s.NFSServer, orNone(s.Node), driver, s.Controller, proxy, s.NodePlugin, class, registration) + if driver == hostPathCSIDriver { + line = fmt.Sprintf("hostpath CSI driver %s on %s, its volumes under %s; controller proxy %s %s; %s; %s", + s.NodePlugin, orNone(s.Node), workspacesExportDir, s.Controller, proxy, class, registration) + } if s.Companion != "" { line += fmt.Sprintf("; the chart's CSI companion (%s preset, the lab's NFS plugin) %s", companionPreset, s.Companion) } @@ -1030,7 +1188,11 @@ func workspacesHint(cfg *config.Config, chartCarries bool, registrant string) st if !cfg.WorkspacesEnabled() { return " Workspace storage is off (platform.workspaces in agentlab.yaml; `agentlab configure --workspaces` turns it on)." } - hint := fmt.Sprintf(" Workspace storage: an NFS server on %s and the NFS CSI driver behind its mTLS proxy, %s;\n"+ + storage := "an NFS server on %s and the NFS CSI driver behind its mTLS proxy" + if workspacesStorageFor(cfg) == config.WorkspaceStorageHostPath { + storage = "the hostpath CSI driver on %s (" + workspacesExportDir + " of the node per volume) behind its mTLS proxy" + } + hint := fmt.Sprintf(" Workspace storage: "+storage+", %s;\n"+ " the read-write-many StorageClass %s. Proof: `agentlab workspaces-test --storage-only`.", WorkspacesNode(cfg), workspacesRegistration(workspacesCSIDriverFor(cfg), registrant), workspacesStorageClass) if cfg.WorkspacesCSICompanion() { hint += fmt.Sprintf("\n Substrate's driver is the chart's CSI companion (%s preset, the lab's NFS plugin serving %s). Proof: `agentlab workspaces-test --storage-only`, its actors mounting through the companion.", companionPreset, companionCSIDriver) diff --git a/internal/lab/workspaceshostpath_test.go b/internal/lab/workspaceshostpath_test.go new file mode 100644 index 00000000..a73ab064 --- /dev/null +++ b/internal/lab/workspaceshostpath_test.go @@ -0,0 +1,262 @@ +package lab + +import ( + "bytes" + "slices" + "testing" + + "gopkg.in/yaml.v3" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + "github.com/giantswarm/agentlab/internal/config" +) + +// withRootless fixes rootlessPodmanOnLinux for one test. +func withRootless(t *testing.T, rootless bool) { + t.Helper() + prev := rootlessPodmanOnLinux + rootlessPodmanOnLinux = func() bool { return rootless } + t.Cleanup(func() { rootlessPodmanOnLinux = prev }) +} + +// An empty platform.workspaces.storage is hostpath under rootless Podman for +// the single-node lab without the companion, nfs otherwise; a set one wins. +func TestWorkspacesStorageFor(t *testing.T) { + for _, tc := range []struct { + name string + rootless bool + storage string + substrateNodes int + companion bool + want string + }{ + {"rootful engine", false, "", 0, false, config.WorkspaceStorageNFS}, + {"rootless Podman", true, "", 0, false, config.WorkspaceStorageHostPath}, + {"rootless Podman with workers", true, "", 1, false, config.WorkspaceStorageNFS}, + {"rootless Podman with the companion", true, "", 0, true, config.WorkspaceStorageNFS}, + {"nfs set under rootless Podman", true, config.WorkspaceStorageNFS, 0, false, config.WorkspaceStorageNFS}, + {"hostpath set on a rootful engine", false, config.WorkspaceStorageHostPath, 0, false, config.WorkspaceStorageHostPath}, + } { + t.Run(tc.name, func(t *testing.T) { + withRootless(t, tc.rootless) + cfg := config.Default() + cfg.Platform.Workspaces.Enabled = true + cfg.Platform.Workspaces.Storage = tc.storage + cfg.Platform.Workspaces.CSICompanion = tc.companion + cfg.SubstrateNodes = tc.substrateNodes + if got := workspacesStorageFor(cfg); got != tc.want { + t.Errorf("workspacesStorageFor = %q, want %q", got, tc.want) + } + }) + } +} + +// The hostpath manifest: no NFS server, the hostpath driver's DaemonSet on +// the control plane with its volumes under the export directory and +// Substrate's directory mounted Bidirectional, the controller Deployment +// the mTLS proxy alone on the driver's socket, the roles bound to the +// DaemonSet's account, and a class without NFS parameters. +func TestWorkspacesManifestHostPath(t *testing.T) { + cfg := config.Default() + cfg.Platform.Workspaces.Enabled = true + cfg.Platform.Workspaces.Storage = config.WorkspaceStorageHostPath + out, err := renderTemplate(cfg, workspacesTemplate, nil) + if err != nil { + t.Fatal(err) + } + var rendered []map[string]any + dec := yaml.NewDecoder(bytes.NewReader(out)) + for { + var doc map[string]any + if err := dec.Decode(&doc); err != nil { + break + } + if doc != nil { + rendered = append(rendered, doc) + } + } + find := func(kind, name string) map[string]any { + for _, d := range rendered { + meta, _ := d["metadata"].(map[string]any) + if d["kind"] == kind && meta["name"] == name { + return d + } + } + return nil + } + named := func(kind, name string) map[string]any { + t.Helper() + d := find(kind, name) + if d == nil { + t.Fatalf("no %s %s in the manifest", kind, name) + } + return d + } + podSpec := func(doc map[string]any) map[string]any { + spec, _ := doc["spec"].(map[string]any) + template, _ := spec["template"].(map[string]any) + pod, _ := template["spec"].(map[string]any) + return pod + } + containers := func(doc map[string]any) map[string]map[string]any { + out := map[string]map[string]any{} + list, _ := podSpec(doc)["containers"].([]any) + for _, c := range list { + container, _ := c.(map[string]any) + name, _ := container["name"].(string) + out[name] = container + } + return out + } + hostPaths := func(doc map[string]any) map[string]string { + out := map[string]string{} + list, _ := podSpec(doc)["volumes"].([]any) + for _, v := range list { + volume, _ := v.(map[string]any) + name, _ := volume["name"].(string) + if hp, ok := volume["hostPath"].(map[string]any); ok { + out[name], _ = hp["path"].(string) + } + } + return out + } + + for _, name := range []string{workspacesNFSServer, workspacesController} { + if find("Deployment", name) != nil { + t.Errorf("Deployment %s rendered with the hostpath storage", name) + } + } + if find("DaemonSet", workspacesNodePlugin) != nil { + t.Errorf("DaemonSet %s rendered with the hostpath storage", workspacesNodePlugin) + } + + driver := named("CSIDriver", hostPathCSIDriver) + if spec, _ := driver["spec"].(map[string]any); spec["attachRequired"] != false { + t.Errorf("CSIDriver attachRequired = %v, want false (the driver runs without its attach step)", spec["attachRequired"]) + } + class := named("StorageClass", workspacesStorageClass) + if class["provisioner"] != hostPathCSIDriver { + t.Errorf("StorageClass provisioner = %v, want %s", class["provisioner"], hostPathCSIDriver) + } + for _, key := range []string{"parameters", "mountOptions"} { + if v, ok := class[key]; ok { + t.Errorf("StorageClass %s = %v, want none (a bind mount takes no NFS options)", key, v) + } + } + + node := named("DaemonSet", hostPathNodePlugin) + if selector, _ := podSpec(node)["nodeSelector"].(map[string]any); selector["kubernetes.io/hostname"] != cfg.ControlPlaneNode() { + t.Errorf("DaemonSet nodeSelector = %v, want the control plane %s", selector, cfg.ControlPlaneNode()) + } + if account := podSpec(node)["serviceAccountName"]; account != hostPathNodePlugin { + t.Errorf("DaemonSet serviceAccountName = %v, want %s", account, hostPathNodePlugin) + } + nodeContainers := containers(node) + for _, name := range []string{"hostpath", "csi-provisioner", "csi-resizer", "node-driver-registrar", "liveness-probe"} { + if nodeContainers[name] == nil { + t.Errorf("DaemonSet has no container %s", name) + } + } + plugin := nodeContainers["hostpath"] + if args, _ := plugin["args"].([]any); !slices.Contains(args, any("--drivername="+hostPathCSIDriver)) { + t.Errorf("hostpath args = %v, want --drivername=%s", args, hostPathCSIDriver) + } + mounts, _ := plugin["volumeMounts"].([]any) + var substrate bool + for _, m := range mounts { + mount, _ := m.(map[string]any) + if mount["mountPath"] == substrateVolumesDir && mount["mountPropagation"] == "Bidirectional" { + substrate = true + } + } + if !substrate { + t.Errorf("hostpath mounts %v, want %s Bidirectional", mounts, substrateVolumesDir) + } + nodeVolumes := hostPaths(node) + if nodeVolumes["data-dir"] != workspacesExportDir || nodeVolumes["socket-dir"] != hostPathNodeSocketDir { + t.Errorf("DaemonSet hostPaths = %v, want data-dir %s and socket-dir %s", nodeVolumes, workspacesExportDir, hostPathNodeSocketDir) + } + + controller := named("Deployment", hostPathController) + if c := containers(controller); len(c) != 1 || c[workspacesProxyContainer] == nil { + t.Errorf("controller containers = %v, want the proxy %s alone", c, workspacesProxyContainer) + } + if hostPaths(controller)["socket-dir"] != hostPathNodeSocketDir { + t.Errorf("controller socket-dir = %v, want the driver's %s", hostPaths(controller)["socket-dir"], hostPathNodeSocketDir) + } + service := named("Service", hostPathControllerService) + if spec, _ := service["spec"].(map[string]any); spec != nil { + if selector, _ := spec["selector"].(map[string]any); selector["app.kubernetes.io/name"] != hostPathController { + t.Errorf("Service selector = %v, want the controller %s", selector, hostPathController) + } + } + named("ConfigMap", "csi-hostpath-proxy") + + for _, role := range []string{"provisioner", "resizer"} { + named("ClusterRole", "agentlab-workspaces-csi-hostpath-"+role) + binding := named("ClusterRoleBinding", "agentlab-workspaces-csi-hostpath-"+role) + subjects, _ := binding["subjects"].([]any) + subject, _ := subjects[0].(map[string]any) + if subject["name"] != hostPathNodePlugin { + t.Errorf("ClusterRoleBinding %s subject = %v, want %s", role, subject["name"], hostPathNodePlugin) + } + } + named("ServiceAccount", hostPathNodePlugin) +} + +// The hostpath registration with Substrate: the driver, the proxy's Service, +// the driver's socket and the Service's service-DNS name, in the values and +// in the CSIDriverConfig the lab applies. +func TestWorkspacesHostPathRegistration(t *testing.T) { + cfg := config.Default() + cfg.Platform.Workspaces.Enabled = true + cfg.Platform.Workspaces.Storage = config.WorkspaceStorageHostPath + v := workspacesValuesFor(cfg) + want := map[string]string{ + "Driver": hostPathCSIDriver, + "ControllerEndpoint": "tcp://csi-hostpath-controller.agentlab-workspaces.svc.cluster.local:50051", + "NodeSocket": "unix://" + hostPathNodeSocketDir + "/csi.sock", + "ServerName": "csi-hostpath-controller.agentlab-workspaces.svc", + } + got := map[string]string{"Driver": v.Driver, "ControllerEndpoint": v.ControllerEndpoint, "NodeSocket": v.NodeSocket, "ServerName": v.ServerName} + for key, value := range want { + if got[key] != value { + t.Errorf("%s = %q, want %q", key, got[key], value) + } + } + if !v.HostPath { + t.Error("HostPath = false, want true") + } + + u := workspacesCSIDriverConfig(workspacesBackendFor(cfg)) + spec, _ := u.Object[crSpec].(map[string]any) + if u.GetName() != hostPathCSIDriver || spec["driverName"] != hostPathCSIDriver || spec["controllerEndpoint"] != want["ControllerEndpoint"] || spec["nodeSocketOverride"] != want["NodeSocket"] { + t.Errorf("CSIDriverConfig %s spec = %v, want the hostpath driver's endpoint and socket", u.GetName(), spec) + } + if tls, _ := spec["tls"].(map[string]any); tls["serverName"] != want["ServerName"] { + t.Errorf("CSIDriverConfig tls = %v, want serverName %s", tls, want["ServerName"]) + } +} + +// The hostpath driver names a volume's directory after its volume handle: a +// directory is a live volume's by the handle as by the PersistentVolume's name. +func TestSplitWorkspacesExportDirsVolumeHandle(t *testing.T) { + pvs := []corev1.PersistentVolume{{ + ObjectMeta: metav1.ObjectMeta{Name: "pvc-ws"}, + Spec: corev1.PersistentVolumeSpec{ + StorageClassName: workspacesStorageClass, + ClaimRef: &corev1.ObjectReference{Namespace: "org-a", Name: "ws"}, + PersistentVolumeSource: corev1.PersistentVolumeSource{ + CSI: &corev1.CSIPersistentVolumeSource{Driver: hostPathCSIDriver, VolumeHandle: "6c1f0e9a-live"}, + }, + }, + }} + live, left := splitWorkspacesExportDirs([]string{"6c1f0e9a-live", "0b7d2c4e-gone"}, pvs) + if len(live) != 1 || live[0] != "6c1f0e9a-live (org-a/ws)" { + t.Errorf("live = %v, want the handle's directory with its claim", live) + } + if len(left) != 1 || left[0] != "0b7d2c4e-gone" { + t.Errorf("left = %v, want the directory no volume backs", left) + } +} diff --git a/internal/lab/workspacestest.go b/internal/lab/workspacestest.go index ab88caef..41850919 100644 --- a/internal/lab/workspacestest.go +++ b/internal/lab/workspacestest.go @@ -183,10 +183,15 @@ func WorkspacesTest(cfg *config.Config, opts WorkspacesTestOptions) error { driver := workspacesCSIDriverFor(cfg) endpoint := substrateDriverEndpointFor(cfg) step("The workspace storage in place (namespace %s)", workspacesNamespace) - if err := proveWorkspacesInstalled(ctx, driver, timeout); err != nil { + backend := workspacesBackendFor(cfg) + if err := proveWorkspacesInstalled(ctx, backend, timeout); err != nil { return err } - note("NFS server, controller with its mTLS proxy and node plugin Ready; StorageClass %s (%s)", workspacesStorageClass, driver) + if backend.nfsServer { + note("NFS server, controller with its mTLS proxy and node plugin Ready; StorageClass %s (%s)", workspacesStorageClass, driver) + } else { + note("hostpath driver and the controller's mTLS proxy Ready; StorageClass %s (%s)", workspacesStorageClass, driver) + } if cfg.WorkspacesCSICompanion() { if err := waitWorkspacesCompanion(ctx); err != nil { return err @@ -214,7 +219,11 @@ func WorkspacesTest(cfg *config.Config, opts WorkspacesTestOptions) error { if err := waitPVCBound(ctx, workspacesTestClaim, timeout); err != nil { return err } - note("Bound (a directory of the export on %s, mounted NFSv4.1)", node) + if backend.nfsServer { + note("Bound (a directory of the export on %s, mounted NFSv4.1)", node) + } else { + note("Bound (a directory under %s on %s, bind-mounted)", workspacesExportDir, node) + } step("A bare mirror seeded on the volume, from a repository with an executable and a symbolic link") out, err := runPod(ctx, workspacesTestNamespace, workspacesTestPod("seed", workspacesSeedScript, []corev1.VolumeMount{{Name: workspacesTestVolume, MountPath: workspacesTestMountPath}}, false), timeout) @@ -264,7 +273,7 @@ func WorkspacesTest(cfg *config.Config, opts WorkspacesTestOptions) error { step("Removing the proof's namespace: the claim's volume goes with it") cleanupWorkspacesTest(ctx, node) - if err := proveWorkspacesVolumeGone(ctx, node, timeout); err != nil { + if err := proveWorkspacesVolumeGone(ctx, node, backend, timeout); err != nil { return err } note("no PersistentVolume of the proof left; nothing of it under %s on %s", workspacesExportDir, node) @@ -281,23 +290,27 @@ func WorkspacesTest(cfg *config.Config, opts WorkspacesTestOptions) error { } // proveWorkspacesInstalled is the precondition: the class, the CSIDriver, -// the NFS server, the controller and the node plugin, each waited for -// briefly. -func proveWorkspacesInstalled(ctx context.Context, driver string, timeout time.Duration) error { +// the NFS server where the storage has one, the controller and the node +// plugin, each waited for briefly. +func proveWorkspacesInstalled(ctx context.Context, backend workspacesBackend, timeout time.Duration) error { for _, o := range []struct{ resource, name, what string }{ {"storageclasses.storage.k8s.io", workspacesStorageClass, "StorageClass"}, - {csiDriversResource, driver, "CSIDriver"}, + {csiDriversResource, backend.driver, "CSIDriver"}, } { if !clusterObjectExists(ctx, o.resource, o.name) { return fmt.Errorf("no %s %s: `agentlab platform` installs the workspace storage", o.what, o.name) } } - for _, name := range []string{workspacesNFSServer, workspacesController} { + deployments := []string{backend.controller} + if backend.nfsServer { + deployments = append([]string{workspacesNFSServer}, deployments...) + } + for _, name := range deployments { if err := waitDeploymentRolledOut(ctx, workspacesNamespace, name, timeout); err != nil { return fmt.Errorf("%s: %w", name, err) } } - return waitDaemonSetReady(ctx, workspacesNamespace, workspacesNodePlugin, timeout) + return waitDaemonSetReady(ctx, workspacesNamespace, backend.nodePlugin, timeout) } // cleanupWorkspacesTest removes the proof's namespace, with its pods and @@ -452,12 +465,18 @@ func waitWorkspacesVolumesGone(ctx context.Context, timeout time.Duration) error // proveWorkspacesVolumeGone is the end: no PersistentVolume of the proof, // and no directory in the export on the node that no PersistentVolume backs. // A directory still backing a live volume (another Workspace's) is left in -// place and named. -func proveWorkspacesVolumeGone(ctx context.Context, node string, timeout time.Duration) error { +// place and named. csi-driver-nfs names a volume's directory after its +// PersistentVolume (pvc-…), the hostpath driver after its volume handle, +// beside its own state.json. +func proveWorkspacesVolumeGone(ctx context.Context, node string, backend workspacesBackend, timeout time.Duration) error { if err := waitWorkspacesVolumesGone(ctx, timeout); err != nil { return fmt.Errorf("a PersistentVolume of the proof is still there %s after its claim went: `kubectl get pv`, the provisioner's log in %s", timeout, workspacesNamespace) } - out, err := outputQuiet(dockerBin, "exec", node, "sh", "-c", "ls -A "+workspacesExportDir+" 2>/dev/null | grep '^pvc-' || true") + list := "ls -A " + workspacesExportDir + " 2>/dev/null | grep '^pvc-' || true" + if !backend.nfsServer { + list = "ls -A " + workspacesExportDir + " 2>/dev/null | grep -v '^state.json$' || true" + } + out, err := outputQuiet(dockerBin, "exec", node, "sh", "-c", list) if err != nil { return fmt.Errorf("reading the export on %s: %w", node, err) } @@ -481,7 +500,8 @@ func proveWorkspacesVolumeGone(ctx context.Context, node string, timeout time.Du // splitWorkspacesExportDirs splits the export's volume directories into // those a PersistentVolume of the class still backs, named with its claim, -// and those left behind. +// and those left behind. A directory is a volume's by the PersistentVolume's +// name or its CSI volume handle. func splitWorkspacesExportDirs(dirs []string, pvs []corev1.PersistentVolume) (live, left []string) { claims := map[string]string{} for _, pv := range pvs { @@ -493,6 +513,9 @@ func splitWorkspacesExportDirs(dirs []string, pvs []corev1.PersistentVolume) (li claim = ref.Namespace + "/" + ref.Name } claims[pv.Name] = claim + if pv.Spec.CSI != nil && pv.Spec.CSI.VolumeHandle != "" { + claims[pv.Spec.CSI.VolumeHandle] = claim + } } for _, d := range dirs { if claim, ok := claims[d]; ok { diff --git a/main.go b/main.go index 8b57a2bc..3e9cad66 100644 --- a/main.go +++ b/main.go @@ -1018,6 +1018,7 @@ func configureCmd() *cobra.Command { var devctlVersion string var serving, workspaces, workspacesCSICompanion, coding, tenants, github, githubSignIn, chatgptSubscription bool var chatgptSubscriptionModel string + var workspacesStorage string var githubSignInClientID string var codingHarnesses []string var codingImages map[string]string @@ -1239,6 +1240,9 @@ func configureCmd() *cobra.Command { if cmd.Flags().Changed("workspaces-csi-companion") { cfg.Platform.Workspaces.CSICompanion = workspacesCSICompanion } + if cmd.Flags().Changed("workspaces-storage") { + cfg.Platform.Workspaces.Storage = workspacesStorage + } if cmd.Flags().Changed("workspaces-github-app-id") { cfg.Platform.Workspaces.GitHub.AppID = workspacesGitHubAppID } @@ -1352,6 +1356,7 @@ func configureCmd() *cobra.Command { cmd.Flags().StringVar(&workspacesProvider, "workspaces-provider", "", "the workspace-manager's provider instances: github (a GitHub App of the lab's own, the default, docs/workspaces.md), gitlab (a GitLab's OAuth application, docs/workspaces.md) or both, github,gitlab") cmd.Flags().StringVar(&workspacesGrantProvider, "workspaces-grant-provider", "", "the provider instance whose App muster pins the workspace-manager to, the one App a person grants (platform.workspaces.grantProvider): one of --workspaces-provider, such as gitlab for 'workspaces-test --part gitlab', or "+config.WorkspaceGrantProviderApps+" to pin none: muster keeps one provider App per instance side by side, each granted once (core_auth_login app=; two or more instances); \"\" returns to the first instance") cmd.Flags().BoolVar(&workspacesCSICompanion, "workspaces-csi-companion", false, "with workspaces on, the connectivity chart's CSI companion is Substrate's workspace driver: its efs preset running the lab's NFS plugin, the lab's NFS driver serving the volumes under the preset's driver name — the lab path for proving a companion chart change (docs/workspaces.md); --workspaces-csi-companion=false returns to the lab's own registration") + cmd.Flags().StringVar(&workspacesStorage, "workspaces-storage", "", "what backs the workspaces' volumes: nfs (an NFS server in the cluster and csi-driver-nfs; needs a rootful engine on Linux), hostpath (csi-driver-host-path: a directory of the single node per volume, bind-mounted; runs under rootless Podman), or \"\" for hostpath under rootless Podman on Linux and nfs elsewhere (docs/workspaces.md)") cmd.Flags().StringVar(&workspacesCleanupAfter, "workspaces-cleanup-after", "", "how long a Session's directory on a workspace volume outlives its last turn, a Go duration (the workspace-manager's sessions.cleanupAfter); \"\" returns to the chart's 30 days") cmd.Flags().StringVar(&workspacesRefreshAfter, "workspaces-refresh-after", "", "the refresh threshold, a Go duration (the workspace-manager's sync.refreshAfter): a Session started on a workspace whose last sync is older requests a sync; \"\" returns to the manager's 1h") cmd.Flags().StringVar(&workspacesGitHubAppID, "workspaces-github-app-id", "", "the App id of the workspaces GitHub App (public; its private key and client secret go into the Secret agent-platform/"+config.WorkspaceGitHubSecretName+")")