feat(workspaces): a hostpath storage the lab runs under rootless Podman

docs/cli.md

@@ -236,9 +236,10 @@ The flags pin a value regardless of the discovery, with or without
 | `--coding-harnesses <runtimes>` | The coding Harnesses `--coding` renders, by runtime: `claude`, `codex`, `omp` or several (`claude,codex,omp`), each named after its runtime; `""` returns to `claude` alone. `codex` needs a kagent line whose chart takes `runtime: codex`; `omp` (oh-my-pi, a BYO Harness on the kagent line's `omp-harness` adapter) one whose chart takes `runtime: byo` and an image from `--coding-images`. A dev build of a coding Harness's image swaps in through `platform.devImages` `claude-harness` / `codex-harness` / `omp-harness`; `claude-harness` and `codex-harness` need `kagent` from the same build, whose controller pins the CLI version ([Coding Harness targets](platform.md#coding-harness-targets)). |
 | `--coding-images <runtime>=<image>` | The image of a coding Harness the chart has none of its own for, pinned by digest (`platform.coding.images`): `omp=gsoci.azurecr.io/giantswarm/kagent/omp-harness@sha256:…`, the `runtimeImages.ompHarness.digest` the release's kagent chart stamps. A dev image (`platform.devImages` `omp-harness`) wins over it. |
 | `--serving[=false]` | Serve models on llm-d in the lab, or turn it off: the KServe llmisvc controller and its CRDs, the well-known runtime configs, the connectivity chart's serving slice with the models Gateway, model-manager's `kserve` backend and one CPU preset of the lab's (needs agents; installs cert-manager; agent-platform 4.44.0 or newer). See [Model serving on llm-d](models.md#model-serving-on-llm-d-platformserving). |
-| `--workspaces[=false]` | Install the workspace storage, or turn it off: an in-cluster NFS server, the NFS CSI driver `nfs.csi.k8s.io` with its controller behind an Envoy mTLS proxy only Agent Substrate's API server may reach (its pod-identity certificate), and the read-write-many StorageClass `agentlab-workspaces`; needs the host kernel's `nfsd` and `nfs` modules (available, not loaded by hand: the kernel loads them). Needs the agents (`--agents`): the workspaces live in kagent's namespace, and the lab refuses them without it. See [Workspaces](workspaces.md). |
+| `--workspaces[=false]` | Install the workspace storage, or turn it off: an in-cluster NFS server, the NFS CSI driver `nfs.csi.k8s.io` with its controller behind an Envoy mTLS proxy only Agent Substrate's API server may reach (its pod-identity certificate), and the read-write-many StorageClass `agentlab-workspaces`; needs the host kernel's `nfsd` and `nfs` modules (available, not loaded by hand: the kernel loads them); under rootless Podman on Linux the hostpath CSI driver backs the volumes instead (`--workspaces-storage`). Needs the agents (`--agents`): the workspaces live in kagent's namespace, and the lab refuses them without it. See [Workspaces](workspaces.md). |
 | `--workspaces-provider <github\|gitlab\|github,gitlab>` | The workspace-manager's provider instances: `github` (the default, `""` too), a GitHub App of the lab's own on github.com, the instance once the App's ids are set and the Secret `agent-platform/workspace-github` carries `private-key` and `client-secret`; `gitlab`, a GitLab's OAuth application; or both. With workspaces on, the base URL `https://workspace-manager.<domain>:<gatewayPort>` and its route are in place regardless; each instance's App registers muster's `/oauth/proxy/callback`. See [Workspaces](workspaces.md#the-provider-instances). |
 | `--workspaces-csi-companion[=false]` | With workspaces on, the connectivity chart's CSI companion is Substrate's workspace driver (`platform.workspaces.csiCompanion`): its `efs` preset running the lab's NFS plugin, the lab's NFS driver serving the volumes as `efs.csi.aws.com` — the lab path for proving a companion chart change; needs agent-platform 4.128.0 or newer. Switching takes the workspace storage apart first, its volumes with it. See [The CSI companion](workspaces.md#the-csi-companion). |
+| `--workspaces-storage <nfs\|hostpath>` | What backs the workspaces' volumes (`platform.workspaces.storage`): `nfs`, an NFS server in the cluster and csi-driver-nfs, which needs a rootful engine on Linux; `hostpath`, csi-driver-host-path, a directory of the single node per volume, bind-mounted, which runs under rootless Podman; `""` (the default) picks `hostpath` under rootless Podman on Linux for the single-node lab without the companion and `nfs` elsewhere. `hostpath` refuses `substrateNodes` above 0 and `--workspaces-csi-companion`. See [Rootless Podman: the hostpath storage](workspaces.md#rootless-podman-the-hostpath-storage). |
 | `--workspaces-cleanup-after <duration>` | How long a Session's directory on a workspace volume outlives its last turn (`platform.workspaces.sessionsCleanupAfter`, the workspace-manager's `sessions.cleanupAfter`): a Go duration; `""` returns to the chart's 30 days. |
 | `--workspaces-refresh-after <duration>` | The refresh threshold (`platform.workspaces.syncRefreshAfter`, the workspace-manager's `sync.refreshAfter`): a Session started on a workspace whose last sync is older requests a sync; a Go duration; `""` returns to the manager's 1h. |
 | `--workspaces-github-app-id`, `--workspaces-github-client-id` | The public ids of the `github` instance's App (`platform.workspaces.github`); its private key and client secret never pass through agentlab. |

docs/troubleshooting.md

@@ -172,6 +172,30 @@ plumbing for model servers under
   A new node leaves them alone; if `up` still reports a change, the warning
   names the key and the command that restores it.
 
+## Rootless Podman's kernel limits
+
+Under rootless Podman on Linux every container of the lab, the kind node's
+own included, runs as your user, so two per-user kernel defaults bound the
+whole lab. Neither can be raised from inside the node.
+
+- **A pod fails to start with `unable to create session key: disk quota
+  exceeded`.** Every container gets a session keyring, counted against your
+  user's key quota (`kernel.keys.maxkeys` 200 and `maxbytes` 20000 by
+  default; `/proc/key-users` shows the use). A full lab comes close to it on
+  its own, and leftover containers of other tools (testcontainers whose
+  reaper never ran) count too. Raise it and keep it:
+  `printf 'kernel.keys.maxkeys = 20000\nkernel.keys.maxbytes = 400000\n' | sudo tee /etc/sysctl.d/90-rootless-keys.conf && sudo sysctl --system`.
+- **Services stop reaching their pods while the pods answer on their own
+  addresses.** kube-proxy logs `Failed to execute iptables-restore` with
+  `sendmsg() failed: Message too long` and keeps the rules it last wrote, so
+  every Service whose pods changed since points at addresses that are gone
+  (an edge's connect to the kagent controller times out, muster cannot reach
+  agent-manager). Inside a user namespace iptables cannot force a socket
+  buffer above `net.core.wmem_default` (212992 bytes by default), and the
+  lab's ruleset outgrows it. Raising it on the host takes effect in the
+  running node at kube-proxy's next sync:
+  `printf 'net.core.wmem_default = 4194304\nnet.core.rmem_default = 4194304\n' | sudo tee /etc/sysctl.d/91-rootless-netlink.conf && sudo sysctl --system`.
+
 ## vm-manager
 
 - **`vm-manager-test` fails at `get_vm_attestation` with `golden mismatch:

docs/workspaces.md

@@ -52,6 +52,50 @@ namespace, or with the cluster on `down`) and the export's bytes with it. A
 server that stopped first would leave a node's NFS client waiting on it for
 good, a node that can then be neither stopped nor removed.
 
+### Rootless Podman: the hostpath storage
+
+Under rootless Podman on Linux the nodes run in your user's user namespace,
+where the kernel mounts neither the `nfsd` filesystem nor an NFS export: the
+`nfs-server` pod logs `Unable to access /proc/fs/nfsd` and never listens, and
+every NFS mount is refused. `platform.workspaces.storage` picks what backs the
+volumes:
+
+| Value | What |
+|---|---|
+| `nfs` | The NFS server and csi-driver-nfs above |
+| `hostpath` | kubernetes-csi/csi-driver-host-path v1.17.1, as Substrate's own `ate-setup csi hostpath` installs it: every volume a directory of the node's `/var/lib/agentlab-workspaces` (beside the driver's `state.json`), published with a bind mount, which a user namespace allows |
+| empty (the default) | `hostpath` under rootless Podman on Linux for the single-node lab without the companion, `nfs` everywhere else |
+
+`agentlab configure --workspaces-storage <nfs|hostpath>` sets it; `""`
+returns to the default. The hostpath storage:
+
+| Piece | What |
+|---|---|
+| The `csi-hostpath-node` DaemonSet (namespace `agentlab-workspaces`) | The driver `hostpath.csi.k8s.io` on the control plane, serving the controller and the node service on `/var/lib/kubelet/plugins/csi-hostpath/csi.sock`, with the provisioner, the resizer and the registrar beside it; Substrate's `/var/lib/ate` mounted `Bidirectional` |
+| The controller endpoint | The `csi-hostpath-controller` Deployment: the same Envoy mTLS proxy as the NFS controller's, alone, dialling the driver's socket on the node, behind the Service `csi-hostpath-controller` (port 50051, TLS server name `csi-hostpath-controller.agentlab-workspaces.svc`) |
+| The StorageClass `agentlab-workspaces` | Read-write-many on the one node, `Immediate` binding, delete-reclaiming, expandable; no mount options |
+| The `CSIDriverConfig` `hostpath.csi.k8s.io` | As the NFS driver's, from the same `workspaces:` values |
+
+A volume is a directory of one node, so the hostpath storage needs the
+single-node lab: `substrateNodes` above 0 is refused with it, and so is
+`csiCompanion`, whose preset mounts the lab's NFS volumes. Nothing is
+installed on the host and no kernel module is needed. The plugin image is
+`registry.k8s.io`'s, pinned by digest (gsoci carries no mirror of it).
+
+The hostpath driver keeps one staging path per volume on a node and refuses
+a second (`volume … is already staged at …`). Two orchestrators stage on the
+node, each at its own path: the kubelet for a pod's claim and atelet for an
+actor's existing volume. So a pod and an actor cannot hold one volume at the
+same time on this storage: a workspace sync beside a running Session, or
+`workspaces-test --storage-only`'s report pod reading the volume its actors
+hold. Actors sharing a volume among themselves (a Session's directory
+read-write and the mirrors read-only, two Sessions of one workspace) need a
+Substrate that stages a multi-node volume once per node; earlier releases
+stage it once per target and are refused at the second mount. The claims,
+the sessions' sub-paths and the controller endpoint's refusal pass on any
+release. csi-driver-nfs advertises no staging, which is why the NFS storage
+meets neither limit; the whole workspace flow needs it.
+
 ### Turning it on
 
 ```sh

internal/config/config.go

@@ -702,8 +702,8 @@ type Tenants struct {
 
 // Workspaces configures the workspace storage in the lab.
 type Workspaces struct {
-	// On, `agentlab platform` installs the NFS server, the NFS CSI driver
-	// behind its mTLS proxy and the StorageClass before the chart, and
+	// On, `agentlab platform` installs the storage (Storage), its CSI
+	// driver behind its mTLS proxy and the StorageClass before the chart, and
 	// renders the chart's `workspaces:` block when the chart carries the
 	// key.
 	Enabled bool `yaml:"enabled"`
@@ -747,6 +747,15 @@ type Workspaces struct {
 	// the preset's driver name. The lab path for proving a companion chart
 	// change (docs/workspaces.md); needs a chart that carries the key.
 	CSICompanion bool `yaml:"csiCompanion,omitempty"`
+	// Storage is what backs the workspaces' volumes: WorkspaceStorageNFS, an
+	// NFS server in the cluster and csi-driver-nfs, or
+	// WorkspaceStorageHostPath, csi-driver-host-path, a directory of the one
+	// node per volume, bind-mounted where it is published. Empty picks
+	// hostpath under rootless Podman on Linux, where the kernel refuses
+	// nfsd and NFS mounts to the node's user namespace, and nfs elsewhere.
+	// hostpath needs the single-node lab (substrateNodes 0) and refuses
+	// csiCompanion, whose preset mounts the lab's NFS volumes.
+	Storage string `yaml:"storage,omitempty"`
 }
 
 // WorkspaceGitLab is the GitLab behind the workspace-manager's provider
@@ -893,6 +902,12 @@ const (
 	WorkspaceGitLabPasswordKey   = "person-password"  // #nosec G101 -- Secret KEY name, not a credential
 )
 
+// The workspace storages platform.workspaces.storage names.
+const (
+	WorkspaceStorageNFS      = "nfs"
+	WorkspaceStorageHostPath = "hostpath"
+)
+
 // WorkspaceGrantProviderApps is the grantProvider that pins no instance:
 // one muster provider App per instance, side by side.
 const WorkspaceGrantProviderApps = "apps"
@@ -2392,6 +2407,18 @@ func (c *Config) Validate() error {
 	if c.Platform.Workspaces.Enabled && c.Platform.Enabled && !c.Platform.Agents {
 		return fmt.Errorf("platform.workspaces requires platform.agents (the workspaces live in kagent's namespace, where every Session runs and mounts its workspace) — `agentlab configure --agents`, or `--workspaces=false`")
 	}
+	switch storage := c.Platform.Workspaces.Storage; storage {
+	case "", WorkspaceStorageNFS:
+	case WorkspaceStorageHostPath:
+		if c.SubstrateNodes > 0 {
+			return fmt.Errorf("platform.workspaces.storage %q publishes a volume as a directory of one node, and substrateNodes %d adds workers whose actors could not mount it: `agentlab configure --workspaces-storage %s`, or substrateNodes 0", storage, c.SubstrateNodes, WorkspaceStorageNFS)
+		}
+		if c.Platform.Workspaces.CSICompanion {
+			return fmt.Errorf("platform.workspaces.storage %q and csiCompanion: the companion's preset mounts the lab's NFS volumes: `agentlab configure --workspaces-storage %s`, or `--workspaces-csi-companion=false`", storage, WorkspaceStorageNFS)
+		}
+	default:
+		return fmt.Errorf("platform.workspaces.storage %q: want %q, %q, or empty (hostpath under rootless Podman on Linux, nfs elsewhere)", storage, WorkspaceStorageNFS, WorkspaceStorageHostPath)
+	}
 	if names, err := parseWorkspaceProviders(c.Platform.Workspaces.Provider); err != nil {
 		return err
 	} else if grant := c.Platform.Workspaces.GrantProvider; grant == WorkspaceGrantProviderApps && len(names) < 2 {

internal/config/workspaces_test.go

@@ -233,3 +233,38 @@ func TestValidateProofRepositories(t *testing.T) {
 		}
 	}
 }
+
+// The workspace storage: empty, nfs and hostpath accepted; hostpath refused
+// beside substrate workers and beside the CSI companion; any other value
+// refused.
+func TestWorkspacesStorage(t *testing.T) {
+	for _, tc := range []struct {
+		name           string
+		storage        string
+		substrateNodes int
+		companion      bool
+		wantErr        string
+	}{
+		{"empty", "", 0, false, ""},
+		{"nfs", WorkspaceStorageNFS, 2, true, ""},
+		{"hostpath", WorkspaceStorageHostPath, 0, false, ""},
+		{"hostpath with workers", WorkspaceStorageHostPath, 1, false, "substrateNodes 1"},
+		{"hostpath with the companion", WorkspaceStorageHostPath, 0, true, "csiCompanion"},
+		{"another storage", "ceph", 0, false, `platform.workspaces.storage "ceph"`},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			cfg := Default()
+			cfg.Platform.Workspaces.Enabled = true
+			cfg.Platform.Workspaces.Storage = tc.storage
+			cfg.Platform.Workspaces.CSICompanion = tc.companion
+			cfg.SubstrateNodes = tc.substrateNodes
+			err := cfg.Validate()
+			switch {
+			case tc.wantErr == "" && err != nil:
+				t.Errorf("Validate: %v, want accepted", err)
+			case tc.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tc.wantErr)):
+				t.Errorf("Validate: %v, want an error naming %q", err, tc.wantErr)
+			}
+		})
+	}
+}

internal/lab/main_test.go

@@ -0,0 +1,14 @@
+package lab
+
+import (
+	"os"
+	"testing"
+)
+
+// TestMain fixes the engine to one that is not rootless Podman, so what the
+// lab picks from it (workspacesStorageFor) does not follow the machine the
+// tests run on; a test of the rootless branch sets it itself.
+func TestMain(m *testing.M) {
+	rootlessPodmanOnLinux = func() bool { return false }
+	os.Exit(m.Run())
+}

internal/lab/runtime.go

@@ -2,6 +2,7 @@ package lab
 
 import (
 	"os"
+	"runtime"
 	"strconv"
 	"strings"
 	"sync"
@@ -59,6 +60,14 @@ func MinPublishablePort() int {
 	return unprivilegedPortStart(os.ReadFile)
 }
 
+// rootlessPodmanOnLinux reports whether the lab's nodes are rootless Podman
+// containers on this machine's kernel: they run in the invoking user's user
+// namespace, where the kernel mounts neither the nfsd filesystem nor an NFS
+// export.
+var rootlessPodmanOnLinux = func() bool {
+	return runtime.GOOS == goosLinux && dockerIsPodman() && os.Geteuid() != 0
+}
+
 // unprivilegedPortStart reads net.ipv4.ip_unprivileged_port_start. An
 // unreadable or unparsable sysctl means the kernel default holds.
 func unprivilegedPortStart(readFile func(string) ([]byte, error)) int {