examples: add gpu-operator kind cluster walkthrough

README.md

@@ -322,6 +322,57 @@ GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-4cf8db2d-06c0-7d70-1a51-e59b25b2c16c)
 GPU 1: NVIDIA A100-SXM4-40GB (UUID: GPU-4404041a-04cf-1ccf-9e70-f139a9b1e23c)
 ```
 
+## Install GPU Operator
+
+If you want to install the NVIDIA GPU Operator in a Docker-based `nvkind`
+environment, follow
+[examples/gpu-operator/README.md](examples/gpu-operator/README.md).
+
+Assuming a cluster has been created as described in the [quickstart
+guide](#quickstart) above, the `gpu-operator` can be installed on the cluster
+as appropriate. For the purposes of this example, we will install the
+`gpu-operator` directly.
+
+First, add the `helm` repo for the `gpu-operator`:
+```bash
+helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
+helm repo update
+```
+
+Then pick the cluster you want to install to:
+```bash
+export KIND_CLUSTER_NAME=evenly-distributed-2-by-4
+```
+
+Create the namespace and apply the required pod security label:
+```bash
+kubectl --context=kind-${KIND_CLUSTER_NAME} create namespace gpu-operator --dry-run=client -o yaml | kubectl apply -f -
+kubectl --context=kind-${KIND_CLUSTER_NAME} label namespace gpu-operator pod-security.kubernetes.io/enforce=privileged --overwrite
+```
+
+And install the `gpu-operator` as follows:
+```bash
+helm upgrade -i     --kube-context=kind-${KIND_CLUSTER_NAME}     --namespace gpu-operator     --create-namespace     --wait     nvidia-gpu-operator nvidia/gpu-operator     --set cdi.enabled=true     --set driver.enabled=false     --set operator.runtimeClass=nvidia
+```
+
+Running the following verifies that the pods for the operator are coming online:
+```bash
+kubectl --context=kind-${KIND_CLUSTER_NAME} get clusterpolicy -A
+kubectl --context=kind-${KIND_CLUSTER_NAME} get pods -n gpu-operator
+kubectl --context=kind-${KIND_CLUSTER_NAME} get ds -n gpu-operator
+```
+
+Running the following verifies that a workload can be deployed and run on a set
+of GPUs in this cluster:
+
+```bash
+kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f examples/gpu-operator/gpu-pod.yml
+kubectl --context=kind-${KIND_CLUSTER_NAME} logs gpu-pod
+```
+
+For a more automated walkthrough, including host preparation and troubleshooting,
+see [examples/gpu-operator/README.md](examples/gpu-operator/README.md).
+
 ## Delete all clusters
 
 The following command can be used to delete all `kind` clusters:

examples/gpu-operator/README.md

@@ -0,0 +1,160 @@
+# GPU Operator Kind Demo
+
+This example uses [gpu-operator-kind-demo.sh](./gpu-operator-kind-demo.sh) to create an `nvkind` cluster, install NVIDIA GPU Operator with `driver.enabled=false`, and verify GPU access with a sample workload.
+
+## Prerequisites
+
+The demo host must have:
+
+- Linux on Ubuntu or Debian
+- An NVIDIA GPU visible on the host
+- A working host NVIDIA driver
+
+Verify the host driver before running the demo:
+
+```bash
+nvidia-smi -L
+```
+
+If `nvidia-smi` is missing, install the host driver first:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y ubuntu-drivers-common
+sudo ubuntu-drivers devices
+sudo ubuntu-drivers autoinstall
+sudo reboot
+```
+
+The script will install or configure these tools if they are missing:
+
+- Go `1.24.3` by default, or `NVKIND_GO_VERSION`
+- `nvkind` via `go install github.com/NVIDIA/nvkind/cmd/nvkind@latest`
+- Docker
+- `kind`
+- `kubectl`
+- `helm`
+- `nvidia-container-toolkit`
+
+## Run The Demo
+
+From `examples/gpu-operator/`:
+
+```bash
+./gpu-operator-kind-demo.sh all
+```
+
+This will:
+
+- check host prerequisites
+- install `nvkind` if it is missing
+- configure Docker for NVIDIA GPUs
+- create the `nvkind-gpu-operator-demo` cluster
+- install GPU Operator
+- run the sample `gpu-pod` workload
+
+## Script Modes
+
+```bash
+./gpu-operator-kind-demo.sh prepare
+./gpu-operator-kind-demo.sh cluster
+./gpu-operator-kind-demo.sh install
+./gpu-operator-kind-demo.sh verify
+./gpu-operator-kind-demo.sh status
+./gpu-operator-kind-demo.sh cleanup
+```
+
+What each mode does:
+
+- `prepare`: checks or installs host-side prerequisites and validates Docker GPU access
+- `cluster`: prepares the host and creates the demo cluster
+- `install`: installs GPU Operator into the current demo cluster
+- `verify`: prints cluster state and runs the sample GPU workload
+- `status`: checks whether the existing demo cluster is healthy end-to-end
+- `cleanup`: deletes the demo cluster
+
+## Status Check
+
+To verify that an existing demo cluster is still healthy:
+
+```bash
+./gpu-operator-kind-demo.sh status
+```
+
+This checks:
+
+- cluster existence
+- node readiness
+- NVIDIA runtime classes
+- GPU Operator pod and daemonset health
+- `ClusterPolicy` readiness
+- end-to-end `gpu-pod` workload execution
+
+## What Success Looks Like
+
+A successful run ends after the `gpu-pod` workload completes, with output like:
+
+```text
+[demo] Demo completed successfully.
+```
+
+A successful status check ends with:
+
+```text
+[demo] Status: kind demo is healthy.
+```
+
+## Troubleshooting
+
+If the script adds your user to the `docker` group and exits, refresh the shell group membership and rerun:
+
+```bash
+newgrp docker
+./gpu-operator-kind-demo.sh all
+```
+
+If host driver validation fails, check:
+
+```bash
+nvidia-smi -L
+docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all ubuntu:22.04 nvidia-smi -L
+```
+
+If `nvidia-smi` is not found, install the host driver first:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y ubuntu-drivers-common
+sudo ubuntu-drivers devices
+sudo ubuntu-drivers autoinstall
+sudo reboot
+```
+
+If cluster creation fails while joining the worker and kubelet logs show `inotify_init: too many open files`, raise the host file and inotify limits before retrying:
+
+```bash
+sudo sysctl -w fs.inotify.max_user_instances=1024
+sudo sysctl -w fs.inotify.max_user_watches=524288
+sudo sysctl -w fs.file-max=2097152
+ulimit -n 1048576
+```
+
+For better persistence across reruns, also raise Docker's file descriptor limit and restart Docker:
+
+```bash
+sudo mkdir -p /etc/systemd/system/docker.service.d
+sudo tee /etc/systemd/system/docker.service.d/limits.conf <<'EOF2'
+[Service]
+LimitNOFILE=1048576
+EOF2
+sudo systemctl daemon-reload
+sudo systemctl restart docker
+```
+
+If the worker still fails to join, inspect the kind worker directly:
+
+```bash
+docker exec nvkind-gpu-operator-demo-worker systemctl status kubelet --no-pager -l
+docker exec nvkind-gpu-operator-demo-worker journalctl -u kubelet --no-pager -n 200
+docker exec nvkind-gpu-operator-demo-worker journalctl -u containerd --no-pager -n 200
+```