nvidia-device-plugin doesn't schedule plugin Pods to workers

[sfc-gh-lshi]

With the currently documented steps, nvidia-device-plugin does not schedule the plugin DaemonSet to any of the cluster workers. This is because the worker nodes are missing required labels.

After installing the Helm chart, the worker(s) need to be labeled thusly:

kubectl label node "${KIND_CLUSTER_NAME}-worker" --overwrite nvidia.com/gpu.present=true

Only afterwards does kubectl --context=kind-${KIND_CLUSTER_NAME} get pod -n nvidia show any Pods.

[LogExE]

It looks like we have to pass --set gfd.enabled=true to helm install command

[jbdalido]

It looks like we have to pass --set gfd.enabled=true to helm install command

+1 @LogExE maybe open a PR to update the yaml?

[wnark]

With the currently documented steps, nvidia-device-plugin does not schedule the plugin DaemonSet to any of the cluster workers. This is because the worker nodes are missing required labels.

After installing the Helm chart, the worker(s) need to be labeled thusly:

kubectl label node "${KIND_CLUSTER_NAME}-worker" --overwrite nvidia.com/gpu.present=true

Only afterwards does kubectl --context=kind-${KIND_CLUSTER_NAME} get pod -n nvidia show any Pods.

Even if the node has been tagged, the device plugin still does not work properly.

$ kubectl describe pod nvidia-device-plugin-sjz9d -n nvidia
Name:                 nvidia-device-plugin-sjz9d
Namespace:            nvidia
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      default
Node:                 kubeflow-worker/172.18.0.3
Start Time:           Wed, 30 Jul 2025 10:27:07 +0800
Labels:               app.kubernetes.io/instance=nvidia-device-plugin
                      app.kubernetes.io/name=nvidia-device-plugin
                      controller-revision-hash=844868695
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   10.244.1.2
IPs:
  IP:           10.244.1.2
Controlled By:  DaemonSet/nvidia-device-plugin
Containers:
  nvidia-device-plugin-ctr:
    Container ID:  containerd://269935f4087ba24849a6d5d470205c0574a442bd31a228b69d67bce04b5bbb7d
    Image:         nvcr.io/nvidia/k8s-device-plugin:v0.17.3
    Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:630596340f8e83aa10b0bc13a46db76772e31b7dccfc34d3a4e41ab7e0aa6117
    Port:          <none>
    Host Port:     <none>
    Command:
      nvidia-device-plugin
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 30 Jul 2025 10:27:37 +0800
      Finished:     Wed, 30 Jul 2025 10:27:37 +0800
    Ready:          False
    Restart Count:  2
    Environment:
      MPS_ROOT:                    /run/nvidia/mps
      NVIDIA_VISIBLE_DEVICES:      all
      NVIDIA_DRIVER_CAPABILITIES:  compute,utility
    Mounts:
      /dev/shm from mps-shm (rw)
      /mps from mps-root (rw)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/run/cdi from cdi-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sz5tt (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:  
  mps-root:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/mps
    HostPathType:  DirectoryOrCreate
  mps-shm:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/mps/shm
    HostPathType:  
  cdi-root:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/cdi
    HostPathType:  DirectoryOrCreate
  kube-api-access-sz5tt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             nvidia.com/gpu:NoSchedule op=Exists
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  52s                default-scheduler  Successfully assigned nvidia/nvidia-device-plugin-sjz9d to kubeflow-worker
  Normal   Pulling    51s                kubelet            Pulling image "nvcr.io/nvidia/k8s-device-plugin:v0.17.3"
  Normal   Pulled     41s                kubelet            Successfully pulled image "nvcr.io/nvidia/k8s-device-plugin:v0.17.3" in 10.019s (10.019s including waiting). Image size: 204067661 bytes.
  Normal   Created    22s (x3 over 41s)  kubelet            Created container: nvidia-device-plugin-ctr
  Normal   Started    22s (x3 over 41s)  kubelet            Started container nvidia-device-plugin-ctr
  Normal   Pulled     22s (x2 over 39s)  kubelet            Container image "nvcr.io/nvidia/k8s-device-plugin:v0.17.3" already present on machine
  Warning  BackOff    10s (x3 over 38s)  kubelet            Back-off restarting failed container nvidia-device-plugin-ctr in pod nvidia-device-plugin-sjz9d_nvidia(e2146167-f6d1-43ba-be26-2b9da46a3f7f)


$ kubectl logs nvidia-device-plugin-sjz9d -n nvidia
I0730 02:28:46.214372       1 main.go:235] "Starting NVIDIA Device Plugin" version=<
	e0a461e1
	commit: e0a461e1e7ad1d239d4708c954f08c3038e2654a
 >
I0730 02:28:46.214430       1 main.go:238] Starting FS watcher for /var/lib/kubelet/device-plugins
I0730 02:28:46.214470       1 main.go:245] Starting OS watcher.
I0730 02:28:46.214827       1 main.go:260] Starting Plugins.
I0730 02:28:46.214865       1 main.go:317] Loading configuration.
I0730 02:28:46.215384       1 main.go:342] Updating config with default resource matching patterns.
I0730 02:28:46.215608       1 main.go:353] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "mpsRoot": "/run/nvidia/mps",
    "nvidiaDriverRoot": "/",
    "nvidiaDevRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": false,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/driver-root"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {}
  },
  "imex": {}
}
I0730 02:28:46.215625       1 main.go:356] Retrieving plugins.
E0730 02:28:46.215734       1 factory.go:112] Incompatible strategy detected auto
E0730 02:28:46.215748       1 factory.go:113] If this is a GPU node, did you configure the NVIDIA Container Toolkit?
E0730 02:28:46.215752       1 factory.go:114] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E0730 02:28:46.215756       1 factory.go:115] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E0730 02:28:46.215763       1 factory.go:116] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E0730 02:28:46.222908       1 main.go:173] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: invalid device discovery strategy

However, based on the operator, it can work normally.just like #48

[wnark]

It looks like we have to pass --set gfd.enabled=true to helm install command

+1 @LogExE maybe open a PR to update the yaml?
When I set --set gfd.enabled=true, pods still fail to start.

$ kubectl logs nvidia-device-plugin-gpu-feature-discovery-zr7h6 -n nvidia
I0730 02:34:19.241263       1 main.go:163] Starting OS watcher.
I0730 02:34:19.241752       1 main.go:168] Loading configuration.
I0730 02:34:19.242227       1 main.go:180] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "gdsEnabled": null,
    "mofedEnabled": null,
    "useNodeFeatureAPI": false,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": null,
      "deviceListStrategy": null,
      "deviceIDStrategy": null,
      "cdiAnnotationPrefix": null,
      "nvidiaCTKPath": null,
      "containerDriverRoot": "/driver-root"
    },
    "gfd": {
      "oneshot": false,
      "noTimestamp": false,
      "sleepInterval": "1m0s",
      "outputFile": "/etc/kubernetes/node-feature-discovery/features.d/gfd",
      "machineTypeFile": "/sys/class/dmi/id/product_name"
    }
  },
  "resources": {
    "gpus": null
  },
  "sharing": {
    "timeSlicing": {}
  },
  "imex": {}
}
I0730 02:34:19.242526       1 main.go:160] Exiting
E0730 02:34:19.242552       1 main.go:127] failed to create resource manager: unsupported strategy auto