How can I resolve a CUDA driver and runtime version mismatch in a Kubernetes containerd GPU cluster?

Error:
CUDA driver version is insufficient for CUDA runtime version

Description:
I am testing a 3 node Kubernetes cluster with GPU compute, and I’m encountering CUDA version mismatch errors when running CUDA samples. The same samples work with Docker runtime, but when using containerd via the GPU Operator, the error occurs. The official documentation suggests using CUDA 12.4 with NVIDIA driver 550.144.03, yet my current container image (nvidia/samples:vectoradd-cuda11.2.1) uses CUDA 11.2, which might be causing the issue.

Below is the execution using Docker that passes:

BASHCopied
sudo docker run --rm --gpus all nvidia/samples:vectoradd-cuda11.2.1

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

cat vectoradd.yml

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  containers:
  - name: cuda-vectoradd
    image: "nvidia/samples:vectoradd-cuda11.2.1"
    resources:
      limits:
        nvidia.com/gpu: 3
  nodeSelector:
    nvidia.com/gpu.product: Tesla-T4
    worker: worker1

kubectl logs cuda-vectoradd

[Vector addition of 50000 elements]
Failed to allocate device vector A (error code CUDA driver version is insufficient for CUDA runtime version)!

Environment:
3 node Kubernetes cluster with GPU compute

nvidia-smi output:

nvcc -V output:

BASHCopied
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

The NVIDIA GPU Operator reports all validations as passed:

BASHCopied
kubectl get pods

NAME                                                              READY   STATUS             RESTARTS       AGE
cuda-vectoradd                                                    0/1     CrashLoopBackOff   42 (16s ago)   3h10m
gpu-feature-discovery-nmkdp                                       1/1     Running            0              32h
gpu-feature-discovery-x5kp5                                       1/1     Running            0              32h
gpu-operator-1738864536-node-feature-discovery-gc-55fcb5f46x45j   1/1     Running            0              32h
gpu-operator-1738864536-node-feature-discovery-master-6946p7p95   1/1     Running            0              32h
gpu-operator-1738864536-node-feature-discovery-worker-4xtr8       1/1     Running            0              32h
gpu-operator-1738864536-node-feature-discovery-worker-pnbrq       1/1     Running            0              32h
gpu-operator-1738864536-node-feature-discovery-worker-px6ks       1/1     Running            0              32h
gpu-operator-84d6c857b4-kllq5                                     1/1     Running            0              32h
nvidia-cuda-validator-45jvj                                       0/1     Completed          0              32h
nvidia-cuda-validator-72bfn                                       0/1     Completed          0              32h
nvidia-dcgm-exporter-7k94s                                        1/1     Running            0              32h
nvidia-dcgm-exporter-bnvqg                                        1/1     Running            0              32h
nvidia-device-plugin-daemonset-6h6mg                              1/1     Running            0              32h
nvidia-device-plugin-daemonset-wxzs9                              1/1     Running            0              32h

The CUDA validator logs also indicate success:

BASHCopied
kubectl logs nvidia-cuda-validator-45jvj

Defaulted container "nvidia-cuda-validator" out of: nvidia-cuda-validator, cuda-validation (init)
cuda workload validation is successful

I would appreciate guidance on whether to upgrade the container’s CUDA runtime to match the host driver version or to update the host drivers to support the image’s CUDA version. Any help to resolve this mismatch is welcome.