CUDA driver version is insufficient for CUDA runtime version

Question

I got the message:

"cutilCheckMsg() CUTIL CUDA error :
kernel launch failure : CUDA driver
version is insufficient for CUDA
runtime version."

While trying to run an example source code. Also happens for the function cutilSafeCall.

I am using:

• Windows 7 64bits
• Visual studio 2008
• CUDA developer driver, toolkit, and SDK 3.1
• Emulation mode

Flex

It sounds like you need to update your driver. And you might want to mention what OS you are using, what CUDA SDK version, etc. – Reaganreagen

Had the same message...in my case it had been just after a windows update, and the process removed Cuda Toolkit...simply required reinstallation. Check version compatibility. Here's my known working config: Windows 10, 64 bit, cuda toolkit v9.0, Visual Studio 2015, Nvidia Quadro P4000, Python version and libraries: Python 3.6.7, Tensorflow-gpu 1.13.1, Cudatoolkit 9.0, cudnn 7.1.4 – Snowmobile

Answer 1

I saw the same at runtime with the latest driver on Mac OS 10.6.

cudaError_t error = cudaGetDevice(&device);
printf("%s\n", cudaGetErrorString(error));

I went back to the developer site, downloaded the driver again and now it runs. http://developer.nvidia.com/object/cuda_3_1_downloads.html#MacOS

Tormentil

Answer 2

You need to ensure that your driver version matches or exceeds your CUDA Toolkit version.

For 2.3 you need a 190.x driver, for 3.0 you need 195.x and for 3.1 you need 256.x (actually anything up to the next multiple of five is ok, e.g. 258.x for 3.1).

You can check your driver version by either running the deviceQueryDrv SDK sample or go into the NVIDIA Control Panel and choose System Information.

Download an updated driver from www.nvidia.com/drivers.

Idiot

Answer 3

Counterintuitively, this error also happens if libcuda.so is not found, even when versions reported by nvidia-smi match perfectly. This library is part of nvidia-drivers package (On CentOS: nvidia-driver-latest-cuda-libs, on Gentoo x11-drivers/nvidia-drivers). It is possible to have the CUDA Tookit with nvcc and libcudart installed and building your app fine, but the drivers part not installed, causing this error.

To diagnose whether this is the reason, use strace:

strace -f -e trace=file ./your_cuda_app

and check for open calls to libcuda.so*, at least one of them should return with a success code, like so:

4928  open("/lib64/libcuda.so.1", O_RDONLY|O_CLOEXEC) = 3 

Heterograft

In my case, I'm using docker image for a service, and I forgot to add ` --runtime=nvidia` in my docker run, which caused libcuda.so to be unavailable. – Poplar

Answer 4

You can either download the latest driver OR use an older toolkit version to compile your code.

Laevorotatory

You also need to make sure that your include / library directories point to the correct version. I have CUDA 5.5 and CUDA 6.0, and even when I changed the build settings to use the 5.5 compiler (in VS project settings), it was still quietly linking to 6.0 libraries (in VS Options / C++ Directories). Now it's working perfectly. Silly me ... – Exergue

Answer 5

My cent,

with Linux/Unix this error may be related to the selected GPU mode (Performance/Power Saving Mode), when you select (with nvidia-settings utiliy) the integrated Intel GPU and you execute the deviceQuery script... you get this error:

-> CUDA driver version is insufficient for CUDA runtime version

But this error is misleading, by selecting back the NVIDIA(Performance mode) with nvidia-settings utility the problem disappears.

It is not a version problem.

Regards

P.s: "Power Saving Mode" tells Optimus to activate the CPU integrated Intel GPU

Urology

Answer 6

CUDA driver version is insufficient for CUDA runtime version: means your GPU can`t been manipulated by the CUDA runtime API, so you need to update your driver.

Drynurse

just download the latest version and install – Drynurse

Answer 7

In my case, I had to run my docker container with nvidia-docker run ... instead of docker run ...

Manganin

Answer 8

I also had similar problem, updated my graphic driver but the problem still remained. I finally decided to remove Cuda 9.2 and install Cuda 8, it solved my issue.

Meridith

Answer 9

This is what has worked for me :

1. Installing TensorFlow version 1.14, installing keras
2. Install CUDA 10.0 from https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exenetwork for windows
3. Download the compatible CuDNN i.e 7.4.2 from https://developer.nvidia.com/rdp/cudnn-archive
4. set the following in the path variable cudnn-10.0-windows10-x64-v7.4.2.24\cuda\bin cudnn-10.0-windows10-x64-v7.4.2.24\cuda\include cudnn-10.0-windows10-x64-v7.4.2.24\cuda\lib\x64
5. Restart the Pc

Sybarite

Answer 10

This problem can also be because of incorrect environment setup, e.g. Docker image setup. Although the driver itself is correct, sufficient for your program. If your LD_LIBRARY_PATH points to the wrong driver, it can throw this error. In my case, i get this error when using /usr/local/nvidia/lib/libcuda.so, and if I use /usr/local/nvidia/lib64/libcuda.so everything goes right.

Howell

Answer 11

I had to add -Wno-deprecated-gpu-targets as a flag during compilation.

Floorwalker

Answer 12

Maybe it is related to the TBB lib: Error OpenCV with CUDA using TBB for multiple GPUs

Try rebuilding it making sure you passed the following parameters to CMake (assuming you already installed "tbb" and "tbb-devel" packages:

-D WITH_TBB=YES -D TBB_INCLUDE_DIRS=/usr/include/tbb

Macrography