Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory;

Question

When I try to run a python script , which uses tensorflow, it shows following error ...

2020-10-04 16:01:44.994797: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-04 16:01:46.780656: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-04 16:01:46.795642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: TITAN X (Pascal) computeCapability: 6.1
coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s
2020-10-04 16:01:46.795699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-04 16:01:46.795808: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64/:/usr/local/cuda-10.0/lib64
2020-10-04 16:01:46.797391: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-04 16:01:46.797707: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-04 16:01:46.799529: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-04 16:01:46.800524: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-04 16:01:46.804150: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-04 16:01:46.804169: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

Output of nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.05    Driver Version: 455.23.05    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    On   | 00000000:03:00.0 Off |                  N/A |
| 23%   28C    P8     9W / 250W |     18MiB / 12194MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1825      G   /usr/lib/xorg/Xorg                  9MiB |
|    0   N/A  N/A      1957      G   /usr/bin/gnome-shell                6MiB |
+-----------------------------------------------------------------------------+

Tensorflow version 2.3.1, Ubuntu - 18.04

I tried to completely remove cuda toolkit and install from scratch but the error remains. Anybody could help me to identify the source of problem??

Hamby

Your Tensorflow is expecting some version of of CUDA 10.x and you have installed some version of CUDA 11.x. You cannot use CUDA 11.x (or any other version of CUDA) as a replacement for CUDA 10.x If your TF build requires, for example, CUDA 10.1, you must install CUDA 10.1 (not the latest CUDA) in order to use that version of TF. – Nettlesome

Answer 1

This usually happens when you run tensorflow with a non compatible version of CUDA. Looks like this has been asked before (could not comment). Refer this question.

Hypoderm

tnx, i had 'libcublas.so.11' in ubuntu 20.04 error but install tensorflow==2.2.0 solved my problem. – Singsong

ls -l /usr/lib/x86_64-linux-gnu/libcudart.so* shows the version you have installed. In my case, 11 was needed, but 10 was installed. Thanks – Osculation

And, on pypi.org/project/tensorflow you can see version 10 (what my apt installed for nvidia-cuda-toolkit, is not supported on all recent versions. (Oldest version on pip is 2.5.0rc0) – Osculation

Answer 2

On Ubuntu 20.04, you can simply install NVIDIAs cuda toolkit cuda:

sudo apt-get update
sudo apt install nvidia-cuda-toolkit

There are also install advices for Windows.

The packge is around 1GB and it took a while to install... Some minutes later you need to export PATH variables so that it can be found:

1. Find Shared Object

sudo find / -name 'libcudart.so*'

/usr/lib/x86_64-linux-gnu/libcudart.so.10.1
/usr/lib/x86_64-linux-gnu/libcudart.so

2. Add the folder to path, so that python finds it

export PATH=/usr/lib/x86_64-linux-gnu${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

3. Permissions

sudo chmod a+r /usr/lib/x86_64-linux-gnu/libcuda*

Helped me

Meat

There's an unncessary ":" in LD_LIBRARY_PATH, should be: export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} – Raspberry

Answer 3

Today I was facing this problem. I went to the CUDA toolkit website, selected the options, and that showed some instructions like this:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.6.2/local_installers/cuda-repo-ubuntu2004-11-6-local_11.6.2-510.47.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-6-local_11.6.2-510.47.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-6-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda         # I have broken packages, so could not invoke this command

So the instructions will change depending on your specifications, DO NOT copy from here/other stackoverflow answer.

I could not invoke the last command, but after some trials and errors, I invoked:

sudo apt install libcudart.so.11.0   # this worked for me!

This worked for me!

Luker

Answer 4

You have to download/update Cuda

If you are looking CUDA Toolkit 10.2 Download use this link: https://developer.nvidia.com/cuda-10.2-download-archive

Then active the virtual environment and set the LD_LIBRARY_PATH, example: Tensorflow Could not load dynamic library 'libcudart.so.10.0 on ubuntu 18.04

Unwritten

Answer 5

Please run these commands, if you are having ubuntu 18.04 installed. or follow the instructions here

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin

sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub

sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"

sudo apt-get update

sudo apt-get -y install cuda 

Sidedress

Seemed to work for me today when I updated to Ubuntu 20.04 - I just replaced all your 1804s with 2004 and now my Jupyter notebook (with tensorflow and tensorflow-gpu 2.5.0 can see my GPU. – Azarria

Answer 6

This worked for me:

sudo apt-get install libcudart10.1

Sanfordsanfourd

Answer 7

Make sure you have installed the compatible GPU/CPU version of tensorflow. I was installing tensorflow in a Pipenv virtual environment on my Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz 2.59 GHz computer. I was using pip install tensorflow and I was getting the same message. The following is the output message after executing a notebook cell or a code including import tensorflow as tf.

2023-02-27 14:53:16.590721: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-27 14:53:16.957000: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-02-27 14:53:16.957019: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-02-27 14:53:18.213523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-02-27 14:53:18.213591: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-02-27 14:53:18.213597: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

When I changed the installation command to pip install tensorflow-cpu the error disappeared. This the new output message after executing the same notebook cell or a code including import tensorflow as tf.

023-02-28 10:01:35.003241: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

pip install tensorflow-gpu may solve the problem. See here for details on tensorflow-gpu. However, check carefully the this official pip repository it seems from december 2022 it is advice to install tensorflow instead of tensorflow-gpu.

Haland