tensorflow gpu can not be called from jupyterhub/jupyter notebook, why?

Question

Well I figure eight hours is enough time trying to fix this on my own, so I'll just ask folks:

I am running tensorflow-gpu 1.1.0 just fine in my virtual environment named 'tensorflow' outside of jupyterhub and Jupyter notebook. That is, I can import tensorflow and run scripts with it using the gpu.

When I'm inside my tensorflow virtualenv and using jupyterhub, Jupyter can not seem to 'see' tensorflow. I get the following error:

ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory

1) This is a common seen error message indicating tensorflow install problems, yet my paths and environment variables seem fine. After all, I can use tensorflow-gpu just fine outside of Jupyter.

2) Typing 'which jupyter' shows ~/anaconda3/envs/hub/bin/jupyter, so I believe that I am referencing jupyter inside my virtualenv. 3) Pip freeze shows that I have jupyterhub and tensorflow-gpu. I even did a pip3 freeze and it shows both packages as well.

Any ideas? Can tensorflow-gpu be run from Jupyter notebooks?

Condensable

Yes, you can run tensorflow (both CPU and GPU versions) from Jupyter. The error seems to indicate that your CUDA is either not installed or not configured correctly. You might want to check that CUDA_HOME environment variable is set and that your LD_LIBRARY_PATH includes your CUDA install location. – Leandroleaning

Mad Wombat, have you used tensorflow gpu and Jupyter from a virtual environment? Would not the issue be jupyter, since I likely have set up tensorflow gpu correctly because it works outside of jupyter – Condensable

I frequently use TensorFlow for GPU in Jupyter from a virtual environment. I do not need to have LD_LIBRARY_PATH set, but CUDA_HOME must be properly configured. You can check if the variable has the expected value from the notebook with import os; os.environ.get('CUDA_HOME'). – Culminant

I have used TensorFlow from Jupyter notebook on my Windows 10 laptop. I did not use virtualenv, I used conda environments, but these should be functionally the same. You do have to install both jupyter and tensorflow in the same virtual environment, of course. – Leandroleaning

Answer 1

I got the solution from here:

[https://github.com/jupyter/notebook/issues/1290][1]

Basically, something was 'wrong' with jupyter in that it could not read my LD_LIBRARY_PATH variable. I did put everything correctly inside .bashrc so I don't know why.

Switch to the command line (terminal). Switch into your virtual environment if you have one.

type in: jupyter notebook --generate-config

It will tell you the directory in which your jupyter configuration file is stored. If you want to list it again type: jupyter --config-dir

Mine jupyter_notebook_config.py file is located here: /home/me/.jupyter/jupyter_notebook_config.py

At the very top of this file, jupyter_notebook_config.py, add in the following code:

import os
c = get_config()
os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-8.0/lib64:usr/local/cuda-8.0/lib64/libcudart.so.8.0'
c.Spawner.env.update('LD_LIBRARY_PATH')

Then restart jupyterhub or jupyter notebook (type in at the command line: jupyter notebook

Tensorflow gpu should work.

The same thing applies even if you are running jupyterhub. Make the changes in jupyter, not jupyterhub. (Each user of jupyterhub will have their own jupyter process, so make the changes not at the 'hub' level, but the jupyter notebook level.

Condensable