nvidia-docker : Unknown runtime specified nvidia

Question

I tried to install the nvidia-docker after installing docker-ce. I followed this : https://github.com/NVIDIA/nvidia-docker to install nvidia-docker. It seems to have installed correctly.

I tried to run:

$ sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.

Although, this works (without --runtime=nvidia):

$ docker container run -ti ubuntu bash

Some additional info on my system: It is an ubuntu server 16.04 with 8 GPUs (Titan Xp) and nvidia driver version 387.26. I can run nvidia-smi -l 1 on the host system and it works as expected.

$ dpkg -l | grep -E '(nvidia|docker)'
ii  docker-ce                              18.06.1~ce~3-0~ubuntu                        amd64        Docker: the open-source application container engine
ii  libnvidia-container-tools              1.0.0-1                                      amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64             1.0.0-1                                      amd64        NVIDIA container runtime library
ii  nvidia-container-runtime               2.0.0+docker18.06.1-1                        amd64        NVIDIA container runtime
ii  nvidia-container-runtime-hook          1.4.0-1                                      amd64        NVIDIA container runtime hook
ii  nvidia-docker2                         2.0.3+docker18.06.1-1                        all          nvidia-docker CLI wrapper



$ cat /etc/docker/daemon.json 
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I have come across: https://github.com/NVIDIA/nvidia-docker/issues/501, but I am not sure how I should go about it.

Chantress

--runtime nvidia is just for nvidia-docker2. --gpus [all|num|dev] should be used instead from Docker 19.03 on. github.com/NVIDIA/nvidia-docker#usage – Unleash

nvidia-docker is deprecated. – Greenbelt

This worked for me: docs.nvidia.com/datacenter/cloud-native/container-toolkit/… – Maryammaryann

Answer 1

From nvidia-docker github repo:

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

Lambeth

Answer 2

Actually, you can try to restart docker daemon by following command.

sudo systemctl daemon-reload
sudo systemctl restart docker

Or you can try to reboot your system. to make nvidia-docker work

Homey

Answer 3

This is how I resolve the above problem for CentOS 7; hopefully it can help anyone who has similar problems.

• Add necessary repos to get nvidia-container-runtime:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.repo | sudo tee /etc/yum.repos.d/nvidia-container-runtime.repo

• (Optional) In my case, I disabled the experimental repos:

sudo yum-config-manager --disable libnvidia-container-experimental
sudo yum-config-manager --disable nvidia-container-runtime-experimental

• Install nvidia-container-runtime package:

sudo yum install nvidia-container-runtime

• Update docker daemon:

sudo vim /etc/docker/daemon.json

with the path to nvidia-container-runtime:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

• Finally, you need to make docker update the path:

sudo pkill -SIGHUP dockerd

Anglican

Answer 4

in docker version above 19.03 or above, nvidia-docker2 is not needed anymore (deprecated) https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html but their document is missing configuration part

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

I believe this would solve original

docker: Error response from daemon: unknown or invalid runtime name: nvidia

If nvidia-container-toolkit and docker are installed properly

Woodson

Answer 5

It seems you may need to purge docker and reinstall it as in the post: github issues

sudo apt remove docker-ce
sudo apt autoremove
sudo apt-get install docker-ce=5:18.09.0~3-0~ubuntu-bionic
sudo apt install nvidia-docker2

Selfrevealing

Links are good but you should also copy paste the relevant contents from there to your post so that even if the link is not available in future your answer will still make sense. – Crape

I believe sudo apt install nvidia-docker2 is deprecated as of Jan 16, 2020. On this page from nvidia-docker, it suggests sudo apt-get install -y nvidia-container-toolkit – Elohist

E: Version '5:18.09.0~3-0~ubuntu-bionic' for 'docker-ce' was not found – Penchant

@Nathan E: Unable to locate package nvidia-container-toolkit – Penchant

@MonaJalal Sorry, somehow I just saw this. Did you get it working? Probably haven't thought about this in quite awhile by now – Elohist

Answer 6

From nvidia-docker Frequently Asked Questions:

Why do I get the error Unknown runtime specified nvidia? Make sure the runtime was registered to dockerd. You also need to reload the configuration of the Docker daemon.

Enfeoff

Posting urls is not enough. They may become invalid. – Macedonia

Answer 7

If you're having trouble installing nvidia-docker then try running this shell script. It worked for me even when nvidia-docker crashed.

Val

Answer 8

Nvidia-docker2 is not supported anymore for newer Ubuntu such as 22.04 LTS Jammy. I managed to get it to work by using instructions from here, in particular the containerd -part (middle):

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Symphonist

Answer 9

See here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker

you may need some configuring jobs after installing nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

Brainwork

Answer 10

Change the --runtime=nvidia tag to --runtine=gpus all hopefully it will run

Pectize