docker --device works with absolute device path, fails with symlink

Question

I have a USB GPIO device with some 3rd party drivers that I want to interface with in a container. Compiles, works fine on host. Also compiles, works fine in the container if I pass --device=/dev/bus/usb/001/$NUM where $NUM is the autogenerated path when the device is plugged in; I presume udev is assigning this. However I want a deterministic bind point, so I modified the udev rules to assign a symlink:

SUBSYSTEM=="usb", ATTR{idVendor}=="09db", 
ATTR{idProduct}=="0075", MODE="0666", 
TAG+="uaccess", TAG+="udev-acl", 
SYMLINK+="mcc_daq"

And that gives me a symlink at /dev/mcc_daq to /dev/bus/usb/whatever. Still works fine on host.

But, if I run with:

docker run --rm -it \
        --device=/dev/mcc_daq \
        mcc_daq1

I get

usb_device_find_USB_MCC: libusb_open failed.: No such file or directory 
Failure, did not find a USB 2408 or 2408_2AO!

which is what the test program spits out when it can't find the device (uses libusb_get_device_descriptor).

When I run in the container udevadm info -q all /dev/mcc_daq, I get the same exact output as udevadm info -q all /dev/bus/usb/001/004 when I start the container with --device=/.../004. However, if I pass in the symlink path to docker, I can't query the absolute path.

The udev output is more verbose on the host system, I'm not sure if this is part of the problem or expected behavior. My intuition is the lack of those extra entries means that libusb can find the device, but it can't find the vendor ID of the product. However, the fact that the udev output is the same in the container (regardless whether symlink or hard path) makes me question that.

Update: New fun fact: when I run strace on the test program, I find:

open("/dev/bus/usb/001/004", O_RDWR)    = -1 ENOENT (No such file or directory)

So that means the driver is looking for ../004, even though I mounted ---device=/dev/mcc_daq. I'm not sure where it's getting this information.

In short, I think using docker run --device=/dev/symlink does not introduce the same udev context as using the hard path, even though output of udevadm info is the same.

Host: udevadm info -q all /dev/mcc_daq

P: /devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
N: bus/usb/001/004
S: mcc_daq
E: BUSNUM=001
E: DEVLINKS=/dev/mcc_daq
E: DEVNAME=/dev/bus/usb/001/004
E: DEVNUM=004
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
E: DEVTYPE=usb_device
E: DRIVER=usb
E: ID_BUS=usb
E: ID_FOR_SEAT=usb-pci-0000_00_14_0-usb-0_8_2
E: ID_MODEL=USB-2408-2AO
E: ID_MODEL_ENC=USB-2408-2AO
E: ID_MODEL_ID=00fe
E: ID_PATH=pci-0000:00:14.0-usb-0:8.2
E: ID_PATH_TAG=pci-0000_00_14_0-usb-0_8_2
E: ID_REVISION=0101
E: ID_SERIAL=MCC_USB-2408-2AO_01DA523C
E: ID_SERIAL_SHORT=01DA523C
E: ID_USB_INTERFACES=:ffff00:
E: ID_VENDOR=MCC
E: ID_VENDOR_ENC=MCC
E: ID_VENDOR_FROM_DATABASE=Measurement Computing Corp.
E: ID_VENDOR_ID=09db
E: MAJOR=189
E: MINOR=3
E: PRODUCT=9db/fe/101
E: SUBSYSTEM=usb
E: TAGS=:uaccess:seat:udev-acl:
E: TYPE=255/255/0
E: USEC_INITIALIZED=19197013

Docker: udevadm info -q all /dev/mcc_daq

P: /devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
N: bus/usb/001/004
E: BUSNUM=001
E: DEVNAME=/dev/bus/usb/001/004
E: DEVNUM=004
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
E: DEVTYPE=usb_device
E: DRIVER=usb
E: MAJOR=189
E: MINOR=3
E: PRODUCT=9db/fe/101
E: SUBSYSTEM=usb
E: TYPE=255/255/0

Docker: udevadm info -q all /dev/bus/usb/001/004

P: /devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
N: bus/usb/001/004
E: BUSNUM=001
E: DEVNAME=/dev/bus/usb/001/004
E: DEVNUM=004
E: DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-8/1-8.2
E: DEVTYPE=usb_device
E: DRIVER=usb
E: MAJOR=189
E: MINOR=3
E: PRODUCT=9db/fe/101
E: SUBSYSTEM=usb
E: TYPE=255/255/0

Countrywoman

Answer 1

From this discussion I concluded that if your docker run command is how you actually use it, you can do a workaround using the bash subsition below helped by the tool 'readlink':

docker run --rm -it \
        --device=$(readlink -f /dev/mcc_daq) \
        mcc_daq1

However if you use docker-compose like me, this workaround get's slightly more awkward because you would need to do some preparatory juggling with environment variables I guess.

One of the problems with this workaround is of course that if you replug your device the path will be invalid. So you would have to restart the container I guess..

Radical

this is literally the solution I ended up using :) I use it with a Makefile instead of docker-compose. I'm hoping it's not going to bite me down the road. – Countrywoman

It will bite you in the $proverbial if the USB devices decide to play musical chairs after a reboot. To make it stick, you cannot rely on docker's restart logic (or docker-compose's "-d" flag, for that matter) to fix things up. This of course totally defeats the purpose of using udev to set up a symlink. If you use docker-compose, you must use a script at system boot time to update the device reference and re-run the "docker-compose up -d". – Winchester

Answer 2

Adding to the answer by Sebastian.

While container restart is still required should the usb device's port name change, we can at least standardize the scripts inside using environment variable.

sudo docker container rm <container name>;
sudo docker run --device=$(readlink -f /dev/<symlink>) -e USBVAR=$(readlink -f /dev/<symlink>) --name <container name> -it <image name> bash

And the scripts inside can just access the environment variable USBVAR to get the device port name. For instance, when one is using espflash or similar tool, just type the below within the bash

espflash board-info $USBVAR

Any kind of attempt to introduce symlink in the device path kept failing in my scenario, so I resorted to this solution.

Fun

Answer 3

I hit this too, and haven't found a clean solution. In my case, I was using the nut-upsd container, with a udev remap rule symlink shared into the container. My error was

+ chgrp -R nut /etc/nut /dev/bus/usb
chgrp: /dev/bus/usb: No such file or directory

and it was clear that this directory was indeed empty, and devices were not going to load properly for upsd. I still haven't found a "clean" solution, but I have resorted to overriding the container's entrypoint, with a custom one that sets up the device properly, then starts the inner entrypoint.

#!/bin/sh -ex
DEVICE=$(lsusb | grep "{{ device_vendor_id }}:{{ device_product_id }}")
BUS=$(echo "$DEVICE" | cut -d' ' -f 2)
DEVICE=$(echo "$DEVICE" | cut -d' ' -f 4 | tr -d :)
DEST="/dev/bus/usb/$BUS/$DEVICE"
echo "Linking {{ device_symlink }} to $DEST"
mkdir -p "/dev/bus/usb/$BUS"
ln -s "/dev/{{ device_symlink }}" "$DEST"

{{ service_main_entrypoint }} $@

In my case, you can see my usage of ansible variables, as this is configured using an ansible playbook.

I'm currently using this for nut-upsd as well as rtl433.

Willett

Ansible is smart. I didn't know much about it when I started that project. In hindsight, ansible (or terraform+nomad) would have solved SO MANY problems. What I learned is that "clean" is less important as long as it's robust and reliable. – Countrywoman

Answer 4

I hope to contribute to this discussion, albeit it has been a while. What I noticed was that the symlink seems to be dereferenced and the device it pointed to is passed to the container.

The musical chairs analogy is very appropriate: if a devices are connected/disconnected in an arbitrary order, the symlinks would be updated, but the container would still used what they pointed to at the time the container was launched.

My solution, albeit blunt, was to restart the container - I could get away with this in that particular application.

For example:

docker run -p 5100:80 -v octoprint_mini1:/octoprint --device /dev/ttyMINI1:/dev/ttyACM0  --name mini1 -dit --restart unless-stopped    octoprint/octoprint

docker run -p 5200:80 -v octoprint_mini2:/octoprint --device /dev/ttyMINI2:/dev/ttyACM0   --name mini2 -dit --restart unless-stopped  octoprint/octoprint

docker run -p 5300:80 -v octoprint_mk3s1:/octoprint --device /dev/ttyMK3S1:/dev/ttyACM0   --name mk3s -dit --restart unless-stopped  octoprint/octoprint

the corresponding udev are:

KERNEL=="ttyACM[0-9]*", SUBSYSTEM=="tty", ATTRS{idVendor}=="2c99", ATTRS{serial}=="CZPX0522X017XC11111", SYMLINK="ttyMINI1", RUN="/usr/bin/docker restart mini1"

KERNEL=="ttyACM[0-9]*", SUBSYSTEM=="tty", ATTRS{idVendor}=="2c99", ATTRS{serial}=="CZPX1022X017XC22222", SYMLINK="ttyMINI2", RUN="/usr/bin/docker restart mini2"

KERNEL=="ttyACM[0-9]*", SUBSYSTEM=="tty", ATTRS{idVendor}=="2c99", ATTRS{serial}=="CZPX3021X004XC33333", SYMLINK="ttyMK3S1", RUN="/usr/bin/docker restart mk3s1"

Euniceeunuch

Thanks for the update, though this project is long in my rearview mirror. I think if the device disconnects, and you get the musical chairs situation, there's no choice but to restart the container. – Countrywoman

Answer 5

My solution was to only add the /dev volume and mount it inside the container and i have the privileged flag set to true. In my compose file:

...
privileged: true
volumes:
   /dev:/dev
...

Aleece