docker volume over fuse : Transport endpoint is not connected

Question

So I have this remote folder /mnt/shared mounted with fuse. It is mostly available, except there shall be some disconnections from time to time.

The actual mounted folder /mnt/shared becomes available again when the re-connection happens.

The issue is that I put this folder into a docker volume to make it available to my app: /shared. When I start the container, the volume is available.

But if a disconnection happens in between, while the /mnt/shared repo on the host machine is available, the /shared folder is not accessible from the container, and I get:

user@machine:~$ docker exec -it e313ec554814 bash
root@e313ec554814:/app# ls /shared 
ls: cannot access '/shared': Transport endpoint is not connected

In order to get it to work again, the only solution I found is to docker restart e313ec554814, which brings downtime to my app, hence is not an acceptable solution.

So my questions are:

1. Is this somehow a docker "bug" not to reconnect to the mounted folder when it is available again?

2. Can I execute this task manually, without having to restart the whole container?

Thanks

Trifurcate

I would try lowering maximal number of sessions / connections for the solution that you use. – Jeffryjeffy

Answer 1

I would try the following solution.

If you mount the volume to your docker like so:

docker run -v /mnt/shared:/shared my-image

I would create an intermediate directory /mnt/base/shared and mount it to docker like so:

docker run -v /mnt/base/shared:/base/shared my-image

and I will also adjust my code to refer to the new path or creating a link from /base/shared to /shared inside the container

Explanation:

The problem is that the mounted directory /mnt/shared is probably deleted on host machine, when there is a disconnection and a new directory is created after connection is back. But, the container started running with directory mapping for the old directory which was deleted. By creating an intermediate directory and mapping to it instead you avoid this mapping issue.

Aquaplane

Do you have an explanation for why this works? – Pomerleau

Answer 2

Another solution that might work is to mount the directory using bind-propagation=shared

e.g:

--mount type=bind,source=/mnt/shared,target=/shared,bind-propagation=shared

See docker docs explaining bind-propogation

Aquaplane

Answer 3

The above answers separately didn't exactly work for me. I had to do a combination of the two.

My files were located at /mnt/union. The container expected the files internally to be at /mnt/union.

This is my Docker compose before the change:

services:
    fuse:
        volumes:
            - /mnt/union:/mnt/union:ro
        restart: always
        privileged: true
        image: user/container:tag

A few notes about this:

1. Privileged is basically required to make fuse mounts work in Docker, otherwise you have to deal with app security, and devices and other things. It's definitely not recommended to do it this way, but it works.
2. This is missing a lot of information, but the important thing to notice is how I am mounting my fuse mount directly.

This is what it looks like now:

services:
    fuse:
        volumes:
            - type: bind
              bind:
                  propagation: rshared
              read_only: true
              source: /mnt
              target: /mnt
        restart: always
        privileged: true
        image: user/container:tag

1. Notice how I did everything long form, it's definitely not required, but I found it easier than the short form, '/mnt:/mnt:ro,rshared' (which does the same thing). If you want write access '/mnt:/mnt:rshared' (or remove the 'ready_only: true' line).
2. Notice the propagation, it is set to rshared, which means that any mounts underneath the source will also be propagated to the container. This is really important as /mnt/union is mounted underneath, and I want the container to see it. This also work with multiple mounts as well. I have other mounts at /mnt/shared, which I have other containers access.
3. The source and target has changed. Look, it went up one directory. This is required, as going up one directory maps that directory internally, rather than trying to map the mount. This is kind of what the accepted answer does, but doesn't exactly explain it well. The reason this works is because Docker only thinks about the directory you give it. If that directory disappears at any point, Docker can't do anything anymore. Instead, we are telling Docker about a folder we know won't change ever, and so Docker maps it. It's hard for me to explain, but just think of it like DNS.

I looked long and hard for this solution, so I hope it comes in handy for others. It's extremely simple, but is not something wildly obvious or logical, which makes it hard to find.

Gauleiter