How to delete volumes in swarm cluster?

Question

I have a swarm cluster with one manager and another normal node , when I create a swarm service I am creating with mount type ,mount source and mount target . It creates the volume with the same name in both manger and node and starts the container and my service is up.

When I release the service the volume created along with the service was not deleted, this is still fine.

The problem I am facing is when I delete the volume with the same endpoint it's only deleting the volume in swarm manager, the volume created in the node while creating the service still exists.

I want the manager to delete all the volumes which is created along with the swarm service. Is there a way ??

Batts

What's the version of docker and is this classic swarm or swarm mode? Given the time difference, I'm guessing the OP and bounty are looking at different things. – Parke

Bounty looks at Swarm Mode. Docker - >=17.x – Ashcroft

Answer 1

After so much of analysis here is the theory.

if you are instructing swarm to create the service with volume, Swarm is only performing actions on creating the services inside the cluster i.e on the multiple nodes yes when you send the volume details yes it does creates the volume as well but while releasing the service it fails to check in the worker nodes for existence of volume while releasing Its the bug in docker

I have raised the bug in docker for it.

As of now there is no other way than manually releasing the volume from worker nodes after releasing the swarm service .

Batts

Do you have the URL link for that issue that yo have raised? – Coltish

You should look after this issue github.com/moby/moby/issues/29158#issuecomment-663793782 – Barbarese

Answer 2

As far as I know a volume is only created on nodes where a container is created. Is it possible that your service fails to start on one node, ends up on the other and somehow swarm doesn't clean up? If thats the case write an issue in github.

Update (from comments):

According to the docker service create documentation:

A named volume is a mechanism for decoupling persistent data needed by your container from the image used to create the container and from the host machine. Named volumes are created and managed by Docker, and a named volume persists even when no container is currently using it. Data in named volumes can be shared between a container and the host machine, as well as between multiple containers. Docker uses a volume driver to create, manage, and mount volumes. You can back up or restore volumes using Docker commands.

So if you are using named volumes the correct answer would be why are they removed on the manager and where they ever created there?

Cynthia

Good perspective of thinking for troubleshooting. I though the same . when you instruct the swarm to create the service with replica 10, It splits and decides how many containers has to run on each node. it only creates the volume in node if there is any service gets allocated as u said. even everything goes well and the containers are successfully running too it fails to release the volumes in the worker node , i have verified manually many number of times. – Batts

And on the manager they are removed? If you have a named volume its supposed to persist right? "A named volume is a mechanism for decoupling persistent data needed by your container from the image used to create the container and from the host machine. Named volumes are created and managed by Docker, and a named volume persists even when no container is currently using it. Data in named volumes can be shared between a container and the host machine, as well as between multiple containers. Docker uses a volume driver to create, manage, and mount volumes." – Cynthia

you are absolutely right herm. i thought in that perspective too , from docker's perspective they identify volume , network and container as a separate service. the volume and the network may be shared across the containers.I agree , in that case why do they delete in manager node is the billion dollar question ? if they are deleting they should delete across the cluster else they should not delete in the manager too is my argument on this. – Batts

Answer 3

There is a way to remove volumes in other machines of the docker swarm cluster.

Considering that the volume name is volume-x, you can run this command bellow:

docker service create -d \
    --name cleaner \
    --restart-condition none \
    --mode global \
    --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock \
    docker \
    sh -c "docker volume rm volume-x"

It will create a global service and run docker volume rm volume-x in each host inside the cluster

To follow the logs:

docker service logs --follow cleaner

Jaime

Answer 4

While waiting from moby/moby issue 29158, Andrey Mikhailov recommends in that same issue:

In a meanwhile, here is the easy way to manage dangling volumes removal across the swarm cluster:

version: '3.9'

services:
  swarm-volumes-cleanup:
    image: "docker:26.1.4-cli"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      NODE_ID: "{{.Node.ID}}"
    deploy:
      mode: global-job
      restart_policy:
        condition: none
    entrypoint:
      - sh
      - -c
      - |
        set -xe
        docker volume ls --filter name="${FILTER_NAME}" --filter "dangling=true" -q | xargs -r docker volume rm
        echo Done - $$NODE_ID

FILTER_NAME="test-infra_" docker stack deploy --detach=false  --compose-file tasks/swarm-volumes-cleanup.yml maintenance

Coltish