rancher-agent fails with "Failed to find container id:\n0::/\n"

Question

I have had my docker host added to Rancher since a long time ago and everything has been working just fine for months. Suddenly, a few days ago, my docker host was marked as "Disconnected" in Rancher. When I check the status of the rancher-agent container I can see that it is restarting all the time:

•100% ➜ sudo docker ps -f name=rancher
CONTAINER ID   IMAGE                   COMMAND         CREATED        STATUS                          PORTS     NAMES
0a12a18ca52c   rancher/agent:v1.2.11   "/run.sh run"   21 hours ago   Restarting (1) 54 seconds ago             rancher-agent

In the log I see this:

•100% ➜ sudo docker container logs 0a12a18ca52c                                                                                                                                                                                                                                                   
time="2021-06-29T09:13:27Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:28Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:29Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:31Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:32Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:35Z" level=fatal msg="Failed to find container id:\n0::/\n" 
time="2021-06-29T09:13:39Z" level=fatal msg="Failed to find container id:\n0::/\n" 

I have tried searching the web for this but found nothing of interest. I have tried recreating the container. I have tried removing everything in /var/lib/rancher and recreating the container. I have even tried to remove my environment in Rancher, stopped docker, removed all the docker data (data-root) on the docker host, again removed the files related to the rancher-agent and recreated the rancher-agent. Every time I recreate the rancher-agent I have used the command that the Rancher GUI gives you when you want to add a new host. I always end up with the same errors in the log.

I have the same version of docker installed on the host as it had when the host was last connected to Rancher. I use Rancher 1.6 (cannot change this) and docker 20.10.6 (also tried 20.10.7) on a machine running Manjaro.

Agnate

Answer 1

it is caused by cgroups v2 to make it work again

echo 'GRUB_CMDLINE_LINUX=systemd.unified_cgroup_hierarchy=false' > /etc/default/grub.d/cgroup.cfg
update-grub

Verbid

Sounds promising. I have since given up on Manjaro and stuck to Ubuntu as it just works without any issues. – Agnate

I get /etc/default/grub.d/cgroup.cfg: Permission denied (even with sudo) – Whenever

Just wanna confirm this worked! Thanks! @Whenever you gotta just edit the file in another way then. What worked for me is: sudo nano /etc/default/grub.d/cgroup.cfg then add this line: GRUB_CMDLINE_LINUX=systemd.unified_cgroup_hierarchy=false to save: press CTRL + X then Y then enter to save the file. then reboot the host and it should work! – Chiasmus

Answer 2

TL;DR: I reinstalled my dockerhost with Ubuntu 20.04 and now everything works.

I tried setting up a new virtual machine with the same OS as my host had (Manjaro Linux) and I got the exact same experience and behavior as on the host. My dockerhost was "Disconnected" and the log said "Failed to find container id:...". I made another virtual machine with Ubuntu 20.04 and everything worked as expected there. What's strange is that the docker version was the same. Not sure about containerd though. I did try different combinations of docker and containerd on my host and guest but they always had the same issue.

Agnate

Same problem here. - on a server with Ubuntu 22.04 LTS, it fails and rancher/agent logs are full of "Failed to find container id…". - on a server with on older Ubuntu (20.04 LTS) it works (I use Docker 20.10.12 on both server). Did you find a solution? Does Rancher Agent is not compatible with Ubuntu 22 ?! – Berndt

Answer 3

also iptables must be old ones otherwise it is broken

Verbid

You might convince more people that your answer is helpful if you add an explanation. Punctuation, capitalisatoin, clearer reference of "also" are optional, but might also help. Copmare How to Answer. And you probably need to make more obvious why this is a separate answer that cannot meaningfully be edited in your other answer post here. I can see that you know how to edit it... – Firdausi

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Kaitlin