Docker daemon fails to process logs

Question

Docker daemon is unable to parse the json log and throws an unexpected EOF error. We are investigating the root cause of the issue.

Environment:

• Debian 4.9.189-3+deb9u2
• Docker engine 19.03.5

Docker daemon log:

Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070677515+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=345
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070695689+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=346
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070712630+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=347
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070732299+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=348
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070755016+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=349
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070773699+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=350

Docker daemon configuration

"log-driver": "json-file",
"log-opts": {
  "mode": "non-blocking",
  "max-size": "500m",
  "max-file": "3"
}

Storage space is available on partition where docker writes its logs.

Your insight in the matter is most welcome.

Leman

We ended up disabling docker logs meanwhile and the cpu usage on containerd process returned back to normal. – Leman

You weren't able to figure out a root cause @Mysteryos? – Reaction

@PaulRichter We have not. Logs are still disabled. It was on debian OS. We are thinking of moving to Ubuntu. – Leman

An update on this situation: We found out that a deployed application container was outputting erroneous JSON logs and had to correct the latter. The best way to mitigate this error is to keep tiny docker log files such as max-size = 10m until you find the culprit. – Leman

Answer 1

The aforementioned issue will cause an abnormal spike in CPU usage of the docker (version <= 19) process and constantly stay at that level until you fix the logs.

To fix the logs & cpu usage, you've to truncate all the container logs (since we are unable to determine which container has faulty logs). It works only if your logs are in json format:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Please see Andre's answer for a permanent fix.

Leman

On mac desktop I had to first log into the vm with docker run -it --rm --privileged --pid=host justincormack/nsenter1 Then run truncate -s 0 /var/lib/docker/containers/*/*-json.log – Salleysalli

Answer 2

Issue=Docker Engine Error json-file: fix sporadic unexpected EOF errors

I had spent some time researching and trying to find errors in the docker logs.

There is a bug in docker engine 20.10.5 which is the version running in docker. A Github PR fixes this. https://github.com/moby/moby/pull/42104

I found the error while running

$ journalctl -u docker.service
level=warning msg="got error while decoding json" error="unexpected EOF" retries=19999

Docker driver for logs has a bug and it wont handle some log formatting. The bug then floods the system's log with retries messages. This situation generates a very annoying CPU load.

Docker engine latest is 20.10.7. The issue has been fixed in release Docker engine 20.10.6 https://docs.docker.com/engine/release-notes/#20106

Sophronia