Docker pull “unexpected EOF”

Question

I faced an issue with docker. The scenario is like this: we use Codebuild+Packer+docker to create AMI, which is used in deploy. During this step we pull image from artifactory and all pass fine except pulling one of the layers which is > 1Gb. After several retries it fails with error: Download failed, retrying: unknown blob and then “unexpected EOF”. Have you ever faced such issue? Any comments or advices are highly appreciated.

Coda

Could be a proxy or load balancer issue, it might be blocking large file downloads. Can you connect directly to artifactory without going through a proxy? Maybe by sshing directly to the artifactory server, or a server on the same network? – Womanhood

I've had problems like this in the past with very large individual layers. I don't have specific advice other than generic tips for building smaller images, like "don't apt-get install build-essential". – Anomaly

What version of docker? – Peeper

@DavidMaze, Yeah indeed that actually how we solved this issue.Thanks! – Coda

If your issue is not with the connection or big files or artifactory, but with the docker registry, check out this answer. – Meantime

I too faced the same issue, it is because of a space issue in our Nginx proxy machine. After we performed the cleanup, it got worked. – Woods

Answer 1

This was mainly because of weak network ( as I was using mobile hotspot )

configured the docker daemon to reduce the tcp packets

$ dockerd --max-concurrent-downloads <int>  

here <int> suggests the number of docker pull layers you want to download concurrently.
default is 3

in mycase i had set to 2

$ dockerd --max-concurrent-downloads 2 &>/dev/null  

downside of doing this is sacrificing your precious time :)
takes time like hell

Cantara

Suit for the users from CN. Meet the same error. Resolve the problem with more strong and stabily network. "This was mainly because of weak network" – Purebred

Answer 2

With Docker Desktop on Windows, could not find the dockerd command, then added the below entry in the daemon.json file and restarted the docker service.

"max-concurrent-downloads": 1

You will find this file at path- C:\Users\<user-name>\.docker\daemon.json.

This will pull the layers in a sequential manner hence it will take time, but yes, this is an alternative solution to download the large image file over the weak network connection.

Hundley

This is a valid answer for Windows users, who do not have the dockerd command. – Amphigory

Answer 3

I had this problem with a very small layer that was corrupted or broken in the registry V2 for some unknown reason. docker pull failed with "unexpected EOF" after retrying the layer (identified as "1f8fd317c5a4" in this case).

Rebuilding the image from source and trying to docker push said "layer already exists", not fixing the issue.

I was able to delete the offending layer using curl like so;

curl -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -sk "https://registry.local/v2/image-name/manifests/1033-develop-7e414712"

(substitute your registry for "registry.local", your image name for "image-name", and your image tag or "latest" for "1033-develop-7e414712".)

Get the complete sha256 digest for layer 1f8fd317c5a4 from the JSON output, and use it in next command:

curl -k -X DELETE "https://registry.local/v2/image-name/blobs/sha256:1f8fd317c5a406a75130dacddc02bd09a9abf44e068e2730dd8f5238666bb390"

Now you will be able to docker push registry.local/image-name:1033-develop-7e414712 to upload the layer you deleted, and everything works.

Triazine

Not sure if this method is transferable to artifactory. I didn't see that tag. – Triazine

I tried that method but despite successfully deleting the offending layers and repushing them I still get the same error after trying to pull again.. :/ – Avalokitesvara

@Avalokitesvara Rebuild the image locally as well before pushing it again – Triazine

I've followed your solution and I've deleted some layers but one of them is still present. I'm building images using Jenkins and pushing them to a private Docker registry. Images are built and pushed successfully and fails when I try to pull them. How can I debug this more please? – Medorra

for me that didn't help. I've got "unknown blob" error and was forced to re-build image, and it solved my problem – Objective

Answer 4

Had the same issue due to a bad connection. In the documentation, here is the dockerd command.

For Linux, simply add:

$ dockerd --max-concurrent-downloads 2
$ dockerd --max-download-attempts 10

For windows docker desktop, open settings -> Docker Engine and pop the following in with the numbers best for you. You can see all the options in the docs as above.

Fiden

Answer 5

1. Stop docker service: sudo service docker stop

2. Run docker service with decreasing max-concurrent-downloads to what suits your internet bandwidth (Unfortunately 1 for me) and increasing max-download-attempts: sudo dockerd --max-concurrent-downloads 1 --max-download-attempts 10

PS: I am not a docker expert. But, I believe there is a better way to do it by adding some config whether to the registry or your docker client.

Decorator

Answer 6

Problem: Unable to pull docker image its giving retrying to pull image and EOF Solution: Update docker software then try to pull image it resolves the issue.

Afterburner

Answer 7

This does not match the situation described by OP perfectly, but I'll post it here for future reference. Docker Desktop 4.15.0 introduced a bug which caused a similar issue for me. Depending on the Docker Desktop version and command used, one of these errors would pop up:

% docker pull alpine:3.7
Error response from daemon: Get "https://registry-1.docker.io/v2/": read tcp 192.168.65.4:55694->192.168.65.5:3128: read: connection reset by peer

% docker-compose up --build
// Some stuff
 => ERROR [container_name internal] load metadata for docker.io/library/alpine:3.7                                                                           0.0s
------
 > [container_name internal] load metadata for docker.io/library/alpine:3.7:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/3.7": unexpected EOF

% docker pull alpine:3.7
Error response from daemon: Get "https://registry-1.docker.io/v2/": unexpected EOF

The solution was to uninstall Docker Desktop and install an older version. I'm posting this here since a lot of guides and instructions recommend updating Docker Desktop to its newest version, but in my case that is exactly what caused the issue. Of course this bug will most likely be patched in a newer version at some point, I have notified Docker support about it.

Edit: It seems that there is a GitHub topic for my issue.

Moxa

Answer 8

After so many trails, here is what worked for me

1. Deleted the image in container registry repo.
2. Rebuild image with --no-cache option and pushed again to container registry repo. docker build --no-cache -t

Spade