Docker pull fails with unexpected EOF

Question

I'm trying to pull an image from a private registry but it fails with unexpected EOF.

be8ec4e48d7f: Already exists 
33b8b485aff0: Already exists 
d887158cc58c: Already exists 
05895bb28c18: Already exists 
3717254b824f: Already exists 
5d1752e32f1f: Already exists 
98554cf2e2ec: Already exists 
3a83ccd2f4ee: Already exists 
c793dcd65f37: Already exists 
c4412ad3121f: Already exists 
eba3bc56bff9: Already exists 
2b1b6e815dce: Already exists 
dda735bf3557: Already exists 
dbece9223ffc: Already exists 
49533680b25f: Already exists 
8d332721c923: Already exists 
d40c3e8ecbdb: Already exists 
75f43ec617dd: Already exists 
225f75c25e6b: Already exists 
81811c4b9e22: Already exists 
409197acab0f: Already exists 
9b5199518afc: Already exists 
b7040ab58553: Already exists 
b6e55490ca80: Already exists 
e04c7210075d: Already exists 
68db66ab7dda: Already exists 
bbe68713195e: Already exists 
1d7c9599b03c: Already exists 
c7472ba4bdbd: Already exists 
3c4082628c2e: Already exists 
50f7a489c209: Already exists 
aaf98f685aa2: Already exists 
8cf31fcd419c: Already exists 
a80092fe6016: Already exists 
9d90bdef5603: Already exists 
9d13d2b62b19: Already exists 
cab2bcedcfdf: Already exists 
41670cbb355b: Already exists 
4036f94db6f4: Retrying in 19 seconds 
42f95b51f0f7: Download complete 
91f3ac158888: Download complete 
df09d420f619: Download complete 
7ca46f747969: Download complete 
34e5e5eb8b2f: Download complete 
ce8b1ad11171: Download complete 
4b124d6694bd: Download complete 
a71b50d0f70c: Download complete 
237b8823183b: Download complete 
4036f94db6f4: Downloading  12.41MB/12.41MB
64858da6aaea: Download complete 
3e88dcb59e3e: Download complete 
aa92021cb41d: Download complete 
0569c05fcc65: Download complete 
63ee9c05e34b: Download complete 
48ac0999fcdb: Download complete 
e10d1975849c: Download complete 
611db2146c6f: Download complete 
7ad4e600c6c6: Download complete 
4261ad6f88e3: Download complete 
c20e2f82fd11: Download complete 
49f62c0b1913: Download complete 
cb37060da14d: Download complete 
f19eda2c4fb2: Download complete 
e600fca97576: Download complete 
508947b05054: Download complete 
652c4860c2f3: Download complete 
358d27eb7aaa: Download complete 
27cfd1cb1501: Download complete 
673d626b28c0: Download complete 
20d0b16ebf52: Download complete 
0cfff8a600f6: Download complete 
00065e08cdfc: Download complete 
bd51e67b9159: Download complete 
7e8e6548bdec: Download complete 
1dea31859db5: Download complete 
f058efbc49a4: Download complete 
3e10bb0abdd8: Download complete 
617d42f04950: Download complete 
798390048c18: Download complete 
2b389764a032: Download complete 
002f5c6ccc90: Download complete 
unexpected EOF

I've tried deleting the layer like suggested here Docker pull “unexpected EOF” but the same error persists even I build the image with a new tag and --no-cache.

My workflow is the following:

• Image is built using Jenkins.
• Image is pushed to private Docker registry.
• Image is pulled in Kubernetes node.

How can I further debug this? Or forcefully delete the layer, and images that depend on it.

Planetoid

Which Kubernetes Network Provider are you using? Flannel, Calico, Canal? ... Might be related to the MTU size – Rn

Answer 1

I had an image I couldn't pull:

$ docker pull registry.com/IMAGE:TAG
TAG: Pulling from IMAGE
97518928ae5f: Already exists
784cd1fd612b: Pull complete
0ec5d186b713: Pull complete
98dc27ad6276: Pull complete
94d536c3c847: Downloading [===============================================>   ]  116.2MB/123.2MB
56a018b1cd83: Download complete
89370282bef7: Download complete
eaf17779ae5d: Downloading [==================================================>]   2.59MB/2.59MB
e031dab02d72: Download complete
unexpected EOF

That's the way it looked like at the end. While pulling the image it looked like so:

TAG: Pulling from IMAGE
97518928ae5f: Already exists
784cd1fd612b: Downloading [========>                                          ]  6.127MB/34.81MB
0ec5d186b713: Download complete
98dc27ad6276: Download complete
94d536c3c847: Downloading [>                                                  ]  540.2kB/123.2MB
56a018b1cd83: Download complete
89370282bef7: Download complete
eaf17779ae5d: Retrying in 4 seconds
e031dab02d72: Waiting

As such, eaf17779ae5d is the culprit. Let's find out the full hash:

$ curl -sSL https://registry.com/v2/IMAGE/manifests/TAG | jq -r '.fsLayers[] | .blobSum'
...
sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d
sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca
...

And try to confirm if we can download it:

$ curl -sSLv https://registry.com/v2/IMAGE/blobs/sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca -o /dev/null
...
> GET /v2/IMAGE/blobs/sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca HTTP/2
> Host: registry.com
> authorization: Basic ...
> user-agent: curl/7.79.1
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host registry.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

$ ssh [email protected] docker exec REGISTRY_CONTAINER_ID ls /var/lib/registry/docker/registry/v2/blobs/sha256/ea/eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca
ls: /var/lib/registry/docker/registry/v2/blobs/sha256/ea/eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca: No such file or directory

We can't, as such let's delete the layer:

$ curl -sS -X DELETE https://registry.com/v2/IMAGE/blobs/sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca

Then push the image again:

$ docker push registry.com/IMAGE:TAG
The push refers to repository [registry.com/IMAGE]
9be3d32c4bdc: Preparing
a17a1099bdbb: Preparing
896990f929fd: Preparing
74cfeaa3d1d8: Preparing
dd2b13d45ffc: Preparing
ff64ee97d76a: Preparing
480f61641fa1: Preparing
b3eaed7a085d: Preparing
1a058d5342cc: Preparing
ff64ee97d76a: Waiting
b3eaed7a085d: Waiting
480f61641fa1: Waiting
1a058d5342cc: Waiting
896990f929fd: Layer already exists
dd2b13d45ffc: Layer already exists
74cfeaa3d1d8: Layer already exists
9be3d32c4bdc: Layer already exists
480f61641fa1: Layer already exists
1a058d5342cc: Layer already exists
ff64ee97d76a: Layer already exists
b3eaed7a085d: Layer already exists
a17a1099bdbb: Pushed
TAG: digest: sha256:5b784cb8080660d7af7222a6b5b0630faa6942b39f0c533354500e946bf92c7c size: 2201

Do note the a17a1099bdbb: Pushed line. It means that the layer was uploaded (pushed). And also do note that the hashes differ (the local one and the one from the registry). During pull it displays registry hashes, during push local hashes. The local hash may be just a random string, the remote hash is most likely a hash of the data. Or maybe they hash different things or differently. To find out what corresponds to what you can obtain a manifest by a digest:

$ curl -sS https://registry.com/v2/IMAGE/manifests/sha256:5b784cb8080660d7af7222a6b5b0630faa6942b39f0c533354500e946bf92c7c | jq
{
  ...
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7810,
    "digest": "sha256:2b71efc18e8f74f9c9ae1a83f5d2a04b09e42e6a4e3a67e381ff68fea1b9a964"
  },
  "layers": [
    ...
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2590453,
      "digest": "sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 553,
      "digest": "sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d"
    }
  ]
}

And then obtain the config (see .config.digest in the output of the previous command):

$ curl -sS https://registry.com/v2/IMAGE/blobs/sha256:2b71efc18e8f74f9c9ae1a83f5d2a04b09e42e6a4e3a67e381ff68fea1b9a964 | jq
{
  ...
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      ...
      "sha256:a17a1099bdbb31acadd8c8ef1da7ab734e14405ed3072383a1699708af512de4",
      "sha256:9be3d32c4bdc87b8ee61ffc775f050bea4fbaf8a32ef4ddb6f6e881a92a0c4a3"
    ]
  }
}

sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca is second to last, and so is sha256:a17a1099bdbb31acadd8c8ef1da7ab734e14405ed3072383a1699708af512de4. That's the match.

Anyways, if you delete just one layer, there should be only one "pushed" line.

Let's confirm if it works:

$ docker pull registry.com/IMAGE:TAG
TAG: Pulling from IMAGE
97518928ae5f: Already exists
784cd1fd612b: Pull complete
0ec5d186b713: Pull complete
98dc27ad6276: Pull complete
94d536c3c847: Pull complete
56a018b1cd83: Pull complete
89370282bef7: Pull complete
eaf17779ae5d: Pull complete
e031dab02d72: Pull complete
Digest: sha256:5b784cb8080660d7af7222a6b5b0630faa6942b39f0c533354500e946bf92c7c
Status: Downloaded newer image for registry.com/IMAGE:TAG
registry.com/IMAGE:TAG

All done. And although my answer basically repeats the one you referenced, I believe I provided more details, and here are a couple of links just in case:

Missing image layer in a docker registry
Examining images and docker registry

Also, if it doesn't help, you might want to run garbage collection:

$ docker exec REGISTRY_CONTAINER_ID registry garbage-collect -m /etc/docker/registry/config.yml

Coz

You'll probably want to add "--delete-untagged" to the garbage-collect command. Also, you'll want to restart the registry afterwards. After that, try re-pushing the image to the registry. You'll likely notice that one or two layers will be pushed as new. – Zebadiah

Answer 2

Just had the same problem and resolved it by:

1. Re-creating registry container (re-start might have been enough)
2. Re-pushing image to registry

Interestingly, after re-creating the registry I got a different error "Unknown blob" which turned out to mean a layer was missing. So a re-push solved the issue.

Sd