docker push fails : manifest invalid

Question

When trying to do docker push <IMAGE> to a private docker image registry, getting an error saying manifest invalid

along with the error, it's giving some [DEPRECATION NOTICE], which might be the cause of the failure as well.

Note: After getting this error and failing the docker push command. (in ci/cd it's failing the job, I'm guessing its throwing exit 1 ), when I check back in the registry, the image is present.

In the same docker registry, there is other images present as well. and they also are going through the same CI/CD pipeline. but for them, there is no issue detected. is there any possibility that the issue is originating from the Dockerfile? (Although it used to work previously, with the same Dockerfile)

I have doubled check that, the image version (e.g: <IMAGE_NAME>:0.0.1-5e90df92ed140-development) I'm pushing is not present in the registry.

I have also referred to multiple questions and issues regarding this and crossed checked the

• The user has permission to push & pull images
• manifests having the same numbers of fsLayers as history

Note: We are using JFrog Artifactory version to 6.12.2

Docker Version (In my local machine, where it fails as well. with the same message):

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Error:

596280599f68: Layer already exists
5d6cbe0dbcf9: Pushed 
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the xxxxxxx.docker.repositories.xxxxxx.com registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
manifest invalid: manifest invalid

Suffice

What version of docker client, engine, and registry? How are you building images? – Rowlett

@Rowlett updated the question with the docker engine and client version. not sure about the registry version though. when I check manifest.json there is schemaVersion:1 – Suffice

@Rowlett I'm right now building using docker build comand, also the same facing when building using Jenkins. – Suffice

Answer 1

Ran into the same problem. The issue was that repo does not let you over-write images tags. I changed the tag to a new major version.

Anomalism

are you saying that the image tag already exists in the repo already and when trying to push? if that is so, then I have checked multiple times, it does not exist till the docker push fails. – Suffice

Answer 2

Ran into the samish problem with 6.16. The issue was authorization related. I had to add the Delete / Override Repository Action as well to the build agent user.

The build agent only had write permissions, and the second time it built the image with the same tag, it could not overwrite the manifest! This made the client to roll back to schema1 for some reason.

Figured it out through the System Logs of Artifactory.

Autobahn

Hey, thanks for the info, but for me, the user I'm using to push, have that permission. – Suffice

Hi, did you find a solution to this issue? – Fortress

@Fortress till now no, – Suffice

Answer 3

This can happen after your system (e.g. Ubuntu) upgrades docker-ce-cli to a newer version e.g. from 5:19.03.15~3-0~ubuntu-bionic to 5:20.10.3~3-0~ubuntu-bionic

If upgrading your docker-registry to a newer version is not an option at the moment, then you might try to downgrade your docker-ce-cli to some previous version. To see the list of available versions:

sudo apt-cache policy docker-ce

sudo apt-cache policy docker-ce-cli

To downgrade to previous version run:

sudo apt install docker-ce=5:19.03.15~3-0~ubuntu-bionic docker-ce-cli=5:19.03.15~3-0~ubuntu-bionic

Same method can be applied to downgrade docker-ce package.

Check version:

docker --version

Then try to push again:

docker push your-docker-registry.com:5000/myprojects/my-postgresql-image:latest

Please note that keeping your software up-to-date might improve security of your system(s)!

Stockpile

Answer 4

I had the same condition as user3749775: The problem was that the image already existed. I just want to add the solution: I added this and then I was able to overwrite images:

docker login -u <user> -p <password> <address of my artifactory>

Strangely, I was able to push the image without logging in, as long as the image did not exist yet.

I would have added this as a comment under the original response. But I don't have reputation to add comments on responses.

Diskson

Answer 5

Publishing my artifacts into artifactory failed with the same reason manifest invalid: manifest invalid due to the previous docker image existed. Removing the docker image resolved the issue.

Error

The push refers to repository [xxxxx.com/foo]
83850d400bdd: Pushed
309acb279757: Pushed
ebf469e844a6: Layer already exists
94c94d3b2e6b: Layer already exists
manifest invalid: manifest invalid
🚨 Error: The command exited with status 1

To remove the docker image

docker image rm <name>

Provision

Answer 6

In our case, it looks as if a manifest file got corrupted somehow,
and the solution was:

1. Delete the images with the tags that failed to upload
   (this includes their manifest files, of course)
2. Manually run the Garbage Collection*
   (go to: Administration > Artifactory > Advanced > Maintenance)

---

*in Artifactory 7.x this requires pressing the Garbage-Collection's Run Now button some 20 times... - see:
Why does removing/deleting old artifacts is not affecting the Artifactory disk space usage?

Inherited

Answer 7

I got these warnings on our project's GitLab Container Registry after pushing some container images built by the Docker CLI that comes with Docker Desktop. Meanwhile, my company policy requires us to use Rancher Desktop instead of Docker Desktop in our workstations.

I have fixed these warnings with 0-size container images by building new container images with the Docker CLI (moby/dockerd) of Rancher Desktop.

Kymberlykymograph