Gitlab pipeline failed : ERROR: Preparation failed: Error response from daemon: toomanyrequests

Question

I have Harbor local docker registry and all needed images are there and connected GitLab to the Harbor and all the images are received from the Harbor but after November 2, Docker put a limit on the number of pulls and it seems dind service pulls from Docker hub. Is it possible to use dind service to pull from Harbor?

Pipeline output:

Running with gitlab-runner 12.10.1 (ce065b93)
  on docker_runner_7 WykGNjC6
Preparing the "docker" executor
30:20
Using Docker executor with image **harbor**.XXX.XXXX.net/library/docker_maven_jvm14 ...
Starting service docker:**dind** ...
**Pulling docker image docker:dind** ...
**ERROR**: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:2s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:4s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)

Ansley

Answer 1

Another way: If you don't want to add daemon.json, you can do this:

1. Pull docker-dind from docker hub

docker pull docker:stable-dind

2. Login to harbor

docker login harbor.XXX.com

3. Tag image to harbor

docker tag docker:stable-dind harbor.XXX.com/library/docker:stable-dind

4. Push to harbor

docker push harbor.XXX.com/library/docker:stable-dind

5. Go to the .gitlab-ci.yml

6. Instead of

services:
    - docker:dind

write:

services:
    
    - name: harbor.XXX.com/library/docker:stable-dind
      alias: docker

My .gitlab-ci.yml :

stages:
  - build_and_push

Build:
  image: ${DOCKER_REGISTRY}/library/docker:ci_tools
  stage: build_and_push
  tags:
    - dind
  services:
    - name: ${DOCKER_REGISTRY}/library/docker:stable-dind
      alias: docker
  script:
    - docker login -u $DOCKER_REGISTRY_USERNAME -p $DOCKER_REGISTRY_PASSWORD $DOCKER_REGISTRY
    - make build test release REGISTRY=${DOCKER_REGISTRY}/library/ TELEGRAF_DOWNLOAD_URL="https://storage.XXX.com/ops/packages/telegraf-1.15.3_linux_amd64.tar.gz" TELEGRAF_SHA256="85a1ee372fb06921d09a345641bba5f3488d2db59a3fafa06f3f8c876523801d"

Ansley

I had the same problem, instead of using Harbor, I have used GitLab Registry. Created a new repo named Docker and pushing there. Everything was ok, thanks for the procedure you've mentioned. – Sleet

Answer 2

I can't found the solution for Gitlab but you can tell the docker to ignore the docker hub registry and go to the local registry.

Add daemon.json in /etc/docker/daemon.json , if doesn't exist you can simply add in the path.

daemon.json

{
  "registry-mirrors": ["https://harbor.XXX.com"]
}

sudo systemctl restart docker

Ansley

Answer 3

I too faced the same issue while deploying some micro-services to kube cluster, here is a blog that I wrote that provides a workaround to optimize the deployment workflow: https://mailazy.com/blog/optimize-docker-pull-gitlab-pipelines/

Maximilianus