gitlab job failed - image pull failed

Question

I am trying to do docker scan by using Trivy and integrating it in GitLab the pipeline is passed. However the job is failed, not sure why the job is failed. the docker image is valid. updated new error after enabled shared runner

gitlab.yml

Trivy_container_scanning:
  stage: test
  image: docker:stable-git
  variables:
    # Override the GIT_STRATEGY variable in your `.gitlab-ci.yml` file and set it to `fetch` if you want to provide a `clair-whitelist.yml`
    # file. See https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html#overriding-the-container-scanning-template
    # for details
    GIT_STRATEGY: none
    IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
  allow_failure: true
  before_script:
    - export TRIVY_VERSION=${TRIVY_VERSION:-v0.20.0}
    - apk add --no-cache curl docker-cli
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin ${TRIVY_VERSION}
    - curl -sSL -o /tmp/trivy-gitlab.tpl https://github.com/aquasecurity/trivy/raw/${TRIVY_VERSION}/contrib/gitlab.tpl
  script:
    - trivy --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "@/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $IMAGE
    #- ./trivy — exit-code 0 — severity HIGH — no-progress — auto-refresh trivy-ci-test
    #- ./trivy — exit-code 1 — severity CRITICAL — no-progress — auto-refresh trivy-ci-test

  cache:
    paths:
      - .trivycache/
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json
  dependencies: []
  only:
    refs:
      - branches

Dockerfile

FROM composer:1.7.2
RUN git clone https://github.com/aquasecurity/trivy-ci-test.git && cd trivy-ci-test && rm Cargo.lock && rm Pipfile.lock
CMD apk add — no-cache mysql-client
ENTRYPOINT [“mysql”]

job error:

Running with gitlab-runner 13.2.4 (264446b2)
  on gitlab-runner-gitlab-runner-76f48bbd84-8sc2l GCJviaG2
Preparing the "kubernetes" executor
30:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image docker:stable-git ...
Preparing environment
30:18
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending
ERROR: Job failed (system failure): prepare environment: image pull failed: Back-off pulling image "docker:stable-git". Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

another error:

Running with gitlab-runner 13.2.4 (264446b2)
  on gitlab-runner-gitlab-runner-76f48bbd84-8sc2l GCJviaG2
Preparing the "kubernetes" executor
30:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image $CI_REGISTRY/devops/docker-alpine-sdk:19.03.15 ...
Preparing environment
30:03
Waiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0t7plc to be running, status is Pending
ERROR: Job failed (system failure): prepare environment: image pull failed: Failed to apply default image tag "/devops/docker-alpine-sdk:19.03.15": couldn't parse image reference "/devops/docker-alpine-sdk:19.03.15": invalid reference format. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Seriate

Answer 1

This is followed by gitlab-org/gitlab-runner issue 27664

• either a GitLab infrastructure issue
• or (comment from Bruce Lau)

After some trial and errors, me and our team figured out the issue is due to the runner failed to use service account secret to pull images.
In order to solve this issue, we use a custom config which specify image_pull_secrets in .dockercfg format in order to pull images successfully.

Content of runner-custom-config-map:

kind: ConfigMap
apiVersion: v1
metadata:
  name: runner-custom-config-map
  namespace: runner-namespace
data:
  config.toml: |-
    [[runners]]
      [runners.kubernetes]
        image_pull_secrets = ["secret_to_docker_cfg_file_with_sa_token"]

Used in the runner operator spec:

spec:
  concurrent: 1
  config: runner-custom-config-map
  gitlabUrl: 'https://example.gitlab.com'
  imagePullPolicy: Always
  serviceaccount: kubernetes-service-account
  token: gitlab-runner-registration-secret

With secret_to_docker_cfg_file_with_sa_token:

kind: Secret
apiVersion: v1
  name: secret_to_docker_cfg_file_with_sa_token
  namespace: plt-gitlab-runners
data:
  .dockercfg: >-
    __docker_cfg_file_with_pull_token__
type: kubernetes.io/dockercfg

---

June 2022: the issue is closed by MR 3399 for GitLab 15.0:
"Check serviceaccount and imagepullsecret availability before creating pod"

To prevent the pod creation when needed resources are not available.

---

May 2023: issue 27664 adds (Gabriel Díaz):

Finally discovered the reason behind this after a lot of time.
The runner just caches the imagepullsecrets during runner registration. No matter what, even when you restart the runner deployment or kill the runner pod.
The only option you can do is to completely unregister the runner, provide a new token, and register again:

If you do so, the runner immediately reads the new imagepullsecrets changes immediately.
Is this behavior documented? It was making me crazy. I would expect the runner to read the secrets value of the imagepullsecrets on every pipeline run, but it only does read it during registration.

Hence, issue 31066: "GitLab runner for Kubernetes only reads imagepullsecrets during runner registration, not on pod /deployment restart and not on new pipeline runs".

Desjardins

i am new to gitlab, where can I modify runner? in gitlab? – Seriate

@user12158726 By following docs.gitlab.com/runner/configuration/… – Desjardins

okay i guess something wrong with runner. saw an active shared runner but warning message "This project does not belong to a group and cannot make use of group runners." – Seriate

@user12158726 OK. This must be linked to the scope of the runner user: docs.gitlab.com/ee/ci/runners/runners_scope.html – Desjardins

i re-setup new error posted at main post about "ERROR: Job failed (system failure): prepare environment: image pull failed: Failed to apply default image tag "/devops/docker-alpine-sdk:19.03.15": couldn't parse image reference "/devops/docker-alpine-sdk:19.03.15": invalid reference format. Check docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information" – Seriate

@user12158726 Maybe an error in gitlab-ci.yml? As in gitlab.com/gitlab-org/gitlab/-/issues/199796 – Desjardins

Answer 2

The root cause is actually no variable being setup in gitlab cicd variables. After defined the registry credentials, all works.

Seriate

Good catch. Could you add, as an illustration, to your answer an example/extract of your gitlab-ci.yml, with the additional part needed to solve the issue? – Desjardins

Here the project variable settings docs.gitlab.com/ee/ci/variables/… – Seriate

OK, and out of those variables, which ones did you need to solve your problem? – Desjardins

Answer 3

I experienced this issue when running a job in GitLab CI.

I was encountering the error below:

Running with gitlab-runner 15.9.0~beta.115.g598a7c91 (598a7c91)
  on blue-1.shared.runners-manager.gitlab.com/default j1aLDqxS, system ID: s_b437a71a38f9
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Resolving secrets
00:00
Preparing the "docker+machine" executor
Using Docker executor with image registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest ...
WARNING: Failed to pull image with policy "always": error pulling image configuration: unknown blob (manager.go:237:0s)
ERROR: Job failed: failed to pull image "registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest" with specified policies [always]: error pulling image configuration: unknown blob (manager.go:237:0s)

I tried to run the job multiple times over a short period of time but it is failed.

How I solved it:

The issue was from GitLab. The GitLab AWS base image registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest was unavailable.

All I had to do was to wait for about 1 to 2 hours and then try building it. This time the pipeline ran perfectly.

Ilailaire