gitlab-runner update failed with GPG error signatures were invalid

Question

I’m unable to update my Gitlab-runner install due to bad keys being detected. Is this a Gitlab update issue or something gone wrong on my system? Update and install was working without problems in 2023.

root@gitlab-runner:~# apt-get update
Hit:1 http://security.debian.org bookworm-security InRelease
Hit:2 http://deb.debian.org/debian bookworm InRelease
Get:3 https://packages.gitlab.com/runner/gitlab-runner/debian bookworm InRelease [23.3 kB]
Err:3 https://packages.gitlab.com/runner/gitlab-runner/debian bookworm InRelease
  The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <[email protected]>
Fetched 23.3 kB in 1s (21.0 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/runner/gitlab-runner/debian bookworm InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <[email protected]>
W: Failed to fetch https://packages.gitlab.com/runner/gitlab-runner/debian/dists/bookworm/InRelease  The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Many suggest to add gitlab apt gpg key like this

root@gitlab-runner:~# curl -s https://packages.gitlab.com/gpg.key | apt-key add -
OK

Still it does not resolve the issue on Debian 12 and Ubuntu 22. Same error on apt update.

Yeasty

Answer 1

To resolve this situation in 2024, especially on old installs, first we need to remove already added gitlab apt gpg key (EXPKEYSIG 3F01618A51312F3F).

Run the command:

sudo apt-key del "F640 3F65 44A3 8863 DAA0 B6E0 3F01 618A 5131 2F3F"

and run latest gitlab runner install script:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash && sudo apt update

That's it, now you can do apt upgrade.

Update from comment below, if you have the same type of issue with self hosted gitlab-ce, please run this instead:

curl -L "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh" | sudo bash && sudo apt update

More details:

Note that apt-key on Debian 12 is obsolete:

root@gitlab-runner:~# apt-key list
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).

So proper way in general should be to put dearmored gpg signature to /etc/apt/trusted.gpg.d, but its not a gitlab case.

If you look at /etc/apt/sources.list.d/runner_gitlab-runner.list file, you will notice gpg key mentioned directly:

# this file was generated by packages.gitlab.com for
# the repository at https://packages.gitlab.com/runner/gitlab-runner

deb [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/debian/ bookworm main
deb-src [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/debian/ bookworm main

This is the reason, why manually adding gpg key with apt-key does not resolve the issue. Executing install script again, would deploy latest key signature.

Yeasty

This worked for me, but I had to modify it to match my gitlab-ce install curl -L "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh" | sudo bash && sudo apt update – Athamas

Your mentioned script is for gitlab-ce, while I had a problem with gitlab ci runner. Still valuable comment, I will amend the answer. – Yeasty

Answer 2

The GitLab keys expire. The last set expired on March 1, 2024. However, GitLab extended them to Feb 27, 2026. You need to update the keys on your system for the new expiration.

Check out the GitLab documentation here: Update keys after expiry extension

1. Determine if you’re using apt-key or signed-by functionality:

grep 'deb \[signed-by=' /etc/apt/sources.list.d/gitlab_gitlab-?e.list

• If this grep returns any lines, you’re using signed-by functionality. This takes precedence over any apt-key usage.
• If this grep returns no lines, you’re using apt-key functionality.

2. For signed-by, the following script (run as root) updates the public keys for GitLab repositories:

    awk '/deb \[signed-by=/{
        pubkey = $2;
        sub(/\[signed-by=/, "", pubkey);
        sub(/\]$/, "", pubkey);
        print pubkey
    }' /etc/apt/sources.list.d/gitlab_gitlab-?e.list | \
    while read line; do
        curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
    done

3. For apt-key, the following script (run as root) updates the public keys for GitLab repositories:

    apt-key del 3F01618A51312F3F
    curl -s "https://packages.gitlab.com/gpg.key" | apt-key add -
    apt-key list 3F01618A51312F3F

Isogloss

Maybe I am dummy, but I couldn't run the second script as is due to Permision denied issue. I had to change it like that curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor | sudo tee $line – Eudiometer

@Eudiometer You did it correctly. In order to write to the signed-by file, you need root. – Subcutaneous

Answer 3

In case you're using signed-by, this single line will fix your issue:

curl -s https://packages.gitlab.com/gpg.key | gpg --dearmor -o /usr/share/keyrings/runner_gitlab.gpg

Just need to be sure about which is the gitlab key filename your apt line is reading from.

After that, you'll be able to update repositories as usual.

Baber

Answer 4

One last things that could also make keyrings not accessible by apt is the file rights on gpg file. The script provided by gitlab doesn't fix the right of the gpg key.

As apt update process is launched in within _apt user, the file is not reachable and will be ignored.

To fix that :

chmod 644 /usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg
apt update # <- should now be ok

Gezira