I have a private repository at gitlab.com that uses the CI feature. Some of the CI jobs create artifacts files that are stored. I just implemented that the artifacts are deleted automatically after one day by adding this to the CI configuration:
expire_in: 1 day
That works great - however, old artifacts won't be deleted (as expected). So my question is:
How can I delete old artifacts or artifacts that do not expire? (on gitlab.com, no direct access to the server)
You can use the GitLab REST API to delete the artifacts from the jobs if you don't have direct access to the server. Here's a sample curl script that uses the API:
#!/bin/bash
# project_id, find it here: https://gitlab.com/[organization name]/[repository name]/edit inside the "General project settings" tab
project_id="3034900"
# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="Lifg_azxDyRp8eyNFRfg"
server="gitlab.com"
# go to https://gitlab.com/[organization name]/[repository name]/-/jobs
# then open JavaScript console
# copy/paste => copy(_.uniq($('.ci-status').map((x, e) => /([0-9]+)/.exec(e.href)).toArray()).join(' '))
# press enter, and then copy the result here :
# repeat for every page you want
job_ids=(48875658 48874137 48873496 48872419)
for job_id in ${job_ids[@]}
do
URL="https://$server/api/v4/projects/$project_id/jobs/$job_id/erase"
echo "$URL"
curl --request POST --header "PRIVATE-TOKEN:${token}" "$URL"
echo "\n"
done
An API call should be easier to script, with GitLab 14.7 (January 2022), which now offers:
Bulk delete artifacts with the API
While a good strategy for managing storage consumption is to set regular expiration policies for artifacts, sometimes you need to reduce items in storage right away.
Previously, you might have used a script to automate the tedious task of deleting artifacts one by one with API calls, but now you can use a new API endpoint to bulk delete job artifacts quickly and easily.
See Documentation, Issue 223793 and Merge Request 75488.
curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \
"https://gitlab.example.com/api/v4/projects/1/artifacts"
As noted by Lubo in the comments:
Response of given API is 202 Accepted. It means for me, that deletion will happen on background.
Also admin area ís updated a bit later than deletion happens
You can use the GitLab REST API to delete the artifacts from the jobs if you don't have direct access to the server. Here's a sample curl script that uses the API:
#!/bin/bash
# project_id, find it here: https://gitlab.com/[organization name]/[repository name]/edit inside the "General project settings" tab
project_id="3034900"
# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="Lifg_azxDyRp8eyNFRfg"
server="gitlab.com"
# go to https://gitlab.com/[organization name]/[repository name]/-/jobs
# then open JavaScript console
# copy/paste => copy(_.uniq($('.ci-status').map((x, e) => /([0-9]+)/.exec(e.href)).toArray()).join(' '))
# press enter, and then copy the result here :
# repeat for every page you want
job_ids=(48875658 48874137 48873496 48872419)
for job_id in ${job_ids[@]}
do
URL="https://$server/api/v4/projects/$project_id/jobs/$job_id/erase"
echo "$URL"
curl --request POST --header "PRIVATE-TOKEN:${token}" "$URL"
echo "\n"
done
copy([...$('a[title="Download artifacts"],a[download]').map((x, e) => /\/([0-9]+)\//.exec(e.href)[1])].join(' ')). –
Woothen Building on top of @David 's answer, @Philipp pointed out that there is now an api endpoint to delete only the job artifacts instead of the entire job.
You can run this script directly in the browser's Dev Tools console, or use node-fetch to run in node.js.
//Go to: https://gitlab.com/profile/personal_access_tokens
const API_KEY = "API_KEY";
//You can find project id inside the "General project settings" tab
const PROJECT_ID = 12345678;
const PROJECT_URL = "https://gitlab.com/api/v4/projects/" + PROJECT_ID + "/"
let jobs = [];
for(let i = 0, currentJobs = []; i == 0 || currentJobs.length > 0; i++){
currentJobs = await sendApiRequest(
PROJECT_URL + "jobs/?per_page=100&page=" + (i + 1)
).then(e => e.json());
jobs = jobs.concat(currentJobs);
}
//skip jobs without artifacts
jobs = jobs.filter(e => e.artifacts);
//keep the latest build.
jobs.shift();
for(let job of jobs)
await sendApiRequest(
PROJECT_URL + "jobs/" + job.id + "/artifacts",
{method: "DELETE"}
);
async function sendApiRequest(url, options = {}){
if(!options.headers)
options.headers = {};
options.headers["PRIVATE-TOKEN"] = API_KEY;
return fetch(url, options);
}
According to the documentation, deleting the entire job log (click on the trash can) will also delete the artifacts.
I am on GitLab 8.17 and am able to remove artifacts for particular job by navigating to storage directory on server itself, default path is:
/var/opt/gitlab/gitlab-rails/shared/artifacts/<year_month>/<project_id?>/<jobid>
Removing both whole folder for job or simply contents, disappears artifact view from GitLab pipline page.
The storage path can be changed as described in docs:
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/administration/job_artifacts.md#storing-job-artifacts
If you don't mind removing entire jobs along with their artifacts in bulk, you can use the glab CLI like this:
glab ci delete --dry-run --older-than 8760h --paginate
This removes all jobs older than 1 year. Just remove --dry-run to make it happen.
The artifacts seem to be deleted asynchronously, so it may take some time for your repository's storage usage to be updated.
If you have deleted all the jobs by accident (thinking the artifacts would be gone, but they didn't) what would be the alternative then brute-forcing a loop range?
I have this code, which does bruteforce on a range of numbers. But since I use the gitlab.com public runners, It's a long-range
# project_id, find it here: https://gitlab.com/[organization name]/[repository name]/edit inside the "General project settings" tab
project_id="xxxxxx" #
# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="yyyyy"
server="gitlab.com"
# Get a range of the oldest known job and the lastet known one, then bruteforce. Used in the case when you deleted pipelines and can't retrive Job Ids.
# https://mcmap.net/q/409272/-for-loop-over-sequence-of-large-numbers-in-bash-duplicate
for (( job_id = 59216999; job_id <= 190239535; job_id++ )) do
echo "$job_id"
echo Job ID being deleted is "$job_id"
curl --request POST --header "PRIVATE-TOKEN:${token}" "https://${server}/api/v4/projects/${project_id}/jobs/${job_id}/erase"
echo -en '\n'
echo -en '\n'
done
This Python solution worked for me with GitLab 13.11.3.
#!/bin/python3
# delete_artifacts.py
import json
import requests
# adapt accordingly
base_url='https://gitlab.example.com'
project_id='1234'
access_token='123412341234'
#
# Get Version Tested with Version 13.11.3
# cf. https://docs.gitlab.com/ee/api/version.html#version-api
#
print(f'GET /version')
x= (requests.get(f"{base_url}/api/v4/version", headers = {"PRIVATE-TOKEN": access_token }))
print(x)
data=json.loads(x.text)
print(f'Using GitLab version {data["version"]}. Tested with 13.11.3')
#
# List project jobs
# cf. https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs
#
request_str=f'projects/{project_id}/jobs'
url=f'{base_url}/api/v4/{request_str}'
print(f'GET /{request_str}')
x= (requests.get(url, headers = {"PRIVATE-TOKEN": access_token }))
print(x)
data=json.loads(x.text)
input('WARNING: This will delete all artifacts. Job logs will remain be available. Press Enter to continue...' )
#
# Delete job artifacts
# cf. https://docs.gitlab.com/ee/api/job_artifacts.html#delete-artifacts
#
for entry in data:
request_str=f'projects/{project_id}/jobs/{entry["id"]}/artifacts'
url=f'{base_url}/api/v4/{request_str}'
print(f'DELETE /{request_str}')
x = requests.delete(url, headers = {"PRIVATE-TOKEN": access_token })
print(x)
I'll keep an updated version here. Feel free to reach out and improve the code.
As you said, it's possible to change the retention time by adding a artifacts:expire_in field in the job settings.
In my testing in Gitlab 15.10.2-ee, the setting is applied retroactively to all matching jobs in the history. The deletion is not instant: it presumably happens once some scheduled job runs, probably once a day.
You could also change the instance-wide setting but that doesn't apply to past artifacts.
expire_build_artifacts_worker_cron in gitlab.rb. –
Cresa Although the answers here are pretty nicely summarised, I am just adding on the Python script used by me for manually cleaning only the artefacts on the latest version of GitLab linux installation (v16.10.3-ee).
I first retrieved the number of pages and projects I have in our GitLab environment, using the values "x-total-pages" and "x-total" of the command curl https://gitlab.company/api/v4/projects?private_token=<token> --head. Then, I iterated through the GitLab projects in the paginated manner, and retrieved the project-IDs into a list. Lastly, I iterated through this list to perform the necessary artefact cleanup.
# This is a sample Python script referenced on the idea from https://mcmap.net/q/400016/-remove-artifacts-from-ci-manually
import json
import requests
def clean_gitlab_artefact():
base_url = "https://gitlab.company"
access_token = "access-token" # check with your GitLab project owner
print(f'GET /version')
x = (requests.get(f"{base_url}/api/v4/version", headers={"PRIVATE-TOKEN": access_token}))
print(x)
data = json.loads(x.text)
print(f'Using GitLab version {data["version"]}. Implemented on 16.10.3-ee!')
# # there were 173 projects at the time of running this script, which can be checked by exploring the
# # value "x-total" of the command `curl https://gitlab.company/api/v4/projects?private_token=<token> --head`
page = 1
total_project_ids = []
while page != 3:
print(f'GET /project-IDs')
projects = (requests.get(f"{base_url}/api/v4/projects?per_page=100&page={page}",
headers={"PRIVATE-TOKEN": access_token}))
page += 1
# print(projects)
data = json.loads(projects.text)
project_ids = [o["id"] for o in data]
total_project_ids += project_ids
print(total_project_ids)
for project_id in total_project_ids:
request_str = f'projects/{project_id}/artifacts'
url = f'{base_url}/api/v4/{request_str}'
print(f'DELETE /{request_str}')
x = (requests.delete(url, headers={"PRIVATE-TOKEN": access_token}))
print(x)
if __name__ == '__main__':
clean_gitlab_artefact()
And, please find below the output of the script which returns soon (with an accepted response 202 if the artefacts would be deleted asynchronously), as cleaning will happen in the background asynchronously, as also explained in other answers here.
None of the API based solutions worked for me because the DELETE API only sets the expiry date on the build. Then it is upto sidekiq to perform the deletion. If there is a bug in gitlab that ignores the expiry date on artifacts even if it is set, then nothing will happen.
Builds can also be removed manually with rails runner. The below script cleans up both artifacts as well as job logs that are older than 1 month. Note that it only does so for the top 20 projects that are consuming diskpace
#!/usr/bin/env ruby
# This is a ruby script to delete build artifacts from gitlab that are older than 1 month
# Copy this file to /tmp/ folder on gitlab server then execute rails runner as below
# gitlab-rails runner /tmp/cleanupArtifacts.rb
include ActionView::Helpers::NumberHelper
ProjectStatistics.order(build_artifacts_size: :desc).limit(20).each do |s|
builds_artifacts = s.project.builds.with_downloadable_artifacts
counter=1
builds_artifacts.find_each do |build|
counter=counter+1
puts "Build #{build.id} \t created at #{build.created_at}"
if build.created_at < 1.month.ago
puts "Build #{build.id} marked for deletion"
build.destroy!
end
end
puts "#{number_to_human_size(s.build_artifacts_size)} \t #{s.project.full_path} \t Builds: #{counter}"
end
© 2022 - 2024 — McMap. All rights reserved.
copy([...$('a[title="Download artifacts"],a[download]').map((x, e) => /\/([0-9]+)\//.exec(e.href)[1])].join(' ')). – Woothen