How to delete gitlab CI jobs pipelines logs/builds and history

Asked 17/11, 2018 at 21:5 Answered 7/11, 2023 at 6:21

Solved gitlab gitlab-ci gitlab-ci-runner

How can we configure gitlab to keep only the last 10 CI jobs/builds and keep deleting the rest?

For example , in Jenkins , we can configure the job to keep only last X builds.

Simmers answered 17/11, 2018 at 21:5 Comment(0)

I think Gitlab doesn't support this feature. But you can create this functionality on your own using Gitlab API and webhooks.

When you push to repo (and pipeline started) it will trigger webhook which can read your CI history via API => you can delete whatever you want.

Here is docs for pipeline events

Here is docs for job API

FYI I use similar solution. I have deployed server for each branch (every brach has MR). When is MR closed it delete deployed server. It is very reliable.

Wafture answered 19/11, 2018 at 14:16 Comment(2)

Is there a way to delete it using gitlab-ctl command ad admin , i mean some command line option for gitlab admin , I am the admin of my gitlab installation – Simmers 19/11, 2018 at 18:38

If you have self-hosted Gitlab you can try this: #38547925 – Wafture 21/11, 2018 at 8:11

UPDATE

As of Gitlab Release 12.6, deleting a pipeline is now an option in the GUI for users in role Owner:

Click the pipeline you want to remove in the pipelines list
Hit the red Delete button in the upper right of the pipeline details page.

As of Gitlab Release 11.6, deleting a pipeline is now an option, for users in Owner role only, via the API.

You need:

An API token
The id of the project
The pipeline_id of the pipeline you wish to remove.

Example using curl from the docs for project id: 1 and pipeline_id: 4:

curl --header "PRIVATE-TOKEN: <your_access_token>" --request "DELETE" "https://gitlab.example.com/api/v4/projects/1/pipelines/46"

Documentation is here

Lissie answered 23/4, 2019 at 15:51 Comment(3)

Is it possible to delete only artifacts and build logs (which are stored in the artifacts directory too), but keep the overall pipeline status in the database? Also, an option to delete everything older than e.g. 3 months would be nice. – Cenac 23/8, 2019 at 9:41

For the id you can use namespaced path encoding. – Phonic 22/10, 2019 at 9:37

Note: you can not delete jobs from the admin gui, you need to delete the pipeline to which the jobs belongs. – Microbalance 6/7, 2020 at 10:46

Mass deletion script fixed for the lazy, delete X pipelines from the oldest (Linux systems).

Note: need jq.

#!/bin/bash
set -e

TOKEN="__SET_YOUR_PERSONAL_ACCESS_TOKEN__"
# Visible under name of project page.
PROJECT_ID="__SET_NUMERIC_PROJECT_ID__"
# Set your gitlab instance url.
GITLAB_INSTANCE="https://gitlab.com/api/v4/projects"
# How many to delete from the oldest, 100 is the maximum, above will just remove 100.
PER_PAGE=100

for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_INSTANCE/$PROJECT_ID/pipelines?per_page=$PER_PAGE&sort=asc" | jq '.[].id') ; do
    echo "Deleting pipeline $PIPELINE"
    curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_INSTANCE/$PROJECT_ID/pipelines/$PIPELINE"
done

Jacobine answered 5/4, 2020 at 18:12 Comment(6)

If you're getting Cannot index string with string "id", you may need an access token with higher privileges. – Dupont 18/1, 2021 at 13:15

It needs personal access token, project token hasn't enough rights. – Pomeroy 29/7, 2021 at 8:41

On Windows, i got curl: (3) URL using bad/illegal format or missing URL because $PIPELINE contained a carriage return. I worked around this by piping the result of jq through dos2unix: ... | jq '.[].id' | dos2unix) ; do – Outshout 9/2, 2022 at 12:7

Also beware that 100 is the maximum per_page value that the GitLab API currently allows, so every value above 100 will just delete 100 pipelines. – Outshout 9/2, 2022 at 16:36

Note that PROJECT should contain a project ID, rather than the name in organization/repo format you usually see. You can find the project ID in the settings. – Stickleback 19/2, 2022 at 8:51

Create a personal access token - docs.gitlab.com/ee/user/profile/personal_access_tokens.html – Firenze 30/1, 2023 at 3:21

Based on previous answers, modified the script to retrieve several projects, and for each one, delete pipelines older than the configured date.

#!/bin/bash
set -e

TOKEN=""
# How many to delete from the oldest.
PER_PAGE=100
UPDATED_BEFORE=2021-02-01T00:00:00Z
GITLAB_URL=


while : ; do
  COUNTER=0

  for PROJECT in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects?per_page=$PER_PAGE" | jq '.[].id') ; do
    echo "Deleting in project $PROJECT"
    for PIPELINE in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc&updated_before=$UPDATED_BEFORE" | jq '.[].id') ; do
        echo "Deleting pipeline $PIPELINE"
        curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
        (( COUNTER++ )) || true
    done
  done

  echo $COUNTER

  if [[ "$COUNTER" -le 0 ]]; then
    break;
  fi
done

Crush answered 19/3, 2021 at 15:53 Comment(4)

what kind of token do I need? – Mating 31/5, 2021 at 5:52

A token that can read the repository (read_repository). – Crush 31/5, 2021 at 8:22

thank you very much, I have modified it to UPDATED_BEFORE=$(date -d "$date -14 days" +"%Y-%m-%dT00:00:00") and set the COUNTER -le 30 so I can put it in a crontab and it will always delete everything older then 14 Days, but leave the last 30 pipelines alone. How is the weather in Lisboa? ;-) – Mating 31/5, 2021 at 18:33

In gitlab 15.8 I had to provide an api-scope token with repository owner access to allow this to work. Also, @Mating I'm pretty sure that $COUNTER -le 30 is not going to do what you want. Instead, you need to add | head -n -30 to the end of the for PIPELINE (before the ")") to always keep the last 30 pipelines around. – Groot 17/2, 2023 at 23:21

Click on Pipeline ID

2.Click on Delete

Bobbette answered 7/7, 2022 at 8:39 Comment(1)

It requires admin access – Bolitho 3/1, 2023 at 12:37

I think Gitlab doesn't support this feature. But you can create this functionality on your own using Gitlab API and webhooks.

When you push to repo (and pipeline started) it will trigger webhook which can read your CI history via API => you can delete whatever you want.

Here is docs for pipeline events

Here is docs for job API

FYI I use similar solution. I have deployed server for each branch (every brach has MR). When is MR closed it delete deployed server. It is very reliable.

Wafture answered 19/11, 2018 at 14:16 Comment(2)

Is there a way to delete it using gitlab-ctl command ad admin , i mean some command line option for gitlab admin , I am the admin of my gitlab installation – Simmers 19/11, 2018 at 18:38

If you have self-hosted Gitlab you can try this: #38547925 – Wafture 21/11, 2018 at 8:11

To delete all pipelines in a project using python the following code can be utilized. You can head over to Jupyter to try it online

import requests

def confirmDeletion():
    confirmDelete = input('Do you want to delete all pipelines in the project Y/N ?')
    if confirmDelete == 'Y' or confirmDelete == 'y'  :
        return True
    else:
        return False

projectId = input('Provide the Gitlab Project ID')
token = input('Provide the Gitlab Access Token')
proceed = bool()
pipelinesQueryUrl = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines'
if confirmDeletion():
    print('Deleting pipelines')
    # The Gitlab API does
    while True:
        response = requests.get(pipelinesQueryUrl, headers = {"PRIVATE-TOKEN": token})
        if response.ok:
            pipelines = response.json()
            if len(pipelines) == 0:
                print('No more pipelines found, exiting')
                break
            else:    
                for pipeline in pipelines:
                    pipelineId = pipeline.get('id')
                    url = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines/{pipelineId}'
                    response = requests.delete(url, headers = {"PRIVATE-TOKEN": token})
                    if response.ok:
                        print(f'Pipeline {pipelineId} succesfully deleted')
                    else:
                        print (f'Deleting pipeline {pipelineId} on path failed: {response.url} : ({response.status_code}) {response.reason}')
                        if response.status_code == 429:
                            # Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
                            print ('Rate Limits have been reached, wait and try again later')
                            break
        else:
            print (f'Querying for pipelines failed: {response.url}: ({response.status_code}) {response.reason}')
            if response.status_code == 429:
                # Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
                print ('Rate Limits have been reached, wait and try again later')
            break
else:
    print('No pipelines deleted')

Mischief answered 23/4, 2021 at 12:22 Comment(0)

Using python gitlab package:

variables:
  GITLAB_TOKEN: ''
  KEEP_DAYS: 14

stages:
  - cleanup

cleanup:
  stage: cleanup
  image: python
  before_script:
    - pip install --upgrade pip pytz python-gitlab
  script: |+
    python -c "
    import os
    import datetime
    import pytz
    import gitlab
    gl = gitlab.Gitlab(os.environ['CI_SERVER_URL'], private_token=os.environ['GITLAB_TOKEN'])
    till = (datetime.datetime.now() - datetime.timedelta(days=int(os.environ['KEEP_DAYS']))).replace(tzinfo=pytz.UTC)
    print(f'Clean before {till}')
    for project in gl.projects.list(iterator=True):
      for pipeline in project.pipelines.list(iterator=True):
        if datetime.datetime.fromisoformat(pipeline.updated_at) > till:
          continue
        print(f'Deleting {project.id}:{pipeline.id}')
        pipeline.delete()
    "
  tags:
    - gitlab

Shall answered 7/11, 2023 at 6:21 Comment(0)

For the lazy, to expand on https://mcmap.net/q/337074/-how-to-delete-gitlab-ci-jobs-pipelines-logs-builds-and-history

Get your PROJECT and TOKEN and run this until all the pipelines are deleted

for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/jobs?per_page=100" | jq '.[].pipeline.id') ; do
    echo "deleting $PIPELINE"
    curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done

Knownothing answered 15/3, 2020 at 14:35 Comment(2)

you make a request to jobs endpoint but delete from /pipelines/ – Chidester 12/7, 2020 at 14:11

It's true, but maybe he wanted to cover some edge case of dangling jobs with already deleted pipeline? (I don't know if that is possible; I'm speculating) – Segmentation 6/10, 2020 at 17:13

_gitlab_pipeline_cleanup() {
echo GITLAB PIPELINE CLEANUP TOOL
echo DELETING ALL EXCEPT RUNNING AND THE MOST RECENT PIPELINE
if [ -z $GITLABURL ]; then echo GITLAB_URL:; read GITLABURL;fi
if [ -z $GITLAB_TOKEN ]; then echo TOKEN:; read GITLAB_TOKEN;fi
if [ -z $PROJECT ]; then echo PROJECT_NUM:; read PROJECT;fi

list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)

echo removing from $GITLABURL Project number $PROJECT
while echo $(echo -n "$list" |wc -c  ) |grep -v ^0$; do 

list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)

for item in $list ;do 
echo -n "| -p $item |"
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECT/pipelines/$item"
done 
done 
echo ; } ;

Then you could do

for PROJECT in 12345 23476234 2138734876238746 ;do _gitlab_pipeline_cleanup ;done

Toro answered 12/7, 2020 at 14:12 Comment(0)

As a small extension to Mog's answer since my gitlab on the one hand provides a maximum of 100 entries per page and on the other hand I wanted to delete depending on the status of the pipeline:

#!/bin/bash
set -e

TOKEN="---your token here--"
PROJECT="PROJECT-ID"
BASEURL="https://gitlab.com/api/v4/projects" # Base Url, if you host your 
# Status values: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled
STATUS=(failed canceled skipped) # separate status by space

# How many to delete from the oldest.
DELETE_CNT=500

# -----
CNT=0
declare -A STATUS_RESULT

for CUR_STATUS in "${STATUS[@]}";
do
  TEMP_CNT=$CNT
  DOLOOP=true
  while [ "$DOLOOP" = true ]
  do
    DOLOOP=false

    for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$BASEURL/$PROJECT/pipelines?per_page=$DELETE_CNT&sort=asc&status=$CUR_STATUS" | jq '.[].id') ; do
      if [[ "$CNT" -lt "$DELETE_CNT" ]]; then
        echo "#$CNT Deleting pipeline $PIPELINE with status $CUR_STATUS"
        curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$BASEURL/$PROJECT/pipelines/$PIPELINE"
        let "CNT+=1"
        DOLOOP=true
      fi
    done

    if [ "$DOLOOP" = true ] ; then
      if [[ "$CNT" -ge "$DELETE_CNT" ]]; then
       DOLOOP=false
      fi
    fi
  done

  TEMP_CNT=$((CNT-TEMP_CNT))
  STATUS_RESULT[$CUR_STATUS]=$TEMP_CNT
  echo "Removed $TEMP_CNT pipelines with status $CUR_STATUS"
done

echo "===================================================="
echo "= Summary of removed pipelines (max: $DELETE_CNT)"
echo "=   Total: $CNT"
echo "="
for key in "${!STATUS_RESULT[@]}"; do
    CNT=${STATUS_RESULT[$key]}
    echo "=   $key: $CNT"
done

Thao answered 4/3, 2021 at 8:59 Comment(1)

Perfect, but status blocked don`t work. Error: jq: error (at <stdin>:0): Cannot index string with string "id" – Infirmary 25/11, 2022 at 12:57

This deletes all pipelines for a project, but I'm sure you can figure out the Perl stuff to skip the first 10

curl -s --header "PRIVATE-TOKEN: ******" --request "GET" "https://gitlab.com/api/v4/projects/********/pipelines?per_page=32767" \
| perl -MJSON -le '$d = decode_json(<>); map { print $_->{"id"} } @$d' - | while read i; do
  curl -s --header "PRIVATE-TOKEN: ********" --request "DELETE" "https://gitlab.com/api/v4/projects/******/pipelines/$i"
done

The project ID is written under the project name (project page) and you get access tokens from "Edit Profile"."Access Tokens" and just check the "API" checkbox.

Install JSON module in Perl on Linux via:

sudo apt -y install libjson-perl

Irairacund answered 7/4, 2021 at 14:30 Comment(0)

My 2 cents. I had issues connecting to the API with curl and I discovered that the R httr library did work fine, so I developed a small R script to have the work done.

library(rjson)
library(httr)
library(optparse)
library(data.table)

#########################
## Define the arguments
option_list = list(
    make_option(c("-n", "--projectName"),
        type="character", default=NULL,
        help="Project name (e.g. cronadmin)",
        metavar="character"),
    make_option(c("-d", "--days"),
        type="integer", default=NULL,
        help="Remove ci/cd pieplines older than this number of days (cannot be used together with the -p/--id option)",
        metavar="integer"),
    make_option(c("-p", "--pid"),
        type="integer", default=NULL,
        help="Identifier of the pipeline that should be removed (cannot be used together with the --days/-d option)",
        metavar="integer"),
    make_option(c("-u", "--gitlabUrl"),
        type="character", 
        help="gitlab URL",
        default=NULL,
        metavar="character"),
    make_option(c("-g", "--gitlabToken"),
        type="character", 
        help="gitlab token",
        default=NULL,
        metavar="character")           
);

errors <- vector();

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

projectName <- opt$projectName
days <- opt$days
pid <- opt$pid
gitlabUrl <- opt$gitlabUrl
gitlabToken <- opt$gitlabToken

if (!is.null(days) && !is.null(pid)) {
    stop(" -p/--pid option cannot be used together with the --days/-d option")
}
if (is.null(days) && is.null(pid)) {
    stop("-p/--pid or --days/-d option must be used")
}
daysVal <- as.numeric(days)
pidVal <- as.numeric(pid)

if (!is.null(days) && (is.na(daysVal) || daysVal != floor(daysVal))) {
    stop(paste("Number of days must be an integer"))
}
if (!is.null(pid) && (is.na(pidVal) || pidVal != floor(pidVal))) {
    stop(paste("Pipelineid id must be an integer"))
}
if (is.null(projectName)) {
    stop("Missing project name (-n)")
}
if (is.null(gitlabUrl)) {
    stop("Missing gitlaburl (-u)")
}
if (is.null(gitlabToken)) {
    stop("Missing gitlabToken (-g)")
}



## GET THE PROJECT ID IN WHICH THE pipes you want to remove are defined

apiUrl <- paste0(gitlabUrl, "/api/v4/")

# GET PROJECTS
projectResults  <- GET( 
      paste0(apiUrl,"/projects"),
      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
    ))
)

projectsList <- content(projectResults)
projectId <-NA
for (i in 1:length(projectsList)) {
    project <- projectsList[[i]]
    projectNamei <- project$name
    if (projectNamei == projectName) {
        projectId <- project$id
    }
    
}


if (is.na(projectId)) {
    stop(paste("Project with name",projectName,"does not exist"))
} else {
    print(paste("Project", projectName, "corresponds to id", projectId))
}

pipelinesToDelete <- vector()

if (!is.null(pid)) {
    # removal by pipeline
    print(paste("Trying to remove pipeline with id", pidVal))
    # check pidVal does exist
    pipelineExistsResults <- GET( 
      paste0(apiUrl,"/projects/",projectId,"/pipelines/",pid),
      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
      ))
    )
    message <- content(pipelineExistsResults)$message
    if (!is.null(message) && message == '404 Not found') {
        stop(paste("Pipeline",pid, "not found"))
    } else {
        pipelinesToDelete <- append(pipelinesToDelete, pid)
    }
} else {
    # removal by date
    allPipelinesResults <- GET( 
      paste0(apiUrl,"/projects/",projectId,"/pipelines?per_page=100"),

      add_headers(c(
        "PRIVATE-TOKEN" = gitlabToken
      ))

    )
    allPipelinesDT <- rbindlist(content(allPipelinesResults))
    allPipelinesDT$date <- as.Date(allPipelinesDT$updated_at)
    allPipelinesDT$daynb <- as.Date(Sys.time()) - allPipelinesDT$date
    
    pipelinesToDelete <- as.vector(allPipelinesDT[daynb > daysVal][["id"]])
}



for (pipelinesToDeletei in pipelinesToDelete) {
    contentDeleteResult <- DELETE(
        url = paste0(apiUrl,"/projects/",projectId,"/pipelines/",pipelinesToDeletei),
        add_headers(c(
            "PRIVATE-TOKEN" = gitlabToken
        ))
    )
    print(contentDeleteResult)
    
}

Precarious answered 22/9, 2022 at 8:11 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

UPDATE

Recommended topics

Hot tags