How do you execute a script after a job is cancelled in GitLab?

I am looking for a way to clean up the runner after a job has been cancelled in GitLab. The reason is we often have to cancel running jobs because the runner is sometimes stuck in the test pipeline and I can imagine a couple of other scenarios where you would want to cancel a job and have a clean up script run after. I am looking for something like after_script but just in the case when a job was cancelled. I checked the GitLab keyword reference but could not find what I need.

The following part of my gitlab-ci.yaml shows the test stage which I would like to gracefully cancel by calling docker-compose down when the job was cancelled.

I am using a single gitlab-runner. Also, I don't use dind.

test_stage:
  stage: test
  only:
  - master
  - tags
  - merge_requests
  image: registry.gitlab.com/xxxxxxxxxxxxxxxxxxx
  variables:
    HEADLESS: "true"
  script:
  - docker login -u="xxxx" -p="${QUAY_IO_PASSWORD}" quay.io  
  - npm install
  - npm run test
  - npm install wait-on  
  - cd example
  - docker-compose up --force-recreate --abort-on-container-exit --build traefik frontend &
  - cd ..
  - apt install -y iproute2
  - export DOCKER_HOST_IP=$( /sbin/ip route|awk '/default/ { print $3 }' )
  - echo "Waiting for ${DOCKER_HOST_IP}/xxxt"
  - ./node_modules/.bin/wait-on "http://${DOCKER_HOST_IP}/xxx" && export BASE_URL=http://${DOCKER_HOST_IP} && npx codeceptjs run-workers 2
  - cd example
  - docker-compose down
  after_script:
  - cd example && docker-compose down 
  artifacts:
    when: always
    paths:
      - /builds/project/tests/output/
  retry:
    max: 2
    when: always
  tags: [custom-runner]

Unfortunately this is not currently possible in GitLab. There have been several tickets opened in their repos, with this one being the most up-to-date.

As of the day that I'm posting this (September 27, 2022), there have been at least 14 missed deliverables for this. GitLab continues to say it's coming, but has never delivered it in the six years that this ticket has been open.

There are mitigations as far as automatic job cancelling, but unfortunately that will not help in your case.

Based on your use case, I can think of two different solutions:

Create a wrapper script that detects when parts of your test job are hanging
Set a timeout on the pipeline (In GitLab you can go to Settings -> CI/CD -> General Pipelines -> Timeout)

Neither of these solutions are as robust as if GitLab themselves implemented a solution, but they can at least prevent you from having a job hang for an eternity and clogging up everything else in the pipeline.

Recommended topics

Hot tags