Gitlab CI/CD job's log exceeded limit

Question

When I run my job on Gitlab CI/CD, after a while I obtain the following error message:

Job's log exceeded limit of 4194304 bytes.

How to change this limit?

Frasco

Answer 1

To change the build log size of your jobs in Gitlab CI/CD, you can edit your config.toml file and add a new limit in kilobytes:

[[runners]]
  output_limit = 10000

According to the documentation

output_limit : Maximum build log size in kilobytes. Default is 4096 (4MB).

For this to take effect, you need to restart the gitlab runner:

sudo gitlab-runner restart

Frasco

I did but the changes did not reflect. I still got the same error message – Stedfast

For me also this change was not showing any effect. I found that the gitlab-runner needs to be restarted, Now its working. Command to restart gitlb-runner --> "sudo gitlab-runner restart" – Logging

Even after restarting the runner, when I manually retried the relevant job in my pipeline I saw the same message for the default limit. And even after manually starting the entire pipeline, I still saw the default limit message. – Melantha

What seemed to finally work was to move the line with output_limit up above the sub-sections starting with the line containing [runners.custom_build_dir]. Maybe this is due to something with parsing the TOML file or specific to TOML syntax? – Melantha

The latest docs also seem to claim that you shouldn't in fact need to restart the runner for this change. – Melantha

Just putting on some extra information. I had to switch to the root user using sudo su in order to access the gitlab runner setting inside /etc/gitlab-runner folder. – Boser

Answer 2

So an answer for those that don't have access to the gitlab-runners configuration file that @Ortomala Lokni refers too.

You can redirect the logger output and archive it quite easily by doing the following (Note: this is done for maven builds).

quality-check:
    extends: .retry-on-job-failure
    stage: quality-check
    timeout: 2 hours
    artifacts:
        name: "$CI_BUILD"
        paths:
            - target/client-quality_check.log
        when: always
        expire_in: 3 days
    only:
        - main
        - merge_requests
    script:
        - echo "Sonar Qube Start"
        - mvn MAVEN_CLI_OPTS sonar:sonar --log-file target/client-quality_check.log \-Dsonar.projectKey=$PROJECT_ KEY \-Dsonar.host.url=$SONAR_HOST_URL \-Dsonar.login=$SONAR_TOKEN 
        - echo "Sonar Qube Complete"

Notice within the maven command I am using the --log-file to redirect the maven output to target/client-quality_check.log and then under artifacts I have set to archive this log file by providing the path to the file.

Once this Job finishes I can then look at the Jobs archives and can see my log file with all the logger output in it.

Zoolatry

Can we run this mvn script without sonar? I don't have the credentials to pass into for SONAR and I am also not sure if my organization has those variables configured i.e $PROJECT_KEY and $SONAR_HOST_URL – Upstroke

You can use the option --log-file on any maven command package, install etc. sonar:sonar is just the line I had in this example. $PROJECT_KEY and $SONAR_HOST_URL are variables specifically for my case. – Zoolatry

Answer 3

Starting from gitlab 14.1 there is another configuration option that effects maximum log size: ci_jobs_trace_size_limit (100MB by default). So altering only runner limit, as described in other answers, is not sufficient anymore.

Since gitlab is all about speed and usability, modifying ci_jobs_trace_size_limit is possible only by executing command directly in rails console of the system (or docker container) where gitlab is running.

root@192:/# gitlab-rails console -e production
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
 GitLab:       14.8.2 (c7be43f6dd3) FOSS
 GitLab Shell: 13.23.2
 PostgreSQL:   12.7
-----------------------------------------------------------[ booted in 122.49s ]
Loading production environment (Rails 6.1.4.6)
irb(main):001:0> Plan.default.actual_limits.update!(ci_jobs_trace_size_limit: 100000000)
=> true
irb(main):002:0> quit

Note: if it seems like gitlab-rails console -e production doesn't do anything and console prompt doesn't pop up you'll need to wait.

Antifouling