How to fix Jenkins java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded

Question

I upgraded from jenkins 2.219 to 2.272 (latest version as of this writing) and now getting the stack trace below when Jenkins starts.

Jenkins docs says that this happens due to the Configuration as Code plugin and to set the jvm args as -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000. The docs also say to increment the value until the error goes away but so far I'm at 480000 and still getting the error. I also don't see that I have the Configuration as Code plugin installed anyway.

How can this be fixed?

java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded
    at jenkins.model.Jenkins.save(Jenkins.java:3379)
    at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3398)
    at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2637)
    at jenkins.model.Jenkins$16.run(Jenkins.java:3342)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1162)
    at jenkins.model.Jenkins.<init>(Jenkins.java:962)
    at hudson.model.Hudson.<init>(Hudson.java:85)
    at hudson.model.Hudson.<init>(Hudson.java:81)
    at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:312)

Extract

Me too facing the exact error – Archibald

Same issue here as well... we also have Configuration as Code plugin and have tried ramping the delay up to stupidly long times. Having learned from a previous instance of this, have reverted to 2.267 war file from the 2.276 – Leeward

fyi I had to "rpm --import pkg.jenkins.io/redhat-stable/jenkins.io.key" to upgrade then hit the IllegalStateException and did "yum downgrade archives.jenkins-ci.org/redhat/jenkins-2.255-1.1.noarch.rpm" to get jenkins working again – Hew

As per my observation, we can safely upgrade to 2.265. Version 2.266 has some critical changes that become blockers for most cases. – Leucopoiesis

Answer 1

I faced the same issue. I have downloaded the jenkins.war , the earlier version (2.263.1) from https://www.jenkins.io/download/ . Stopped the jenkins and replaced the war with my install directory(C:\Program Files\Jenkins). And started the jenkins.This works like a charm.

Archibald

Yes I had to downgrade as well. – Extract

Answer 2

For anyone having this issue, yes downgrading to https://get.jenkins.io/war-stable/2.263.1/ first is required. Then restart the service, then upgrade all of your plugins, then try the LTS upgrade again, at least for me it worked just fine.

More details about the issue here: https://www.jenkins.io/doc/upgrade-guide/2.204/

SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded If you encounter this, you can tell the plugin to delay configuration for an amount of time to give Jenkins time to load the global configuration before the configuration is applied by the plugin.

To enable this set the io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to a number of milliseconds to delay the initialisation. The required value will be dependant on aspects of your system (cpu/disk) and configuration, and how it can be found is mostly a trial and error. A suggestion would be to start with 5000 (5 Seconds) and then increment by 2000 (2 seconds) until you no longer exhibit the issue and finally add 1000 (1 second) for some extra safety. For example, to delay the configuration by 9 seconds you would use something like the following command java -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000 -jar jenkins.war. Exactly how and where you specify this option depends on the installation method used to install Jenkins

However for me the workaround didnt work without first downgrading, upgrading all plugins, then finally upgrading the core again.

Cannery

Thanks. simply replacing war file worked for me as someone had accidentally upgraded jenkins , when we had absolutely no need to. – Feudal

That is a very very bad mindset to have. Jenkins always need to be up to date. Most of those updates are security issues. If you dont update Jenkins and I get on your network I can own the local administrator or root account on the box it is hosted on. Don't be the reason your source code is stolen or you face a ransomware attack, ALWAYS upgrade your stuff, its part of the management of the platform, if you dont have time to deal with it, dont use Jenkins. – Cannery

Answer 3

To avoid this issue beforehand when you actually have not installed the Casc plugin:

Update to https://github.com/jenkinsci/role-strategy-plugin/releases/tag/role-strategy-3.1 and you should be OK.

When jenkins is already in the faulty state:

download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. https://updates.jenkins.io/download/plugins/role-strategy/

(information taken from: https://github.com/jenkinsci/configuration-as-code-plugin/issues/1531)

Skid

Thanks. I have updated plugin first and then Jenkins. My problem is sovled. – Irremovable

Thanks this solved the problem, downgraded updated the plugin and then upgraded again jenkins. – Novelia

Answer 4

You can download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. Usually the jenkins plugin directory will be under - /var/lib/jenkins/plugins/.

The link to get Role strategy hpi plugin - https://updates.jenkins.io/latest/role-strategy.hpi. And restart Jenkins using this command: systemctl restart jenkins.

Villose

Answer 5

My scenario was to upgrade the Jenkins to the latest version and then restore the data in it to work like it was working before.

While upgrading from Jenkins v2.263 to v2.303, I had the same error. The only thing which worked for me was to upgrade my global plugin i.e Role-based authorization strategy first, and then Jenkins. It worked like a crisp.

Cirone

i still have same situation, when upgrading jenkins jenkins_2.263.4 to 2.289.3 or higher , is there any solution than down grading. i already have already upgraded role-strategy.hpi. to 3.2 . still no help . any other solution? – Reckoning

Hey chandu. I have already mentioned the solution other than downgrading the Jenkins version. You need to identify the plugin, which has the global view in your Jenkins. In my case, I was using a Role-based authorization strategy. I upgraded my plugin first, and then upgraded the Jenkins. The trick worked. – Cirone

thanks !, I already updated role-based authorization strategy plugin. But in my case, i have updated all other plugins that helped, i think its mainly matri-auth and ldap updates made difference, i guess. – Reckoning

Answer 6

We had the same issue after we upgraded to the latest version. After some searching we decided to go with a lower version which has LTS support, 2.263.1 in this case. That version worked instantly so I think this error is a bug.

Stelu

Answer 7

we has similar issue, resolved by downgrading the jenkins.war file downloaded from URL: https://get.jenkins.io/war-stable/2.263.1/

Shrimp

Answer 8

I have the same error on Jenkins 2.332.2 and updated plugin role-strategy v483.v17281966f5c3.
I had to download the version 3.2.0 of the plugin and put it directly in the jenkins plugins folder. (previously I remove the existing '.jpi' file)
Then restarting Jenkins /etc/init.d/jenkins restart, and the issue has gone.
Thank you for the intel!

Modigliani

this works for me. for 2.263.4, 483 is the latest available version for role-strategy, so I upgraded to that version. This downgrading to 3.2.0 worked like a charm. – Eladiaelaeoptene

Answer 9

I have faced similar issue, and found that we don't need to downgrade version. Below are the steps I performed and it worked for me

1. upgrade global plugin i.e Role-based authorization strategy first from Jenkins Console.

2. Here are steps which I have performed: #on ubuntu, in /usr/share/jenkins:

sudo service jenkins stop
sudo mv jenkins.war jenkins.war.old
sudo wget https://updates.jenkins-ci.org/latest/jenkins.war
sudo chown -R jenkins:jenkins jenkins.war
sudo service jenkins start

Equiponderate

Isn't that line with 'debian-stable/binary/jenkins_2.235.5_all.deb' still a downgrade? Seeing that 2.263.4 is a more recent version. – Andromede

@analyst47 yes its now downgraded but its just an example, any version of choice can be used. – Equiponderate