How do I change log level in runtime without restarting Spring Boot application

Question

I have deployed a Spring Boot application in PCF. I want to log the message based on the environment variable. What should I do so that the run time log level change will work without restarting the application?

Sofa

Answer 1

Changing the log level in Spring Boot 1.5+ can be done with a http-endpoint

Add

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

and than you can use

curl -X "POST" "http://localhost:8080/loggers/de.springbootbuch" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d $'{
  "configuredLevel": "WARN"
}'  

Where everything beyond /loggers/ is the name of the logger.

If you running this in PCF it get's even better: This is directly supported from their backend.

Thrasonical

It's worth emphasizing that the Actuator loggers endpoint was added in 1.5.1 and if you have an older Spring Boot app using < 1.5.1 with Actuator, even though there are other Actuator endpoints provided, you won't find the one for loggers. I know because I just spent ages trying to work out why it wouldn't turn on :-) – Ritualist

For Spring Boot 2.1 you have to make sure that the loggers endpoint is exposed to web by adding loggers value to management.endpoints.web.exposure.include setting. If you don't have such line in your application.properties file, than add: management.endpoints.web.exposure.include=health,info,loggers, since its default is health,info – Nicholasnichole

In this article is described the process: blog.codeleak.pl/2017/03/… . Do not forget to add the attribute scan=true in the logback.xml as written here: https://mcmap.net/q/301975/-how-do-i-change-log-level-in-runtime-without-restarting-spring-boot-application – Accusatory

The only problem is when you have a lot of instances running. You will hit the API and it will go to 1 instance and change the log level. But now if you have to revert it, the chances are very less that your load balancer will send that request to the exact same instance. It becomes more difficult when the number of instances is more. – Kami

Answer 2

For Spring Boot 2.1.5+:

First, you need the actuator Plugin:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Second, you need to expose the endpoint like Dennis said in his comment (loggers is disabled by default):

management.endpoints.web.exposure.include=health,info,loggers

Finally, you can use the Rest Endpoints to get Information about the loggers and set the logging levels.

curl -X "GET" "http://localhost:8080/actuator/loggers"

To set the Root logging Level you can use

curl -X "POST" "http://localhost:8080/actuator/loggers/ROOT" -H "Content-Type: application/json; charset=utf-8"   -d $'{ "configuredLevel": "INFO" }'

Relent

Thanks! The management.endpoints.web.exposure.include=...,loggers property was the missing piece! – Rockyrococo

Answer 3

This is an extension of @Michael Simons answer. With this method you will have a UI for doing that:

This method is a bit longer but it solves much much more. We are going to use a tool called Spring Boot Admin Server.

1. First you need to include some dependencies

   <!--Dependency for registering your app as a Spring Boot Admin Server-->
   <dependency>
       <groupId>de.codecentric</groupId>
       <artifactId>spring-boot-admin-server</artifactId>
       <version>1.3.3</version>
   </dependency>
   
   <!--Provide a nice looking ui-->
   <dependency>
       <groupId>de.codecentric</groupId>
       <artifactId>spring-boot-admin-server-ui</artifactId>
       <version>1.3.3</version>
   </dependency>
   
   <!--Dependency for registering your app as a Spring Boot Admin Client-->
   <dependency>
       <groupId>de.codecentric</groupId>
       <artifactId>spring-boot-admin-starter-client</artifactId>
       <version>1.3.0</version>
   </dependency>
   <dependency>
           <groupId>org.jolokia</groupId>
           <artifactId>jolokia-core</artifactId>
   </dependency>
   

2. Enable your app to be a Spring Boot Admin Server using the annotation @EnableAdminServer.

   @SpringBootApplication
   @EnableAdminServer
   public class Application {
      public static void main(String[] args) {
         // ... your code as before ...
      }
   }
   

3. In your application.properties add the following:

   Register your app to the Spring Boot Admin Server which is still your app

   spring.boot.admin.url=http://localhost:8031
   

   Instruct Spring Boot Admin Server where to find the client

   // For versions 2.*.*
   spring.boot.admin.client.url=http://localhost:8031
   // For versions 1.*.*
   spring.boot.admin.client.service-url=http://localhost:8031
   spring.boot.admin.client.management-url=http://localhost:8031
   spring.boot.admin.client.health-url=http://localhost:8031/health
   

4. In your logback.xml just add the following line <jmxConfigurator/>. This allows configuration of logback via JMX. More info here

... and voila you are done. Now you can change the debug level for any logger at runtime.

i. Just visit the url for your Spring Boot Admin Server-in our case here (http:/localhost:8031).

ii. A list of applications (clients) registered will be displayed on the home page.

iii. Click Details against the registered clients which will take you to another page.

iv. Click the Logging tab which will list all loggers registered in your application.

v. You can change the log levels it will change your logging level at runtime. Here is a snippet of what you expect

Hacksaw

Would this work if I have multiple instances running? Say for example in a K8s cluster in a given namespace, if I have multiple pods (instances) running for a specific service.. Will the change be effected in all the instances? – Elastance

@Elastance Yes. Just tested, it changes for all instances – Hacksaw

Answer 4

If you are using logback api to configure logging in the project then you can use the AutoScan feature of logback api. As per documentation

logback-classic will scan for changes in its configuration file and automatically reconfigure itself when the configuration file changes. In order to instruct logback-classic to scan for changes in its configuration file and to automatically re-configure itself set the scan attribute of the element to true.

<configuration scan="true"> 
  ... 
</configuration> 

Scan frequency: "By default, the configuration file will be scanned for changes once every minute". See the logback API documentation for more details.

Blackfish

Answer 5

Since Spring Boot 1.5.x, you can use logger endpoint to POST desired logging level.

Pandurate

Answer 6

You can use following code in controller and call the API to change log level

@PostMapping("/changeloglevel")
public void changeloglevel(@RequestParam String loglevel)
{
    LoggerContext loggerContext = (LoggerContext)LoggerFactory.getILoggerFactory();
    loggerContext.getLogger("package where springboot main class resides").setLevel(Level.toLevel(loglevel));   
}

The loglevel can be ERROR, INFO, WARN , etc

Luckless

java.lang.ClassCastException: class org.apache.logging.slf4j.Log4jLoggerFactory cannot be cast to class org.apache.logging.log4j.core.LoggerContext – Iridis

@PetersonV, review your imports and make sure you use the correct logback classes not log4j classes or others – Seaworthy

plus If you need to set the log level globally for all classes [for root], you can by loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).setLevel(Level.valueOf(loglevel)); – Seaworthy

Answer 7

The default logging provider is logback. To setup your system so that the logging level can be changed at runtime you need to perform the following steps:

Firstly in src/main/resources create a custom logback configuration named logback-spring.xml that includes spring's default configurator and then adds the directive that exposes logback configuration over JMX:

<configuration>
  <include resource="org/springframework/boot/logging/logback/base.xml"/>
  <jmxConfigurator />    
</configuration>

Now add a dependency on the Jolokia JMX-over-HTTP bridge: org.jolokia:jolokia-core.

You should now be able to hit /jolokia endpoints on your spring boot application. The protocol is documented here. It's not pretty. To get you started, here's a few GET examples that you can hit straight from a browser:

Show ROOT logger level:

/jolokia/exec/ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator/getLoggerLevel/ROOT

Change ROOT logger level to debug:

/jolokia/exec/ch.qos.logback.classic:Name=default,Type=ch.qos.logback.classic.jmx.JMXConfigurator/setLoggerLevel/ROOT/debug

spring-boot-actuator is aware of the /jolokia endpoint and it is marked sensitive=true so if you have spring-security on the classpath then it will require authentication.

Perkin

Answer 8

There are 3 ways to do it.

1. Spring Actuator - leveraging /loggers endpoint.
2. Spring Boot Admin.
3. LogBack auto-scan.

Do check out this blog by Amy DeGregorio for more details.

Viridian

Answer 9

For IntelliJ users: I had a similar scenario and ended up writing my own Intellij plugin, which by far was the simplest solution for me. It is essentially a wrapper over the actuator log endpoints

1. Enable the log endpoint in Spring Boot Actuator as described in the above comment
2. Install the IntellIj Plugin LogBoot
3. Connect to your Spring Boot App and you are ready to go

Check it out here: https://plugins.jetbrains.com/plugin/17101-logboot LogBoot plugin

Dogvane

Answer 10

You can create a jsp and using it directly like https://gist.github.com/iamkristian/943918/043ac51bd80321a0873d93277979c8a9a20a9a48#file-log4jadmin-jsp

<%@ page language="java" contentType="text/html;charset=UTF-8" %>
<%@ page import="org.apache.log4j.Level" %>
<%@ page import="org.apache.log4j.LogManager" %>
<%@ page import="org.apache.log4j.Logger" %>
<%@ page import="java.util.HashMap" %>
<%@ page import="java.util.Enumeration" %>
<%@ page import="java.util.Set" %>
<%@ page import="java.util.Arrays" %>
<% long beginPageLoadTime = System.currentTimeMillis();%>

<html>
<head>
    <title>Log4J Administration</title>
    <style type="text/css">
        <!--
        #content {
            margin: 0px;
            padding: 0px;
            text-align: center;
            background-color: #ccc;
            border: 1px solid #000;
            width: 100%;
        }
        body {
            position: relative;
            margin: 10px;
            padding: 0px;
            color: #333;
        }
        h1 {
            margin-top: 20px;
            font: 1.5em Verdana, Arial, Helvetica sans-serif;
        }
        h2 {
            margin-top: 10px;
            font: 0.75em Verdana, Arial, Helvetica sans-serif;
            text-align: left;
        }
        a, a:link, a:visited, a:active {
            color: red;
            text-decoration: none;
            text-transform: uppercase;
        }
        table {
            width: 100%;
            background-color: #000;
            padding: 3px;
            border: 0px;
        }
        th {
            font-size: 0.75em;
            background-color: #ccc;
            color: #000;
            padding-left: 5px;
            text-align: center;
            border: 1px solid #ccc;
            white-space: nowrap;
        }
        td {
            font-size: 0.75em;
            background-color: #fff;
            white-space: nowrap;
        }
        td.center {
            font-size: 0.75em;
            background-color: #fff;
            text-align: center;
            white-space: nowrap;
        }
        .filterForm {
            font-size: 0.9em;
            background-color: #000;
            color: #fff;
            padding-left: 5px;
            text-align: left;
            border: 1px solid #000;
            white-space: nowrap;
        }
        .filterText {
            font-size: 0.75em;
            background-color: #fff;
            color: #000;
            text-align: left;
            border: 1px solid #ccc;
            white-space: nowrap;
        }
        .filterButton {
            font-size: 0.75em;
            background-color: #000;
            color: #fff;
            padding-left: 5px;
            padding-right: 5px;
            text-align: center;
            border: 1px solid #ccc;
            width: 100px;
            white-space: nowrap;
        }
        -->
    </style>
</head>
<body onLoad="javascript:document.logFilterForm.logNameFilter.focus();">
<%
    String containsFilter = "Contains";
    String beginsWithFilter = "Begins With";
    String[] logLevels = {"debug", "info", "warn", "error", "fatal", "off"};
    String targetOperation = (String) request.getParameter("operation");
    String targetLogger = (String) request.getParameter("logger");
    String targetLogLevel = (String) request.getParameter("newLogLevel");
    String logNameFilter = (String) request.getParameter("logNameFilter");
    String logNameFilterType = (String) request.getParameter("logNameFilterType");
%>
<div id="content">
<h1>Log4J Administration</h1>
<div class="filterForm">
    <form action="log4jAdmin.jsp" name="logFilterForm">Filter Loggers:&nbsp;&nbsp;
        <input name="logNameFilter" type="text" size="50" value="<%=(logNameFilter == null ? "":logNameFilter)%>"
               class="filterText"/>
        <input name="logNameFilterType" type="submit" value="<%=beginsWithFilter%>" class="filterButton"/>&nbsp;
        <input name="logNameFilterType" type="submit" value="<%=containsFilter%>" class="filterButton"/>&nbsp;
        <input name="logNameClear" type="button" value="Clear" class="filterButton"
               onmousedown='javascript:document.logFilterForm.logNameFilter.value="";'/>
        <input name="logNameReset" type="reset" value="Reset" class="filterButton"/>
        <param name="operation" value="changeLogLevel"/>
    </form>
</div>
<table cellspacing="1">
    <tr>
        <th width="25%">Logger</th>
        <th width="25%">Parent Logger</th>
        <th width="15%">Effective Level</th>
        <th width="35%">Change Log Level To</th>
    </tr>
    <%
        Enumeration loggers = LogManager.getCurrentLoggers();
        HashMap loggersMap = new HashMap(128);
        Logger rootLogger = LogManager.getRootLogger();
        if (!loggersMap.containsKey(rootLogger.getName())) {
            loggersMap.put(rootLogger.getName(), rootLogger);
        }
        while (loggers.hasMoreElements()) {
            Logger logger = (Logger) loggers.nextElement();
            if (logNameFilter == null || logNameFilter.trim().length() == 0) {
                loggersMap.put(logger.getName(), logger);
            } else if (containsFilter.equals(logNameFilterType)) {
                if (logger.getName().toUpperCase().indexOf(logNameFilter.toUpperCase()) >= 0) {
                    loggersMap.put(logger.getName(), logger);
                }
            } else {
// Either was no filter in IF, contains filter in ELSE IF, or begins with in ELSE
                if (logger.getName().startsWith(logNameFilter)) {
                    loggersMap.put(logger.getName(), logger);
                }
            }
        }
        Set loggerKeys = loggersMap.keySet();
        String[] keys = new String[loggerKeys.size()];
        keys = (String[]) loggerKeys.toArray(keys);
        Arrays.sort(keys, String.CASE_INSENSITIVE_ORDER);
        for (int i = 0; i < keys.length; i++) {
            Logger logger = (Logger) loggersMap.get(keys[i]);
// MUST CHANGE THE LOG LEVEL ON LOGGER BEFORE GENERATING THE LINKS AND THE
// CURRENT LOG LEVEL OR DISABLED LINK WON'T MATCH THE NEWLY CHANGED VALUES
            if ("changeLogLevel".equals(targetOperation) && targetLogger.equals(logger.getName())) {
                Logger selectedLogger = (Logger) loggersMap.get(targetLogger);
                selectedLogger.setLevel(Level.toLevel(targetLogLevel));
            }
            String loggerName = null;
            String loggerEffectiveLevel = null;
            String loggerParent = null;
            if (logger != null) {
                loggerName = logger.getName();
                loggerEffectiveLevel = String.valueOf(logger.getEffectiveLevel());
                loggerParent = (logger.getParent() == null ? null : logger.getParent().getName());
            }
    %>
    <tr>
        <td><%=loggerName%>
        </td>
        <td><%=loggerParent%>
        </td>
        <td><%=loggerEffectiveLevel%>
        </td>
        <td class="center">
            <%
                for (int cnt = 0; cnt < logLevels.length; cnt++) {
                    String url = "log4jAdmin.jsp?operation=changeLogLevel&logger=" + loggerName + "&newLogLevel=" + logLevels[cnt] + "&logNameFilter=" + (logNameFilter != null ? logNameFilter : "") + "&logNameFilterType=" + (logNameFilterType != null ? logNameFilterType : "");
                    if (logger.getLevel() == Level.toLevel(logLevels[cnt]) || logger.getEffectiveLevel() == Level.toLevel(logLevels[cnt])) {
            %>
            [<%=logLevels[cnt].toUpperCase()%>]
            <%
            } else {
            %>
            <a href='<%=url%>'>[<%=logLevels[cnt]%>]</a>&nbsp;
            <%
                    }
                }
            %>
        </td>
    </tr>
    <%
        }
    %>
</table>
<h2>
    Revision: 1.0<br/>
    Page Load Time (Millis): <%=(System.currentTimeMillis() - beginPageLoadTime)%>
</h2>
</div>
</body>
</html>

Snarl

Answer 11

You can also add a settings page in the web service to update the log level. This can then be done using ajax. The following example includes login and csrf token:

First, add some form to specify the new log level. Can be improved for example by using a select element.

<form>
    <input type="text" id="logClassName" name="logClassName"/>
    <input type="text" id="logLevel" name="logLevel" />
    <button onclick="submitLogLevelChange(); return false;">Submit</button>
</form>

Then, the request is sent:

function submitLogLevelChange() {
    var className = document.getElementById('logClassName').value;
    var logLevel = document.getElementById("logLevel").value;
    $.ajax({
        // Set up security, see below.
        beforeSend: setHeader,
        type: 'POST',
        // specify the logger to be modified
        url: "/loggers/" + className,
        // specify the new log level
        data: '{"configuredLevel":"' + logLevel + '"}',
        contentType: 'application/json',
        processData: false,
        }).done(function(data, textStatus, jqXHR) {
            if (jqXHR.status === 200) {
                // Happy
            } else if (jqXHR.status === 401) {
                // Logged out or not enough user rights
            } else {
                //Some other problem
            }
        })
        .fail(function(jqXHR, textStatus ) {
            if (jqXHR.status === 200) {
                // Actually was successful, FireFox has some issues...
            } else {
                // Failure
            }
        });
    }

The following function injects the csrf token to the POST request:

function setHeader(xhr) {
  var token = $("meta[name='_csrf']").attr("content");
  var header = $("meta[name='_csrf_header']").attr("content");
  xhr.setRequestHeader(header, token);
}

Spermicide

Answer 12

If you use Log4j 2 for logging you can easily configuration it to set the log level to use based on an environment variable or system property. If you do it this way you won't need to modify the file just because the environment changed.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR" monitorInterval="300">
<properties>
  <property name="LOG_DIR">${sys:user.dir}/logs/</property>
  <property name="log_env">${sys:env:-lab}</property>
  <property name="flow_lab">${sys:flow_match:-ACCEPT}</property>
  <property name="flow_prod">NEUTRAL</property>
  <property name="level_lab">DEBUG</property>
  <property name="level_prod">INFO</property>
</properties>
<MarkerFilter marker="FLOW" onMatch="${flow_${log_env}}" onMismatch="NEUTRAL"/>
<Appenders>
    <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout pattern="%d{ABSOLUTE} %-5level # %class.%method %m%n" />
    </Console>

    <RollingFile name="log4j" fileName="${LOG_DIR}/log4j.txt" filePattern="${LOG_DIR}/archive/log4j.txt.%d{yyyyMMdd_HHmmss}-%i">
        <PatternLayout>
            <MarkerPatternSelector defaultPattern="%d [%t] %-5p %X{requestId, sessionId, loginId, userId, ipAddress, corpAcctNumber} %C{1.}.%M:%L - %m%n">
                <PatternMatch key="FLOW" pattern="%d [%t] %-5p %X{requestId, sessionId, loginId, userId, ipAddress, corpAcctNumber} -------- %C{1.}.%M:%L %msg --------%n"/>
            </MarkerPatternSelector>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="30 MB"/>
        </Policies>
        <!-- A max of 20 will allow 20 files per second with the date pattern specified on the RollingFile declaration.
             Hopefully that is a ridiculous value -->
        <DefaultRolloverStrategy min="1" max="20">
            <Delete basePath="${LOG_DIR}/archive">
                <!-- Nested conditions: the inner condition is only evaluated on files for which the outer conditions are true. -->
                <IfFileName glob="log4j.txt.*">
                    <!-- Only allow 1 GB of files to accumulate -->
                    <IfAccumulatedFileSize exceeds="1 GB"/>
                </IfFileName>
            </Delete>
        </DefaultRolloverStrategy>
    </RollingFile>
</Appenders>
<Loggers>
    <Logger name="com.mycorp.package1" level="${level_${log_env}}" additivity="false">
        <AppenderRef ref="log4j"/>
    </Logger>
    <Logger name="com.mycorp.package2" level="info" additivity="false">
        <AppenderRef ref="log4j"/>
    </Logger>
    <Root level="${level_${log_env}}">
        <AppenderRef ref="log4j" />
    </Root>
</Loggers>

Henceforth

Changing system property requires restart. – Disvalue