Is there a recommended way to get spring boot to JSON format logs with logback

Question

Using spring boot 2.1.1.RELEASE one can seemingly format logs as JSON by providing a logback-spring.xml file as follows:

<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
            <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
            <timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
            <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
                <prettyPrint>true</prettyPrint>
            </jsonFormatter>
        </layout>
    </encoder>
</appender>

<root level="INFO">
    <appender-ref ref="stdout" />
</root>

and adding to the pom.xml

<dependency>
            <groupId>ch.qos.logback.contrib</groupId>
            <artifactId>logback-json-classic</artifactId>
            <version>0.1.5</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback.contrib</groupId>
            <artifactId>logback-jackson</artifactId>
            <version>0.1.5</version>
        </dependency>

indeed leading to messages like:

{
  "timestamp" : "2018-12-11T18:20:25.641Z",
  "level" : "INFO",
  "thread" : "main",
  "logger" : "com.netflix.config.sources.URLConfigurationSource",
  "message" : "To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.",
  "context" : "default"
}

Why?

I'm trialing logz.io which appears to behave more favourably when logs are JSON formatted, some o the shippers struggle with multiline logs like we see in java stack traces and when formatting in JSON it can automatically parse fields like level and message and if there is MDC data it automatically gets that.

I had some not so great experiences with a few of the methods of shipping logs to logzio, like their docker image and using rsyslog without using JSON formatted log messages.

Issues With This Approach

It works ok for console appending, but spring boot provides like logging.file=test.log, logging.level.com.example=WARN, logging.pattern.console. I can indeed import the managed configuration from spring-boot-2.1.1.RELEASE.jar!/org/springframework/boot/logging/logback/base.xml which in turn imports a console-appender.xml andfile-appender.xml`.

An example of the console-appender

<included>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
        </encoder>
    </appender>
</included>

An example of the file appender

<included>
    <appender name="FILE"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
            <pattern>${FILE_LOG_PATTERN}</pattern>
        </encoder>
        <file>${LOG_FILE}</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
            <maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
            <maxHistory>${LOG_FILE_MAX_HISTORY:-0}</maxHistory>
        </rollingPolicy>
    </appender>
</included>

These two are exactly what I need to support spring configuration of the properties, but they don't include the encoder/layout I'd need.

It appears in my initial tests that I can't simple name my appender the same as those and provide my layouts. For example:

<configuration>

    <include resource="org/springframework/boot/logging/logback/base.xml"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
                <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
                <timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
                <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
                    <prettyPrint>true</prettyPrint>
                </jsonFormatter>
            </layout>
        </encoder>
    </appender>



    <root level="INFO">
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>

leads to the message being logged in both JSON and plain text format.

I can indeed just copy and paste the contents of these 3 files into my custom config rather than import them at all. Then I may override what I want to customise.

However, as spring evolves and new releases are made which may add features, I'd be forever forcing myself to keep up, copy and paste the new files and make my changes and test them.

Is there any better way that I can either:

• Just make additive changes to the appenders rather than entirely redefine them, e.g. keep the config from spring but provide my own encoder or layout to be used by those appenders.
• Configure spring to JSON log via properties entirely without any config - I doubt this :S

---

Footnote: logzio do provide a dependency one can import, but I dislike the idea of coupling the logging provider into the code directly. I feel that if the servoce happens to produce JSON logs to stdout or a file, it's easy for any provider to process those and ship them to some destination.

Mikkimiko

Answer 1

I am not using any dependency. Simply, doing it via application.yml, that's all. This solution solves, multiline log issue, too.

logging:
  pattern:
    console: "{\"time\": \"%d\", \"level\": \"%p\", \"correlation-id\": \"%X{X-Correlation-Id}\", \"source\": \"%logger{63}:%L\", \"message\": \"%replace(%m%wEx{6}){'[\r\n]+', '\\n'}%nopex\"}%n"

Padre

Works great for me, thanks. A note - had to remove Correlation-Id, as it is not out of the box in Spring Boot – Altaf

This will break if the log message includes double-quotes. – Discrepant

We are using application.properties file. \\n did not work for us. Instead, this worked for us: %replace(%message){'\n', '\\\\n'}. – Translation

For message that includes double-quotes, this can work: {"message": "%replace(%replace(%message){'\n', '\\\\n'}){'"', '\\\\"'}"} – Translation

@NileshKevlani where should we replace the {"message": "%replace(%replace(%message){'\n', '\\\\n'}){'"', '\\\\"'}"} in given answer. Can you please share whole pattern which includes double-quotes – Nucleoplasm

Hey, in my use case, I was using application.properties file and I was setting this property like this: logging.pattern.console={"message": "%replace(%replace(%message){'\n', '\\\\n'}){'"', '\\\\"'}"} – Translation

Answer 2

I use something like the following, has always worked fine.

Spring Boot recommendation is to name the file logback-spring.xml and place it under src/main/resources/, this enables us to use spring profiles in logback. So in the file below you will see that for LOCAL profile you can log in the standard fashion but for the deployments on the server or a container you can you a different logging strategy.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %5p [YourApp:%thread:%X{X-B3-TraceId}:%X{X-B3-SpanId}] %logger{40} - %msg%n
            </pattern>
        </encoder>
    </appender>
    <appender name="jsonstdout" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <providers>
                <timestamp>
                    <timeZone>EST</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "level": "%level",
                        "service": "YourApp",
                        "traceId": "%X{X-B3-TraceId:-}",
                        "spanId": "%X{X-B3-SpanId:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "message": "%message"
                        }
                    </pattern>
                </pattern>
                <stackTrace>
                    <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                        <maxDepthPerThrowable>30</maxDepthPerThrowable>
                        <maxLength>2048</maxLength>
                        <shortenedClassNameLength>20</shortenedClassNameLength>
                        <rootCauseFirst>true</rootCauseFirst>
                    </throwableConverter>
                </stackTrace>
            </providers>
        </encoder>
    </appender>
    <root level="info">
    <springProfile name="LOCAL">
      <appender-ref ref="stdout" />
    </springProfile>
    <springProfile name="!LOCAL">
      <appender-ref ref="jsonstdout" />
    </springProfile>
  </root>

</configuration>

Trillium

Can you post an example @Chad . I have had no problems. We have about 30 microservices and no issues so far. Best part is a readable stack trace – Trillium

if you logged the string I have a quote ". I think it would break the json because the quote wouldn't be escaped. – Typhoon

Answer 3

Sounds like you need to copy paste with modifications 3 out of 4 files from here https://github.com/spring-projects/spring-boot/tree/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback into your configuration.

The good news is that you don't need to copy paste https://github.com/spring-projects/spring-boot/blob/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml

That can be included like so <include resource="org/springframework/boot/logging/logback/default.xml"/>

That will get you some of spring's default config without copying and pasting

Laura

Answer 4

If you switch to log4j2 using the method specified in the Spring Boot documentation

implementation "org.springframework.boot:spring-boot-starter-log4j2"
modules {
  module("org.springframework.boot:spring-boot-starter-logging") {
    replacedBy("org.springframework.boot:spring-boot-starter-log4j2", "Use Log4j2 instead of Logback")
  }
}

You can have a file log4j2.properties that contains the following.

appender.stdout.type=Console
appender.stdout.name=json
appender.stdout.json.type=JsonTemplateLayout
appender.stdout.json.eventTemplateUri=classpath:LogstashJsonEventLayoutV1.json
appender.console.type=Console
appender.console.name=console
rootLogger.appenderRef.stdout.ref=${env:LOGGING_APPENDER:-json}

I used LOGGING_APPENDER to match my environment level overrides.

Note one odd flaw though... this does not work when you're on JDK17.

UPDATE for Spring boot 3.x, but you still cannot control the logging via env vars

log4j-spring.xml

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration status="WARN">
    <Appenders>
        <Console name="json" target="SYSTEM_OUT" follow="true">
            <JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="INFO">
            <AppenderRef ref="json"/>
        </Root>
    </Loggers>
</Configuration>

And these depenedencies

    runtimeOnly("org.springframework.boot:spring-boot-starter-log4j2")
    runtimeOnly("org.apache.logging.log4j:log4j-layout-template-json")
    modules {
        module("org.springframework.boot:spring-boot-starter-logging") {
            replacedBy("org.springframework.boot:spring-boot-starter-log4j2", "Use Log4j2 instead of Logback")
        }
    }

Laburnum

Answer 5

This approach is for using Logback with one additional dependency

runtimeOnly("net.logstash.logback:logstash-logback-encoder:7.4")

And logback-spring.xml which contains

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>
    <root level="warn">
        <appender-ref ref="${LOGGING_APPENDER:-JSON}"/>
    </root>
    <!-- this silences warnings when the appenders are not active -->
    <logger name="nil">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="JSON"/>
    </logger>
</configuration>

Laburnum