spring kafka - set consumer log level manually (spring 5)

Question

We are working on spring project and integrated with spring-kafka recently. It's working as expected where we are able produce and consume messages properly.

But as we moved project to PROD, we are getting DEBUG level log statements and amount of statements overwhelmed. We are trying to set logging levels to WARN or ERROR specifically for Kafka logs.

In process we added logback.xml (added in main/resources) by following below article but no luck, also tried other solutions we came across in internet. (adding log4j.properties and log4j2.xml)

• https://www.baeldung.com/logback

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration name="default">

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
       <encoder>
          <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="org.apache.kafka"                 level="error" />
    <logger name="org.apache.kafka.common.metrics"  level="error"/>
    <logger name="org.apache.kafka.clients"         level="error"/>
    <logger name="org.apache.kafka.clients.consumer.internals"          level="error"/>
    <logger name="org.apache.kafka.clients.consumer.internals.Fetcher"  level="error"/>
     
    <root level="error" name="org.apache.kafka" additivity="false">
        <appender-ref ref="STDOUT" />
    </root> 
    
</configuration>

Sample logs:

[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=client-01, groupId=publisher] Cluster ID: Udhjf-fdbfds
[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=client-01, groupId=publisher] Discovered group coordinator 127.....0 (id: 65654654654 rack: null)
[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator -  [Consumer clientId=client-01, groupId=publisher] (Re-)joining group
[WebSphere non-WLM Dispatch Thread t=008abcf0] WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.
[WebSphere non-WLM Dispatch Thread t=008abcf0] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.7.1
[WebSphere non-WLM Dispatch Thread t=008abcf0] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 616tyu85d0d567566

Anyone came across similar situation or any leads will be really helpful. Thanks in advance. :)

Kentkenta

Have you tried using Boot's logging config in application.properties/yml? logging.level.org.apache.kafka=warn ? docs.spring.io/spring-boot/docs/current/reference/html/… – Bereave

@GaryRussell we are working with spring (not spring boot), so NO. But tried log4j.logger.org.apache.kafka=warn in log4j.proerties. could you please suggest any other way with spring.. – Kentkenta

Which logging library are you actually using? logback XML only works if you have slf4j-log4j12 bridge and logback dependency. log4j2 configs only work if you have that and also the slf4j-log4j12 bridge. Out of the box, Kafka uses log4j.properties (I assume your post as a typo and is missing the j). You may also need to modify your websphere logging configs – Diddle

@Diddle we are using mentioned dependencies (logback-core, slf4j-api, logback-classic) in article, will add slf4j-log4j12 bridge and update. While testing with log4j.properties we are having dependencies of log4j-core, log4j-api, log4j-slf4j-impl (2.4.1). Please let me know websphere logging config details we need any. – Kentkenta

Those last three would pick up a log4j2 file, not log4j. The file shown in the question is for logback, though – Diddle

You can debug logback.xml by adding debug="true" to the opening <configuration>. With log4j2 it's status="debug" logging.apache.org/log4j/2.x/manual/configuration.html – Bereave

@Kentkenta If you found any solution please post it as answer. I am also facing the same issue. – Thebaine

@ParthaSarathiGhosh Please check posted answer which helped us to resolve the issues. – Kentkenta

logging.level.org.apache.kafka: INFO @GaryRussell, It’s work for me, using Spring Boot 3.2.6 in application.yaml, thanks 🙏🏼 – Works

Answer 1

Late answer, but might be helpful for someone.

We did few changes in same logback.xml file and also added log4j2.xml in order to make it work in our application. Below are the changes from both files, please give it a try.

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration name="default">
   <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
     <encoder>
        <pattern> %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
     </encoder>
  </appender>
  
  <logger name="org.apache.log4j.xml"            level="error"/>
  <logger name="org.apache.http"                 level="error" />
  <logger name="org.apache.http.wire"            level="error"/>
  <logger name="org.apache.commons.httpclient"   level="error"/>
  
  <logger name="org.springframework" level="error">
      <appender-ref ref="STDOUT" />
  </logger> 
  
  <logger name="org.springframework.kafka" level="error" additivity="false">
      <appender-ref ref="STDOUT" />
  </logger>

  <logger name="org.apache.kafka" level="info" additivity="false">
      <appender-ref ref="STDOUT" />
  </logger>
   
  <root level="error">
      <appender-ref ref="STDOUT" />
  </root> 

</configuration>

log4j2.xml

<?xml version="1.0" encoding="UTF-8" ?>  
<!DOCTYPE log4j:configuration PUBLIC "-//log4j/log4j Configuration//EN" "log4j.dtd">  
<log4j:configuratio xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">  

   <appender name="consoleAppender" class="org.apache.log4j.ConsoleAppender">  
       <layout class="org.apache.log4j.PatternLayout">  
          <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />  
       </layout>  
   </appender>  

   <appender name="fileAppender" class="org.apache.log4j.RollingFileAppender">  
       <param name="append" value="true" />  
       <param name="maxFileSize" value="10MB" />  
       <param name="maxBackupIndex" value="10" />  
       <param name="file" value="${catalina.home}/logs/myStruts1App.log" />  
       <layout class="org.apache.log4j.PatternLayout">  
          <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />  
       </layout>  
   </appender>  

   <logger name="org.springframework">
      <level value="error" />
   </logger>

   <logger name="org.springframework.kafka">
      <level value="error" />
   </logger>

   <logger name="org.apache.http">
      <level value="warn" />
   </logger>
   
   <logger name="org.apache.http.wire">
      <level value="warn" />
   </logger>

   <logger name="org.apache.commons.httpclient">
      <level value="warn" />
   </logger>

   <logger name="org.apache.kafka">
      <level value="error" />
   </logger>

   <root>  
       <!-- DEBUG level for all other loggers -->
       <priority value="debug" />  
       <appender-ref ref="consoleAppender" />  
       <appender-ref ref="fileAppender" />  
   </root>  

</log4j:configuration>  

We can observe few changes here related to addivity, try to check below articles to know more about it.

• https://newsbasis.com/what-is-additivity-in-logback/

• Logback logger logging twice

Kentkenta