Using MDC in log4j to dynamically name the log file

Question

Is it possible some how to use MDC to name the log file at run time.

I have a single web application which is being called by different names at the same time using tomcat docbase. So i need to have separate log files for each of them.

Freemasonry

Answer 1

This can be accomplished in Logback, the successor to Log4J.

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.

See the documentation for Sifting Appender

The SiftingAppender is unique in its capacity to reference and configure nested appenders. In the above example, within the SiftingAppender there will be nested FileAppender instances, each instance identified by the value associated with the "userid" MDC key. Whenever the "userid" MDC key is assigned a new value, a new FileAppender instance will be built from scratch. The SiftingAppender keeps track of the appenders it creates. Appenders unused for 30 minutes will be automatically closed and discarded.

In the example, they generate a separate log file for each user based on an MDC value. Other MDC values could be used depending on your needs.

Menis

But using Logback would mean that all the logging statements have to be changed right ?` – Freemasonry

Answer 2

This is also possible with log4j. You can do this by implementing your own appender. I guess the easiest way is to subclass AppenderSkeleton.

All logging events end up in the append(LoggingEvent event) method you have to implement.

In that method you can access the MDC by event.getMDC("nameOfTheKeyToLookFor");

Then you could use this information to open the file to write to. It may be helpful to have a look at the implementation of the standard appenders like RollingFileAppender to figure out the rest.

I used this approach myself in an application to separate the logs of different threads into different log files and it worked very well.

Burnight

I am not using RollingFileAppender. Is it possible in FileAppender ? – Freemasonry

I mentioned RollingfileAppender only as example of how to implement an appender. Basically you can do whatever you want in the append method. – Burnight

Answer 3

I struggled for a while to find SiftingAppender-like functionality in log4j (we couldn't switch to logback because of some dependencies), and ended up with a programmatic solution that works pretty well, using an MDC and appending loggers at runtime:

//  this can be any thread-specific string
String processID = request.getProcessID();  

Logger logger = Logger.getRootLogger();

//  append a new file logger if no logger exists for this tag
if(logger.getAppender(processID) == null){

  try{
    String pattern = "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n";
    String logfile = "log/"+processID+".log";

    FileAppender fileAppender = new FileAppender(
        new PatternLayout(pattern), logfile, true);
    fileAppender.setName(processID);

    // add a filter so we can ignore any logs from other threads
    fileAppender.addFilter(new ProcessIDFilter(processID));

    logger.addAppender(fileAppender);
  }catch(Exception e){
    throw new RuntimeException(e);
  }
}

//  tag all child threads with this process-id so we can separate out log output
MDC.put("process-id", processID);

//whatever you want to do in the thread
LOG.info("This message will only end up in "+processID+".log!");

MDC.remove("process-id");

The filter appended above just checks for a specific process id:

public class RunIdFilter extends Filter {

  private final String runId;

  public RunIdFilter(String runId) {
    this.runId = runId;
  }

  @Override
  public int decide(LoggingEvent event) {
    Object mdc = event.getMDC("run-id");

    if (runId.equals(mdc)) {
      return Filter.ACCEPT;
    }

    return Filter.DENY;
  }
}

Hope this helps a bit.

Platonism

How do I write same filter in Log4j 2? – Restraint

@NaiveCoder anywhere you want to start MDC filtering. – Platonism

Answer 4

As of 20-01-2020, this is now a default functionality of Log4j.

To achieve that you just need to use a RoutingAppender with MDC.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" monitorInterval="30">
  <Appenders>
    <Routing name="Analytics" ignoreExceptions="false">
      <Routes>
        <Script name="RoutingInit" language="JavaScript"><![CDATA[
             // This script must return a route name
             //
             // Example from https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender
             // on how to get a MDC value
             // logEvent.getContextMap().get("event_type");
             //
             // but as we use only one route with dynamic name, we return 1

            1
            ]]>
        </Script>
        <Route>
          <RollingFile
            name="analytics-${ctx:event_type}"
            fileName="logs/analytics/${ctx:event_type}.jsonl"
            filePattern="logs/analytics/$${date:yyyy-MM}/analytics-${ctx:event_type}-%d{yyyy-dd-MM-}-%i.jsonl.gz">
            <PatternLayout>
              <pattern>%m%n</pattern>
            </PatternLayout>
            <Policies>
              <TimeBasedTriggeringPolicy/>
              <SizeBasedTriggeringPolicy size="250 MB"/>
            </Policies>
          </RollingFile>
        </Route>
      </Routes>
      <!-- Created appender TTL -->
      <IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
    </Routing>
  </Appenders>
  <Loggers>
    <Logger name="net.bytle.api.http.AnalyticsLogger" level="debug" additivity="false">
      <AppenderRef ref="Analytics"/>
    </Logger>
  </Loggers>
</Configuration>

To known more, see Log4j - How to route message to log file created dynamically.

Garpike