Wildcard pattern for RoutingAppender of Log4j2

Question

I am trying to use the new RoutingAppender of Log4j2 to route the different logs based on the MDC (ThreadContext in Log4j2). What I want to do is the following:

• If MDC map has $contextId -> Append to $contextId appender (specific log)
• If MDC does not have $contextId -> Append to main appender (general log)

I want to achieve this using a wildcard pattern in the tag and then filter using the key parameter in the for contextId (${ctx:contextId}) and using the default (without key paramenter) for the main appender, however I don't know which value is that wildcard.

Any help is appreciated, maybe I am approaching this from the wrong path. I have been reading about Filters but don't seem to work as I want.

Thanks!

Kaon

Answer 1

Thanks for the link Remko, I have found a temporary solution until that feature gets improved from the guys of Log4j2. The solution is using both RoutingAppender and Filters. This is how my log4j2 config looks like (I have properties defined but I am not showing them here):

<appenders>
    <appender name="applicationAppender" type="RollingFile" fileName="${logFileName}" filePattern="${logFileNamePattern}" bufferedIO="true" immediateFlush="true" append="true">
        <layout type="PatternLayout" pattern="${logPattern}" />
        <Policies>
            <TimeBasedTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="${logFileSize}" />
        </Policies>
        <DefaultRolloverStrategy max="${logFileCount}" />
    </appender>

    <Routing name="contextSpecificAppender">
        <Routes pattern="$${ctx:contextId}">
            <Route>
                <appender name="Rolling-${ctx:contextId}" type="RollingFile" fileName="logs/${ctx:contextId}.log" filePattern="${logFileNamePattern}" bufferedIO="true" immediateFlush="true" append="true">
                    <layout type="PatternLayout" pattern="${logPattern}" />
                    <Policies>
                        <TimeBasedTriggeringPolicy />
                        <SizeBasedTriggeringPolicy size="${logFileSize}" />
                    </Policies>
                    <DefaultRolloverStrategy max="${logFileCount}" />
                </appender>
            </Route>
        </Routes>
    </Routing>
</appenders>

<loggers>
    <root level="info">
        <appender-ref ref="contextSpecificAppender">
            <ThreadContextMapFilter onMatch="DENY" onMismatch="ACCEPT">
                <KeyValuePair key="contextId" value="" />
            </ThreadContextMapFilter>
        </appender-ref>
        <appender-ref ref="applicationAppender">
            <ThreadContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
                <KeyValuePair key="contextId" value="" />
            </ThreadContextMapFilter>
        </appender-ref>
    </root>
</loggers>

What I do it is calling ThreadContext.put("contextId", "") or ThreadContext.put("contextId", "something") depending on what appender I want to log. I hope the wildward feature gets implemented soon, but for the meantime, this solution is enough for me.

Thanks!

Kaon

I found a solution so that you don't have to use the filter in junction with toggling the ThreadContext key from populated to empty) by using two routes. Like this example explains logging.apache.org/log4j/2.x/faq.html#separate_log_files You can use a the key attribute on route to create a route that is chosen if ThreadContext has no value for the key. I then use this route to reference my primary appender. If you set up this second "route" you do not have to use a ThreadContextMapFilter at all. Furthermore you will only need one appender-ref in your root logger. – Constructive

Answer 2

Thanks hveiga for following up and posting your solution, it was helpful. I wanted to say you can avoid your filter solution by adding a second 'route' that routes all messages with no value for your routing key as explained here: http://logging.apache.org/log4j/2.x/faq.html#separate_log_files

So your updated log4j config would look like this.

<appenders>
    <appender name="applicationAppender" type="RollingFile" fileName="${logFileName}" filePattern="${logFileNamePattern}" bufferedIO="true" immediateFlush="true" append="true">
        <layout type="PatternLayout" pattern="${logPattern}" />
        <Policies>
            <TimeBasedTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="${logFileSize}" />
        </Policies>
        <DefaultRolloverStrategy max="${logFileCount}" />
    </appender>

    <Routing name="contextSpecificAppender">
        <Routes pattern="$${ctx:contextId}">
            <Route>
                <appender name="Rolling-${ctx:contextId}" type="RollingFile" fileName="logs/${ctx:contextId}.log" filePattern="${logFileNamePattern}" bufferedIO="true" immediateFlush="true" append="true">
                    <layout type="PatternLayout" pattern="${logPattern}" />
                    <Policies>
                        <TimeBasedTriggeringPolicy />
                        <SizeBasedTriggeringPolicy size="${logFileSize}" />
                    </Policies>
                    <DefaultRolloverStrategy max="${logFileCount}" />
                </appender>
            </Route>
            <Route ref="applicationAppender" key="$${ctx:contextId}">
            </Route>
        </Routes>
    </Routing>
</appenders>

<loggers>
    <root level="info">
        <appender-ref ref="contextSpecificAppender"/>
    </root>
</loggers>

And in your application, you can just set the ThreadContext by calling ThreadContext.put("contextId", "something") and clear it when you are done by calling ThreadContext.clear() OR ThreadContext.remove("contextId")

Lastly, I used the

<RollingFile>

element (like the examples linked above) instead of

<appender type="RollingFile"> 

element you used. I believe this is preferred when you migrate to log4j2 from log4j.

Constructive

Answer 3

I was unhappy with the solution to define the fallback route with the trick described in https://issues.apache.org/jira/browse/LOG4J2-326 and http://logging.apache.org/log4j/2.x/faq.html#separate_log_files, because this forces me to duplicate the appender configuration contained in the routes. I do not need different appender configuration for the default route, but just a proper file name for the general log.

Given that the default property map is looked up for an property which is undefined in its context, see https://logging.apache.org/log4j/2.x/manual/configuration.html#PropertySubstitution, I think the most straightforward way is just to define the default, e.g.

<Properties>
    <Property name="fruits">any_fruit</Property>
</Properties>

and in case the thread context does not have ${ctx:fruits} the "any_fruit" is taken.

Murrumbidgee

Answer 4

Creating Dynamic Multiple log files using RoutingAppender in lo4j2.properties

status=debug
name=PropertiesConfig

filter.threshold.type=ThresholdFilter
filter.threshold.level=debug

appenders=routing
appender.routing.type=Routing
appender.routing.name=Routing
appender.routing.routes.type=Routes
appender.routing.routes.pattern=$${ctx:keyname}

appender.routing.routes.route.type=Route

appender.routing.routes.route.rolling.type=RollingFile
appender.routing.routes.route.rolling.name=RollingFile
appender.routing.routes.route.rolling.fileName=${ctx:keyname}.log
appender.routing.routes.route.rolling.filePattern=${ctx:keyname}-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz
appender.routing.routes.route.rolling.layout.type=PatternLayout
appender.routing.routes.route.rolling.layout.pattern=%m%n
appender.routing.routes.route.rolling.policies.type=Policies
appender.routing.routes.route.rolling.policies.time.type=TimeBasedTriggeringPolicy
appender.routing.routes.route.rolling.policies.time.interval=2
appender.routing.routes.route.rolling.policies.time.modulate=true
appender.routing.routes.route.rolling.policies.size.type=SizeBasedTriggeringPolicy
appender.routing.routes.route.rolling.policies.size.size=1KB
appender.routing.routes.route.rolling.strategy.type=DefaultRolloverStrategy
appender.routing.routes.route.rolling.strategy.max=5

appender.routing.routes.route2.type=Route

appender.routing.routes.route2.key=P:/TestLogging/specialspecial
#appender.routing.routes.route.ref=Routes

appender.routing.routes.route2.rolling.type=RollingFile
appender.routing.routes.route2.rolling.name=RollingFile
appender.routing.routes.route2.rolling.fileName=${ctx:keyname}.log
appender.routing.routes.route2.rolling.filePattern=${ctx:keyname}-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz
appender.routing.routes.route2.rolling.layout.type=PatternLayout
appender.routing.routes.route2.rolling.layout.pattern=%d %p %C{1.} [%t] %m%n
appender.routing.routes.route2.rolling.policies.type=Policies
appender.routing.routes.route2.rolling.policies.time.type=TimeBasedTriggeringPolicy
appender.routing.routes.route2.rolling.policies.time.interval=2
appender.routing.routes.route2.rolling.policies.time.modulate=true
appender.routing.routes.route2.rolling.policies.size.type=SizeBasedTriggeringPolicy
appender.routing.routes.route2.rolling.policies.size.size=1KB
appender.routing.routes.route2.rolling.strategy.type=DefaultRolloverStrategy
appender.routing.routes.route2.rolling.strategy.max = 5
loggers=routing
logger.routing.level=debug
logger.routing.name=com.infy.demo
logger.routing.additivity=false
logger.routing.appenderRef.routing.ref=Routing
rootLogger.level=debug





public class TestLog4j2Logging {

private static final Logger log = LogManager.getLogger(TestLog4j2Logging.class);


public static void testMsg()
{
    
    String[] arr = {"Msg1111111111111","Msg222222222222222","Msg3333333333333","Msg44444444444"};
    for (String string : arr) {
        log.info(string);
        log.error(string);
        ThreadContext.remove("keyname");
        ThreadContext.put("keyname", "P:/TestLogging/specialspecial");
        log.debug(string);
    }
}


public static void main(String[] args) throws FileNotFoundException, IOException
{
    TestLog4j2Logging testLog4j2Logging = new TestLog4j2Logging();

    ThreadContext.put("keyname","P:/TestLogging/rollingtest");
    Configurator.initialize(null, "./properties/log4j2.properties");
    
    
    TestLog4j2Logging.testMsg();
    System.out.println("Messages are getting Logged");
}

}

Gramicidin