Using log4j2, how to log key value pairs

Question

I need to create logs with key value pairs as below. Is there any support in PatternLayout to do this for the static fields in a thread like log_level, class_name, event_id etc with the log4j2.xml.

Sample log:

2014-06-18 11:57:46,719 log_level="INFO" class_name="com.abc.dgl.App:main(158)" name="Application start event" event_id="b88f7ea0-4cb1-438f-a728-ac7c2bdac578" app="Test App" severity="info" action="loaded sfor file processing" desc="props was read and loaded" result="success" reason="abc" transaction_id="b88f7ea0-4cb1-438f-a728-ac7c2bdac578"

Oil

Answer 1

Yes, this is possible.

You can either use a MapMessage, which is supported by the map (or K) conversion pattern of PatternLayout: an example layout pattern would be "%-5p [%t]: %m %map%n".

Logging a MapMessage looks like this: Map<String,String> myMap = getMyMap(); Logger.debug(new MapMessage(myMap));

Another way to do this is to use the ThreadContext map. This is supported by the mdc (or X) conversion pattern of PatternLayout. Example pattern: "%-5p [%t]: %m %mdc%n". A common usage is putting a user ID in the Thread Context Map when a user logs in, and having this user ID shown in all log messages emitted by that thread until the user logs out.

Instead of logging the whole map you can also log specific keys only by specifying the key in the layout pattern: e.g. "%-5p [%t]: %m %mdc{userID}%n".

Scone

Thanks a ton. I used a combination of the ThreadContext, map and Pattern to address the fixed vs the dynamic part of logs. I want the logs to be ingested to Splunk and splunk likes key/value pairs. – Oil

Pattern Layout: <PatternLayout pattern="%d log_level='%p' thread_name='%t' event_id='%X{event_id}' transaction_id='%X{transaction_id}' app_name='%X{app_name}' class_name='%c' method_name='%M(%L)' %m %n"/> – Oil

ThreadContext: ThreadContext.put("event_id", UUID.randomUUID().toString()); ThreadContext.put("transaction_id", UUID.randomUUID().toString()); ThreadContext.put("app_name", "splunk sample app"); – Oil

Map<String,String> myMap = new HashMap<String,String>(); myMap.put("card_token", "abcdefg"); myMap.put("action", "start of sample app"); myMap.put("desc", "start of sample splunk app"); logger.error(new MapMessage(myMap)); – Oil

myMap.clear(); myMap.put("action", "inside the loop"); myMap.put("desc", "inside the loop of the sample app"); myMap.put("loop_count", String.valueOf(i)); logger.error(new MapMessage(myMap)); – Oil

Does this look ok for a general logging strategy? I want to implement log4j2 Aysnclogger with Splunk compatable logging for a wide range of applications. – Oil

Final log: 2014-06-18 19:06:22,756 log_level='ERROR' thread_name='main' event_id='e8e81656-92cb-4a21-8ffd-3f2bd9a368ba' transaction_id='a30486cc-915c-43f4-9ae1-c98d45fd86ae' app_name='splunk sample app' class_name='com.abc.splunk.asyncsample.AppAsyncMain' method_name='main(47)' action="start of sample app" card_token="abcdefg" desc="start of sample splunk app" – Oil

I am not that familiar with Splunk, but two comments on your code/config: to use a MapMessage, your PatternLayout should include a %K or %map conversion pattern. Also, if you are logging asynchronously then it is not a good idea to re-use your myMap instance. Better to create a new Map for every log MapMessage. The reason is that if you call myMap.clear() in your application thread before the logging thread has had a chance to write the message to disk, you erase the message contents before it is written... – Scone

good point on Map.clear(), will do that. Using %m gives me what i was expecting. When i use %map or %K, it add a curly bracket in the log. Is there an issue if i use %m ? – Oil

2014-06-19 13:33:07,279 log_level='ERROR' thread_name='main' event_id='541d9070-30ae-4450-976d-d16ba01fdae1' transaction_id='25dbb726-880f-470e-9229-84aec3b92f9e' app_name='splunk sample app' class_name='com.abc.splunk.asyncsample.AppAsyncMain' method_name='main(62)' {action=end of sample app, desc=end of sample splunk app} – Oil

No issue, I was just looking at the docs. If you are happy with the results for %m, no problem. – Scone

That answers it. Thank you Remko. – Oil

Answer 2

Yes. Use %kvp as the pattern in your PatternLayout.

Unfortunately, it isn't very configurable. This was added in 1.3.0 according to the source.

Boulogne