Parsing Log4j Layouts from Log Files [closed]

Question

Is there any open source tool out there that can read all Layout implementations of Apache Log4j 1.2.x into something meaningful (e.g., Log4j's own LogEvent objects)? So far I haven't found an application (Chainsaw included) that can do that.

I am looking for something straightforward, that converts from a text file (plain or XML), into a Java object, given the file name and the pattern parameters (e.g., the format pattern in PatternLayout). If it has a clear API that makes it easy to integrate into an application, even better, but this is not necessary.

Surly

You have described what you want it to do, but not what problem you are trying to solve. – Brahma

Why are you looking for a different implementation than Log4j itself? Isn't it possible to simply use the log4j libs? – Hoedown

Answer 1

Here is the code Chainsaw uses to convert patternlayouts into something useful, using only log4j APIs. In the case of the latest developer snapshot of Chainsaw, it is used to build a Chainsaw config directly from log4j xml or properties file fileappender entries.

Use the converters and fields populated by PatternParser#parse to do what you want. The PatternParser class is included in the log4j 'extras' companion.

From http://svn.apache.org/repos/asf/logging/chainsaw/trunk/src/main/java/org/apache/log4j/chainsaw/LogFilePatternLayoutBuilder.java

public static String getLogFormatFromPatternLayout(String patternLayout) {
    String input = OptionConverter.convertSpecialChars(patternLayout);
    List converters = new ArrayList();
    List fields = new ArrayList();
    Map converterRegistry = null;

    PatternParser.parse(input, converters, fields, converterRegistry, PatternParser.getPatternLayoutRules());
    return getFormatFromConverters(converters);
}

If you actually want something that will read a log file conforming to a specific PatternLayout and generate log4j LoggingEvents, see LogFilePatternReceiver. Chainsaw uses the getLogFormatFromPatternLayout method to convert the patternLayout into the LogFormat required by the receiver.

By the way, the developer snapshot of Chainsaw contains a lot of additional new features - if you want to try it out, you can get it here:

http://people.apache.org/~sdeboy

Reeves

Thanks very much for the very useful pointers, which do save time. My impression, too, is increasingly becoming that I can only do what I am looking for via resorting to the source code of Log4j and Chainsaw. I wish it were already there, but at least it is a starting point. – Surly

Since you have a configuration file with a patternlayout and a log file, the helpers in LogFilePatternLayoutBuilder plus a LogFilePatternReceiver will give you LoggingEvents from that log file. You just have to connect them together. – Reeves

Answer 2

I'm developing open source log viewer OtrosLogViewer. You can use it as library in your application. Take a look at this example.

//Define log4j properties
Properties p = new Properties();
p.put("type", "log4j");
p.put("pattern", "TIMESTAMP LEVEL [THREAD]  MESSAGE");
p.put("dateFormat", "yyyy-MM-dd HH:mm:ss,SSS");
Log4jPatternMultilineLogParser logParser = new Log4jPatternMultilineLogParser();    
LogImporterUsingParser importerUsingParser = new LogImporterUsingParser(logParser);
importerUsingParser.init(p);

//
ParsingContext context = new ParsingContext();

//Create log collector, it capture all parsed log events
ProxyLogDataCollector dataCollector = new ProxyLogDataCollector();

//Create input stream from file
InputStream in = new FileInputStream("log4j/log4j.txt");

//parse log file
importerUsingParser.importLogs(in, dataCollector, context);

Hooky

The link associated with this example is not working. – Homy

Answer 3

I have come across your project and it is a nice job, however ideally I would like something that takes a native Log4j conversion pattern (e.g., %-5p [%t]: %m%n) and a file formatted using that pattern, and reads the corresponding information.

Essentially, I am looking for the reverse of the Layout.format(LoggingEvent) method of the base Layout class of Log4j. This method produces a String that is stored in the log file, so the reverse operation would be able to parse that file and create LoggingEvent objects, or something similar (something like a deserialization procedure).

Thanks!

EDIT: The conversion is straightforward, by just doing

LogFilePatternLayoutBuilder.getLogFormatFromPatternLayout(conversionPattern);

via the Chainsaw class org.apache.log4j.chainsaw.LogFilePatternLayoutBuilder.

Surly

By the way, this referred to the nice OtrosLogViewer project. :-) – Surly