Which Appenders should be used in distributed system? How to configure them?

Question

I am trying to add logging component to distributed system. It is written in AspectJ to avoid chaining current source-code. I use socket appender to send logs, but I'd like to try something more effective.

I've heard I should use JMSAppender and AsyncAppender, but I failed to configure it. Should I create Receiver which gathers logs and pass them to database and to GUI (I use ChainSaw)?

I tried to follow turorial1 and tutorial2 , but they aren't clear enough.

Edit:

In a small demo I've prepared I sent 6 logs for a request (simulation of 3 components)

[2012-08-08 15:40:28,957] [request1344433228957] [Component_A] [start]
[2012-08-08 15:40:32,050] [request1344433228957] [Component_B] [start]
[2012-08-08 15:40:32,113] [request1344433228957] [Component_C] [start]
[2012-08-08 15:40:32,113] [request1344433228957] [Component_C] [end - throwing]
[2012-08-08 15:40:32,144] [request1344433228957] [Component_B] [end]
[2012-08-08 15:40:32,175] [request1344433228957] [Component_A] [end]

Using socket Appender. So my log4j.properties is:

log4j.rootLogger=DEBUG, server

log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=4712
log4j.appender.server.RemoteHost=localhost
log4j.appender.server.ReconnectionDelay=1000

so I run

>java -classpath log4j-1.2.17.jar org.apache.log4j.net.SimpleSocketServer 4712 log4j-server.properties

with configuration

log4j.rootLogger=DEBUG, CA, FA

#
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=[%d] [%t] [%c] [%m]%n

#
log4j.appender.FA=org.apache.log4j.FileAppender
log4j.appender.FA.File=report.log
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
log4j.appender.FA.layout.ConversionPattern=[%d] [%t] [%c] [%m]%n

Then I send my logs from file to Chainsaw:

It is absolutely basic, but I want to learn how to do it better. First of all, I'd like to send logs asynchronously. Then create very simple Receiver, which e.g. can pass logs to a file.

I tried to follow tutorials I listed above, but I failed. So question is: could you provide some example configuration? Example of Receiver.java and log4.properties files?

Finnish

Answer 1

Finally I've found how to configure it. I put 2 files into src folder.

jndi.properties

topic.logTopic=logTopic

and log4j-jms.properties

log4j.rootLogger=INFO, stdout, jms

## Be sure that ActiveMQ messages are not logged to 'jms' appender
log4j.logger.org.apache.activemq=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern= 

## Configure 'jms' appender. You'll also need jndi.properties file in order to make it work
log4j.appender.jms=org.apache.log4j.net.JMSAppender
log4j.appender.jms.InitialContextFactoryName=org.apache.activemq.jndi.ActiveMQInitialContextFactory
log4j.appender.jms.ProviderURL=tcp://localhost:61616
log4j.appender.jms.TopicBindingName=logTopic
log4j.appender.jms.TopicConnectionFactoryBindingName=ConnectionFactory

Then I run my program with VM argument

-Dlog4j.configuration=log4j-jms.properties

and receive logs in class Receiver.java

public class Receiver implements MessageListener {

    PrintWriter pw = new PrintWriter("result.log");
    Connection conn;
    Session sess;
    MessageConsumer consumer;

    public Receiver() throws Exception {


        ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
        Connection conn = factory.createConnection();
        Session sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
        conn.start();
        MessageConsumer consumer = sess.createConsumer(sess.createTopic("logTopic"));
        consumer.setMessageListener(this);
    }

    public static void main(String[] args) throws Exception {
        new Receiver();

    }

    public void onMessage(Message message) {
        try {
            LoggingEvent event = (LoggingEvent) ((ActiveMQObjectMessage) message).getObject();

            DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS"); 
            String nowAsString = df.format(new Date(event.getTimeStamp())); 

            pw.println("["+ nowAsString + "]" + 
                    " [" + event.getThreadName()+"]" +
                    " ["+ event.getLoggerName() + "]" +
                    " ["+ event.getMessage()+"]");
            pw.flush();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Finnish

Answer 2

I would use NFS or CDFS and mount a drive on all the machines. Have each application instance write to a different file. You will be able to find all the logs in one directory (or drive) no matter how many machines you use.

I wouldn't use NFS or CDFS over a global WAN with a high latency e.g. > 50 ms round trip. In this cause I have used JMS (but I didn't use log4j)

Antonina

I need to split logs to different files by request type, not by machine name. Is it possible? – Finnish

Do you need to do this in near real time or can it be processed over night? A file per request could be very inefficient. If you are doing something time sensitive I wouldn't use log4j any way. ;) – Antonina

My idea was: Messages are sent asynchronously by JMS appender to Receiver. Receiver chooses the longeest requests or one request in group of 50, then pass them to ChainSaw and data base. Should't I use log4j in this situation? I'm looking for an easy solution – Finnish

You can use log4j for this with JMS. In my case there reason I was collecting this data was for measuring the time the request passed through the system and the log4j and JMS were affecting the results (slowing them down) If the latency is not a concern, or you are measuring time in many milli-seconds, its not a problem. – Antonina

Answer 3

My two cents.. Whatever you do, make sure that you use asynchronous mechanism to deliver your logs to the receiver, otherwise it will eventually stall your apps. Another point, to deliver logs reliably you should consider a fail over mechanism built into the appender itself - receivers may go offline for short or long time, if you care for the logs, the fail over is definitely required. We have built similar system you describe (sorry for the add), but if you like you can use our appender (look in downloads), it's free and has the sources. There is also a video tutorial. It has fail over and flexible asynchronous mechanism plus a backup fall back.

How many appenders should you use? One appender per jvm will do all right. Config files should probably be per jvm, not sure how you intend to implement the receiver, in any case the appenders need to find your receiver which is usually host port pair at least. Regarding the database, my experience is very sour with RDBMS (we are moving to nosql) but if you don't go above couple of hundred million records, most commercial databases will do with some effort. Not a simple task I must say, took us couple of years to build commercial quality system you just drawn with few skinny rectangles :)

Aroma

It is not simple, for sure :) Definitely not for 2nd year student. LogFaces looks great and I'll be inspired by it, no doubts. Your advice is very "high-level", so I`ll edit my question in a minute to provide more details. I am new to all of technologies I use, definitely need support. – Finnish

Answer 4

I'd recommend syslog and the built in syslog appender. Use TCP for reliable logging (+Asyc appender maybe) or UDP for fire-and-forget logging. I have a rsyslog config if you need.

Misgive

Negative of using syslog appender is that you lose a LOT of the structured information that comes with log4j. If the starting point wasn't log4j certainly would be a good option. – Brightwork

Agree, now, 3.5 years later, We don't use syslog anymore. (logback + logentries appender). – Misgive