Log4Net and logging from parallel instances

Question

I'm using log4net in my project and there is one problem. The major function of the program takes a long time and I use logging to save information about it. I use FileAppender for saving log to file.

My application is on the shared(local) folder and there could be several instances of the application running from one path. In this case I could log information only from the first program, other instances of my applications couldn't log info because log file is locked.

When I'm using "log4net.Appender.FileAppender+MinimalLock" option there are cases of information loss. Not all logs from both instances are saved to file.

How can I solve this problem and log info from parallel instances? Also what about performance degradation when I use "MinimalLock" option?

Thanks. Hope for your help.

Hennahane

could you write the log to a database instead of a file? – Goudy

Main reason to use file is that i need to exclude all DB connections. – Hennahane

Answer 1

Simply include the process id of the application in the log file name. Different instances of your app will then log to different files. Here is an example:

<appender name="MyRollingFileAppender" type="log4net.Appender.RollingFileAppender">
  <file type="log4net.Util.PatternString">
    <conversionPattern value="log_%processid.log" />
  </file>
<!-- ... -->

Yvette

Note to others: Be very careful not to miss the type="log4net.Util.PatternString" attribute, or it will not work! – Homunculus

Also, from https://mcmap.net/q/904750/-process-id-not-coming-in-log4net-log-file-name, <file type="log4net.Util.PatternString" value="c:\\testLogs\\TwoProjects-[%processid].txt" /> also works (i.e., no need for <conversionPattern ... />. – Homunculus

Won't there be collisions? Wouldn't it be better to use the time that the process started plus process id? – Fescennine

Answer 2

I think it's a typical situation when centralized logging solution is desirable. Instead of worrying about files and suffer from performance bottlenecks, I'd rather pump log statements asynchronously to some remote service which will take care of storing and handling the logs. Have a look at this log aggregator called logFaces, it was designed with the purpose of decoupling applications from managing their logs. It should work with standard log4net UDP appender and will partition your log data per applications, hosts, threads, etc.. while letting you create log files any time when they really needed.

Disclosure: I am the author of this product.

Curcio

Seem overkill for simple desktop app that allows multiple instances. – Feller

Answer 3

You can create a custom appender that opens the file to write and then closes it. Should it hit a locked file, it could pause and retry a small number of times.

In the custom appender you could also open the file in shared write mode that would allow multiple writers, but this won't prevent pieces of log lines from being merged together.

If you aren't writing a lot of data, the open/close mechanism, listed above, is probably your best option. Note, because of the constant opening and closing of the file, you could see a noticeable performance impact if you are logging a lot of data.

A more complicated mechanism, but one that could provide a high performance logging path: Write a logging service that receives log lines via TCP or UDP. The service would be responsible for buffering data and writing it to disk. We've used this approach in the past (not via Log4Net, but as a general solution) to improve log writing efficiency.

Octavie

Could you provide some example how I can create custom file appender. – Hennahane

Check out: karlagius.com/2008/01/02/writing-a-custom-appender-for-log4net – Octavie

Answer 4

Perhaps you log to different files from each instance? Otherwise, you'd probably need to set up a separate process that is dedicated to logging. Each instance of your program would send it's log messages there and it would take care of appending it to the file. This could be accomplished using SocketAppender, perhaps. Also, I find that breaking up log output into chucks using RollingFileAppender is much easier to deal with.

Nonagon

Answer 5

Definitely consider creating different log files for each process, perhaps with unique filenames generated using a timestamp.

Karakul

How can I specify logfile name from runtime? I thought that is available only from config. file? – Hennahane

You're right, I shot from the hip... timestamp works, but configuring a new log file name for a new process gets tricky. Answer amended, thanks. – Karakul

Answer 6

Instead of MinimalLock using InterProcessLock can reduce the data loss incase of multiple processes accessing single logfile.

log4net.Appender.FileAppender+InterProcessLock

Rundgren