viability of c++ logging library for asynchronous capturing of high throughput data?

Question

I am working with a real time system written in C++. We are looking to use either boost or pantheios for logging. The system has some standard logging requirements which I'm confident can be met by either framework, but in addition we want to be capable of logging all input captured by this system. This input will be captured by multiple threads, including some threads that have real-time constraints and can not afford significant delays from inefficient logging. This should result in a high throughput of data to be logged.

I primarily want to know whether either framework can be trusted to manage such high throughput logging from multiple threads without delaying my time critical threads. In addition we may need to do some data scrubbing which would require adding some sort of hook which is capable of identifying the capture inputs which have secure data, run our data scrubbing hook, and maintain a buffer containing mappings of values that were already scrubbed.

I believe both logging platforms can do this, but it's not clear to me with a quick glance at their API. Can anyone who has used either of these logging tools give me some feedback on how efficient they are in this context, how easy it would be to implement what I described, or their preference between the two logging frameworks? Really any information would useful.

Thanks

Scot

Answer 1

I am using the Boost logging library written by Jon Torjo and this one offers to do all writing to log files from a dedicated thread, so you have no I/O delays on the thread doing the logging. This has the disadvantage thought, that when the system crashes some log statements might not be logged, as it makes use of an internal queue.

But generally this library is performing very well, gives you a lot of different options and I think could be a good option for you, if you're willing to sacrifice on messages.

If that is not an option, you'll have to do I/O from the thread needing to log, which is really not ideal on a real-time system.

If you're running on Windows, you know it is not a RT OS, right?

Osteotomy

The library you mentioned is NOT a boost library, it went through the review process but was unanimously rejected by the Boost community, as for Torjo he hasn't been heard of since on the Boost MLs – Bastia

Answer 2

I have done logging in big multi-threaded situations. My experience was:

• Logging is there for developer benefit only. Nobody else will read or understand log files.

• Decouple the generation of the log messages and how they get logged.

• If you want to make the generation of log messages more efficient, use a context first that is quick, check if anything is logging this context and if not do not generate. Otherwise generate the message. (The most common use of this technique is a "level" which can be "debug" "info" etc. and if nothing is logging to that level we do not create the message).

• Each use case should have a special id generated that remains with that use case and everything logged from it will display this use-case id.

• Also log the thread-id that generates the message.

• Use a separate thread to do the logging than the ones that generate the message. (boost's logging library does it this way)

• Beware of going "macro-mad" although light macros to add in things like FILE and LINE automatically are ok.

Porosity

Answer 3

You may wish to consider http://www.logog.org . It sounds like you're dealing with many of the same problems I was dealing with when I wrote it. I originally wrote it for logging a multithreaded audio DSP engine, which of course is the essence of time-critical rendering.

See also https://stackoverflow.com/questions/696321/best-logging-framework-for-native-c

Numinous