Is there a difference between RotatingFileHandler and logrotate.d + WatchedFileHandler for Python log rotation?

Question

Python has its own RotatingFileHandler which is supposed to automatically rotate log files. As part of a linux application which would need to rotate it's log file every couple of weeks/months, I am wondering if it is any different than having a config file in logrotate.d and using a WatchedFileHandler instead.

Is there any difference in how they operate? Is one method safer, more efficient, or considered superior to the other?

Offshoot

logrotate.d depends on the logger closing and reopening its file pointer on each log event. I'm not sure if the standard python file logger does that. – Photometry

@Photometry it doesn't, but there's a good alternative inside logging, see the bottom of my answer – Moneybag

I noticed you edited you question to mention WatchedFileHandler. Did you mean RotatingFileHandler? – Moneybag

no. WatchedFileHandler is used with logrotate. RotatingFileHandler takes care of rotating the logs itself, and replaces logrotate. – Offshoot

Answer 1

What is the intended audience of your program?

If you're creating a desktop application and most users can't be expected to read the logs, you should handle it for them. Not only rotating, but also deleting old ones - you don't want to fill the poor user's hard drive!

On the other hand, if the audience is experienced UNIX sysadmins, you'll have to take a different approach.

Sysadmins will need features you cannot possibly anticipate. Send them by email, write them to append-only storage, you name it. For this audience, it's best if your logging is as flexible as possible. Flexible (in UNIX) means simple - so just write to a file and consider it done.

Also, sysadmins don't want to re-learn how to do logging all over again. Even if you want to provide this kind of feature, make sure the default is reasonable within this assumption.

Finally. tdelaney raised a important point: the standard FileHandler doesn't pay much attention to the file it's writing to. You should use a WatchedFileHandler, which was written specifically for this purpose

Moneybag

That's a good argument for the syslog and NTEventLog handlers which log to the expected places. The app shouldn't have write access to \var\log but can write its logs there anyway via the standard syslog calls. – Photometry

Good point, it doesn't directly my question though. But I am guessing you mean advanced *nix users would expect a log rotating mechanism to be handled by logrotate, if at all? – Offshoot

@Photometry syslog and NTEventLog are indeed a better choice if you want to explicitly write code for different platforms – Moneybag

@Offshoot yes, that's my point. It's not unheard of having a daemon perform rotation, but it's inconvenient. Which aspects of your question do you feel are not addressed here? – Moneybag

@goncalopp your answer only outlines the way a programmer should think about logging, depending the targeted audience. It doesn't answer the question of wether to use python's RotatingFileHandler or the logrotate tool, as both could be used to accomplish the same task. I have accepted it though, since your comment makes things a bit clearer, but you might want to edit it to directly answer the question for future reference. – Offshoot

It's not entirely unlikely that you may want an action taken when a log is rotated (e.g. restarting rsyslog). I'm unsure if this is possible with the python logging framework; it is a core functionality of logrotate. – Longitude

Answer 2

RotatingFileHandler allows a log file to grow up to size N, and then immediately and automatically rotates to a new file.

logrotate.d runs once per day usually. If you want to limit a log file's size, logrotate.d is not the most helpful because it only runs periodically.

Lebron

You should note that logrotate does have a size option, which will rotate a log once it reaches/exceeds a set size.. Reference site: thegeekstuff.com/2010/07/logrotate-examples – Cachepot

But logrotate will still only perform the rotation when it runs during its set time. So there is a lag period where the log can continue to grow. Changing logrotate to run hourly will mitigate this somewhat of course. – Miltiades