What is a good size (in bytes) for a log file?

Question

I am using the python logging modules RotatingFileHandler, and you can set the maximum size of each log file. What is a good maximum size for a log file? Please give your answer in bytes.

Virtuoso

Answer 1

My default logging setup:

RotatingFileHandler(filename, maxBytes=10*1024*1024, backupCount=5)

Mammalogy

In other words: 10 MiB. – Polyhistor

copy this one, maxBytes=10485760 – Groovy

I love this answer because 10 MiB is not really a special value, but finding the best value is so specific and complicated. 10 MiB is as good as it gets to start playing around, and may as well become a Schelling point only for having been posted here :) – Franco

it should be noted that often building a value is far more often more useful that straight up using the resulting value. – Luker

Answer 2

As the other answers have said, there is a no hard and fast answer. It depends so much on your app and your environment. Here's some guidelines I use.

For a multi-user app on a typical server: Configure your logging to generate no more than 1 or 2 entries per user action for production, and then rotate it daily. Keep as many days as you have disk space for, or your data retention/privacy policies allow for. If you want auditing, you probably want a separate solution.

For a single-user app: Try and keep enough information to diagnose anything weird that might happen. No more than 2 or 3 entries per user action though, unless you are doing batch operations. Don't put more than 2MB in a file, so the user can email it you. Don't keep more than 50MB of logs, because it's probably not your space you are wasting here.

Caswell

Good point about emailing logs, that actually tipped the scales on a decision I had to make. Thanks. – Clone

Now Gmail apparently does up to 25MB. 2009, the good old days :) – Sailesh

Answer 3

Size isn't as important to me as dividing at sensible chronological points. I prefer one log file per day, however, if the file won't open with any notepad program you have at your disposal, it is too big and you might want to go with hourly logs.

Girgenti

But don't use the Windows notepad.exe as a measurement tool, because it's really bad at opening anything even slightly largeish (at least up to Windows XP, maybe that got better). – Ofris

Joachim: Still, if that's all you have then you'd rather want a log file you can open. – Tisbee

Answer 4

It completely depends on the external variables of the system. For instance:

• Are you running on an embedded device whose only external storage is a 1MB SD card, or do you have full access to a 1TB hard drive?
• Are you logging each time you enter/exit a function, or are you only logging one or two caught exceptions throughout the whole system?
• Is the purpose of these logs to be sent back to the developer for support? A 1kb log file isn't going to help you much, but you probably don't need 200MB of logs for a single support issue.

Without these kinds of details, there is no good answer to your question (and there might not be a good answer even with these details).

Osteoblast

I'm logging each time I enter/exit a function in debug mode. I find it useful so in that case what should I set the size of the log file? Functions are crud functions for database via SQLAlchemy. – Hedger

@Ivan: Since it's running in debug mode, you probably have some room to tweak settings as necessary. Start out with a value like 250k, and if your application is too database-intensive to fit its log in there, increase the size as necessary. – Osteoblast