How to create a new log file every time the application runs?

Question

Currently, this is what I have (testlog.py):

import logging
import logging.handlers

filename = "example.log"

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")

handler = logging.handlers.RotatingFileHandler(filename, mode = 'w', backupCount = 5)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger.addHandler(handler)

ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)

for i in range(10):
   logger.debug("testx") #where I alternate x from 1 thru 9 to see output

It currently successfully prints out to the console and to example.log, which is what I want.

Every time I run it, it makes a new file and replaces the old example.log like so:

• run with logger.debug("test1") - example.log will contain test1 10 times like it should.

• run with logger.debug("test2") - it rewrites example.log to contain test2 10 times.

• etc...

However, I would like for the code to make a new log file every time I run the program so that I have:

example.log
example.log1
example.log2 
...
example.log5

In conclusion, I'd like for this file to print the log message to the console, to the log file, and I would like a new log file (up to *.5) whenever I run the program.

Tangram

SOLVED! all I did was add logging.handlers.RotatingFileHandler.doRollover(handler) at the end – Tangram

Don't do it at the end - if your script fails mid-point end doesn't reach the end you won't be getting the new log file on the next start. Do it when initializing your logging. – Bulbar

So symptomatic of the logging module that everything is so complicated. And, in this case, so un-obvious (no, one would not expect it on RotatingFileHandler's API). Upvoted, had same Q. Not to mention that I have dict/json-based logging setup, so now I need to figure out how to translate answers here to that format. grrr. – Whaling

Answer 1

logging.handlers.RotatingFileHandler rotates your logs based either on size or on date, but you can force it to rotate using RotatingFileHandler.doRollover() so something like:

import logging.handlers
import os

filename = "example.log"

# your logging setup

should_roll_over = os.path.isfile(filename)
handler = logging.handlers.RotatingFileHandler(filename, mode='w', backupCount=5)
if should_roll_over:  # log already exists, roll over!
    handler.doRollover()

# the rest of your setup...

Should work like a charm.

Bulbar

it creates empty log files on each run – Hurty

set delay = True, it avoids creation of empty log files – Ramage

How do I add a timestamp instead of .log.1, .log.2 ? – Knurled

@Knurled RotatingFileHandler uses the base filename to determine the rotated log name, you cannot set it to use timestamps or anything else except the <base_name>.<n> for its rotated files. That being said, this being Python, you can always override the internals to have it do your bidding, you can check out one way to do it in this answer. – Bulbar

@ybonda, it works for me if I remove mode = 'w' from RotatingFileHandler – Vasilikivasilis

Answer 2

If you change the mode to 'a' in the solution posted by zwer, then you get a single empty log file at the first run, after that it will work as intended. Just increase the backupCount by +1 :-)

Vientiane

Answer 3

OK, going by the title only and only interested in starting a new, same-name file each time, at least for unittests.

Goal:

When running test_foo.py, start with a new test_foo.log each time.

---

(yes, I know that's not necessarily what OP intended, but searching for "create new log file each time" got me here, so...)

This is the general idea, not claiming best practices. YMMV, but it may help some people who came here by way of the title.

PA_ARGV1 = Path(sys.argv[0])

IS_TEST_FILE = PA_ARGV1.name.startswith("test_") and PA_ARGV1.parent.name == "tests"

try:
    # tries to account for `pytest` and `nose`/`nose2` as well
    IS_TEST = (
        IS_TEST_FILE or "nose" in sys.modules.keys() or "nose2" in sys.modules.keys() or sys.argv[0].endswith("/pytest")
    )
except (Exception,) as e:
    IS_TEST = False

def get_config(somevar : Any = None) -> dict[str,Any]:
    """call a function to load the config, rather than hardcoding it"""

    LOGGING = {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {

        ....
        "handlers": {
            "default": {
                "class": "logging.FileHandler",👈
                "level": LOG_LEVEL_FILE,
                "formatter": "standard",
                "filename": str(fnp_applog),
                "mode": "a",👈
                "encoding": "utf-8",
            },
        }
        ....

    if IS_TEST:
        # and I probably could have used `RotatingFileHandler` normally
        # and set it to `FileHandler` in this special case.
        LOGGING["handlers"]["default"]["mode"] = "w"

    return LOGGING

Whaling