How to make SMTPHandler not block

Question

I installed a local SMTP server and used logging.handlers.SMTPHandler to log an exception using this code:

import logging
import logging.handlers
import time
gm = logging.handlers.SMTPHandler(("localhost", 25), '[email protected]', ['[email protected]'], 'Hello Exception!',)
gm.setLevel(logging.ERROR)
logger.addHandler(gm)
t0 = time.clock()
try:
    1/0
except:
    logger.exception('testest')
print time.clock()-t0

It took more than 1sec to complete, blocking the python script for this whole time. How come? How can I make it not block the script?

Toneless

Answer 1

Here's the implementation I'm using, which I based on this Gmail adapted SMTPHandler.
I took the part that sends to SMTP and placed it in a different thread.

import logging.handlers
import smtplib
from threading import Thread

def smtp_at_your_own_leasure(mailhost, port, username, password, fromaddr, toaddrs, msg):
    smtp = smtplib.SMTP(mailhost, port)
    if username:
        smtp.ehlo() # for tls add this line
        smtp.starttls() # for tls add this line
        smtp.ehlo() # for tls add this line
        smtp.login(username, password)
    smtp.sendmail(fromaddr, toaddrs, msg)
    smtp.quit()

class ThreadedTlsSMTPHandler(logging.handlers.SMTPHandler):
    def emit(self, record):
        try:
            import string # for tls add this line
            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                            self.fromaddr,
                            string.join(self.toaddrs, ","),
                            self.getSubject(record),
                            formatdate(), msg)
            thread = Thread(target=smtp_at_your_own_leasure, args=(self.mailhost, port, self.username, self.password, self.fromaddr, self.toaddrs, msg))
            thread.start()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)

Usage example:

logger = logging.getLogger()

gm = ThreadedTlsSMTPHandler(("smtp.gmail.com", 587), 'bugs@my_company.com', ['admin@my_company.com'], 'Error found!', ('[email protected]', 'top_secret_gmail_password'))
gm.setLevel(logging.ERROR)

logger.addHandler(gm)

try:
    1/0
except:
    logger.exception('FFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUU-')

Toneless

Answer 2

You could use QueueHandler and QueueListener. Taken from the docs:

Along with the QueueListener class, QueueHandler can be used to let handlers do their work on a separate thread from the one which does the logging. This is important in Web applications and also other service applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via SMTPHandler) are done on a separate thread.

Alas they are only available from Python 3.2 onward.

Toneless

+1, and they are available for older Python versions through the logutils project: plumberjack.blogspot.com/2010/10/… – Bonin

Answer 3

The simplest form of asynchronous smtp handler for me is just to override emit method and use the original method in a new thread. GIL is not a problem in this case because there is an I/O call to SMTP server which releases GIL. The code is as follows

class ThreadedSMTPHandler(SMTPHandler):
    def emit(self, record):
        thread = Thread(target=SMTPHandler.emit, args=(self, record))
        thread.start()

Alkane

Answer 4

Most probably you need to write your own logging handler that would do the sending of the email in the background.

Shotgun

e.g. Calling out to the local sendmail program that probably came with your MTA and submits to your local SMTP Server without using SMTP. – Gavin

Answer 5

A thing to keep in mind when coding in Python is the GIL (Global Interpreter Lock). This lock prevents more than one process from happening at the same time. there are many number of things that are 'Blocking' activities in Python. They will stop everything until they completed.

Currently the only way around the GIL is to either push off the action you are attempting to an outside source like aix and MattH are suggesting, or to implement your code using the multiprocessing module (http://docs.python.org/library/multiprocessing.html) so that one process is handling the sending of messages and the rest is being handled by the other process.

Tabular

Answer 6

As the OP pointed out, QueueHandler and QueueListener can do the trick! I did some research and adapted code found on this page to provide you with some sample code:

# In your init part,
# assuming your logger is given by the "logger" variable
# and your config is storded in the "config" dictionary

logging_queue = Queue(-1)
queue_handler = QueueHandler(logging_queue)
queue_handler.setLevel(logging.ERROR)
queue_handler.setFormatter(logging_formatter)
logger.addHandler(queue_handler)

smtp_handler = SMTPHandler(mailhost=(config['MAIL_SERVER'], config['MAIL_PORT']),
                           fromaddr=config['MAIL_SENDER'],
                           toaddrs=[config['ERROR_MAIL']],
                           subject='Application error',
                           credentials=(config['MAIL_USERNAME'], config['MAIL_PASSWORD']),
                           secure=tuple())
smtp_handler.setLevel(logging.ERROR)
smtp_handler.setFormatter(logging_formatter)

queue_listener = QueueListener(logging_queue, smtp_handler)
queue_listener.start()

# Let's test it. The warning is not mailed, the error is.
logger.warning('Test warning')
logger.error('Test error')

What I am not sure about is whether it is necessary to use setLevel and setFormatter twice, probably not.

Equation

Answer 7

Here's the implementation I'm using, which I based on Jonathan Livni code.

import logging.handlers
import smtplib
from threading import Thread

# File with my configuration
import credentials as cr

host = cr.set_logSMTP["host"]
port = cr.set_logSMTP["port"]
user = cr.set_logSMTP["user"]
pwd = cr.set_logSMTP["pwd"]
to = cr.set_logSMTP["to"]



def smtp_at_your_own_leasure(
    mailhost, port, username, password, fromaddr, toaddrs, msg
):
    smtp = smtplib.SMTP(mailhost, port)
    if username:
        smtp.ehlo()  # for tls add this line
        smtp.starttls()  # for tls add this line
        smtp.ehlo()  # for tls add this line
        smtp.login(username, password)
    smtp.sendmail(fromaddr, toaddrs, msg)
    smtp.quit()


class ThreadedTlsSMTPHandler(logging.handlers.SMTPHandler):
    def emit(self, record):
        try:
            # import string  # <<<CHANGE THIS>>>

            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                self.fromaddr,
                ",".join(self.toaddrs),  # <<<CHANGE THIS>>>
                self.getSubject(record),
                formatdate(),
                msg,
            )
            thread = Thread(
                target=smtp_at_your_own_leasure,
                args=(
                    self.mailhost,
                    port,
                    self.username,
                    self.password,
                    self.fromaddr,
                    self.toaddrs,
                    msg,
                ),
            )
            thread.start()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)


# Test
if __name__ == "__main__":
    logger = logging.getLogger()

    gm = ThreadedTlsSMTPHandler((host, port), user, to, "Error!:", (user, pwd))
    gm.setLevel(logging.ERROR)

    logger.addHandler(gm)

    try:
        1 / 0
    except:
        logger.exception("Test ZeroDivisionError: division by zero")

Usherette