What's the proper way to run some Python code asynchronously?

Question

I needed to send mail from my plain Flask app, so I thought the simplest way would be to send it using smtplib. But I had to do it asynchronously - you can't just insert a 3 second delay into the request - right? So I add the email to a queue (psql table), and send it from another program that reads this table and uses smptlib.

This second program (maildonkey) is running as a separate process, in an independent upstart service.

Now I need another one of those little asynchoronous services, and I'm thinking if I should write another python script (third, counting my Flask app and 'maildonkey') or should I use something like Python's 'multiprocess', or even 'threads' and rewrite the second program?

(When I was programming in Clojure, I could easily run code in a separate thread with 'futures', so normally I would do that.)

Freiman

Answer 1

You should consider using Celery. It is very widely used in web frameworks for asynchronous processing and supports a lot of different backends like AMQP, databases etc.

Hendrika

Thanks, but I think this would be overkill for such simple things - I already have a working system and introducing a new component would just increase its complexity - see teddziuba.com/2011/02/the-case-against-queues.html. I'm interested in solving it AND simplifying it at the same time. – Freiman

Answer 2

Try Gevent.
You can create Greenlet object for your long task.
This greenlet is green thread.

from gevent import monkey
monkey.patch_all()
import gevent
from gevent import Greenlet

class Task(Greenlet):
    def __init__(self, name):
        Greenlet.__init__(self)
        self.name = name    
    def _run(self):
        print "Task %s: some task..." % self.name

t1 = Task("long task")
t1.start()
# here we are waiting task
gevent.joinall([t1])

Also you can use Gevent as a server for Flask:

from gevent.wsgi import WSGIServer
from yourapplication import app

http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()

Ochlophobia