Eventlet or gevent or Stackless + Twisted, Pylons, Django and SQL Alchemy

Question

We're using Twisted extensively for apps requiring a great deal of asynchronous io. There are some cases where stuff is cpu bound instead and for that we spawn a pool of processes to do the work and have a system for managing these across multiple servers as well - all done in Twisted. Works great. The problem is that it's hard to bring new team members up to speed. Writing asynchronous code in Twisted requires a near vertical learning curve. It's as if humans just don't think that way naturally.

We're considering a mixed approach perhaps. Maybe keep the xmlrpc server part and process management in Twisted and implement the other stuff in code that at least looks synchronous to some extent while not being as such. Then again I like explicit over implicit so I have to think about this a bit more. Anyway onto greenlets - how well does that stuff work? So there's Stackless and as you can see from my Gallentean avatar I'm well aware of the tremendous success in it's use for CCP's flagship EVE Online game first hand. What about Eventlet or gevent? Well for now only Eventlet works with Twisted. However gevent claims to be faster as it's not a pure python implementation but rather relies upon libevent instead. It also claims to have fewer idiosyncrasies and defects. gevent It's maintained by 1 guy as far as I can tell. This makes me somewhat leery but all great projects start this way so... Then there's PyPy - I haven't even finished reading about that one yet - just saw it in this thread: Drawbacks of Stackless.

So confusing - I'm wondering what the heck to do - sounds like Eventlet is probably the best bet but is it really stable enough? Anyone out there have any experience with it? Should we go with Stackless instead as it's been around and is proven technology - just like Twisted is as well - and they do work together nicely. But still I hate having to have a separate version of Python to do this. what to do....

This somewhat obnoxious blog entry hit the nail on the head for me though: Asynchronous IO for Grownups I don't get the Twisted is being like Java remark as to me Java is typically where you are in the threading mindset but whatever. Nevertheless if that monkey patch thing really works just like that then wow. Just wow!

Discernment

Can you comment further on what you mean by MySQL not being good for OLTP? – Meridel

I took that out as it wasn't relevant to the rest of the article. But what makes it poor is, abysmal support for views, slow triggers and stored procedures, a rather primitive query optimizer and difficult to decipher / not enough info for query plans (no plans available for updates / deletes / inserts either). Requires indexes on all foreign key columns often resulting in so many useless indexes that you end up having to simply not implement foreign keys at all to avoid severe performance degradation. I could go on and on... – Discernment

Twisted is one of the best known Python libraries. How would switching to way more obscure projects help anyone "get up to speed" faster? – Grunberg

It's called twisted for a reason :) Writing callback based code is an extremely confusing experience and feels convoluted to most developers accustomed to writing and thinking in terms of synchronous code. Also, if you write a library that you want to have work in a threaded server environment like Pylons as well as an asynchronous and typically non-threaded one (well unless you have cpu bound stuff that you defer to a thread or stuff that accesses a database interface that blocks - which they pretty much all do) it's not an obvious task to solve. – Discernment

Why are callbacks an issue with Twisted? Doesn't @inlineCallbacks solve the problem of callbacks convoluting the code? – Earnestineearnings

+1 to @inlineCallbacks. That is the answer for human issues. But make sure your kids understand generators first, otherwise they'll never get it. – Tanana

Answer 1

You might want to check out:

• Comparing gevent to eventlet
• Reports from users who moved from twisted or eventlet to gevent

Eventlet and gevent are not really comparable to Stackless, because Stackless ships with a standard library that is not aware of tasklets. There are implementations of socket for Stackless but there isn't anything as comprehensive as gevent.monkey. CCP does not use bare bones Stackless, it has something called Stackless I/O which AFAIK is windows-only and was never open sourced (?).

Both eventlet and gevent could be made to run on Stackless rather than on greenlet. At some point we even tried to do this as a GSoC project but did not find a student.

Dumbhead

Thanks so much Denis - I'd already read through the comparison of gevent to eventlet but the Reports from users who moved from twisted or eventlet to gevent was enlightening. – Discernment

@Discernment It's been a year a half since you've asked this question. I was wondering if you could share your experience since then. What did you end up doing? How did it work out for you? – Castleman

The comparison is biassed. Eventlet has a lot of improvements since that time, and now has bigger support. – Harpp

Answer 2

Answering part of your question - if you look at http://speed.pypy.org you'll see that using twisted on top of PyPy may give you some speedups. This depends of course on your workload, but it's probably worth checking out.

Cheers,
fijal

Gliadin

Using PyPy gives you a huge speedup, so it's worth to consider. There is also Tornado which is simpler, much faster then twisted, works more stable on high load. – Harpp

Answer 3

I've built a little real time web app on top of eventlet and repoze.bfg (I gave up on django quite a while ago). I've found eventlet and monkey patching to be just as easy as Ted says.

Solfatara

Answer 4

Gevent isn't pure Python, and it strictly depends on CPython. From web frameworks you mentioned Eventlet (OpenStack) and Tornado (FriendsFeed, Quora) has the biggest deploy.

Harpp