Which Python async library would be best suited for my code? Asyncore? Twisted?
Asked Answered
S

3

18

I have a program I'm working on that will be reading from two 'network sources' simultaneously. I wanted to try out an asynchronous approach rather than use threading. This has lead me to wonder which library to use...

I've come up with some simple example code that kind of demonstrates what my program will be doing:

import sniffer

def first():
    for station in sniffer.sniff_wifi():
        log(station.mac())

def second():
    for station in sniffer.sniff_ethernet():
        log(station.mac())

first()
second()

The two sniffer methods look somewhat like this:

def sniff_wifi(self):

    while True:
        yield mac_address

The while True loop obviously makes them blocking.

I want to use asyncore for this as it is part of the standard library. No 3rd party dependencies are a bonus. However, that doesn't mean I won't use it if you recommend I do...

Can I achieve what I'm trying to do with asyncore? If so, could you show me how to convert my example code to 'asyncore code'? Do you know of any good asyncore tutorials?

Sainted answered 8/12, 2010 at 5:8 Comment(0)
F
50

Twisted is better in pretty much every possible way. It's more portable, more featureful, simpler, more scalable, better maintained, better documented, and it can make a delicious omelette. Asyncore is, for all intents and purposes, obsolete.

It's hard to demonstrate all the ways in which Twisted is superior in a short answer (how could I demonstrate a http/dns/ssh/smtp/pop/imap/irc/xmpp/process-spawning/multi-threading server in a short example?), so instead I'll focus on one of the most common misconceptions that people seem to have about Twisted: that it's somehow more complex or harder to use than asyncore.

Let's start with an asyncore example. In order to avoid a biased presentation, I'll use an example from someone else who still likes asyncore a bit. Here's a simple asyncore example taken from Richard Jones' weblog (with comments elided for brevity).

First, here's the server:

import asyncore, socket

class Server(asyncore.dispatcher):
    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.bind(('', port))
        self.listen(1)

    def handle_accept(self):
        socket, address = self.accept()
        print 'Connection by', address
        EchoHandler(socket)

class EchoHandler(asyncore.dispatcher_with_send):
    def handle_read(self):
        self.out_buffer = self.recv(1024)
        if not self.out_buffer:
            self.close()

s = Server('', 5007)
asyncore.loop()

and here's the client:

import asyncore, socket

class Client(asyncore.dispatcher_with_send):
    def __init__(self, host, port, message):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.connect((host, port))
        self.out_buffer = message

    def handle_close(self):
        self.close()

    def handle_read(self):
        print 'Received', self.recv(1024)
        self.close()

c = Client('', 5007, 'Hello, world')
asyncore.loop()

There are a few obscure cases that this code doesn't handle correctly, but explaining them is boring and complicated, and the code has already made this answer long enough.

Now, here's some code that does basically the same thing, with Twisted. First, the server:

from twisted.internet import reactor, protocol as p

class Echo(p.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)

class EchoFactory(p.Factory):
    def buildProtocol(self, addr):
        print 'Connection by', addr
        return Echo()

reactor.listenTCP(5007, EchoFactory())
reactor.run()

And now, the client:

from twisted.internet import reactor, protocol as p

class EchoClient(p.Protocol):
    def connectionMade(self):
        self.transport.write(self.factory.data)

    def dataReceived(self, data):
        print 'Received:', data
        self.transport.loseConnection()

class EchoClientFactory(p.ClientFactory):
    protocol = EchoClient
    def __init__(self, data):
        self.data = data

reactor.connectTCP('localhost', 5007, EchoClientFactory('Hello, world'))
reactor.run()

There are a couple of things that I'd like to draw your attention to. First of all, the Twisted example is 25% shorter, even for something this trivial. 40 lines for asyncore, only 30 for Twisted. As your protocol grows more complex, this difference will get bigger and bigger, as you need to write more and more support code for asyncore that would have been provided for you by Twisted.

Second, Twisted provides a complete abstraction. With the asyncore example, you have to use the socket module to do the actual networking; asyncore provides only multiplexing. This is a problem if you need portable behavior on platforms such as Windows. It also means that asyncore completely lacks facilities for doing asynchronous sub-process communication on other platforms; you can't stuff arbitrary file descriptors into a select() call on Windows.

Third, the Twisted example is transport neutral. None of Echo and EchoFactory and EchoClient and EchoClientFactory is in any way specific to TCP. You can make these classes into a library that can be connected via SSH, or SSL, or a UNIX socket, or a pipe, only by changing the one connectTCP/listenTCP call at the bottom. This is important, as supporting something like TLS directly in your protocol logic is very tricky. For example, a 'write' in TLS will trigger a 'read' at the lower level. So, you need to separate these concerns out to get them right.

Finally, specific to your use-case, if you're dealing with MAC addresses and ethernet frames directly, Twisted contains Twisted Pair, a low-level library for dealing with IP and ethernet-level networking. This isn't the most actively maintained part of Twisted; the code is quite old. But, it should work, and if it doesn't we will take any bugs in it seriously and (eventually) see that they get fixed. As far as I'm aware, there's no comparable library for asyncore, and it certainly doesn't contain any such code itself.

Fink answered 8/12, 2010 at 8:55 Comment(7)
This would be a fine introduction from the author of twisted.Uveitis
+1. Very detailed and to the point but I expected nothing less from you. :)Trifacial
Definitely a great answer, not to mention that it's from the lead architect of twisted...Radium
Sorry, but as soon as I see the word factory in Python code, I can't take it seriously... What are the "obscure cases" asyncore does not handle correctly? Do you know any alternative network library (callback based)? Or a pythonic wrapper on top of Twisted? :)Aho
Just because Java has abused the word doesn't mean that it's a useless concept. en.wikipedia.org/wiki/Factory_method_pattern :-)Fink
Some such "obscure" case that asyncore will not handle is EMFILE or ENOMEM coming back from accept, or EWOULDBLOCK coming back from recv. These only happen in fairly obscure conditions (the length limit on comments would prevent me from describing them fully), but Twisted will handle them silently and asyncore will crash its mainloop.Fink
@Fink No, that is not exactly true as every exception (without exit exception) is caught: hg.python.org/cpython/file/2.7/Lib/asyncore.py#l79. However, not handling EWOULDBLOCK on recv seems like a bug.Aho
T
3

Asyncore is nice but not very feature rich so you might run into problems later when your app grows. That being said, it's great to prototype stuff. The approach is quite simple. You define methods to handle certain events in a class (when read is possible, when write is possible etc) and then subclass that from the asyncore.dispatcher (I think) class.

The official docs for the module as well as Doug Hellmann's excellent PyMOTW article on it are good sources to check out for docs and examples.

If your protocol is conversational (e.g. send this, receive that), you can check out the asynchat module also distributed with the standard library for ideas.

Twisted is a much more heavy duty approach. I'm sure it will work better for larger projects given how much it's used but I can't say anything more because I don't have any first hand experience with it.

Trifacial answered 8/12, 2010 at 7:30 Comment(3)
asyncore/asynchat seem to only be useful when tied directly into the socket handling. In my code, sniffer handles all the socket stuff and yields the results. All I want asyncore to do is run my first and second functions and allow me to issue a callback when the sniffers return data. How would I do this? (I didn't downvote you, btw)Sainted
asyncore/asynchat gives you an object oriented wrapper around the select function call (which is usually what all async things are built over). select operates on file descriptors so that's what asyncore/asynchat does as well. Give me a few minutes to get something workable for you.Trifacial
I think glyph's answer is more complete that mine will ever be.Trifacial
P
0

Curl has been designed to be non blocking in all perspectives and avoids using select which is a costly operation during async I/O. At low level, curl is using most optimal possible solutions thus to the date there is no framework which has been able to perform better than curl though there might be frameworks which could give similar performance.

That being said, how about writing your own sockets? Its very easy in Python and can give you amazing performance once you know what you are doing and is clear with your goals.

Pearl answered 10/4, 2012 at 13:38 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.