How to defer a Django DB operation from within Twisted?

Question

I have a normal Django site running. In addition, there is another twisted process, which listens for Jabber presence notifications and updates the Django DB using Django's ORM.

So far it works as I just call the corresponding Django models (after having set up the settings environment correctly). This, however, blocks the Twisted app, which is not what I want.

As I'm new to twisted I don't know, what the best way would be to access the Django DB (via its ORM) in a non-blocking way using deferreds.

1. deferredGenerator ?
2. twisted.enterprise.adbapi ? (circumvent the ORM?)
3. ???

If the presence message is parsed I want to save in the Django DB that the user with jid_str is online/offline (using the Django model UserProfile). I do it with that function:

def django_useravailable(jid_str, user_available):
    try:
        userhost = jid.JID(jid_str).userhost()
        user = UserProfile.objects.get(im_jabber_name=userhost)
        user.im_jabber_online = user_available
        user.save()
        return jid_str, user_available
    except Exception, e:
        print e
    raise jid_str, user_available,e

Currently, I invoke it with:

d = threads.deferToThread(django_useravailable, from_attr, user_available)
d.addCallback(self.success)
d.addErrback(self.failure)

Azaleeazan

Please post code, especially the part which "blocks" your app. – Hibbler

Answer 1

"I have a normal Django site running."

Presumably under Apache using mod_wsgi or similar.

If you're using mod_wsgi embedded in Apache, note that Apache is multi-threaded and your Python threads are mashed into Apache's threading. Analysis of what's blocking could get icky.

If you're using mod_wsgi in daemon mode (which you should be) then your Django is a separate process.

Why not continue this design pattern and make your "jabber listener" a separate process.

If you'd like this process to be run any any of a number of servers, then have it be started from init.rc or cron.

Because it's a separate process it will not compete for attention. Your Django process runs quickly and your Jabber listener runs independently.

Denney

I would like to have the opportunity to migrate the presence notificaiton handler to another server when needed. With the process solution that is not easily possible. Furthermore, I like twisted async appraoch, which I think can handle massive amounts of notification changes. I suppose there will be a lot more notificatinos than web site hits. – Azaleeazan

Answer 2

I have been successful using the method you described as your current method. You'll find by reading the docs that the twisted DB api uses threads under the hood because most SQL libraries have a blocking API.

I have a twisted server that saves data from power monitors in the field, and it does it by starting up a subthread every now and again and calling my Django save code. You can read more about my live data collection pipeline (that's a blog link).

Are you saying that you are starting up a sub thread and that is still blocking?

Hereford

But the reason to using twisted is that we do not want use threads. If I have to stick with the current threaded version I have no benefits in using twisted. I then might as well use threads from the Django process directly. – Azaleeazan

Yes, I hate it too for that very reason. At least you can ensure that there are a limited number of database threads started (i.e manage 1000 client connections and cleverly pool their database access and threads). Database usage is only a small part of my long term connections managed by twisted, so its okay to spin up a thread on demand for me. – Hereford

Answer 3

I have a running Twisted app where I use Django ORM. I'm not deferring it. I know it's wrong, but hadd no problems yet.

Thereupon