asyncore callbacks launching threads... ok to do?

Question

I'm unfamiliar with asyncore, and have very limited knowledge of asynchronous programming except for a few intro to twisted tutorials.

I am most familiar with threads and use them in all my apps. One particular app uses a couchdb database as its interface. This involves longpolling the db looking for changes and updates. The module I use for couchdb is couchdbkit. It uses an asyncore loop to watch for these changes and send them to a callback.

So, I figure from this callback is where I launch my worker threads. It seems a bit crude to mix asynchronous and threaded programming. I really like couchdbkit, but would rather not introduce issues into my program.

So, my question is, is it safe to fire threads from an async callback?

Here's some code...

def dispatch(change):
    global jobs, db_url # jobs is my queue
    db = Database(db_url)
    work_order = db.get(change['id']) # change is an id to the document that changed. 
                                  # i need to get the actual document (workorder)

    worker = Worker(work_order, db) # fire the thread
    jobs.append(worker)
    worker.start()
    return


main()
.
.
.

consumer.wait(cb=dispatch, since=update_seq, timeout=10000) #wait constains the asyncloop.

Update:

After looking over this more, I have an additional question for the couchdbkit gurus. There will potentially be hundreds of threads using the database. As you can see in my code sample, I am instantiating a couchdbkit.Database object per thread. I think this may be wasteful. So, is it ok for a single database object to be used globally among the threads?

Lindeberg

Answer 1

Wouldn't this create a new thread every time the server returns a new document? I would guess that you are best creating a pool of worker threads before you call anything on the server, and just add a job to the queue that these threads are reading their work from in the dispatch method.

But there is no reason why mixing threading and asynchronous programming should be considered dangerous.

Pendragon

Yea creating a new thread per work order is the idea. Each worker will open a tunnel and perform various maintenance tasks on a remote device. I haven't considered a thread pool, but maybe given how callbacks are used this may e a good idea. Thanks for for the input. – Lindeberg