job queue implementation for python

T

9

14

Do you know/use any distributed job queue for python? Can you share links or tools

Trotyl answered 26/8, 2009 at 18:1 Comment(0)

M

4

In addition to multiprocessing there's also the Celery project, if you're using Django.

Miniaturize answered 26/8, 2009 at 18:6 Comment(5)

Thanks for the link? Is it strictly used with django? can we use it for standard python projects? – Trotyl 28/8, 2009 at 3:20

I don't see why not, with suitable adaptation (not sure how much work that'll be - depends on your exact requirement). – Miniaturize 28/8, 2009 at 8:20

Celery has an underlying library, called Carrot, that you can use without Django. – Septuplicate 8/1, 2010 at 3:56

Both Celery and Carrot works without Django. Or that is, you can use it from outside of a Django project. Recently someone even implemented paste support: bitbucket.org/twillis/celery-paste – Sonia 1/2, 2010 at 16:25

Celery is now designed to be used outside of Django (but still has Django support if you need it) – Pusillanimity 28/10, 2012 at 17:36

G

12

Pyres is a resque clone built in python. Resque is used by Github as their message queue. Both use Redis as the queue backend and provide a web-based monitoring application.

http://binarydud.github.com/pyres/intro.html

Goren answered 21/7, 2010 at 20:8 Comment(0)

M

4

In addition to multiprocessing there's also the Celery project, if you're using Django.

Miniaturize answered 26/8, 2009 at 18:6 Comment(5)

Thanks for the link? Is it strictly used with django? can we use it for standard python projects? – Trotyl 28/8, 2009 at 3:20

I don't see why not, with suitable adaptation (not sure how much work that'll be - depends on your exact requirement). – Miniaturize 28/8, 2009 at 8:20

Celery has an underlying library, called Carrot, that you can use without Django. – Septuplicate 8/1, 2010 at 3:56

Both Celery and Carrot works without Django. Or that is, you can use it from outside of a Django project. Recently someone even implemented paste support: bitbucket.org/twillis/celery-paste – Sonia 1/2, 2010 at 16:25

Celery is now designed to be used outside of Django (but still has Django support if you need it) – Pusillanimity 28/10, 2012 at 17:36

N

3

There's also "bucker" by Sylvain Hellegouarch which you can find here:

http://trac.defuze.org/wiki/bucker

It describes itself like this:

bucker is a queue system that supports multiple storage for the queue (memcached, Amazon SQS for now) and is driven by XML messages sent over a TCP connections between a client and the queue server.

Ninny answered 26/8, 2009 at 18:10 Comment(0)

V

2

Look at beanstalkd

Virago answered 26/8, 2009 at 18:18 Comment(0)

D

2

redqueue? It's implemented in python+tornado framework, speaks memcached protocol and is optionally persistent into log files. Currently it is also able to behave like beanstalkd, the reserve/delete way in memcache protocol as well.

REDQUEUE

Dorweiler answered 8/1, 2010 at 3:44 Comment(1)

At present, I have been using celery. I will look at the redqueue. Thanks for answer – Trotyl 8/1, 2010 at 15:3

P

2

If you think that Celery is too heavy for your needs then you might want to look at the simple distributed task queue:

Poll answered 18/2, 2012 at 15:5 Comment(0)

L

1

It's a year late or whatever, but this is something I've hacked together to make a queue of Processes executing them only X number at a time. http://github.com/goosemo/job_queue

Lail answered 21/7, 2010 at 20:4 Comment(0)

G

-1

You probably want to look at multiprocessing's Queue. Included in Python 2.6, get it on PyPI for earlier versions of Python.

Standard library documentation: http://docs.python.org/library/multiprocessing.html On PyPI: http://pypi.python.org/pypi/multiprocessing

Recommended topics

Hot tags