How to check if celery result backend is working

Question

I am using celery with Redis.

Current Redis is used as a broker and as a result backend.

BROKER_TRANSPORT = 'redis'
BROKER_URL = 'redis://domain:8888/0'
CELERY_RESULT_BACKEND = 'redis://domain:8888/0'

I want to clear few things:

1. What is the benefit of using the result backend? I mean what I will get by using it
2. How can I see that it is working? I mean will something be stored in Redis? Will that storage is permanent? How can I query that? Will that storage grow with time etc
3. Can I monitor that result backend stuff with the celery flower?

Oldster

Answer 1

A result backend is exactly what it sounds like, all it does is store results from tasks.

Let's say that you have the following task that actually returns a value.

@task
def sum(x, y):
    return x + y

At some point, you call this task. If you do not have a result backend, get() will throw an error (or a warning, I forget which). If you do have a result backend (and assuming it's properly configured), task.get() will poll your redis-backend for a result from the task_id associated with task and then return it to you via whatever serializer you specified.

from tasks import sum

task = sum.delay(3, 4)
task.get()

You can see that it works by just calling get() (and waiting to completion) on a task that you've sent off to the broker. You can read more about working with celery results from the official documentation.

You can in principle poll your redis database from the redis-cli, but I see no reason to. You can view results in flower by going to one of the actual task detail views and checking the "result" field under the "Basic Task Options" table. e.g. http://flower.myserver.com/task/

Ozonosphere

Thanks Tu , I got that . One thing which is not clear to me that where does the celery store all thos einformation like task_id, which worker was that task executed etc. Does it go in Redis . Is that information permanent> Because i see that if i restart celery flower then all the tasks in task tab is gone . It look like that information is not persistent – Oldster

Yes, it is all stored in the broker. In your specific case, that appears to be a redis-database which is an in-memory database. This information is indeed not persistent (by design!) If you want to persist your redis-database, read up on how to do that from the redis-documentation regarding redis-peristence. If persistence is going to be a big part of your use-case, you may want to look into a more robust broker such as RabbitMQ which is more involved, but will allow data in the queue to be preserved on abrupt shutdowns. – Ozonosphere

Do you think it make sense that i manually store my tasks status in database in the task execution code to see the full history of whats happening so that if in case i need to check some task_id from log messages then i can see the related steps , results etc from databse directly – Oldster

Whether or not it makes sense is entirely up to you since you understand what your needs and goals are. That said, here are my $0.02. If you do not plan on actually using the results in your code, there is no point in using a result backend, especially if the only enduse is for periodically cross referencing your logs. You could instead log the result from within the task, rather than returning it alongside the task id (see celery docs on logging.) Redis is meant to be lightweight, so I would say, keep it that way! – Ozonosphere

Celery is the broker to manage tasks and retries. Redis is an in-memory db, but you should enable to persistent feature in case the server is rebooted so it will read the queue back in and resume work. Flower listens to Redis using Celery and then stores it in its own database which l you can enable persistent storage on as well. I also store my status events in my database so I can leverage it for the user to see what is working and not working for troubleshooting issues. – Defant

Answer 2

The question is already answered. So, I would like to show the snapshots of what celery stores in DB just to give an idea.

Here, I have used the default settings of celery with MySQL database as result_backend.

It has created two tables:

1. celery_taskmeta

and

2. celery_tasksetmeta

Jobye