Celery task history

Question

I am building a framework for executing tasks on top of Celery framework.

I would like to see the list of recently executed tasks (for the recent 2-7 days).

Looking on the API I can find app.backend object, but cannot figure out how to make a query to fetch tasks.

For example I can use backends like Redis or database. I do not want to explicitly write SQL queries to database.

Is there a way to work with task history/results with API?

I tried to use Flower, but it can only handle events and cannot get history before its start.

Volcanic

check out django-celery, I think it already provides an admin for tasks – Chrestomathy

It seems you might be right. I did not want to use django in this project, but it is likely I need the same functionality that djcelery provides. – Volcanic

Answer 1

You can use the persisent option，eg: flower -A ctq.celery --persistent=True

Hollington

Why was that down voted? It seems reasonable. Whoever down voted please clarify because I'd like to know why that is a bad idea. – Pruinose

this is not bad idea. The down vote because I used Chinese! 23333 – Fermentation

Answer 2

You need to keep the task results in a backend, for example Redis. The Celery documentation contains information about how to do this here:

http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#keeping-results

Also you want to set the CELERY_TASK_RESULT_EXPIRES configuration parameter, because by default the results are discarded after one day.

If you do this, then Flower will show you the history of the task execution, regardless from when it started.

Zigzag

Yes, I have tried that way, but it does not work. That's why I am asking. In the Flower's sources I have found that it works with events only. With backend api it is possible to get past task's results, but only knowing it's ID. It is likely djcelery would be the best for my needs. – Volcanic

@Kapucko, I left an idea to solve this with celery. And I create a new record in the database when a task starts. I finally switched the project to django. – Volcanic

@Volcanic thank you for your response. Could you tell me what advantages does django provide in this case? – Enki

@Kapucko, not only this exact case, but I prefer Django for its flexibility and I like it's project structure. Managing database with models and migrations is pretty mature and easy. – Volcanic

Just wanted to note that Celery introduced new lowercase settings as of version 4.0, so CELERY_TASK_RESULT_EXPIRES is now configured with result_expires. – Lorileelorilyn