How to prevent the same task to be executed by celery?

Question

I'm implementing a cache server that uses a celery task to update the cache in background. There is only one task that I call it with different arguments (cache keys).

Since after connecting this server to my main production server it will receive tens of requests per second for the same cache key I want to make sure there are never more than one of the update tasks with the same cache key inside celery queue (working as a queue and a set at the same time).

I thought of using a redis set to make sure of that before running the task but I'm looking for a better way.

Mattock

Is your requirement really "only one task with the same cache key inside celery QUEUE" or is it "only one task with the same cache key running (updating cache) concurrently"? – Automate

@MuhammadTahir only one task with the same cache key inside celery QUEUE – Mattock

See if this works for you docs.celeryproject.org/en/latest/tutorials/…. This will work as it is for the second case that I mentioned but it can be modified for your version (the first I mentioned) too. – Automate

Possible duplicate of Running "unique" tasks with celery – Bridesmaid

@Bridesmaid Possibly yes. I will try to find the code I was writing at the time to check if there is any difference in requirements for both questions. Though I remember I solved my problem using this github.com/cameronmaske/celery-once . – Mattock

Answer 1

There is only one way, implement your own lock mechanism. The official doc has a nice example page.. The only limit is your imagination.

Hope this helps.

Abdication