Custom prefix for redis keys with Celery

Question

I am using redis as a broker between Django and Celery. The redis instance I have access to is shared with many other applications and so the broker is not reliable (the redis keys it uses are deleted by others, the messages often get sent to workers in other applications). Changing redis database does not solve the problem (there are few databases and many applications).

How can I configure Celery to prefix all the keys it uses with a custom string? The docs mention ways to add prefixes to queue names, but that does not affect the redis keys. The underlying library (Kombu) does not seem to let the user prefix the keys it uses as far as I can tell.

Fogged

Answer 1

The functionality to add prefix to all the redis keys has been added as part of this. Now you can configure it like this:

BROKER_URL = 'redis://localhost:6379/0'
celery = Celery('tasks', broker=BROKER_URL, backend=BROKER_URL)
celery.conf.broker_transport_options = {'global_keyprefix': "prefix"}

Dunford

Answer 2

This is not supported by Celery yet. A pull request on this subject is currently stalled due to a lack of workforce:

https://github.com/celery/kombu/pull/912

Fogged

Answer 3

You can just override the prefix value of your celery task.

@shared_task(bind=True)
def task(self, params):
    self.backend.task_keyprefix = b'new-prefix'

Ballflower

From which release of celery is this feature available? And, can we provide prefix in the format {prefix}. Require this prefix format for resolving redis crossslot issue. – Anthropogeography