Django model object as parameter for celery task raises EncodeError - 'object of type someModelName is not JSON serializable'

Question

Im working with a django project(im pretty new to django) and running into an issue passing a model object between my view and a celery task.

I am taking input from a form which contains several ModelChoiceField fields and using the selected object in a celery task. When I queue the task(from the post method in the view) using someTask.delay(x, y, z) where x, y and z are various objects from the form ModelChoiceFields I get the error object of type <someModelName> is not JSON serializable.

That said, if I create a simple test function and pass any of the same objects from the form into the function I get the expected behavior and the name of the object selected in the form is logged.

def test(object):
    logger.debug(object.name)

I have done some poking based on the above error and found django serializers which allows for a workaround by serializing the object using serializers.serialize('json', [template]), in the view before passing it to the celery task.

I can then access the object in the celery task by using template = json.loads(template)[0].get('fields') to access its required bits as a dictionary -- while this works, it does seem a bit inelegant and I wanted to see if there is something I am missing here.

Im obviously open to any feedback/guidance here however my main questions are:

• Why do I get the object...is not JSON serializable error when passing a model object into a celery task but not when passing to my simple test function?

• Is the approach using django serializers before queueing the celery task considered acceptable/correct or is there a cleaner way to achieve this goal?

Any suggestions would be greatly appreciated.

Traceback: I tried to post the full traceback here as well however including that caused the post to get flagged as 'this looks like spam'

Internal Server Error: /build/
Traceback (most recent call last):
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 49, in _reraise_errors
    yield
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 220, in dumps
    payload = encoder(data)
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 65, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 55, in default
    return super().default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Template is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)

Lump

Answer 1

Add this lines to settings.py

# Project/settings.py
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Then instead of passing object, send JSON with id/pk if you're using a model instance call the task like this..

test.delay({'pk': 1})

Django model instance is not available in celery environment, as it runs in a different process

How you can get the model instance inside task then? Well, you can do something like below -

def import_django_instance():
    """
    Makes django environment available 
    to tasks!!
    """
    import django
    import os
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Project.settings')
    django.setup()


# task
@shared_task(name="simple_task")
def simple_task(data):
    import_django_instance()
    from app.models import AppModel

    pk = data.get('pk')
    instance = AppModel.objects.get(pk=pk)
    # your operation

Interradial

Thanks for your suggestion. So to clarify, is it not possible to send a python object directly to a celery task as a positional parameter? – Lump

Not to my knowledge no! Because celery runs in a different environment so you can't have Django application functionalities in there directly. But what you can do instead is follow my updated answer.... – Interradial

Thanks for the clarification. I have achieved my goal by using the your suggestion of querying the object inside the celery task and passing the pk for query criteria. I wanted to note here that this method does not seem to require settings modifications with respect to celery. – Lump

It will, I guess when you run the celery task as a different service :-) e.g. celery -A Project worker -l INFO – Interradial

This is close to what I used but it won't work if the state of your model has changed and you sent the task to work with the state of the model when the task was fired. – Foresaid

Answer 2

I had similar problem and found out two solutions:

1. OP's first approach

# view
...
orders_json = serializers.serialize('json', orders)
cancel_order_task.delay(orders_json)
...

# task
def resend_order_task(orders_json: dict):
    orders = list(serializers.deserialize('json', orders_json))
    for order in orders:
        order = order.object
...

Key difference from OP's approach is order = order.object. It allows you to access attributes of an object naturally: order.id, order.params, etc.

Pros:

• You send an entire object (just how OP wanted to initially)
• Less operations with DB

Cons:

• Cost of (de)serialization scale linearly with object size and complexity and might slow down performance for larger and more complex objects
• Havier messages to send to celery

2. Raihan K. suggestion

# view
...
order_ids = [order.id for order in orders]
resend_order_task.delay(order_ids)
...

# task
@shared_task
def resend_order_task(order_ids: list):
    orders = Order.objects.filter(id__in=order_ids)
    for order in orders:
...

Pros:

• No compute power for (de)serializing
• Less traffic needed to send task
• You get an actual object from DB and can manipulate it right away

Cons:

• More DB operations required

Sweetener