Celery task returning a list of pydantic BaseModel subclass: not json serializable

Question

Consider this code:

from pydantic import BaseModel

class MyModel(BaseModel):
  x: int

appc = Celery(...)

@appc.task(bind=True)
def mytask(self):
    return [MyModel(x=0)]

res = mytask.delay().get()  # ERROR

Error is: kombu.exceptions.EncodeError: TypeError('Object of type MyModel is not JSON serializable')

I tried working with kombu serializers or with pydantic encoder but I really don't understand what is going wrong.

Doti

maybe return [MyModel(x=0).dict()] will work? – Hydrometer

It does, but it requires to modify the task signature to return List[Dict] instead of List[MyModel] loosing all the benefits of pydantic... – Doti

Solved with custom serializer, as in: #21632378 – Doti

I was about to say that. Perhaps it would be nice if you write a good answer to your question (since you already solved the problem), and accept it so other people may know the solution without going through the comments... – Theophrastus

Answer 1

I solved it in a easy way using pickle serialization:

appc = Celery(
    name=__name__,
    # ...
)


class CeleryConfig:
    task_serializer = "pickle"
    result_serializer = "pickle"
    event_serializer = "json"
    accept_content = ["application/json", "application/x-python-serialize"]
    result_accept_content = ["application/json", "application/x-python-serialize"]


appc.config_from_object(CeleryConfig)

In this way tasks and results are serialized with pickle and, as shown here (https://pydantic-docs.helpmanual.io/usage/exporting_models/#pickledumpsmodel), Pydantic models are dumpable and loadable with pickle without issues.

In this way:

res = mytask.delay().get()

not only works, but res it is indeed a true List[model] so i.e. res[0].x == 0 is valid and true.

Doti

Answer 2

Celery versions >= 5.5.0 will support serialization, deserialization and validation of Pydantic objects. Documentation: https://docs.celeryq.dev/en/latest/userguide/tasks.html#argument-validation-with-pydantic

Sorcerer