Flask/Celery: AttributeError("Can't pickle local object 'celery_init_app.<locals>.FlaskTask'")

Question

python 3.10 running in venv on Windows 10 pro.

I am trying to follow the tutorial for the Celery and Flask integration: https://flask.palletsprojects.com/en/latest/patterns/celery/

# example.py

from celery import Celery, Task
from flask import Flask


def celery_init_app(app: Flask) -> Celery:
    class FlaskTask(Task):
        def __call__(self, *args: object, **kwargs: object) -> object:
            with app.app_context():
                return self.run(*args, **kwargs)

    celery_app = Celery(app.name, task_cls=FlaskTask)
    celery_app.config_from_object(app.config["CELERY"])
    celery_app.set_default()
    app.extensions["celery"] = celery_app
    return celery_app


def create_app() -> Flask:
    app = Flask(__name__)
    app.config.from_mapping(
        CELERY=dict(
            # Redis Docker container connection string
            broker_url="redis://default:redispw@localhost:55000",
            result_backend="redis://default:redispw@localhost:55000",
            task_ignore_result=True,
        ),
    )
    app.config.from_prefixed_env()
    celery_init_app(app)
    return app

# make_celery.py

from example import create_app

flask_app = create_app()
celery_app = flask_app.extensions["celery"]

Command line call: celery -A make_celery worker --loglevel INFO

Runs into:

Unrecoverable error: AttributeError("Can't pickle local object 'celery_init_app.<locals>.FlaskTask'")
[...]
.venv\lib\site-packages\billiard\reduction.py", line 123, in steal_handle
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access is denied

The same error and traceback occurs on the example repo: https://github.com/pallets/flask/tree/main/examples/celery

How can I resolve this? What alternatives are available?

Tried moving the class FlaskTask(Task) out of the local scope of the celery_init_app function, but then I obviously lose access to the app variable.

Deictic

Answer 1

I ran into the same issue. Changing celery_init_app to read like this fixed it

celery_app = Celery(app.name)
celery_app.config_from_object(app.config["CELERY"])
celery_app.Task = FlaskTask

I wish I could give an explanation as to the "why". If anybody knows please post it

Twentytwo

Thank you. quite a few hrs i have been going round in circles. seems there are no up to date examples of flask with app factory and celery. unless you already know what you are doing. – Arenaceous

@Twentytwo What is the full code for this area? I couldn't get it to work. Unfortunately I have seen reports that celery no longer works with windows. – Felicity

If you do ``` celery_app.task_cls = FlaskTask ``` the linter will not complain. – Fencesitter

Answer 2

If you're using windows, run the worker like this,

celery -A celery_app worker -l info -P solo

Celery uses the prefork pool, which relies on Unix-specific features that are not available on Windows.

Centigrade