Debugging Celery with VSCode

Question

I am using VSCode for web-developing with framework Django. There is no questions for debugging Django, but when i try to use Celery - debugger dosn't stop at breakpoints. I use this configuration for run Celery and Celery Beat:

{
    "name": "Beat",
    "type": "python",
    "request": "launch",
    "pythonPath": "/home/MyName/job/MyProject/venv/bin/python",
    "program": "/home/MyName/job/MyProject/venv/bin/celery",
    "console": "integratedTerminal",
    "args": [
        "-A",
        "bgp",
        "beat",
        "-l",
        "info"
    ]
},
{
    "name": "Celery",
    "type": "python",
    "request": "launch",
    "pythonPath": "/home/MyName/job/MyProject/venv/bin/python",
    "program": "/home/MyName/job/MyProject/venv/bin/celery",
    "console": "integratedTerminal",
    "args": [
        "-A",
        "bgp",
        "worker",
        "-l",
        "info",
        "-Q",
        "ssh",
        "--concurrency=1",
    ]
},  

When i run Celery - i get this traceback:

[2018-11-29 13:18:34,112: CRITICAL/MainProcess] Unrecoverable error: RuntimeError('already started',)
Traceback (most recent call last):
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/concurrency/base.py", line 131, in start
    self.on_start()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/concurrency/prefork.py", line 112, in on_start
    **self.options)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 432, in __init__
    super(AsynPool, self).__init__(processes, *args, **kwargs)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/pool.py", line 1007, in __init__
    self._create_worker_process(i)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 449, in _create_worker_process
    return super(AsynPool, self)._create_worker_process(i)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/pool.py", line 1116, in _create_worker_process
    w.start()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/billiard/popen_fork.py", line 72, in _launch
    self.pid = os.fork()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 488, in new_fork
    _on_forked_process()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 56, in _on_forked_process
    pydevd.settrace_forked()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 1723, in settrace_forked
    patch_multiprocessing=True,
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 1488, in settrace
    stop_at_frame,
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 1536, in _locked_settrace
    debugger.connect(host, port)  # Note: connect can raise error.
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 484, in connect
    s = start_client(host, port)
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/pydevd_hooks.py", line 125, in <lambda>
    _start_client = (lambda h, p: start_client(daemon, h, p))
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/pydevd_hooks.py", line 71, in start_client
    sock, start_session = daemon.start_client((host, port))
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/daemon.py", line 208, in start_client
    with self.started():
  File "/usr/local/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/daemon.py", line 109, in started
    self.start()
  File "/home/MyName/job/MyProject/venv/lib/python2.7/site-packages/ptvsd/daemon.py", line 144, in start
    raise RuntimeError('already started')
RuntimeError: already started
[2018-11-29 13:18:34,158: INFO/MainProcess] Connected to amqp://project:**@127.0.0.1:5672/project
[2018-11-29 13:18:34,210: INFO/MainProcess] mingle: searching for neighbors
[2018-11-29 13:18:35,292: INFO/MainProcess] mingle: all alone
[2018-11-29 13:18:35,353: WARNING/MainProcess] /home/MyName/job/MyProject/venv/lib/python2.7/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2018-11-29 13:18:35,354: INFO/MainProcess] celery@MyName-vm ready..  

Celery is working fine. But when i put breakpoint into any task - Celery thread dosn't stop on it. How can i fix it?
My Celery version: celery[redis]==4.2.0

Dx

I'm not alone – Dx

I need this to be figured out as well. – Anergy

Answer 1

You can try adding "-P solo" into args of Celery configuration. Refer to https://github.com/Microsoft/ptvsd/issues/1046

Here is my Celery configuration. It is working properly for me.

    {
        "name": "Python: Celery",
        "type": "python",
        "request": "launch",
        "module": "celery",
        "console": "integratedTerminal",
        "args": [
            "-A",
            "tsbc",
            "worker",
            "-l",
            "info",
            "-P",
            "solo",
        ]
    }

Cnidus

This worked for me, it saved my many hours of manual debugging. Thank you very much @Kevin Z. Li. – Varlet

Maybe obvious, but if you are installing your code with a python setup.py install be sure you are setting breakpoints in the site-packages version not your project's working directory or the breakpoints won't trigger. – Emulsify

It might also be nessary to add this line: "env": { "DJANGO_SETTINGS_MODULE": "settings" }, – Hyracoid

Answer 2

This my configuration, works just fine.

{
  "name": "Python: Django Shell",
  "type": "python",
  "request": "launch",
  "program": "${workspaceFolder}/manage.py",
  "args": [
    "shell"
  ],
  "django": true
},
{
  "name": "Python: Celery Workers",
  "type": "python",
  "request": "launch",
  "module": "celery",
  "console": "integratedTerminal",
  "envFile": "${workspaceFolder}/.env",
  "args": ["-A", "yourproject", "worker", "-l", "debug", "-Q", "queueName"]

}

Musing

Answer 3

I figured out a way that uses debugpy in VS Code to have a better debugging experience than the Celery pdb. I use docker-compose, but even if you don't you can still use the idea.

So first start up you celery with debugpy

  celery:
    command:
      [
        "sh",
        "-c",
        "pip install debugpy -t /tmp && python /tmp/debugpy --listen 0.0.0.0:6900 -m celery -A backend.celery worker -l info",
      ]
    ports:
      - 6900:6900

Then here is the launch.json

{
    "name": "Celery: Remote Attach",
    "type": "python",
    "request": "attach",
    "connect": {
        "host": "localhost",
        "port": 6900
    },
    "pathMappings": [{
        "localRoot": "${workspaceFolder}",
        "remoteRoot": "/app"
    }],
    // "preLaunchTask": "docker-compose up",
    "django": true,
},

That's it, enjoy debugging! You can set break points and evaluate variables as you usually do.

I have a post about this for more

Hoeg

I tried to follow the approach and the debugger pauses at the breakpoints perfectly. But when I disconnect the debugger, it doesn’t get disconnected even though the debug subprocesses are terminated – Collaborationist

Disconnecting(detach) won't stop the celery, you could just docker-compose stop/restart celery (if I understand your question correctly) – Hoeg

Is it a constraint from debugpy / vscode ? Also, the hot reload doesn’t seem to be working with Celery. It works fine for the app without —noreload —nothreading. – Collaborationist

hot reload is a feature of django, not celery. If your code changes, you have to restart manually the celery – Hoeg

Answer 4

None of the answers worked for me where I don't want to install an additional debugger. Also putting "-P solo" in the arguments is not required as pointed in one of the answer. Here's the configuration which worked for me

The only two things to take care of are

1. cwd or current working directory should be the same folder as where you run celery from in the terminal. In my case, clickserver is the django folder which contains celery.py so I will run it above that folder
2. PYTHONPATH should be the same folder as cwd.

It should work seamlessly

 "configurations": [
            {
                "name": "Python: Celery",
                "type": "python",
                "cwd": "${workspaceFolder}/clickstream-server/django/clickserver",
                "request": "launch",
                "module": "celery",
                "console": "integratedTerminal",
                "env": {
                    "PYTHONPATH": "${workspaceFolder}/clickstream-server/django/clickserver"
                  },
                "args": [
                    "-A",
                    "clickserver",
                    "worker",
                    "-l",
                    "info",
                ]
            },

Bright

Answer 5

Debugging Celery with VS Code can now be quite simpler, by invoking it as Python package. Example configuration in launch.json:

{
  "name": "Python: Celery",
  "type": "debugpy",
  "module": "celery",
  "args": [
    "-A",
    "jobspot",
    "worker",
    "--loglevel=info",
    "--beat",
    "--pool=solo",
    "--concurrency=1",
    "--events"
  ],
  "request": "launch",
}

Breakpoints work and for me it works like a charm, if you are working in dev containers.

Blodget