Although celery reports no problems at start and says it successfully connected to redis (see log), I get this error running celery inspect ping
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
return retry_over_time(
File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
self._connection = self._establish_connection()
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
conn.connect()
File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
self.transport.connect()
File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 129, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.8/site-packages/amqp/transport.py", line 184, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/celery", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 217, in main
return celery(auto_envvar_prefix="CELERY")
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 134, in caller
return f(ctx, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/bin/control.py", line 136, in inspect
replies = inspect._request(action,
File "/usr/local/lib/python3.8/site-packages/celery/app/control.py", line 106, in _request
return self._prepare(self.app.control.broadcast(
File "/usr/local/lib/python3.8/site-packages/celery/app/control.py", line 741, in broadcast
return self.mailbox(conn)._broadcast(
File "/usr/local/lib/python3.8/site-packages/kombu/pidbox.py", line 328, in _broadcast
chan = channel or self.connection.default_channel
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
self._ensure_connection(**conn_opts)
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
return retry_over_time(
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: [Errno 111] Connection refused
It is a docker-compose app with the redis initiated in docker-compose.yml, so we can see the logs from all containers. Also we can see the celery-beat tasks run successfully, but I want to call task.delay() in one of the views, and there it fails with the same error. Logs:
db_1 |
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1 |
db_1 | 2022-08-18 09:16:02.923 UTC [1] LOG: starting PostgreSQL 13.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
db_1 | 2022-08-18 09:16:02.923 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2022-08-18 09:16:02.923 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2022-08-18 09:16:02.930 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
frontend_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
frontend_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
redis_1 | 1:C 18 Aug 2022 09:16:02.869 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 18 Aug 2022 09:16:02.869 # Redis version=7.0.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 18 Aug 2022 09:16:02.869 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
db_1 | 2022-08-18 09:16:02.942 UTC [21] LOG: database system was shut down at 2022-08-18 09:13:45 UTC
frontend_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
redis_1 | 1:M 18 Aug 2022 09:16:02.870 * monotonic clock: POSIX clock_gettime
db_1 | 2022-08-18 09:16:02.947 UTC [1] LOG: database system is ready to accept connections
frontend_1 | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
redis_1 | 1:M 18 Aug 2022 09:16:02.871 * Running mode=standalone, port=6379.
redis_1 | 1:M 18 Aug 2022 09:16:02.871 # Server initialized
redis_1 | 1:M 18 Aug 2022 09:16:02.871 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
frontend_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * The AOF directory appendonlydir doesn't exist
frontend_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * Loading RDB produced by version 7.0.0
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * RDB age 134 seconds
frontend_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * RDB memory usage when created 1.18 Mb
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * Done loading RDB, keys loaded: 2, keys expired: 0.
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * DB loaded from disk: 0.000 seconds
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: using the "epoll" event method
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: nginx/1.22.0
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: built by gcc 11.2.1 20220219 (Alpine 11.2.1_git20220219)
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: OS: Linux 5.15.0-46-generic
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
redis_1 | 1:M 18 Aug 2022 09:16:02.872 * Ready to accept connections
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker processes
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 22
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 23
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 24
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 25
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 26
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 27
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 28
frontend_1 | 2022/08/18 09:16:04 [notice] 1#1: start worker process 29
celery-beat_1 | [2022-08-18 09:16:05,012: INFO/MainProcess] beat: Starting...
celery_1 |
celery_1 | -------------- celery@934c4c8d4628 v5.2.7 (dawn-chorus)
celery_1 | --- ***** -----
celery_1 | -- ******* ---- Linux-5.15.0-46-generic-x86_64-with-glibc2.2.5 2022-08-18 09:16:05
celery_1 | - *** --- * ---
celery_1 | - ** ---------- [config]
celery_1 | - ** ---------- .> app: MyFavouriteSite:0x7f2188204a00
celery_1 | - ** ---------- .> transport: redis://redis:6379//
celery_1 | - ** ---------- .> results: redis://redis:6379/
celery_1 | - *** --- * --- .> concurrency: 8 (prefork)
celery_1 | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery_1 | --- ***** -----
celery_1 | -------------- [queues]
celery_1 | .> celery exchange=celery(direct) key=celery
celery_1 |
celery_1 |
celery_1 | [tasks]
celery_1 | . app1.tasks.task1
celery_1 | . app2.tasks.task1
celery_1 |
celery_1 | [2022-08-18 09:16:05,516: INFO/MainProcess] Connected to redis://redis:6379//
celery_1 | [2022-08-18 09:16:05,519: INFO/MainProcess] mingle: searching for neighbors
celery_1 | [2022-08-18 09:16:06,530: INFO/MainProcess] mingle: all alone
celery_1 | [2022-08-18 09:16:06,614: WARNING/MainProcess] /usr/local/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
celery_1 | leak, never use this setting in production environments!
celery_1 | warnings.warn('''Using settings.DEBUG leads to a memory
celery_1 |
celery_1 | [2022-08-18 09:16:06,614: INFO/MainProcess] celery@934c4c8d4628 ready.
celery-beat_1 | [2022-08-18 09:30:00,104: INFO/MainProcess] Scheduler: Sending due task task1_name_in_settings (app1.tasks.task1)
celery_1 | [2022-08-18 09:30:00,123: INFO/MainProcess] Task app1.tasks.task1[09c3c690-a379-4a30-a7b4-25cfa57d679a] received
celery_1 | [2022-08-18 09:30:00,143: INFO/ForkPoolWorker-7] Task app1.tasks.task1[09c3c690-a379-4a30-a7b4-25cfa57d679a] succeeded in 0.0180537269989145s: None