How to start a custom RQ worker within a Docker Container (Python, Flask and Redis)

I followed the excellent Flask Mega Tutorial by Miguel Grinberg and have successfully setup a Flask web app with a Redis task queue and RQ workers, all in Docker containers.

To improve task queue performance, I now need to use my own custom worker, rather than the default RQ worker.

Unfortunately, I'm struggling to understand how I start a custom worker within docker.

To start a default RQ worker, the Flask Mega Tutorial uses the method of overriding the Docker entrypoint with "venv/bin/rq" and then supplying the argument "worker -u redis://redis-server:6379/0 microblog-tasks".

The executable name is supplied with the --entrypoint flag, whilst the command arguments are passed at the very end, after the name of the container image.

Here is the full command - only the last two lines are relevant to this question.

$ docker run --name rq-worker -d --rm -e SECRET_KEY=my-secret-key \
-e MAIL_SERVER=smtp.googlemail.com -e MAIL_PORT=587 -e MAIL_USE_TLS=true \
-e MAIL_USERNAME=<your-gmail-username> -e MAIL_PASSWORD=<your-gmail-password> \
--link mysql:dbserver --link redis:redis-server \
-e DATABASE_URL=mysql+pymysql://microblog:<database-password>@dbserver/microblog \
-e REDIS_URL=redis://redis-server:6379/0 \
--entrypoint venv/bin/rq \
microblog:latest worker -u redis://redis-server:6379/0 microblog-tasks

I have my own custom worker with the following code, taken directly from the RQ documentation:

#!/usr/bin/env python
import sys
from rq import Connection, Worker

# Preload libraries
import library_that_you_want_preloaded

# Provide queue names to listen to as arguments to this script,
# similar to rq worker
with Connection():
    qs = sys.argv[1:] or ['default']

    w = Worker(qs)
    w.work()

Given that my custom worker is located within the Docker container at "home/dashboard/app/custom_worker.py", which commands do I need to supply upon starting the Docker container to create an RQ worker using my customised worker script? So far I have tried the following:

$ docker run --name rq-worker -d --rm -e SECRET_KEY=my-secret-key \
-e MAIL_SERVER=smtp.googlemail.com -e MAIL_PORT=587 -e MAIL_USE_TLS=true \
-e MAIL_USERNAME=<your-gmail-username> -e MAIL_PASSWORD=<your-gmail-password> \
--link mysql:dbserver --link redis:redis-server \
-e DATABASE_URL=mysql+pymysql://microblog:<database-password>@dbserver/microblog \
-e REDIS_URL=redis://redis-server:6379/0 \
--entrypoint venv/bin/rq \
microblog:latest /home/dashboard/app/custom_worker.py -u redis://redis-server:6379/0 microblog-tasks

and also...

$ docker run --name rq-worker -d --rm -e SECRET_KEY=my-secret-key \
-e MAIL_SERVER=smtp.googlemail.com -e MAIL_PORT=587 -e MAIL_USE_TLS=true \
-e MAIL_USERNAME=<your-gmail-username> -e MAIL_PASSWORD=<your-gmail-password> \
--link mysql:dbserver --link redis:redis-server \
-e DATABASE_URL=mysql+pymysql://microblog:<database-password>@dbserver/microblog \
-e REDIS_URL=redis://redis-server:6379/0 \
--entrypoint /home/dashboard/app \
microblog:latest custom_worker -u redis://redis-server:6379/0 microblog-tasks

Any help would be greatly appreciated. There are a lot of posts online about creating a custom RQ worker, but I've not found much detail on how you practically use your custom worker in deployment.

Thank you kindly, Robin

Recommended topics

Hot tags