How to run sidekiq in a separate different docker container apart from application

Question

How to run sidekiq in a separate different docker container apart from application. We are using whenever for the sidekiq jobs, but the jobs are getting triggered in all web containers

1. How do we setup the web container (rails app) and the sidekiq workers to scale horizontally as well (preferably as separate containers).
2. How should we manage database migrations since sidekiq, ui containers use the same image
3. I am trying to use the following snippet, is this correct or should it be any different?

version: '3.8'

services:
  foo-db:
    image: postgres:$POSTGRES_VERSION
    container_name: foo-db-container
    restart: unless-stopped
    env_file: .env
    volumes:
      - /var/lib/postgresql/data
    networks:
      - $FOO_NETWORK

  foo-redis:
    image: redis:$REDIS_VERSION
    container_name: foo-redis-container
    init: true
    sysctls:
      net.core.somaxconn: 511
    env_file: .env
    volumes:
      - /var/lib/redis/data
    networks:
      - $FOO_NETWORK

  foo-sidekiq:
    depends_on:
      - foo-db
      - foo-redis
    build: ./foo-ui
    command: bundle exec sidekiq
    env_file: .env
    volumes:
      - /var/lib/redis/data
    networks:
      - $FOO_NETWORK

  foo-service:
    build: foo-service
    # image: gcr.io/foo/foo-service:latest
    container_name: foo-service-container
    ports:
      - "$FOO_SERVICE_PORT:$FOO_SERVICE_PORT"
    env_file: .env
    networks:
      - $FOO_NETWORK

  foo-ui:
    build: ./foo-ui
    # image: gcr.io/foo/foo-ui:latest
    container_name: foo-ui-container
    depends_on:
      - foo-db
      - foo-redis
      - foo-sidekiq
      - foo-service
    ports:
      - "$FOO_UI_PORT:$FOO_UI_PORT"
    env_file: .env
    networks:
      - $FOO_NETWORK

networks:
  foo-network:

Lewiss

is this the actual compose.yml file you are using? It doesn't seem like it is nested properly (ie. foo-sidekiq: seems to be indented more than it should) – Shutdown

Yes its formatted properly, appreciate any inputs on how to manage this – Lewiss

Answer 1

Here's the list of things you should setup:

• Images

You should have a different image for your sidekiq and web application. That way, you can ensure whenever -w will be called only in the background (sidekiq) image, and not inside the web image.

Example image: https://gist.github.com/cesartalves/69f6440b97c89e9dee6cfffbdf9b7790 (adapt to your needs)

Important points:

• apt-get update && apt-get install -y cron - install crontab
• bundle exec whenever -w - running whenever
• CMD [ "bundle", "exec", "sidekiq" - run sidekiq as main process
• ENTRYPOINT [ "./docker-entrypoint.sh" ] - run cron in the background and other aditional commands you might need to do

To check if crontab is running, you can do one of the following after your container is running https://askubuntu.com/questions/85558/verify-if-crontab-works

About the "horizontal scalling", this should be done automatically if you have multiple sidekiq instances consuming the same redis pool - the more sidekiq instances, the faster your jobs run.

If by "horizontal" you mean being able to set up on which servers your crons will run, you could have multiple schedule.rb files, and each of your background containers should deploy only one of them to the crontab, by doing whenever -w -f config/schedule_specific_cron1.rb. You could pass the file as an environment variable.

2. Migrations

If all the application containers use the same database, have only one of them execute migrations. This will speed up your container starting and avoid any locking errors. I suggesting doing that in the "web" image since you will have multiple sidekiq container instances.

3. Gems

All your images will use the same gems. Therefore, you want to create a volume to store them, to speed up their building process:

foo-sidekiq:
    depends_on:
      - foo-db
      - foo-redis
    build: ./foo-ui
    command: bundle exec sidekiq
    env_file: .env
    volumes:
      - /var/lib/redis/data
      - gem_cache:/gems
    networks:
      - $FOO_NETWORK

volumes:
  gem_cache:

Inside your image:

ENV BUNDLE_PATH /gems

4. Orchestration

Which container orchestration system are you using?

Dependecies should be run on this order:

1. Run database container
2. Run redis container
3. Run web containers and background task containers

Ideally, you should be using something which allows you to specify how many sidekiq container instances you want.

--

I think that covers the gist. Let me know if you need more information.

Wald

1. How do I build a sidekiq only image, do you have a sample Dockerfile for that? – Lewiss

Do you have a base image already? Add it to your question, that makes it easier for me to adapt it to your scenario – Wald

foo-sidekiq, foo-service is supposed to share the same rails app container, which I feel is inefficient but there is no easy way to separate the runtime needed for foo-sidekiq – Lewiss

I'll add an example: gist.github.com/cesartalves/69f6440b97c89e9dee6cfffbdf9b7790 – Wald

You are still copying the entire app in the above Dockerfile Is there a way to optimize it – Lewiss

How do you make sure that foo-service doesn't run sidekiq nor whenever, what changes should be made to the Dockerfile and how should we optimize it – Lewiss

What does this mean "# latest two configs are not necessary", can you be explicit – Lewiss

"You are still copying the entire app in the above Dockerfile Is there a way to optimize it". As regards to the source code, no. You'll need an exact copy of the files need to run on both web and sidekiq containers. If there are parts that only sidekiq or the web process uses, you can alter the COPY command on each image to include only those. As regards the gems, they're already optimized, because you're installing them in a folder mapped to a folder and therefore, they will be cached across containers. – Wald

Let us continue this discussion in chat. – Wald