Python module not found in docker container

Question

I'm trying to run a django-cookiecutter project, but I'm getting a ModuleNotFoundError: No module named 'iprestrict' error in some of the containers. I checked out our repository that my colleagues can successfully build and run using both Docker for Windows and Docker for Linux. I've been debugging this for a while, but now I'm at a loss and asking for your help.

This is my setup:

$ uname -r
5.0.0-36-generic

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.04
Release:    19.04
Codename:   disco

$ docker -v
Docker version 19.03.3, build a872fc2f86

$ docker-compose -v
docker-compose version 1.25.0, build 0a186604

$ service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 13:02:07 CET; 1h 59min ago
     Docs: https://docs.docker.com
 Main PID: 9382 (dockerd)
    Tasks: 41
   Memory: 518.6M
   CGroup: /system.slice/docker.service
           ├─ 9382 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─14817 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5432 -container-ip 172.30.0.2 -container-port 5432
           └─14829 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8025 -container-ip 172.30.0.3 -container-port 8025

To get a clean start I ran the following commands in my project directory:

$ docker-compose down --volumes
$ docker-compose build --no-cache

The project is successfully built without error messages, these are the last lines, having deleted a bunch of packages for readability. What can be seen is that django-iprestrict got successfully installed:

Successfully installed [...] django-environ-0.4.5 django-extensions-2.1.5 django-iprestrict-1.7.0 django-model-utils-3.1.2 [...]

This is the relevant part from the Dockerfile:

FROM python:3.6.8-alpine

ENV PYTHONUNBUFFERED 1

[...]

COPY ./requirements /requirements
RUN pip install -r /requirements/local.txt

Among the packages in local.txt are these:

[...]
django==2.1.5  # https://www.djangoproject.com/
django-environ==0.4.5  # https://github.com/joke2k/django-environ
django-model-utils==3.1.2  # https://github.com/jazzband/django-model-utils
django-iprestrict==1.7.0  # https://github.com/muccg/django-iprestrict
[...]

From the docker-compose.yml:

services:
  django:
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    image: companyname_local_django
    depends_on:
      - postgres

    # these configs need to be overriden by the local and prod service configurations
    volumes:
      - .:/app
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres

And from docker-compose.override.yml:

services:
  django: &django
    ports:
      - "8000:8000"
    command: /start
    environment:
      - TERM=xterm-256color
    restart: on-failure

    # these configs are overriding the values in docker-compose.yml django service configuration
    volumes:
      - webpack_build:/app/build
      - .:/app
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres

  postgres:
    volumes:
      - local_postgres_data:/var/lib/postgresql/data
      - local_postgres_data_backups:/backups
    ports:
      - "5432:5432"

  celeryworker:
    <<: *django
    image: companyname_local_celeryworker
    depends_on:
      - postgres
    ports: []
    command: /start-celeryworker

  celerybeat:
    <<: *django
    image: companyname_local_celerybeat
    depends_on:
      - postgres
    ports: []
    command: /start-celerybeat

  flower:
    <<: *django
    image: companyname_local_flower
    ports:
      - "5555:5555"
    command: /start-flower

Then I try to run it using $ docker-compose up and I get the following error messages:

celerybeat_1    | celery beat v4.2.1 (windowlicker) is starting.
celerybeat_1    | Traceback (most recent call last):
celerybeat_1    |   File "/usr/local/bin/celery", line 10, in <module>
celerybeat_1    |     sys.exit(main())
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
celerybeat_1    |     _main()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
celerybeat_1    |     cmd.execute_from_commandline(argv)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
celerybeat_1    |     super(CeleryCommand, self).execute_from_commandline(argv)))
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
celerybeat_1    |     return self.handle_argv(self.prog_name, argv[1:])
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in handle_argv
celerybeat_1    |     return self.execute(command, argv)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 420, in execute
celerybeat_1    |     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279, in run_from_argv
celerybeat_1    |     sys.argv if argv is None else argv, command)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 363, in handle_argv
celerybeat_1    |     return self(*args, **options)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 238, in __call__
celerybeat_1    |     ret = self.run(*args, **kwargs)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/bin/beat.py", line 109, in run
celerybeat_1    |     return beat().run()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/apps/beat.py", line 79, in run
celerybeat_1    |     self.init_loader()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/apps/beat.py", line 128, in init_loader
celerybeat_1    |     self.app.loader.init_worker()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/loaders/base.py", line 114, in init_worker
celerybeat_1    |     self.import_default_modules()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/loaders/base.py", line 108, in import_default_modules
celerybeat_1    |     raise response
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/utils/dispatch/signal.py", line 288, in send
celerybeat_1    |     response = receiver(signal=self, sender=sender, **named)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/fixups/django.py", line 82, in on_import_modules
celerybeat_1    |     self.worker_fixup.validate_models()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/fixups/django.py", line 120, in validate_models
celerybeat_1    |     self.django_setup()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/celery/fixups/django.py", line 116, in django_setup
celerybeat_1    |     django.setup()
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
celerybeat_1    |     apps.populate(settings.INSTALLED_APPS)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/django/apps/registry.py", line 89, in populate
celerybeat_1    |     app_config = AppConfig.create(entry)
celerybeat_1    |   File "/usr/local/lib/python3.6/site-packages/django/apps/config.py", line 90, in create
celerybeat_1    |     module = import_module(entry)
celerybeat_1    |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
celerybeat_1    |     return _bootstrap._gcd_import(name[level:], package, level)
celerybeat_1    |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
celerybeat_1    |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
celerybeat_1    |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
celerybeat_1    | ModuleNotFoundError: No module named 'iprestrict'

Same error for celeryworker_1 and flower_1 containers.

Then I checked the installed packages in the different containers and found that iprestrict is missing in celerybeat, celeryworker and flower, but not in django (which they expand):

$ docker-compose run --rm django sh
/app $ pip freeze | grep django-
django-allauth==0.38.0
django-cors-headers==2.4.0
django-coverage-plugin==1.6.0
django-crispy-forms==1.7.2
django-debug-toolbar==1.11
django-environ==0.4.5
django-extensions==2.1.5
django-iprestrict==1.7.0
django-model-utils==3.1.2
django-react-templatetags==5.2.1
django-redis==4.10.0
django-simple-history==2.7.0
django-webpack-loader==0.6.0

/app $ python
Python 3.6.8 (default, Feb  6 2019, 01:56:13) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import iprestrict
>>> 

$ docker-compose run --rm flower sh
/app $ pip freeze | grep django-
django-allauth==0.38.0
django-cors-headers==2.4.0
django-coverage-plugin==1.6.0
django-crispy-forms==1.7.2
django-debug-toolbar==1.11
django-environ==0.4.5
django-extensions==2.1.5
django-model-utils==3.1.2
django-react-templatetags==5.2.1
django-redis==4.10.0
django-simple-history==2.7.0
django-webpack-loader==0.6.0

/app $ python
Python 3.6.8 (default, Feb  6 2019, 01:56:13) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import iprestrict
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'iprestrict'
>>> 

As you can see, all the packages with the correct versions as defined in local.txt are there except django-iprestrict==1.7.0.

Does anyone have an idea what could be the reason for this? Thanks!

Unintelligible

What does the celerybeat Dockerfile install? You say it expands the django Dockerfile - have you checked if during the build of the celerybeat image (especially the installation of packages in celerybeat itself) any warnings happen. Maybe there is a package/version conflict. – Tc

Try running apt-get update before RUN pip install -r /requirements/local.txt – Imprecate

@VarunMukundhan I'm doing that, just omitted it in the excerpt of the file I posted. Since it's an alpine image I'm running RUN apk update as the first command. – Unintelligible

@Tc Thanks for the hint, there is no separate Dockerfile for the other containers, though. They're using the django one, just executing different commands when launching. When I execute docker-compose build, the output is flower uses an image, skipping celerybeat uses an image, skipping celeryworker uses an image, skipping Building frontend Building postgres Building django. I'm locking closer into the logs during build of the django container now. – Unintelligible

@Unintelligible if you are using the same image for all of them, and it works for your colleagues then maybe you have cached images that are used for the other services. Though that seems far fetched - you probably have deleted all your older stuff and caches several times by now. – Tc

Update: I managed to fix it and it works on all machines I'm using (Windows, Ubuntu and macOS). Although I don't really understand why because it works for my colleagues and used to work on my machine as well... What I did was changing the image in the docker-compose file for the services celeryworker, celerybeat and flower to companyname_local_django:latest. My reasoning was that the image with this name has definitely been built before and it should be used as the base for these services. It works now, but I still don't understand why it stopped working in the first place. – Unintelligible