ModuleNotFoundError: No module named 'kafka.vendor.six.moves' in Dockerized Django Application

Question

I am facing an issue with my Dockerized Django application. I am using the following Dockerfile to build my application:

FROM python:alpine

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV DJANGO_SUPERUSER_PASSWORD datahub

RUN mkdir app
WORKDIR /app
COPY ./app .
RUN mkdir -p volumes

RUN apk update
RUN apk add --no-cache gcc python3-dev musl-dev mariadb-dev

RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt

RUN apk del gcc python3-dev musl-dev

CMD python3 manage.py makemigrations --noinput &&\
    while ! python3 manage.py migrate --noinput; do sleep 1; done && \ 
    python3 manage.py collectstatic --noinput &&\
    python3 manage.py createsuperuser --user datahub  --email admin@localhost --noinput;\
    python3 manage.py runserver 0.0.0.0:8000

In my requirements.txt file:

kafka-python==2.0.2

When I run my application inside the Docker container, I encounter the following error:

ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

Compelete Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/usr/local/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/usr/local/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/usr/local/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/usr/local/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/usr/local/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/usr/local/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/usr/local/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

I have already tried updating the Kafka package, checking dependencies, and installing the six package manually. However, the issue still persists. Can anyone provide insights on how to resolve this error?

Thank you in advance for your help!

Conventionality

have you considered making the runserver line your entrypoint? rather than running right out of the build of the image. once you have that change and have run the image, consider use bash in the container to poke around and see what got installed. might also check the build log for any issues installing the dependencies? – Meg

also, I note you did python3 on everything but the runserver line at the end. possible that you had 2 pythons in that image? – Meg

I just have python3. I edit that line – Conventionality

have you considered the change I suggested (not running the server from the build)? Then going in and starting in interactive mode with /bin/bash or /bin/sh so you can check for the module in the container's file system. – Meg

Yes. I consider. don't work. I'm use debain base distro instead of alpine. – Conventionality

Answer 1

This appears to be a Python 3.12 issue, I have the same error but in an entirely different context.

Instead of FROM python:alpine

I suggest you use FROM python:3.11

3.12 is still very new are there are many projects still trying to work out issues.

Spoilsport

Answer 2

This is issue with the current version of kafka-python

Instead use kafka-python-ng

$  pip install kafka-python-ng

Sensitometer

Answer 3

Replace Lib\site-packages\kafka\vendor\six.py with https://github.com/dpkp/kafka-python/blob/master/kafka/vendor/six.py, the failure is away against python 3.12 windows amd64 version.

Fieldsman

Answer 4

I had this issue for Python 3.12.

Function which raises this error responsible for snappy compression - this is a some 'range' function. In my case I've fixed it by this code. Don't know is snappy works now - I am not using this feature.

import sys, types

m = types.ModuleType('kafka.vendor.six.moves', 'Mock module')
setattr(m, 'range', range)
sys.modules['kafka.vendor.six.moves'] = m

Pickle

Answer 5

Make an external IMPORT, In my case:

if sys.version_info >= (3, 12, 0): sys.modules['kafka.vendor.six.moves'] = six.moves

Celt

The solution works, but there are two things to look out for: - The version must be an exact match. In my case, I needed to change sys.version_info >= (3, 12, 0) to sys.version_info >= (3, 12, 1) because I used version 3.12.1 - The if statement needs to be before the kafka-python import – Faiyum

Answer 6

I faced the same problem with Python 3.12.2 and KAFKA-Python 2.0.2.

So i decreased the Kafka version to 2.0.0 and changed the import lines in the codec.py file from :

from kafka.vendor import six
from kafka.vendor.six.moves import range

to

from kafka.vendor import six
from six.moves import range

and in my case, the problem is solved

Owsley

Answer 7

This issue is related to Python version 3.12, until it is fixed, you can try the below way I already tried and solved

1. Install the kafka-python-ng package :

pip install kafka-python-ng

2. Use this snippet before the main script

import six
import sys
if sys.version_info >= (3, 12, 0):
    sys.modules['kafka.vendor.six.moves'] = six.moves

Mitchell