Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

Question

I am using the confluent-kafka Python client in my project. I'm trying to create a Docker image with this client.

I am facing the following error:-

#11 8.015 [pipenv.exceptions.InstallError]:       In file included from /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/Admin.c:17:
#11 8.015 [pipenv.exceptions.InstallError]:       /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#11 8.015 [pipenv.exceptions.InstallError]:          23 | #include <librdkafka/rdkafka.h>
#11 8.015 [pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
#11 8.015 [pipenv.exceptions.InstallError]:       compilation terminated.
#11 8.015 [pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
#11 8.016 [pipenv.exceptions.InstallError]:       [end of output]

Based on my search it is related to Apple M1 build for librdkafka.

Downswing

Please edit to post the relevant parts of your Dockerfile. Please include the base image FROM, and any other libraries/packages you tried to install, including confluent-kafka. – Gilded

If you're trying to build an ARM image, then yes, you'll probably have to use kafka-python instead – Diadelphous

I had the same issue on my M1 mac. I run docker build via docker-compose. I was able to resolved it by adding platforms: - "linux/amd64" under the build field in the docker-compose file. See docs.docker.com/compose/compose-file/compose-file-v2/#platform – Rupture

Answer 1

If you are installing confluent-kafka inside Docker container on M1, this helped me.

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka.git && \
  cd librdkafka && git checkout tags/v1.9.0 && \
  ./configure && make && make install && \
  cd ../ && rm -rf librdkafka

Rna

Specifically referring to the question about building the image in Docker - This works. I had to bump the tag to v2.02 but otherwise solves the problem. – Corruption

Works flawlessly for me on Ubuntu 22.04, BUT I had to change the tags/v1.9.0 to tags/1.8.2 in my case. This is the version of your confluent-kafka module (e.g. confluent-kafka==1.8.2 in requirements.txt). – Estovers

Answer 2

Please refer to this Github issue. The problem with M1 is that Homebrew is installed in a different location and so these variables need to be added to the environment by including these lines in your .zshrc file

export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib

Scopoline

why should it matter when building a docker image? the host platform should be completely independent right? – Maecenas

The same problem occurs on a MacOS host. The title of the question doesn't refer to a docker. This answer is not about the docker. – Clinkerbuilt

This fixed my issue. I was not trying to do anything with Docker so I wasn't in the same situation as OP. Keep in mind you might need to update the version in the path if your librdkafka version ever changes. You can find what version you're using with brew info librdkafka after it's installed. – Malignant

Answer 3

The Confluent engineers are obviously very focused on their paying customers, and many, many months after the release of Python 3.10, they still haven't released 3.10 wheels that include the binaries for the package.

https://github.com/confluentinc/confluent-kafka-python/issues/1219

You have two choices. 1, run Python 3.9; 2, install librdkafka-dev via apt before installing confluent-kafka (for a debian base image).

The issue is the same with Apple M1 - i.e, there are no pre-built wheels for the platform containing the binaries, but it is much easier to fix (by installing librdkafka-dev) if you are trying to build a amd64 Linux image.

Honduras

Will the 1st choice work for Alphine images as well? – Susian

Answer 4

I added these to my docker-compose.yml (according to https://mcmap.net/q/516136/-unable-to-install-confluent-kafka-quot-fatal-error-librdkafka-rdkafka-h-no-such-file-or-directory-quot and https://github.com/confluentinc/confluent-kafka-python/issues/65#issuecomment-269964346)

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka && \
  cd librdkafka && git checkout tags/v2.0.2 && \
  ./configure && make && make install && \
  ldconfig &&\
  cd ../ && rm -rf librdkafka

RUN pip install confluent-kafka==2.0.2

Delfeena

Answer 5

On a non-M1 mac, Python 3.10.2, I solved this with brew install librdkafka.

Dove

Answer 6

Prebuilt binary wheels of confluent-kafka-python for your Apple M1 seems to not exist.

Thus, in order to install the package, pip tries to build it itself from source. But librdkafka looks like to not be installed as per librdkafka/rdkafka.h: No such file or directory error you got.

You can find specific instructions to install from source for different platforms here.

Osteal

Answer 7

I solved this problem trying with Python 3.9.13 on a virtual environment.

Galata

Could you add some detail about your virtual env set up? – Crispation

Sure. 1 Navigate to the folder using cd 2 python3 -m venv env 3 source env/bin/activate Now you can install Python 3.9.13 and install confluent-kafka normally. Ctrl+D for quit from virtual env. – Galata