Docker kafka on MacOS M1 Issues stuck on configuring

Question

I use macOS M1 Big Sur 11.2.3, but my kafka cannot running well and cannot create/list the topics. I don't know its because the OS or not, but the log for kafka is only like this:

docker-compose logs

list the topics logs

here's my docker compose:

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
    networks:
      - kafka_net
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - 9092:9092
    expose:
      - 29092
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
    restart: always
    networks:
      - kafka_net
networks:
  kafka_net:
    driver: bridge

I think kafka not yet to running, So i cannot list/create a topics. Do you guys have any idea of this? i already search possibilities about this, but i still don't have the problem solving. Thanks

Professoriate

as @santanu mentioned in his answer below, the latest Kafka does not require zookeeper and can simplify this setup – Chickenlivered

Answer 1

For startup kafka in docker on mac m1 i do next things :

1. Clone kafka repo and build on my macbook pro m1 (./gradlew clean releaseTarGz)
2. Create docker-compose project dir with next structure

• root dir
  • kafka_m1
    • kafka (unziped kafka distr)
    • Dockerfile
  • docker-compose.yml

3. Configure server.properties file (root dir/kafka_m1/kafka/config/server.properties). Main params:

   zookeeper.connect=zookeeper:2181
   listeners=PLAINTEXT://:9092
   advertised.listeners=PLAINTEXT://127.0.0.1:9092
   listener.security.protocol.map=PLAINTEXT:PLAINTEXT

4. kafka`s Dockerfile (openjdk:15.0.2-jdk was build for m1):

    FROM openjdk:15.0.2-jdk
    
    WORKDIR /
    COPY . /
    
    EXPOSE 9092
    EXPOSE 8092
    EXPOSE 9092
    EXPOSE 10092
    EXPOSE 11092
    EXPOSE 12092
    
    ENTRYPOINT [ "/kafka/bin/kafka-server-start.sh", "/kafka/config/server.properties" ]

5. docker-compose.yml (i choose zookeeper:3.7.0 because it has arm build):

  version: "2"
  services:
  zookeeper:
    image: zookeeper:3.7.0
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    build: ./kafka_m1
    ports:
      - "127.0.0.1:9092:9092"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

6. Start docker-compose with kafka_m1 image building.

It`s fast receipt for run local kafka in docker

Deboer

Does anyone tried this as I have the same situation now? – Waterrepellent

You do not need to clone or build the kafka Repo to use it in a docker container. I don't think this answer is correct – Chickenlivered

Answer 2

Just add platform to your docker-compose.yml file:

platform: linux/amd64

services:
  zookeeper:
    image: wurstmeister/zookeeper
    platform: linux/amd64
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
    networks:
      - kafka_net
    ...

If above option doesn't help then you should try the following:

version: '3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Eisenhart

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Bailor

Thanks! wurstmeister images are working fine on my Apple M1 – Patriciate

Answer 3

This mentioned docker-compose.yml is working for me on Mac M1-Pro. The latest Kafka also doesn't need zookeeper. This also contains Kafka-UI container.

version: '3'

networks:
  app-tier:
    driver: bridge

services:
  kafka:
    image: 'bitnami/kafka:latest'
    container_name: kafka
    hostname: kafka
    ports:
      - '9092:9092'
    networks:
      - app-tier
    environment:
      KAFKA_CFG_NODE_ID: 0
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka:29093
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: 'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8090:8080"
    depends_on:
      - kafka
    networks:
      - app-tier
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS: kafka:29092

Screenshot of Kafka-UI

Link: https://hub.docker.com/r/bitnami/kafka

Raynold

This is exactly what I needed. Simple and it works great on Mac. Thanks! I didn't know we could get rid of zookeeper with the latest version – Chickenlivered

Answer 4

the docker-compose.yaml file is ok, you just have to execute the following commands into the kafka container:

1. kafka-topics.sh --zookeeper zookeeper:2181 --topic test --create --partitions 3 --replication-factor 1
2. kafka-topics.sh --zookeeper zookeeper:2181 --list

btw: you can enter to the kafka container using the command: docker exec -it CONTAINER_ID bash

Stereotropism

Answer 5

Let me provide the docker-compose file that I spent a whole day creating.

Below docker-compose file builds containers for the zookeeper, Kafka, and kafka-manager CMAK

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    restart: always
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    restart: always
    ports:
      - "9092:9092"
      - "1099:1099"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: kafka
      JMX_PORT: 1099
  kafka_manager:
    image: hlebalbau/kafka-manager:stable
    restart: always
    ports:
      - "9000:9000"
    environment:
      ZK_HOSTS: zookeeper:2181
    command: -Dpidfile.path=/dev/null

docker-compose up --build -d

Then for managing cluster through CMAK you require to execute following

docker exec -it zookeeper bash
./bin/zkCli.sh
create /kafka-manager/mutex ""
create /kafka-manager/mutex/locks ""
create /kafka-manager/mutex/leases ""

Saline

I am getting error: error getting credentials - err: docker-credential-desktop resolves to executable in current directory (./docker-credential-desktop), out: `` I am using Colima, instead of Docker Desktop – Tarsuss