Not able to create kafka topic using docker-compose

Question

I am trying to set up Kafka rest proxy using a docker. But the topic I am providing in the configuration isn't creating.

I am checking Kafka Topic with API: curl "http://metrics-kafka-rest:38082/topics" and I am getting this response: ["__confluent.support.metrics","_schemas"]

Below in the config that I have used in docker-compose:

    image: confluentinc/cp-zookeeper:5.3.0
    container_name: 'metrics-zookeeper'
    restart: always
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SASL_ENABLED: "FALSE"

  metrics-kafka:
    image: confluentinc/cp-kafka:5.3.0
    container_name: 'metrics-kafka'
    restart: always
    ports:
      - "29092:29092"
    depends_on:
      - metrics-zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_ZOOKEEPER_CONNECT: metrics-zookeeper:32181
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://metrics-kafka:29092
      KAFKA_CREATE_TOPICS: "Notification:1:1"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_ENABLE: 'false'

  metrics-schema-registry:
    image: confluentinc/cp-schema-registry:5.3.0
    container_name: 'metrics-schema-registry'
    restart: always
    ports:
      - "38081:38081"
    depends_on:
      - metrics-kafka
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: metrics-zookeeper:32181
      SCHEMA_REGISTRY_HOST_NAME: metrics-schema-registry
      SCHEMA_REGISTRY_LISTENERS: "http://metrics-schema-registry:38081"
      SCHEMA_REGISTRY_DEBUG: "true"

  metrics-kafka-rest:
    image: confluentinc/cp-kafka-rest:5.3.0
    container_name: 'metrics-kafka-rest'
    restart: always
    ports:
      - "38082:38082"
    depends_on:
      - metrics-schema-registry
    environment:
      KAFKA_REST_ZOOKEEPER_CONNECT: metrics-zookeeper:32181
      KAFKA_REST_SCHEMA_REGISTRY_URL: "http://metrics-schema-registry:38081"
      KAFKA_REST_HOST_NAME: metrics-kafka-rest
      KAFKA_REST_LISTENERS: "http://metrics-kafka-rest:38082"
      KAFKA_REST_DEBUG: "true"

I expect that when I hit API of getting the list of topics then it should contain topic Notification.

Miyokomizar

Answer 1

KAFKA_CREATE_TOPICS is not a supported Environment variable for the cp-kafka image that you're using.

Since you already have KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" then you can just start using the broker and topics will be created as and when they're first referenced by the producer or consumer.

Osteoarthritis

So how should we create default topics with cp-kafka at container startup time? – Elspeth

Thanks a bunch...just spent 20mins pulling hair ! – Malraux

But is it possible to define a name previously? [Edited] I think that it will answer my question environment: KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact" See: registry.hub.docker.com/r/wurstmeister/kafka – Alterative

Answer 2

Agree with @Robin, and give some config references because some guys mix the env variables KAFKA_CREATE_TOPICS for images wurstmeister/kafka: https://registry.hub.docker.com/r/wurstmeister/kafka and confluentinc/cp-kafka:https://docs.confluent.io/platform/current/installation/docker/config-reference.html#optional-confluent-enterprise-ak-settings This is to say KAFKA_CREATE_TOPICS is not a supported Environment variable for the cp-kafka, it's for wurstmeister/kafka

Levorotatory