Login/Register
Apple M1: can't start the confluent control center using docker
Asked Answered
Solved dockerapache-kafkaconfluent-platformapple-m1confluent-control-center
U

2

7

I would like to start the Confluent control center using docker as prescribed in the document:

https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html

This is the docker-compose.yaml file they provided;

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:5.5.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  schema-registry:
    image: confluentinc/cp-schema-registry:5.5.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'

  connect:
    image: cnfldemos/cp-server-connect-datagen:0.3.2-5.5.0
    hostname: connect
    container_name: connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      # CLASSPATH required due to CC-2422
      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-5.5.0.jar
      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR

  control-center:
    image: confluentinc/cp-enterprise-control-center:5.5.0
    hostname: control-center
    container_name: control-center
    depends_on:
      - zookeeper
      - broker
      - schema-registry
      - connect
      - ksqldb-server
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
      CONTROL_CENTER_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      CONTROL_CENTER_CONNECT_CLUSTER: 'connect:8083'
      CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
      CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
      CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
      CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
      CONFLUENT_METRICS_TOPIC_REPLICATION: 1
      PORT: 9021

  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:5.5.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
      - connect
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"

  ksqldb-cli:
    image: confluentinc/cp-ksqldb-cli:5.5.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - connect
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

  ksql-datagen:
    image: confluentinc/ksqldb-examples:5.5.0
    hostname: ksql-datagen
    container_name: ksql-datagen
    depends_on:
      - ksqldb-server
      - broker
      - schema-registry
      - connect
    command: "bash -c 'echo Waiting for Kafka to be ready... && \
                       cub kafka-ready -b broker:29092 1 40 && \
                       echo Waiting for Confluent Schema Registry to be ready... && \
                       cub sr-ready schema-registry 8081 40 && \
                       echo Waiting a few seconds for topic creation to finish... && \
                       sleep 11 && \
                       tail -f /dev/null'"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      STREAMS_BOOTSTRAP_SERVERS: broker:29092
      STREAMS_SCHEMA_REGISTRY_HOST: schema-registry
      STREAMS_SCHEMA_REGISTRY_PORT: 8081

  rest-proxy:
    image: confluentinc/cp-kafka-rest:5.5.0
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - 8082:8082
    hostname: rest-proxy
    container_name: rest-proxy
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'

I run the services using the command:

docker-compose up -d

After a while the services look -


enter image description here


and I was not able to reach the control center in the localhost:9021 they the port is listening. I look for the container logs and find these:


control-center


control-center  | [2021-12-03 06:12:52,137] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-4-consumer, groupId=_confluent-controlcenter-5-5-0-1] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,143] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-4-consumer, groupId=_confluent-controlcenter-5-5-0-1] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,142] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-8-consumer, groupId=_confluent-controlcenter-5-5-0-1] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,146] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-8-consumer, groupId=_confluent-controlcenter-5-5-0-1] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,151] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-2-consumer, groupId=_confluent-controlcenter-5-5-0-1] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,155] INFO [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-2-consumer, groupId=_confluent-controlcenter-5-5-0-1] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
control-center  | [2021-12-03 06:12:52,277] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-7-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 8 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [2021-12-03 06:12:52,278] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-4-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 6 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [2021-12-03 06:12:52,296] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-3-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 6 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [2021-12-03 06:12:52,298] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-5-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 7 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [2021-12-03 06:12:52,306] INFO streams in state=REBALANCING (io.confluent.controlcenter.streams.KafkaStreamsManager)
control-center  | [2021-12-03 06:12:52,297] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-8-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 8 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [2021-12-03 06:12:52,310] INFO tocheck=[Store{name=KSTREAM-OUTEROTHER-0000000105-store, rollup=false}, Store{name=TriggerActionsStore, rollup=false}, Store{name=TriggerEventsStore, rollup=false}, Store{name=AlertHistoryStore, rollup=false}, Store{name=MonitoringMessageAggregatorWindows, rollup=true}, Store{name=MetricsAggregateStore, rollup=false}, Store{name=Group, rollup=true}, Store{name=MonitoringTriggerStore, rollup=false}, Store{name=group-aggregate-store, rollup=true}, Store{name=monitoring-aggregate-rekey-store, rollup=false}, Store{name=aggregatedTopicPartitionTableWindows, rollup=true}, Store{name=MonitoringVerifierStore, rollup=false}, Store{name=KSTREAM-OUTERTHIS-0000000104-store, rollup=false}, Store{name=aggregate-topic-partition-store, rollup=false}, Store{name=MonitoringStream, rollup=true}] (io.confluent.controlcenter.streams.KafkaStreamsManager)
control-center  | [2021-12-03 06:12:52,310] INFO streams in state=REBALANCING (io.confluent.controlcenter.streams.KafkaStreamsManager)
control-center  | #
control-center  | # A fatal error has been detected by the Java Runtime Environment:
control-center  | #
control-center  | #  SIGSEGV[thread 283969722112 also had an error]
control-center  | [thread 283968669440 also had an error]
control-center  |  (0xb) at pc=0x000000400c04f320, pid=1, tid=0x0000004225fa2700
control-center  | #
control-center  | # JRE version: OpenJDK Runtime Environment (Zulu 8.38.0.13-CA-linux64) (8.0_212-b04) (build 1.8.0_212-b04)
control-center  | # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
control-center  | # Problematic frame:
control-center  | # [thread 284374480640 also had an error]
control-center  | J 2840 C1 org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup()V (48 bytes) @ 0x000000400c04f320 [0x000000400c04f300+0x20]
control-center  | #
control-center  | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
control-center  | #
control-center  | # An error report file with more information is saved as:
control-center  | # //hs_err_pid1.log
control-center  | [thread 284375533312 also had an error]
control-center  | [thread 284378691328 also had an error]
control-center  | [thread 284377638656 also had an error]
control-center  | [thread 279529854720 also had an error]
control-center  | [2021-12-03 06:12:52,431] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-6-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 8 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [thread 284376585984 also had an error]
control-center  | #
control-center  | # If you would like to submit a bug report, please visit:
control-center  | #   http://www.azulsystems.com/support/
control-center  | #
control-center  | qemu: uncaught target signal 6 (Aborted) - core dumped


broker logs


    broker  | [2021-12-03 06:15:52,919] TRACE [Broker id=1] Cached leader info UpdateMetadataPartitionState(topicName='docker-connect-status', partitionIndex=4, controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], zkVersion=0, replicas=[1], observers=[], offlineReplicas=[]) for partition docker-connect-status-4 in response to UpdateMetadata request sent by controller 1 epoch 1 with correlation id 74 (state.change.logger)
    broker  | [2021-12-03 06:15:52,919] TRACE [Broker id=1] Cached leader info UpdateMetadataPartitionState(topicName='docker-connect-status', partitionIndex=1, controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], zkVersion=0, replicas=[1], observers=[], offlineReplicas=[]) for partition docker-connect-status-1 in response to UpdateMetadata request sent by controller 1 epoch 1 with correlation id 74 (state.change.logger)
    broker  | [2021-12-03 06:15:52,919] TRACE [Broker id=1] Cached leader info UpdateMetadataPartitionState(topicName='docker-connect-status', partitionIndex=3, controllerEpoch=1, leader=1, leaderEpoch=0, isr=[1], zkVersion=0, replicas=[1], observers=[], offlineReplicas=[]) for partition docker-connect-status-3 in response to UpdateMetadata request sent by controller 1 epoch 1 with correlation id 74 (state.change.logger)
    broker  | [2021-12-03 06:15:52,940] TRACE [Controller id=1 epoch=1] Received response {error_code=0,_tagged_fields={}} for request UPDATE_METADATA with correlation id 74 sent to broker broker:29092 (id: 1 rack: null) (state.change.logger)

I believe this control center log provided the main error:


# A fatal error has been detected by the Java Runtime Environment:
control-center  | #
control-center  | #  SIGSEGV[thread 283969722112 also had an error]
control-center  | [thread 283968669440 also had an error]
control-center  |  (0xb) at pc=0x000000400c04f320, pid=1, tid=0x0000004225fa2700
control-center  | #
control-center  | # JRE version: OpenJDK Runtime Environment (Zulu 8.38.0.13-CA-linux64) (8.0_212-b04) (build 1.8.0_212-b04)
control-center  | # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
control-center  | # Problematic frame:
control-center  | # [thread 284374480640 also had an error]
control-center  | J 2840 C1 org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup()V (48 bytes) @ 0x000000400c04f320 [0x000000400c04f300+0x20]
control-center  | #
control-center  | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
control-center  | #
control-center  | # An error report file with more information is saved as:
control-center  | # //hs_err_pid1.log
control-center  | [thread 284375533312 also had an error]
control-center  | [thread 284378691328 also had an error]
control-center  | [thread 284377638656 also had an error]
control-center  | [thread 279529854720 also had an error]
control-center  | [2021-12-03 06:12:52,431] WARN [Consumer clientId=_confluent-controlcenter-5-5-0-1-34f98e4d-4e61-42b9-93f5-dcd8ea94e837-StreamThread-6-consumer, groupId=_confluent-controlcenter-5-5-0-1] Error while fetching metadata with correlation id 8 : {_confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-MonitoringStream-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-repartition=UNKNOWN_TOPIC_OR_PARTITION, _confluent-controlcenter-5-5-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-repartition=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
control-center  | [thread 284376585984 also had an error]
control-center  | #
control-center  | # If you would like to submit a bug report, please visit:
control-center  | #   http://www.azulsystems.com/support/
control-center  | #
control-center  | qemu: uncaught target signal 6 (Aborted) - core dumped

This is the Java version that I use:


$ java --version
openjdk 11.0.9.1 2020-11-04
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode)


Previosuly, I never had an issue with the confluent platform using the docker, so, is this particular to the Apple m1 chip and how do I solve it until confluent provides an official solution?



Unsightly answered 3/12, 2021 at 7:8 Comment(12)
java --version on your Mac has nothing to do with the Java used in the containers. The JRE that is in the containers is for AMD64, not ARM, so you could try adding a platform directive to the Compose file for each of the containers. Secondly, it is unclear how much memory you gave Docker, but the recommendation for actually running Control Center on its own is over 4GB, and you need an additional 4GB for running everything else, especially Connect and KSQL (so give Docker daemon at least 8-10GB). Or you can remove all the services you don't actually need(REST Proxy? datagen? The KSQL CLI?) – Breslau
The memory is good at 10 GB - I see no reason to provide more than this. I tried to comment out the other services but this doesn't solve the issue. – Unsightly
I don't have an M1 Mac to test with (maybe people in the Confluent Forums do), but you may want to remove the ksql logs from the question if you're only wanting to ask about Control Center – Breslau
How about broker? I tried run your docker-compose.yml in my M1, it fails. The broker log says qemu: uncaught target signal 11 (Segmentation fault) - core dumped. – Sharpedged
@J.Song I get similar error from the control center: control-center | qemu: uncaught target signal 6 (Aborted) - core dumped The broker looks fine to me. Is there a solution for the issue? I have similar issue with lessio.io docker image as well, this works after I modify the Dockerfile – Unsightly
At the moment I only run 3 services: ` NAME COMMAND SERVICE STATUS PORTS broker "/etc/confluent/dock…" broker running 0.0.0.0:9092->9092/tcp control-center "/etc/confluent/dock…" control-center running 0.0.0.0:9021->9021/tcp zookeeper "/etc/confluent/dock…" zookeeper running 0.0.0.0:2181->2181/tcp ` – Unsightly
@Breslau The ksql log was removed. A long list of people having the same issues: github.com/docker/for-mac/issues/5123 – Unsightly
Ok, here are my thoughts. The images are for amd64 architecture, and the JVMs inside docker container are Zulu8. I think this JVM does not compatible with M1. I tested my kafka image running on openjdk-11, it works very fine. My container image is also amd64 architecture. – Sharpedged
Do you use openjdk-11 in a local computer and this makes it work? I would really appreciate it if you write an answer with detailed instructions as I'm stuck for the last few days. My goal would be to up the confluent platform. – Unsightly
@Arefe, I made my own kafka container image on top of openjdk:11. It works fine. I suggest, you can download confluent platform community edition, so you can build your own confluent image with other JVM. But first of all, DO YOU really have to run the container on your M1? – Sharpedged
@J.Song I need to run it using containers, and when running through the source code this works fine. So no issue there. Would you mind writing a detailed answer about how you make these images? Thanks. – Unsightly
Let us continue this discussion in chat. – Sharpedged
T
8

Confluent has released Confluent Platform Docker images that work on M1 machines.

I just validated using https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html. I was able to log in to Control Center on my Macbook M1.

Tenebrific answered 10/7, 2022 at 19:0 Comment(0)
B
3

I experienced the same - boosting memory and CPU, even all the way up, did not help. Confluent has mentioned they are working on it - in the issue the suggestion is to use a hypervisor or a cheap hosted instance:

In the meantime, I recommend running an M1-supported hypervisor, using some cloud instances for development, or using Confluent Cloud basic clusters for development. - @addisonhuddy

This other SO post explains how to create your own docker-compose for Kafka using ARM supported images, but is a manual setup not Confluent (as asked in original question).

Baggott answered 10/2, 2022 at 6:57 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.