error while starting kafka broker

Asked 28/9, 2016 at 23:19 Answered 18/7 at 14:26

I was able to successfully set up zookeeper and one kafka broker yesterday. Everything worked as expected. I shut down kafka (ctrl + c) and then zookeeper.

Today I started zookeeper and when I started kafka (bin/kafka-server-start.sh config/server0.properties), I get the following error. I tried various remedies suggested (removing completely my kafka installation and doing it again from scratch). Still I get same error.

[2016-09-28 16:15:55,895] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
        at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
        at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:244)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)
[2016-09-28 16:15:55,896] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

All set up in mac

Dunkin answered 28/9, 2016 at 23:19 Comment(0)

I have faced the same issue while setting up the kafka and zookeeper multinode cluster.

Root Cause :

As zookeeper ensemble takes some time to setup so when we start kafka brokers, all zookeeper nodes might have not joined zookeeper ensemble yet.

So It may be the case that one kafka node connected to a zookeeper node which is not yet in the ensemble.

Another reason can be kafka broker is killed instead of stopped. When you kill your kafka process or it crashes and it does not close its zookeeper connection correctly.

Solution :

To avoid this situation make sure your zookeeper ensemble is up and running before starting kafka brokers.

Add proper waits between zookeeper and kafka broker services start and stop.

Update

As of April 2021, Confluent built KRaft Protocol which doesn't require zookeeper management separately and it is built within kafka brokers. Goal was to remove Apache Kafka’s dependency on ZooKeeper for metadata management so now with latest versions we can avoid synchronisation issues between kafka and zookeeper.

Ebner answered 19/12, 2017 at 9:56 Comment(4)

But how do you fix the issue once it occurs? – Odontoid 11/1, 2018 at 16:6

You can delete the node in zookeeper that's listed in the log. I'm surprised that the node isn't an ephemeral node cleaned on disconnect. – Exert 9/10, 2018 at 15:52

Go to zookeeper.properties and find the zookeeper data dir. Delete the contents of data dir in zookeeper. Restart zookeeper and kafka. It will work – Nelan 6/12, 2019 at 4:42

I am in windows environment and there is a point, I think, where you say: Anothre reason...broker killed; there is a bat for stopping kafka and for stopping zookeeper and I did nou use them - so I fall into that problem. Now I know that each time I stop the proceses - not killing them (or closing the cmd where they are running) - I am doing the right thing! Thank you for the tip @Shubham ! (since I am in tests, I start and stop it offen!) – Romo 20/1, 2020 at 15:2

@ShubhWIP's anwwer is correct, but is missing a vital information.

Add proper waits between zookeeper and kafka broker services start and stop.

Introduce a 20 second delay between zookeeper and kafka. The zookeeper has a session expiry time of 18000ms. It needs this time to declare the old session dead. If the Kafka broker is brought up before this happens, the broker shuts down with "Error while creating ephemeral at /broker/ids/id, node already exists".

Minstrelsy answered 17/3, 2021 at 6:1 Comment(1)

on docker-compose can do something like entrypoint: sh -c 'sleep 30 && /etc/confluent/docker/run' – Alyose 23/6, 2021 at 20:21

I found similar issue in my AWS server. The problem was zookeeper was running. So I killed the process first and tried again and it worked.

$ ps aux | grep zookeeper
$kill -9 <zookeeper.properties process>

Ebonee answered 8/6, 2017 at 3:14 Comment(1)

For me this happened on my local machine. I had a set of docker commands running on start up. One of which, I did not realise, was zookeeper. Once I killed the docker process I was able to started my own zookeeper and then the kafka server worked. – Odontoid 11/1, 2018 at 16:19

Under /tmp directory following directories have been removed:

kafka-log directory
Zookeeper-log directory

Then, I've restarted kafka service.

It works !!

Salmonoid answered 7/1, 2020 at 13:1 Comment(3)

this could have been a comment – Reflux 7/1, 2020 at 13:39

Bad answer. You destroy the precious data. Do you format your harddisk because you couldn't open a file? – Digger 8/6, 2021 at 7:27

I took a backup of log files, deleted the *-log folder from /tmp. It worked. – Ishmaelite 27/4, 2022 at 14:57

I came across the same issue and there are 2 ways to go about it

[Recommended] You need to clean up the broker ids in the zookeeper path /brokers/ids/[]. Use the zk-cli tool delete command to clean up the paths. Start your brokers and verify it registers with the coordinator.
The other way of resolving this is to change your broker-id from kafka server config and restarting the broker. However, this would corrupt your partitions and data is not recommended

Advice answered 8/4, 2017 at 9:22 Comment(1)

Worked for me, and possibly a better solution than the somewhat brutal deletion of zk and kafka temp directories. Many thanks, I would just add, for those not familiar, you can also use the ./bin/zookeeper-shell.sh in standalone kafka to do the job (eg ./kafka/kafka_2.13-2.8.0/bin/zookeeper-shell.sh 0.0.0.0:2181). Check the brokerid with ls /brokers/ids and then delete it with e.g. deleteall /brokers/ids/0. – Besot 4/8, 2021 at 10:16

The below approach worked for me.

Change the broker ID in config/server-1.properties to a different value. By default it is 0. Change it to 1 and test things again.

Mcnamee answered 19/5, 2018 at 2:23 Comment(0)

I had the when running multiple containers with docker-compose up and ending the processes with ctrl-C. Fixed it with docker-compose down.

Acrolith answered 18/11, 2020 at 8:27 Comment(1)

this deletes the attached volume as well... not helpful! I wanted to restart the docker after adding jdbc-driver to the broker. – Stick 9/11, 2022 at 8:48

I had similar issue in with docker container containing zookeeper and kafka (it isn't best practise to put two services in one container, but it was for testing purpose).

this was the error

[2018-11-21 12:54:24,434] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,434] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,435] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,453] INFO Loading logs. (kafka.log.LogManager)
[2018-11-21 12:54:24,457] INFO Logs loading complete in 4 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,468] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,469] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,694] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2018-11-21 12:54:24,728] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,750] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2018-11-21 12:54:24,790] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2018-11-21 12:54:24,802] ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72057672936325120' does not match current session '72057674644258816' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-11-21 12:54:24,803] INFO Result of znode creation at /brokers/ids/0 is: NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-11-21 12:54:24,808] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
    at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1525)
    at kafka.zk.KafkaZkClient.registerBrokerInZk(KafkaZkClient.scala:84)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:257)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)

Quickfix for me was to start kafka-server a few seconds later;

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties && (bin/kafka-server-start.sh config/server.properties || echo "kafka start failed. restarting ... " && sleep 3 && bin/kafka-server-start.sh config/server.properties)

Bookstand answered 21/11, 2018 at 13:39 Comment(0)

Make sure the kafka server shut down correctly by checking the default prot 9092 is already in use or not. Kill the process on 9092, and restart kafka server.

If not works, might be the existing kafka server (with broker id 0) is already in use. Give it a try to change broker.id to 1 instead of 0 in "server0.properties" (btw, why it was called server0, did you change any config here? by default, it should be server.properties),

Cardigan answered 29/9, 2016 at 0:52 Comment(0)

I was in the same boat in windows environment! And I am just in tests, so I have to close the processes every day since they are installed on my laptop which is put down daily! @Shubham gave me the idea but he did not elaborate so, I feel it's right to put it as an answer so that others who come in the same boat know what to do: It is important that the processes are not killed - in my case I closed the windows cmd in which the two processes were running. This is WRONG and the logs will not allow after that to start the kafka broker properly. I know now to stop the processes by running the proper bat (in \bin\windows folder where the other commands - for starting the servers- are). I will not kill the processes any more by closing the cmd. To go out of the pit I did erase the the kafka logs - but that is not the solution! The solution is to avoid killing the processes!

Romo answered 20/1, 2020 at 15:31 Comment(0)

I encountered this in a production environment and could not simply change the Kafka broker ID. After executing a rolling restart of all Zookeeper instances, I was able to successfully restart the Kafka broker. If you are running against a single Zookeeper instance, then restarting the single Zookeeper instance should suffice.

Quan answered 8/11, 2020 at 9:53 Comment(0)

I also encountered the same problem. You can follow my steps to do it:

A cluster with three nodes as an example:

For node h01: set broker.id=0 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory(log.dirs) and you will find a file meta.properties: open and set broker.id=0.

Next do similar work:

For node h02: set broker.id=1 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory (log.dirs) and you will find a file meta.properties: open and set broker.id=1.

For node h03: set broker.id=2 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory (log.dirs) and you will find a file meta.properties: open and set broker.id=2.

Last restart kafka.

Molybdenous answered 18/8, 2021 at 19:4 Comment(0)

The same issue was faced in my Windows machine.

First, start your zookeeper. Go to your main Kafka file location, like C:\kafka_2.13-3.6.1.

After that, manually put the path of zookeeper-server-start.bat file and also zookeeper.properties file.

This is because zookeeper-server-start.bat file is present inside the bin\windows under the Kafka folder and your config file is present in the main Kafka folder. So, you need to put manually path for both files same follow the procedure for starting the Kafka server.

Something like this:

C:\kafka_2.13-3.6.1>bin\windows\zookeeper-server-start.bat config\zookeeper.properties

C:\kafka_2.13-3.6.1>bin\windows\kafka-server-start.bat config\server.properties

Emendate answered 13/12, 2023 at 6:5 Comment(0)

For those who use Zookeeper and Kafka as a service on Linux I just added to exclude the temp files when service is started:

/etc/systemd/system/zookeeper.service

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStartPre=+/bin/bash -c 'rm -rf /tmp/zookeeper/*'
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

/etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/root/.sdkman/candidates/java/11.0.23-tem"
ExecStartPre=+/bin/bash -c 'rm -rf /tmp/kafka-logs/*'
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target

The temp files are configured as dataDir at /usr/local/kafka/config/zookeeper.properties and as log.dirs at /usr/local/kafka/config/server.properties

Underglaze answered 18/7 at 14:26 Comment(0)

-7

You just have to execute the zookeeper-server-stop.[sh|bat] script.

Squint answered 24/1, 2017 at 19:13 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Update

Recommended topics

Hot tags