Can't start zookeeper

Question

I'm using confluent platform, the zookeeper is active with status lookup. but when I try to start kafka with confluent it shows zookeeper is down.

$ sudo service zookeeper status
Redirecting to /bin/systemctl status  zookeeper.service
● zookeeper.service - Zookeeper
   Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-08-08 17:25:34 PDT; 16h ago
     Docs: http://kafka.apache.org/documentation.html
  Process: 3774 ExecStop=/var/www/confluent/bin/zookeeper-server-stop (code=exited, status=1/FAILURE)
 Main PID: 3785 (java)
   CGroup: /system.slice/zookeeper.service
           └─3785 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log...

 zookeeper[3785]: [2017-08-08 17:26:09,005] INFO Processed session termination for sessionid: 0x15dc460fd0c0000 (org.apache.zooke...Processor)
 zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0004, timeout of 60000ms exceeded (org.apache.zook...perServer)
 zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0002, timeout of 60000ms exceeded (org.apache.zook...perServer)
 zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0003, timeout of 60000ms exceeded (org.apache.zook...perServer)
 zookeeper[3785]: [2017-08-08 17:26:39,001] INFO Processed session termination for sessionid: 0x15dc4364baf0004 (org.apache.zooke...Processor)
 zookeeper[3785]: [2017-08-08 17:26:39,002] INFO Processed session termination for sessionid: 0x15dc4364baf0002 (org.apache.zooke...Processor)
 zookeeper[3785]: [2017-08-08 17:26:39,002] INFO Processed session termination for sessionid: 0x15dc4364baf0003 (org.apache.zooke...Processor)
 zookeeper[3785]: [2017-08-09 09:56:26,711] INFO Accepted socket connection from /127.0.0.1:46446 (org.apache.zookeeper.server.NI...xnFactory)
 zookeeper[3785]: [2017-08-09 09:59:14,796] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -72...erverCnxn)
 zookeeper[3785]: [2017-08-09 09:59:14,796] INFO Closed socket connection for client /127.0.0.1:46446 (no session established for...erverCnxn)
Hint: Some lines were ellipsized, use -l to show in full.

---

$ confluent start kafka
Starting zookeeper
|Zookeeper failed to start
zookeeper is [DOWN]
Cannot start Kafka, Zookeeper is not running. Check your deployment

Ciaracibber

It looks a lot like someone is sending malformed data to zookeeper, which causes it to crash. Does this happen straight after you start Kafka? If so, what does the Kafka log say? – Kieffer

The confluent cli will start zookeeper as well so you don't need to start it separately with systemd. You can get the zookeeper logs using the command confluent log zookeeper to help debug why it failed but I think its just failing to start a second zookeeper since you already have one running. – Grate

Answer 1

This is because zookeeper is already running, you can check the process with

ps aux|grep zookeeper

and kill the process manually and it is gonna work.

Resort

Thanks a lot i was stuck because of this for almost half a day ! – Conveyance

Answer 2

The most common cause for the message you are seeing when running:

confluent start kafka

and informs you that zookeeper is down, is that there's another zookeeper instance that is currently running, and the new zookeeper instance can not bind to its required port (by default this port is 2181).

A few options at your disposal to figure out what's the other zookeeper instance that is currently running when you try to issue confluent start kafka are:

1. run jps to see the running java processes. Zookeeper is the process named QuorumPeerMain next to its process ID. (equivalent to running ps xuaww | grep -i zookeeper or equivalent).
2. run lsof -i :2181 to figure out what the process that is running and has reserved the default zookeeper port (in this example 2181, but might be different in your system).

Try running confluent start kafka again after stopping the above process.

Convince

Answer 3

I received the same message. In my case I didn't set $JAVA_HOME variable properly.

Mathew

Answer 4

You are mixing two installations.

confluent start kafka would depend on you running confluent start zookeeper.

Rather, it seems you already have systemctl running Zookeeper, so you should ideally just configure your server.properties and use the regular kafka-server-start script. And/or create a systemctl file for Kafka.

Bilander

if i have my kafka and zookeeper started via ambari should i just start it from there? i try to stop my zookeeper and start it by using confluent local start they started well but when i check my ambari, zookeeper server stopped. – Mcalpin

it just happend as it is. Zookeeper, kafka is running and green state on ambari view. and when ssh confluent local status everything is down. it returning zookeeper is [DOWN] . When i stop zookeeper service, and then start it on ssh confluent local start everything is up. Even everything is up but when i check it on ambari, zookeper server is stop. what happened?? – Mcalpin

@Mcalpin You should never use confluent command if you are using Ambari. Always use Ambari, if you have it. Ambari cannot share/manage the same processes – Bilander

HI, thank you so much for your respond, i really appreciate that. Alright noted that, is everything can be done in ambari? i dont find enough documentation about add services that not provided inside ambari (like confluent) so i install it via apt-get. What about creating topics can we do it in ambari or its just okay ssh? start confluent in ambari? if you could can you please provide me some source. So i know what i can and cant to do in ambari. Thank you so much i really appreciate that. – Mcalpin

i've locate kafka-connect and found there are some installed plugin, but i cant fine kafka-connect-hdfs3 which is what i need. Is there any way you can sugggest me? Thank you – Mcalpin

@Mcalpin There is no such thing as "confluent kafka". Ambari starts Apache Kafka. You do not need Amari to start other components, but Ambari server is managed through Python and JSON templates. You can add different services to it. I suggest you use Puppet or Ansible instead. Please create a new post rather than comment on un-related questions, though – Bilander

Answer 5

1. run $ confluent log zookeeper you will be able to see the log for any errors

2. there is high chance zookeeper is already running and using the port 2181,

use $ sudo lsof -i :1-2181 to see which process is using that port and try to kill and try again or

run $ sudo netstat -plten | grep java to see processes and ports they are on.

run kill -9 <pid> to kill the process.

Dariusdarjeeling