How to configure multi-node Apache Storm cluster

Asked 29/5, 2015 at 9:13 Answered 6/11, 2015 at 7:8

I'm following http://jayatiatblogs.blogspot.com/2011/11/storm-installation.html & http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup to set up Apache Storm cluster in Ubuntu 14.04 LTS at AWS EC2.

My master node is 10.0.0.185. My slave nodes are 10.0.0.79, 10.0.0.124 & 10.0.0.84 with myid of 1, 2 and 3 in their zookeeper-data respectively. I set up an ensemble of Apache Zookeeper consists of all the 3 slave nodes.

Below are my zoo.cfg for my slave nodes:

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/home/ubuntu/zookeeper-data
clientPort=2181

server.1=10.0.0.79:2888:3888
server.2=10.0.0.124:2888:3888
server.3=10.0.0.84:2888:3888

autopurge.snapRetainCount=3
autopurge.purgeInterval=1

Below are my storm.yaml for my slave nodes:

########### These MUST be filled in for a storm configuration
 storm.zookeeper.server:
     - "10.0.0.79"
     - "10.0.0.124"
     - "10.0.0.84"
#     - "localhost"
 storm.zookeeper.port: 2181

# nimbus.host: "localhost"
 nimbus.host: "10.0.0.185"

 storm.local.dir: "/home/ubuntu/storm/data"
 java.library.path: "/usr/lib/jvm/java-7-oracle"

 supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703
     - 6704
#
# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "backtype.storm.metric.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"

Below are the storm.yaml for my master node:

########### These MUST be filled in for a storm configuration
 storm.zookeeper.servers:
     - "10.0.0.79"
     - "10.0.0.124"
     - "10.0.0.84"
#     - "localhost"
#
 storm.zookeeper.port: 2181

 nimbus.host: "10.0.0.185"
# nimbus.thrift.port: 6627
# nimbus.task.launch.secs: 240

# supervisor.worker.start.timeout.secs: 240
# supervisor.worker.timeout.secs: 240

 ui.port: 8772

#  nimbus.childopts: "‐Xmx1024m ‐Djava.net.preferIPv4Stack=true"

#  ui.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"
#  supervisor.childopts: "‐Djava.net.preferIPv4Stack=true"
#  worker.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"

 storm.local.dir: "/home/ubuntu/storm/data"

 java.library.path: "/usr/lib/jvm/java-7-oracle"

# supervisor.slots.ports:
#     - 6700
#     - 6701
#     - 6702
#     - 6703
#     - 6704

# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"

# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "backtype.storm.metric.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"

I start my zookeeper in all my slave nodes, then start my storm nimbus in my master node, then start storm supervisor in all my slave nodes. However, when I view in my Storm UI, there is only 1 supervisor with total 5 slots in the cluster summary & only 1 supervisor information in the supervisor summary, why so?

How many slave nodes is actually working if I submit a topology in this case?

Why it is not 3 supervisors with total 15 slots?

What should I do in order to have 3 supervisors?

When I check in the supervisor.log in the slave nodes, the causes is as below:

2015-05-29T09:21:24.185+0000 b.s.d.supervisor [INFO] 5019754f-cae1-4000-beb4-fa0
16bd1a43d still hasn't started

Herd answered 29/5, 2015 at 9:13 Comment(2)

See failed supervisor logs about connection problems – Hyperbaric 29/5, 2015 at 10:42

Added the supervisor logs. It seem like only 1 supervisor will be started at the same time, other will not be started, although I issue the "storm supervisor" command as well. – Herd 29/5, 2015 at 22:36

What you are doing perfect and its works too.

The only thing you should change is your storm.dir. It is same in the slave and the master nodes just change the path in the storm.dir path in nimbus & supervisor nodes (don't use same local path). When you use same local path the nimbus and supervisor share same id. They come into play but you don’t see 8 slots they just show you 4 slots as workers.

Change the (storm.local.dir:/home/ubuntu/storm/data) and don`t use same path in supervisor and nimbus.

Pore answered 6/11, 2015 at 7:8 Comment(6)

Thank you for your help, however, the problem still persists although I change the storm.local.dir:/home/ubuntu/storm/data in the storm.yaml of nimbus node to /home/ubuntu/storm/data2, meanwhile remain the storm.local.dir:/home/ubuntu/storm/data in the storm.yaml of slave node. – Herd 29/11, 2015 at 12:0

When I issue storm rebalance, instead of allowing new slave run together with previous node, the new slave node replace previous node to run – Herd 2/12, 2015 at 3:15

okay !! can i know how many supervisor nodes you are running!! does the path dir in the supervisor changed or using same path in all the supervisors nodes?if your running same local path in supervisors change it.ex path in supervisor 1 node storm.local.dir:/home/ubuntu/storm/data and in supervisor 2 node storm.local.dir:/home/ubuntu/storm/data2.they should differ ! if it doesn't work check the logs of previous node after replacement. – Pore 4/12, 2015 at 4:5

I'm running 3 slave nodes now. Thank you very much, it works after I put different directory for storm.local.dir in the storm.yaml file of each slave node. – Herd 5/12, 2015 at 6:9

Great ! you can look at storm trouble-shoot page if you faced any new errors.Good day – Pore 7/12, 2015 at 4:58

sorry i'm new to storm i've same problem but i have one worker for one supervisor , what if i will run it in single machine first ? can you see my problem here #34574966 – Glimmer 3/1, 2016 at 7:57

Are you referring to Nimbus as the master node?

Generally, Zookeeper cluster should be started first and then nimbus and then the supervisors. Zookeeper and Nimbus should be always available for the Storm cluster to function correctly.

You should check the supervisor logs to check for the failures. The Nimbus host and the Zookeeper machines should be accessible from to Supervisor machines.

Glottic answered 29/5, 2015 at 10:59 Comment(1)

Yes, I do start the start the Zookeeper cluster first, then start storm Nimbus, then start the storm Supervisor. I just type in wrongly in the question I asked. Edited my question. – Herd 29/5, 2015 at 21:56

Recommended topics

Hot tags