HADOOP / YARN - Are the ResourceManager and the hdfs NameNode always installed on the same host?

Question

Are the “resource manager” and the “hdfs namenode” always installed on the same host?

1) When I want to send an http request (YARN REST API) to get new application id I am using this web uri:

http://<rm http address:port>/ws/v1/cluster/apps/new-application

This port is the Resource-Manager Web UI http port, it’s default value is 8088 as shown in img1: source for img1

2) When I want to send an http request (WebHDFS REST API) for hdfs commands, for example to get file status I am using this web uri:

http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILESTATUS

This <PORT> is the NameNode WebUI http Port, it’s default value is 50070 as shown in img2: source for img2

Are the hosts of those components (ResourceManager and NaneNode) are always installed on the same host?

Any help would be appreciated, Thanks!

Hwahwan

You can install the YARN Resource Manager and the HDFS Namenode to any host you like. They don't need to share the host. – Reinaldoreinaldos

Answer 1

It is not necessary to run resourceManager and nameNode on a single machines, we have such configuration when working in a single node/small cluster---with few nodes---configuration. When there is large cluster, master nodes usually run resourceManager, nameNode and secondaryNamenode on different machines.

Have a look at these links:

Master Nodes in Hadoop Clusters

HortonWorks: Typical Hadoop Cluster

Hesperus

The note says that Datanodes, NodeManagers, regionservers are co-deployed. Do you mean that each worker node has a datanode, a nodemanager and a regionserver ?! – Everyway