name node Vs secondary name node

Question

Hadoop is Consistent and partition tolerant, i.e. It falls under the CP category of the CAP theoram.

Hadoop is not available because all the nodes are dependent on the name node. If the name node falls the cluster goes down.

But considering the fact that the HDFS cluster has a secondary name node why cant we call hadoop as available. If the name node is down the secondary name node can be used for the writes.

What is the major difference between name node and secondary name node that makes hadoop unavailable.

Thanks in advance.

Portly

Answer 1

The namenode stores the HDFS filesystem information in a file named fsimage. Updates to the file system (add/remove blocks) are not updating the fsimage file, but instead are logged into a file, so the I/O is fast append only streaming as opposed to random file writes. When restaring, the namenode reads the fsimage and then applies all the changes from the log file to bring the filesystem state up to date in memory. This process takes time.

The secondarynamenode job is not to be a secondary to the name node, but only to periodically read the filesystem changes log and apply them into the fsimage file, thus bringing it up to date. This allows the namenode to start up faster next time.

Unfortunatley the secondarynamenode service is not a standby secondary namenode, despite its name. Specifically, it does not offer HA for the namenode. This is well illustrated here.

See Understanding NameNode Startup Operations in HDFS.

Note that more recent distributions (current Hadoop 2.6) introduces namenode High Availability using NFS (shared storage) and/or namenode High Availability using Quorum Journal Manager.

Reversal

Hi Thanks for the awesome video, I have one last doubt-> The secondary name node creates periodical check point by merging the edit log data into fsimage. The other day I was reading about some ckechpoint nodes in HDFS that runs in a different machines, After reading that I've got an understanding that the functionality of the secondary name node and checkpoint nodes are quite similar, then what makes checkpount nodes different from a secondary name node. Are they the same? – Portly

The checkpoint node is a new service (post Hadoop 0.21) that replaces the secondary name node. Look at the comments on HADOOP-4539 – Reversal

Answer 2

Things have been changed over the years especially with Hadoop 2.x. Now Namenode is highly available with fail over feature.

Secondary Namenode is optional now & Standby Namenode has been to used for failover process.

Standby NameNode will stay up-to-date with all the file system changes the Active NameNode makes .

HDFS High availability is possible with two options : NFS and Quorum Journal Manager but Quorum Journal Manager is preferred option.

How does Hadoop Namenode failover process works?

Regarding your queries on CAP theory for Hadoop:

1. It can be strong consistent
2. HDFS is almost highly Available unless you met with some bad luck ( If all three replicas of a block are down, you won't get data)
3. Supports data Partition

Stretcherbearer

Answer 3

Name Node is a primary node in which all the metadata into is stored into fsimage and editlog files periodically. But, when name node down secondary node will be online but this node only have the read access to the fsimage and editlog files and dont have the write access to them . All the secondary node operations will be stored to temp folder . when name node back to online this temp folder will be copied to name node and the namenode will update the fsimage and editlog files.

Vegetal

Answer 4

Even in HDFS High Availability, where there are two NameNodes instead of one NameNode and one SecondaryNameNode, there is not availability in the strict CAP sense. It only applies to the NameNode component, and even there if a network partition separates the client from both of the NameNodes then the cluster is effectively unavailable.

Phenolphthalein

Answer 5

If I explain it in simple way, suppose Name Node as a men(working/live) and secondary Name Node as a ATM machine(storage/data storage)
So all the functions carried out by NN or men only but if it goes down/fails then SNN will be useless it doesn’t work but later it can be used to recover your data or logs

Mandatory

Answer 6

When NameNode starts, it loads FSImage and replay Edit Logs to create latest updated namespace. This process may take long time if size of Edit Log file is big and hence increase startup time. The job of Secondary Name Node is to periodically check edit log and replay to create updated FSImage and store in persistent storage. When Name Node starts it doesn't need to replay edit log to create updated FSImage, it uses FSImage created by secondary name node.

Outgoings

Answer 7

The namenode is a master node that contains metadata in terms of fsimage and also contains the edit log. The edit log contains recently added/removed block information in the namespace of the namenode. The fsimage file contains metadata of the entire hadoop system in a permanent storage. Every time we need to make changes permanently in fsimage, we need to restart namenode so that edit log information can be written at namenode, but it takes a lot of time to do that.

A secondary namenode is used to bring fsimage up to date. The secondary name node will access the edit log and make changes in fsimage permanently so that next time namenode can start up faster.

Basically the secondary namenode is a helper for namenode and performs housekeeping functionality for the namenode.

Higa