Failed to get system directory - hadoop

Question

Using hadoop multinode setup (1 mater , 1 salve)

After starting up start-mapred.sh on master , i found below error in TT logs (Slave an)

org.apache.hadoop.mapred.TaskTracker: Failed to get system directory

can some one help me to know what can be done to avoid this error

I am using Hadoop 1.2.0 jetty-6.1.26 java version "1.6.0_23"

mapred-site.xml file

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>master:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
<property>
  <name>mapred.map.tasks</name>
    <value>1</value>
      <description>
          define mapred.map tasks to be number of slave hosts
</description>
</property>

<property>
  <name>mapred.reduce.tasks</name>
    <value>1</value>
      <description>
          define mapred.reduce tasks to be number of slave hosts
</description>
</property>

</configuration>

core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hduser/workspace</value>
</property>
</configuration>

Overplus

Please post your hdfs-site.xml and core-site.xml too, if you can. – Rocco

Also the DN and TT logs. – Sapor

2013-07-29 13:38:55,728 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. got this error after specifying a path to hadoop.tmp.dir – Overplus

Have you formatted it? – Sapor

yes i formatted it @ Tariq – Overplus

I am facing disk-error , so i added hadoop.tmp.dir path ,and this started throwing this error. – Overplus

Answer 1

It seems that you just added hadoop.tmp.dir and started the job. You need to restart the Hadoop daemons after adding any property to the configuration files. You have specified in your comment that you added this property at a later stage. This means that all the data and metadata along with other temporary files is still in the /tmp directory. Copy all those things from there into your /home/hduser/workspace directory, restart Hadoop and re run the job.

Do let me know the result. Thank you.

Sapor

Re-format filesystem in /tmp/hadoop-root/dfs/name ? (Y or N) y Format aborted in /tmp/hadoop-root/dfs/name ; issue on format , can u advise @Sapor – Overplus

this woked fine after removing temp directory – Overplus

but issue with connceting to name node , org.apache.hadoop.mapred.JobTracker: Problem connecting to HDFS Namenode... re-trying @ Tariq – Overplus

Answer 2

If, it is your windows PC and you are using cygwin to run Hadoop. Then task tracker will not work.

Tantalize

@Tariq-- I didn't get your question. Please explain what you want to say :) – Tantalize

I would like to know that how did you arrive at the conclusion that TT will not run if you are using Hadoop through cygwin. Thank you. – Sapor

@Tariq--This is my personal experience, When I was installing Hadoop on my Windows PC. All Job got started successfully. but, while running the MR program. I got error while checking tasktracker log Tasktracker currently supporting linux machine only. and My job got fail :) Please check these link as well hadoop.apache.org/docs/current/hadoop-mapreduce-client/… and hadoop.apache.org/docs/stable/cluster_setup.html – Tantalize

Thank you for the pointers. You can do it, IMHO. Process is not smooth though. – Sapor