Saturday, October 19, 2013

0

Hadoop-2.2.0 YARN Installation in Ubuntu Single node cluster

  • Saturday, October 19, 2013
  • Share
  • Install Hadoop-2-2-0   YARN in ubuntu 



    Requirements;

    •        Ubuntu OS
    •        JDK



    1.  hadoop -2.2.0 (YARN)  Download Hadoop-2.2.0 from here
    2.Extract Hadoop-2.2.0.tar.gz by following command
              $tar -xvf hadoop-2.2.0.tar.gz 



    create home  Directory for hadoop  I created hadoop/yarn under home  
    $mkdir -p /home/hadoop/yarn
    and move hadoop-2.2.0 folder  in to /home/hadoop/yarn
    For example if hadoop-2.2.0  is in Download folder
                   $sudo mv /home/Download  /home/hadoop/yarn
                   $sudo chmod -R 755 to /home/hadoop/yarn/hadoop-2.2.0

    set these following properties in ./bashrc
    export HADOOP_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_MAPRED_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_COMMON_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_HDFS_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export YARN_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_CONF_DIR=home/hadoop/yarn/hadoop-2.2.0/etc/hadoop
    Next create two directories for namenode and datanode
    by following command


         $mkdir -p yarn/yarn_data/hdfs/namenode
         $mkdir -p yarn/yarn_data/hdfs/datanode

    enter  to hadoop-2.2.0 by 
           $cd /home/hadoop/yarn/hadoop-2.2.0
    Finally set hadoop configuration in following files
    Use gedit its very simple than other editors


     $sudo gedit etc/hadoop/yarn-site.xml 
    .<property>
    
    
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>
    <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
     $sudo gedit etc/hadoop/core-site.xml 
    <property>
    
       <name>fs.default.name</name>
    
       <value>hdfs://localhost:9000</value>
    
    </property>
    
     $sudo gedit etc/hadoop/hdfs-site.xml 
     <property>
    
       <name>dfs.replication</name>
    
       <value>1</value>
    
     </property>
    
     <property>
    
       <name>dfs.namenode.name.dir</name>
    
    <value>file:/home/jaikumar123/hadoop/yarn/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode      </value>
    
     </property>
    
     <property>
    
       <name>dfs.datanode.data.dir</name>
    
      <value>file:/home/jaikumar123/hadoop/yarn/hadoop-2.2.0/yarn/yarn_data/hdfs/datanode </value>
    
     </property>
    
       $sudo gedit etc/hadoop/mapred-site.xml
    <?xml version="1.0"?>
    
    <configuration>
    
       <property>
    
          <name>mapreduce.framework.name</name>
    
          <value>yarn</value>
    
       </property>
    
    </configuration>
    
     Format the namenode

        $sudo bin/hadoop namenode -format

    start hadoop daemon  services
        $sbin/hadoop-daemon.sh start namenode
        $sbin/hadoop-daemon.sh start datanode

    yarn daemon service
        $sbin/yarn-daemon.sh start resourcemanager
        $sbin/yarn-daemon.sh start nodemanager
        $sbin/mr-jobhistory-daemon.sh start historyserver

    if you have problem with  user privilege  switch in to root and try again
    Check Daemons are they started ...

         $jps
        6450 JobHistoryServer
        6485 Jps
        6242 DataNode
        6377 NodeManager
        6155 NameNode
        6310 ResourceManager
    open resourcemanager page following URL in your browser

    http://localhost:50070

    0 Responses to “Hadoop-2.2.0 YARN Installation in Ubuntu Single node cluster”

    Subscribe