Saturday, October 19, 2013

0

Hadoop-2.2.0 YARN Installation in Ubuntu Single node cluster

  • Saturday, October 19, 2013
  • Install Hadoop-2-2-0   YARN in ubuntu 



    Requirements;

    •        Ubuntu OS
    •        JDK



    1.  hadoop -2.2.0 (YARN)  Download Hadoop-2.2.0 from here
    2.Extract Hadoop-2.2.0.tar.gz by following command
              $tar -xvf hadoop-2.2.0.tar.gz 



    create home  Directory for hadoop  I created hadoop/yarn under home  
    $mkdir -p /home/hadoop/yarn
    and move hadoop-2.2.0 folder  in to /home/hadoop/yarn
    For example if hadoop-2.2.0  is in Download folder
                   $sudo mv /home/Download  /home/hadoop/yarn
                   $sudo chmod -R 755 to /home/hadoop/yarn/hadoop-2.2.0

    set these following properties in ./bashrc
    export HADOOP_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_MAPRED_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_COMMON_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_HDFS_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export YARN_HOME=home/hadoop/yarn/hadoop-2.2.0
    
    export HADOOP_CONF_DIR=home/hadoop/yarn/hadoop-2.2.0/etc/hadoop
    Next create two directories for namenode and datanode
    by following command


         $mkdir -p yarn/yarn_data/hdfs/namenode
         $mkdir -p yarn/yarn_data/hdfs/datanode

    enter  to hadoop-2.2.0 by 
           $cd /home/hadoop/yarn/hadoop-2.2.0
    Finally set hadoop configuration in following files
    Use gedit its very simple than other editors


     $sudo gedit etc/hadoop/yarn-site.xml 
    .<property>
    
    
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>
    <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
     $sudo gedit etc/hadoop/core-site.xml 
    <property>
    
       <name>fs.default.name</name>
    
       <value>hdfs://localhost:9000</value>
    
    </property>
    
     $sudo gedit etc/hadoop/hdfs-site.xml 
     <property>
    
       <name>dfs.replication</name>
    
       <value>1</value>
    
     </property>
    
     <property>
    
       <name>dfs.namenode.name.dir</name>
    
    <value>file:/home/jaikumar123/hadoop/yarn/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode      </value>
    
     </property>
    
     <property>
    
       <name>dfs.datanode.data.dir</name>
    
      <value>file:/home/jaikumar123/hadoop/yarn/hadoop-2.2.0/yarn/yarn_data/hdfs/datanode </value>
    
     </property>
    
       $sudo gedit etc/hadoop/mapred-site.xml
    <?xml version="1.0"?>
    
    <configuration>
    
       <property>
    
          <name>mapreduce.framework.name</name>
    
          <value>yarn</value>
    
       </property>
    
    </configuration>
    
     Format the namenode

        $sudo bin/hadoop namenode -format

    start hadoop daemon  services
        $sbin/hadoop-daemon.sh start namenode
        $sbin/hadoop-daemon.sh start datanode

    yarn daemon service
        $sbin/yarn-daemon.sh start resourcemanager
        $sbin/yarn-daemon.sh start nodemanager
        $sbin/mr-jobhistory-daemon.sh start historyserver

    if you have problem with  user privilege  switch in to root and try again
    Check Daemons are they started ...

         $jps
        6450 JobHistoryServer
        6485 Jps
        6242 DataNode
        6377 NodeManager
        6155 NameNode
        6310 ResourceManager
    open resourcemanager page following URL in your browser

    http://localhost:50070
    Read more...

    Friday, July 12, 2013

    0

    How to Install PyBrain on Ubuntu 12.04

  • Friday, July 12, 2013
  • Install PyBrain on Ubuntu 12.04

    PyBrain is a modular Machine Learning Library for Python. 
    open terminal:
     put following commands one by one

    sudo apt-get install python-dev python-scipy python-pip
    sudo pip install pybrain
     
    #########################################


    Read more...

    Monday, September 17, 2012

    0

    Twenty20 World Cup-2012 - schedule

  • Monday, September 17, 2012
  • Group A - England (A1), India (A2) and Afghanistan
    Group B - Australia (B1), West Indies (B2) and Ireland
    Group C - South Africa (C1), Sri Lanka (C2) and Zimbabwe
    Group D - New Zealand (D1), Pakistan (D2) and Bangladesh


    Twenty20 World Cup-2012


    Sep-2012
    Tue 18 Day/Night
    14:30 GMT, 19:30 local
    1st Match, Group C - Sri Lanka v Zimbabwe
    Mahinda Rajapaksa International Cricket Stadium, Hambantota

    Wed 19 Day/Night
    10:00 GMT, 15:30 local
    2nd Match, Group B - Australia v Ireland
    R.Premadasa Stadium, Colombo

    Wed 19 Day/Night
    14:00 GMT, 19:30 local
    3rd Match, Group A - India v Afghanistan
    R.Premadasa Stadium, Colombo

    Thu 20 Day/Night
    14:00 GMT, 19:30 local
    4th Match, Group C - South Africa v Zimbabwe
    Mahinda Rajapaksa International Cricket Stadium, Hambantota

    Fri 21 Day/Night
    10:00 GMT, 15:30 local
    5th Match, Group D - Bangladesh v New Zealand
    Pallekele International Cricket Stadium, Kandy

    Fri 21 Day/Night
    14:00 GMT, 19:30 local
    6th Match, Group A - England v Afghanistan
    R.Premadasa Stadium, Colombo

    Sat 22 Day/Night
    10:00 GMT, 15:30 local
    7th Match, Group C - Sri Lanka v South Africa
    Mahinda Rajapaksa International Cricket Stadium, Hambantota

    Sat 22 Day/Night
    14:00 GMT, 19:30 local
    8th Match, Group B - Australia v West Indies
    R.Premadasa Stadium, Colombo

    Sun 23 Day/Night
    10:00 GMT, 15:30 local
    9th Match, Group D - New Zealand v Pakistan
    Pallekele International Cricket Stadium, Kandy

    Sun 23 Day/Night
    14:00 GMT, 19:30 local
    10th Match, Group A - India v England
    R.Premadasa Stadium, Colombo

    Mon 24 Day/Night
    14:00 GMT, 19:30 local
    11th Match, Group B - West Indies v Ireland
    R.Premadasa Stadium, Colombo

    Tue 25 Day/Night
    14:00 GMT, 19:30 local
    12th Match, Group D - Pakistan v Bangladesh
    Pallekele International Cricket Stadium, Kandy

    Thu 27 Day/Night
    10:00 GMT, 15:30 local
    13th Match, Super Eights, Group 1 - C1 v D2
    Pallekele International Cricket Stadium, Kandy

    Thu 27 Day/Night
    14:00 GMT, 19:30 local
    14th Match, Super Eights, Group 1 - A1 v B2
    Pallekele International Cricket Stadium, Kandy

    Fri 28 Day/Night
    10:00 GMT, 15:30 local
    15th Match, Super Eights, Group 2 - D1 v C2
    R.Premadasa Stadium, Colombo

    Fri 28 Day/Night
    14:00 GMT, 19:30 local
    16th Match, Super Eights, Group 2 - B1 v A2
    R.Premadasa Stadium, Colombo

    Sat 29 Day/Night
    10:00 GMT, 15:30 local
    17th Match, Super Eights, Group 1 - C1 v B2
    Pallekele International Cricket Stadium, Kandy

    Sat 29 Day/Night
    14:00 GMT, 19:30 local
    18th Match, Super Eights, Group 1 - B1 v C2
    Pallekele International Cricket Stadium, Kandy

    Sun 30 Day/Night
    10:00 GMT, 15:30 local
    19th Match, Super Eights, Group 2 - D1 v A2
    R.Premadasa Stadium, Colombo

    Sun 30 Day/Night
    14:00 GMT, 19:30 local
    20th Match, Super Eights, Group 2 - B2 v D2
    R.Premadasa Stadium, Colombo
    Oct-2012
    Mon 01 Day/Night
    10:00 GMT, 15:30 local
    21st Match, Super Eights, Group 1 - A1 v C1
    Pallekele International Cricket Stadium, Kandy

    Mon 01 Day/Night
    14:00 GMT, 19:30 local
    22nd Match, Super Eights, Group 1 - B1 v D1
    Pallekele International Cricket Stadium, Kandy

    Tue 02 Day/Night
    10:00 GMT, 15:30 local
    23rd Match, Super Eights, Group 2 - A2 v C2
    R.Premadasa Stadium, Colombo

    Tue 02 Day/Night
    14:00 GMT, 19:30 local
    24th Match, Super Eights, Group 2 - A1 v D2
    R.Premadasa Stadium, Colombo

    Thu 04 Day/Night
    14:00 GMT, 19:30 local
    1st Semi-Final - TBC v TBC
    R.Premadasa Stadium, Colombo

    Fri 05 Day/Night
    14:00 GMT, 19:30 local
    2nd Semi-Final - TBC v TBC
    R.Premadasa Stadium, Colombo

    Sun 07 Day/Night
    14:00 GMT, 19:30 local
    Final - TBC v TBC
    R.Premadasa Stadium, Colombo
    Read more...

    Subscribe