Hadoop: Setup a single host test system

Tests performed on a single calxeda SOC with ubuntu 12.10

Prerequisites

Install Java/JRE

  apt-get update
  apt-get install default-jre

Setup Passwordless Access

Setup passwordless ssh for user/root (I used root in this example, separate hadoop user should really be setup!)

  ssh-keygen -t rsa
  # dont enter a passphrase, just hit enter twice for a blank passphrase
  cd .ssh
  cat id_rsa.pub >> authorized_keys
  chmod 600 authorized_keys

Install Hadoop

Get latest stable release

The latest release is available from: http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/

  wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/hadoop-1.0.3.tar.gz
  cd /opt
  tar zxvf /path/to/download/hadoop-1.0.3.tar.gz

Setup Config Files

All files in question here are found in /opt/hadoop-1.0.3

conf/core-site.xml: <xml> <configuration>

    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>

</configuration> </xml>

conf/hdfs-site.xml: <xml> <configuration>

    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

</configuration> </xml>

conf/mapred-site.xml: <xml> <configuration>

    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>

</configuration> </xml>

conf/hadoop-env/sh:

  export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf/jre

Format the namenode

  ./bin/hadoop namenode -format