Difference between revisions of "Hadoop: Setup a single host test system"

From Define Wiki
Jump to navigation Jump to search
Line 30: Line 30:
  
 
=== Setup Config Files ===
 
=== Setup Config Files ===
 +
All files in question here are found in <tt>/opt/hadoop-1.0.3</tt>
 +
 +
<tt>conf/core-site.xml:</tt>
 +
<xml>
 +
<configuration>
 +
    <property>
 +
        <name>fs.default.name</name>
 +
        <value>hdfs://localhost:9000</value>
 +
    </property>
 +
</configuration>
 +
</xml>
 +
 +
<tt>conf/hdfs-site.xml:</tt>
 +
<xml>
 +
<configuration>
 +
    <property>
 +
        <name>dfs.replication</name>
 +
        <value>1</value>
 +
    </property>
 +
</configuration>
 +
</xml>
 +
 +
<tt>conf/mapred-site.xml:</tt>
 +
<xml>
 +
<configuration>
 +
    <property>
 +
        <name>mapred.job.tracker</name>
 +
        <value>localhost:9001</value>
 +
    </property>
 +
</configuration>
 +
</xml>
 +
 +
<tt>conf/hadoop-env/sh:</tt>
 +
<syntaxhighlight>
 +
  export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf/jre
 +
</syntaxhighlight>

Revision as of 16:35, 17 July 2012

Tests performed on a single calxeda SOC with ubuntu 12.10

Prerequisites

Install Java/JRE

  apt-get update
  apt-get install default-jre

Setup Passwordless Access

Setup passwordless ssh for user/root (I used root in this example, separate hadoop user should really be setup!) 

  ssh-keygen -t rsa
  # dont enter a passphrase, just hit enter twice for a blank passphrase
  cd .ssh
  cat id_rsa.pub >> authorized_keys
  chmod 600 authorized_keys

Install Hadoop

Get latest stable release

The latest release is available from: http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/ 

  wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/hadoop-1.0.3.tar.gz
  cd /opt
  tar zxvf /path/to/download/hadoop-1.0.3.tar.gz

Setup Config Files

All files in question here are found in /opt/hadoop-1.0.3

conf/core-site.xml: <xml> <configuration> 

    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>

</configuration> </xml>

conf/hdfs-site.xml: <xml> <configuration> 

    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

</configuration> </xml>

conf/mapred-site.xml: <xml> <configuration> 

    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>

</configuration> </xml>

conf/hadoop-env/sh: 

  export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf/jre
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Hadoop:_Setup_a_single_host_test_system&oldid=812"