K mouza: /* Configure for the all workloads */

2015-03-27T10:21:07Z

Configure for the all workloads

K mouza: Created page with "HiBench is a Hadoop benchmark suite. It is good for testing the correctness and health of a Hadoop installation. *Download HiBench from github: git clone https://githu..."

2015-03-27T10:17:04Z

Created page with "HiBench is a Hadoop benchmark suite. It is good for testing the correctness and health of a Hadoop installation. *Download HiBench from github: <syntaxhighlight> git clone https://githu..."

New page

HiBench is a Hadoop benchmark suite. It is good for testing the correctness and health of a Hadoop installation.

*Download HiBench from github:
<syntaxhighlight>
git clone https://github.com/intel-hadoop/HiBench.git
</syntaxhighlight>

== Getting Started ==

=== Prerequisites ===

*Setup HiBench
*:Make sure these things are installed: maven. Then, locate into HiBench/common/hibench and run mvn process-sources to get dependencies.

*Setup Hadood
*:Before you run any workload in the package, please verify the Hadoop framework is running correctly. All the workloads have been tested with Cloudera Distribution of Hadoop 5(cdh5.1.0) and Hadoop version 1.0.4 and 2.2.0

*Setup Hive (for hivebench)
*:Please make sure you have properly set up Hive in your cluster if you want to test hivebench. Or the benchmark willuse the default release fetched by maven.

=== Configure for the all workloads ===

You need to set some global environment variables in the <tt>bin/hibench-config.sh</tt> file located in the root dir.

<syntaxhighlight>
export JAVA_HOME=/usr/lib/jvm/java
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.5.1/
export HADOOP_EXECUTABLE=hadoop
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/
export HADOOP_EXAMPLES_JAR=/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar
export MAPRED_EXECUTABLE=mapred
#Set the varaible below only in YARN mode
export HADOOP_JOBCLIENT_TESTS_JAR=/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.1-mapr-1501-tests.jar
</syntaxhighlight>

These variables are for use with MapR. (see [[MapR: Landing Page]]

← Older revision		Revision as of 10:21, 27 March 2015
Line 34:		Line 34:
	</syntaxhighlight>		</syntaxhighlight>

−	These variables are for use with MapR. (see [[MapR: Landing Page]]	+	These variables are for use with MapR. (see [[MapR: Landing Page]])
		+
		+	=== Configure each workload ===
		+
		+	You can modify the <tt>conf/configure.sh</tt> file under each workload folder if it exists. All the data size and options related to the workload are defined in this file.
		+
		+	=== Synchronize the time on all nodes ===
		+
		+	This is required for dfsioe, and optional for others)
		+
		+	== Running ==
		+
		+	=== Run several workloads together ===
		+
		+	The <tt>conf/benchmarks.lst</tt> file under the package folder defines the workloads to run when you execute the <tt>bin/run-all.sh</tt> script under the package folder. Each line in the list file specifies one workload. You can use # at the beginning of each line to skip the corresponding bench if necessary.
		+
		+	=== Run workload in throughput test mode ===
		+
		+	The <tt>conf/benchmarks-concurrent.lst</tt> file under the package folder defines the workloads to run when you execute the <tt>bin/run-concurrent.sh</tt> script under the package folder. The number at the end of each line indicates the number of each workload you want to submit simultaneously. Before running, execute the script <tt>bin/prepare-concurrent.sh</tt>.
		+
		+	For the workload hivebench, please run the metastore as a service first to support concurrency: <tt>hive --service metastore</tt>.
		+
		+	=== Run each workload separately ===
		+
		+	You can also run each workload separately. In general, there are 3 different files under one workload folder.
		+
		+	<syntaxhighlight>
		+	conf/configure.sh Configuration file contains all parameters such as data size and test options.
		+	bin/prepare*.sh Generate or copy the job input data into HDFS.
		+	bin/run*.sh Execute the workload
		+	</syntaxhighlight>
		+
		+	Follow the steps below to run a workload
		+
		+	#Configure the benchmark:
		+	#:set your own configurations by modifying configure.sh if necessary
		+	#Prepare data:
		+	#:bin/prepare.sh (bin/prepare-read.sh for dfsioe) to prepare input data in HDFS for running the benchmark
		+	#Run the benchmark:
		+	#:bin/run*.sh to run the corresponding benchmark

HiBench Testing - Revision history

K mouza: /* Configure for the all workloads */

K mouza: Created page with "HiBench is a Hadoop benchmark suite. It is good for testing the correctness and health of a Hadoop installation. *Download HiBench from github: git clone https://githu..."