Bright:Deploying Hadoop - Revision history

David: /* Log from Install */

2015-06-06T21:00:27Z

Log from Install

David at 20:45, 6 June 2015

2015-06-06T20:45:49Z

K mouza: /* Installing from terminal */

2015-02-11T10:58:04Z

Installing from terminal

K mouza: /* Installing from terminal */

2015-02-11T10:57:43Z

Installing from terminal

K mouza: /* Installing from CMGUI */

2015-02-11T10:53:09Z

Installing from CMGUI

K mouza: /* Installing from CMGUI */

2015-02-11T10:51:26Z

Installing from CMGUI

K mouza: /* Installing from CMGUI */

2015-02-11T10:42:46Z

Installing from CMGUI

K mouza: /* Installing from CMGUI */

2015-02-11T10:39:52Z

Installing from CMGUI

K mouza: /* Installing from CMGUI */

2015-02-11T10:26:35Z

Installing from CMGUI

K mouza at 10:26, 11 February 2015

2015-02-11T10:26:00Z

← Older revision		Revision as of 21:00, 6 June 2015
Line 197:		Line 197:


		+	[root@head-Boston ~]# cm-hadoop-setup -u ubs-test
		+	Requested removal of Hadoop instance 'ubs-test'.
		+	Stopping all Hadoop services for instance 'ubs-test'...
		+	Stopping all Hadoop services for instance 'ubs-test'... done.
		+	Instance 'ubs-test' found in cmdaemon. Removing...
		+	Unable to remove ubs-test, the following objects depend on it:
		+	Remove failed but no errors
		+	Instance 'ubs-test' found in cmdaemon. Removing... FAILED.
		+	Waiting a few seconds before removing files from local fs...
		+	Waiting a few seconds before removing files from local fs... done.
		+	Removing Hadoop instance 'ubs-test'

		+	Removing:
		+	/etc/hadoop/ubs-test
		+	/var/log/hadoop/ubs-test
		+	/var/run/hadoop/ubs-test
		+	/tmp/hadoop/ubs-test/
		+	/etc/hadoop/ubs-test/zookeeper
		+	/var/lib/zookeeper/ubs-test
		+	/var/log/zookeeper/ubs-test
		+	/var/run/zookeeper/ubs-test
		+	/etc/hadoop/ubs-test/hbase
		+	/var/log/hbase/ubs-test
		+	/var/run/hbase/ubs-test
		+	/var/lib/hadoop/ubs-test/
		+	/etc/init.d/hadoop-ubs-test-*
		+	Updating images...
		+	Updating images... done.
		+	Instance 'ubs-test' removed from images.
		+	Removing log files.
		+	Module file(s) deleted.
		+	Removal successfully completed.
	</syntaxhighlight>		</syntaxhighlight>

← Older revision		Revision as of 20:45, 6 June 2015
Line 48:		Line 48:

	Then follow the on screen instructions, which are the same as the CMGUI ones. A configuration file will be then created.		Then follow the on screen instructions, which are the same as the CMGUI ones. A configuration file will be then created.
		+
		+	== Log from Install ==
		+	<syntaxhighlight>
		+	# went through the cm-hadoop-setup ncurses setup, created teh file below
		+	[root@head-Boston hadoop]# cp /tmp/hadoop-ubs-test.xml /home/david/hadoop-scripts/
		+	[root@head-Boston hadoop]# cm-hadoop-setup -c /home/david/hadoop-scripts/hadoop-ubs-test.xml
		+	Reading config from file '/home/david/hadoop-scripts/hadoop-ubs-test.xml'... done.
		+	Hadoop flavor 'Hortonworks', release '2.6.0.2.2.0.0-2041'
		+	Hadoop is already installed in /cm/shared/apps/hadoop/Hortonworks/2.6.0.2.2.0.0-2041
		+	Will now configure instance 'ubs-test'
		+	ZooKeeper is already installed in /cm/shared/apps/hadoop/Hortonworks/zookeeper-3.4.6.2.2.0.0-2041
		+	HBase is already installed in /cm/shared/apps/hadoop/Hortonworks/hbase-0.98.4.2.2.0.0-2041-hadoop2
		+	Creating module file... done.
		+	Configuring Hadoop instance on local filesystem and images... done.
		+	Updating images...
		+	Updating images... done.
		+	Creating Hadoop instance in cmdaemon...
		+	Creating Hadoop instance in cmdaemon... done.
		+	Formatting HDFS...
		+	Formatting HDFS... done.
		+	Waiting for NameNode service...
		+	Waiting for NameNode service... done.
		+	Waiting for DataNodes to come up...
		+	Waiting for DataNodes to come up... done.
		+	Waiting for NameNode to be ready...
		+	Waiting for NameNode to be ready... done.
		+	Setting up HDFS...
		+	Setting up HDFS... done.
		+	Setting up HBase hosts...
		+	Setting up HBase hosts... done.
		+	Setting up YARN hosts...
		+	Setting up YARN hosts... done.
		+	Waiting for Hadoop to be ready for validation test...
		+	Waiting for Hadoop to be ready for validation test... done.
		+	Validating Hadoop setup...
		+	Validating Hadoop setup... done.
		+	Checking ZooKeeper installation...
		+	Checking ZooKeeper installation... done.
		+	Checking HBase installation...
		+	Checking HBase installation... done.
		+	Installation successfully completed.
		+	Finished.
		+	[root@head-Boston ~]# cmsh
		+	[head-Boston]% user
		+	[head-Boston->user]% use user david
		+	[head-Boston->user[david]]% set hadoophdfsaccess ubs-test; commit
		+	[Ctrl+d]
		+	[david@head-Boston ~]$ ssh openstack1
		+	Last login: Sat Jun 6 15:42:30 2015 from head-boston.cm.cluster
		+	[david@openstack1 ~]$ module load hadoop/ubs-test/Hortonworks/2.6.0.2.2.0.0-2041
		+	[david@openstack1 ~]$ module load zookeeper/ubs-test/Hortonworks/3.4.6.2.2.0.0-2041
		+	[david@openstack1 ~]$ module load hbase/ubs-test/Hortonworks/0.98.4.2.2.0.0-2041-hadoop2
		+	[david@openstack1 ~]$ cd $HADOOP_PREFIX/bin
		+	[david@openstack1 bin]$ ./hadoop fs -ls /
		+	Found 6 items
		+	drwx------ - root hadoop 0 2015-06-06 16:13 /accumulo
		+	drwx------ - hbase hadoop 0 2015-06-06 16:13 /hbase
		+	drwxrwxrwt - hdfs hadoop 0 2015-06-06 16:18 /staging
		+	drwxr-xr-x - hdfs hadoop 0 2015-06-06 20:59 /system
		+	drwxrwxrwt - hdfs hadoop 0 2015-06-06 16:12 /tmp
		+	drwxr-xr-x - hdfs hadoop 0 2015-06-06 21:32 /user
		+	[david@openstack1 bin]$ hadoop jar /cm/shared/apps/hadoop/ubs-test/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 1 5
		+	Number of Maps = 1
		+	Samples per Map = 5
		+	Wrote input for Map #0
		+	Starting Job
		+	15/06/06 21:35:42 INFO client.RMProxy: Connecting to ResourceManager at openstack3.cm.cluster/173.16.0.55:8032
		+	15/06/06 21:35:42 INFO input.FileInputFormat: Total input paths to process : 1
		+	15/06/06 21:35:42 INFO mapreduce.JobSubmitter: number of splits:1
		+	15/06/06 21:35:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433603618985_0005
		+	15/06/06 21:35:42 INFO impl.YarnClientImpl: Submitted application application_1433603618985_0005
		+	15/06/06 21:35:43 INFO mapreduce.Job: The url to track the job: http://openstack3:8088/proxy/application_1433603618985_0005/
		+	15/06/06 21:35:43 INFO mapreduce.Job: Running job: job_1433603618985_0005
		+	15/06/06 21:35:49 INFO mapreduce.Job: Job job_1433603618985_0005 running in uber mode : false
		+	15/06/06 21:35:49 INFO mapreduce.Job: map 0% reduce 0%
		+	15/06/06 21:35:54 INFO mapreduce.Job: map 100% reduce 0%
		+	15/06/06 21:36:00 INFO mapreduce.Job: map 100% reduce 100%
		+	15/06/06 21:36:00 INFO mapreduce.Job: Job job_1433603618985_0005 completed successfully
		+	15/06/06 21:36:00 INFO mapreduce.Job: Counters: 49
		+	File System Counters
		+	FILE: Number of bytes read=28
		+	FILE: Number of bytes written=219263
		+	FILE: Number of read operations=0
		+	FILE: Number of large read operations=0
		+	FILE: Number of write operations=0
		+	HDFS: Number of bytes read=277
		+	HDFS: Number of bytes written=215
		+	HDFS: Number of read operations=7
		+	HDFS: Number of large read operations=0
		+	HDFS: Number of write operations=3
		+	Job Counters
		+	Launched map tasks=1
		+	Launched reduce tasks=1
		+	Data-local map tasks=1
		+	Total time spent by all maps in occupied slots (ms)=2938
		+	Total time spent by all reduces in occupied slots (ms)=3030
		+	Total time spent by all map tasks (ms)=2938
		+	Total time spent by all reduce tasks (ms)=3030
		+	Total vcore-seconds taken by all map tasks=2938
		+	Total vcore-seconds taken by all reduce tasks=3030
		+	Total megabyte-seconds taken by all map tasks=3008512
		+	Total megabyte-seconds taken by all reduce tasks=3102720
		+	Map-Reduce Framework
		+	Map input records=1
		+	Map output records=2
		+	Map output bytes=18
		+	Map output materialized bytes=28
		+	Input split bytes=159
		+	Combine input records=0
		+	Combine output records=0
		+	Reduce input groups=2
		+	Reduce shuffle bytes=28
		+	Reduce input records=2
		+	Reduce output records=0
		+	Spilled Records=4
		+	Shuffled Maps =1
		+	Failed Shuffles=0
		+	Merged Map outputs=1
		+	GC time elapsed (ms)=62
		+	CPU time spent (ms)=2090
		+	Physical memory (bytes) snapshot=447819776
		+	Virtual memory (bytes) snapshot=2117320704
		+	Total committed heap usage (bytes)=402653184
		+	Shuffle Errors
		+	BAD_ID=0
		+	CONNECTION=0
		+	IO_ERROR=0
		+	WRONG_LENGTH=0
		+	WRONG_MAP=0
		+	WRONG_REDUCE=0
		+	File Input Format Counters
		+	Bytes Read=118
		+	File Output Format Counters
		+	Bytes Written=97
		+	Job Finished in 18.315 seconds
		+	Estimated value of Pi is 4.00000000000000000000
		+	[david@openstack1 ~]$ sudo su -
		+	[root@openstack1 ~]# rpm -e cluster-tools-slave-7.0-41_cm7.0.noarch
		+	[root@openstack1 ~]# yum install cluster-tools-7.0-4764_cm7.0.noarch
		+	[root@openstack1 ~]# exit
		+	[david@openstack1 hadoop]$ ./cm-hadoop-tests.sh ubs-test
		+	hadoop/ubs-test/Hortonworks/2.6.0.2.2.0.0-2041
		+	hbase/ubs-test/Hortonworks/0.98.4.2.2.0.0-2041-hadoop2
		+
		+	# STUFF RUNS FOR A WHILE - GO THROUGH GUI
		+
		+	# Lets uninstall
		+
		+
		+
		+	</syntaxhighlight>

@@ Line 40: / Line 40: @@
 * If you have an xml Hadoop configuration file, just run:
-<syntaxhighlight> cm-hadoop-setup -c <config.xml> </syntaxhighlight>
+<syntaxhighlight> cm-hadoop-setup -c <config>.xml </syntaxhighlight>
 Some example configuration files are located in <tt> /cm/local/apps/cluster-tools/hadoop/conf/ </tt>

← Older revision		Revision as of 10:57, 11 February 2015
Line 38:		Line 38:

	== Installing from terminal ==		== Installing from terminal ==
		+
		+	* If you have an xml Hadoop configuration file, just run:
		+	<syntaxhighlight> cm-hadoop-setup -c <config.xml> </syntaxhighlight>
		+
		+	Some example configuration files are located in <tt> /cm/local/apps/cluster-tools/hadoop/conf/ </tt>
		+
		+	* If you don't have a configuration file then run:
		+	<syntaxhighlight> cm-hadoop-setup </syntaxhighlight>
		+
		+	Then follow the on screen instructions, which are the same as the CMGUI ones. A configuration file will be then created.

@@ Line 34: / Line 34: @@
 * Review the configuration file and save it if it's needed:
 [[File:hadoop-install8.png | 500px]]
 == Installing from terminal ==

@@ Line 11: / Line 11: @@
 [[File:hadoop-install0.png | 800px]]
-* sd
+* Provide instance name, java installation path and Hadoop data path:
 [[File:hadoop-install1.png | 500px]]
-*dd
+* Provide Hadoop distribution and make sure the tarball names are correct:
 [[File:hadoop-install2.png | 500px]]
-*ff
+* Select namenodes and Yarn server host:
 [[File:hadoop-install3.png | 500px]]
-*yy
+* Select the category of the Hadoop nodes:
 [[File:hadoop-install4.png | 500px]]
-*uu
+* Select any other optional nodes to act as data nodes:
 [[File:hadoop-install5.png | 500px]]
-*hh
+* Select Zookeeper nodes (at least 3):
 [[File:hadoop-install6.png | 500px]]
-*uu
+* Select HBASE nodes:
 [[File:hadoop-install7.png | 500px]]
-*uu
+* Review the configuration file and save it if it's needed:
 [[File:hadoop-install8.png | 500px]]
 == Installing from terminal ==

@@ Line 9: / Line 9: @@
 * Go to the "''Hadoop HDFS''" tab and click "''Add HDFS Instance''"
-[[File:hadoop-install0.png | 800x800px]]
+[[File:hadoop-install0.png | 800px]]
 * sd
-[[File:hadoop-install1.png | 800x800px]]
+[[File:hadoop-install1.png | 500px]]
 *dd
-[[File:hadoop-install2.png | 800x800px]]
+[[File:hadoop-install2.png | 500px]]
 *ff
-[[File:hadoop-install3.png | 800x800px]]
+[[File:hadoop-install3.png | 500px]]
 *yy
-[[File:hadoop-install4.png | 800x800px]]
+[[File:hadoop-install4.png | 500px]]
 *uu
-[[File:hadoop-install5.png | 800x800px]]
+[[File:hadoop-install5.png | 500px]]
 *hh
-[[File:hadoop-install6.png | 800x800px]]
+[[File:hadoop-install6.png | 500px]]
 *uu
-[[File:hadoop-install7.png | 800x800px]]
+[[File:hadoop-install7.png | 500px]]
 *uu
-[[File:hadoop-install8.png | 800x800px]]
+[[File:hadoop-install8.png | 500px]]
 == Installing from terminal ==

← Older revision		Revision as of 10:26, 11 February 2015
Line 9:		Line 9:

	* Go to the "''Hadoop HDFS''" tab and click "''Add HDFS Instance''"		* Go to the "''Hadoop HDFS''" tab and click "''Add HDFS Instance''"
		+	*

	== Installing from terminal ==		== Installing from terminal ==

← Older revision		Revision as of 10:26, 11 February 2015
Line 1:		Line 1:
−	Before any installation, make sure you have the appropriate licence to install Hadoop nodes.	+	== First steps ==
		+
		+	* Before any installation, make sure you have the appropriate licence to install Hadoop nodes.
		+	* Preferably, create a new Node Category, which will include all the Hadoop nodes (nameservers, datanodes etc.)
		+	* Install the <tt>cm-hortonworks-hadoop</tt> package, through Bright:
		+	<syntaxhighlight> sudo yum install cm-hortonworks-hadoop</syntaxhighlight>

	== Installing from CMGUI ==		== Installing from CMGUI ==