Hadoop Setup - Pseudo Cluster Installation Setup

2011-07-10T13:32:00.001-07:00

Hello Everybody,

In this post I would be going over hadoop installation on a single machine for development purpose.

Start by creating a user to run hadoop as admin, though it is easy to run hadoop using your own account, I prefer creating a dedicated user to run hadoop. A user "hadoop" is created for this purpose.

Download and extract the latest hadoop release tar file from the Apache site http://hadoop.apache.org/mapreduce/releases.html

tar xzf hadoop-0.21.0.tar.gz

Lets create an environmental variable HADOOP_INSTALL to point to hadoop installation and update the path so that we can execute hadoop commands from any location.

export HADOOP_INSTALL=/Users/hadoop/hadoop-0.21.0
export PATH=$PATH:$HADOOP_INSTALL/bin

Hadoop, the default configuration is set for "Standalone Mode". In the Standalone mode the default filesystem used is the local file system native to the operating system. Hadoop Configuration is mainly controlled by three files core-site.xml, hdfs-site.xml and mapred.xml.

Core-site.xml is mainly responsible for hadoop-core settings and the possible parameters for the configuration can be found at $HADOOP_INSTALL/common/docs/core-default.html. For Psuedo Cluster configuration add the following property to core-site.xml

fs.default.name = hdfs://localhost:9000

Hdfs-site.xml is the configuration file that controls the hadoop distributed file system settings and the parameters can be found at $HADOOP_INSTALL/hdfs/docs/hdfs-default.html. Pseudo Cluster setup properties are mentioned below.

dfs.replication = 1

Mapred-site.xml is the configuration file that controls the properties for Map Reduce Job Tracker and Task tracker settings and the properties can be found at $HADOOP_INSTALL/docs/mapred-default.html. Pseudo Cluster setup properties are given below.

mapred.job.tracker = localhost:9001

Before adding any files to hdfs you can format the Hadoop file system with the command "hadoop namenode -format" .

Hadoop requires SSH access to the local node as well as remote nodes, for our pseudo cluster setup we need to configure ssh access to localhost for the user "hadoop". Follow the steps below to create the rsa key pairs and add them to the authorized keys.

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

"ssh localhost" would confirm if the above steps have worked properly or not. To debug ssh you can issue "ssh -vvv localhost" and investigate the error in detail.

Now hadoop is all set to run in a pseudo cluster configuration. Hadoop can be started with "start-all.sh" which starts all the required daemons. If you want to have a finer control on what components need to be started then you can start dfs with "start-dfs.sh" and mapred trackers with "start-mapred.sh" and configration files can also be passed as parameters to these scripts of you conf files lies outside the conf directory "-config path-to-conf directory".

Issue a "start-all.sh" and you can use the jvm process tool "jps" to monitor the java processes of hadoop have started as expected or not. The five components that will get started are NameNode, DataNode, SecondaryNameNode, JobTracker, TaskTracker.

Hadoop comes with several web interfaces which are available at the below locations.

http://localhost:50030/ - Web UI for MapReduce Job Trackers
http://localhost:50060/ - Web UI for Task Trackers
http://localhost:50070/ - Web UI for HDFS name Nodes

In the next part of the series I will be posting on how to setup Cloudera's Distribution of Hadoop and HBase.

Cheers,

Vijay Akkineni.

Computing Junkie

Hadoop Setup - Pseudo Cluster Installation Setup