Hadoop 2.7.1 on a Virtual Machines Cluster

This time I will use FEDORA Server 24 to create four virtual machines. The first one is called “FedoraMaster”:

vm1The configuration is going to be as similar as it is in the Raspberry Pi architecture, with 1GB of memory and 16 GB of disk space and the cpu is Intel 3. The settings of storage and Network must be changed as you need:

Screenshot from 2016-07-13 03-05-41During the installation part, the interfaces must be set as is also needed:

Screenshot from 2016-07-13 03-11-34Generate the keygen and store it in the authorized_keys file:

Screenshot from 2016-07-13 05-05-19Set the /etc/hostname and the /etc/hosts to ensure the remote access.

hadoop2.7Install the package of Java and then set the Download file to store the hadoop2.7.1 installation packages and save the unzip files into the /opt file by using the tar -xvzf hadoop-2.7.1.tar.gz -C /opt/ command.

Set the path to the files .bashrc and hadoop-env.sh:

Screenshot from 2016-07-13 05-29-32Configure the hadoop xml files as it is suggested, starting with the core:

vm3Followed by the mapred-site.xml file, that has copied from the template:

Screenshot from 2016-07-13 05-56-45Finally, the yarn file that contains more specification for hardware:

Screenshot from 2016-07-13 06-01-12Finally we run the hadoop namenode -format command and the scripts:

Screenshot from 2016-07-13 06-07-38To run it as a VMCluster, there are some considerations such as:

1.- Set the IPs and hostname, hosts and update the .bashrc and script to the hadoop environment.

2.- Add the IPs in the slave file, which is located inside the hadoop file.

3.- Change the IP on core-site.xml and create a path to store the datanode on each slave node. For more details, you can visit this Website.

About Julita Inca

Ingeniero de Sistemas UNAC, Magíster en Ciencias de la Computación PUCP, OPW GNOME 2011, Miembro de la GNOME Foundation desde el 2012, Embajadora Fedora Perú desde el 2012, ganadora del scholarship of the Linux Foundation 2012, experiencia como Admin Linux en GMD y Especialista IT en IBM, con certificaciones RHCE, RHCSA, AIX 6.1, AIX 7 Administrator e ITILv3. Experiencia académica en universidades como PUCP, USIL y UNI. HPC researcher, a simple mortal, like you!
This entry was posted in τεχνολογια :: Technology and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s