Hadoop 2.8.0 Deployment and Installation Tutorial

Hadoop deployment preparation: install Linux locally under VMware. In the Linux family, the CentOS 7 download address is http://101.110.118.58/isoredirect.centos.org/centos/7/isos/x86\_64/CentOS-7-x86\_64-Minimal-1611.iso The matching Java version is 1.8.0_121 Hadoop version 2.8.0 download address http://mirror.bit.edu.cn/apache/hadoop/common/ you can download whichever version you like

1. Basic configuration

First install a CentOS 7 and get Java configured. Java environment configuration: my Java install location is /usr/java/ jdk1.8.0_121 edit the Java environment with vi ~/.bash_profile, add or modify export JAVA_HOME=/usr/java/jdk1.8.0_121 export PATH=$JAVA_HOME/bin:$PATH then run . ~/.bash_profile to make the variables take effect

2. SSH configuration

First run yum install ssh to install it then run ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa to generate the key

3. Hadoop configuration

Extract the Hadoop you downloaded. (Mine is in the /roo directory, so the full path is /root/hadoop) Configure the environment variables: vi ~/.bash_profile export HADOOP_HOME=/root/hadoop export PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$HADOOP_HOME/bin then run . ~/.bash_profile to make the variables take effect

4. Hadoop file configuration

First give this machine a name: for example mine is s204 so run hostnamectl set-hostname s204 to change the hostname then, in the /root/hadoop directory, edit the following files in order

vim etc/hadoop/core-site.xml

In add fs.defaultFS hdfs://s204:9000 hadoop.tmp.dir file:/root/hadoop/tmp io.file.buffer.size 131702 hadoop.proxyuser.hadoop.hosts * hadoop.proxyuser.hadoop.groups *

vim etc/hadoop/hdfs-site.xml

In add dfs.namenode.name.dir file:/root/hadoop/hdfs/name dfs.datanode.data.dir file:/root/hadoop/hdfs/data dfs.replication 3 dfs.namenode.secondary.http-address s204:9001 dfs.webhdfs.enabled true

vim etc/hadoop/mapred-site.xml

In add mapreduce.framework.name yarn mapreduce.jobhistory.address s204:10020 mapreduce.jobhistory.webapp.address s204:19888

vim etc/hadoop/yarn-site.xml

In add yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.auxservices.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.resourcemanager.address s204:8032 yarn.resourcemanager.scheduler.address s204:8030 yarn.resourcemanager.resource-tracker.address s204:8031 yarn.resourcemanager.admin.address s204:8033 yarn.resourcemanager.webapp.address s204:8088 yarn.nodemanager.resource.memory-mb 6078

vim etc/hadoop/yarn-env.sh

Find export JAVA_HOME, uncomment it and edit the Java path to export JAVA_HOME=/usr/java/jdk1.8.0_121 Find JAVA_HEAP_MAX=-Xmx1000m and change it to JAVA_HEAP_MAX=-Xmx1024m

vim etc/hadoop/slaves

Clear it and add s204

Network configuration

My IP is 192.168.5.9 edit the network to use a fixed IP: vim /etc/sysconfig/network-scripts/ifcfg-ens33 set the fixed IP TYPE=”Ethernet” #BOOTPROTO=”dhcp” DEFROUTE=”yes” PEERDNS=”yes” PEERROUTES=”yes” IPV4_FAILURE_FATAL=”no” IPV6INIT=”yes” IPV6_AUTOCONF=”yes” IPV6_DEFROUTE=”yes” IPV6_PEERDNS=”yes” IPV6_PEERROUTES=”yes” IPV6_FAILURE_FATAL=”no” IPV6_ADDR_GEN_MODE=”stable-privacy” NAME=”ens33” UUID=”b9fe1e5c-be20-47f1-a2d3-e12f5ddb6aa1” DEVICE=”ens33” ONBOOT=”yes” IPADDR0=192.168.5.9 PREFIX0=24 GATEWAY0=192.168.5.2 DNS1=114.114.114.114 then restart the network with systemctl restart network run ip add to check whether the network IP matches what you set

Starting Hadoop

Go into the /root/hadoop directory run the format command ./bin/hdfs namenode –format if “Exiting with status 0” shows up on the fifth line from the bottom, it succeeded then start it with ./sbin/start-all.sh once it’s up, run ./bin/hdfs dfsadmin –report to see whether there are nodes; if it returns “cannot connect”, startup failed run systemctl stop firewalld.service to turn off the firewall then enter s204:8088 in a browser and you’ll see the Hadoop interface

Configuring the other nodes

Once the above is configured, shut down CentOS 7 and do a full clone. In the newly cloned system, change the IP address and hostname. Note that in the Hadoop config file etc/hadoop/hdfs-site.xml the file address in dfs.datanode.data.dir file:/root/hadoop/hdfs/data must not be the same. Mine are three different paths: file:/root/hadoop/hdfs/data file:/root/hadoop/hdfs/data/205 file:/root/hadoop/hdfs/data/206 After cloning, and once the config files, IP and hostname are changed, edit vim etc/hadoop/slaves on the s204 machine and add s205 s206 copy the SSH token for passwordless login, for example copy it to s205 with scp ~/.ssh/authorized_keys root@s205:~/.ssh/ do the same on the other machine the other machines also need to be formatted then stop Hadoop on s204 with ./sbin/stop-all.sh start it again with ./sbin/start-all.sh and you’ll see all three nodes in the browser.