Hadoop 的运行模式
Hadoop 一般有三种运行模式, 分别是:
单机模式 (Standalone Mode), 默认情况下, Hadoop 即处于该模式, 使用本地文件系统, 而不是分布式文件系统., 用于开发和调试.
伪分布式模式 (Pseudo Distrubuted Mode), 使用的是分布式文件系统, 守护进程运行在本机机器, 模拟一个小规模的集群, 在一台主机模拟多主机, 适合模拟集群学习.
完全分布式集群模式 (Full Distributed Mode),Hadoop 的守护进程运行在由多台主机搭建的集群上, 是真正的生产环境.
这里介绍的就是如何搭建一个 Hadoop 完全分布式集群.
安装环境介绍
准备了四个服务器, IP 为 192.168.0.236,192.168.0.237,192.168.0.238,192.168.0.239, 其中 192.168.0.236 作为主节点, 其他 3 个作为从节点. 具体版本信息如下:
- CentOS 7.4
- JDK 8
- Hadoop 2.10.0
准备安装环境
设置主机名
在各个服务器上修改对应的主机名:
- # 在 192.168.0.236 上执行:
- hostnamectl set-hostname onemore-hadoop-master
- # 在 192.168.0.237 上执行:
- hostnamectl set-hostname onemore-hadoop-slave1
- # 在 192.168.0.238 上执行:
- hostnamectl set-hostname onemore-hadoop-slave2
- # 在 192.168.0.239 上执行:
- hostnamectl set-hostname onemore-hadoop-slave3
关闭 SELINUX
编辑 / etc/selinux/config 文件:
vi /etc/selinux/config
把
SELINUX=enforcing
修改为:
SELINUX=disabled
重启服务器
reboot
设置 hosts
- cat>> /etc/hosts <<EOF
- 192.168.0.236 onemore-hadoop-master
- 192.168.0.237 onemore-hadoop-slave1
- 192.168.0.238 onemore-hadoop-slave2
- 192.168.0.239 onemore-hadoop-slave3
- EOF
关闭防火墙
停止防火墙
systemctl stop firewalld.service
禁止防火墙开机启动
systemctl disable firewalld.service
设置免密登录
分布式集群搭建需要主节点能够免密登录至各个从节点上. 因此, 需要在主节点上生成公钥, 把将主节点的公钥在从节点中加入授权.
在 192.168.0.236 上生成公钥.
SSH-keygen -t rsa
在 192.168.0.236 上, 把公钥发送到各个从节点
- scp ~/.SSH/id_rsa.pub 192.168.0.237:~/.SSH
- scp ~/.SSH/id_rsa.pub 192.168.0.238:~/.SSH
- scp ~/.SSH/id_rsa.pub 192.168.0.239:~/.SSH
这时还不是免密登录登录的, 需要输入用户名和密码.
将公钥追加到各个从节点的授权里.
在每个从节点执行一下命令:
cat ~/.SSH/id_rsa.pub>> ~/.SSH/authorized_keys
安装 JDK
参见之前的《详解在 Linux 系统中安装 JDK https://mp.weixin.qq.com/s/emm17Xx8XZ8newBIi1Vv3Q 》, 这里就不再赘述了.
Hadoop 环境配置
主节点配置
下载 Hadoop
从北京理工大学的镜像上下载 Hadoop:
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz
创建文件夹
mkdir /usr/local/hadoop
解压
tar -xzvf hadoop-2.10.0.tar.gz -C /usr/local/hadoop
- cat>> /etc/profile <<EOF
- #Hadoop
- export HADOOP_HOME=/usr/local/hadoop/hadoop-2.10.0
- export PATH=\$PATH:\$HADOOP_HOME/bin
- EOF
- <configuration>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>file:/usr/local/hadoop/tmp</value>
- <description>Abase for other temporary directories.</description>
- </property>
- <property>
- <name>fs.defaultFS</name>
- <value>hdfs://onemore-hadoop-master:9000</value>
- </property>
- </configuration>
- <configuration>
- <property>
- <name>dfs.replication</name>
- <value>3</value>
- </property>
- <property>
- <name>dfs.name.dir</name>
- <value>/usr/local/hadoop/hdfs/name</value>
- </property>
- <property>
- <name>dfs.data.dir</name>
- <value>/usr/local/hadoop/hdfs/data</value>
- </property>
- </configuration>
- <configuration>
- <property>
- <name>mapreduce.framework.name</name>
- <value>yarn</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>http://onemore-hadoop-master:9001</value>
- </property>
- </configuration>
- <configuration>
- <property>
- <name>yarn.nodemanager.aux-services</name>
- <value>mapreduce_shuffle</value>
- </property>
- <property>
- <name>yarn.resourcemanager.hostname</name>
- <value>onemore-hadoop-master</value>
- </property>
- </configuration>
- onemore-hadoop-slave1
- onemore-hadoop-slave2
- onemore-hadoop-slave3
- cat>> /etc/profile <<EOF
- #Hadoop
- export HADOOP_HOME=/usr/local/hadoop/hadoop-2.10.0
- export PATH=\$PATH:\$HADOOP_HOME/bin
- EOF
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/core-site.xml onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/hdfs-site.xml onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/mapred-site.xml onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/yarn-site.xml onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/masters onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- /usr/local/hadoop/hadoop-2.10.0/sbin/stop-all.sh
- /usr/local/hadoop/hadoop-2.10.0/sbin/start-all.sh
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/hadoop-env.sh onemore-hadoop-slave1:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/hadoop-env.sh onemore-hadoop-slave2:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
- scp -r /usr/local/hadoop/hadoop-2.10.0/etc/hadoop/hadoop-env.sh onemore-hadoop-slave3:/usr/local/hadoop/hadoop-2.10.0/etc/hadoop/
来源: https://www.cnblogs.com/heihaozi/p/12020688.html