Hapoop 搭建 (三)hadoop 集群搭建 https://www.cnblogs.com/xuchen0117/p/12459610.html
Hapoop 搭建 (五) 搭建 zookeeper 集群环境 https://www.cnblogs.com/xuchen0117/p/12454758.html
Hapoop 搭建 (七)HDFS HA 搭建 https://www.cnblogs.com/xuchen0117/p/12466245.html
结构以及服务器角色
一, yarn-site.xml 文件配置
在 centos01 的 hadoop 安装目录的配置文件夹中 配置 yarn-site.xml (路径:/opt/modules/hadoop-2.8.2/etc/hadoop ), 完整文件如下:
- <configuration>
- <!-- 指定在 YARN 上面可以运行 MapReduce 程序 -->
- <property>
- <name>yarn.nodemanager.aux-services</name>
- <value>mapreduce_shuffle</value>
- </property>
- <!-- YARN HA 配置 -->
- <!-- 开启 ResourceManager HA 高可用 -->
- <property>
- <name>yarn.resourcemanager.ha.enabled</name>
- <value>true</value>
- </property>
- <!-- 指定 RM 的 cluster id -->
- <property>
- <name>yarn.resourcemanager.cluster-id</name>
- <value>yarncluster</value>
- </property>
- <!-- 集群中 ResourceManager 的 Id 列表 -->
- <property>
- <name>yarn.resourcemanager.ha.rm-ids</name>
- <value>rm1,rm2</value>
- </property>
- <!-- 分别指定 RM 的地址 所在节点主机名 -->
- <property>
- <name>yarn.resourcemanager.hostname.rm1</name>
- <value>centos01</value>
- </property>
- <property>
- <name>yarn.resourcemanager.hostname.rm2</name>
- <value>centos02</value>
- </property>
- <!-- 分别指定 web 页面的访问地址 -->
- <property>
- <name>yarn.resourcemanager.webapp.address.rm1</name>
- <value>centos01:8088</value>
- </property>
- <property>
- <name>yarn.resourcemanager.webapp.address.rm2</name>
- <value>centos02:8088</value>
- </property>
- <!-- 指定 zk 集群地址 -->
- <property>
- <name>yarn.resourcemanager.zk-address</name>
- <value>centos01:2181,centos02:2181,centos03:2181</value>
- </property>
- <!-- 启用自动恢复 -->
- <property>
- <name>yarn.resourcemanager.recovery.enabled</name>
- <value>true</value>
- </property>
- <!-- 指定 resourcemanager 的状态信息存储在 zookeeper 集群上 -->
- <property>
- <name>yarn.resourcemanager.store.class</name>
- <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
- </property>
- <property>
- <name>yarn.log-aggregation-enable</name>
- <value>true</value>
- </property>
- <property>
- <name>yarn.log-aggregation.retain-seconds</name>
- <value>86400</value>
- </property>
- </configuration>
二, 同步到其它节点
将 yarn-site.xml 配置文件复制到另 2 个节点上
- scp /opt/modules/hadoop-2.8.2/etc/hadoop/yarn-site.xml [email protected]:/opt/modules/hadoop-2.8.2/etc/hadoop/
- scp /opt/modules/hadoop-2.8.2/etc/hadoop/yarn-site.xml [email protected]:/opt/modules/hadoop-2.8.2/etc/hadoop/
三, 启动 zookeeper 集群
停止
sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh stop
启动
- sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh start
- [[email protected] ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh status
- ZooKeeper JMX enabled by default
- Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
- Mode: leader
- [[email protected] ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh stop
- ZooKeeper JMX enabled by default
- Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
- Stopping zookeeper ... STOPPED
- [[email protected] ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh status
- ZooKeeper JMX enabled by default
- Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
- Error contacting service. It is probably not running.
- [[email protected] ~]$ sh /opt/modules/zookeeper-3.4.14/bin/zkServer.sh start
- ZooKeeper JMX enabled by default
- Using config: /opt/modules/zookeeper-3.4.14/bin/../conf/zoo.cfg
- Starting zookeeper ... STARTED
- [[email protected] ~]$
四, 启动 YARN 集群
分别进入 centos01,centos02 的 hadoop 的安装目录启动 ResourceManager
sh /opt/modules/hadoop-2.8.2/sbin/yarn-daemon.sh start resourcemanager
进入 3 各节点的 hadoop 安装目录启动 NodeManager
sh /opt/modules/hadoop-2.8.2/sbin/yarn-daemon.sh start nodemanager
五, 查看节点进程
- [[email protected] sbin]$ jps
- 96960 ResourceManager
- 64850 JournalNode
- 97427 NodeManager
- 64164 DataNode
- 34004 QuorumPeerMain
- 65305 DFSZKFailoverController
- 98876 Jps
- 36702 NameNode
- [[email protected] ~]$ jps
- 59364 DataNode
- 84087 NodeManager
- 83686 ResourceManager
- 85798 Jps
- 61513 DFSZKFailoverController
- 20457 QuorumPeerMain
- 59980 JournalNode
- 82735 NameNode
- [[email protected] hadoop]$ jps
- 62977 NodeManager
- 49509 JournalNode
- 65191 Jps
- 130810 QuorumPeerMain
- 48845 DataNode
六, 测试 YARN 故障转移
访问 centos01 活动的 ResourceManager
http://192.168.0.171:8088/cluster
访问 centos02 活动的 ResourceManager, 此时活动的是 centos01 所以会自动跳转到 centos01 上
- http://192.168.0.172:8088/cluster
- [[email protected] sbin]$ jps
- 96960 ResourceManager
- 64850 JournalNode
- 97427 NodeManager
- 64164 DataNode
- 34004 QuorumPeerMain
- 65305 DFSZKFailoverController
- 98876 Jps
- 36702 NameNode
- [[email protected] sbin]$ kill -9 96960
- [[email protected] sbin]$
自动切换到 centos02 上
来源: http://www.bubuko.com/infodetail-3478694.html