hadoop高可用核心组件与机制1.HDFS NameNode 高可用通过Active/Standby双 NameNode 架构实现。两个 NameNode 共享一个编辑日志存储如 QJM 或 NFSActive 节点写入Standby 节点实时回放保持元数据同步。各 DataNode 同时向两个 NameNode 发送块报告确保元数据一致。故障转移由ZKFailoverControllerZKFC进程监控基于 ZooKeeper 实现自动切换。2. YARN ResourceManager 高可用同样采用 Active/Standby 模式。多个 ResourceManager 通过 ZooKeeper 选举出 Active 节点Standby 节点同步状态如应用运行信息。当 Active 宕机Standby 自动接管正在运行的任务可继续部分作业可能需要重试。3.节点实现server1:192.168.80.151 NN进程 h1节点主节点server6:192.168.80.156 NN进程 h2节点NN备用节点server2:192.168.80.152 DN进程server3:192.168.80.153 DN进程server5:192.168.80.155 DN进程先停掉服务所有datanode节点删除用户hadoop tmp临时目录内容[hadoopserver1 hadoop]$ sbin/stop-all.sh[hadoopserver6 root]$ rm -fr tmp/*zookeeper 集群部署安装zookeeper[rootserver1 ~]# mv apache-zookeeper-3.8.6-bin.tar.gz /home/hadoop[hadoopserver1 ~]$ tar zxf apache-zookeeper-3.8.6-bin.tar.gzvim zoo.cfgserver.1192.168.80.152:2888:3888各节点配置文件相同并且需要在/tmp/zookeeper目录中创建myid文件写入一个唯一的数字取值范围在1-255。比如192.168.36.167节点的myid文件写入数字“1”此数字与配置文件中的定义保持一致server.1192.168.36.167:2888:3888其它节点依次类推。id 按照zoo.cfg配置写入/tmp/hadoop/myid[hadoopserver2 ~]$ mkdir -r tmp/zookeeper -r递归先建立tmp/,再在tmp/下建立zookeeper[hadoopserver2 ~]$ echo 1 tmp/zookeeper/myid[hadoopserver3 root]$ mkdir -r tmp/zookeeper[hadoopserver3 root]$ echo 2 tmp/zookeeper/myid[hadoopserver5 ~]$ mkdir-r tmp/zookeeper[hadoopserver5 ~]$ echo 3 /tmp/zookeeper/myid在各节点启动服务server2、server3、server5bin/zkServer.sh start查看节点状态bin/zkServer.sh statushadoop配置配置core-site.xml文件[hadoopserver1 hadoop]$ vim core-site.xmlconfigurationpropertynamefs.defaultFS/namevaluehdfs://masters/value/propertypropertynameha.zookeeper.quorum/namevalue192.168.80.152:2181,192.168.80.153:2181,192.168.80.155:2181/value/property/configuration配置 hdfs-site.xml 文件hdfs-site.xml 是 Hadoop HDFS 的核心配置文件在高可用HA模式下需要特别配置以下几个关键参数[hadoopserver1 hadoop]$ vim hdfs-site.xmlconfigurationpropertynamedfs.replication/namevalue3/value/propertypropertynamedfs.nameservices/namevaluemasters/value/propertypropertynamedfs.ha.namenodes.masters/namevalueh1,h2/value/propertypropertynamedfs.namenode.rpc-address.masters.h1/namevalue192.168.80.151:9000/value/propertypropertynamedfs.namenode.http-address.masters.h1/namevalue192.168.80.151:9870/value/propertypropertynamedfs.namenode.rpc-address.masters.h2/namevalue192.168.80.156:9000/value/propertypropertynamedfs.namenode.http-address.masters.h2/namevalue192.168.80.156:9870/value/propertypropertynamedfs.namenode.shared.edits.dir/namevalueqjournal://192.168.80.152:8485;192.168.80.153:8485;192.168.80.155:8485/masters/value/propertypropertynamedfs.journalnode.edits.dir/namevalue/tmp/journaldata/value/propertypropertynamedfs.ha.automatic-failover.enabled/namevaluetrue/value/propertypropertynamedfs.client.failover.proxy.provider.masters/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value/propertypropertynamedfs.ha.fencing.methods/namevaluesshfenceshell(/bin/true)/value/propertypropertynamedfs.ha.fencing.ssh.private-key-files/namevalue/home/hadoop/.ssh/id_rsa/value/propertypropertynamedfs.ha.fencing.ssh.connect-timeout/namevalue30000/value/property/configuration启动hdfs集群1、确认启动zookeeper集群三节点用jps查看2、在三个DN上依次启动journalnode(第一次启动hdfs必须先启动journalnode)[hadoopserver3 ~]$ cd hadoop[hadoopserver3 hadoop]$ bin/hdfs --daemon start journalnode[hadoopserver3 hadoop]$ jps6355JournalNode6085 QuorumPeerMain6431 Jps3、格式化HDFS集群server1上执行[hadoopserver1 hadoop]$ bin/hdfs namenode -formatNamenode数据默认存放在/tmp需要把数据拷贝到h2[hadoopserver1 hadoop]$ scp -r /tmp/hadoop-hadoop server6:/tmp4、格式化zookeeper 只需在h1上执行即可[hadoopserver1 hadoop]$ bin/hdfs zkfc -formatZ5、启动hdfs集群只需在h1上执行即可[hadoopserver1 hadoop]$ sbin/start-dfs.sh6、查看各节点状态[hadoopserver1 hadoop]$ jps21732 Jps20869 NameNode21310 DFSZKFailoverController[hadoopserver6 hadoop-hadoop]$ jps10258 Jps9764 NameNode9882 DFSZKFailoverController[hadoopserver2 hadoop]$ jps16435 JournalNode16629 DataNode16777 Jps10606 QuorumPeerMain[hadoopserver3 hadoop]$ jps7072 DataNode6355 JournalNode6085 QuorumPeerMain7214 Jps[hadoopserver5 hadoop]$ jps11297 Jps9410 QuorumPeerMain9638 JournalNode11209 DataNode主备节点设置成功7、测试故障自动切换[hadoopserver1 hadoop]$ jps20869 NameNode23594 Jps21310 DFSZKFailoverController[hadoopserver1 hadoop]$ kill -9 20869强行关掉NN进程后依然可以访问此时h2转为active状态接管NN重新启动NN进程为standby状态[hadoopserver1 hadoop]$ bin/hdfs --daemon start namenodeyarn高可用配置编辑 mapred-site.xml[hadoopserver1 hadoop]$ vim mapred-site.xmlconfigurationpropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynamemapreduce.application.classpath/namevalue$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value/property/configuration编辑 yarn-site.xml[hadoopserver1 hadoop]$ vim yarn-site.xmlconfigurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value/propertypropertynameyarn.resourcemanager.ha.enabled/namevaluetrue/value/propertypropertynameyarn.resourcemanager.cluster-id/namevalueRM_CLUSTER/value/propertypropertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/value/propertypropertynameyarn.resourcemanager.hostname.rm1/namevalueserver1/value/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueserver6/value/propertypropertynameyarn.resourcemanager.recovery.enabled/namevaluetrue/value/propertypropertynameyarn.resourcemanager.store.class/namevalueorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore/value/propertypropertynameyarn.resourcemanager.zk-address/namevalue192.168.80.152:2181,192.168.80.153:2181,192.168.80.155:2181/value/property/configuration启动yarn服务[hadoopserver1 hadoop]$ cd ../..[hadoopserver1 hadoop]$ sbin/start-yarn.sh各节点状态[hadoopserver1 hadoop]$ jps43344 Jps24938 NameNode43148 ResourceManager21310 DFSZKFailoverController[hadoopserver6 hadoop-hadoop]$ jps9764 NameNode11365 Jps9882 DFSZKFailoverController11293 ResourceManager[hadoopserver2 hadoop]$ jps16435 JournalNode17459 NodeManager16629 DataNode17541 Jps10606 QuorumPeerMain[hadoopserver3 hadoop]$ jps7072 DataNode6355 JournalNode6085 QuorumPeerMain7785 Jps7662 NodeManager[hadoopserver5 hadoop]$ jps9410 QuorumPeerMain14612 Jps9638 JournalNode14550 DataNode13707 NodeManager测试yarn故障切换[hadoopserver1 hadoop]$ jps24938 NameNode43148 ResourceManager21310 DFSZKFailoverController43982 Jps[hadoopserver1 hadoop]$ kill -9 43148[hadoopserver1 hadoop]$ bin/yarn --daemon start resourcemanager 手动恢复主备切换原理下面我们就来看看YARN是如何实现多个ResourceManager之间的主备切换的。创建锁节点 在ZooKeeper上会有一个/yarn-leader-election/RM_CLUSTER的锁节点所有的 ResourceManager 在启动的时候都会去竞争写一个 Lock 子节点/yarn-leader-election/RM_CLUSTER/ActiveBreadCrumb该节点是临时节点。ZooKeepr能够为我们保证最终只有一个 ResourceManager 能够创建成功。创建成功的那个ResourceManager就切换为Active状态没有成功的那些ResourceManager则切换为Standby状态。[hadoopserver2 apache-zookeeper-3.8.6-bin]$ bin/zkCli.shls /yarn-leader-election[RM_CLUSTER]get /yarn-leader-election/RM_CLUSTER/ActiveBreadCrumbRM_CLUSTERrm2可以看到此时集群中ResourceManager2为Active。注 册 Watcher 监 听 所 有 Standby 状 态 的 ResourceManager 都 会 向/yarn-leader-election/RM_CLUSTER/ActiveBreadCrumb节点注册一个节点变更的Watcher监听利用临时节点的特性能够快速感知到Active状态的ResourceManager的运行情况。主备切换当Active状态的ResourceManager出现诸如宕机或重启的异常情况时其在ZooKeeper 上 连 接 的 客 户 端 会 话 就 会 失 效 因 此/yarn-leader-election/RM_CLUSTER/ActiveBreadCrumb节点就会被删除。此时其余各个Standby状态的ResourceManager就都会接收到来自ZooKeeper服务端的Watcher事件通知然后会重复进行步骤1的操作。以上就是利用ZooKeeper来实现ResourceManager的主备切换的过程实现了ResourceManager的HA。HDFS中NameNode的HA的实现原理跟YARN中ResourceManager的HA的实现原理相同。其锁节点为/hadoop-ha/masters/ActiveBreadCrumb。YARN的主备切换就像一群人抢一把唯一能发号施令的“麦克风”Active状态。谁先在ZooKeeper上成功插上自己的“小旗子”创建临时节点谁就是老大Active没抢到的就是替补Standby。一旦老大倒下了宕机他插的旗子会自动消失临时节点特性剩下的人立刻重新抢这把麦克风。hadoop高可用关闭[hadoopserver1 hadoop]$ sbin/stop-yarn.sh[hadoopserver1 hadoop]$ sbin/stop-dfs.sh[hadoopserver2 hadoop]$ cd[hadoopserver2 ~]$ cd apache-zookeeper-3.8.6-bin/[hadoopserver2 apache-zookeeper-3.8.6-bin]$ bin/zkServer.sh stop除server1其他5节点取消挂载umount /home/hadoop/rm -fr ~/tmp/*