Hadoop 3.2.4 云服务器ECS部署阿里云3节点集群配置与安全组规则详解1. 云原生环境下的Hadoop集群规划在阿里云ECS上部署Hadoop集群首先需要明确资源规划。与本地虚拟机部署不同云环境需要考虑实例规格、网络拓扑和存储类型的选型。以下是针对中型数据处理场景的推荐配置节点角色实例规格系统盘数据盘数量备注NameNodeecs.g6e.xlarge100GB500GB ESSD1高可用可选Secondary NNResourceManagerecs.g6e.xlarge100GB-1与NameNode分设提高可用性DataNodeecs.d1ne.2xlarge100GB4TB×2 HDD3本地SSD提升I/O性能关键配置建议网络规划所有节点应部署在同一可用区使用专有网络VPC确保低延迟通信存储策略NameNode使用ESSD云盘保证元数据访问速度DataNode挂载本地HDD盘实现高吞吐数据存储系统优化关闭swap分区设置vm.swappiness1避免GC时触发swap提示生产环境建议DataNode节点不少于3个HDFS副本数设置为3以保证数据可靠性2. ECS实例初始化配置2.1 系统基础环境准备所有节点需执行以下初始化操作# 更新系统并安装基础工具 sudo yum update -y sudo yum install -y epel-release sudo yum install -y jq telnet ncdu tree lsof # 优化内核参数 echo vm.swappiness 1 | sudo tee -a /etc/sysctl.conf echo net.ipv4.tcp_tw_reuse 1 | sudo tee -a /etc/sysctl.conf sudo sysctl -p # 创建专用用户 sudo groupadd hadoop sudo useradd -g hadoop hadoop echo hadoop ALL(ALL) NOPASSWD:ALL | sudo tee /etc/sudoers.d/hadoop2.2 JDK环境部署Hadoop 3.2.4需要JDK8环境推荐使用OpenJDK# 安装OpenJDK sudo yum install -y java-1.8.0-openjdk-devel # 配置环境变量 echo export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk | sudo tee -a /etc/profile echo export PATH$JAVA_HOME/bin:$PATH | sudo tee -a /etc/profile source /etc/profile验证安装java -version # 应输出openjdk version 1.8.0_3623. 安全组与网络配置3.1 安全组入方向规则阿里云安全组需开放以下端口端口范围协议授权对象用途说明22TCP管理IP段SSH管理9870TCP0.0.0.0/0NameNode Web UI8088TCP0.0.0.0/0ResourceManager Web UI9864-9869TCP集群内网段DataNode数据传输8030-8033TCP集群内网段YARN服务通信配置示例代码# 创建安全组如果不存在 ALIYUN_SG_ID$(aliyun ecs CreateSecurityGroup \ --RegionId cn-hangzhou \ --SecurityGroupName hadoop-cluster-sg \ --VpcId vpc-xxxxxx \ --Description Hadoop Cluster Security Group | jq -r .SecurityGroupId) # 添加入方向规则 for port in 22 9870 8088 {9864..9869} {8030..8033}; do aliyun ecs AuthorizeSecurityGroup \ --RegionId cn-hangzhou \ --SecurityGroupId $ALIYUN_SG_ID \ --IpProtocol tcp \ --PortRange $port/$port \ --SourceCidrIp 0.0.0.0/0 done3.2 跨节点SSH免密登录在NameNode节点执行# 生成密钥对 sudo -u hadoop ssh-keygen -t rsa -P -f ~/.ssh/id_rsa # 分发公钥到所有节点 for node in namenode resourcemanager datanode{1..3}; do sudo -u hadoop ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop$node done # 测试免密登录 sudo -u hadoop ssh datanode1 hostname4. Hadoop集群部署实战4.1 软件包分发与安装在NameNode节点操作# 下载Hadoop 3.2.4 wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz tar -zxvf hadoop-3.2.4.tar.gz -C /opt/ chown -R hadoop:hadoop /opt/hadoop-3.2.4 ln -s /opt/hadoop-3.2.4 /opt/hadoop # 同步到其他节点 for node in resourcemanager datanode{1..3}; do rsync -avz /opt/hadoop-3.2.4/ $node:/opt/hadoop-3.2.4/ ssh $node ln -s /opt/hadoop-3.2.4 /opt/hadoop done4.2 关键配置文件优化core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://namenode:8020/value /property property namehadoop.tmp.dir/name value/data/hadoop/tmp/value /property property namehadoop.proxyuser.hadoop.groups/name value*/value /property /configurationhdfs-site.xmlconfiguration property namedfs.namenode.name.dir/name value/data/hadoop/namenode/value /property property namedfs.datanode.data.dir/name value/data/hadoop/datanode/value /property property namedfs.replication/name value3/value /property property namedfs.client.use.datanode.hostname/name valuetrue/value /property /configurationyarn-site.xmlconfiguration property nameyarn.resourcemanager.hostname/name valueresourcemanager/value /property property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.resource.memory-mb/name value12288/value /property /configuration4.3 集群初始化与启动# 格式化HDFS仅在首次执行 sudo -u hadoop hdfs namenode -format # 启动HDFS sudo -u hadoop start-dfs.sh # 启动YARN sudo -u hadoop start-yarn.sh # 验证服务 jps # NameNode应显示NameNode、SecondaryNameNode # DataNode应显示DataNode # ResourceManager应显示ResourceManager # NodeManager应显示NodeManager5. 运维监控与调优5.1 基础监控指标组件关键指标监控方法NameNode堆内存使用、文件系统容量JMX端口9870DataNode磁盘使用率、块报告延迟dfs.datanode.https.port9864ResourceManager应用队列状态、容器分配Web UI8088NodeManager容器资源使用、健康状态yarn.nodemanager.webapp.address:80425.2 性能调优参数hadoop-env.sh内存配置export HDFS_NAMENODE_OPTS-Xmx8g -Xms8g export HDFS_DATANODE_OPTS-Xmx4g -Xms4g export YARN_RESOURCEMANAGER_OPTS-Xmx4g -Xms4g export YARN_NODEMANAGER_OPTS-Xmx4g -Xms4gmapred-site.xml优化property namemapreduce.map.memory.mb/name value2048/value /property property namemapreduce.reduce.memory.mb/name value4096/value /property