Hadoop 3.3.6 Spark 3.5.2 环境部署Ubuntu 22.04 集群搭建 5 节点实战1. 环境规划与准备在开始部署之前我们需要明确集群的架构设计。一个典型的HadoopSpark集群包含以下角色节点主节点Master1台运行NameNode、ResourceManager、Spark Master等核心服务从节点Slave4台运行DataNode、NodeManager、Spark Worker等计算存储服务客户端节点可选1台用于提交作业和管理集群硬件配置建议主节点8核CPU/16GB内存/500GB存储从节点16核CPU/32GB内存/2TB存储网络千兆以太网建议配置hosts文件实现节点间域名解析操作系统要求# 检查系统版本 lsb_release -a # 确保所有节点已安装SSH sudo apt update sudo apt install -y openssh-server2. 基础环境配置2.1 系统参数优化在所有节点上执行以下优化配置# 关闭交换分区 sudo swapoff -a sudo sed -i /swap/s/^/#/ /etc/fstab # 调整文件描述符限制 echo * soft nofile 65536 | sudo tee -a /etc/security/limits.conf echo * hard nofile 65536 | sudo tee -a /etc/security/limits.conf # 内核参数优化 cat EOF | sudo tee -a /etc/sysctl.conf vm.swappiness 10 net.ipv6.conf.all.disable_ipv6 1 net.ipv6.conf.default.disable_ipv6 1 net.ipv6.conf.lo.disable_ipv6 1 EOF sudo sysctl -p2.2 创建专用用户sudo groupadd hadoop sudo useradd -g hadoop hduser sudo passwd hduser2.3 SSH免密登录配置在主节点上生成密钥并分发到所有节点su - hduser ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys # 将公钥复制到所有节点包括自己 for node in master slave1 slave2 slave3 slave4; do ssh-copy-id hduser$node done3. Java环境安装Hadoop和Spark都需要Java运行环境推荐安装OpenJDK 11sudo apt install -y openjdk-11-jdk # 验证安装 java -version环境变量配置echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 ~/.bashrc source ~/.bashrc4. Hadoop集群部署4.1 下载与安装wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz tar -xzvf hadoop-3.3.6.tar.gz -C /opt/ mv /opt/hadoop-3.3.6 /opt/hadoop chown -R hduser:hadoop /opt/hadoop4.2 核心配置文件1. hadoop-env.shecho export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 /opt/hadoop/etc/hadoop/hadoop-env.sh echo export HADOOP_HOME/opt/hadoop /opt/hadoop/etc/hadoop/hadoop-env.sh2. core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/opt/hadoop/data/tmp/value /property /configuration3. hdfs-site.xmlconfiguration property namedfs.replication/name value3/value /property property namedfs.namenode.name.dir/name value/opt/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name value/opt/hadoop/data/datanode/value /property /configuration4. mapred-site.xmlconfiguration property namemapreduce.framework.name/name valueyarn/value /property /configuration5. yarn-site.xmlconfiguration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.resourcemanager.hostname/name valuemaster/value /property /configuration6. workers文件slave1 slave2 slave3 slave44.3 分发Hadoop到所有节点for node in slave1 slave2 slave3 slave4; do rsync -av /opt/hadoop hduser$node:/opt/ ssh hduser$node sudo chown -R hduser:hadoop /opt/hadoop done4.4 格式化HDFS并启动集群# 格式化NameNode仅在首次部署时执行 hdfs namenode -format # 启动HDFS start-dfs.sh # 启动YARN start-yarn.sh # 验证服务 jps5. Spark集群部署5.1 下载与安装wget https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3.tgz tar -xzvf spark-3.5.2-bin-hadoop3.tgz -C /opt/ mv /opt/spark-3.5.2-bin-hadoop3 /opt/spark chown -R hduser:hadoop /opt/spark5.2 核心配置文件1. spark-env.shcp /opt/spark/conf/spark-env.sh.template /opt/spark/conf/spark-env.sh echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 /opt/spark/conf/spark-env.sh echo export HADOOP_CONF_DIR/opt/hadoop/etc/hadoop /opt/spark/conf/spark-env.sh echo export SPARK_MASTER_HOSTmaster /opt/spark/conf/spark-env.sh2. workers文件slave1 slave2 slave3 slave45.3 分发Spark到所有节点for node in slave1 slave2 slave3 slave4; do rsync -av /opt/spark hduser$node:/opt/ ssh hduser$node sudo chown -R hduser:hadoop /opt/spark done5.4 启动Spark集群/opt/spark/sbin/start-all.sh # 验证Spark UI # 访问 http://master:80806. 集群验证与测试6.1 Hadoop功能测试# 创建HDFS目录 hdfs dfs -mkdir -p /user/hduser # 上传测试文件 hdfs dfs -put /opt/hadoop/LICENSE.txt /user/hduser/ # 运行WordCount示例 hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/hduser/LICENSE.txt /user/hduser/output # 查看结果 hdfs dfs -cat /user/hduser/output/*6.2 Spark功能测试# 启动PySpark交互式环境 pyspark # 执行简单计算 rdd sc.parallelize(range(1000)) print(rdd.count()) # 运行Spark Pi示例 /opt/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn /opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar 1007. 性能调优与监控7.1 Hadoop参数优化yarn-site.xml新增配置property nameyarn.nodemanager.resource.memory-mb/name value24576/value !-- 根据实际内存调整 -- /property property nameyarn.scheduler.maximum-allocation-mb/name value8192/value /property7.2 Spark参数优化spark-defaults.confspark.master yarn spark.driver.memory 4g spark.executor.memory 8g spark.executor.cores 4 spark.dynamicAllocation.enabled true spark.shuffle.service.enabled true7.3 监控工具部署1. Hadoop监控# 启用YARN Timeline Server echo export YARN_TIMELINESERVER_HEAPSIZE2048 /opt/hadoop/etc/hadoop/yarn-env.sh yarn --daemon start timelineserver2. Spark监控# 启用Spark History Server mkdir /opt/spark/history echo export SPARK_HISTORY_OPTS-Dspark.history.fs.logDirectoryhdfs://master:9000/spark-logs /opt/spark/conf/spark-env.sh hdfs dfs -mkdir -p /spark-logs /opt/spark/sbin/start-history-server.sh8. 安全配置与维护8.1 防火墙规则sudo ufw allow 22 sudo ufw allow 8080/tcp # Spark UI sudo ufw allow 8088/tcp # YARN ResourceManager sudo ufw allow 9870/tcp # HDFS NameNode sudo ufw enable8.2 定期维护脚本日志清理脚本#!/bin/bash # 清理30天前的Hadoop日志 find /opt/hadoop/logs -name *.log* -mtime 30 -exec rm {} \; # 清理Spark事件日志 hdfs dfs -rm -r /spark-logs/app-*$(date -d 30 days ago %Y%m%d)*8.3 备份策略NameNode元数据备份# 创建每日备份 tar -czf /backup/hadoop-nn-metadata-$(date %Y%m%d).tar.gz /opt/hadoop/data/namenode