Spark 3.5.1 与 Hadoop 3.3.6 集群部署3节点 Standalone 模式配置详解在当今数据驱动的商业环境中构建高效可靠的大数据处理平台已成为企业数字化转型的核心需求。作为业界领先的分布式计算框架组合Spark与Hadoop的协同部署能够为企业级数据分析提供强大的计算能力和存储支持。本文将深入探讨如何在生产级环境中部署三节点Spark Standalone集群与Hadoop 3.3.6的集成方案涵盖从基础环境准备到高级调优的全流程实践。1. 集群规划与环境准备1.1 硬件资源配置建议对于生产环境的三节点集群建议采用以下硬件配置作为基准节点角色CPU核心数内存容量存储空间网络带宽Master节点8核32GB500GB10GbpsWorker节点116核64GB2TB10GbpsWorker节点216核64GB2TB10Gbps实际配置应根据业务负载特点进行调整CPU密集型作业适当增加每节点核心数内存密集型作业扩展内存容量并配置合理的off-heap内存IO密集型作业考虑使用SSD或NVMe存储1.2 系统环境配置所有节点需确保一致的运行环境# 检查并安装基础依赖 sudo yum install -y epel-release sudo yum install -y java-11-openjdk-devel openssl ntp pdsh sshpass # 配置主机名解析所有节点执行 cat EOF | sudo tee -a /etc/hosts 192.168.1.101 spark-master 192.168.1.102 spark-worker1 192.168.1.103 spark-worker2 EOF # 配置SSH免密登录在master节点执行 ssh-keygen -t rsa -P -f ~/.ssh/id_rsa ssh-copy-id spark-master ssh-copy-id spark-worker1 ssh-copy-id spark-worker2 # 验证时钟同步 sudo systemctl enable ntpd sudo systemctl start ntpd ntpq -p1.3 软件版本选择本次部署选用经过充分验证的稳定版本组合Spark 3.5.12024年发布的最新稳定版包含AQE自适应查询执行优化Hadoop 3.3.6支持EC擦除编码和S3A连接器增强OpenJDK 11LTS版本提供长期支持下载地址wget https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz2. Hadoop集群部署与配置2.1 基础安装步骤在所有节点执行以下安装流程# 解压Hadoop安装包 tar -xzf hadoop-3.3.6.tar.gz -C /opt ln -s /opt/hadoop-3.3.6 /opt/hadoop # 配置环境变量 cat EOF | sudo tee /etc/profile.d/hadoop.sh export HADOOP_HOME/opt/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_LOG_DIR/var/log/hadoop EOF source /etc/profile.d/hadoop.sh # 创建必要目录 sudo mkdir -p /var/log/hadoop sudo chown -R $(whoami):$(whoami) /var/log/hadoop mkdir -p $HADOOP_HOME/data/{namenode,datanode}2.2 核心配置文件优化编辑$HADOOP_CONF_DIR下的关键配置文件core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://spark-master:8020/value /property property namehadoop.tmp.dir/name value/opt/hadoop/data/tmp/value /property property nameio.file.buffer.size/name value131072/value /property /configurationhdfs-site.xmlconfiguration property namedfs.replication/name value2/value /property property namedfs.namenode.name.dir/name valuefile:///opt/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name valuefile:///opt/hadoop/data/datanode/value /property property namedfs.blocksize/name value256m/value /property property namedfs.namenode.handler.count/name value100/value /property /configurationworkers文件spark-worker1 spark-worker22.3 集群初始化与启动在Master节点执行以下命令# 格式化HDFS首次部署执行 hdfs namenode -format # 启动HDFS集群 start-dfs.sh # 验证服务状态 hdfs dfsadmin -report预期输出应显示两个活跃的DataNode节点。通过Web UI可访问NameNode状态页面http://spark-master:98703. Spark Standalone集群部署3.1 Spark安装与基础配置在所有节点执行以下步骤# 解压Spark安装包 tar -xzf spark-3.5.1-bin-hadoop3.tgz -C /opt ln -s /opt/spark-3.5.1-bin-hadoop3 /opt/spark # 配置环境变量 cat EOF | sudo tee /etc/profile.d/spark.sh export SPARK_HOME/opt/spark export PATH$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin export PYSPARK_PYTHON/usr/bin/python3 EOF source /etc/profile.d/spark.sh3.2 集群配置文件定制spark-env.shMaster节点配置cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh cat EOF $SPARK_HOME/conf/spark-env.sh export SPARK_MASTER_HOSTspark-master export SPARK_MASTER_PORT7077 export SPARK_WORKER_CORES16 export SPARK_WORKER_MEMORY48G export SPARK_WORKER_INSTANCES1 export SPARK_DAEMON_MEMORY4G export SPARK_LOCAL_DIRS/opt/spark/work export SPARK_LOG_DIR/var/log/spark export SPARK_PID_DIR/var/run/spark export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop EOFworkers文件spark-worker1 spark-worker23.3 集群服务管理在Master节点启动集群# 启动Spark集群 $SPARK_HOME/sbin/start-all.sh # 验证节点状态 $SPARK_HOME/bin/spark-shell --master spark://spark-master:7077通过Web UI可查看集群状态http://spark-master:80804. 高级配置与调优4.1 网络与安全配置SSL加密配置# 生成密钥库 keytool -keystore spark-keystore.jks -alias spark -validity 365 -genkey -keyalg RSA # 配置spark-defaults.conf spark.ssl.enabled true spark.ssl.keyPassword yourpassword spark.ssl.keyStore /path/to/spark-keystore.jks spark.ssl.keyStorePassword yourpassword spark.ssl.trustStore /path/to/spark-truststore.jks spark.ssl.trustStorePassword yourpassword4.2 资源调度优化动态资源分配配置spark.dynamicAllocation.enabledtrue spark.dynamicAllocation.initialExecutors2 spark.dynamicAllocation.minExecutors2 spark.dynamicAllocation.maxExecutors10 spark.shuffle.service.enabledtrue4.3 监控与日志聚合Prometheus监控集成# 配置metrics.properties *.sink.prometheusServlet.classorg.apache.spark.metrics.sink.PrometheusServlet *.sink.prometheusServlet.path/metrics/prometheus master.sink.prometheusServlet.port4040 worker.sink.prometheusServlet.port4041日志聚合配置!-- 在spark-defaults.conf中配置 -- spark.eventLog.enabled true spark.eventLog.dir hdfs://spark-master:8020/spark-logs spark.history.fs.logDirectory hdfs://spark-master:8020/spark-logs启动历史服务器$SPARK_HOME/sbin/start-history-server.sh5. 集群验证与维护5.1 健康检查脚本创建cluster_health_check.sh脚本#!/bin/bash # 检查HDFS状态 hdfs dfsadmin -report | grep -i Live datanodes # 检查Spark Worker状态 curl -s http://spark-master:8080 | grep -A5 Workers # 检查资源使用情况 pdsh -w spark-worker1,spark-worker2 free -h df -h # 检查服务进程 pdsh -w spark-master,spark-worker1,spark-worker2 jps5.2 常见问题排查指南Worker节点无法注册检查防火墙设置sudo firewall-cmd --list-ports验证网络连通性pdsh -w spark-worker1,spark-worker2 ping -c 3 spark-master检查Worker日志tail -100 /var/log/spark/spark-*-worker-*.log任务执行失败# 检查Executor日志 hdfs dfs -ls /spark-logs/*/executors # 分析GC情况 $SPARK_HOME/bin/spark-submit --conf spark.executor.extraJavaOptions-XX:PrintGCDetails5.3 版本升级策略滚动升级步骤逐个停止Worker节点更新软件包并验证配置重新加入集群最后升级Master节点回退方案保留旧版本安装目录维护配置版本控制准备快速回退脚本