1. 项目概述在当今云原生技术蓬勃发展的时代Kubernetes已经成为容器编排领域的事实标准。对于企业级生产环境而言高可用性High Availability不是可选项而是必选项。Rocky Linux作为RHEL的完美替代品以其稳定性和长期支持特性成为企业级基础设施的首选操作系统之一。我最近为一个中型金融科技公司完成了基于Rocky Linux的多Master节点Kubernetes高可用集群部署项目。这个方案不仅通过了严格的压力测试还在实际生产环境中稳定运行了6个月无故障。本文将详细分享从系统准备到集群调优的全套实施方案特别适合需要构建生产级Kubernetes集群的运维团队参考。2. 环境准备与系统配置2.1 硬件规划建议对于生产级Kubernetes集群硬件资源配置需要根据实际负载精心规划。在我们的实施案例中采用了以下配置Master节点3台必须奇数台满足etcd选举要求CPU: 8核以上内存: 32GB以上存储: 100GB系统盘 200GB数据盘用于etcd网络: 双万兆网卡bonding模式4Worker节点根据业务需求动态扩展CPU: 16核以上内存: 64GB以上存储: 100GB系统盘 根据容器需求配置额外存储重要提示Master节点必须使用物理服务器虚拟化环境可能导致性能问题和选举异常。我们在测试阶段曾尝试在VMware上部署结果在节点故障模拟时出现了etcd脑裂情况。2.2 Rocky Linux基础配置所有节点统一安装Rocky Linux 8.6当前最新LTS版本并进行以下基础配置# 禁用SELinuxKubernetes兼容性要求 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config setenforce 0 # 关闭防火墙由Kubernetes网络策略管理 systemctl stop firewalld systemctl disable firewalld # 配置时间同步 dnf install chrony -y systemctl enable chronyd systemctl start chronyd chronyc sources # 加载内核模块 cat EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF # 配置内核参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system3. 高可用Kubernetes集群部署3.1 容器运行时安装我们选择containerd作为容器运行时相比Docker更加轻量且CNCF认证# 安装containerd dnf config-manager --add-repohttps://download.docker.com/linux/centos/docker-ce.repo dnf install containerd.io -y # 生成默认配置并启用SystemdCgroup containerd config default /etc/containerd/config.toml sed -i s/SystemdCgroup false/SystemdCgroup true/g /etc/containerd/config.toml # 启动服务 systemctl enable containerd systemctl start containerd3.2 Kubernetes组件安装配置Kubernetes官方yum源并安装必要组件cat EOF /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF dnf install kubelet kubeadm kubectl -y --disableexcludeskubernetes systemctl enable kubelet3.3 高可用控制平面部署多Master高可用集群的核心在于负载均衡器我们选择HAProxyKeepalived方案etcd集群控制平面组件的多实例部署首先在所有Master节点上安装HAProxy和Keepaliveddnf install haproxy keepalived -y配置HAProxy所有Master节点相同配置cat EOF /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice daemon defaults log global timeout connect 5000 timeout client 50000 timeout server 50000 frontend k8s-api bind *:6443 mode tcp default_backend k8s-api backend k8s-api mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2 EOF配置Keepalived注意每个节点的priority和ip要不同# Master1配置 cat EOF /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER priority 100 virtual_router_id 51 advert_int 1 authentication { auth_type PASS auth_pass 42 } virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } } EOF启动服务systemctl enable haproxy keepalived systemctl start haproxy keepalived3.4 使用kubeadm初始化集群在第一个Master节点上执行初始化kubeadm init --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers初始化完成后按照提示在其他Master节点上执行join命令kubeadm join 192.168.1.100:6443 --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane --certificate-key key3.5 网络插件安装我们选择Calico作为网络插件它支持网络策略且性能优异kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml4. 生产环境关键配置与优化4.1 etcd性能调优etcd是Kubernetes集群的大脑需要特别优化# 在每个Master节点上修改etcd环境文件 cat EOF /etc/etcd.env ETCD_HEARTBEAT_INTERVAL100 ETCD_ELECTION_TIMEOUT500 ETCD_SNAPSHOT_COUNT10000 ETCD_MAX_REQUEST_BYTES15728640 ETCD_QUOTA_BACKEND_BYTES8589934592 EOF # 重启etcd systemctl restart etcd4.2 kubelet资源预留防止系统关键进程因资源不足被OOM Killer杀死cat EOF /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS--kube-reservedcpu500m,memory1Gi,ephemeral-storage1Gi \ --system-reservedcpu1000m,memory2Gi,ephemeral-storage1Gi \ --eviction-hardmemory.available500Mi,nodefs.available10% EOF systemctl daemon-reload systemctl restart kubelet4.3 控制平面组件高可用确保API Server、Controller Manager和Scheduler的多实例部署# 检查组件状态 kubectl get pods -n kube-system -l tiercontrol-plane # 手动平衡Leader选举Controller Manager和Scheduler for i in {0..2}; do kubectl patch leases kube-controller-manager -n kube-system \ --type merge \ --patch {\spec\:{\holderIdentity\:\k8s-master-${i}\}} kubectl patch leases kube-scheduler -n kube-system \ --type merge \ --patch {\spec\:{\holderIdentity\:\k8s-master-${i}\}} done5. 集群验证与故障排查5.1 高可用性测试节点故障测试# 随机停止一个Master节点上的kubelet systemctl stop kubelet # 观察集群状态应在30秒内完成故障转移 watch kubectl get nodes网络分区测试# 模拟网络分区 iptables -A INPUT -p tcp --dport 2379:2380 -j DROP # 检查etcd集群健康状态 ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key endpoint health5.2 常见问题排查指南问题现象排查步骤解决方案etcd成员无法加入集群1. 检查网络连通性2. 检查证书有效期3. 检查时钟同步1. 修复网络配置2. 重新生成证书3. 确保NTP服务正常API Server间歇性不可用1. 检查负载均衡器日志2. 检查Master节点资源使用率3. 检查kube-apiserver日志1. 调整HAProxy配置2. 增加Master节点资源3. 检查证书和认证配置Worker节点NotReady1. 检查kubelet服务状态2. 检查容器运行时状态3. 检查网络插件Pod状态1. 重启kubelet2. 重启containerd3. 重新部署Calico6. 生产环境最佳实践6.1 备份与恢复策略etcd定期备份# 创建每日备份脚本 cat EOF /usr/local/bin/etcd-backup.sh #!/bin/bash DATE\$(date %Y%m%d) ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /backup/etcd-snapshot-\$DATE.db EOF # 设置定时任务 echo 0 3 * * * root /usr/local/bin/etcd-backup.sh /etc/cron.d/etcd-backup关键资源备份# 备份所有Kubernetes资源 kubectl get all --all-namespaces -o yaml all-resources-backup.yaml6.2 安全加固措施RBAC最小权限原则# 创建只读权限的ClusterRole kubectl create clusterrole readonly \ --verbget,list,watch \ --resource* # 绑定到开发团队 kubectl create clusterrolebinding dev-team-readonly \ --clusterrolereadonly \ --groupdev-teamPod安全策略# 创建限制性PSP cat EOF | kubectl apply -f - apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - projected - secret - downwardAPI - persistentVolumeClaim hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: MustRunAs ranges: - min: 1 max: 65535 fsGroup: rule: MustRunAs ranges: - min: 1 max: 65535 readOnlyRootFilesystem: false EOF6.3 监控与告警配置部署Prometheus Operatorhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --set grafana.adminPasswordadmin关键告警规则示例# API Server可用性告警 - alert: KubeAPIDown expr: sum(up{jobapiserver}) 1 for: 5m labels: severity: critical annotations: summary: Kubernetes API server down description: API server has been down for more than 5 minutes # etcd写入延迟告警 - alert: HighEtcdWriteLatency expr: histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)) 0.5 for: 10m labels: severity: warning annotations: summary: High etcd write latency description: 99th percentile etcd write latency is high for {{ $labels.instance }}7. 集群维护与升级策略7.1 滚动升级Master节点升级kubeadm工具dnf upgrade kubeadm -y升级第一个Master节点kubeadm upgrade apply v1.24.3升级其他Master节点kubeadm upgrade node升级kubelet和kubectldnf upgrade kubelet kubectl -y systemctl restart kubelet7.2 Worker节点升级策略优雅驱逐Podkubectl drain node-name --ignore-daemonsets --delete-emptydir-data升级节点组件dnf upgrade kubeadm kubelet kubectl -y systemctl restart kubelet重新加入集群kubectl uncordon node-name7.3 证书轮换管理检查证书有效期kubeadm certs check-expiration手动更新证书kubeadm certs renew all systemctl restart kubelet更新kubeconfig文件mv $HOME/.kube/config $HOME/.kube/config.old cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config8. 性能调优实战经验8.1 API Server参数优化修改/etc/kubernetes/manifests/kube-apiserver.yamlspec: containers: - command: - kube-apiserver - --max-requests-inflight3000 - --max-mutating-requests-inflight1000 - --watch-cache-sizessecrets100,configmaps100 - --default-watch-cache-size1008.2 kube-proxy调优切换为ipvs模式并优化参数kubectl edit configmap kube-proxy -n kube-system修改配置mode: ipvs ipvs: minSyncPeriod: 5s syncPeriod: 30s scheduler: rr8.3 容器运行时优化调整containerd配置/etc/containerd/config.toml[plugins.io.containerd.grpc.v1.cri] sandbox_image registry.aliyuncs.com/google_containers/pause:3.6 [plugins.io.containerd.grpc.v1.cri.containerd] snapshotter overlayfs [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc] runtime_type io.containerd.runc.v2 [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options] SystemdCgroup true [plugins.io.containerd.grpc.v1.cri.registry] [plugins.io.containerd.grpc.v1.cri.registry.mirrors] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io] endpoint [https://registry-1.docker.io]9. 真实生产环境踩坑记录时钟漂移导致etcd集群故障现象etcd频繁leader选举日志中出现clock difference警告 解决方案在所有节点部署chrony时间同步配置NTP服务器为内部时间源设置chrony的maxpoll为16提高同步频率IPVS模式下的连接泄漏现象长连接服务出现随机断开 解决方案调整ipvs连接超时参数ipvsadm --set 900 120 300在Service中配置sessionAffinity: ClientIP大内存节点上的kubelet OOM现象节点突然变为NotReady状态 解决方案调整kubelet内存限制echo KUBELET_EXTRA_ARGS--memory-request-override1Gi /etc/sysconfig/kubelet systemctl restart kubelet配置合理的kube-reserved和system-reservedCalico IP分配耗尽现象新Pod无法获取IP地址 解决方案扩展IP池范围calicoctl patch ippool default-ipv4-ippool --patch {spec: {cidr: 10.244.0.0/16}}启用IP回收功能apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: stalePodCleanupDelay: 30m10. 扩展与未来演进多集群管理方案部署Kubefed实现多集群联邦配置集群间网络等连接实现跨集群服务发现服务网格集成评估Istio vs Linkerd性能开销渐进式部署策略关键指标监控方案GPU资源调度优化NVIDIA设备插件配置多GPU卡拓扑感知调度共享GPU内存分配策略边缘计算扩展KubeEdge架构评估边缘节点自治策略离线场景下的数据同步方案在实际部署过程中我们发现Rocky Linux作为Kubernetes基础操作系统表现出色特别是在安全更新和内核稳定性方面。一个特别有用的技巧是在部署完成后使用kube-bench工具进行CIS基准测试可以快速发现安全配置问题。我们团队已经将这套部署方案标准化后续所有新集群都采用这种架构运维效率提升了60%以上。