1. 项目背景与核心挑战在分布式系统中etcd作为Kubernetes的核心数据存储组件其稳定性和扩展性直接影响整个集群的可靠性。当业务规模扩大时原有的单节点或三节点etcd集群可能面临性能瓶颈。将etcd从单节点扩容至双节点虽然不能形成法定多数quorum但作为过渡方案或特定场景下的容灾配置仍具有实用价值。注意生产环境强烈建议至少3节点集群。双节点仅适用于测试环境或特殊容灾场景因为网络分区时可能出现脑裂问题。2. 扩容前准备2.1 环境检查清单现有etcd版本etcd --version证书有效期openssl x509 -in /opt/etcd/ssl/server.pem -noout -dates防火墙规则确保2379客户端、2380节点间端口双向开放磁盘空间df -h检查数据目录剩余空间2.2 证书体系更新双节点扩容需要重新生成包含新节点信息的证书# 生成CA配置 cat ca-config.json EOF { signing: { default: { expiry: 87600h }, profiles: { www: { expiry: 87600h, usages: [ signing, key encipherment, server auth, client auth ] } } } } EOF # 生成CSR配置注意hosts字段包含新旧节点IP cat server-csr.json EOF { CN: etcd, hosts: [ 192.168.1.10, 192.168.1.11, localhost ], key: { algo: rsa, size: 2048 }, names: [ { C: CN, ST: Beijing, L: Beijing, O: etcd, OU: System } ] } EOF # 使用cfssl生成证书 cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cfssl gencert -caca.pem -ca-keyca-key.pem \ -configca-config.json -profilewww server-csr.json | cfssljson -bare server3. 节点扩容实操步骤3.1 现有节点配置更新备份原证书mkdir -p /opt/etcd/ssl_backup cp /opt/etcd/ssl/* /opt/etcd/ssl_backup/部署新证书cp ca.pem ca-key.pem server.pem server-key.pem /opt/etcd/ssl/ chmod 600 /opt/etcd/ssl/*-key.pem chown etcd:etcd /opt/etcd/ssl/*.pem修改etcd服务配置示例为systemd# /usr/lib/systemd/system/etcd.service [Unit] DescriptionEtcd Server Afternetwork.target [Service] Typenotify Useretcd EnvironmentFile-/etc/etcd/etcd.conf ExecStart/opt/etcd/bin/etcd \ --nameetcd-1 \ --cert-file/opt/etcd/ssl/server.pem \ --key-file/opt/etcd/ssl/server-key.pem \ --peer-cert-file/opt/etcd/ssl/server.pem \ --peer-key-file/opt/etcd/ssl/server-key.pem \ --trusted-ca-file/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file/opt/etcd/ssl/ca.pem \ --initial-advertise-peer-urlshttps://192.168.1.10:2380 \ --listen-peer-urlshttps://0.0.0.0:2380 \ --listen-client-urlshttps://0.0.0.0:2379,http://127.0.0.1:2379 \ --advertise-client-urlshttps://192.168.1.10:2379 \ --initial-cluster-tokenetcd-cluster \ --initial-clusteretcd-1https://192.168.1.10:2380,etcd-2https://192.168.1.11:2380 \ --initial-cluster-stateexisting \ --data-dir/var/lib/etcd Restarton-failure LimitNOFILE65536 [Install] WantedBymulti-user.target重启服务并验证systemctl daemon-reload systemctl restart etcd etcdctl --endpointshttps://192.168.1.10:2379 \ --cacert/opt/etcd/ssl/ca.pem \ --cert/opt/etcd/ssl/server.pem \ --key/opt/etcd/ssl/server-key.pem \ endpoint health3.2 新节点部署同步必要文件# 从现有节点复制 scp -r /opt/etcd root192.168.1.11:/opt/ scp /usr/lib/systemd/system/etcd.service root192.168.1.11:/usr/lib/systemd/system/修改新节点配置# 关键修改项 --nameetcd-2 --initial-advertise-peer-urlshttps://192.168.1.11:2380 --advertise-client-urlshttps://192.168.1.11:2379启动新节点mkdir -p /var/lib/etcd chown -R etcd:etcd /var/lib/etcd /opt/etcd systemctl daemon-reload systemctl enable --now etcd4. 集群验证与调优4.1 健康检查# 检查集群成员 etcdctl --write-outtable \ --endpointshttps://192.168.1.10:2379,https://192.168.1.11:2379 \ --cacert/opt/etcd/ssl/ca.pem \ member list # 检查leader状态 etcdctl --endpointshttps://192.168.1.10:2379 \ endpoint status -w table4.2 性能调优参数在/etc/etcd/etcd.conf中添加# 提高网络吞吐 heartbeat-interval: 100 election-timeout: 500 # 优化存储性能 auto-compaction-mode: periodic auto-compaction-retention: 1h # 内存分配建议 max-request-bytes: 33554432 # 32MB quota-backend-bytes: 8589934592 # 8GB5. 常见问题排查5.1 证书错误症状x509: certificate is valid for 192.168.1.10, not 192.168.1.11解决方案确认server-csr.json包含所有节点IP重新生成证书后同步到所有节点按顺序重启节点先follower后leader5.2 集群不健康症状failed to check member health排查步骤# 检查节点日志 journalctl -u etcd -n 50 --no-pager # 测试基础网络 nc -zv 192.168.1.11 2380 tcping 192.168.1.11 2380 # 临时调低日志级别调试 ETCD_LOG_LEVELdebug etcd --config-file/etc/etcd/etcd.conf5.3 数据不一致处理方案# 先备份数据 etcdctl snapshot save backup.db # 从健康节点恢复数据 ETCDCTL_API3 etcdctl snapshot restore backup.db \ --initial-clusteretcd-1https://192.168.1.10:2380,etcd-2https://192.168.1.11:2380 \ --initial-advertise-peer-urlshttps://192.168.1.10:2380 \ --nameetcd-16. 后续维护建议监控指标配置关键指标etcd_server_leader_changes_seen_total、etcd_disk_wal_fsync_duration_secondsPrometheus示例配置- job_name: etcd static_configs: - targets: [192.168.1.10:2379,192.168.1.11:2379] scheme: https tls_config: ca_file: /path/to/ca.pem cert_file: /path/to/client.pem key_file: /path/to/client-key.pem insecure_skip_verify: false证书自动续期方案# 使用cert-manager示例配置 apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: etcd-server spec: secretName: etcd-server-tls duration: 2160h # 90d renewBefore: 360h # 15d issuerRef: name: ca-issuer kind: ClusterIssuer commonName: etcd dnsNames: - localhost ipAddresses: - 192.168.1.10 - 192.168.1.11 usages: - server auth - client auth升级为三节点的建议步骤准备第三个节点的证书提前加入hosts字段修改现有节点的initial-cluster参数按相同流程部署第三个节点更新所有节点的--initial-cluster-state为existing后重启