此版本的NFS为单点,仅为练习使用,生产环境建议使用cephfs的卷类型,避免单点。或者通过keepalived加Sersync的方案对NFS作容灾处理即可用于生产环境。当然,对于开发或测试环境,方便起见,直接使用单点的NFS加mongodb statefulSet方案是最为清晰简便的。
mongodb集群部署分两种情况,一是只部署副本机制,不使用分片,另一种情况是使用分片集群。使用分片的情况下会略复杂,但基本部署方法和只部署副本的方法是差不多的,只不过是多了一些角色。这里只介绍部署副本集群到k8s集群。
部署前提:一个高可用k8s集群和NFS storage class。
部署服务器规划
主机名 | 业务概述 |
---|---|
k8s-register-node | harbor私服 |
lb-node-1 | nginx负载、helm |
lb-node-2 | nginx负载 |
k8s-master-1 | k8s master(controlplane,worker,etcd) |
k8s-master-2 | k8s master(controlplane,worker,etcd) |
k8s-master-3 | k8s master(controlplane,worker,etcd) |
k8s-storage-3 | NFS-Server |
部署一主一从一仲裁的集群,可以看出,因为NFS服务只有一个,因此这里是存在单点故障的,除此单点外,mongodb本身的服务具有高可用,副本机制如下图:
如上图,共部署三个副本,其中有一个仲裁节点,此节点并不保存数据。主节点负责读写,从节点通过Raft协议同步数据, 可作读节点。
准备keyFile
mongodb副本集必须有3个以上成员,且成员个数必须为奇数。主备节点存储数据,仲裁节点不存储数据。客户端同时连接主节点与备节点,不连接仲裁节点。 仲裁节点是一种特殊的节点,它本身并不存储数据,主要的作用是决定哪一个备节点在主节点挂掉之后提升为主节点,所以客户端不需要连接此节点。
每个服务器都必须有keyFile文件,且该keyFile文件必须相同,各服务器才能正常交互。
openssl rand -base64 756 > /home/kmning/mongodb/mongodb.key
statefulSet定义
PV我们直接使用NFS storage class的动态制备,因此我们定义好statefulset和Service就可以把mongodb集群安装好,最后手动初始化集群即可。
下面的kefile内容即是上面生成的mongodb.key对应的内容。
mongodb-statefulset.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:name: mongodb-rs-cm
data:keyfile: |上面生成的keyfile内容mongod_rs.conf: |+systemLog:destination: filelogAppend: truepath: /data/mongod.logstorage:dbPath: /datajournal:enabled: truedirectoryPerDB: truewiredTiger:engineConfig:cacheSizeGB: 4directoryForIndexes: trueprocessManagement:fork: truetimeZoneInfo: /usr/share/zoneinfopidFilePath: /data/mongod.pidnet:port: 27017bindIp: 0.0.0.0maxIncomingConnections: 5000security:keyFile: /data/configdb/keyfileauthorization: enabledreplication:oplogSizeMB: 5120replSetName: xxxReplSet
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: mongodb-rs
spec:serviceName: mongodb-rsreplicas: 3selector:matchLabels:app: mongodb-rstemplate:metadata:labels:app: mongodb-rsspec:containers:- name: mongoimage: xxx.com:443/mongo:4.4.10ports:- containerPort: 27017name: mongo-pod-portcommand: ["sh"]args:- "-c"- |set -exmongod --config /data/configdb/mongod_rs.confsleep infinity env:- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIPvolumeMounts:- name: confmountPath: /data/configdbreadOnly: false- name: datamountPath: /datareadOnly: falsevolumes:- name: confconfigMap:name: mongodb-rs-cmdefaultMode: 0600volumeClaimTemplates:- metadata:name: dataspec:accessModes: ["ReadWriteOnce"]resources:requests:storage: 300GistorageClassName: managed-nfs-storage
configmap也一同建立,这里以默认的namespace进行的部署,实际情况因选择自己的namespace。 这里要注意,keyfile权限设置600否则无法启动mongod。 fork如果配置true的话,在启动命令后我写了sleep infinity,否则容器立马会退出。
注意上面,configmap相当于定义了一个keyfile文件和一个mongod_rs.conf配置文件,在mongod_rs.conf配置文件中直接使用keyfile,然后在StatefulSet中对副本模板定义时直接挂载了mongod_rs.conf中定义的数据目录和配置文件目录,这样一来k8s就可以利用这些配置文件和副本状态集模板定义构建StatefulSet,而存储的PV则交给了NFS的storage Class。
注意上面的keyfile内容定义,以一行的形式保存,不能有tab。
安装上面的statefuleSet
kmning@k8s-master-1:~/mongodb$ kubectl create -f mongodb-statefulset.yaml
configmap/mongodb-rs-cm created
statefulset.apps/mongodb-rs created
查看状态
kmning@k8s-master-1:~/mongodb$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mongodb-rs-0 1/1 Running 0 4m26s
mongodb-rs-1 1/1 Running 0 4m1s
mongodb-rs-2 1/1 Running 0 3m35s
可见,三个pod正在有序创建完成,再查看一下PV即可发现,三块PV和PVC由NFS的storage class给我们动态制备了。
kmning@k8s-master-1:~/mongodb$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-14d12757-c32e-4e26-857e-7e6fb22c637f 10Gi RWX Delete Bound default/data-nacos-0 managed-nfs-storage 6d6h
pvc-50a5d0e7-0ae1-41a3-ae43-29c8279da460 300Gi RWO Delete Bound default/data-mongodb-rs-2 managed-nfs-storage 3m6s
pvc-a213fa79-bf84-465d-b7ee-fe9a2713d329 300Gi RWO Delete Bound default/data-mongodb-rs-0 managed-nfs-storage 3m57s
pvc-b9629010-ce4b-4c04-83fb-ee2feae32040 300Gi RWO Delete Bound default/data-mongodb-rs-1 managed-nfs-storage 3m31s
pvc-bb1beed7-5303-4fea-9bc1-4b1c3f762b18 10Gi RWX Delete Bound default/data-nacos-2 managed-nfs-storage 6d6h
pvc-cb9de849-bfaa-4c42-9328-c90831336c68 10Gi RWX Delete Bound default/data-nacos-1 managed-nfs-storage 6d6h
kmning@k8s-master-1:~/mongodb$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-mongodb-rs-0 Bound pvc-a213fa79-bf84-465d-b7ee-fe9a2713d329 300Gi RWO managed-nfs-storage 3m54s
data-mongodb-rs-1 Bound pvc-b9629010-ce4b-4c04-83fb-ee2feae32040 300Gi RWO managed-nfs-storage 3m28s
data-mongodb-rs-2 Bound pvc-50a5d0e7-0ae1-41a3-ae43-29c8279da460 300Gi RWO managed-nfs-storage 3m2s
data-nacos-0 Bound pvc-14d12757-c32e-4e26-857e-7e6fb22c637f 10Gi RWX managed-nfs-storage 6d6h
data-nacos-1 Bound pvc-cb9de849-bfaa-4c42-9328-c90831336c68 10Gi RWX managed-nfs-storage 6d6h
data-nacos-2 Bound pvc-bb1beed7-5303-4fea-9bc1-4b1c3f762b18 10Gi RWX managed-nfs-storage 6d6h
Service暴露服务
上面已经成功创建了mongodb的statefulset对应的三个Pod,但目前外部尚不可访问,先创建一个ClusterIP类型的Service,用以达到被其他pod访问的目的。
mongod-svc.yml
apiVersion: v1
kind: Service
metadata:name: mongodb-rsnamespace: default
spec:ports:- port: 27017protocol: TCPtargetPort: mongo-pod-portselector:app: mongodb-rstype: ClusterIP
注意上面的service name要和StatefulSet中定义的serviceName对应起来。另外 ,targetPort我使用了mongodb定义的port的名称,这样一来,即使后端的mongodb端口更换了,Service也不用修改。
创建
euht@k8s-master-1:~/mongodb$ kubectl create -f mongod-svc.yml
service/mongodb-rs created
创建后k8s就给这个Service分配了集群IP,查看如
kmning@k8s-master-1:~/mongodb$ kubectl get svc mongodb-rs -o yaml
apiVersion: v1
kind: Service
metadata:creationTimestamp: "2023-04-27T02:20:58Z"name: mongodb-rsnamespace: defaultresourceVersion: "3746349"uid: 758812a2-9d5c-457f-ab91-264fafbf9998
spec:clusterIP: 10.43.149.68clusterIPs:- 10.43.149.68internalTrafficPolicy: ClusteripFamilies:- IPv4ipFamilyPolicy: SingleStackports:- port: 27017protocol: TCPtargetPort: mongo-pod-portselector:app: mongodb-rssessionAffinity: Nonetype: ClusterIP
status:loadBalancer: {}
然后,在k8s集群内,我们都可以利用集群IP 10.43.232.149去访问到后端mongodb的pod。
kmning@k8s-master-1:~/mongodb$ kubectl describe svc mongodb-rs
Name: mongodb-rs
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=mongodb-rs
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.149.68
IPs: 10.43.149.68
Port: <unset> 27017/TCP
TargetPort: mongo-pod-port/TCP
Endpoints: 10.42.0.33:27017,10.42.2.36:27017,10.42.4.13:27017
Session Affinity: None
Events: <none>
可以看到,这个集群IP后面拥有三个Endpoints,对应三个mongodb pod。
现在我们可以通过域名的方式来设置replica set,也不用关心pod会变化的ip地址了。 域名为:$(podname).$(service name).$(namespace).svc.cluster.local
可以进行一次验证,mongodb-rs-0通过域名连接mongodb-rs-2
kmning@k8s-master-1:~/mongodb$ kubectl exec -it mongodb-rs-0 -- mongo mongodb-rs-2.mongodb-rs.default.svc.cluster.local
MongoDB shell version v4.4.10
connecting to: mongodb://mongodb-rs-2.mongodb-rs.default.svc.cluster.local:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("64c8e67e-50ca-480a-ab5d-473b291239b2") }
MongoDB server version: 4.4.10
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, seehttps://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forumshttps://community.mongodb.com
>
可以看出,正常连接。事实上通过域名连接,k8s会先通过dns服务查找到集群IP,然后通过Service找到Pod对应的IP,然后再进行连接。后续mongodb的pod实例怎么变,此域名都不会变,这样一来用域名设置的有状态配置就不会因为pod的IP改变而变得不可用。
集群初始化
上面已经搭建好mongodb的各个副本,但它们之间尚未有联系,随便登录一个副本进行初始化以建立副本集。
可以登录任何的一个副本进行设置,注意host字段要使用服务域名,因为statefulset的pod的域名是稳定不会变的。
kubectl exec -it mongodb-rs-0 -- mongo
初始化
use admin;
rs.initiate({_id: "yourReplSet",members: [{ _id : 0, host : "mongodb-rs-0.mongodb-rs.default.svc.cluster.local:27017",priority: 50 },{ _id : 1, host : "mongodb-rs-1.mongodb-rs.default.svc.cluster.local:27017" ,priority: 60},{ _id : 2, host : "mongodb-rs-2.mongodb-rs.default.svc.cluster.local:27017",arbiterOnly: true }]}
)
查看状态
rs.status();
yourReplSet:SECONDARY> rs.status();
{"set" : "yourReplSet","date" : ISODate("2023-04-26T10:13:56.173Z"),"myState" : 2,"term" : NumberLong(2),"syncSourceHost" : "mongodb-rs-1.mongodb-rs.default.svc.cluster.local:27017","syncSourceId" : 1,"heartbeatIntervalMillis" : NumberLong(2000),"majorityVoteCount" : 2,"writeMajorityCount" : 2,"votingMembersCount" : 3,"writableVotingMembersCount" : 2,"optimes" : {"lastCommittedOpTime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"lastCommittedWallTime" : ISODate("2023-04-26T10:13:39.160Z"),"readConcernMajorityOpTime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"readConcernMajorityWallTime" : ISODate("2023-04-26T10:13:39.160Z"),"appliedOpTime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"durableOpTime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"lastAppliedWallTime" : ISODate("2023-04-26T10:13:39.160Z"),"lastDurableWallTime" : ISODate("2023-04-26T10:13:39.160Z")},"lastStableRecoveryTimestamp" : Timestamp(1682503991, 4),"electionParticipantMetrics" : {"votedForCandidate" : true,"electionTerm" : NumberLong(2),"lastVoteDate" : ISODate("2023-04-26T10:13:22.875Z"),"electionCandidateMemberId" : 1,"voteReason" : "","lastAppliedOpTimeAtElection" : {"ts" : Timestamp(1682503992, 1),"t" : NumberLong(1)},"maxAppliedOpTimeInSet" : {"ts" : Timestamp(1682503992, 1),"t" : NumberLong(1)},"priorityAtElection" : 50,"newTermStartDate" : ISODate("2023-04-26T10:13:09.158Z"),"newTermAppliedDate" : ISODate("2023-04-26T10:13:23.890Z")},"members" : [{"_id" : 0,"name" : "mongodb-rs-0.mongodb-rs.default.svc.cluster.local:27017","health" : 1,"state" : 2,"stateStr" : "SECONDARY","uptime" : 3246,"optime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"optimeDate" : ISODate("2023-04-26T10:13:39Z"),"syncSourceHost" : "mongodb-rs-1.mongodb-rs.default.svc.cluster.local:27017","syncSourceId" : 1,"infoMessage" : "","configVersion" : 1,"configTerm" : 2,"self" : true,"lastHeartbeatMessage" : ""},{"_id" : 1,"name" : "mongodb-rs-1.mongodb-rs.default.svc.cluster.local:27017","health" : 1,"state" : 1,"stateStr" : "PRIMARY","uptime" : 55,"optime" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"optimeDurable" : {"ts" : Timestamp(1682504019, 1),"t" : NumberLong(2)},"optimeDate" : ISODate("2023-04-26T10:13:39Z"),"optimeDurableDate" : ISODate("2023-04-26T10:13:39Z"),"lastHeartbeat" : ISODate("2023-04-26T10:13:55.921Z"),"lastHeartbeatRecv" : ISODate("2023-04-26T10:13:54.904Z"),"pingMs" : NumberLong(0),"lastHeartbeatMessage" : "","syncSourceHost" : "","syncSourceId" : -1,"infoMessage" : "","electionTime" : Timestamp(1682503992, 2),"electionDate" : ISODate("2023-04-26T10:13:12Z"),"configVersion" : 1,"configTerm" : 2},{"_id" : 2,"name" : "mongodb-rs-2.mongodb-rs.default.svc.cluster.local:27017","health" : 1,"state" : 7,"stateStr" : "ARBITER","uptime" : 55,"lastHeartbeat" : ISODate("2023-04-26T10:13:55.922Z"),"lastHeartbeatRecv" : ISODate("2023-04-26T10:13:54.932Z"),"pingMs" : NumberLong(0),"lastHeartbeatMessage" : "","syncSourceHost" : "","syncSourceId" : -1,"infoMessage" : "","configVersion" : 1,"configTerm" : 2}],"ok" : 1,"$clusterTime" : {"clusterTime" : Timestamp(1682504019, 1),"signature" : {"hash" : BinData(0,"KM4OLkjcMxpk7RqlTC2omtRQ0pY="),"keyId" : NumberLong("7226299616734478340")}},"operationTime" : Timestamp(1682504019, 1)
}
可见,副本集群已成功在k8s上运行。