statefulset 容器 IP 地址的变化不可控, 所以证书必须实时生成, 当前的做法是将一个自认证的证书打包进 etcd 镜像, 起 pod 时自己生成, 而访问 etcd 所需的证书也需要从容器中获取.
当前三台机器都只对某一块 nas 盘有访问权限, 所以每个 pod 的落点都是固定的, 如 etcd-0 对应 100.68.34.8, 若有偏差只能手动调整.
etcd 参数只使用了最基本的参数, docker 打包镜像的代码和部署文件已添加在附件
环境在
- [root@ecam40931 etcd]# kubectl get pod -o wide|grep etcd
- etcd-0 1/1 Running 0 16m 172.1.50.2 100.68.34.8
- etcd-1 1/1 Running 0 16m 172.1.34.2 100.68.34.9
- etcd-2 1/1 Running 0 16m 172.1.95.2 100.68.34.10
当前已经可使用加密方式正常访问
- [root@ecam40931 etcd]# kubectl exec -it etcd-0 -- sh
- / # etcdctl --ca-file /etc/etcd/ssl/ca.pem --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --endpoints=https://172.1.34.2:2379 cluster-health
- member 1293bb6c66f7bfa1 is healthy: got healthy result from https://172.1.34.2:2379
- member 5fefc8eefc1469cb is healthy: got healthy result from https://172.1.50.2:2379
- member e38762190fc12c09 is healthy: got healthy result from https://172.1.95.2:2379
- cluster is healthy
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv0001
- spec:
- capacity:
- storage: 100Gi
- accessModes:
- - ReadWriteOnce
- persistentVolumeReclaimPolicy: Recycle
- storageClassName: nas-etcd
- nfs:
- path: /csp_csmp_id100020_vol1004_prd
- server: 100.68.21.4
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv0002
- spec:
- capacity:
- storage: 100Gi
- accessModes:
- - ReadWriteOnce
- persistentVolumeReclaimPolicy: Recycle
- storageClassName: nas-etcd
- nfs:
- path: /csp_csmp_id100020_vol1005_prd
- server: 100.68.21.4
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv0003
- spec:
- capacity:
- storage: 100Gi
- accessModes:
- - ReadWriteOnce
- persistentVolumeReclaimPolicy: Recycle
- storageClassName: nas-etcd
- nfs:
- path: /csp_csmp_id100020_vol1006_prd
- server: 100.68.21.4
- ---
- apiVersion: v1
- kind: Service
- metadata:
- name: etcd
- namespace: kube-system
- spec:
- selector:
- App: etcd
- clusterIP: None
- ports:
- ports:
- - port: 2379
- targetPort: 2379
- name: port2379
- - port: 2380
- targetPort: 2380
- name: port2380
- ---
- apiVersion: apps/v1beta1
- kind: StatefulSet
- metadata:
- name: etcd
- namespace: kube-system
- spec:
- serviceName: "etcd"
- replicas: 3
- template:
- metadata:
- labels:
- App: etcd
- spec:
- terminationGracePeriodSeconds: 10
- nodeSelector:
- caas_cluster: storage
- # host_name: ecam41060
- containers:
- - name: etcd
- image: hub.yun.paic.com.cn/etcd:test
- ports:
- - containerPort: 2379
- name: port2379
- containerPort: 2380
- name: port2380
- volumeMounts:
- - name: datadir
- mountPath: /var/lib/etcd
- volumeClaimTemplates:
- - metadata:
- name: datadir
- namespace: etcd
- spec:
- accessModes: [ "ReadWriteOnce" ]
- resources:
- requests:
- storage: 10Gi
- storageClassName: nas-etcd
来源: https://yq.aliyun.com/articles/689052