目录
前言
.... 前言
本文选用 Stolon 的方式搭建 PostgreSQL 高可用方案, 主要为 Harbor 提供高可用数据库, Harbor 搭建可查看 kubernetes 搭建 Harbor 无坑及 Harbor 仓库同步, 之后会提供 Redis 高可用及 Harbor 高可用方案搭建
方案比较
几种 PostgreSQL 高可用方案简单比较:
引用 https://studygolang.com/articles/19002?fr=sidebar
首先 repmgr 这种方案的算法有明显缺陷, 非主流分布式算法, 直接 pass;
Stolon 和 Patroni 相对于 Crunchy 更加 Cloud Native, 后者是基于 pgPool 实现.
Crunchy 和 Patroni 相对于 Stolon 有更多的使用者, 并且提供了 Operator 对于以后的管理和扩容
根据上面简单的比较, 最终选择的 stolon, 作者选择的是 Patroni, 感觉实际区别并不大.
一, Stolon 概述:
- Stolon( https://github.com/sorintlab/stolon
- )
是由 3 个部分组成的:
keeper: 他负责管理 PostgreSQL 的实例汇聚到由 sentinel(s)提供的 clusterview.
sentinel:it 负责发现并且监控 keeper, 并且计算最理想的 clusterview.
proxy: 客户端的接入点. 它强制连接到右边 PostgreSQL 的 master 并且强制关闭连接到由非选举产生的 master.
Stolon 用 etcd 或者 consul 作为主要的集群状态存储.
二, Installation
- Git clone https://github.com/sorintlab/stolon.git
- cd XXX/stolon/examples/kubernetes
如图所示
如有兴趣可查看官网搭建:
如下为 YAML 中注意修改的地方
stolon-keeper.YAML 中设置 PostgreSQL 用户名
- - name: STKEEPER_PG_SU_USERNAME
- value: "postgres"
stolon-keeper.YAML 中设置 stolon 挂载卷
- volumeClaimTemplates:
- - metadata:
- name: data
- spec:
- accessModes:
- - "ReadWriteOnce"
- resources:
- requests:
- storage: "512Mi"
- storageClassName: nfs
secret.YAML 中设置用户密码
- apiVersion: v1
- kind: Secret
- metadata:
- name: stolon
- type: Opaque
- data:
- password: cGFzc3dvcmQx
如下是作者整理的完整的 stolon 的编排文件, 可直接修改使用
- # This is an example and generic rbac role definition for stolon. It could be
- # fine tuned and split per component.
- # The required permission per component should be:
- # keeper/proxy/sentinel: update their own pod annotations
- # sentinel/stolonctl: get, create, update configmaps
- # sentinel/stolonctl: list components pods
- # sentinel/stolonctl: get components pods annotations
- apiVersion: rbac.authorization.k8s.io/v1beta1
- kind: Role
- metadata:
- name: stolon
- namespace: default
- rules:
- - apiGroups:
- - ""
- resources:
- - pods
- - configmaps
- - events
- verbs:
- - "*"
- ---
- apiVersion: rbac.authorization.k8s.io/v1beta1
- kind: RoleBinding
- metadata:
- name: stolon
- namespace: default
- roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: stolon
- subjects:
- - kind: ServiceAccount
- name: default
- namespace: default
- ---
- apiVersion: extensions/v1beta1
- kind: Deployment
- metadata:
- name: stolon-sentinel
- spec:
- replicas: 2
- template:
- metadata:
- labels:
- component: stolon-sentinel
- stolon-cluster: kube-stolon
- annotations:
- prometheus.io/scrape: "true"
- prometheus.io/port: "8080"
- spec:
- containers:
- - name: stolon-sentinel
- image: sorintlab/stolon:master-pg10
- command:
- - "/bin/bash"
- - "-ec"
- - |
- exec gosu stolon stolon-sentinel
- env:
- - name: POD_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.name
- - name: STSENTINEL_CLUSTER_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.labels['stolon-cluster']
- - name: STSENTINEL_STORE_BACKEND
- value: "kubernetes"
- - name: STSENTINEL_KUBE_RESOURCE_KIND
- value: "configmap"
- - name: STSENTINEL_METRICS_LISTEN_ADDRESS
- value: "0.0.0.0:8080"
- ## Uncomment this to enable debug logs
- #- name: STSENTINEL_DEBUG
- # value: "true"
- ports:
- - containerPort: 8080
- ---
- apiVersion: v1
- kind: Secret
- metadata:
- name: stolon
- type: Opaque
- data:
- password: cGFzc3dvcmQx
- ---
- # PetSet was renamed to StatefulSet in k8s 1.5
- # apiVersion: apps/v1alpha1
- # kind: PetSet
- apiVersion: apps/v1beta1
- kind: StatefulSet
- metadata:
- name: stolon-keeper
- spec:
- serviceName: "stolon-keeper"
- replicas: 2
- template:
- metadata:
- labels:
- component: stolon-keeper
- stolon-cluster: kube-stolon
- annotations:
- pod.alpha.kubernetes.io/initialized: "true"
- prometheus.io/scrape: "true"
- prometheus.io/port: "8080"
- spec:
- terminationGracePeriodSeconds: 10
- containers:
- - name: stolon-keeper
- image: sorintlab/stolon:master-pg10
- command:
- - "/bin/bash"
- - "-ec"
- - |
- # Generate our keeper uid using the pod index
- IFS='-' read -ra ADDR <<< "$(hostname)"
- export STKEEPER_UID="keeper${ADDR[-1]}"
- export POD_IP=$(hostname -i)
- export STKEEPER_PG_LISTEN_ADDRESS=$POD_IP
- export STOLON_DATA=/stolon-data
- chown stolon:stolon $STOLON_DATA
- exec gosu stolon stolon-keeper --data-dir $STOLON_DATA
- env:
- - name: POD_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.name
- - name: STKEEPER_CLUSTER_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.labels['stolon-cluster']
- - name: STKEEPER_STORE_BACKEND
- value: "kubernetes"
- - name: STKEEPER_KUBE_RESOURCE_KIND
- value: "configmap"
- - name: STKEEPER_PG_REPL_USERNAME
- value: "repluser"
- # Or use a password file like in the below supersuser password
- - name: STKEEPER_PG_REPL_PASSWORD
- value: "replpassword"
- - name: STKEEPER_PG_SU_USERNAME
- value: "postgres"
- - name: STKEEPER_PG_SU_PASSWORDFILE
- value: "/etc/secrets/stolon/password"
- - name: STKEEPER_METRICS_LISTEN_ADDRESS
- value: "0.0.0.0:8080"
- # Uncomment this to enable debug logs
- #- name: STKEEPER_DEBUG
- # value: "true"
- ports:
- - containerPort: 5432
- - containerPort: 8080
- volumeMounts:
- - mountPath: /stolon-data
- name: data
- - mountPath: /etc/secrets/stolon
- name: stolon
- volumes:
- - name: stolon
- secret:
- secretName: stolon
- # Define your own volumeClaimTemplate. This example uses dynamic PV provisioning with a storage class named "standard" (so it will works by default with minikube)
- # In production you should use your own defined storage-class and configure your persistent volumes (statically or dynamically using a provisioner, see related k8s doc).
- volumeClaimTemplates:
- - metadata:
- name: data
- spec:
- accessModes:
- - "ReadWriteOnce"
- resources:
- requests:
- storage: "512Mi"
- storageClassName: nfs
- ---
- apiVersion: v1
- kind: Service
- metadata:
- name: stolon-proxy-service
- spec:
- ports:
- - port: 5432
- targetPort: 5432
- selector:
- component: stolon-proxy
- stolon-cluster: kube-stolon
- ---
- apiVersion: extensions/v1beta1
- kind: Deployment
- metadata:
- name: stolon-proxy
- spec:
- replicas: 2
- template:
- metadata:
- labels:
- component: stolon-proxy
- stolon-cluster: kube-stolon
- annotations:
- prometheus.io/scrape: "true"
- prometheus.io/port: "8080"
- spec:
- containers:
- - name: stolon-proxy
- image: sorintlab/stolon:master-pg10
- command:
- - "/bin/bash"
- - "-ec"
- - |
- exec gosu stolon stolon-proxy
- env:
- - name: POD_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.name
- - name: STPROXY_CLUSTER_NAME
- valueFrom:
- fieldRef:
- fieldPath: metadata.labels['stolon-cluster']
- - name: STPROXY_STORE_BACKEND
- value: "kubernetes"
- - name: STPROXY_KUBE_RESOURCE_KIND
- value: "configmap"
- - name: STPROXY_LISTEN_ADDRESS
- value: "0.0.0.0"
- - name: STPROXY_METRICS_LISTEN_ADDRESS
- value: "0.0.0.0:8080"
- ## Uncomment this to enable debug logs
- #- name: STPROXY_DEBUG
- # value: "true"
- ports:
- - containerPort: 5432
- - containerPort: 8080
- readinessProbe:
- tcpSocket:
- port: 5432
- initialDelaySeconds: 10
- timeoutSeconds: 5
三, 部署 stolon
- kubectl applay -f stolon.YAML
- Initialize the cluster(大概意思是 stolon 初始化 k8s 集群, 可以大概看下官网解释)
All the stolon components wait for an existing clusterdata entry in the store. So the first time you have to initialize a new cluster. For more details see the cluster initialization doc. You can do this step at every moment, now or after having started the stolon components.
- You can execute stolonctl in different ways:
- as a one shot command executed inside a temporary pod:
- kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
- from a machine that can access the store backend:
- stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
later from one of the pods running the stolon components.
执行
kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
如图所示, 部署成功
四, 卸载 PostgreSQL 数据库
- kubectl delete -f stolon.YAML
- kubectl delete pvc data-stolon-keeper-0 data-stolon-keeper-1
五, 验证 PostgreSQL 安装成功(也可简单测试下)
1, 验证数据同步
连接 master 并且建立 test 表
- psql --host --port 30543 postgres -U stolon -W
- postgres=# create table test (id int primary key not null,
- value text not null);
- CREATE TABLE
- postgres=# insert into test values (1, 'value1');
- INSERT 0 1
- postgres=# select * from test;
- id | value
- ---- -------- 1 | value1
- (1 row)
也可进入 Pod 执行 PostgreSQL 命令
- kubectl exec -ti stolon-proxy-5977cdbcfc-csnkq bash
- # 登入 sql
- psql --host localhost --port 5432 postgres -U postgres
- \l #列出所有数据库
- \c dbname #切换数据库
- CREATE TABLE
- insert into test values (1, 'value1');
- INSERT 0 1
- select * from test;
- \d #列出当前数据库的所有表
- \q #退出数据库
连接 slave 并且检查数据. 你可以写一些信息以便确认请求已经被 slave 处理了.
- psql --host --port 30544 postgres -U stolon -W
- postgres=# select * from test;
- id | value
- ---- -------- 1 | value1
- (1 row)
2, 测试 failover
这个案例是官方代码库中 statefullset 的一个例子.
简单的说, 就是为模拟了 master 挂掉, 我们先删除了 master 的 statefulset 又删除了 master 的 pod.
- kubectl delete statefulset stolon-keeper --cascade=false
- kubectl delete pod stolon-keeper-0
然后, 在 sentinel 的 log 中我们可以看到新的 master 被选举出来了.
- no keeper info available db=cb96f42d keeper=keeper0
- no keeper info available db=cb96f42d keeper=keeper0
- master db is failed db=cb96f42d keeper=keeper0
- trying to find a standby to replace failed master
- electing db as the new master db=087ce88a keeper=keeper1
现在, 在刚才的那两个终端中如果我们重复上一个命令, 我们可以看到如下输出.
- postgres=# select from test;
- server closed the connection unexpectedly
- This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset:
Succeeded.
- postgres=# select from test;
- id | value
- ---- -------- 1 | value1
- (1 row)
Kubernetes 的 service 把不可用的 pod 去掉, 把请求转到可用的 pod 上. 所以新的读取连接被路由到了健康的 pod 上.
. 也可用 chaoskube 模拟随机的 pod 挂掉(准生产可以测试下)
另一个测试集群弹性 (resilience) 的好方法是用 https://github.com/linki/chaoskube .Chaoskube 是一个小的服务程序, 它可以周期性的在集群里随机的 kill 掉一些的 pod. 它也可以用 helm charts 部署.
- helm install --set labels="release=factualcrocodile,
- component!=factual-crocodine-etcd" --set
- interval=5m stable/chaoskube
这条命令会运行 chaoskube, 它会每 5 分钟删除一个 pod. 它会选择 label 中 release=factual-crocodile 的 pod, 但是会忽略 etcd 的 pod.
本文按照官网搭建, 主要为之后的 Harbor 高可用做准备, 有情趣的伙伴点个赞, 之后会续写 Redis,Harbor 高可用
参考资料:
- https://my.oschina.net/u/2306127/blog/2991474
- https://studygolang.com/articles/19002?fr=sidebar
来源: https://www.cnblogs.com/keep-live/p/11543770.html