kuberntes 系统使用 etcd 存储所有数据, 是最重要的组件之一, 注意 etcd 集群只能有奇数个节点(1,3,5...), 本文档使用 3 个节点做集群.
一, 基础环境
软件包
etcd 下载地址: https://github.com/coreos/etcd/releases
服务器
架构图
二, 生成 etcd 证书与私钥
创建 etcd 配置文件
- {
- "CN": "etcd",
- "hosts": [
- "127.0.0.1",
- "192.168.1.11",
- "192.168.1.12",
- "192.168.1.13",
- "etcd1",
- "etcd2",
- "etcd3"
- ],
- "key": {
- "algo": "rsa",
- "size": 2048
- },
- "names": [
- {
- "C": "CN",
- "ST": "BeiJing",
- "L": "BeiJing",
- "O": "Ctyun",
- "OU": "ops"
- }
- ]
- }
hosts: 定授权使用该证书的 etcd 节点 IP(应包含集群内所有节点的 IP 与 hostname)
C: 国家
ST: 省份
L: 城市
O: 公司
OU: 部门
生成私钥, 证书请求文件, 证书
CA 证书在前面的章节创建 Kubernetes 证书相关(CFSSL) https://www.jianshu.com/p/944f2003c829
- cfssl gencert -ca=/opt/ssl/k8sca/ca.pem \
- -ca-key=/opt/ssl/k8sca/ca-key.pem \
- -config=/opt/ssl/k8sca/ca-config.JSON \
- -profile=kubernetes /opt/ssl/etcd/etcd-csr.JSON | cfssljson -bare etcd
-ca: 指定 CA 证书路径
-ca-key: 指定 CAKey 的路径
-config: 指定 CA 证书签署策略配置文件
二, 启动和配置 etcd
创建 etcd 配置文件 etcd.conf, 此配置文件是定义一些变量, 方便 etcd.service 文件中直接引用, 方便后期维护.
配置文件的所有含义都在创建 etcd.service 配置文件后有详解.
- mkdir -p /etc/etcd
- VIM /etc/etcd/etcd.conf
- # [member]
- ETCD_NAME=etcd1
- ETCD_DATA_DIR="/var/lib/etcd"
- ETCD_LISTEN_PEER_URLS="https://192.168.1.11:2380"
- ETCD_LISTEN_CLIENT_URLS="https://192.168.1.11:2379"
- #[cluster]
- ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.11:2380"
- ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
- ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.11:2379"
创建 etcd.service 配置文件
etcd.service 的变量都是引用 etcd 配置文件中
- mkdir /var/lib/etcd
- VIM /usr/lib/systemd/system/etcd.service
- [Unit]
- Description=Etcd Server
- After=network.target
- After=network-online.target
- Wants=network-online.target
- Documentation=https://github.com/coreos
- [Service]
- Type=notify
- WorkingDirectory=${ETCD_DATA_DIR}
- EnvironmentFile=/etc/etcd/etcd.conf
- ExecStart=/usr/local/bin/etcd \
- --name=etcd1 \
- --cert-file=/opt/ssl/etcd/etcd.pem \
- --key-file=/opt/ssl/etcd/etcd-key.pem \
- --peer-cert-file=/opt/ssl/etcd/etcd.pem \
- --peer-key-file=/opt/ssl/etcd/etcd-key.pem \
- --trusted-ca-file=/opt/ssl/k8sca/ca.pem \
- --peer-trusted-ca-file=/opt/ssl/k8sca/ca.pem \
- --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
- --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
- --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
- --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
- --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
- --initial-cluster etcd1=https://192.168.1.11:2380,etcd2=https://192.168.1.12:2380,etcd3=https://192.168.1.13:2380 \
- --initial-cluster-state=new \
- --data-dir=/var/lib/etcd
- Restart=on-failure
- RestartSec=5
- LimitNOFILE=65536
- [Install]
- WantedBy=multi-user.target
--initial-cluster: 集群当中的其他节点
--cert-file:etcd 证书路径
--key-file:etcd 私钥路径
--peer-cert-file: 对等证书 (双向证书) 路径
--peer-key-file: 对等证书 (双向证书) 私钥路径
--trusted-ca-file: 作为客户端时的 CA 证书路径
--peer-trusted-ca-file: 对等证书的 CA 证书路径
--initial-advertise-peer-urls: 列出集群成员通信的 URL, 用于通告集群其他成员
--listen-peer-urls: 用于监听集群其他成员的 URL 列表
--listen-client-urls: 用于监听客户端通讯的 URL 列表
--advertise-client-urls: 通告客户端的 URL, 用于列出所有客户端
--initial-cluster-token:etcd 集群的初始集群令牌, 服务器必须通过令牌才能加入 etcd 集群
启动 etcd 集群
集群所有节点都配置好配置文件, 同时启动.
systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
不使用变量的配置方法
前面分别创建了 / etc/etcd/etcd.conf 与 / var/lib/systemd/system/etcd.service 两个配置文件, 且 etcd.service 引用了 etcd.conf 重定义的变量.
如若不想引用变量, 可以不写 etcd.conf. 仅创建 etcd.service 如下
- [root@etcd1 k8sca]# cat /usr/lib/systemd/system/etcd.service
- [Unit]
- Description=Etcd Server
- After=network.target
- After=network-online.target
- Wants=network-online.target
- Documentation=https://github.com/coreos
- [Service]
- Type=notify
- WorkingDirectory=/var/lib/etcd/
- ExecStart=/usr/local/bin/etcd \
- --name etcd1 \
- --cert-file=/opt/ssl/etcd/etcd.pem \
- --key-file=/opt/ssl/etcd/etcd-key.pem \
- --peer-cert-file=/opt/ssl/etcd/etcd.pem \
- --peer-key-file=/opt/ssl/etcd/etcd-key.pem \
- --trusted-ca-file=/opt/ssl/k8sca/ca.pem \
- --peer-trusted-ca-file=/opt/ssl/k8sca/ca.pem \
- --initial-advertise-peer-urls=https://192.168.1.11:2380 \
- --listen-peer-urls=https://192.168.1.11:2380 \
- --listen-client-urls=https://192.168.1.11:2379,http://127.0.0.1:2379 \
- --advertise-client-urls=https://192.168.1.11:2379 \
- --initial-cluster-token=etcd-cluster-0 \
- --initial-cluster etcd1=https://192.168.1.11:2380,etcd2=https://192.168.1.12:2380,etcd3=https://192.168.1.13:2380 \
- --initial-cluster-state=new \
- --data-dir=/var/lib/etcd
- Restart=on-failure
- RestartSec=5
- LimitNOFILE=65536
- [Install]
- WantedBy=multi-user.target
三, 测试集群状态
集群中所有主机配置均一致, 仅 IP 地址不同.
- $ etcdctl --ca-file=/opt/ssl/k8sca/ca.pem --cert-file=/opt/ssl/etcd/etcd.pem --key-file=/opt/ssl/etcd/etcd-key.pem cluster-health
- member aa869cb0f2e7ed31 is healthy: got healthy result from https://192.168.1.11:2379
- member b08a644fd7247c5e is healthy: got healthy result from https://192.168.1.13:2379
- member bb9bd2baaebf7d95 is healthy: got healthy result from https://192.168.1.12:2379
常见问题
publish error: etcdserver: request timed out
我部署时先使用一台进行测试, 无论如何启动 etcd 都无法启动, 提示
publish error: etcdserver: request timed out
. 后来发现 etcd.service 中制定了其他 etcd 主机, 所以当单独启动一台是连接其他主机不通, 那么 etcd 就启动失败. 所以要把 etcd 集群中所有主机都配置好, 同时启动就 OK 了.
参考
- http://blog.51cto.com/sgk2011/2108542
- https://github.com/gjmzj/kubeasz/blob/master/docs/setup/02-install_etcd.md
- https://blog.csdn.net/qq_33199919/article/details/80623055
- https://skyao.gitbooks.io/learning-etcd3/content/
来源: https://www.cnblogs.com/aubin/p/9993674.html