1. 简介
Kubernetes v1.13 版本发布后, kubeadm 才正式进入 GA, 可以生产使用, 用 kubeadm 部署 kubernetes 集群也是以后的发展趋势. 目前 Kubernetes 的对应镜像仓库, 在国内阿里云也有了镜像站点, 使用 kubeadm 部署 Kubernetes 集群变得简单并且容易了很多, 本文使用 kubeadm 带领大家快速部署 Kubernetes v1.13.2 版本.
注意: 请不要把目光仅仅放在部署上, 如果你是新手, 推荐先熟悉用二进制文件部署后, 再来学习用 kubeadm 部署. 二进制文件部署请查看我博客的其他文章.
2. 架构信息
系统版本: CentOS 7.6
内核: 3.10.0-957.el7.x86_64
- Kubernetes: v1.13.2
- Docker-ce: 18.06
推荐硬件配置: 2 核 2G
Keepalived 保证 apiserever 服务器的 IP 高可用
Haproxy 实现 apiserver 的负载均衡
为了减少服务器数量, haproxy,keepalived 配置在 node-01 和 node-02.
节点名称 | 角色 | IP | 安装软件 |
---|---|---|---|
负载 VIP | VIP | 10.31.90.200 | |
node-01 | master | 10.31.90.201 | kubeadm、kubelet、kubectl、docker、haproxy、keepalived |
node-02 | master | 10.31.90.202 | kubeadm、kubelet、kubectl、docker、haproxy、keepalived |
node-03 | master | 10.31.90.203 | kubeadm、kubelet、kubectl、docker |
node-04 | node | 10.31.90.204 | kubeadm、kubelet、kubectl、docker |
node-05 | node | 10.31.90.205 | kubeadm、kubelet、kubectl、docker |
node-06 | node | 10.31.90.206 | kubeadm、kubelet、kubectl、docker |
service 网段 | 10.245.0.0/16 |
2. 部署前准备工作
1) 关闭 selinux 和防火墙
- sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
- setenforce 0
- systemctl disable firewalld
- systemctl stop firewalld
2) 关闭 swap
swapoff -a
3) 为每台服务器添加 host 解析记录
- cat>>/etc/hosts<<EOF
- 10.31.90.201 node-01
- 10.31.90.202 node-02
- 10.31.90.203 node-03
- 10.31.90.204 node-04
- 10.31.90.205 node-05
- 10.31.90.206 node-06
- EOF
4) 创建并分发密钥
在 node-01 创建 SSH 密钥.
- [root@node-01 ~]# SSH-keygen -t rsa
- Generating public/private rsa key pair.
- Enter file in which to save the key (/root/.SSH/id_rsa):
- Created directory '/root/.ssh'.
- Enter passphrase (empty for no passphrase):
- Enter same passphrase again:
- Your identification has been saved in /root/.SSH/id_rsa.
- Your public key has been saved in /root/.SSH/id_rsa.pub.
- The key fingerprint is:
- SHA256:26z6DcUarn7wP70dqOZA28td+K/erv7NlaJPLVE1BTA root@node-01
- The key's randomart image is:
- +---[RSA 2048]----+
- | E..o+|
- | . o|
- | . |
| . . |
- | S o . |
- | .o X oo .|
- | oB +.o+oo.|
- | .o*o+++o+o|
- | .++o+Bo+=B*B|
- +----[SHA256]-----+
分发 node-01 的公钥, 用于免密登录其他服务器
for n in `seq -w 01 06`;do SSH-copy-id node-$n;done
5) 配置内核参数
- cat <<EOF> /etc/sysctl.d/k8s.conf
- net.bridge.bridge-nf-call-ip6tables = 1
- net.bridge.bridge-nf-call-iptables = 1
- net.ipv4.ip_nonlocal_bind = 1
- net.ipv4.ip_forward = 1
- vm.swappiness=0
- EOF
- sysctl --system
6) 加载 ipvs 模块
- cat> /etc/sysconfig/modules/ipvs.modules <<EOF
- #!/bin/bash
- modprobe -- ip_vs
- modprobe -- ip_vs_rr
- modprobe -- ip_vs_wrr
- modprobe -- ip_vs_sh
- modprobe -- nf_conntrack_ipv4
- EOF
- chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
7) 添加 yum 源
- cat <<EOF> /etc/yum.repos.d/kubernetes.repo
- [kubernetes]
- name=Kubernetes
- baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
- enabled=1
- gpgcheck=1
- repo_gpgcheck=1
- gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
- EOF
- wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/CentOS-Base.repo
- wget http://mirrors.aliyun.com/repo/epel-7.repo -O /etc/yum.repos.d/epel.repo
- wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
2. 部署 keepalived 和 haproxy
1) 安装 keepalived 和 haproxy
在 node-01 和 node-02 安装 keepalived 和 haproxy
yum install -y keepalived haproxy
2) 修改配置
keepalived 配置
node-01 的 priority 为 100,node-02 的 priority 为 90, 其他配置一样.
- [root@node-01 ~]# cat /etc/keepalived/keepalived.conf
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- feng110498@163.com
- }
- notification_email_from Alexandre.Cassen@firewall.loc
- smtp_server 127.0.0.1
- smtp_connect_timeout 30
- router_id LVS_1
- }
- vrrp_instance VI_1 {
- state MASTER
- interface eth0
- lvs_sync_daemon_inteface eth0
- virtual_router_id 88
- advert_int 1
- priority 100
- authentication {
- auth_type PASS
- auth_pass 1111
- }
- virtual_ipaddress {
- 10.31.90.200/24
- }
- }
haproxy 配置
node-01 和 node-02 的 haproxy 配置是一样的. 此处我们监听的是 10.31.90.200 的 8443 端口, 因为 haproxy 是和 k8s apiserver 是部署在同一台服务器上, 都用 6443 会冲突.
- global
- chroot /var/lib/haproxy
- daemon
- group haproxy
- user haproxy
- log 127.0.0.1:514 local0 warning
- pidfile /var/lib/haproxy.pid
- maxconn 20000
- spread-checks 3
- nbproc 8
- defaults
- log global
- mode tcp
- retries 3
- option redispatch
- listen https-apiserver
- bind 10.31.90.200:8443
- mode tcp
- balance roundrobin
- timeout server 15s
- timeout connect 15s
- server apiserver01 10.31.90.201:6443 check port 6443 inter 5000 fall 5
- server apiserver02 10.31.90.202:6443 check port 6443 inter 5000 fall 5
- server apiserver03 10.31.90.203:6443 check port 6443 inter 5000 fall 5
3) 启动服务
- systemctl enable keepalived && systemctl start keepalived
- systemctl enable haproxy && systemctl start haproxy
3. 部署 kubernetes
1) 安装软件
由于 kubeadm 对 Docker 的版本是有要求的, 需要安装与 kubeadm 匹配的版本.
由于版本更新频繁, 请指定对应的版本号, 本文采用 1.13.2 版本, 其它版本未经测试.
- yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 ipvsadm ipset docker-ce-18.06.1.ce
- # 启动 docker
- systemctl enable docker && systemctl start docker
- # 设置 kubelet 开机自启动
- systemctl enable kubelet
2) 修改初始化配置
- [root@node-01 ~]# cat kubeadm-init.YAML
- apiVersion: kubeadm.k8s.io/v1beta1
- bootstrapTokens:
- - groups:
- - system:bootstrappers:kubeadm:default-node-token
- token: abcdef.0123456789abcdef
- ttl: 24h0m0s
- usages:
- - signing
- - authentication
- kind: InitConfiguration
- localAPIEndpoint:
- advertiseAddress: 10.31.90.201
- bindPort: 6443
- nodeRegistration:
- criSocket: /var/run/dockershim.sock
- name: node-01
- taints:
- - effect: NoSchedule
- key: node-role.kubernetes.io/master
- ---
- apiVersion: kubeadm.k8s.io/v1beta1
- kind: ClusterConfiguration
- apiServer:
- timeoutForControlPlane: 4m0s
- certificatesDir: /etc/kubernetes/pki
- clusterName: kubernetes
- controlPlaneEndpoint: "10.31.90.200:8443"
- dns:
- type: CoreDNS
- etcd:
- local:
- dataDir: /var/lib/etcd
- imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
- kubernetesVersion: v1.13.2
- networking:
- dnsDomain: cluster.local
- podSubnet: ""serviceSubnet:"10.245.0.0/16"
- scheduler: {}
- controllerManager: {}
- ---
- apiVersion: kubeproxy.config.k8s.io/v1alpha1
- kind: KubeProxyConfiguration
- mode: "ipvs"
3) 预下载镜像
- [root@node-01 ~]# kubeadm config images pull --config kubeadm-init.YAML
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.13.2
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.13.2
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.13.2
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.13.2
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.2.24
- [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.2.6
4) 初始化
- [root@node-01 ~]# kubeadm init --config kubeadm-init.YAML
- [init] Using Kubernetes version: v1.13.2
- [preflight] Running pre-flight checks
- [preflight] Pulling images required for setting up a Kubernetes cluster
- [preflight] This might take a minute or two, depending on the speed of your internet connection
- [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
- [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
- [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
- [kubelet-start] Activating the kubelet service
- [certs] Using certificateDir folder "/etc/kubernetes/pki"
- [certs] Generating "ca" certificate and key
- [certs] Generating "apiserver" certificate and key
- [certs] apiserver serving cert is signed for DNS names [node-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.12.0.1 10.31.90.201 10.31.90.200]
- [certs] Generating "apiserver-kubelet-client" certificate and key
- [certs] Generating "etcd/ca" certificate and key
- [certs] Generating "etcd/server" certificate and key
- [certs] etcd/server serving cert is signed for DNS names [node-01 localhost] and IPs [10.31.90.201 127.0.0.1 ::1]
- [certs] Generating "etcd/peer" certificate and key
- [certs] etcd/peer serving cert is signed for DNS names [node-01 localhost] and IPs [10.31.90.201 127.0.0.1 ::1]
- [certs] Generating "etcd/healthcheck-client" certificate and key
- [certs] Generating "apiserver-etcd-client" certificate and key
- [certs] Generating "front-proxy-ca" certificate and key
- [certs] Generating "front-proxy-client" certificate and key
- [certs] Generating "sa" key and public key
- [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
- [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
- [kubeconfig] Writing "admin.conf" kubeconfig file
- [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
- [kubeconfig] Writing "kubelet.conf" kubeconfig file
- [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
- [kubeconfig] Writing "controller-manager.conf" kubeconfig file
- [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
- [kubeconfig] Writing "scheduler.conf" kubeconfig file
- [control-plane] Using manifest folder "/etc/kubernetes/manifests"
- [control-plane] Creating static Pod manifest for "kube-apiserver"
- [control-plane] Creating static Pod manifest for "kube-controller-manager"
- [control-plane] Creating static Pod manifest for "kube-scheduler"
- [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
- [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
- [apiclient] All control plane components are healthy after 22.503955 seconds
- [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
- [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
- [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node-01" as an annotation
- [mark-control-plane] Marking the node node-01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
- [mark-control-plane] Marking the node node-01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
- [Bootstrap-token] Using token: abcdef.0123456789abcdef
- [Bootstrap-token] Configuring Bootstrap tokens, cluster-info ConfigMap, RBAC Roles
- [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
- [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
- [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
- [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
- [addons] Applied essential addon: CoreDNS
- [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
- [addons] Applied essential addon: kube-proxy
- Your Kubernetes master has initialized successfully!
- To start using your cluster, you need to run the following as a regular user:
- mkdir -p $HOME/.kube
- sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
- sudo chown $(id -u):$(id -g) $HOME/.kube/config
- You should now deploy a pod network to the cluster.
- Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
- https://kubernetes.io/docs/concepts/cluster-administration/addons/
- You can now join any number of machines by running the following on each node
- as root:
- kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1
kubeadm init 主要执行了以下操作:
[init]: 指定版本进行初始化操作
[preflight] : 初始化前的检查和下载所需要的 Docker 镜像文件
[kubelet-start] : 生成 kubelet 的配置文件 "/var/lib/kubelet/config.yaml", 没有这个文件 kubelet 无法启动, 所以初始化之前的 kubelet 实际上启动失败.
[certificates]: 生成 Kubernetes 使用的证书, 存放在 / etc/kubernetes/pki 目录中.
[kubeconfig] : 生成 KubeConfig 文件, 存放在 / etc/kubernetes 目录中, 组件之间通信需要使用对应文件.
[control-plane]: 使用 / etc/kubernetes/manifest 目录下的 YAML 文件, 安装 Master 组件.
[etcd]: 使用 / etc/kubernetes/manifest/etcd.YAML 安装 Etcd 服务.
[wait-control-plane]: 等待 control-plan 部署的 Master 组件启动.
[apiclient]: 检查 Master 组件服务状态.
[uploadconfig]: 更新配置
[kubelet]: 使用 configMap 配置 kubelet.
[patchnode]: 更新 CNI 信息到 Node 上, 通过注释的方式记录.
[mark-control-plane]: 为当前节点打标签, 打了角色 Master, 和不可调度标签, 这样默认就不会使用 Master 节点来运行 Pod.
[Bootstrap-token]: 生成 token 记录下来, 后边使用 kubeadm join 往集群中添加节点时会用到
[addons]: 安装附加组件 CoreDNS 和 kube-proxy
5) 为 kubectl 准备 Kubeconfig 文件
kubectl 默认会在执行的用户家目录下面的. kube 目录下寻找 config 文件. 这里是将在初始化时 [kubeconfig] 步骤生成的 admin.conf 拷贝到. kube/config.
- [root@node-01 ~]# mkdir -p $HOME/.kube
- [root@node-01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
- [root@node-01 ~]# chown $(id -u):$(id -g)$HOME/.kube/config
在该配置文件中, 记录了 API Server 的访问地址, 所以后面直接执行 kubectl 命令就可以正常连接到 API Server 中.
6) 查看组件状态
- [root@node-01 ~]# kubectl get cs
- NAME STATUS MESSAGE ERROR
- scheduler Healthy ok
- controller-manager Healthy ok
- etcd-0 Healthy {
- "health": "true"
- }
- [root@node-01 ~]# kubectl get node
- NAME STATUS ROLES AGE VERSION
- node-01 NotReady master 14m v1.13.2
目前只有一个节点, 角色是 Master, 状态是 NotReady.
7) 其他 master 部署
在 node-01 将证书文件拷贝至其他 master 节点
- USER=root
- CONTROL_PLANE_IPS="node-02 node-03"
- for host in ${
- CONTROL_PLANE_IPS
- }; do
- SSH "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
- scp /etc/kubernetes/pki/ca.* "${USER}"@$host:/etc/kubernetes/pki/
- scp /etc/kubernetes/pki/sa.* "${USER}"@$host:/etc/kubernetes/pki/
- scp /etc/kubernetes/pki/front-proxy-ca.* "${USER}"@$host:/etc/kubernetes/pki/
- scp /etc/kubernetes/pki/etcd/ca.* "${USER}"@$host:/etc/kubernetes/pki/etcd/
- scp /etc/kubernetes/admin.conf "${USER}"@$host:/etc/kubernetes/
- done
在其他 master 执行, 注意 --experimental-control-plane 参数
kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1 --experimental-control-plane
8) node 部署
在 node-04,node-05,node-06 执行, 注意没有 --experimental-control-plane 参数
kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1
9) 部署网络插件 weave
Master 节点 NotReady 的原因就是因为没有使用任何的网络插件, 此时 Node 和 Master 的连接还不正常. 目前最流行的 Kubernetes 网络插件有 Flannel,Calico,Canal,Weave 这里选择使用 Weave.
[root@node-01 ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d'\n')"
10) 查看节点状态
所有的节点已经处于 Ready 状态.
- [root@node-01 ~]# kubectl get node
- NAME STATUS ROLES AGE VERSION
- node-01 Ready master 35m v1.13.2
- node-02 Ready master 36m v1.13.2
- node-03 Ready master 36m v1.13.2
- node-04 Ready <none> 40m v1.13.2
- node-05 Ready <none> 40m v1.13.2
- node-06 Ready <none> 40m v1.13.2
查看 pod
- [root@node-01 ~]# kubectl get pod -n kube-system
- NAME READY STATUS RESTARTS AGE
- coredns-89cc84847-j8mmg 1/1 Running 0 1d
- coredns-89cc84847-rbjxs 1/1 Running 0 1d
- etcd-node-01 1/1 Running 1 1d
- etcd-node-02 1/1 Running 0 1d
- etcd-node-03 1/1 Running 0 1d
- kube-apiserver-node-01 1/1 Running 0 1d
- kube-apiserver-node-02 1/1 Running 0 1d
- kube-apiserver-node-03 1/1 Running 0 1d
- kube-controller-manager-node-01 1/1 Running 2 1d
- kube-controller-manager-node-02 1/1 Running 0 1d
- kube-controller-manager-node-03 1/1 Running 0 1d
- kube-proxy-jfbmv 1/1 Running 0 1d
- kube-proxy-lvkms 1/1 Running 0 1d
- kube-proxy-qx7kh 1/1 Running 0 1d
- kube-proxy-xst5v 1/1 Running 0 1d
- kube-proxy-zfwrk 1/1 Running 0 1d
- kube-proxy-ztg6j 1/1 Running 0 1d
- kube-scheduler-node-01 1/1 Running 1 1d
- kube-scheduler-node-02 1/1 Running 1 1d
- kube-scheduler-node-03 1/1 Running 1 1d
- weave.NET-2xr7j 2/2 Running 0 1d
- weave.NET-6x2gn 2/2 Running 1 1d
- weave.NET-l6fdw 2/2 Running 1 1d
- weave.NET-mswmj 2/2 Running 2 1d
- weave.NET-skdw7 2/2 Running 1 1d
- weave.NET-zm7zf 2/2 Running 1 1d
查看 ipvs 的状态
- [root@node-01 ~]# ipvsadm -L -n
- IP Virtual Server version 1.2.1 (size=4096)
- Prot LocalAddress:Port Scheduler Flags
- -> RemoteAddress:Port Forward Weight ActiveConn InActConn
- TCP 10.245.0.1:443 rr
- -> 10.31.90.201:6443 Masq 1 2 0
- -> 10.31.90.202:6443 Masq 1 0 0
- -> 10.31.90.203:6443 Masq 1 2 0
- TCP 10.245.0.10:53 rr
- -> 10.32.0.3:53 Masq 1 0 0
- -> 10.32.0.4:53 Masq 1 0 0
- TCP 10.245.90.161:80 rr
- -> 10.45.0.1:80 Masq 1 0 0
- TCP 10.245.90.161:443 rr
- -> 10.45.0.1:443 Masq 1 0 0
- TCP 10.245.149.227:1 rr
- -> 10.31.90.204:1 Masq 1 0 0
- -> 10.31.90.205:1 Masq 1 0 0
- -> 10.31.90.206:1 Masq 1 0 0
- TCP 10.245.181.126:80 rr
- -> 10.34.0.2:80 Masq 1 0 0
- -> 10.45.0.0:80 Masq 1 0 0
- -> 10.46.0.0:80 Masq 1 0 0
- UDP 10.245.0.10:53 rr
- -> 10.32.0.3:53 Masq 1 0 0
- -> 10.32.0.4:53 Masq 1 0 0
至此 kubernetes 集群部署完成. 如有问题欢迎在下面留言交流. 希望大家多多关注和点赞, 谢谢!
来源: http://blog.51cto.com/billy98/2350660