Master apiserver 启用 TLS 认证后, Node 节点 kubelet 组件想要加入集群, 必须使用 CA 签发的有效证书才能与 apiserver 通信, 当 Node 节点很多时, 签署证书是一件很繁琐的事情, 因此有了 TLS Bootstrapping 机制, kubelet 会以一个低权限用户自动向 apiserver 申请证书, kubelet 的证书由 apiserver 动态签署.
认证大致工作流程如图所示:
准备二进制文件
- scp kubelet kube-proxy 192.168.0.125:/opt/kubernetes/bin/
- scp kubelet kube-proxy 192.168.0.126:/opt/kubernetes/bin/
部署 kubelet
创建角色绑定
kubectl create clusterrolebinding kubelet-Bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-Bootstrap
创建 kubelet bootstrapping kubeconfig 文件
- # 设置集群参数
- kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.130:6443 \
- --kubeconfig=Bootstrap.kubeconfig
- # 设置客户端认证参数
- kubectl config set-credentials kubelet-Bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=Bootstrap.kubeconfig
- # 设置上下文参数
- kubectl config set-context default --cluster=kubernetes --user=kubelet-Bootstrap --kubeconfig=Bootstrap.kubeconfig
- # 设置默认上下文
- kubectl config use-context default --kubeconfig=Bootstrap.kubeconfig
将 Bootstrap.kubeconfig 文件拷贝到 node 节点
- scp Bootstrap.kubeconfig 192.168.0.125:/opt/kubernetes/cfg/
- scp Bootstrap.kubeconfig 192.168.0.126:/opt/kubernetes/cfg/
创建 kubelet 配置文件
- [root@k8s-node01 ~]# VIM /opt/kubernetes/cfg/kubelet
- KUBELET_OPTS="--logtostderr=false \
- --v=4 \
- --log-dir=/opt/kubernetes/log \
- --hostname-override=192.168.0.125 \
- --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
- --Bootstrap-kubeconfig=/opt/kubernetes/cfg/Bootstrap.kubeconfig \
- --config=/opt/kubernetes/cfg/kubelet.config \
- --cert-dir=/opt/kubernetes/ssl \
- --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
参数说明:
--hostname-override 在集群中显示的主机名
--kubeconfig 指定 kubeconfig 文件位置, 会自动生成
--Bootstrap-kubeconfig 指定刚才生成的 Bootstrap.kubeconfig 文件
--cert-dir 颁发证书存放位置
--pod-infra-container-image 管理 Pod 网络的镜像
其中 / opt/kubernetes/cfg/kubelet.config 配置文件如下:
- [root@k8s-node01 ~]# VIM /opt/kubernetes/cfg/kubelet.config
- kind: KubeletConfiguration
- apiVersion: kubelet.config.k8s.io/v1beta1
- address:
- port: 10250
- readOnlyPort: 10255
- cgroupDriver: cgroupfs
- clusterDNS:
- - 10.0.0.2
- clusterDomain: cluster.local.
- failSwapOn: false
- authentication:
- anonymous:
- enabled: true
创建 kubelet 系统服务
- [root@k8s-node02 ~]# VIM /usr/lib/systemd/system/kubelet.service
- [Unit]
- Description=Kubernetes Kubelet
- After=docker.service
- Requires=docker.service
- [Service]
- EnvironmentFile=/opt/kubernetes/cfg/kubelet
- ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
- Restart=on-failure
- KillMode=process
- [Install]
- WantedBy=multi-user.target
启动 kubelet
- systemctl daemon-reload
- systemctl enable kubelet
- systemctl restart kubelet
- systemctl status kubelet
查看 csr 请求
- [root@k8s-master1 ~]# kubectl get csr
- NAME AGE REQUESTOR CONDITION
- node-csr-h0XFLgAXsCQvIQdUN5_fHGJbwYJaekO3zzhEK_wDcNY 103s kubelet-Bootstrap Pending
- node-csr-xECZ6WkPvlSzu9fE4CJQlMjPfCxJlUpidvSuKWOGpZE 90s kubelet-Bootstrap Pending
批准 kubelet 的 TLS 证书请求
- [root@k8s-master1 ~]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
- certificatesigningrequest.certificates.k8s.io/node-csr-h0XFLgAXsCQvIQdUN5_fHGJbwYJaekO3zzhEK_wDcNY approved
- certificatesigningrequest.certificates.k8s.io/node-csr-xECZ6WkPvlSzu9fE4CJQlMjPfCxJlUpidvSuKWOGpZE approved
查看 node 已经加入集群
- [root@k8s-master1 ~]# kubectl get node
- NAME STATUS ROLES AGE VERSION
- 192.168.0.125 Ready <none> 68s v1.13.0
- 192.168.0.126 Ready <none> 69s v1.13.0
kube-proxy 部署
配置 kube-proxy 使用 LVS
yum install -y ipvsadm ipset conntrack
创建 kube-proxy 证书请求
- [root@k8s-master1 ssl]# VIM kube-proxy-csr.JSON
- {
- "CN": "system:kube-proxy",
- "hosts": [],
- "key": {
- "algo": "rsa",
- "size": 2048
- },
- "names": [
- {
- "C": "CN",
- "ST": "BeiJing",
- "L": "BeiJing",
- "O": "k8s",
- "OU": "System"
- }
- ]
- }
生成证书
cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.JSON -profile=kubernetes kube-proxy-csr.JSON | cfssljson -bare kube-proxy
将证书分发到 node 节点
- scp kube-proxy*.pem 192.168.0.125:/opt/kubernetes/ssl/
- scp kube-proxy*.pem 192.168.0.126:/opt/kubernetes/ssl/
创建 kube-proxy kubeconfig 文件
- kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.130:6443 \
- --kubeconfig=kube-proxy.kubeconfig
- kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
- kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
- kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
分发 kubeconfig 配置文件
- scp kube-proxy.kubeconfig 192.168.0.125:/opt/kubernetes/cfg/
- scp kube-proxy.kubeconfig 192.168.0.126:/opt/kubernetes/cfg/
创建 kube-proxy 配置文件
- [root@k8s-node01 ~]# VIM /opt/kubernetes/cfg/kube-proxy
- KUBE_PROXY_OPTS="--logtostderr=false \
- --v=4 \
- --log-dir=/opt/kubernetes/log \
- --hostname-override=192.168.0.125 \
- --cluster-cidr=10.0.0.0/24 \
- --proxy-mode=ipvs \
- --ipvs-min-sync-period=5s \
- --ipvs-sync-period=5s \
- --ipvs-scheduler=rr \
- --masquerade-all=true \
- --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
创建 kube-proxy 系统服务
- [root@k8s-node01 ~]# VIM /usr/lib/systemd/system/kube-proxy.service
- [Unit]
- Description=Kubernetes Proxy
- After=network.target
- [Service]
- EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
- ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
- Restart=on-failure
- RestartSec=5
- LimitNOFILE=65536
- [Install]
- WantedBy=multi-user.target
启动 kube-proxy
- systemctl daemon-reload
- systemctl enable kube-proxy
- systemctl restart kube-proxy
- systemctl status kube-proxy
运行一个测试示例
- kubectl run nginx --image=nginx --replicas=3
- kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort
查看 pod,service
- [root@k8s-master1 ~]# kubectl get pod,svc -o wide
- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
- pod/nginx-7cdbd8cdc9-g9658 0/1 ImagePullBackOff 0 51s 172.17.84.2 192.168.0.125 <none> <none>
- pod/nginx-7cdbd8cdc9-wmh46 0/1 ContainerCreating 0 51s <none> 192.168.0.126 <none> <none>
- pod/nginx-7cdbd8cdc9-zwmxd 0/1 ContainerCreating 0 51s <none> 192.168.0.126 <none> <none>
- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
- service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 21h <none>
- service/nginx NodePort 10.0.0.171 <none> 88:48652/TCP 48s run=nginx
访问部署的 nginx
查看访问日志
- [root@k8s-master1 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
- [root@k8s-master1 ~]# kubectl logs nginx-7cdbd8cdc9-g9658
- 172.17.84.1 - - [18/Dec/2018:01:59:43 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ApplewebKit/537.36 (Khtml, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "-"
查看 LVS 状态
- [root@k8s-node02 ~]# ipvsadm -L -n
- IP Virtual Server version 1.2.1 (size=4096)
- Prot LocalAddress:Port Scheduler Flags
- -> RemoteAddress:Port Forward Weight ActiveConn InActConn
- TCP 127.0.0.1:48652 rr
- -> 172.17.34.2:80 Masq 1 0 0
- -> 172.17.34.3:80 Masq 1 0 0
- -> 172.17.84.2:80 Masq 1 0 0
- TCP 172.17.34.0:48652 rr
- -> 172.17.34.2:80 Masq 1 0 0
- -> 172.17.34.3:80 Masq 1 0 0
- -> 172.17.84.2:80 Masq 1 0 0
- TCP 172.17.34.1:48652 rr
- -> 172.17.34.2:80 Masq 1 0 0
- -> 172.17.34.3:80 Masq 1 0 0
- -> 172.17.84.2:80 Masq 1 0 0
- TCP 192.168.0.126:48652 rr
- -> 172.17.34.2:80 Masq 1 0 0
- -> 172.17.34.3:80 Masq 1 0 0
- -> 172.17.84.2:80 Masq 1 0 0
- TCP 10.0.0.1:443 rr
- -> 192.168.0.123:6443 Masq 1 0 0
- -> 192.168.0.124:6443 Masq 1 0 0
- TCP 10.0.0.171:88 rr
- -> 172.17.34.2:80 Masq 1 0 0
- -> 172.17.34.3:80 Masq 1 0 0
- -> 172.17.84.2:80 Masq 1 0 0
来源: http://www.bubuko.com/infodetail-2889682.html