一 Nginx 代理实现 kube-apiserver 高可用
1.1 Nginx 实现高可用
基于 nginx 代理的 kube-apiserver 高可用方案.
控制节点的 kube-controller-manager,kube-scheduler 是多实例部署, 所以只要有一个实例正常, 就可以保证高可用;
集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver, kube-dns 会自动解析出多个 kube-apiserver 节点的 IP, 所以也是高可用的;
在每个节点起一个 nginx 进程, 后端对接多个 apiserver 实例, nginx 对它们做健康检查和负载均衡;
kubelet,kube-proxy,controller-manager,scheduler 通过本地的 nginx(监听 127.0.0.1)访问 kube-apiserver, 从而实现 kube-apiserver 的高可用;
从而基于 nginx 4 层透明代理功能实现 K8S 节点 ( master 节点和 worker 节点) 高可用访问 kube-apiserver .
1.2 下载编译 Nginx
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz
- [[email protected] work]# tar -xzvf nginx-1.15.3.tar.gz
- [[email protected] ~]# cd /opt/k8s/work/nginx-1.15.3/
- [[email protected] nginx-1.15.3]# mkdir nginx-prefix
- [[email protected] nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
- [[email protected] ~]# cd /opt/k8s/work/nginx-1.15.3/
- [[email protected] nginx-1.15.3]# make && make install
解释:
--with-stream: 开启 4 层透明转发 (TCP Proxy) 功能;
--without-xxx: 关闭所有其他功能, 这样生成的动态链接二进制程序依赖最小.
- [[email protected] ~]# cd /opt/k8s/work/nginx-1.15.3/
- [[email protected] nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
1.3 验证编译后的 Nginx
- [[email protected] ~]# cd /opt/k8s/work/nginx-1.15.3
- [[email protected] nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
- nginx version: nginx/1.15.3
- [[email protected] nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx # 查看 nginx 动态链接的库
- Linux-vdso.so.1 => (0x00007ffdda980000)
- libdl.so.2 => /lib64/libdl.so.2 (0x00007feb37300000)
- libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000)
- libc.so.6 => /lib64/libc.so.6 (0x00007feb36d17000)
- /lib64/ld-Linux-x86-64.so.2 (0x00007feb37504000)
提示: 由于只开启了 4 层透明转发功能, 所以除了依赖 libc 等操作系统核心 lib 库外, 没有对其它 lib 的依赖(如 libz,libssl 等), 以便达到精简编译的目的.
1.4 安装和部署 Nginx
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
- done # 创建 Nginx 目录
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx [email protected]${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
- SSH [email protected]${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
- SSH [email protected]${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
- done # 分发 Nginx 二进制
1.5 配置 Nginx 四层透明转发
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# cat> kube-nginx.conf <<EOF
- worker_processes 1;
- events {
- worker_connections 1024;
- }
- stream {
- upstream backend {
- hash $remote_addr consistent;
- server 172.24.8.71:6443 max_fails=3 fail_timeout=30s;
- server 172.24.8.72:6443 max_fails=3 fail_timeout=30s;
- server 172.24.8.73:6443 max_fails=3 fail_timeout=30s;
- }
- server {
- listen 127.0.0.1:8443;
- proxy_connect_timeout 1s;
- proxy_pass backend;
- }
- }
- EOF
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- scp kube-nginx.conf [email protected]${master_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
- done # 分发 Nginx 四层透明代理配置文件
1.6 配置 Nginx system
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# cat> kube-nginx.service <<EOF
- [Unit]
- Description=kube-apiserver nginx proxy
- After=network.target
- After=network-online.target
- Wants=network-online.target
- [Service]
- Type=forking
- ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
- ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
- ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
- PrivateTmp=true
- Restart=always
- RestartSec=5
- StartLimitInterval=0
- LimitNOFILE=65536
- [Install]
- WantedBy=multi-user.target
- EOF
1.7 分发 Nginx systemd
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- scp kube-nginx.service [email protected]${master_ip}:/etc/systemd/system/
- done
二 启动并验证
2.1 启动 Nginx
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- SSH [email protected]${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
- done
2.2 检查 Nginx 服务
- [[email protected] ~]# cd /opt/k8s/work
- [[email protected] work]# source /opt/k8s/bin/environment.sh
- [[email protected] work]# for master_ip in ${MASTER_IPS[@]}
- do
- echo ">>> ${master_ip}"
- SSH [email protected]${master_ip} "systemctl status kube-nginx |grep'Active:'"
- done
来源: http://www.bubuko.com/infodetail-3289937.html