利用 ldirectord 实现 RS 的高可用性 --- 实验: 搭建 Ldirectord, 实现 Real Server 的高可用性
LVS 高可用性 --- RS 的高可用
1 Director 不可用, 整个系统将不可用;
SPoF Single Point of Failure 单点失败
解决方案: 高可用
- keepalived(实现相对容易的, 轻量级的解决方案)
- heartbeat/corosync(重量级的实现方法, 在生产中用的越来越少)
2 某 RS 不可用时, Director 依然会调度请求至此 RS
解决方案: 由 Director 对各 RS 健康状态进行检查, 失败时禁用, 成功时启用
- keepalived
- heartbeat/corosync
ldirectord(更好的解决方案, 可以放便的实现健康状态检查功能, 此外, 还带有 IPVS 策略的定义, 此前定义 IPVS 策略是手动键入命令, 但是 ldirectord 自动就把 IPVS 策略配置好了, 不用手动配置了)
检测方式
(a) 网络层检测, icmp
(b) 传输层检测, 端口探测
(c) 应用层检测, 请求某关键资源
RS 全不可用时: backup server, sorry server
LVS 在具体实现的时候存在单点失败的问题, 例如 LVS 本身出故障了.
Ldirectord 策略可以替代 ipvsadm 策略, 所以, 安装 Ldirectord 就可以不要 ipvsadm 策略了
[root@localhost ~]# ipvsadm -C
Ldirectord 只能够配置 LVS 服务器, 配置不了 Real Server
ldirectord 官网: http://horms.net/projects/ldirectord/ http://horms.net/projects/ldirectord/
ldirectord ldirectord is a daemon to monitor and administer real servers in a LVS cluster of load balanced virtual servers. ldirectord typically used as a resource for Linux-HA , but can also be run from the command line.
ldirectord ldirectord 是监控和管理实际服务器守护进程在 LVS 集群负载均衡的虚拟服务器. linux - ha ldirectord 通常作为一个资源, 但也可以从命令行运行.
ldirectord 属于高可用集成套件中的一个包而已
安装 ldirectord
[root@LVS ~]# yum -y install ldirectord-3.9.6-0rc1.1.2.x86_64.rpm
ldirectord 依赖很多 perl 包
安装后生成的文件
- [root@LVS ~]# rpm -ql ldirectord
- /etc/ha.d
- /etc/ha.d/resource.d
/etc/ha.d/resource.d/ldirectord
/etc/logrotate.d/ldirectord
/usr/lib/ocf/resource.d/heartbeat/ldirectord
/usr/lib/systemd/system/ldirectord.service
/usr/sbin/ldirectord
搭建 Ldirectord, 实现 Real Server 的高可用性
在 RS 机器上需要运行的脚本
- [root@rs1 ~]# cat lvs_dr_rs.sh
- #!/bin/bash
- vip=10.0.0.100
- mask='255.0.0.0'
- dev=lo:1
- rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
- service httpd start &> /dev/null && echo "The httpd Server is Ready!"
- echo "`hostname`"> /var/www/html/index.html
- case $1 in
- start)
- echo 1> /proc/sys/net/ipv4/conf/all/arp_ignore
- echo 1> /proc/sys/net/ipv4/conf/lo/arp_ignore
- echo 2> /proc/sys/net/ipv4/conf/all/arp_announce
- echo 2> /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
- echo "The RS Server is Ready!"
- ;;
- stop)
- ifconfig $dev down
- echo 0> /proc/sys/net/ipv4/conf/all/arp_ignore
- echo 0> /proc/sys/net/ipv4/conf/lo/arp_ignore
- echo 0> /proc/sys/net/ipv4/conf/all/arp_announce
- echo 0> /proc/sys/net/ipv4/conf/lo/arp_announce
- echo "The RS Server is Canceled!"
- ;;
- *)
- echo "Usage: $(basename $0) start|stop"
- exit 1
- ;;
- esac
- [root@rs1 ~]#
在 LVS 机器上需要运行的脚本
- [root@LVS ~]# cat lvs_dr_vs.sh
- #!/bin/bash
- vip='10.0.0.100'
- iface='ens34:1'
- mask='255.255.255.255'
- port='80'
- rs1='192.168.159.102'
- rs2='192.168.159.103'
- scheduler='wrr'
- type='-g'
- rpm -q ipvsadm &> /dev/null || yum -y install ipvsadm &> /dev/null
- case $1 in
- start)
ifconfig $iface $vip netmask $mask #broadcast $vip up
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
- echo "The VS Server is Ready!"
- ;;
- stop)
- ipvsadm -C
- ifconfig $iface down
- echo "The VS Server is Canceled!"
- ;;
- *)
- echo "Usage: $(basename $0) start|stop"
- exit 1
- ;;
- esac
- [root@LVS ~]#
配置 ldirectord, 把配置文件模板, 复制到 / etc/ha.d 下, 当做配置文件
[root@localhost ~]# cp /usr/share/doc/ldirectord-3.9.6/ldirectord.cf /etc/ha.d/
配置文件中大部分都是注释, 注释大部分都是范例, 可以根据自己的生产环境, 参考例子改改就可以了
Ldirectord 重要的作用就是健康性检查功能, 检查后端的 Real Server 是不是可用
配置文件内容
[root@LVS ~]# vim /etc/ha.d/ldirectord.cf
checktimeout=3 检查超时时间, 探测一次 3 秒不回应, 就认为死了
checkinterval=1 探测间隔 (一秒探测一次, 探测时间太久用户就可能会发觉服务不可用)
#fallback=127.0.0.1:80 备用服务器的地址 (如果服务器全挂了会看到拒绝访问, 入如果配置了这个选项, 用户就会看到这个服务器提供的页面, 一般配置成 LVS 服务器所在的地址, 所以要确保 LVS 可以提供网页服务. 去掉注释, 启用)
在 LVS 服务器上搭建备用服务器
- yum -y install httpd && systemctl start httpd
- echo Sorror,Server Down!> /var/www/html/index.html
- [root@LVS ~]# curl 10.0.0.100
- Sorror,Server Down!
- #fallback6=[::1]:80 IPV6 地址
autoreload=yes IPV6 地址, 不用管它. 这个配置文件将来需不需要改完以后通过 systemctl restart 的方式生效. 不需要配置, 修改完以后自动生效 (第一次还需要把 ldirectord 服务手动的起来, 起来以后, 在修改这个文件, 就会自动生效了, 而不用重启服务)
- #logfile="/var/log/ldirectord.log" 日志
- #logfile="local0" 日志级别
- # Sample for an http virtual service VIP
- virtual=192.168.6.240:80
- real=192.168.6.2:80 gate 1 gate(DR 模型) 1(权重)
- real=192.168.6.3:80 gate
- real=192.168.6.6:80 gate
service=http 服务
scheduler=rr 调度算法
- #persistent=600 超时时间
- #netmask=255.255.255.255
protocol=tcp 协议
checktype=negotiate 测试的类型 (健康性检查的方式)
checkport=80 健康性就检查 80 端口, 看测试页是否可以被访问可以访问就没问题
request="index.html" 要探测的页面 (准备一个测试页比较好)
receive="Test Page" 探测页面中的关键字符串
virtualhost=www.x.y.z 不用加, 注释掉
修改配置文件
- [root@LVS ~]# vim /etc/ha.d/ldirectord.cf
- # Global Directives
- checktimeout=3
- checkinterval=1
- fallback=127.0.0.1:80
- #fallback6=[::1]:80
- autoreload=yes
- logfile="/var/log/ldirectord.log"
- logfile="local0"
- # Sample for an http virtual service
- virtual=10.0.0.100:80
- real=192.168.111.102 gate 1
- real=192.168.111.103 gate 3
- # fallback=127.0.0.1:80 gate
- service=http
- scheduler=wrr
- #persistent=600
- #netmask=255.255.255.255
- protocol=tcp
- checktype=negotiate
- checkport=80
- request="test.html"
- receive="test"
- # virtualhost=www.x.y.z
准备测试页
- [root@RS1 ~]# echo test> /var/www/html/test.html
- [root@RS2 ~]# echo test> /var/www/html/test.html
启动服务
[root@localhost ~]# systemctl start ldirectord
注意: 没有手工加 ipvsadm 策略, 启动服务的时候会自动根据配置文件, 生成的 ipvsadm 策略
- [root@LVS ~]# ipvsadm -ln
- IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
- TCP 10.0.0.100:80 wrr
- -> 192.168.111.102:80 Route 1 0 0
- -> 192.168.111.103:80 Route 3 0 0
- [root@client ~]# for i in {1..10}; do curl 10.0.0.100; done
- server1.ding.com
- server2.ding.com
- server2.ding.com
- server1.ding.com
- server2.ding.com
- ......
- [root@client ~]# for i in {1..100};do curl 10.0.0.100; sleep 0.2; done
- RS1
- RS2
- RS2
- RS2
- [root@LVS ~]# ipvsadm -ln
- IP Virtual Server version 1.2.1 (size=4096)
- TCP 10.0.0.100:80 wrr
- -> 192.168.111.103:80 Route 3 0 49
- [root@localhost ~]#
- [root@client ~]# curl 10.0.0.100
- Sorror,Server Down!
- [root@LVS ~]# tail -f /var/log/ldirectord.log
- [Mon Mar 5 20:33:05 2018|ldirectord|2559] Deleted fallback server: 127.0.0.1:80 (192.168.6.240:80)
- [Mon Mar 5 20:33:05 2018|ldirectord|2559] Deleted real server: 192.168.6.2:80 (192.168.6.240:80)
- [root@RS1 ~]# tail -f /var/log/httpd/access_log
- 192.168.111.100 - - [05/Mar/2018:20:55:12 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"
- 192.168.111.100 - - [05/Mar/2018:20:55:13 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"
- 192.168.111.100 - - [05/Mar/2018:20:55:14 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"
- [root@LVS /etc/ha.d]# vim ldirectord.cf
- # Sample configuration for a fwmark based service For an explanation of
- # fwmark see the ipvsadm(8) man page
- #virtual=1 标签
- # real=192.168.6.2 gate
- # real=192.168.6.3 gate
- # real=192.168.6.6 gate
- # fallback=127.0.0.1:80 gate
- # service=http
- # scheduler=rr
- # #persistent=600
- # #netmask=255.255.255.255
- # protocol=fwm
- # checktype=negotiate
- # checkport=80
- # request="index.html"
- # receive="Test Page"
- # virtualhost=x.y.z
- # Sample for an http virtual service
- virtual=10
- # fallback=127.0.0.1:80 gate
- service=http
- scheduler=wrr
- #persistent=600 持久连接, 启用后就会一直往一个服务器上调度了
- #netmask=255.255.255.255
- protocol=fwm #这个加不加都可以
- checktype=negotiate
- checkport=80
- request="test.html"
- receive="test"
- # virtualhost=www.x.y.z
- [root@LVS ~]# ipvsadm -ln
- IP Virtual Server version 1.2.1 (size=4096)
- FWM 10 wrr
- -> 192.168.111.102:0 Route 1 0 0
- -> 192.168.111.103:0 Route 3 0 0
- [root@LVS ~]#
- [root@client ~]# for i in {1..100} ; do curl -k https://10.0.0.100; curl 10.0.0.100; done
- [root@client ~]# ssh 10.0.0.100
- root@10.0.0.100's password:
- Last login: Thu Apr 26 11:51:32 2018 from 192.168.3.204
- [root@LVS ~]#
- [root@client ~]# for i in {1..100} ;do curl -k https://10.0.0.100;curl 10.0.0.100;done
- RS2
- RS1
- [root@client ~]# ssh 10.0.0.100
- ssh: connect to host 10.0.0.100 port 22: Connection refused
- [root@LVS ~]# ipvsadm -ln
- IP Virtual Server version 1.2.1 (size=4096)
- -> 192.168.111.102:0 Route 1 0 0
- -> 192.168.111.103:0 Route 3 0 0
- [root@LVS ~]#
- [root@client ~]# for i in {1..100} ;do curl -k https://10.0.0.100;curl 10.0.0.100;done
- RS2
- RS2
- RS2
- RS2
来源: http://blog.51cto.com/13335066/2108040