背景
本篇主要讲 Redis Cluster 的安装, 让我们先用起来, 感受一下到底是怎么回事, 后面再继续学习有关如何 Cluster 如何 failover, 添加节点, 删除节点, 迁移 slots 等功能.
Redis Cluster 特点
(1)Redis Cluster 共有 16384(0-16383) 个 hash slots, 数据写入时, 根据 CRC16('key')%16384 hash slots 分配到不同的节点上;
(2) 当整个集群部分节点 crash 不影响继续使用, 如有 A,B,C 三个 master 和 A1,B1,C1 三个 slave 组成的 cluster, 如果 B 宕机, 则 B1 会自动提成为 master, 当 B 恢复之后, 也会自动成为 B1 的 slave, 保证集群的高可用性;
(3) 当集群负载比较高, 不足以支撑现在的业务, 可以添加空节点, 然后 rebalance slot 或者 reshard slot;
安装 Redis Cluster
(1) 环境准备
Redis 版本都是 3.2.9, 有 6 个 redis 实例分别在 6 台不同的主机上运行, 其中 3 个作为 Master, 另外 3 个分别作为它们的 Slave 存在, 端口统一使用默认的端口 6379;
- Master1: sht-sgmhadoopcm-01(172.16.101.54)
- Slave1: sht-sgmhadoopdn-01(172.16.101.58)
- Master2: sht-sgmhadoopnn-01(172.16.101.55)
- Slave2: sht-sgmhadoopdn-02(172.16.101.59)
- Master3: sht-sgmhadoopnn-02(172.16.101.56)
- Slave3: sht-sgmhadoopdn-03(172.16.101.60)
- # 修改配置文件, 每个节点都需要执行, 修改其他节点对应的 IP 地址
- [root@sht-sgmhadoopcm-01 redis]# vim redis.conf
- port 6379
- bind 172.16.101.54
- appendonly yes
- cluster-enabled yes
- cluster-config-file nodes-6379.conf
- cluster-node-timeout 15000
(2) 在需要 ruby 的节点上安装 ruby
- [root@sht-sgmhadoopcm-01 redis]# yum install ruby -y
- # 下面这一步可能报错, 因为 CentOS7.3 默认安装的 ruby2.0.0, 版本太低
- [root@sht-sgmhadoopcm-01 redis]# gem install redis
- Fetching: redis-4.0.1.gem (100%)
- Successfully installed redis-4.0.1
- Parsing documentation for redis-4.0.1
- Installing ri documentation for redis-4.0.1
- Done installing documentation for redis after 2 seconds
- 1 gem installed
(3) 启动所有节点
- [root@sht-sgmhadoopcm-01 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopnn-02 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopnn-01 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopdn-01 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopdn-02 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopdn-03 redis]# src/redis-server redis.conf
- [root@sht-sgmhadoopnn-02 redis]# ps -ef|grep redis
- root 1716 1 0 12:40 ? 00:00:00 src/redis-server 172.16.101.56:6379 [cluster]
- root 1720 1504 0 12:41 pts/2 00:00:00 grep --color=auto redis
(4) 使用 redis-trib.rb 脚本创建 cluster
Create 的过程中已经给三个 master 分配好了 hash slots, 并且制定好 Slave;
选项 --replicas 1 表示为每个创建的主服务器节点创建一个从服务器节点;
- [root@sht-sgmhadoopcm-01 redis]# ruby src/redis-trib.rb create --replicas 1 172.16.101.54:6379 172.16.101.55:6379 172.16.101.56:6379 172.16.101.58:6379 172.16.101.59:6379 172.16.101.60:6379
- >>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
- Using 3 masters:
- 172.16.101.54:6379
- 172.16.101.55:6379
- 172.16.101.56:6379
- Adding replica 172.16.101.58:6379 to 172.16.101.54:6379
- Adding replica 172.16.101.59:6379 to 172.16.101.55:6379
- Adding replica 172.16.101.60:6379 to 172.16.101.56:6379
- M: 610aa83831404be545b25cc7f7322e987da1dd33 172.16.101.54:6379
- slots:0-5460 (5461 slots) master
M: 4c7348ac575a1a3bd4357e937a8f94f1654193cf 172.16.101.55:6379
slots:5461-10922 (5462 slots) master
M: ece3a80f6989fa7f8316d73ee30ceea84340f24e 172.16.101.56:6379
slots:10923-16383 (5461 slots) master
S: 28e8d5c8fd12533622d8110f8d262cb50120ca02 172.16.101.58:6379
replicates 610aa83831404be545b25cc7f7322e987da1dd33
S: b50ba506525ee547823b3d9a7e5a095933e3ff42 172.16.101.59:6379
replicates 4c7348ac575a1a3bd4357e937a8f94f1654193cf
S: c79de1092011ed395ee772969274c0567c2b5dd1 172.16.101.60:6379
replicates ece3a80f6989fa7f8316d73ee30ceea84340f24e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....
- >>> Performing Cluster Check (using node 172.16.101.54:6379)
- M: 610aa83831404be545b25cc7f7322e987da1dd33 172.16.101.54:6379
- slots:0-5460 (5461 slots) master
- 1 additional replica(s)
S: b50ba506525ee547823b3d9a7e5a095933e3ff42 172.16.101.59:6379
slots: (0 slots) slave
replicates 4c7348ac575a1a3bd4357e937a8f94f1654193cf
S: 28e8d5c8fd12533622d8110f8d262cb50120ca02 172.16.101.58:6379
slots: (0 slots) slave
replicates 610aa83831404be545b25cc7f7322e987da1dd33
M: 4c7348ac575a1a3bd4357e937a8f94f1654193cf 172.16.101.55:6379
- slots:5461-10922 (5462 slots) master
- 1 additional replica(s)
M: ece3a80f6989fa7f8316d73ee30ceea84340f24e 172.16.101.56:6379
- slots:10923-16383 (5461 slots) master
- 1 additional replica(s)
S: c79de1092011ed395ee772969274c0567c2b5dd1 172.16.101.60:6379
slots: (0 slots) slave
replicates ece3a80f6989fa7f8316d73ee30ceea84340f24e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
(5) 查看 Cluster 基本信息
- [root@sht-sgmhadoopcm-01 redis]# src/redis-cli -c -h 172.16.101.54 -p 6379
- 172.16.101.54:6379> info replication
- # Replication
- role:master
- connected_slaves:1
- slave0:ip=172.16.101.58,port=6379,state=online,offset=2591,lag=0
- master_repl_offset:2591
- repl_backlog_active:1
- repl_backlog_size:1048576
- repl_backlog_first_byte_offset:2
- repl_backlog_histlen:2590
- 172.16.101.54:6379> cluster info
- cluster_state:ok
- cluster_slots_assigned:16384
- cluster_slots_ok:16384
- cluster_slots_pfail:0
- cluster_slots_fail:0
- cluster_known_nodes:6
- cluster_size:3
- cluster_current_epoch:6
- cluster_my_epoch:1
- cluster_stats_messages_sent:4462
- cluster_stats_messages_received:4462
- 172.16.101.54:6379> cluster nodes
b50ba506525ee547823b3d9a7e5a095933e3ff42 172.16.101.59:6379 slave 4c7348ac575a1a3bd4357e937a8f94f1654193cf 0 1532497136876 5 connected
610aa83831404be545b25cc7f7322e987da1dd33 172.16.101.54:6379 myself,master - 0 0 1 connected 0-5460
28e8d5c8fd12533622d8110f8d262cb50120ca02 172.16.101.58:6379 slave 610aa83831404be545b25cc7f7322e987da1dd33 0 1532497135872 4 connected
4c7348ac575a1a3bd4357e937a8f94f1654193cf 172.16.101.55:6379 master - 0 1532497133867 2 connected 5461-10922
ece3a80f6989fa7f8316d73ee30ceea84340f24e 172.16.101.56:6379 master - 0 1532497134870 3 connected 10923-16383
c79de1092011ed395ee772969274c0567c2b5dd1 172.16.101.60:6379 slave ece3a80f6989fa7f8316d73ee30ceea84340f24e 0 1532497137878 6 connected
[root@sht-sgmhadoopcm-01 redis]# cat nodes-6379.conf
b50ba506525ee547823b3d9a7e5a095933e3ff42 172.16.101.59:6379 slave 4c7348ac575a1a3bd4357e937a8f94f1654193cf 0 1532493960094 5 connected
610aa83831404be545b25cc7f7322e987da1dd33 172.16.101.54:6379 myself,master - 0 0 1 connected 0-5460
28e8d5c8fd12533622d8110f8d262cb50120ca02 172.16.101.58:6379 slave 610aa83831404be545b25cc7f7322e987da1dd33 0 1532493955077 4 connected
4c7348ac575a1a3bd4357e937a8f94f1654193cf 172.16.101.55:6379 master - 0 1532493958087 2 connected 5461-10922
ece3a80f6989fa7f8316d73ee30ceea84340f24e 172.16.101.56:6379 master - 0 1532493956080 3 connected 10923-16383
c79de1092011ed395ee772969274c0567c2b5dd1 172.16.101.60:6379 slave ece3a80f6989fa7f8316d73ee30ceea84340f24e 0 1532493959091 6 connected
vars currentEpoch 6 lastVoteEpoch 0
FAQ
Error1: ruby2.0.0 版本太低
- [root@sht-sgmhadoopcm-01 redis]# gem install redis
- Fetching: redis-4.0.1.gem (100%)
ERROR: Error installing redis:
redis requires Ruby version>= 2.2.2.
解决方法:
- [root@sht-sgmhadoopcm-01 redis]# ruby --version
- ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
- [root@sht-sgmhadoopcm-01 redis]# gpg --keyserver hkp:// http://keys.gnupg.net/ --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
- [root@sht-sgmhadoopcm-01 redis]# curl -sSL https://get.rvm.io/ | bash -s stable
- [root@sht-sgmhadoopcm-01 redis]# find / -name rvm -print
- /usr/local/rvm
- /usr/local/rvm/src/rvm
- /usr/local/rvm/src/rvm/bin/rvm
- /usr/local/rvm/src/rvm/lib/rvm
- /usr/local/rvm/src/rvm/scripts/rvm
- /usr/local/rvm/bin/rvm
- /usr/local/rvm/lib/rvm
- /usr/local/rvm/scripts/rvm
- [root@sht-sgmhadoopcm-01 redis]# source /usr/local/rvm/scripts/rvm
- [root@sht-sgmhadoopcm-01 redis]# rvm list known
- [root@sht-sgmhadoopcm-01 redis]# rvm install 2.4.1
- [root@sht-sgmhadoopcm-01 redis]# rvm use 2.4.1
- Using /usr/local/rvm/gems/ruby-2.4.1
- [root@sht-sgmhadoopcm-01 redis]# rvm use 2.4.1 --default
- Using /usr/local/rvm/gems/ruby-2.4.1
- [root@sht-sgmhadoopcm-01 redis]# rvm remove 2.3.4
- ruby-2.3.4 - #already gone
- Using /usr/local/rvm/gems/ruby-2.4.1
- [root@sht-sgmhadoopcm-01 redis]# ruby --version
- ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
Error2: 添加节点数据不为空
- [root@sht-sgmhadoopcm-01 redis]# src/redis-trib.rb add-node 172.16.101.66:6379 172.16.101.54:6379
- ......
[ERR] Node 172.16.101.66:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
解决方法:
(1) 删除 aof 和 rdb 文件
(2) 删除 nodes-6379.conf 配置文件
(3) 执行 flushdb and flushall
参考链接
https://redis.io/topics/cluster-tutorial
来源: http://www.bubuko.com/infodetail-2704547.html