菜萝卜子

【Redis】redis集群

Redis集群模式

既然我们都有主从模式了，为何还需要集群模式呢？

主从模式特点

数据全量存储：所有节点存储完整数据副本，主节点负责写操作，从节点异步复制数据
水平扩展限制：单个master的复制负载随数据量增长而增加，无法通过简单增加节点实现写能力扩展

集群的特点

支持海量数据的读写存储操作（水平扩展）
支持读写分离，实现高性能
支持数据高可用（一个数据丢了，还有其他数据可用）
集群自带（哨兵）故障转移机制
连接集群的时候，只需要连接任意一台可用的节点即可使用集群
槽位slot负责分配到各个物理节点。

Redis Cluster核心机制

Redis集群的槽位slot
Redis没有使用一致性hash,而且是引入了“哈希槽”的概念，哈希槽分片，每个Redis集群有16384个虚拟槽位（0-16383）
数据分布公式：slot = CRC16(key) % 16384
槽位不一定是连续的，槽位分配支持动态调整，每个节点维护槽位映射表

部署Redis集群

这里我部署三master三slave集群模式，使用两台机器模拟的方式
环境如下：

master1：192.168.100.30:6379
master2：192.168.100.30:6389
master3：192.168.100.30:6399
slave1：192.168.100.31:6379
slave2：192.168.100.31:6389
slave3：192.168.100.31:6399

修改配置

修改配置文件，所有节点配置如下集群信息，数字代表默认的行号（使用vim快速定位）

 88 bind 0.0.0.0
 112 protected-mode no
 310 daemonize yes
 342 pidfile /var/run/redis_cluster_6379.pid
 356 logfile /var/log/redis/cluster/redis_cluster6379.log
 516 dir /var/lib/redis/cluster/
 493 dbfilename dump6379.rdb
 1398 appendonly yes
 1425 appendfilename "appendonly6379.aof"
 1431 appenddirname "appendonlydir"
 1051 requirepass 123456
 1052 masterauth 123456
#集群配置
 1598 cluster-enabled yes
 1606 cluster-config-file nodes-6379.conf
 1612 cluster-node-timeout 15000

mkdir -p /var/log/redis/cluster/
mkdir -p /var/lib/redis/cluster/

按照上面的模板修改三个配置文件（因为我是一台机器部署三redis）

#快速修改模板文件
sed s/6379/6389/g redis6379.conf > redis6389.conf
sed s/6379/6399/g redis6379.conf > redis6399.conf

root@redis-slave:~/redis_cluster> ls
redis6379.conf  redis6389.conf  redis6399.conf 
#将配置文件分发
cp redis6379.conf  redis6389.conf  redis6399.conf /etc/
scp redis6379.conf  redis6389.conf  redis6399.conf 192.168.100.30:/etc/

所有节点编写一个启动脚本

root@redis-sentinel:~> cat start_redis_cluster.sh
#!/bin/bash
redis-server /etc/redis6379.conf
redis-server /etc/redis6389.conf
redis-server /etc/redis6399.conf

启动所有redis，查看启动情况

查询启动情况
#192.168.100.30节点
root@redis-master:~# ps -ef |grep redis
root      372376       1  0 15:35 ?        00:00:00 redis-server 0.0.0.0:6379 [cluster]
root      372378       1  0 15:35 ?        00:00:00 redis-server 0.0.0.0:6389 [cluster]
root      372380       1  0 15:35 ?        00:00:00 redis-server 0.0.0.0:6399 [cluster]

#192.168.100.31节点
root@redis-slave:~# ps -ef |grep redis
root      209847       1  0 15:32 ?        00:00:01 redis-server 0.0.0.0:6379 [cluster]
root      209849       1  0 15:32 ?        00:00:01 redis-server 0.0.0.0:6389 [cluster]
root      209860       1  0 15:32 ?        00:00:01 redis-server 0.0.0.0:6399 [cluster]

创建集群

创建集群命令如下

redis-cli --cluster create <ip1:port1> <ip2:port2> ... --cluster-replicas <num>

redis-cli -a 123456 --cluster create --cluster-replicas 1 192.168.100.30:6379 192.168.100.30:6389 192.168.100.30:6399 192.168.100.31:6379 192.168.100.31:6389 192.168.100.31:6399

执行结果如下

root@redis-sentinel:~# redis-cli -a 123456 --cluster create --cluster-replicas 1 192.168.100.30:6379 192.168.100.30:6389 192.168.100.30:6399 192.168.100.31:6379 192.168.100.31:6389 192.168.100.31:6399
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.100.31:6399 to 192.168.100.30:6379
Adding replica 192.168.100.30:6399 to 192.168.100.31:6379
Adding replica 192.168.100.31:6389 to 192.168.100.30:6389
M: a15bf78a91754d338865c8b12318ada86a7286a1 192.168.100.30:6379
   slots:[0-5460] (5461 slots) master
M: bd3ebec253530977ded2e5b500c9574b26c95153 192.168.100.30:6389
   slots:[10923-16383] (5461 slots) master
S: 9cc1277bde188dee1159cf2893a78cc463962101 192.168.100.30:6399
   replicates dede6f3d307350d1a53c29ab6e2e5be8f74b08c5
M: dede6f3d307350d1a53c29ab6e2e5be8f74b08c5 192.168.100.31:6379
   slots:[5461-10922] (5462 slots) master
S: 41c4dde02ec5858eb0487d969e6a48310ff92b78 192.168.100.31:6389
   replicates bd3ebec253530977ded2e5b500c9574b26c95153
S: 918452535acb916e8ae30103ce083ffc0184b0d7 192.168.100.31:6399
   replicates a15bf78a91754d338865c8b12318ada86a7286a1
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 192.168.100.30:6379)
M: a15bf78a91754d338865c8b12318ada86a7286a1 192.168.100.30:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 9cc1277bde188dee1159cf2893a78cc463962101 192.168.100.30:6399
   slots: (0 slots) slave
   replicates dede6f3d307350d1a53c29ab6e2e5be8f74b08c5
M: bd3ebec253530977ded2e5b500c9574b26c95153 192.168.100.30:6389
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 918452535acb916e8ae30103ce083ffc0184b0d7 192.168.100.31:6399
   slots: (0 slots) slave
   replicates a15bf78a91754d338865c8b12318ada86a7286a1
S: 41c4dde02ec5858eb0487d969e6a48310ff92b78 192.168.100.31:6389
   slots: (0 slots) slave
   replicates bd3ebec253530977ded2e5b500c9574b26c95153
M: dede6f3d307350d1a53c29ab6e2e5be8f74b08c5 192.168.100.31:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

检查集群状态

查看当前实例的主从关系

127.0.0.1:6389> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.100.31,port=6389,state=online,offset=14999,lag=1
master_failover_state:no-failover
master_replid:e4872ac3e94b1a3b5aa2982d5cf9650f1115367c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:14999
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:14999

故障转移机制

down掉主节点，主节点对应的从节点会变成主节点
重新启动原来的主节点，原来的主节点会变成从节点

查询节点信息

查看节点信息（对应关系和槽位信息）

127.0.0.1:6389> cluster nodes
a15bf78a91754d338865c8b12318ada86a7286a1 192.168.100.30:6379@16379 master - 0 1745048730268 1 connected 0-5460
9cc1277bde188dee1159cf2893a78cc463962101 192.168.100.30:6399@16399 slave dede6f3d307350d1a53c29ab6e2e5be8f74b08c5 0 1745048730000 4 connected
41c4dde02ec5858eb0487d969e6a48310ff92b78 192.168.100.31:6389@16389 slave bd3ebec253530977ded2e5b500c9574b26c95153 0 1745048729257 2 connected
bd3ebec253530977ded2e5b500c9574b26c95153 192.168.100.30:6389@16389 myself,master - 0 0 2 connected 10923-16383
918452535acb916e8ae30103ce083ffc0184b0d7 192.168.100.31:6399@16399 slave a15bf78a91754d338865c8b12318ada86a7286a1 0 1745048730773 1 connected
dede6f3d307350d1a53c29ab6e2e5be8f74b08c5 192.168.100.31:6379@16379 master - 0 1745048729056 4 connected 5461-10922

#或者使用下面的命令
redis-cli -a 123456 --cluster check 192.168.100.30:6379

查集群的信息

127.0.0.1:6389> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_ping_sent:659
cluster_stats_messages_pong_sent:656
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:1316
cluster_stats_messages_ping_received:656
cluster_stats_messages_pong_received:660
cluster_stats_messages_received:1316
total_cluster_links_buffer_limit_exceeded:0

测试写入

如果连接集群添加数据时，槽位不匹配，会出现如下报错

127.0.0.1:6389> set k8 1
(error) MOVED 8331 192.168.100.31:6379

客户端通过MOVED错误自动重定向（需使用-c参数）,会自动切换到对应的节点操作（注意下面的IP与端口的变化）

127.0.0.1:6399> set k1 2
-> Redirected to slot [12706] located at 192.168.100.30:6389
OK
192.168.100.30:6389> set k1 3
OK
192.168.100.30:6389> set k2 3
-> Redirected to slot [449] located at 192.168.100.30:6379
OK
192.168.100.30:6379> set k1 3
-> Redirected to slot [12706] located at 192.168.100.30:6389
OK
192.168.100.30:6389> set k9 3
OK
192.168.100.30:6389> set k10 3
OK
192.168.100.30:6389> set k11 3
OK
192.168.100.30:6389> set k112 3
-> Redirected to slot [5613] located at 192.168.100.31:6379
OK

扩容集群

准备配置文件

准备一份新的配置文件，两个节点都添加一台机器，添加一主一从

cp redis6400.conf /etc/
scp redis6400.conf [email protected]:/etc/

启动新的redis

默认情况下启动，新加入的状态是master

#启动机器
redis-server /etc/redis6400.conf

#查看状态
root@redis-slave:~/redis_cluster# redis-cli -a 123456 -p 6400
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6400> info replication
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:115f77bd2d4e7f426050825ead2909a353d71b4a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

加入集群

将192.168.100.30:6400 作为主加入集群

机器要加入集群，必须是一个空的库，里面没有数据

redis-cli -a 123456 --cluster add-node <新节点ip:port> <原集群ip:port>
redis-cli -a 123456 --cluster add-node 192.168.100.30:6400 192.168.100.30:6379

检查集群状态，可以看到92.168.100.30:6400作为Master加入了集群

root@redis-master:~# redis-cli -a 123456 --cluster check 192.168.100.30:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.31:6399 (91845253...) -> 2 keys | 5461 slots | 1 slaves.
192.168.100.30:6400 (3f342966...) -> 0 keys | 0 slots | 0 slaves.
192.168.100.30:6389 (bd3ebec2...) -> 4 keys | 5461 slots | 1 slaves.
192.168.100.31:6379 (dede6f3d...) -> 3 keys | 5462 slots | 1 slaves.
[OK] 9 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.100.30:6379)
S: a15bf78a91754d338865c8b12318ada86a7286a1 192.168.100.30:6379
   slots: (0 slots) slave
   replicates 918452535acb916e8ae30103ce083ffc0184b0d7
M: 918452535acb916e8ae30103ce083ffc0184b0d7 192.168.100.31:6399
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 3f3429665c4a7f11793ca90cfee375a6340fb894 192.168.100.30:6400
   slots: (0 slots) master
M: bd3ebec253530977ded2e5b500c9574b26c95153 192.168.100.30:6389
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9cc1277bde188dee1159cf2893a78cc463962101 192.168.100.30:6399
   slots: (0 slots) slave
   replicates dede6f3d307350d1a53c29ab6e2e5be8f74b08c5
S: 41c4dde02ec5858eb0487d969e6a48310ff92b78 192.168.100.31:6389
   slots: (0 slots) slave
   replicates bd3ebec253530977ded2e5b500c9574b26c95153
M: dede6f3d307350d1a53c29ab6e2e5be8f74b08c5 192.168.100.31:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

从上面的内容来看，目前的环境为4主3从
但是此时3主，已经分配完所有的slot号了
使用命令重新分配slot

redis-cli --cluster reshard <新master的ip:port>

分配多少个slots：4096
接收的目标hashID：3f3429665c4a7f11793ca90cfee375a6340fb894
从哪个主机匀出来： all