Redis step analysis sentinel cluster

This article mainly introduces the implementation steps of Redis sentinel cluster. The article introduces it in detail through sample code. It has certain reference learning value for everyone's study or work. Let's take a look at it together. ,I hope everyone has to help.

Recommended learning: Redis video tutorial

1. Overview of Redis sentinel cluster

(1) Redis Sentinel Overview

*Sentinel Sentinel: This is a distributed system. This process is used to monitor the working status of the Master server in the Redis cluster. When the Master server fails, it can realize the control of the Master and Slave servers. Second-level switching ensures that the system has a Master server, which provides high availability of the Redis cluster. It was added in the Redis2.6 version and became stable after the 2.8 version.

Redis Sentinel and Redis The difference between master and slave:

Redis Sentinel: After the master server fails, there will be a slave server to replace the master server

Redis Master-slave: After the master server fails, the slave server will not do anything

(2) Working mechanism of Redis Sentinel

The sentry only needs to be deployed on the master server

Worker process:

Monitoring: Sentinel will continuously check whether each server in the cluster is operating normally through gossip protocols

Notification: When a redis monitored by Sentinel When there is a problem with the server, Sentinel can send notifications to the administrator or other applications through the API (Application Programming Interface)

Automatic failover: If a Master server in the cluster fails, Sentinel will start an automatic fault migration operation through the voting protocol (Agreement Protocols). It will select a Slave server with more complete data to upgrade to the master server. When the client tries to connect When the Master server fails, the cluster will also return the address of the new Master server to the client, so that the cluster can use the current Master to replace the failed Master.

After the Master and Slave are switched, the contents of the Master's redis main configuration file, the Slave's redis main configuration file and the sentinel's configuration file will all change accordingly, that is, the original Master's redis The main configuration file will have one more line of Slave server configuration, and then the monitoring target of Sentinel will be changed to the current Master server

(3) Sentinel's three scheduled monitoring tasks

Every 10 seconds, each Sentinel node will send the info command to the master node and slave node to obtain the information of the Redis data node

Function:

Obtain the information of the slave node by executing the info command on the master node. This is why the Sentinel node does not need to explicitly configure the monitoring slave node. When a new slave node joins, it can be detected immediately. When the node is unreachable or after failover, the node topology information can be updated in real time through the info command.

Every 1 second, each Sentinel node will send a ping command to the master node and slave node for a heartbeat check to confirm whether these nodes are currently reachableIf the master node hangs up, sentinel will select one of the remaining slave nodes with relatively complete data to be the master node

2. Deploy the Redis Sentinel System

(1) Experimental environment

##Centos7.4192.168.100.202masterRedis-5.0.4Redis: 6379 Sentinel: 26379Master##Centos7.4Centos7.4

（2）实验步骤 -在每台服务器上都安装Redis

安装步骤相同，主机名、ip不同，下面只写Master配置

[root@Centos7 ~]# hostnamectl set-hostname master
[root@Centos7 ~]# su
[root@master ~]# systemctl stop firewalld
[root@master ~]# setenforce 0
setenforce: SELinux is disabled
[root@master ~]# mount /dev/cdrom /mnt/
mount: /dev/sr0 写保护，将以只读方式挂载
mount: /dev/sr0 已经挂载或 /mnt 忙
       /dev/sr0 已经挂载到 /mnt 上
[root@master ~]# ll
总用量 1928
-rw-------. 1 root root    1264 1月  12 18:27 anaconda-ks.cfg
-rw-r--r--  1 root root 1966337 6月   9 01:16 redis-5.0.4.tar.gz
[root@master ~]# tar xf redis-5.0.4.tar.gz
[root@master ~]# cd redis-5.0.4
[root@master redis-5.0.4]# make
[root@master redis-5.0.4]# mkdir -p /usr/local/redis
[root@master redis-5.0.4]# cp /root/redis-5.0.4/src/redis-server /usr/local/redis/
[root@master redis-5.0.4]# cp /root/redis-5.0.4/src/redis-cli /usr/local/redis/
[root@master redis-5.0.4]# cp /root/redis-5.0.4/redis.conf  /usr/local/redis/ 
[root@master redis-5.0.4]# vim /usr/local/redis/redis.conf   #修改
。。。。。。
  68 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  69 bind 192.168.100.202  #修改为本机地址，如果为127.0.0.1就只能本机访问
  70 
。。。。。。
  87 # are explicitly listed using the "bind" directive.
  88 protected-mode no  #关闭redis的保护模式，如果为yes的话其他客户端就无法连接到此服务器
  89 
。。。。。。
 135 # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
 136 daemonize yes  #开启redis的后台守护程序，即在redis开启之后是放在后台运行的
 137 
。。。。。。
 262 # Note that you must specify a directory here, not a file name.
 263 dir /usr/local/redis/rdb
 264 
。。。。。。
 506 #
 507 requirepass 123123  #去掉注释，修改redis的密码为123123
 508 
#保存退出
[root@slave2 redis-5.0.4]# mkdir /usr/local/redis/rdb
[root@master redis-5.0.4]# vim /etc/init.d/redis
#!/bin/sh
# chkconfig: 2345 80 90
# description: Start and Stop redis
#PATH=/usr/local/bin:/sbin:/usr/bin:/bin
REDISPORT=6379
EXEC=/usr/local/redis/redis-server
REDIS_CLI=/usr/local/redis/redis-cli
PIDFILE=/var/run/redis_6379.pid
CONF="/usr/local/redis/redis.conf"
AUTH="123123"
LISTEN_IP=$(netstat -utpln |grep redis-server |awk '{print $4}'|awk -F':' '{print $1}')

case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                $EXEC $CONF
        fi
        if [ "$?"="0" ]
        then
              echo "Redis is running..."
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                $REDIS_CLI -h $LISTEN_IP -p $REDISPORT -a $AUTH SHUTDOWN
                while [ -x ${PIDFILE} ]
               do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
   restart|force-reload)
        ${0} stop
        ${0} start
        ;;
  *)
    echo "Usage: /etc/init.d/redis {start|stop|restart|force-reload}" >&2
        exit 1
esac
[root@master redis-5.0.4]# chkconfig --add redis
[root@master redis-5.0.4]# chmod 755 /etc/init.d/redis
[root@master redis-5.0.4]# ln -s /usr/local/redis/* /usr/local/bin/
[root@master redis-5.0.4]# /etc/init.d/redis start 
Starting Redis server...
5233:C 09 Jun 2021 01:25:53.069 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5233:C 09 Jun 2021 01:25:53.069 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=5233, just started
5233:C 09 Jun 2021 01:25:53.069 # Configuration loaded
Redis is running...
[root@master redis-5.0.4]# netstat -anpt | grep 6379
tcp        0      0 192.168.100.202:6379    0.0.0.0:*               LISTEN      5234/redis-server 1

-做redis主从

******（1）Master配置
[root@master redis-5.0.4]# vim /usr/local/redis/redis.conf #修改
。。。。。。
 292 #
 293  masterauth 123123  #配置主服务器密码，哨兵有一个问题，就是当主服务器坏掉，切换到从服务器时，原来的主服务器可以正常运行之后，再次加入集群是加不进去的，因为哨兵没有配置主服务器的密码，所以无法连接，所以在使用哨兵集群时，要把每台的主服务器密码都配置上，每台redis的密码最好都一样
 294 
。。。。。。
 456 #
 457  min-replicas-to-write 1 #设置slave服务器的数量，当slave服务器少于这个数量时，Master主服务器会停止接收客户端的一切写请求
 458  min-replicas-max-lag 10 #设置主服务器和从服务器之间同步数据的超时时间，当超过此时间时，master主服务器会停止客户端的一切写操作，单位为秒
 459 #
。。。。。。
[root@master redis-5.0.4]# /etc/init.d/redis restart   #重启redis
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped
Starting Redis server...
5291:C 09 Jun 2021 02:04:39.132 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5291:C 09 Jun 2021 02:04:39.132 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=5291, just started
5291:C 09 Jun 2021 02:04:39.132 # Configuration loaded
Redis is running...

******（2）Slave1配置
[root@slave1 redis-5.0.4]# vim /usr/local/redis/redis.conf 
。。。。。。
 285 #
 286 replicaof 192.168.100.202 6379  #在从服务器上指定主服务器的ip和端口
 287 
。。。。。。
 292 #
 293 masterauth 123123  #指定主服务器上redis的密码
 294
。。。。。。
#保存退出
[root@slave redis-5.0.4]# /etc/init.d/redis restart  #重启服务
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped
Starting Redis server...
5304:C 09 Jun 2021 02:11:32.241 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5304:C 09 Jun 2021 02:11:32.241 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=5304, just started
5304:C 09 Jun 2021 02:11:32.241 # Configuration loaded
Redis is running...

******（3）Slave2配置
[root@slave2 redis-5.0.4]# vim /usr/local/redis/redis.conf 
。。。。。。
 286  replicaof 192.168.100.204 6379
 287 
 288 # If the master is password protected (using the "requirepass" configuration
 289 # directive below) it is possible to tell the replica to authenticate before
 290 # starting the replication synchronization process, otherwise the master will
 291 # refuse the replica request.
 292 #
 293  masterauth 123123
 294 
。。。。。。
#保存退出
[root@slave2 redis-5.0.4]# /etc/init.d/redis restart 
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped
Starting Redis server...
5253:C 09 Jun 2021 17:50:25.680 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5253:C 09 Jun 2021 17:50:25.680 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=5253, just started
5253:C 09 Jun 2021 17:50:25.680 # Configuration loaded
Redis is running...

******（3）验证主从是否成功
[root@master ~]# redis-cli -h 192.168.100.202 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.202:6379> set aaa bbb
OK
192.168.100.202:6379> set bbb ccc 
OK
192.168.100.202:6379> keys *
1) "aaa"
2) "bbb"

[root@slave1 ~]# redis-cli -h 192.168.100.203 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.203:6379> keys *
1) "bbb"
2) "aaa"
192.168.100.203:6379> set ttt fff
(error) READONLY You can't write against a read only replica.  #从服务器无法写入数据

[root@slave2 redis-5.0.4]# redis-cli -h 192.168.100.204 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.204:6379> keys *
1) "aaa"
2) "bbb"
192.168.100.204:6379> set ggg aaa
(error) READONLY You can't write against a read only replica.

#主从配置完成

-配置哨兵

******（1）在master上配置sentinel哨兵
[root@master ~]# cp redis-5.0.4/src/redis-sentinel /usr/local/redis/  #复制哨兵启动脚本
[root@master ~]# cp redis-5.0.4/sentinel.conf /usr/local/redis/  #复制哨兵配置文件
[root@master ~]# mkdir -p /var/redis/data  #创建日志文件存放位置	
[root@master ~]# vim /usr/local/redis/sentinel.conf   #修改哨兵配置文件
。。。。。。
 21 port 26379  #指定端口默认为26379
 22 
 23 # By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
 24 # Note that Redis will write a pid file in /var/run/redis-sentinel.pid when
 25 # daemonized.
 26 daemonize yes  #yes为放在后台运行，使用no放在前台运行可以看到主从切换时候的信息
 27 
。。。。。。
 64 # unmounting filesystems.
 65 dir /var/redis/data  #指定日志存放位置，就是刚才创建的路径
 66 
。。。。。。
 83 # The valid charset is A-z 0-9 and the three characters ".-_".
 84 sentinel monitor mymaster 192.168.100.202  6379 1  #指定用户为mymaster，ip为202，端口为6379，1表示当有一台master出现故障时，就进行切换
 85 
 86 # sentinel a
。。。。。。
112 # Default is 30 seconds.
113 sentinel down-after-milliseconds mymaster 3000  #指定master的失效时间，单位为毫秒3000为3秒，表示master超过3秒没响应就判定为故障
114 
。。。。。。
145 # Default is 3 minutes.
146 sentinel failover-timeout mymaster 180000  #切换操作完成的超时时间，单位为毫秒180000为180秒，在主从切换超过这个时间就判定为切换失败
147 
148 # SCRIPTS EXE
。。。。。。
102 #
103  sentinel auth-pass mymaster 123123  #连接master和slave的密码
104 sentinel config-epoch mymaster  1  #切换后最多有多少节点可以于新的master进行同步数据
105 
#保存退出
[root@master ~]# /usr/local/redis/redis-sentinel /usr/local/redis/sentinel.conf  #启动哨兵
1118:X 09 Jun 2021 18:09:29.027 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1118:X 09 Jun 2021 18:09:29.027 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1118, just started
1118:X 09 Jun 2021 18:09:29.027 # Configuration loaded
[root@master ~]# netstat -anpt | grep 26379
tcp        0      0 0.0.0.0:26379           0.0.0.0:*               LISTEN      1119/redis-sentinel 
tcp6       0      0 :::26379                :::*                    LISTEN      1119/redis-sentinel 
[root@master ~]# kill -9 1119  #先关闭哨兵
[root@master ~]# netstat -anpt | grep 26379
[root@master ~]# sed -i '26s/yes/no/g' /usr/local/redis/sentinel.conf  #修改为前台启动
[root@master ~]# /usr/local/redis/redis-sentinel /usr/local/redis/sentinel.conf  #再次开启哨兵，稍等一段时间会有提示
1129:X 09 Jun 2021 18:11:02.585 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1129:X 09 Jun 2021 18:11:02.585 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1129, just started
1129:X 09 Jun 2021 18:11:02.585 # Configuration loaded
1129:X 09 Jun 2021 18:11:02.586 * Increased maximum number of open files to 10032 (it was originally set to 1024).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.4 (00000000/0) 64 bit
  .-`` .-".  "\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 1129
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

1129:X 09 Jun 2021 18:11:02.586 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1129:X 09 Jun 2021 18:11:02.586 # Sentinel ID is fce7776020cf12792fd239f6f9d34f2d3fdef98c
1129:X 09 Jun 2021 18:11:02.586 # +monitor master mymaster 192.168.100.202 6379 quorum 1
1129:X 09 Jun 2021 18:18:04.434 * +reboot slave 192.168.100.204:6379 192.168.100.204 6379 @ mymaster 192.168.100.202 6379  #看到新增两条消息，从服务器增加了203和204主服务器时202
1129:X 09 Jun 2021 18:18:14.478 * +reboot slave 192.168.100.203:6379 192.168.100.203 6379 @ mymaster 192.168.100.202 6379

#哨兵配置完成

-测试哨兵的故障切换

******（1）把master服务器在开启一个终端，在新开启的终端中关闭redis，测试是否可以主从切换
[root@master ~]# /etc/init.d/redis stop 
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped

******（2）切换到开启哨兵的终端，查看新弹出的信息
1129:X 09 Jun 2021 18:20:36.588 # +failover-end master mymaster 192.168.100.202 6379
1129:X 09 Jun 2021 18:20:36.588 # +switch-master mymaster 192.168.100.202 6379 192.168.100.203 6379
1129:X 09 Jun 2021 18:20:36.588 * +slave slave 192.168.100.204:6379 192.168.100.204 6379 @ mymaster 192.168.100.203 6379  #发现主服务器变成了203
1129:X 09 Jun 2021 18:20:36.588 * +slave slave 192.168.100.202:6379 192.168.100.202 6379 @ mymaster 192.168.100.203 6379
1129:X 09 Jun 2021 18:20:39.607 # +sdown slave 192.168.100.202:6379 192.168.100.202 6379 @ mymaster 192.168.100.203 6379‘

******（3）在203上测试主从复制是否可以正常同步
[root@slave1 ~]# redis-cli -h 192.168.100.203 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.203:6379> keys *
1) "aaa"
2) "bbb"
192.168.100.203:6379> set yyy aaa
OK
192.168.100.203:6379> keys *
1) "yyy"
2) "aaa"
3) "bbb"

[root@slave2 redis-5.0.4]# redis-cli -h 192.168.100.204 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.204:6379> keys *  #发现同步成功
1) "yyy"
2) "bbb"
3) "aaa"

******（4）此时重新开启202的redis，并且查看哨兵的提示消息
[root@master ~]# /etc/init.d/redis start 
Starting Redis server...
1167:C 09 Jun 2021 18:23:39.756 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1167:C 09 Jun 2021 18:23:39.756 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1167, just started
1167:C 09 Jun 2021 18:23:39.756 # Configuration loaded
Redis is running...

1129:X 09 Jun 2021 18:23:50.324 * +convert-to-slave slave 192.168.100.202:6379 192.168.100.202 6379 @ mymaster 192.168.100.203 6379   #提示增加了一台slave

******（5）在202的新终端上查看redis的数据是否成功同步
[root@master ~]# redis-cli -h 192.168.100.202 -a 123123
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.202:6379> keys *  #发现已经成功同步
1) "bbb"
2) "aaa"
3) "yyy"

#测试故障切换缓存，发现在master主机出现故障然后重新连接到集群后，master角色不会进行转移

-哨兵日志分析

#把哨兵放在前台运行时，日志信息会直接输出到终端上，放到后台运行时，日志会写到指定的路径中
+reset-master <instance details>   #当master被重置时.
+slave <instance details>  #当检测到一个slave并添加进slave列表时.
+failover-state-reconf-slaves <instance details> #Failover状态变为reconf-slaves状态时
+failover-detected <instance details> #当failover发生时
+slave-reconf-sent <instance details> #sentinel发送SLAVEOF命令把它重新配置时
+slave-reconf-inprog <instance details> #slave被重新配置为另外一个master的slave，但数据复制还未发生时。
+slave-reconf-done <instance details> #slave被重新配置为另外一个master的slave并且数据复制已经与master同步时。
-dup-sentinel <instance details> #删除指定master上的冗余sentinel时,当一个sentinel重新启动时，可能会发生这个事件
+sentinel <instance details> #当master增加了一个sentinel时。
+sdown <instance details> #进入SDOWN状态时;
-sdown <instance details> #离开SDOWN状态时。
+odown <instance details> #进入ODOWN状态时。
-odown <instance details> #离开ODOWN状态时。
+new-epoch <instance details>  #当前配置版本被更新时。
+try-failover <instance details> #达到failover条件，正等待其他sentinel的选举。
+elected-leader <instance details> #被选举为去执行failover的时候。
+failover-state-select-slave <instance details> #开始要选择一个slave当选新master时。
no-good-slave <instance details> #没有合适的slave来担当新master
selected-slave <instance details> #找到了一个适合的slave来担当新master
failover-state-send-slaveof-noone <instance details> #当把选择为新master的slave的身份进行切换的时候。
failover-end-for-timeout <instance details>   #failover由于超时而失败时。
failover-end <instance details> #failover成功完成时。
switch-master <master name> <oldip> <oldport> <newip> <newport> #当master的地址发生变化时。通常这是客户端最感兴趣的消息了。
+tilt #进入Tilt模式。
-tilt #退出Tilt模式。

推荐学习：Redis视频教程

System	ip	Host name	Redis version	Port	Play role
192.168.100.203	slave1	Redis-5.0.4	Redis:6379	Slave
192.168.100.204	slave2	Redis-5.0.4	Redis: 6379	Slave

The above is the detailed content of Redis step analysis sentinel cluster. For more information, please follow other related articles on the PHP Chinese website!