An in-depth analysis of sentinel failover in Redis

This article will take you through the failover (sentinel) in Redis, I hope it will be helpful to you!

When two or more Redis instances form a master-slave relationship, the cluster they form has a certain degree of high availability: when the master fails, the slave can become The new master provides external reading and writing services, and this operating mechanism is called failover. [Related recommendations: Redis Video Tutorial]

So who will discover the master’s fault and make failover decisions?

One way is to maintain a daemo process and monitor all master-slave nodes, as shown in the following figure:

There are One master and two slaves, this daemon process monitors these three nodes. However, the daemon is a single node and its availability cannot be guaranteed. Multiple daemons need to be introduced, as shown in the following figure:

Multiple daemons solve the availability problem, but consistency problems arise. How to reach an agreement on whether a certain master is available? ? For example, in the above figure, the two networks of daemon1 and master are blocked, but the connection between daemon and master is smooth. Does the master node need to failover at this time?

Redis'ssentinel provides a set of interaction mechanisms between multiple daemons. Multiple daemons form a cluster and become a sentinel cluster. Daemon nodes are also called sentinel nodes. As shown in the figure below:

These nodes communicate, elect, and negotiate with each other, and the fault of the master node is found,# Demonstrate consistency in ##failover decisions.

The sentinel cluster monitors any number of masters and slaves under the master, and automatically upgrades the offline master from a slave under it to a new master instead of continuing to process command requests.

Start and initialize Sentinel

To start a Sentinel, you can use the command:

./redis-sentinel ../sentinel.conf

or the command:

./redis-server ../sentinel.conf --sentinel

When a Sentinel is started , it needs to perform the following steps:

Initialize the server

Sentinel is essentially a Redis server running in a special mode. It is different from ordinary The Redis servers perform different tasks, and the initialization process is not exactly the same. For example, ordinary Redis server initialization will load RDB or AOF files to restore data, but Sentinel will not load it when it starts because Sentinel does not use a database.

Replace the code used by ordinary Redis servers with Sentinel-specific codes

Replace some of the codes used by ordinary Redis servers with Sentinel-specific codes. For example, an ordinary Redis server uses server.c/redisCommandTable as the server's command table:

truct redisCommand redisCommandTable[] = {
    {"module",moduleCommand,-2,"as",0,NULL,0,0,0,0,0},
    {"get",getCommand,2,"rF",0,NULL,1,1,1,0,0},
    {"set",setCommand,-3,"wm",0,NULL,1,1,1,0,0},
    {"setnx",setnxCommand,3,"wmF",0,NULL,1,1,1,0,0},
    {"setex",setexCommand,4,"wm",0,NULL,1,1,1,0,0},
    {"psetex",psetexCommand,4,"wm",0,NULL,1,1,1,0,0},
    {"append",appendCommand,3,"wm",0,NULL,1,1,1,0,0},
    .....
    {"del",delCommand,-2,"w",0,NULL,1,-1,1,0,0},
    {"unlink",unlinkCommand,-2,"wF",0,NULL,1,-1,1,0,0},
    {"exists",existsCommand,-2,"rF",0,NULL,1,-1,1,0,0},
    {"setbit",setbitCommand,4,"wm",0,NULL,1,1,1,0,0},
    {"getbit",getbitCommand,3,"rF",0,NULL,1,1,1,0,0},
    {"bitfield",bitfieldCommand,-2,"wm",0,NULL,1,1,1,0,0},
    {"setrange",setrangeCommand,4,"wm",0,NULL,1,1,1,0,0},
    {"getrange",getrangeCommand,4,"r",0,NULL,1,1,1,0,0},
    {"substr",getrangeCommand,4,"r",0,NULL,1,1,1,0,0},
    {"incr",incrCommand,2,"wmF",0,NULL,1,1,1,0,0},
    {"decr",decrCommand,2,"wmF",0,NULL,1,1,1,0,0},
    {"mget",mgetCommand,-2,"rF",0,NULL,1,-1,1,0,0},
    {"rpush",rpushCommand,-3,"wmF",0,NULL,1,1,1,0,0},
    {"lpush",lpushCommand,-3,"wmF",0,NULL,1,1,1,0,0}
    ......
    }

Sentinel uses sentinel.c/sentinelcmds as the server list, as shown below:

struct redisCommand sentinelcmds[] = {
    {"ping",pingCommand,1,"",0,NULL,0,0,0,0,0},
    {"sentinel",sentinelCommand,-2,"",0,NULL,0,0,0,0,0},
    {"subscribe",subscribeCommand,-2,"",0,NULL,0,0,0,0,0},
    {"unsubscribe",unsubscribeCommand,-1,"",0,NULL,0,0,0,0,0},
    {"psubscribe",psubscribeCommand,-2,"",0,NULL,0,0,0,0,0},
    {"punsubscribe",punsubscribeCommand,-1,"",0,NULL,0,0,0,0,0},
    {"publish",sentinelPublishCommand,3,"",0,NULL,0,0,0,0,0},
    {"info",sentinelInfoCommand,-1,"",0,NULL,0,0,0,0,0},
    {"role",sentinelRoleCommand,1,"l",0,NULL,0,0,0,0,0},
    {"client",clientCommand,-2,"rs",0,NULL,0,0,0,0,0},
    {"shutdown",shutdownCommand,-1,"",0,NULL,0,0,0,0,0},
    {"auth",authCommand,2,"sltF",0,NULL,0,0,0,0,0}
}

Initialize Sentinel state

The server will initialize a sentinel.c/sentinelState structure (save all status related to Sentinel functions in the server).

struct sentinelState {
 
    char myid[CONFIG_RUN_ID_SIZE+1]; /* This sentinel ID. */
    
    //当前纪元，用于实现故障转移
    uint64_t current_epoch;         /* Current epoch. */
    
    //监视的主服务器
    //字典的键是主服务器的名字
    //字典的值则是一个指向sentinelRedisInstances结构的指针
    dict *masters;      /* Dictionary of master sentinelRedisInstances.
                           Key is the instance name, value is the
                           sentinelRedisInstance structure pointer. */
    //是否进入tilt模式
    int tilt;           /* Are we in TILT mode? */
    
    //目前正在执行的脚本数量
    int running_scripts;    /* Number of scripts in execution right now. */
    
    //进入tilt模式的时间
    mstime_t tilt_start_time;       /* When TITL started. */
    
    //最后一次执行时间处理器的时间
    mstime_t previous_time;         /* Last time we ran the time handler. */
    
    // 一个FIFO队列，包含了所有需要执行的用户脚本
    list *scripts_queue;            /* Queue of user scripts to execute. */
    
    char *announce_ip;  /* IP addr that is gossiped to other sentinels if
                           not NULL. */
    int announce_port;  /* Port that is gossiped to other sentinels if
                           non zero. */
    unsigned long simfailure_flags; /* Failures simulation. */
    int deny_scripts_reconfig; /* Allow SENTINEL SET ... to change script
                                  paths at runtime? */
}

According to the given configuration file, initialize Sentinel's monitoring master server list

Initialization of Sentinel status will trigger an update of the masters dictionary Initialization, and the master dictionary is initialized based on the loaded Sentinel configuration file.

The key of the dictionary is the name of the monitoring main server, and the value of the dictionary is the sentinel.c/sentinelRedisInstance structure corresponding to the monitored main server.

The properties of the sentinelRedisInstance structure are as follows:

typedef struct sentinelRedisInstance {
    //标识值，记录了实例的类型，以及该实例的当前状态
    int flags;      /* See SRI_... defines */
    
    //实例的名字
    //主服务器的名字由用户在配置文件中设置
    //从服务器以及Sentinel的名字由Sentinel自动设置
    //格式为ip:port,例如“127.0.0.1:26379”
    char *name;     /* Master name from the point of view of this sentinel. */
    
    //实例运行的ID
    char *runid;    /* Run ID of this instance, or unique ID if is a Sentinel.*/
    
    //配置纪元，用于实现故障转移
    uint64_t config_epoch;  /* Configuration epoch. */
    
    //实例的地址
    sentinelAddr *addr; /* Master host. */
    
    //sentinel down-after-milliseconds选项设定的值
    //实例无响应多少毫秒之后才会被判断为主观下线（subjectively down）
    mstime_t down_after_period; /* Consider it down after that period. */
    
    //sentinel monitor <master-name> <ip> <redis-port> <quorum>选项中的quorum
    //判断这个实例为客观下线（objective down）所需的支持投票的数量
    unsigned int quorum;/* Number of sentinels that need to agree on failure. */  
    //sentinel parallel-syncs <master-name> <numreplicas> 选项的numreplicas值
    //在执行故障转移操作时，可以同时对新的主服务器进行同步的从服务器数量
    int parallel_syncs; /* How many slaves to reconfigure at same time. */
    
    //sentinel failover-timeout <master-name> <milliseconds>选项的值
    //刷新故障迁移状态的最大时限
    mstime_t failover_timeout;      /* Max time to refresh failover state. */
}

For example, when starting Sentinel, the following configuration file is configured:

# sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel monitor master1 127.0.0.1 6379 2

# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds master1 30000

# sentinel parallel-syncs <master-name> <numreplicas>
sentinel parallel-syncs master1 1

# sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout master1 900000

Then Sentinel will create the main server master1 as shown in the figure below The example structure shown:

The structure of Sentinel status and masters dictionary is as follows:

Create a network connection to the main server

Create a network connection to the monitored main server. Sentinel will become the client of the main server, send commands to the main server and obtain responses from the commands. information.

Sentinel will create two asynchronous network connections to the main server:

Command connection, used to send commands to the main server and receive command replies
Subscribe Connect and subscribe to the _sentinel_:hello channel of the main server

Sentinel发送信息和获取信息

• Sentinel默认会以每十秒一次的频率，通过命令连接向被监视的master和slave发送INFO命令。

  通过master的回复可获取master本身信息，包括run_id域记录的服务器运行ID，以及role域记录的服务器角色。另外还会获取到master下的所有的从服务器信息，包括slave的ip地址和port端口号。Sentinel无需用户提供从服务器的地址信息，由master返回的slave的ip地址和port端口号，可以自动发现slave。

  当Sentinel发现master有新的slave出现时，Sentinel会为这个新的slave创建相应的实例外，Sentinel还会创建到slave的命令连接和订阅连接。

  根据slave的INFO命令的回复，Sentinel会提取如下信息：

  1.slave的运行ID run_id

  2.slave的角色role

  3.master的ip地址和port端口

  4.master和slave的连接状态master_link_status

  5.slave的优先级slave_priority

  6.slave的复制偏移量slave_repl_offset

• Sentinel在默认情况下会以每两秒一次的频率，通过命令连接向所有被监视的master和slave的_sentinel_:hello频道发送一条信息

  发送以下格式的命令：

     PUBLISH _sentinel_:hello   "<s_ip>,<s_port>,<s_runid>,<s_epoch>,<m_name>,<m_ip>,<m_port>,<m_epoch>"

以上命令相关参数意义：

参数	意义
s_ip	Sentinel的ip地址
s_port	Sentinel的端口号
s_runid	Sentinel的运行ID
s_runid	Sentinel的运行ID
m_name	主服务器的名字
m_ip	主服务器的IP地址
m_port	主服务器的端口号
m_epoch	主服务器当前的配置纪元

• Sentinel与master或者slave建立订阅连接之后，Sentinel就会通过订阅连接发送对_sentinel_:hello频道的订阅,订阅会持续到Sentinel与服务器的连接断开为止

命令如下所示：

SUBSCRIBE sentinel:hello

如上图所示，对于每个与Sentinel连接的服务器 ，Sentinel既可以通过命令连接向服务器频道_sentinel_:hello频道发送信息，又通过订阅连接从服务器的_sentinel_:hello频道接收信息。

• sentinel间会相互感知，新加入的sentinel会向master的_sentinel_:hello频道发布一条消息，包括自己的消息，其它该频道订阅者sentinel会发现新的sentinel。随后新的sentinel和其它sentinel会创建长连接。

相互连接的各个Sentinel可以进行信息交换。Sentinel为master创建的实例结构中的sentinels字典保存了除Sentinel本身之外，所有同样监视这个主服务器的其它Sentinel信息。

前面也讲到sentinel会为slave创建实例（在master实例的slaves字典中）。现在我们也知道通过sentinel相互信息交换，也创建了其它sentinel的实例（在master实例的sentinels字典中）。我们将一个sentinel中保存的实例结构大概情况理一下，如下图所示：

从上图可以看到slave和sentinel字典的键由其ip地址和port端口组成，格式为ip:port,其字典的值为其对应的sentinelRedisInstance实例。

master的故障发现

主观不可用

默认情况下Sentinel会以每秒一次的频率向所有与它创建了命令连接的master（包括master、slave、其它Sentinel）发送PING命令，并通过实例返回的PING命令回复来判断实例是否在线。

PING命令回复分为下面两种情况：

• 有效回复：实例返回 +PONG、-LOADING、-MASTERDOWN三种回复的一种

• 无效回复：除上面有效回复外的其它回复或者在指定时限内没有任何返回

Sentinel配置文件中的设置down-after-milliseconds毫秒时效内(各个sentinel可能配置的不相同)，连续向Sentinel返回无效回复，那么sentinel将此实例置为主观下线状态，在sentinel中维护的该实例flags属性中打开SRI_S_DOWN标识，例如master如下所示：

客观不可用

在sentinel发现主观不可用状态后，它会将“主观不可用状态”发给其它sentinel进行确认，当确认的sentinel节点数>=quorum，则判定该master为客观不可用，随后进入failover流程。

上面说到将主观不可用状态发给其它sentinel使用如下命令：

SENTINEL is-master-down-by-addr <ip> <port> <current_epoch> <runid>

各个参数的意义如下：

• ip：被sentinel判断为主观下线的主服务器的ip地址
• port: 被sentinel判断为主观下线的主服务器的port地址
• current_epoch:sentinel的配置纪元，用于选举领头Sentinel
• runid：可以为*号或者Sentinel的运行ID，*号代表检测主服务器客观下线状态。Sentinel的运行ID用于选举领头Sentinel

接受到以上命令的sentinel会反回一条包含三个参数的Multi Bulk回复：

1）<down_state> 目标sentinel对该master检查结果，1:master已下线 2:master未下线

2）<leader_runid> 两种情况，*表示仅用于检测master下线状态 ，否则表示局部领头Sentinel的运行ID（选举领头Sentinel）

3）<leader_epoch> 当leader_runid为时，leader_epoch始终为0。不为时则表示目标Sentinel的局部领头Sentinel的配置纪元（用于选举领头Sentinel）

其中节点数量限制quorum为sentinel配置文件中配置的

sentinel monitor <master-name> <ip> <redis-port> <quorum>

quorum选项，不同的sentinel配置的可能不相同。

当sentinel认为master为客观下线状态，则会将master属性中的flags的SRI_O_DOWN标识打开，例如master如下图所示：

Elect Sentinel Leader

When a master goes down, multiple sentinel nodes may discover and confirm each other's "subjective unavailability status" through interaction at the same time, while achieving the "objective Unavailable status" and intends to initiate a failover. But in the end there can only be one sentinel node as the failover initiator, then Sentinel Leader needs to be elected, and a Sentinel Leader election process needs to be started.

Redis's Sentinel mechanism implements this election algorithm similar to the Raft protocol:

1. The epoch variable of sentinelState is similar to the term (election round) in the raft protocol.

2. Each sentinel node that confirms that the master is "objectively unavailable" will broadcast its own election request to the surroundings(SENTINEL is-master-down-by-addr , current_epoch is its own configuration epoch, run_id is its own running ID)

3. Each sentinel node that has received the election request has not received it yet When other candidates are requested, it will set the intention of this round as the first candidate sentinel and reply to it (First come, first served); if the intention has been expressed in this round, other candidates will be rejected. and will respond with intention (Multi Bulk reply with three parameters as introduced above, down_state is 1, leader_runid is the running ID of the source sentinel that initiated the election request received for the first time, leader_epoch is the first received election request The configuration epoch of the sentinel that is the source of the request)

4. If each sentinel node that initiates a candidate request receives more than half of the intentions to agree to a candidate sentinel (maybe itself), Then it is determined that the sentinel is the leader. If this round lasts long enough and no leader is elected, the next round will be started.

leader sentinel After confirmation, leader sentinel selects one from all slaves of the master as the new master according to certain rules.

Failover failover

After the Sentinel Leader is elected, the sentinel leader performs failover on the offline master:

• sentinel leader selects a slave that is in good condition and has complete data among all the slaves of the offline master, and then sends the:SLAVEOF no one command to this slave to change this slave is converted to master.

  Let’s take a look at how the new master is selected? The Sentinel leader will save all offline slaves into a list, and then filter according to the following rules:

• slave with the highest priority, identified by the replica-priority option in the redis.conf configuration, the default is 100, the lower the replica-priority, the higher the priority. 0 is a special priority, indicating that it cannot be upgraded to master.

• If there are multiple slaves with equal priorities, the slave with the largest copy offset will be selected (the data is more complete)

• If there are multiple slaves with equal priorities and the largest maximum replication offset, select the slave with the smallest running ID

After selecting the slave that needs to be upgraded to the new master, the Sentinel Leader will send the SLAVEOF no one command to the slave. Afterwards, Sentinel will send INFO to the upgraded slave once per second (usually once every ten seconds). When the reply role changes from slave to master, the Sentinel Leader will know that it has been upgraded to master.

• ##The sentinel leader sends the SLAVEOF command (SLAVEOF ) to the slave under the offline master to

  copy the new master .

• Set the old master as the slave of the new master and continue to monitor it. When it comes back online, Sentinel will execute the command to make it the slave of the new master.

##Summary

Sentinel is a high-availability solution for Redis. The number of nodes in the Sentinel cluster needs to be >= 3.

By default, Sentinel executes info on the master and slave every ten seconds to discover master change information, master-slave relationship and discover new slave nodes.

By default, sentinel sends a message to the _sentinel_:hello channel of all monitored masters and slaves through command connections every two seconds to interact with other sentinels

By default, sentinel sends Master, slave, and other sentinels send PING commands to determine whether the other party is offline

Sentinel Leader is elected according to certain rules.

The Sentinel Leader performs a failover operation and elects a new master to replace the offline master.

For more programming-related knowledge, please visit:

Introduction to Programming

! !

The above is the detailed content of An in-depth analysis of sentinel failover in Redis. For more information, please follow other related articles on the PHP Chinese website!