This article will introduce you to the relevant knowledge of Redis and take you through master-slave replication, Sentinel, and clustering, so as to take your Redis level to a higher level!

1. Master-slave replication

1. Introduction

Master-slave replication is the cornerstone of Redis distribution and the high availability of Redis Assure. In Redis, the server being replicated is called the master server (Master), and the server replicating the master server is called the slave server (Slave). [Related recommendations: Redis Video Tutorial]

The configuration of master-slave replication is very simple, and there are three ways (including IP-master server IP address/PORT -Main server Redis service port):

• Configuration file - redis.conf file, configure slaveof ip port

• command - enter Redis client executes slaveof ip port

• Startup parameters——./redis-server --slaveof ip port

2. Master-slave replication The evolution

The master-slave replication mechanism of Redis was not as perfect as the 6.x version at the beginning, but it was iterated from version to version. It has generally gone through three versions of iteration:

• Before 2.8

• ##2.8~4.0

• After 4.0

• Synchronization (sync): refers to updating the data status of the slave server to the current data status of the main server, which mainly occurs during initialization or subsequent full synchronization.

• Command propagate: When the data status of the master server is modified (write/delete, etc.) and the data status between the master and slave is inconsistent, the master service will change the data. The command is propagated to the slave server to bring the status between the master and slave servers back to consistency.

• The slave server receives the slaveof ip prot command sent by the client, and the slave server creates it to the master server based on ip:port Socket connection

• After the socket is successfully connected to the main server, the slave server will associate a file event handler specifically used to handle replication work with the socket connection. Subsequent RDB files and propagated commands sent by the master server

• start to be copied, and the slave server sends sync commands to the master server

• master server After receiving the sync command, execute the bgsave command. The sub-process of the main process fork of the main server will generate an RDB file and record all write operations after the RDB snapshot is generated in the buffer.

## After the #bgsave command is executed, the master server sends the generated RDB file to the slave server. After receiving the RDB file, the slave server will first clear all its own data, then load the RDB file, and update its data status to the master server. The data status of the RDB file
The master server sends the buffer write command to the slave server, and the slave server receives the command and executes it.
Master-slave replication synchronization step is completed
2.1.2 Command propagation

When the synchronization work is completed, the master-slave replication It is necessary to maintain the consistency of data status through command propagation. As shown in the figure below, after the synchronization work between the current master and slave servers is completed, the master service deletes K6 after receiving the DEL K6 instruction from the client. At this time, K6 still exists on the slave server, and the master-slave data status is inconsistent. In order to maintain the consistent status of the master and slave servers, the master server will propagate commands that cause its own data status to change to the slave server for execution. When the slave server also executes the same command, the data status between the master and slave servers will remain consistent.

2.1.3 Defects

From the above, we can’t see any flaws in the master-slave replication of versions before 2.8. This is because we have not considered network fluctuations. Brothers who understand distribution must have heard of CAP theory. CAP theory is the cornerstone of distributed storage systems. In CAP theory, P (partition network partition) must exist, and Redis master-slave replication is no exception. When a network failure occurs between the master and slave servers, resulting in the failure of communication between the slave server and the master server for a period of time. When the slave server reconnects to the master server, if the data status of the master server changes during this period, then the master-slave server Inconsistencies in data status will occur between servers. In master-slave replication versions before Redis 2.8, the way to solve this data state inconsistency is to resend the sync command. Although sync can ensure that the data status of the master and slave servers is consistent, it is obvious that sync is a very resource-consuming operation.

When the sync command is executed, the resources required by the master and slave servers:

• The master server executes BGSAVE to generate RDB files, which will occupy a large amount of CPU, disk I/O and memory resources.

• The master server sends the generated RDB file to the slave server, which will occupy a lot of network bandwidth.

• The slave server receives the RDB file and loads it , will cause the slave server to be blocked and unable to provide services

As can be seen from the above three points, the sync command will not only cause the response ability of the master server to decrease, but also cause the slave server to Refuse to provide services to outsiders.

2.2 Version 2.8-4.0

2.2.1 Improvement points

For versions before 2.8, Redis will reconnect to the slave server after 2.8 Data status synchronization has been improved. The direction of improvement is to reduce the occurrence of full resynchronization and use partial resynchronization as much as possible. After version 2.8, the psync command is used instead of the sync command to perform synchronization operations. The psync command has both full synchronization and incremental synchronization functions:

• Full synchronization with the previous version (sync) Consistent

• In incremental synchronization, different measures will be taken according to the situation for replication after disconnection and reconnection; if conditions permit, only part of the data missing from the service will still be sent.

2.2.2 How to implement psync

In order to achieve incremental synchronization after disconnection and reconnection from the server, Redis adds three auxiliary parameters:

• Replication offset

• Replication backlog

• Server running id (run id)

2.2.2.1 Replication offset

A replication offset will be maintained in both the master server and the slave server

• The master server sends data to the slave service, propagating N bytes of data, and the replication offset of the master service is increased by N

• from the slave server Receive the data sent by the master server, receive N bytes of data, and increase the replication offset of the slave server by N

The normal synchronization situation is as follows:

By comparing whether the replication offsets between the master and slave servers are equal, you can know whether the data status between the master and slave servers is consistent. Assuming that A/B propagates normally and C is disconnected from the server, the following situation will occur:

It is obvious that there is a copy offset Later, after the slave server C is disconnected and reconnected, the master server only needs to send the 100 bytes of data missing from the slave server. But how does the master server know what data is missing from the slave server?

2.2.2.2 Copy backlog buffer

The copy backlog buffer is a fixed-length queue with a default size of 1MB. When the data status of the master server changes, the master server synchronizes the data to the slave server and saves a copy to the replication backlog buffer.

In order to match the offset, the copy backlog buffer not only stores the data content, but also records the offset corresponding to each byte:

When the slave server is disconnected and reconnected, the slave server sends its replication offset (offset) to the master server through the psync command, and the master server can use this offset Use the amount to determine whether to perform incremental propagation or full synchronization.

• If the data at offset 1 is still in the copy backlog buffer, then perform incremental synchronization operation

• Otherwise, perform full synchronization Operation, consistent with sync

The default copy backlog buffer size of Redis is 1MB. How to set it if you need to customize it? Obviously, we want to use incremental synchronization as much as possible, but we don't want the buffer to occupy too much memory space. Then we can set the size of the replication backlog buffer S by estimating the reconnection time T after the Redis slave service is disconnected and the memory size M of the write commands received by the Redis master server per second.

S ​​= 2 * M * T

Note that the expansion of 2 times here is to leave a certain amount of room to ensure that most of the breaks Incremental synchronization can be used for line reconnection.

2.2.2.3 Server running ID

After seeing this, do you think that the incremental synchronization of disconnection and reconnection can already be achieved, and you also need to run the ID dry Well? In fact, there is another situation that has not been considered, that is, when the master server goes down, a slave server is elected as the new master server. In this case, we can distinguish it by comparing the running ID.

• The run ID (run id) is 40 random hexadecimal strings automatically generated when the server starts. Both the master service and the slave server will generate run IDs

• When the slave server synchronizes the data of the master server for the first time, the master server will send its running ID to the slave server, and the slave server will save it in the RDB file

• When the slave server is disconnected and reconnected, the slave server will send the previously saved master server running ID to the master server. If the server running ID matches, it proves that the master server has not changed, and you can try incremental synchronization

• If the server running ID does not match, full synchronization will be performed

2.2.3 Complete psync

The complete psync process is very complicated. It has been very perfect in the master-slave replication version of 2.8-4.0. The parameters sent by the psync command are as follows:

psync

When the slave server has not replicated any master server (it is not the first time that the master-slave replicated , because the master server may change, but the slave server is fully synchronized for the first time), the slave server will send:

psync ? -1

The complete psync process is as follows:

• Received from the server Go to SLAVEOF 127.0.0.1 6379 command

• Return OK from the server to the command initiator (this is an asynchronous operation, return OK first, and then save the address and port information)

• The slave server saves the IP address and port information to the Master Host and Master Port

• The slave server actively initiates a socket to the master server based on the Master Host and Master Port connection, and at the same time, the slave service will associate a file event handler specifically used for file copying with this socket connection for subsequent RDB file copying and other work

• Master server After receiving the socket connection request from the slave server, create a corresponding socket connection for the request, and look at the slave server as a client (in master-slave replication, the master server and the slave server are actually clients of each other. end and server)

• The socket connection is established, and the slave server actively sends a PING command to the main service. If the main server returns PONG within the specified timeout period, the socket is proved The word connection is available, otherwise disconnect and reconnect

• If the master server sets a password (masterauth), then the slave server sends the AUTH masterauth command to the master server for authentication. Note that if the slave server sends a password but the master service does not set a password, the master server will send a no password is set error; if the master server requires a password but the slave server does not send a password, the master server will send a NOAUTH error; If the passwords do not match, the master server sends an invalid password error.

• The slave server sends REPLCONF listening-port xxxx (xxxx represents the port of the slave server) to the master server. After receiving the command, the master server will save the data. When the client uses INFO replication to query the master-slave information, it can return the data.

• The slave server sends the psync command. Please see the above for this step. Figure two situations of psync

• The master server and the slave server are clients of each other, performing data requests/responses

• Master server The heartbeat packet mechanism is used between the server and the slave server to determine whether the connection is disconnected. The slave server sends a command to the master server every 1 second, REPLCONF ACL offset (replication offset of the slave server). This mechanism can ensure the correct synchronization of data between the master and slave. If the offsets are not equal, the master server will take Incremental/full synchronization measures are used to ensure consistent data status between master and slave (the choice of incremental/full depends on whether the data of offset 1 is still in the replication backlog buffer)

2.3 Version 4.0

Redis versions 2.8-4.0 still have some room for improvement. Can incremental synchronization be performed when the main server is switched? Therefore, Redis 4.0 version has been optimized to deal with this problem, and psync has been upgraded to psync2.0. pync2.0 abandoned the server running ID and used replid and replid2 instead. Replid stores the running ID of the current main server, and replid2 saves the running ID of the previous main server.

• Replication offset

• Replication backlog

• Main server running id (replid)

• Last main server running id (replid2)

We can solve the main server through replid and replid2 When switching, the problem of incremental synchronization:

• If replid is equal to the running id of the current main server, then determine the synchronization method incremental/full synchronization

• If the replicas are not equal, determine whether replicas 2 are equal (whether they belong to the slave server of the previous master server). If they are equal, you can still choose incremental/full synchronization. If they are not equal, you can only perform full synchronization.

2. Sentinel

1. Introduction

Master-slave replication lays the foundation for Redis distribution The basis, but ordinary master-slave replication cannot achieve high availability. In the ordinary master-slave replication mode, if the master server goes down, the operation and maintenance personnel can only manually switch the master server. Obviously, this solution is not advisable. In response to the above situation, Redis officially launched a high-availability solution that can resist node failures - Redis Sentinel. Redis Sentinel: A Sentinel system composed of one or more Sentinel instances. It can monitor any number of master and slave servers. When the monitored master server goes down, the master server will be automatically offline, and the slave server will be upgraded to New master server.

The following example: When the old Master's offline time exceeds the upper limit of offline time set by the user, the Sentinel system will perform a failover operation on the old Master. The failover operation includes three steps:

• Select the latest data in the Slave as the new Master

• Send new replication instructions to other Slaves to make other slave servers become the new Master's Slave

• Continue to monitor the old Master, and if it comes online, set the old Master as the Slave of the new Master

This article is based on the following resource list:

struct sentinelState {
   
    //当前纪元，故障转移使用
 uint64_t current_epoch; 
  
    // Sentinel监视的主服务器信息 
    // key -> 主服务器名称 
    // value -> 指向sentinelRedisInstance指针
    dict *masters; 
    // ...
} sentinel;

daemonize yes
port 26379
protected-mode no
dir "/usr/local/soft/redis-6.2.4/sentinel-tmp"
sentinel monitor redis-master 192.168.211.104 6379 2
sentinel down-after-milliseconds redis-master 30000
sentinel failover-timeout redis-master 180000
sentinel parallel-syncs redis-master 1

typedef struct sentinelRedisInstance {
 
    // 标识值，标识当前实例的类型和状态。如SRI_MASTER、SRI_SLVAE、SRI_SENTINEL
    int flags;
    
    // 实例名称 主服务器为用户配置实例名称、从服务器和Sentinel为ip:port
    char *name;
    
    // 服务器运行ID
    char *runid;
    
    //配置纪元，故障转移使用
 uint64_t config_epoch; 
    
    // 实例地址
    sentinelAddr *addr;
    
    // 实例判断为主观下线的时长 sentinel down-after-milliseconds redis-master 30000
    mstime_t down_after_period; 
    
    // 实例判断为客观下线所需支持的投票数 sentinel monitor redis-master 192.168.211.104 6379 2
    int quorum;
    
    // 执行故障转移操作时，可以同时对新的主服务器进行同步的从服务器数量 sentinel parallel-syncs redis-master 1
    int parallel-syncs;
    
    // 刷新故障迁移状态的最大时限 sentinel failover-timeout redis-master 180000
 mstime_t failover_timeout;
    
    // ...
} sentinelRedisInstance;

sentinel down-after-milliseconds redis-master 30000

SENTINEL is-master-down-by-addr <ip> <port> <current_epoch> <runid>

SENTINEL is-master-down-by-addr <ip> <port> <current_epoch> <runid>

typedef struct clsuterNode {

    // 创建时间
    mstime_t ctime;
    
    // 节点名字，由40位随机16进制的字符组成（与sentinel中讲的服务器运行id相同）
    char name[REDIS_CLUSTER_NAMELEN];
    
    // 节点标识，可以标识节点的角色和状态
    // 角色 -> 主节点或从节点 例如：REDIS_NODE_MASTER(主节点) REDIS_NODE_SLAVE(从节点)
    // 状态 -> 在线或下线 例如：REDIS_NODE_PFAIL(疑似下线) REDIS_NODE_FAIL(下线) 
    int flags;
    
    // 节点配置纪元，用于故障转移，与sentinel中用法类似
    // clusterState中的代表集群的配置纪元
    unit64_t configEpoch;
    
    // 节点IP地址
    char ip[REDIS_IP_STR_LEN];
    
    // 节点端口
    int port;
    
    // 连接节点的信息
    clusterLink *link;
    
    // 一个2048字节的二进制位数组
    // 位数组索引值可能为0或1
    // 数组索引i位置值为0，代表节点不负责处理槽i
    // 数组索引i位置值为1，代表节点负责处理槽i
    unsigned char slots[16384/8];
    
    // 记录当前节点处理槽的数量总和
    int numslots;
    
    // 如果当前节点是从节点
    // 指向当前从节点的主节点
    struct clusterNode *slaveof;
    
    // 如果当前节点是主节点
    // 正在复制当前主节点的从节点数量
    int numslaves;
    
    // 数组——记录正在复制当前主节点的所有从节点
    struct clusterNode **slaves;
    
} clsuterNode;

typedef struct clusterState {

    // 连接创建时间
    mstime_t ctime;
   
    // TCP 套接字描述符
    int fd;
    
    // 输出缓冲区，需要发送给其他节点的消息缓存在这里
    sds sndbuf;
    
    // 输入缓冲区，接收打其他节点的消息缓存在这里
    sds rcvbuf;
    
    // 与当前clsuterNode节点代表的节点建立连接的其他节点保存在这里
    struct clusterNode *node;
} clusterState;

typedef struct clusterState {

    // 当前节点指针，指向一个clusterNode
    clusterNode *myself;
    
    // 集群当前配置纪元，用于故障转移，与sentinel中用法类似
    unit64_t currentEpoch;
    
    // 集群状态 在线/下线
    int state;
    
    // 集群中处理着槽的节点数量总和
    int size;
    
    // 集群节点字典，所有clusterNode包括自己
    dict *node;
    
    // 集群中所有槽的指派信息
    clsuterNode *slots[16384];
    
    // 用于槽的重新分配——记录当前节点正在从其他节点导入的槽
    clusterNode *importing_slots_from[16384];
    
    // 用于槽的重新分配——记录当前节点正在迁移至其他节点的槽
    clusterNode *migrating_slots_to[16384];
    
    // ...
    
} clusterState;

typedef struct clusterState {

    // ...
    
    // 用于槽的重新分配——记录当前节点正在从其他节点导入的槽
    clusterNode *importing_slots_from[16384];
    
    // 用于槽的重新分配——记录当前节点正在迁移至其他节点的槽
    clusterNode *migrating_slots_to[16384];
    
    // ...
    
} clusterState;