Redis Sentinel Principle, I have tolerated you for a long time!

❝

There is a saying in the role of redis master-slave replication: "Master-slave replication is the cornerstone of high availability." So what is high availability? High availability is to reduce the system The time that cannot be provided is the commonly heard reference to six nines. Sentinels and clusters are essential to achieve high availability. This article mainly introduces the sentinel mechanism.

❞

Preface

##❝Kaka compiled a road map to create an interview guide , I plan to write articles according to such a roadmap, and later I found that the knowledge points that have not been added are being added. I also look forward to your partners helping to add them. See you in the comment area!

❞

Insert picture description here

The main content of this article Introducing Sentinel around the following aspects

• Sentinel introduction
• Sentinel configuration
• Working principle of sentry

Implementation environment of this article

• centos7.3 redis4.0
• redis working directory /usr/local/redis
• ##Perform simulation operation in virtual machine

1. What is a sentinel

Let’s briefly talk about how we are configuring the master-slave One situation during replication is that the master node is down. Who will provide the service!

When the master node is down, master-slave replication has no meaning. In an era where data is king, what is the point without data? High availability.

At this time, a big brother named Sentinel came out of the blue. The big brother said that I will help you deal with this problem.

Since the main node master, as the boss, will not let you play. I will select a boss from among the four of you, and then you will play with him.

When the boss who doesn't play with you comes back, his identity will be invalid, and he will no longer be your boss. He can only play with the boss I selected.

The above dialogue process is what exactly is the meaning of configuring sentinels. Whoever plays with them will give data to whom. Once we know the role of sentinels, we will continue.

"Finally, we use professional terms to explain what a sentinel is."

Sentinel, the English name is sentinel, is a distributed system used for master-slave structure Each server in monitors. When the master node fails, a new master node is selected through the voting mechanism, and all slave nodes are connected to the new master node.

2. The role of the sentinel

The dialogue process we talked about above is one of the functions of the sentinel. Automatically Failover.

When it comes to its role, it must be exactly what this sentinel does at work. Let’s first describe it using a relatively dry concept, and then we’ll talk about the working principles one by one below.

Three functions of SentinelMonitoring, notification, and automatic fault transfer

• Monitoring

  • Who is being monitored? To support the master-slave structure, one is the master node and the other is the slave node, so it must be monitoring these two.
  • Monitor whether the master node and slave nodes are running normally
  • Detect whether the master node is alive and the running status of the master node and slave nodes
• Notification

  • When there is a problem with the server detected by the Sentinel, notifications will be sent to other Sentinels. The Sentinels are equivalent to one WeChat group, every problem discovered by the sentry will be posted in this group.
• Automatic failover

  • When the main node is detected to be down, disconnect from the downed main node. of all slave nodes, select one of the slave nodes as the master node, and then connect other slave nodes to this latest master node. And inform the client of the latest server address.

One thing to note here is that Sentinel is also a redis server, but it does not provide any services to the outside world.

Configure it as an odd number when configuring the sentinel. So why configure the number of Sentinel servers to be an odd number? With this question you will find the answer you want below.

2. How to configure Sentinel

1. Preparation

In this chapter we begin to configure the sentry and prepare for the preliminary work. The picture below shows Kaka’s preparations. Start 8 clients, three sentinels, one master node, two slave nodes, one master node client, and one slave node client.

2. Interpretation of sentinel.conf configuration

The configuration file used by sentinel is sentinel.confLet's interpret the sentinel.conf configuration informationBut most of them are comments. Here Kaka provides you with a command to filter these useless informationcat sentinel.conf | grep -v '#' | grep -v '^$'

• port 26379: External service port number
• dir /tmp: Storage of sentinel work information
• ##sentinel monitor mymaster 127.0.0.1 6379 2: Who is being monitored? The name can be customized. The 2 behind it represents that if two sentinels determine that the master node is down, then the master node will be down. It is usually set to the number of sentinels. Half plus one.
• sentinel down-after-milliseconds mymaster 30000: The sentinel down-after-milliseconds mymaster 30000: How long does it take for the sentinel to connect to the master node without responding, which means it hangs. The following 30000 is milliseconds, which is 30 seconds.
• sentinel parallel-syncs mymaster 1: This configuration item refers to the maximum number of slave nodes that can synchronize with the new master node during failover. The smaller the value, the longer it will take to complete the failover. The larger the value, the more slave nodes will be unavailable due to data synchronization.
• sentinel failover-timeout mymaster 180000: During the synchronization process, how long it takes to complete is considered effective. The system default value is 3 minutes.

3. Start configuring

Use the command

cat sentinel.conf | grep -v '#' | grep -v '^$' > ./data/sentinel-26379.confMove the filtered information in sentinel.conf to/usr/local/redis/confDownloadThen opensentinel-26379.confModify the information storage directoryThen quickly copy the two sentinel configuration files, the ports are 26380 and 26381. sed 's/26379/26381/g' sentinel-26379.conf > sentinel-26381.conf

Insert picture description here

Test that the master-slave replication is in normal working condition, start three redis servers, and the ports are 6379, 6380, 6381 Check the master node information. There are two slave nodes connected, and the ports are 6380 and 6381 respectively.

A small point here is that one of the lags is 1 and the other is 0! lag is the delay time. I am testing locally, so there will be a situation of 0. This rarely happens when using a cloud server. The values ​​of lag are 0 and 1, which are normal. Test the master node to add a hash value, hset kaka name kakaGet the kaka values ​​from slave1 and slave2 respectively, and check whether the master-slave replication is running normally.

After testing, our master-slave structure is running normally. Start a sentinelredis-sentinel 26379-sentinel.confConnect to 26379 sentinel, mainly the last line, the monitored master node is named mymaster, the status is normal, and there are two slave nodes The number of sentinels is 1. Let’s check the sentinel configuration information of 26379. It has been changed at this time. Start a sentinel of 26380, redis-sentinel 26380-sentinel .conf, please note that there is an extra piece of information in the last line. This ID is the newly added ID of our 26379 configuration file. Then we come to the client of Sentinel 26379, which is also newly added. The ID of 26380 Sentinel. At this time, we are checking the configuration file of 26379 Sentinel. The first time we check the configuration file, 26380 Sentinel is not configured. The second time we check the configuration file, the information added after 26380 Sentinel is configured. Finally we need to start the Sentinel client 3, the port number is 26381. After starting up, our configuration information and server information will also be changed. If we add the information of Sentinel 26380, Sentinel 26381 will also have it.

Up to this point our configuration of Sentinel is over. Next, we will shut down the master node. After waiting for 30 seconds, we will come to the client of 26379 Sentinel. Some new information has been added here. So what does this information do? Let’s break it down.

We need to know a few things about the information here first

• sdown: This information means that one of the three sentinels thinks that the master node is down
• odown: This information means that the other two sentries connected to the master node and found that the master node was indeed down.
• Then they initiated an Round voting, here Kaka uses redis4.0, this information is slightly different between versions
• switch-master mymaster 127.0.0.1 6379 127.0.0.1 6380: Until here is the sentinel As a result of the voting, redis with port 6380 is elected as the master node
• slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6380: Here the port is 6381 Made a connection with 6379 and the new master node 6380
• sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380: The last sentence is the port 6379 or It was not online, so it was kicked offline

When we put the 6379 redis server online again, we can see that the Sentinel server responded with two sentences. One sentence is to remove 6379 offline. The last sentence is to reconnect 6379 to the new master node. At this time, the master node is 6380. Set the value in the redis client of 6380 to check whether the master-slave replication is working properly.

Add list type to new master node 6380Get this value at 6379 and 6381, that’s it! Our sentry mode configuration is complete.

3. Working Principle of Sentinel

After configuring the sentinel, you need to The working principle has been analyzed. Only by knowing its working process can we have a better understanding of Sentinel.

The principles explained in this article are not so dry! Allows you to read a technical article as a story.

Getting to the point, the role of sentry is monitoring, notification, and failover. Then the working principle also revolves around these three points.

1. Monitoring Workflow

Insert picture description here

1. The sentinel sends the info command and saves all sentinel status, master node and slave node information
2. The master node will record the redis instance The information recorded by the master node and the information recorded by the sentinel appear to be the same, but in fact there are still some differences.
3. The sentinel will send the info command to the corresponding slave node based on the slave node information obtained from the master node
4. Then Sentinel 2 comes In the same way, the master node will be changed to send the info command and a cmd connection will be established.
5. At this time, Sentinel 2 will also save the same information as Sentinel 1, but it will only save the sentinel information. It's 2.
6. At this time, a publish and subscribe is established between them so that the information of each sentinel is consistent. In order to ensure long-term symmetry of information between sentinels, they will also send ping commands to each other.
7. When another Sentinel 3 comes, it will do the same thing and send info to the master node and slave node. And establish connections with Sentinel 1 and Sentinel 2.

2. Notification workflow

Sentinel will send The command obtains its status and publishes the information to the Sentinel subscription.

3. Failover principle (the focus of this article)

Here Insert picture description

• The sentinel will keep sending publish sentinel: hello to the master node until the sentinel reports sdown. This word may seem familiar to you. That's right, it's the information reported by the Sentinel server after we disconnected the master node above. After Sentinel reports the master node sdown, it is not finished yet. Sentinel will also publish a message to the intranet to indicate that the master node is down. The command sent is sentinel is-master-down-by-address-port
• After the other sentinels received the command, did the master node hang up? Let me go and see if it hangs or not. The message sent is also hello. The rest of the sentinels will also send the information they received and send the command sentinel is-master-down-by-address-port to their own intranet, confirm that the first one sent sentinel is- The sentry at master-down-by-address-port said you are right, this guy is indeed dead. When everyone thinks that the master node is down, its status will be modified to odown. When a sentinel thinks that the master node is hung up, the status is sdown, and when half of the sentinels think that the master node is hung up, the status is odown. This is why the sentinel is configured with an odd number.
• For one sentinel who thinks that the main node is down, it is called subjective offline, and half of the sentinels think that the main node is down, which is called guest official offline.
• Once the master node guest officer is considered offline, the sentinel will proceed to the next step

❝

At this time, the sentinel has detected the problem, so which sentinel is responsible for selecting the new master node! It cannot be that Zhang San also goes, Li Si also goes, and Wang Wu also goes, it will be chaotic, so We need to choose the leader among all the sentinels, so how to choose! Please see the picture below.

❞

This time! The five sentinels will have a meeting together. All the sentinels are in an intranet, and then one thing they will do is that the five sentinels will send commands at the same time sentinel is-master-down-by-address-portAnd bring your election times and runid. Each sentinel is both a candidate and a voter. Each sentinel has one vote, and the envelope represents its voting rights. When sentinel1 and sentinel4 send instructions to the group at the same time to prepare for the election, sentinel2 says at this time that I will vote for whoever receives the instruction first. If sentinel1 is released early, then sentinel2's vote will be cast for sentinel1. Voting will be initiated according to this rule until one sentinel's votes are half of the total number of sentinels. Assume that sentinel1 will be elected after the number of votes sentinel1 reaches more than half of the total number of sentinels. At this time, the next stage is reached. Above, Sentinel has selected sentinel1 as the representative to go to all the slave nodes to find one as the master node. There are certain rules for selecting a master node, not just picking one at random.

Get rid of those who are not online first

Kill the one with slow response. Sentinel will send information to all redis, and the one with slow response will be killed. Kill the one that has been disconnected from the original master node for the longest time. Since the demonstration is not enough here, all A new slave5 has been added, which makes no sense! After judging from the above three points, there are still salve4 and slave5, and they will be filtered according to the priority principle.

• First, it will be based on the priority. If the priority is the same, other judgments will be made.
• Judge the offset offset and judge the data synchronization. If Say the offset of slave4 is 90 and the offset of slave5 is 100. Then the sentinel will think that there is something wrong with slave4's network! Then slave5 will be selected as the new master node. What if the offsets of slave4 and slave5 are the same? There is one last judgment
• The last step is to judge the runid, which is the seniority ranking in the workplace. In other words, it is judged based on the creation time of the runid. The one with the earliest time will be ranked first.

After selecting the new master node, instructions must be sent to all nodes.

4. Summary

I have finished talking about all the knowledge points about Sentinel, this article is the most important This is how Sentinel works. Let's briefly review its working principle.

• Monitor first, and all sentinels synchronize information

• Sentinels publish information to subscriptions

• Failover

  • Sentinel found that the master node was offline
  • Sentinel started voting for the person in charge
  • The person in charge will elect a new one The master node
  • The new master node disconnects the original master node, and other slave nodes connect to the new master node. The original master node connects as a slave node after it comes online.

The above is Kaka’s understanding of Sentinels. If mistakes can be made, Kaka will correct them in time.

❝

Persistence in learning, persistence in blogging, and persistence in sharing are the beliefs that Kaka has always upheld since his career. I hope that Kaka’s articles in the huge Internet can bring you a little Please help. See you in the next issue.

❞

Recommended: "Redis Tutorial"

The above is the detailed content of Redis Sentinel Principle, I have tolerated you for a long time!. For more information, please follow other related articles on the PHP Chinese website!