A brief discussion on three methods of redis to implement real-time subscription push

How can 20w push users be completed concurrently in seconds? This article will introduce to you three methods for redis to implement real-time subscription push: MQ, traditional scheduled tasks, and Redis' SortSet queue. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to everyone.

[Related recommendations: Redis Video Tutorial]

A while ago, we developed a project for the company’s coupon collection center. This project is based on Redis is implemented as a key technology.

Let’s talk about the coupon collection center project first. This project is similar to the coupon collection center of JD.com app. Of course, the picture is taken from JD.com, not the company’s. . .

There is a function called subscription push for receiving coupons.

What is the subscription push for coupon collection?

means that the user has subscribed to the push notification of the coupon, and the reminder information will be pushed to the user's app one minute before it can be claimed.

Originally, this subscription function was supposed to be implemented by the message center, but they said it could not be implemented in a short time. So I, the person in charge of coupons, did it -.-!. The specific plan is to reach the specific push time point. The coupon system calls the push interface of the message center to push the information out.

Let’s analyze the business scenario of this function. The company currently has 6000W registered users, so don’t ask who it is. . . For example, if there is a no-threshold discount coupon that offers an instant discount of 20 yuan when placing an order, more people will grab this coupon. We conservatively estimate it to be 10W, and it is hard to say if it is a million yuan. Our initial target is 200,000 people, so these 200,000 push messages will be pushed out in one minute! And one user can subscribe to multiple coupons. So we know that there are two outstanding difficulties with this subscription function:

• Effectiveness of push: If the push is slow, users will complain that they have not been notified in time and missed the opportunity to start grabbing.

• The volume of the push is large: a popular coupon that everyone wants to grab!

However, the volume of push will affect the effectiveness of push. This is really a headache!

Then let us solve the problems one by one!

Issues with the effectiveness of push: When a user subscribes to a coupon collection reminder in the coupon collection center, a user's subscription reminder record will be generated in the background, which records the time point at which it was given to the user. Send push messages. So the question becomes how the system can quickly select which records to push in real time!

Option 1:

Delayed delivery of MQ. Although MQ supports delayed delivery of messages, the scale is too large, 1s 5s 10s 30s 1m, and it cannot be used for precise time point delivery! And if the user cancels the subscription after executing the subscription, the operation of deleting the MQ message sent is a bit cumbersome and difficult to implement in a short time! And users can cancel and then subscribe, which again involves the problem of deduplication. Therefore, MQ’s plan is rejected.

Option 2:

Traditional scheduled tasks. This is relatively simple. To use a scheduled task, load the user's subscription reminder records in the db and select the records that can currently be pushed. But there is a saying that goes well: Any design that is divorced from actual business is a rogue. Let's analyze whether traditional scheduled tasks are suitable for our business!

Can it support multiple machines running at the same time?	Generally not, the same It can only be run alone at all times.
Storage data source	It is usually mysql or other traditional database, and it is a single table storage
Frequency	Supports seconds, minutes, hours and days, generally not too fast

To sum up, we know that general traditional scheduled tasks have the following shortcomings:

1. Performance bottleneck. Only one machine is processing it, which is unable to cope with the large amount of data!

2. Poor effectiveness. The frequency of scheduled tasks cannot be too high. If it is too high, it will put a lot of pressure on the business database!

3. Single point of failure.

If the running machine hangs up, then the entire business will be unavailable -. - This is a terrible thing! Therefore, traditional scheduled tasks are not suitable for this business. . . So are we at our wits’ end? Actually no! We just need to make a simple transformation of the traditional scheduled tasks! You can turn it into a scheduled task cluster that can run on multiple machines at the same time, and the effectiveness can be accurate to the second level, and reject single points of failure! This requires the help of our powerful redis.

Option 3:

Scheduled task cluster First we need to define the three problems that the scheduled task cluster needs to solve!

1. The effectiveness must be high

2. The throughput must be large

3. The service must be stable and there must be no single point of failure. The following is the architecture diagram of the entire scheduled task cluster. .

The architecture is very simple: we store the user's subscription push records in the sortedSet queue of the redis cluster, and use the reminder timestamp as the score value, and then in our personal Each business server starts a timer with a frequency of seconds. My setting is 1s. Then after load balancing, the user records to be pushed are obtained from a queue and pushed. Next we analyze the following architecture.​

1. Performance: excluding bandwidth and other factors, it is basically linearly related to the number of machines. The greater the number of machines, the greater the throughput. When the number of machines is small, the relative throughput decreases.​

2. Effectiveness: It has been improved to the second level, and the effect is acceptable.​

3. Single point of failure? nonexistent! Unless the redis cluster or all servers are down. . . .

Here is an analysis of why redis is used?

First, redis can be used as a high-performance storage db. Its performance is much better than MySQL, and it supports persistence and has good stability.

The second redis SortedSet queue naturally supports sorting based on time as a condition, which perfectly satisfies us in selecting the records to be pushed.

ok~ Now that the plan is available, how can we implement it within one day? Yes, it only took me one day from designing this plan to completing the basic coding. . . Because time is too late.

First we use user_id as the key, and then mod the queue number hash into the redis SortedSet queue. Why is this? Because if the user subscribes to two coupons at the same time and the push time is very close, the two pushes can be merged into one~, and the hash is relatively even. The following is a screenshot of part of the code:

Then we need to determine the number of queues. Generally speaking, we define as many queues as we have for processing servers. Because too few queues may cause queue competition, too many may result in records not being processed in a timely manner. However, the best practice is that the number of queues should be dynamically configurable, because the number of online cluster machines will often change.

We will add more machines during the big promotion, right? And as the business volume increases, the number of machines will also increase, right? So I borrowed Taobao's diamond to dynamically configure the number of queues.

How many records we take from the queue each time can also be dynamically configured

This way, we can configure it at any time The actual production situation adjusts the throughput of the entire cluster~. So our scheduled task cluster still has a feature that supports dynamic adjustment~. The last key component is load balancing. This is very important!

Because if this is not done well, it may cause multiple machines to compete to process a queue at the same time, affecting the efficiency of the entire cluster! When time was very tight, I used a simple and practical algorithm that uses redis to auto-increment the key and then mod the number of queues. This will largely ensure that no two machines will compete for a queue at the same time~.

For more programming-related knowledge, please visit: programmingvideo! !

The above is the detailed content of A brief discussion on three methods of redis to implement real-time subscription push. For more information, please follow other related articles on the PHP Chinese website!