How to ensure the high availability of message queue?

Message queue is an essential skill in high-concurrency scenarios. With our use, there are many problems in the production environment, such as: How to achieve high availability of message queue?

There are many types of middleware in scenarios. Here we prepare some commonly used ones for analysis and processing.

1. High availability of RabbitMQ

RabbitMQ is relatively representative because it is based on master-slave (non-distributed) high availability, so we use RabbitMQ is an example to explain how to implement the high availability of the first MQ.

RabbitMQ has three modes: stand-alone mode, normal cluster mode, and mirror cluster mode.

Single-player mode

Stand-alone mode is Demo level. Usually you start it locally for fun. No one uses stand-alone mode for production.

Normal cluster mode (no high availability)

Normal cluster mode means starting multiple RabbitMQ instances on multiple machines, one for each machine. The queue you create will only be placed on one RabbitMQ instance, but each instance synchronizes the metadata of the queue (metadata can be thought of as some configuration information of the queue. Through metadata, you can find the instance where the queue is located).

When you consume, if you are actually connected to another instance, then that instance will pull data from the instance where the queue is located.

This method is really troublesome and not very good. It does not achieve the so-called distribution and is just an ordinary cluster. Because this leads to either the consumer randomly connecting to an instance each time and pulling data, or the consumer fixedly connecting to the instance where the queue is located to consume data. The former has the overhead of data pulling, and the latter leads to a single-instance performance bottleneck.

And if the instance that holds the queue goes down, other instances will not be able to pull from that instance. If you enable message persistence and let RabbitMQ store messages, the messages may not necessarily be Lost, you have to wait for this instance to recover before you can continue to pull data from this queue.

So this matter is more embarrassing. There is no so-called high availability. This solution is mainly to improve throughput, that is, to allow multiple nodes in the cluster to serve the read and write operations of a certain queue. .

Mirror cluster mode (high availability)

This mode is the so-called high availability mode of RabbitMQ. What is different from ordinary cluster mode is that in mirror cluster mode, the queue you create, regardless of metadata or messages in the queue, will exist on multiple instances. That is to say, each RabbitMQ node has a complete copy of this queue. Mirroring means containing all the data of the queue. Then every time you write a message to the queue, the message will be automatically synchronized to the queues of multiple instances.

So how to enable this mirror cluster mode? In fact, it is very simple. RabbitMQ has a good management console, which is to add a policy in the background. This policy is a mirror cluster mode policy. When specified, you can require data to be synchronized to all nodes, or to a specified number. When the node creates the queue again, applying this strategy will automatically synchronize the data to other nodes.

In this case, the advantage is that if any of your machines goes down, it will be fine. Other machines (nodes) still contain the complete data of this queue, and other consumers can go to other nodes to consume data.

The disadvantage is that, first, the performance overhead is too great. The messages need to be synchronized to all machines, resulting in heavy network bandwidth pressure and consumption!

Second, these games are not distributed and have no scalability. If a queue is heavily loaded and you add a machine, the new machine will also contain all the data of the queue. , and there is no way to linearly expand your queue.

2. High availability of Kafka

The most basic architectural understanding of Kafka: it is composed of multiple brokers, each broker is a node; you create a topic, This topic can be divided into multiple partitions, each partition can exist on different brokers, and each partition stores a part of the data.

This is a natural distributed message queue, which means that the data of a topic is scattered on multiple machines, and each machine stores a part of the data.

In fact, RabbmitMQ and the like are not distributed message queues. They are traditional message queues. They just provide some clustering and HA (High Availability, High Availability) mechanisms, because no matter how you play Yes, the data of a queue in RabbitMQ is placed on one node. Under the mirror cluster, the complete data of the queue is also placed on each node.

Before Kafka 0.8, there was no HA mechanism. If any broker went down, the partition on that broker would be useless and could not be written or read. There was no high availability at all.

For example, we assume that we create a topic and specify that the number of partitions is 3, each on three machines. However, if the second machine goes down, 1/3 of the data on this topic will be lost, so this cannot be highly available.

After Kafka 0.8, a HA mechanism is provided, which is the replica mechanism. The data of each partition will be synchronized to other machines to form its own multiple replicas. All replicas will elect a leader, then production and consumption will deal with this leader, and other replicas will be followers. When writing, the leader will be responsible for synchronizing the data to all followers. When reading, just read the data on the leader directly. Can only read and write leader?

It’s very simple. If you can read and write each follower at will, then you have to take care of the data consistency issue. The system complexity is too high and problems can easily occur. Kafka will evenly distribute all replicas of a partition on different machines, so as to improve fault tolerance.

If you do this, there is so-called high availability, because if a certain broker goes down, it’s okay. The partitions on that broker have copies on other machines. If the downed broker has a leader of a certain partition, a new leader will be re-elected from the followers, and everyone can continue to read and write the new leader. This is called high availability.

When writing data, the producer writes to the leader, and then the leader writes the data to the local disk. Then other followers take the initiative to pull data from the leader. Once all followers have synchronized the data, they will send acks to the leader. After the leader receives the acks from all followers, it will return a successfully written message to the producer. (Of course, this is just one of the modes, and this behavior can be adjusted appropriately)

When consuming, it will only be read from the leader, but only when a message has been synchronized by all followers and successfully returned ack. , this message will be read by consumers.

For more related php knowledge, please visit php tutorial!

The above is the detailed content of How to ensure the high availability of message queue?. For more information, please follow other related articles on the PHP Chinese website!