How to solve the Big Key problem in Redis

1. What is Big Key?

In simple terms, Big Key means that the value corresponding to a certain key is very large and takes up a large amount of redis space. It is essentially a large value problem. . The key can often be set by the program itself, and the value is often not controlled by the program, so the value may be very large.

The values ​​corresponding to these Big Keys in redis are very large and take a lot of time in the serialization/deserialization process. Therefore, when we operate Big Key, it is usually time-consuming, which may lead to Redis blocks, thereby reducing redis performance.

Use several practical examples to describe the characteristics of large Key:

● A String type Key, its value is 5MB (the data is too large);

● A Key of type List, the number of its lists is 20,000 (the number of lists is too many);

● A Key of type ZSet, the number of its members is 10,000 (the number of members is too many);

● A Key in Hash format has only 1,000 members, but the total value size of these members is 100MB (the member size is too large);

In actual business , the determination of big keys still needs to be comprehensively judged based on the actual usage scenarios and business scenarios of Redis. It is usually judged by the size of the data and the number of members.

2. The scene where Big Key occurs?

1. Improper use of redis data structure

Using Redis in scenarios that are not suitable for its capabilities will cause the value of the Key to be too large, such as using String type Key stores large-volume binary file data.

2. Failure to clean up junk data in a timely manner

Due to failure to clean up invalid data regularly, the number of members in the HASH type key continues to increase. Value will increase infinitely because it will only continue to receive data without any deletion mechanism.

3. Inaccurate business estimates

Insufficient consideration in planning and design before business launch, failure to reasonably split members in Key, resulting in inaccuracies in individual Keys Too many members.

4. List of fans of celebrities and Internet celebrities, and a list of comments on a certain hot news item

Suppose we use the List data structure to save the fans of a certain celebrity/Internet celebrity , or save the comment list of hot news. Because the number of fans is huge, hot news will have a lot of click-through rates and comments, so there will be a lot of elements stored in the List collection, which may cause the value to be too large, resulting in a Big Key problem.

3. What are the dangers of Big Key?

1. Blocking request

The value corresponding to Big Key is larger. When we read and write it, it takes a long time, which may cause blocking. Subsequent request processing. The core thread of Redis is single-threaded, which means that all requests will be processed serially. If the previous request is not completed, the subsequent request will not be processed.

2. Memory increase

The memory consumed to read the Big Key will increase compared with the normal Key. If it continues to increase, it may cause OOM (memory overflow), or reaching the maximum memory maxmemory setting value of redis, causing write blocking or important keys being evicted.

3. Blocking the network

When reading a large single value, it will occupy more bandwidth of the server network card, slow down itself and may affect other servers on the server. Redis instance or application.

4. Impact on master-slave synchronization and master-slave switching

Deleting a large Key will cause the main library to be blocked for a long time and cause synchronization interruption or master-slave switching.

4. How to identify Big Key?

1. Use the command identification that comes with redis

For example, you can use the official Redis client redis-cli plus the --bigkeys parameter to find an instance 5 The largest key of a data type (String, hash, list, set, zset).
The advantage is that it can be scanned online without blocking the service; the disadvantage is that there is less information and the content is not accurate enough.

2. Use the debug object key command

to analyze the Key according to the incoming object (the name of the Key) and return a large amount of data, in which the value of serializedlength is The serialized length of the Key. It should be noted that the serialized length of the Key is not equal to its real length in the memory space. In addition, the debug object is a debugging command, which is expensive to run, and when it is running, enter The remaining requests to Redis will be blocked until they are completed. And only the information of a single key can be found at a time, so it is not officially recommended.

3. redis-rdb-tools open source tool

This method is to execute bgsave on the redis instance. bgsave will trigger the snapshot backup of redis and generate rdb persistence. file, and then analyze the dumped rdb file to find the big key in it.

The advantage is that the obtained key information is detailed, there are many optional parameters, and it supports customized requirements. The result information can be selected in json or csv format, and subsequent processing is convenient. The disadvantage is that it requires offline operation and it takes a long time to obtain the results.

5. How to solve the Big Key problem?

To solve the Big Key problem, it is nothing more than reducing the size of the value corresponding to the key, that is, for the String data structure, reducing the length of the stored string; for the List, Hash, Set, and ZSet data structures It is to reduce the number of elements in the collection.

1. Split the big Key

Split a Big Key into multiple small Keys such as key-value, and ensure that the number or size of each key is within a reasonable range, and then store it, by getting different keys or using mget to obtain them in batches .

2. Clean up the big keys

Clean up the big keys in Redis and delete such data from Redis. Redis has provided the UNLINK command since 4.0, which can slowly and gradually clean up incoming keys in a non-blocking manner. Through UNLINK, you can safely delete large keys or even extra-large keys.

3. Monitor Redis’s memory, network bandwidth, timeout and other indicators

By monitoring the system and setting reasonable Redis memory alarm thresholds to remind us that there may be large events at this time Key is being generated, such as: Redis memory usage exceeds 70%, Redis memory growth rate exceeds 20% within 1 hour, etc.

4. Regularly clean up invalid data

Accumulation of a large amount of invalid data will occur. If a certain Key has been writing a large amount of data incrementally, but ignored it Data timeliness. Invalid data can be cleaned up through scheduled tasks.

5. Compress value

Use serialization and compression algorithms to control the size of the key, but it should be noted that both serialization and deserialization will cause certain performance consumption. If the value is still very large after compression, you can consider further splitting the key.

Supplementary knowledge: key design

(1)[Recommendation]: Readability and manageability

Prefix with the business name (or database name) (to prevent key conflicts), separated by colons, such as business name: table name: id
o2o:order:1

(2) [Suggestion]: Simplicity

Under the premise of ensuring semantics, control the length of the key. When there are many keys, the memory usage cannot be ignored. For example:
user:{uid}:friends:messages:{mid} is simplified to u:{uid} m:{mid}

(3)[Mandatory]: Do not include special characters

Counter example: include spaces, newlines, single and double quotes, and other escape characters

The above is the detailed content of How to solve the Big Key problem in Redis. For more information, please follow other related articles on the PHP Chinese website!