Redis optimization guide: network, memory, disk, blocking points

Since Redis is a memory-based operation, the CPU is not its performance bottleneck. On the contrary, the server's memory utilization, network IO, and disk reads and writes play a key role in the performance of Redis.

Therefore, we will focus on optimization from aspects such as network, memory, disk and blocking points. If any terminology is unclear, it is recommended to refer to the redis content in previous issues or consult relevant information.

Network Optimization

If the client requests the server, that is, in the "request-response" mode, use batch processing as much as possible to reduce network IO overhead.

Batch processing technology: atomic m batch processing instructions, pipline technology, redis, transactions, lua scripts.

Batch processing reduces network IO overhead

Atomic m batch processing instructions: string type, it is recommended to use mget/mset instead of get/set; hash type, it is recommended to use hmget/hmset instead of hget/hset.

pipline technology: Pipeline technology can be used when there are batch operations when using list, set and zset.

redis transaction: recommended when special business requirements ensure multiple instructions.

lua script: It is recommended to use lua script when you need to ensure the atomicity of multiple instructions. Specific examples include distributed unlocking, flash sales and inventory reduction.

Inter-node network optimization

Build a cluster in the same LAN;

Control the number of nodes that split the cluster. The hash slots allocated on the redis instance need to be transferred between different instances. When the debt balancing instance is deleted, the data will be transferred between different instances. But the hash slot information is not large, and the data migration is gradual, but it is not the main problem;

Memory optimization

Control the length of the key: It is recommended to define the specifications before development to ensure that the key is simple and clear, and abbreviate the key according to the business as much as possible.

Avoid bigkey: The recommended size of string type is within 20KB. It is recommended to control the threshold of hash, list, set and zset, and it is recommended to control it within 5000.

Key setting expires: Make full use of memory.

Choose the appropriate data structure

String type, it is recommended to use integer type. Its underlying encoding will choose integer encoding, which has low memory overhead;

Hash type, it is recommended to control the element threshold. When there are few elements, the bottom layer will use a compressed list data structure, which has a small memory overhead; list type, it is recommended to control the element threshold. When there are few elements, the bottom layer will use a compressed list data structure, which has a small memory overhead; set type, It is recommended to store integer type. The underlying encoding will choose integer encoding, which has low memory overhead;

zset type, it is recommended to control the element threshold. When there are few elements, the bottom layer will use a compressed list data structure, which has low memory overhead;

Data compression: The client can use snappy, gzip and other compression algorithms to compress data before writing to redis to reduce memory usage. However, the client needs to decompress the data after reading the data, which will consume more CPU.

Enable memory elimination strategy

Avoid the default memory elimination strategy. Please choose an appropriate elimination strategy based on actual business to improve memory utilization.

LRU: Focus on the number of accesses, eliminate the least recently used keys, and have a wide range of usage scenarios. Redis's LRU uses an algorithm similar to LRU, adding an extra field with a length of 24 bits to the key, which is the timestamp of the last access. Adopt a lazy way to process: when performing a write operation, if the maximum memory is exceeded, the LRU elimination algorithm is executed once, 5 (the number can be set) keys are randomly sampled, and the oldest key is eliminated. If the maximum memory is still exceeded after elimination, the elimination will continue. .

LFU: Focus on access frequency, recommended when dealing with cache pollution.

Memory fragmentation optimization problem

Reasons: One is caused by the allocation strategy of the memory allocator. The memory allocator allocates according to a fixed size, rather than according to the actual requested size. If the requested bytes are within the requested bytes, 32 bytes are actually allocated; the other It is the memory fragmentation caused by part of the space that will be released after the redis key-value pair is deleted.

Positioning: Observe the indicator of men_fragmentation_ratio through the instruction INFO memory; if the indicator is between 1-1.5, it is normal; if the indicator is greater than 1.5, the memory fragmentation rate has exceeded 50%, and memory fragmentation needs to be processed;

Solution: Restart the redis instance;

Enable redis automatic memory fragmentation cleaning function.

Disk Optimization

Physically build the redis service: When persisting, redis uses the method of creating a sub-process (which will call the fork system of the operating system), and the execution of fork in a virtual machine environment is slower than that of a physical machine.

Persistence Optimization Mechanism

Do not enable persistence: redis is only used for caching, so there is no need to enable persistence to reduce disk overhead;

AOF optimization: process AOF in the background, configure apenfsync everyec to put the data persistence flush operation into the background thread for execution, and try to reduce the impact of redis writing to disk on performance;

Do not build high-frequency AOF persistence. The default frequency of AOF persistence is once per second. It is not recommended to modify this configuration. It can already guarantee that data will be lost for up to 1 second;

Enable hybrid persistence, redis4.0 supports hybrid persistence RDB incremental AOF;

Enable multi-thread configuration. Before redis 6.0, persistence was handled through the main thread fork sub-process, but forks were blocked synchronously. After 6.0, multiple processes are supported to handle persistence operations;

Cluster optimization

Slave performs persistence optimization: the master does not perform persistence and shares the pressure of master disk IO as much as possible; master-slave optimization: incremental mode, the master-slave synchronization method is designated as incremental mode, and the full RDB mode will not be selected. The mode is very performance consuming; when using the cascade synchronization mode, with one master and multiple slaves, multiple slaves come to the master to synchronize data, which will directly drag down the performance of the master.

Regarding this problem, redis supports cascade synchronization, that is, the master only synchronizes data to one salve, and then the data of other salves are synchronized from this salve to relieve the pressure on the master.

The actual size is recommended not to exceed 6G. If the instance is too large, the master-slave synchronization will be stuck, and in serious cases it will bring down the master.

AOF will be replayed during abnormal restart. If the instance is too large, data recovery will be abnormally slow.

Choke point optimization

Analysis: Since Redis is single-threaded when processing requests and instructions, its performance bottleneck is the synchronization blocking problem.

bigkeyquestion

Hazards: Reading and writing bigkey may cause timeout, and redis operates data in a single thread, which may seriously block the entire redis service. Moreover, a key will only be divided into one node, so the pressure of sketching cannot be shared. Bigkey detection: comes with the command bredis-cli-bigkeys. Redis comes with instructions that can only find the largest key among the five data types, which is not very useful and is not recommended. python scanning script. It can locate specific keys, but the accuracy is not high and is not recommended.

rdb_bigkeys tool. A tool written in Go, it is fast and highly accurate. It can also be directly exported to a csv file for easy viewing and recommendation.

Optimization: For non-string type bigkey, the element set can be split into multiples. For example, if a bigkey is split into 1000 keys, the suffix of the key uses hash modulo 1000.

Use local cache. For example, store the business ID version number in redis. Put the specific content in the local cache. Check the redis cache first for each query, and then check the version number with the local cache.

Optimizing bigkey is generally a laborious process. It is recommended to define specifications during development to avoid bigkey problems.

Expiration Policy

Scheduled deletion: Each expired key is given a scheduled job, and it is deleted directly when it expires. It has high memory utilization and high CPU usage. Lazy deletion: When the key is queried, it is judged whether the key has expired. If it has expired, it will be deleted. This results in low CPU usage and high memory utilization. Periodic deletion: scan every once in a while, expired keys are deleted directly, and the CPU and memory utilization are average.

1. Greedy strategy. Redis will set the expired key in a separate dictionary.

2. Scanning process. Select 20 keys from the expired dictionary and delete the expired keys among the 20 keys. If the proportion of deleted keys exceeds 1/4, repeat step 1.

Based on the above logic, in order to solve the problem of thread stuck caused by excessive looping, a timeout mechanism is added to the algorithm. The default time is 25ms.

3. Scan frequency: redis defaults to 10 expiration scans per second.

Redis defaults to lazy deletion and regular deletion.

Optimization: Turn on lazy-free, and the time-consuming operation of releasing memory will be executed in the background thread, supported by redis4.0.

Enable multi-threading mode. Before redis6.0, the expiration policy was a synchronous operation of the main thread. After 6.0, multi-threading is used for processing.

High complexity instructions

It is recommended to use scan to query in batches, and do not use keys. Not applicable to aggregation operations: redis uses a single-threaded model to process requests. When executing commands that are too complex (consuming more CPU resources), subsequent requests will be queued and cause delays, such as SINTER, SINTERSTORW, ZUNIONSTORE, ZINTERSTORE, etc. It is recommended to use scan to find out the elements in the collection in batches and perform aggregation calculations on the client.

Container class data operation: When there are too many container class elements, direct query will cause delays due to the network. It is recommended to query in batches.

When there are too many container elements, deleting keys directly may cause redis to freeze. It is recommended to delete them in batches.

The above is the detailed content of Redis optimization guide: network, memory, disk, blocking points. For more information, please follow other related articles on the PHP Chinese website!