Introduction to two persistence defects in redis

1. RDB persistence mode defects

1. Problem description:

Concurrent 200 channels, simulating continuous writing to Redis, after 4 hours, A large number of interface calls began to fail, and the error message was as follows:

{"data":{"sendResult":null},"base":{"returncode":"99999","returndesc":"系统异常：MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error."},"qrybase":{"total":0,"count":0,"start":0}}

2. Cause analysis:

Interpreted the error message and thought it was caused by insufficient disk space. It was found that 42% of the disk was left, as follows Display:

So I opened the Redis log according to the error message prompts and continued the stress test. The interface still reported an error, but

can be obtained from the Redis log information. Can't save in background: fork: Cannot allocate memory

It is related to the improper use of memory by the process. Check the memory occupied by the Redis main process as follows: occupying nearly 55%*4G memory

Specific reason: In order to avoid the main process suspended animation when saving data to the hard disk, Redis needs to Fork a copy of the main process, and then complete the operation of saving the data to the hard disk in the Fork process. If the main process uses 2.2GB Of memory, an additional 2.2GB is needed when Fork child process. At this time, the memory is not enough, Fork fails, and the data saving hard disk also fails.

3. Mitigation plan (cannot fundamentally solve the problem):

3.1 Modify the configuration item stop-writes-on-bgsave-error no in the redis.conf file (the default value is yes), that is When an error occurs in the bgsave snapshot operation, stop writing data to the disk. In this way, any subsequent write errors will fail. In order not to affect subsequent write operations, you need to change this value to no

3.2 Modify the kernel parameters (3 below) method), but requires root permissions:

（1）	编辑/etc/sysctl.conf ，改vm.overcommit_memory=1，然后sysctl -p 使配置文件生效
（2）sysctl vm.overcommit_memory=1
（3）echo 1 > /proc/sys/vm/overcommit_memory

2. AOF persistence mode defects

1. Description of problem 1:

Both Redis master and slave nodes enable AOF mode. There were 200 concurrent connections, and the simulation continued to write to Redis. After 15 minutes, a large number of interface calls began to fail, and the Linux virtual server where Redis was located hung.

The interface error is as follows:

{"data":null,"base":{"returndesc":"系统异常","returncode":"999999"},"qrybase":null}
Biz（dubbo）接口报错如下：
2015-06-05 11:28:28.760 [DubboServerHandler-X.X.X.X:20882-thread-173] ERROR  - error while validate jedis!
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out

Cause analysis:

From the dubbo interface error message, it is caused by the interface API operation Redis timeout. Judging from the system logs and IO monitoring, it is shown that the above problems are caused by IO bottlenecks (system IO is too busy), as shown below:

It can also be seen from the system log that the IO blocking time exceeds 120 seconds, and the machine hangs due to the system security mechanism.

Summary
The test results prove that the AOF mode has the most obvious flaw, that is, IO will become a performance bottleneck when access pressure is high, resulting in service unavailability.

3. Mitigation plan (cannot fundamentally solve the problem)

Edit /etc/sysctl.conf and add the following configuration:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

Then sysctl -p makes the configuration file take effect.

Description of Problem 2:

Whether AOF mode or RDB (snapshot mode) is used, when the size of the two files (.aof or .rdb) exceeds 80% of the system memory, the Redis process will be killed by the system down, causing the service to become unavailable.

Summary

The above problems indicate that we need to plan the system memory capacity in advance when using Redis, because once Redis crashes, a large amount of data will be lost and it is unrecoverable.

For more redis knowledge, please pay attention to the redis introductory tutorial column.

The above is the detailed content of Introduction to two persistence defects in redis. For more information, please follow other related articles on the PHP Chinese website!