服务器IO问题（疑似Redis导致的）

Question

某生产环境服务器（多个服务都安装在1台服务器上）

服务器负载一致居高不下，升级过硬件以后有好转，CPU和内存的有所缓解，不过磁盘IO的依然如故；

使用nmon查看由多块磁盘组成一个LVM分区，写一直是100%；

使用lsof查看，该分区的主要读写的是MySQL和Redis；

使用iotop查看io排行，前几个分别是：

[jbd2/dm-0-8]
[flush-253:0]
[redis]
[mysql]

对前2个进程不太了解，有没有那位比较熟悉的大神指点下，前2个进程的用途以及他们和Redis的IO优化建议；

Answer 1

Troubleshoot this problem from three aspects:
1. Application layer

• Redis status monitoring to see if there are any application performance issues

• MySQL status monitoring, check slow query logs and other related problem points

2. System layer
flush-253:0 and jbd2/dm-0-8 are all processes of the operating system that actually deal with the disk and file system. In essence, they are system operations generated by the application layer.

3. Hardware
Check the disk status and server hardware problems

Answer 2

I have encountered the problem of high redis io, which is related to the rdb mode of saving.
redis default configuration
save 900 1
save 300 10
save 60 10000
The above means that the service has been performed at least once within
900 seconds Modification or the server has been modified at least 10 times within 300 seconds or the server has been modified at least 10,000 times within 60 seconds. A save will be triggered. The rdb method is to fork the main process into a child process, and then fork the child process. All the data of the process is saved to the disk. Note that it is all. In other words, for example, if you have a redis occupying 10G of memory, then every 900 seconds at most, all the data in the 10G memory will be saved to the disk. If you The device IO capability is relatively poor, or the redis data volume is larger, then the save operation will not be completed within a save cycle. In this case, the save will continue, which will cause IO high.
The solution is
1 Extend the redis save cycle.
2 Split redis and do not save too much data in a single redis process.
3 Try to use the aof method.