Open source load balancing LVS random self-start exception diagnosis case analysis

Open source load balancing LVS random self-start exception diagnosis case analysis_PHP tutorial

WBOY

Release： 2016-07-12 09:01:06

Original

1062 people have browsed it

Open source load balancing LVS random self-start abnormal diagnosis case analysis

In response to the country’s call to advocate the security, autonomy and controllability of IT information system software and hardware facilities, our company also Efforts are being made to promote the localization and open source of IT software and hardware. Therefore, the open source load balancing software LVS was introduced and multiple LVS systems were deployed in production. A strange problem was discovered during use. After configuring random auto-start, some LVS systems can start normally with the server when the operating system is restarted. However, one of the systems cannot start the service automatically with the operating system. In order to solve this problem, the following is provided.
1. Problem phenomenon
The specific symptoms of this set of LVS failures are as follows:
LVS has been configured as a system service normally and set to 2345 to start automatically at boot. However, after restarting the server, it was found that the service did not start as expected. By checking the system's diagnostic log tail –f /var/log/messages, we found that ipvs frequently flushes logs to the messages file.
The specific error message is as follows: Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available. After manually executing ipvsadm -Ln, the service returns to normal.
2. Analysis process:
1. Through the diagnostic log prompts, we can confirm that the root cause of the problem is that the ipvs module was not loaded normally when the system restarted
2. By executing the following command lsmod|grep ip_vs It also further confirms that the corresponding module is not loaded into the kernel
So what is the reason why the ipvs module is not loaded into the Linux kernel?
3. Debug specific ideas:
1. Test the settings and restart without starting keepalived. Only start ipvsadm to see if it can be started. The problem remains after restarting, indicating that the problem is not with keepalived for the time being;
2. Replace high I changed the version of ipvsadm to 1.25 and the problem remained after the test;
3. I began to suspect that the installation sequence was the reason, so I tested both versions of ipvsadm in a different installation sequence than keepalived, and matched the startup sequence, but the problem remained;
4. Accidentally use the /etc/init.d/ipvsadm start command to start the ipvs service and then restart the operating system. ipvsadm is successfully pulled up by the system and the snapshot is saved immediately.
5. So I checked the startup script of ipvsadm. The script is relatively cumbersome and involves many files. I found a configuration file /etc/sysconfig/ipvsadm. Suddenly I got an idea, is there something wrong with the configuration file? After searching for the ipvsadm file, the problem was found. There is one missing configuration file under /etc/sysconfig;
6. Manually create the ipvsadm configuration file touch ipvsadm, and then restart the system. The ipvsadm service started successfully
4. Best solution Solution
After deploying and configuring LVS, execute the service ipvsadm save command
5. Cause of this problem
The relevant deployment personnel did not follow the deployment manual and missed relevant steps, which resulted in the random self-starting of ipvs. related configuration files.