Record traffic exception handling_PHP tutorial

记一次流量异常处理

前两天接到一个做开发的朋友电话，说他们客户一台服务器开机后，所有一个网段的机器上网都变慢了，他远程操作这台服务器也一卡一卡的。
我第一反应就是机器被人攻击过了，因为我之前也遇到过类似的现象。大概都是tomcat管理密码设置的比较弱，被人上传了一些war包，导致服务器拼命往外发包，或者是被人恶意上传了一些php文件，也是往外发送大量的数据包。总而言之，往外发送大量数据包基本都是被人攻击过啦！下面看看我是怎么处理的。
1、首先我给他一个脚本，确认一下是网卡异常流量引起。注意网卡改成你的外网网卡名称。

<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>while : ; do</li><li>time=`date "+%Y-%m-%d %H:%M:%S"`</li><li>rx_before=`ifconfig eth0|sed -n "8"p|awk '{print $2}'|cut -c7-`</li><li>tx_before=`ifconfig eth0|sed -n "8"p|awk '{print $6}'|cut -c7-`</li><li>sleep 2</li><li>rx_after=`ifconfig eth0|sed -n "8"p|awk '{print $2}'|cut -c7-`</li><li>tx_after=`ifconfig eth0|sed -n "8"p|awk '{print $6}'|cut -c7-`</li><li>rx_result=$[(rx_after-rx_before)/256]</li><li>tx_result=$[(tx_after-tx_before)/256]</li><li>echo "$time Now_In_Speed: "$rx_result"kbps Now_OUt_Speed: "$tx_result"kbps"</li><li>sleep 2</li><li>done</li></ol>

然后运行这个脚本

<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>sh traffic.sh</li></ol>

执行之后我们会看到偶尔流出的流量惊人。

<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>2016-02-03 13:32:01 Now_In_Speed: 5kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:05 Now_In_Speed: 2kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:09 Now_In_Speed: 1kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:13 Now_In_Speed: 1kbps Now_OUt_Speed: 664567kbps</li><li>2016-02-03 13:32:17 Now_In_Speed: 6kbps Now_OUt_Speed: 657895kbps</li><li>2016-02-03 13:32:21 Now_In_Speed: 3kbps Now_OUt_Speed: 568462kbps</li><li>2016-02-03 13:32:25 Now_In_Speed: 4kbps Now_OUt_Speed: 0kbps</li></ol>

2、问题确定了，我们就好办了，上图我截图很少，而且我还发现了很有规律的事情，大概每二十多秒就会发出3-4个左右相当大的数据包。既然有规律那就肯定是后台有程序在运行。我查看了一下服务器是不是运行了tomcat？结果webapps目录下没有一些异常的jar包。我再查了一下是不是apache什么的，结果服务器上就只发现运行了oracle，根据自己的排查故障经验，我和朋友说了把oracle关闭。缩小故障查找范围，好确认不是oracle引起的。
3、在用ps -ef看了一下基本看不出，因为进程太多了，而且很多系统的进程我也不认识，没有看到什么异常进程。只有一个tomcat进程，kill之后一会又起来了，很奇怪。肯定是什么守护程序一直启动。
4、上面说了一开机启动就会出现这个现象，那么还肯定是启动服务或者启动脚本里面写了什么代码，结果rc.local文件也正常。那么看/etc/init.d目录下的启动脚本，有没有新增的或者可疑的？果然发现了一个functions和DbSecuritySpt文件，我将这两个文件移走，然后故障依旧。看了一下DbSecuritySpt文件里面内容：

<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>#!/bin/bash</li><li>/usr/local/apache-tomcat-6.0.44/webapps/eei/gfty</li></ol>

初步一看这是一个很正常的脚本文件啊！一般病毒文件都是打不开的。问了我朋友说不是他们写的，那我只能将这个文件移走，然后也很二逼似的把那个functions文件也移走了，结果他们重启机器后，告诉我服务器起不来啦！截图如下：

一看上图的报错我心想肯定是那个functions文件移走报错了。不过还好这是个虚拟机，我远程连接宿主机上。
然后在上图界面输入root密码后，执行mount -o remount rw /后将functions文件移到/etc/init.d目录下重新启动。但是重启还是报错，说要检查文件系统块文件，又执行fsck -y /dev/sda后提示重启，重启后系统正常运行。没有tomcat那个进程了，但是还是偶尔往外拼命往外发包啊！
5、我又上网查了一下很多网友说是将/tmp目录下有一些的文件里面写了PID号，但是我根据这些PID号没有找到这些进程，我把这两个文件移走了。重启系统故障依旧。而且操作很卡真的很恶心，加上我自己的笔记本一上午关机7次，应该硬件老化的原因，比较2010年买的。哎！此处心中一万个草泥马飘过。
6、又咨询了朋友说iftop工具能看的出来，我试了也不行，爆卡，后来又是是iptraf工具，这两个工具都没安装，又花了很多安装时间，结果也看不出来啊！拿一个iptraf工具我们看看，如下图所示：

7、又看了一下chkconfig开机启动有没有异常的服务，一看服务太多了，也很难发现。
8、我在想是不是每次连续发送几个大包的时候是不是那个进程也会占用很高的CPU使用率呢？再一边观察流量脚本运行的情况，一边又执行top看看是不是哪个进程导致。有一个getty进程偶尔能跑到70%多，结果一查看是有6个终端，然后关闭了多余的终端，但是还是异常。也没发现别的进程占用很高的CPU使用率。
9、最后我想用netstat -an | more测试，一个一个排查当前服务器开放端口，因为我朋友说客户也不是很懂linux，开放了很多端口暴露在互联网上。发现了一个xxxx.51545->119.147.145.221:6001异常，然后我查看了一下这个51545端口对应的进程，如下所示：

This process executes the getty command, which indicates that the getty process above me occasionally reaches a usage rate of more than 70%, which is exactly consistent with what I checked. I tried to kill the 1587 process number, and after observing for a while, the script outflow traffic was basically 0, which is normal. That is to say, this server connects to port 6001 of the server 119.147.145.221 on the Internet through port 51545. It is found that the IP address belongs to Guangdong Telecom.
But the matter is not over yet. As mentioned above, it must be running in the boot program, and the destination address 119.147.145.221 must be hidden in the virus file. I went into the /etc/rc.d directory and took a look:

It seems that virus files have been placed at every startup level!

It’s all software! However, this file was removed in my fourth step, and then we also saw a suspicious selinux file, because the timestamps of it and the DbSecuritySpt file were different from other startup script files. This is a test of whether your eyes are sharp~~

Look at the stuff in the /usr/bin/bsd-port directory again!

Hurry up and delete the /etc/init.d/selinux file and the /usr/bin/bsd-port directory. Then restart and try again, the system will be fine! Network card traffic is also normal. Then I changed the root password, but unfortunately I couldn’t find out whether the code file was maliciously uploaded due to a system vulnerability or a program vulnerability.

Summary of troubleshooting:
1. The server must use a password as little as possible to log in, preferably a key plus a password.
2. Open fewer ports that have nothing to do with business.
3. When checking problems, you must use the elimination method. Eyes should be sharp.
4. I am still not very familiar with many scripts and processes in the system.
4. Although it has been solved, this machine has still been uploaded. I am not sure if there are still some unstable factors. It is recommended that it is best to reinstall the system.

http://www.bkjia.com/PHPjc/1109309.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/1109309.htmlTechArticleRemember the handling of traffic anomalies two days ago. I received a call from a friend who was doing development, saying that their client had a server powered on. Afterwards, the Internet access of all machines in a network segment became slow, and he remotely operated this...