Table of Contents
记一次流量异常处理
Home php教程 php手册 记一次流量异常处理

记一次流量异常处理

Jun 13, 2016 am 08:44 AM
android

记一次流量异常处理

前两天接到一个做开发的朋友电话,说他们客户一台服务器开机后,所有一个网段的机器上网都变慢了,他远程操作这台服务器也一卡一卡的。
我第一反应就是机器被人攻击过了,因为我之前也遇到过类似的现象。大概都是tomcat管理密码设置的比较弱,被人上传了一些war包,导致服务器拼命往外发包,或者是被人恶意上传了一些php文件,也是往外发送大量的数据包。总而言之,往外发送大量数据包基本都是被人攻击过啦!下面看看我是怎么处理的。
1、首先我给他一个脚本,确认一下是网卡异常流量引起。注意网卡改成你的外网网卡名称。
<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>while : ; do</li><li>time=`date "+%Y-%m-%d %H:%M:%S"`</li><li>rx_before=`ifconfig eth0|sed -n "8"p|awk '{print $2}'|cut -c7-`</li><li>tx_before=`ifconfig eth0|sed -n "8"p|awk '{print $6}'|cut -c7-`</li><li>sleep 2</li><li>rx_after=`ifconfig eth0|sed -n "8"p|awk '{print $2}'|cut -c7-`</li><li>tx_after=`ifconfig eth0|sed -n "8"p|awk '{print $6}'|cut -c7-`</li><li>rx_result=$[(rx_after-rx_before)/256]</li><li>tx_result=$[(tx_after-tx_before)/256]</li><li>echo "$time Now_In_Speed: "$rx_result"kbps Now_OUt_Speed: "$tx_result"kbps"</li><li>sleep 2</li><li>done</li></ol>
Copy after login
然后运行这个脚本
<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>sh traffic.sh</li></ol>
Copy after login
执行之后我们会看到偶尔流出的流量惊人。
<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>2016-02-03 13:32:01 Now_In_Speed: 5kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:05 Now_In_Speed: 2kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:09 Now_In_Speed: 1kbps Now_OUt_Speed: 0kbps</li><li>2016-02-03 13:32:13 Now_In_Speed: 1kbps Now_OUt_Speed: 664567kbps</li><li>2016-02-03 13:32:17 Now_In_Speed: 6kbps Now_OUt_Speed: 657895kbps</li><li>2016-02-03 13:32:21 Now_In_Speed: 3kbps Now_OUt_Speed: 568462kbps</li><li>2016-02-03 13:32:25 Now_In_Speed: 4kbps Now_OUt_Speed: 0kbps</li></ol>
Copy after login
2、问题确定了,我们就好办了,上图我截图很少,而且我还发现了很有规律的事情,大概每二十多秒就会发出3-4个左右相当大的数据包。既然有规律那就肯定是后台有程序在运行。我查看了一下服务器是不是运行了tomcat?结果webapps目录下没有一些异常的jar包。我再查了一下是不是apache什么的,结果服务器上就只发现运行了oracle,根据自己的排查故障经验,我和朋友说了把oracle关闭。缩小故障查找范围,好确认不是oracle引起的。
3、在用ps -ef看了一下基本看不出,因为进程太多了,而且很多系统的进程我也不认识,没有看到什么异常进程。只有一个tomcat进程,kill之后一会又起来了,很奇怪。肯定是什么守护程序一直启动。
4、上面说了一开机启动就会出现这个现象,那么还肯定是启动服务或者启动脚本里面写了什么代码,结果rc.local文件也正常。那么看/etc/init.d目录下的启动脚本,有没有新增的或者可疑的?果然发现了一个functions和DbSecuritySpt文件,我将这两个文件移走,然后故障依旧。看了一下DbSecuritySpt文件里面内容:
<ol style="margin:0 1px 0 0px;padding-left:40px;" start="1" class="dp-css"><li>#!/bin/bash</li><li>/usr/local/apache-tomcat-6.0.44/webapps/eei/gfty</li></ol>
Copy after login
初步一看这是一个很正常的脚本文件啊!一般病毒文件都是打不开的。问了我朋友说不是他们写的,那我只能将这个文件移走,然后也很二逼似的把那个functions文件也移走了,结果他们重启机器后,告诉我服务器起不来啦!截图如下:

一看上图的报错我心想肯定是那个functions文件移走报错了。不过还好这是个虚拟机,我远程连接宿主机上。
然后在上图界面输入root密码后,执行mount -o remount rw /后将functions文件移到/etc/init.d目录下重新启动。但是重启还是报错,说要检查文件系统块文件,又执行fsck -y /dev/sda后提示重启,重启后系统正常运行。没有tomcat那个进程了,但是还是偶尔往外拼命往外发包啊!
5、我又上网查了一下很多网友说是将/tmp目录下有一些的文件里面写了PID号,但是我根据这些PID号没有找到这些进程,我把这两个文件移走了。重启系统故障依旧。而且操作很卡真的很恶心,加上我自己的笔记本一上午关机7次,应该硬件老化的原因,比较2010年买的。哎!此处心中一万个草泥马飘过。
6、又咨询了朋友说iftop工具能看的出来,我试了也不行,爆卡,后来又是是iptraf工具,这两个工具都没安装,又花了很多安装时间,结果也看不出来啊!拿一个iptraf工具我们看看,如下图所示:

7、又看了一下chkconfig开机启动有没有异常的服务,一看服务太多了,也很难发现。
8、我在想是不是每次连续发送几个大包的时候是不是那个进程也会占用很高的CPU使用率呢?再一边观察流量脚本运行的情况,一边又执行top看看是不是哪个进程导致。有一个getty进程偶尔能跑到70%多,结果一查看是有6个终端,然后关闭了多余的终端,但是还是异常。也没发现别的进程占用很高的CPU使用率。
9、最后我想用netstat -an | more测试,一个一个排查当前服务器开放端口,因为我朋友说客户也不是很懂linux,开放了很多端口暴露在互联网上。发现了一个xxxx.51545->119.147.145.221:6001异常,然后我查看了一下这个51545端口对应的进程,如下所示:

这个进程执行的正是getty命令,说明和我上面一个getty进程偶尔跑到70%多使用率,查的正好符合。我尝试将这个1587进程号kill掉,再观察一段时间脚本流出流量基本为0了,也就是正常了。也就是说这台服务器通过51545端口去连接互联网上的119.147.145.221这台服务器的6001端口,查看了这个IP地址是广东电信的。
但是事情还没有完结,上面说了肯定是开机启动程序里面运行的,而且这个目的119.147.145.221地址肯定是在病毒文件里面隐藏的。我进到/etc/rc.d目录下看了一下:

看样子每个启动级别都被人置放病毒文件啦!

都是软件连哈!不过这个文件被我第四步的时候移走了,然后我们还看到了一个可疑的selinux文件,因为它和DbSecuritySpt文件的时间戳和别的启动脚本文件不一样。这就测试你的眼睛尖不尖了哈~~

再看看这个/usr/bin/bsd-port目录下的东东哈!

赶紧删除/etc/init.d/selinux文件和/usr/bin/bsd-port目录。然后重启再试试看,系统一切正常!网卡流量也一切正常。然后修改root密码,但是很遗憾这里查不出是因为系统的漏洞还是程序的漏洞导致被恶意上传代码文件的。

故障处理总结:
1、服务器一定要尽少量的使用密码登录,最好是密钥加密码登录。
2、少开一些和业务无关的端口。
3、查问题一定要用排除法。眼睛要尖。
4、还是对系统很多脚本和进程不是很熟悉。
4、虽然解决了,但是这个机器还是被人上传过的,不确定是不是还会有一些不稳定的因素,建议最好还是重新安装系统。


Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

New report delivers damning assessment of rumoured Samsung Galaxy S25, Galaxy S25 Plus and Galaxy S25 Ultra camera upgrades New report delivers damning assessment of rumoured Samsung Galaxy S25, Galaxy S25 Plus and Galaxy S25 Ultra camera upgrades Sep 12, 2024 pm 12:23 PM

In recent days, Ice Universe has been steadily revealing details about the Galaxy S25 Ultra, which is widely believed to be Samsung's next flagship smartphone. Among other things, the leaker claimed that Samsung only plans to bring one camera upgrade

Samsung Galaxy S25 Ultra leaks in first render images with rumoured design changes revealed Samsung Galaxy S25 Ultra leaks in first render images with rumoured design changes revealed Sep 11, 2024 am 06:37 AM

OnLeaks has now partnered with Android Headlines to provide a first look at the Galaxy S25 Ultra, a few days after a failed attempt to generate upwards of $4,000 from his X (formerly Twitter) followers. For context, the render images embedded below h

IFA 2024 | TCL\'s NXTPAPER 14 won\'t match the Galaxy Tab S10 Ultra in performance, but it nearly matches it in size IFA 2024 | TCL\'s NXTPAPER 14 won\'t match the Galaxy Tab S10 Ultra in performance, but it nearly matches it in size Sep 07, 2024 am 06:35 AM

Alongside announcing two new smartphones, TCL has also announced a new Android tablet called the NXTPAPER 14, and its massive screen size is one of its selling points. The NXTPAPER 14 features version 3.0 of TCL's signature brand of matte LCD panels

New report delivers damning assessment of rumoured Samsung Galaxy S25, Galaxy S25 Plus and Galaxy S25 Ultra camera upgrades New report delivers damning assessment of rumoured Samsung Galaxy S25, Galaxy S25 Plus and Galaxy S25 Ultra camera upgrades Sep 12, 2024 pm 12:22 PM

In recent days, Ice Universe has been steadily revealing details about the Galaxy S25 Ultra, which is widely believed to be Samsung's next flagship smartphone. Among other things, the leaker claimed that Samsung only plans to bring one camera upgrade

Vivo Y300 Pro packs 6,500 mAh battery in a slim 7.69 mm body Vivo Y300 Pro packs 6,500 mAh battery in a slim 7.69 mm body Sep 07, 2024 am 06:39 AM

The Vivo Y300 Pro just got fully revealed, and it's one of the slimmest mid-range Android phones with a large battery. To be exact, the smartphone is only 7.69 mm thick but features a 6,500 mAh battery. This is the same capacity as the recently launc

Motorola Razr 50s shows itself as possible new budget foldable in early leak Motorola Razr 50s shows itself as possible new budget foldable in early leak Sep 07, 2024 am 09:35 AM

Motorola has released countless devices this year, although only two of them are foldables. For context, while most of the world has received the pair as the Razr 50 and Razr 50 Ultra, Motorola offers them in North America as the Razr 2024 and Razr 2

Xiaomi Redmi Note 14 Pro Plus arrives as first Qualcomm Snapdragon 7s Gen 3 smartphone with Light Hunter 800 camera Xiaomi Redmi Note 14 Pro Plus arrives as first Qualcomm Snapdragon 7s Gen 3 smartphone with Light Hunter 800 camera Sep 27, 2024 am 06:23 AM

The Redmi Note 14 Pro Plus is now official as a direct successor to last year'sRedmi Note 13 Pro Plus(curr. $375 on Amazon). As expected, the Redmi Note 14 Pro Plus heads up the Redmi Note 14 series alongside theRedmi Note 14and Redmi Note 14 Pro. Li

Samsung Galaxy S24 FE billed to launch for less than expected in four colours and two memory options Samsung Galaxy S24 FE billed to launch for less than expected in four colours and two memory options Sep 12, 2024 pm 09:21 PM

Samsung has not offered any hints yet about when it will update its Fan Edition (FE) smartphone series. As it stands, the Galaxy S23 FE remains the company's most recent edition, having been presented at the start of October 2023. However, plenty of

See all articles