java - Tomcat的Spring-MVC项目在重启一段时间后变卡

Question

最近更新了程序之后，发现网页在tomcat重启一阵子之后变得异常的卡。不知道为什么。发现了好多内存泄漏的警告，觉得是不是因为不正常的关闭导致内存不足呢，就试了几个方法。

最先试着把tomcat的context.xml里面设置缓存最大值，貌似设到了100000，启动后发现速度不错，但过了一段时间又卡得不得了了。
再之后把服务器的内存调大了，问题还是照样出现。而且每次系统的缓存只会越来越多，不会减少。

上网看了一些类似问题的回答，有人说是java获取数据库的效率不高而造成的，但是我觉得我这个问题应该不是出在这，因为有很多需要获取数据库的函数代码都变，以前就没有出现过这种问题，为什么现在变成这样呢？不过最近倒是有一些改动，例如添加了两个TimerTask。倒觉得应该问题不回出在这里吧。

package x.xx.xxx;

import java.util.Calendar;
import java.util.Date;
import java.util.List;
import java.util.Timer;
import java.util.TimerTask;

import javax.annotation.Resource;

import org.apache.log4j.Logger;

import x.xx.xxx.ManagementStation;
import x.xx.xxx.ManagementStationService;
/**
 * 源代码来自《java定时任务，每天定时执行任务》
 * http://www.cnblogs.com/cvst/articles/5818233.html
 *
 */

public class TimerManager {
    @Resource
    RemoteControlController remoteControlController;
    
    @Resource
    ManagementStationService managementStationService;

    @Resource
    ControllerStatusController controllerStatusController;

    // 时间间隔
    private static final long PERIOD_DAY = 24 * 60 * 60 * 1000;
    private static final int START_TIME = 1;

    public void initTimerManager() {
        Calendar calendar = Calendar.getInstance();

        /*** 定制每日1:00执行方法 ***/

        calendar.set(Calendar.HOUR_OF_DAY, START_TIME);
        calendar.set(Calendar.MINUTE, 0);
        calendar.set(Calendar.SECOND, 0);

        Date date = calendar.getTime(); // 第一次执行定时任务的时间
        Date date2 = calendar.getTime();

        // 如果第一次执行定时任务的时间 小于 当前的时间
        // 此时要在第一次执行定时任务的时间加一天，以便此任务在下个时间点执行。如果不加一天，任务会立即执行。
        if (date.before(new Date())) {
            date = this.addDay(date, 1);
        }

        Timer timer = new Timer();
        Timer timer2 = new Timer();

        UpdateDailyEletricPowerTimerTask task = new UpdateDailyEletricPowerTimerTask();
        // 安排指定的任务在指定的时间开始进行重复的固定延迟执行。
        timer.schedule(task, date, PERIOD_DAY);

        UpdateLampStatusTimerTask task2 = new UpdateLampStatusTimerTask();
        timer2.schedule(task2, date2, PERIOD_DAY);
    }

    // 增加或减少天数
    public Date addDay(Date date, int num) {
        Calendar startDT = Calendar.getInstance();
        startDT.setTime(date);
        startDT.add(Calendar.DAY_OF_MONTH, num);
        return startDT.getTime();
    }

    public class UpdateDailyEletricPowerTimerTask extends TimerTask {

         private Logger log = Logger.getLogger(UpdateLampStatusTimerTask.class);

        @Override
        public void run() {
            try {
                /**
                 * 查询前昨两天日冻结正向有功总电量
                 */
                Calendar now = Calendar.getInstance();
                int year = now.get(Calendar.YEAR) - 2000;
                int month = now.get(Calendar.MONTH) + 1;
                int day = now.get(Calendar.DATE);
                remoteControlController.dailyPositiveElectricPowerCollecting(year, month, day);

            } catch (Exception e) {
                log.info("-------------NFDFlightDataTimerTask解析信息发生异常--------------");
            }
        }
    }
    
    public class UpdateLampStatusTimerTask extends TimerTask {

         private Logger log = Logger.getLogger(UpdateLampStatusTimerTask.class);

        @Override
        public void run() {
            try {
                /**
                 * 更新全部灯具状态
                 */
                List<ManagementStation> mlist = managementStationService.getManagementStationList();
                StringBuffer temp=new StringBuffer("");
                for(ManagementStation m:mlist) temp.append(m.getMid()+",");
                String[] arr = temp.toString().split(",");
                boolean realTime = false;
                controllerStatusController.UpdateControllerStatus(arr,realTime);

            } catch (Exception e) {
                log.info("-------------UpdateLampStatusTimerTask解析信息发生异常--------------");
            }
        }
    }

不过系统变得死卡死卡的的时候其实是开启thread和进行socket通讯之后。
不过不知道问题出在前置机于客户端的通讯那，还是我这边的系统程序问题，下面是系统接收返回信息的线程，请各位大虾过目过目，谢谢。

private Status<OperationReturnMsgInfo> GetReturnedInfo(int frame_no) throws Exception{
        FrameController.addFrameToSendingQueue(frame_no);
        Status<OperationReturnMsgInfo> status=new Status<OperationReturnMsgInfo>();
        long beginTime = System.currentTimeMillis();
        boolean over_runtime = false;
        /**
         * 如果超过最大接收时间RUNTIME_MAX就直接把接收到的信息返回页面
         * 其他的报文继续在后台进行接收
         */
        while(FrameController.getExpectReutrnedFrameAmount(frame_no) != 0){
            Thread.sleep(1000);
            long nowTime = System.currentTimeMillis();
            over_runtime = nowTime - beginTime > RUNTIME_MAX? true : false;
            if(over_runtime) break;//超时跳出
        }
        List<byte[]> return_frames = FrameController.GetReturnedFrame(frame_no);

        List<OperationReturnMsgInfo> olist= new ArrayList<OperationReturnMsgInfo>();
        if(return_frames !=null){
            for(byte[] return_frame:return_frames){
                List<OperationReturnMsgInfo> olisttmp = parseFrontEndMsg(return_frame);
                if(olisttmp != null)
                    olist.addAll(olisttmp);
            }
            status.setCode(1);
        }else{
            status.setCode(0);
        }
        status.setList(olist);
        
        if(!over_runtime){
            FrameController.RecallFrameNo(frame_no);
        }else{
            /**
             * 后台处理超时没有返回的报文
             */
            final int tmp_frame_no = frame_no;
            new Thread(new Runnable(){

                @Override
                public void run() {
                    int frame_no_in_the_thread = tmp_frame_no;//FIXME 有点问题：如果tmp_frame_no在运行到这行时被其他线程改变了，怎么办？

                    long beginTime = System.currentTimeMillis();
                    boolean over_runtime = false;

                    int lastExpectReutrnedFrameAmount = FrameController.getExpectReutrnedFrameAmount(frame_no_in_the_thread);
                    while(FrameController.getExpectReutrnedFrameAmount(frame_no_in_the_thread) != 0){
                        try {
                            Thread.sleep(1000);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                        long nowTime = System.currentTimeMillis();
                        over_runtime = nowTime - beginTime > BG_RUNTIME_MAX? true : false;
                        if(over_runtime) {
                            //再超时便强制回收帧序号，虽然暂时和直接回收没有区别。
                            FrameController.CompulsivelyRecallFrameNo(frame_no_in_the_thread);
                            return;
                        }
                        /**
                         * 如果该帧返回报文有继续更新，重置beginTime
                         * 主要用于十分花时间的多报文返回操作RUNTIME_MAX的话就直接返回到页面。
                         * 相当于如果下一个报文返回间隔的时间大于RUNTIME_MAX就强行关闭对该帧的返回报文的接收
                         */
                        if(lastExpectReutrnedFrameAmount != FrameController.getExpectReutrnedFrameAmount(frame_no_in_the_thread)){
                            lastExpectReutrnedFrameAmount = FrameController.getExpectReutrnedFrameAmount(frame_no_in_the_thread);
                            beginTime = System.currentTimeMillis();
                        }
                    FrameController.RecallFrameNo(frame_no_in_the_thread);
                }
                
            }).start();
        }
        return status;
    }

Answer 1

The questioner can find some analysis tools to capture some useful information first.
StuckThe most likely reason for slow program execution is resource preemptioncpu.

First of all, the poster may use jstack pid to print out the thread stack and analyze how the thread mainly works, which line of code the program runs to, and what calculations are being performed? I’m going to check the corresponding code issues one by one

Ifjstack不够楼主还可以使用xrebel/yourkitsuch tools assist positioning

Answer 2

Looking at the description in the title, it should be caused by resource grabbing caused by memory leaks.
Finding memory leaks cannot be solved in one answer. It must be combined with specific code and situation analysis. Here are troubleshooting ideas for reference:

1. Configure Tomcat to leak and export heap files
   For details, please refer to how to configure tomcat to generate heapdump

2. Use tools to analyze headdump files and locate abnormal stack information.
   Common analysis tools include IBM HeapAnalyzer, jhat, and jmap. You can refer to using JMAP dump and analyzing dump files

3. Analyze whether there are exceptions in the creation and destruction of the exception stack in the code. If it is not destroyed normally, it will lead to leaks

It is recommended that the poster try to master the method to solve the problem, so that he can bypass the analogy when encountering such problems in the future